(Tool) Stack Dump Checker

[Verstort]

View File

---

Summary:

 This is a python script that checks for stack dumps in papyrus logs.

 

Background:

If too many papyrus scripts exist in memory (the papyrus engine stack) all at once, the engine will dump some of its stack to garbage to prevent a stack overflow. Those dumped scripts are not retrieved nor revived, some will restart from scratch with all their state reset, but some will never restart on their own. Script dependencies can break, threads can break, state is lost and thus; Mods can break.

 

Once you get a single stack dump your save is broken. Maybe you can save clean the affected mods from your save to reset them with a save cleaner, but you have to clean every affected mod those scripts are attached to, and you lose settings/state/quests and everything saved to that mod anyway. I'm speaking theoretically, as I have never successfully fixed a save this way myself without realizing it wasn't worth it and stopping early.

 

The claim above that stack dumps can corrupt saves has been challenged in the support thread. See their counter argument post here:

 

Stack dumps are silent too, sometimes you get a brief pause as the game locks for maybe a full second as it dumps memory, but there is no other alert or notification to a user that a dump occurred. You just start getting really weird and specific bugs that would never happen otherwise, and are therefor impossible to find/fix for mod authors and yourself.

 

So I built this script to detect when a dump happens after every night of playing Skyrim. I save frequently enough that I can just delete any save I make after a crash dump to keep my save intact, and then I look at the stack dump time and scripts dumped to help me figure out what caused it to avoid it in the future.

 

I made this years ago, but didn't bother uploading it anywhere, and haven't seen another similar script or program anywhere, which is weird considering its a problem I've seen many users have. If something like this but better already exists feel free to tell me or post it in the support thread.

 

Instructions:

Move the script file to your skyrim logs folder, typically:

C:/Users/*your user name*/Documents/My Games/Skyrim/Logs/Script/ <- there should be papyrus logs here

From there, assuming you have python installed, you can just double click the script and it should search all the logs in that folder for stack dumps and print to a small text window. That's it.

 

Explaining the details printed about each stack dump:

The file is listed, the line count so you know how big the file at a glance.

Each file can contain multiple dumps, counted here

The line in the file where the dump starts, making it easier to find/search if you want to look at the actual file later

Time is printed per dump as it was printed in the file, most likely uses the same clock as your operating system. This can help you figure out when the dump occurred if the dump contents aren't enough info.

-----------------------------------------------------------

Then the scripts that were dumped from the stack are listed in decending order of frequency.

- the left number is the count of how many scripts with the following name were found, if its 10 then there were 10 scripts like it in the dump with the same name.

- if a dash is shown instead, that means the script next to the dash is part of a mini stack with the script above it, showing you what scripts called which subscripts.

the hex of the object the script is attached to is printed next, if its 0x0000014, that's the player, the script was attached to the player's actor, otherwise the hex will often tell you which mod a quest (the script is attached to) belongs to. The first two characters are in base 16 the load order for that mod.

then finally the name of the script and the functions being run, on the far right are shown

 

 

Maybe one day I'm modify this so that it reads the main papyrus script in realtime and will notify you while you play the game, but for now it works well enough for me. Once I started using it, and reverted to an early save before the first stack dump, I can just check every night and that's frequent enough to stop it being an issue.

 

Permissions:

Unlicensed open source, its a really basic script do what you want with its code

 

Thanks to:

@bicobus for pointing out some of the awful things I do when I'm in a hurry to bang out a quick script so I could fix them

---

• Submitter
  Verstort
• Submitted
  02/14/2019
• Category
  Other
• Requires
  Python 2.X or 3.X
• Special Edition Compatible

---

View File

 

[2Dimm]

why is it "Not Applicable" to Special Edition?

[Verstort]

I haven't actually played with SE all that much, I'm assuming the script works fine with logs generated by SE, although I haven't tested. By all means try it out and tell me if it's not working, but I don't feel comfortable changing it to Yes until I test it.

[El_Duderino]

Thanks very much for sharing your work with the community!

 

I'm a player who regularly uses heavy SL+DD builds on top of a heavily modded game for long playthroughs, so I do get the occasional dump when my check my logs after playing. This script will help making the attempt to analyze the dump a lot less tedious.

[worik]

That sounds very convenient for me.

A question: Does logging strictly need to be enabled when papyrus "stack dumps" or will it create logs in these situations regardless of your ini setting? ?

 

I ask, because I usually have logging switched off. It adds more strain to a running skyrim, so I only switch on when I have suspicions....

[GenioMaestro]

18 hours ago, Verstort said:

Those dumped scripts are not retrieved nor revived, some will restart from scratch with all their state reset, but some will never restart on their own. Script dependencies can break, threads can break, state is lost and thus; Mods can break.

 

Once you get a single stack dump your save is broken. Maybe you can save clean the affected mods from your save to reset them with a save cleaner, but you have to clean every affected mod those scripts are attached to, and you lose settings/state/quests and everything saved to that mod anyway. I'm speaking theoretically, as I have never successfully fixed a save this way myself without realizing it wasn't worth it and stopping early.

But what are you saying?

 

Please, do consistent and repeatable tests before saying something like that.

Your tool can help you see the Stack Dumps in the log, but nothing more.

 

Take a look at my ScriptTest and read the blog, especially chapter 3.

Download my ScriptTest and start doing tests to convince yourself.

 

The game is NEVER corrupted by a Stack Dumps or by hundreds of Stack Dumps. NEVER.

 

First, because the log is disabled by default and no log file is generated, so that the Stack Dump can not be seen.
Second, because if the Stack Dump was a problem for the game in certain circumstances, simply the game would not Stack Dumps or would alert it in a very noticeable way. It would be stupid to make Stack Dumps and break the game completely silently without giving any notice of any kind.
Third, if REALLY the game is broken every time there is a Stack Dump I am sure that STEP, NEXUS and thousands of users would say it VERY clearly and we would have many tools for locate the Stack Dumps and alert about the danger.
Fourth, mod programmers can not know in any way or form WHEN a Stack Dump is generated. If REALLY the game is broken every time there is a Stack Dump, programmers should be able to detect it in order to control it and be able to re-launch the scripts.

When the game make a Stack Dump simply list the stack of calls to the log. Do nothing more. Is a simple Warning.

The game can stop a script at any time. Really, the game launches and stops ALL the scripts in each frame, because the scripts ONLY run for 1.2 milliseconds in each frame.
That means that when the Stack Dump occurs, ALL the scripts are suspended.

The Stack Dump only reports that there are too many scripts running at the same times and nothing more.

 

Take my ScriptTest and start doing consistent checks. My test can launch 3200 simultaneous events and that can generate Stack Dump very easily, in fact generate a lot, and NOTHING happens. I launch the event one time and ONLY one time and my 3200 events are processed, each and every one of them, and a lot of times the game record Stack Dump in midle of the process, and the game NEVER loses even one of them and NEVER fails to execute a single line of code.

 

The scripts are compiled and converted into opcode. You can open any saved game with ReSaver and look at the Active Scripts to see the opcode lines.
In each frame the game can execute thousands of opcode lines and when the time of 1.2 milliseconds ends, the script is suspended. In the next frame, the script is restarted EXACTLY on that line and the game NEVER loses a single opcode line. And it does it in every frame, at 60 FPS it does it 60 times per second. At 60 FPS the Scripts are ONLY executed for 72 milliseconds of every second.

 

All this means, and I repeat, that the game is NEVER corrupted by a Stack Dumps or by hundreds of Stack Dumps. NEVER.

Because when the game write the Stack Dump ALL the script are in SUSPEND state.

After write the Stack Dump ALL the script are RESUMED in the next opcode line.

 

The only REAL limit in the call stack of the Script Engine is 32768 pending calls = Active Scripts.

When the game have exactly 32768 pending calls = Active Scripts made CTD.

But before that, you can have a Stack Dump with 32767 references whitout any problem.

And the Script Engine can manage ALL the 32767 calls and can execute all of them, one after another, and NEVER lost one pending call. 

While developing my ScriptTest I get some savegames with 28k and 30k Active Scripts reported in ReSaver and the game load it and execute it whitout any problem. And i can stop it and lower the Active Scripts count to below 100, of course, with a lot of gigantic Stack Dump in the log.

But the game is NEVER corrupted for the Stack Dumps.

 

------------------------------

 

The game can be corrupted when an error occurs in a memory variable and ONLY in that case the scripts can start to give problems.
This can happen when using PapyrusUtil and/or Jcontainers because these dll's are not perfect and sometimes they do not store/return all the information. Or, when the game is recorded and read, SKSE does not store and correctly retrieve all the information.

 

An example of this is in the own SexLab that uses PapyrusUtil. SexLab stores experience statistics for each npc and sometimes those statistics are lost. I always have the same followers and sometimes I look at those statistics and sometimes my followers lost their stats. And SexLab has no way of knowing it because it simply looks if the npc is in the matrix and if it is not assumed to be a new npc without statistics and adds it. But what really happens is that, sometimes, PapyrusUtil loses some of that information.

 

In PAHE + HSH something similar happens with Jcontainers. Sometimes one of the slaves disappears and it is for the same reason. Because a hole is produced in the matrix. And if you install Sexual Fame you will see a message that says exactly THAT: you can LOSE the data with the statistics.

 

The corruption problem can be generated whitout mods and whitout PapyrusUtil /Jcontainers, in the pure base game, because when save and load manage a lot of data, can be up to 5 MB of data, and, sometimes, not ALL things are correctly stored/retrieved.

 

The only solution to the corruption is discard that bad savegame and load a stable save.

Follow the golden rule: If the game make strange things and not works as expected NEVER save the game. Load another save and repeat.

 

So do not look where there is not and, please, correct your message.

[GenioMaestro]

1 hour ago, worik said:

That sounds very convenient for me.

A question: Does logging strictly need to be enabled when papyrus "stack dumps" or will it create logs in these situations regardless of your ini setting? ?

 

I ask, because I usually have logging switched off. It adds more strain to a running skyrim, so I only switch on when I have suspicions....

The log is disabled by defaut and you need enable it for record the Stack Dump.

Enable log or/and debug not affect the performance of the game or alter the frame rate in any way.

Take a look to the blog in the chapter 2.

[worik]

38 minutes ago, GenioMaestro said:

The log is disabled by defaut and you need enable it for record the Stack Dump.

Enable log or/and debug not affect the performance of the game or alter the frame rate in any way.

Take a look to the blog in the chapter 2.

Thank you

[Verstort]

7 hours ago, GenioMaestro said:

The game is NEVER corrupted by a Stack Dumps or by hundreds of Stack Dumps. NEVER.

 

Please, do consistent and repeatable tests before saying something like that.

Interesting. I haven't done any direct test and I'll check out your work tonight, although I am skeptical that all the memory corruption I see after a stack dump is only save/load inaccuracies, that would be even more alarming honestly, and not intuitive with my anecdotal experience.

 

7 hours ago, GenioMaestro said:

Your tool can help you see the Stack Dumps in the log, but nothing more.

 ...

Follow the golden rule: If the game make strange things and not works as expected NEVER save the game. Load another save and repeat.

I never claimed otherwise. I only ever used it as a warning system and that's what I recommended its use as.

[Bane Master]

Well, I'm not qualified to get into a a technical discussion about how stack dumps do or don't do whatever they do - but I have to say that when they have occurred in my games  various quests & mods start failing or behaving strangely contemporaneously with the dump occurring. 

 

As a result I personally would roll back a save to prior to the dump and fix the issue rather than play on.

[GenioMaestro]

1 hour ago, Verstort said:

Interesting. I haven't done any direct test and I'll check out your work tonight, although I am skeptical that all the memory corruption I see after a stack dump is only save/load inaccuracies, that would be even more alarming honestly, and not intuitive with my anecdotal experience.

Yes, not doubt it. All the corruption problems in the game are caused exclusively when save or load.

Sometimes a mod can have problems and made strange things but is limited to that mod.

Maybe you not have a good load order or have some faulting compatility patch.

Leaving that things apart the game works perfect, at least for me. Take a look to my savegame.

Spoiler

I'm playing the same match for 5 months, i'm level 108, saved the game more than 2k times and my savegame 2046 have 27.7 MB and near cero problems. I have 199 plugins now and near 240 mods. 

I change from NMM to Vortex near 2 weeks ago and, of course, i use the same save game.

As you can understand, is imposible that i can have exactly the same load order for 5 months and, of course, change my load order when change the mod manager, but that is not a problem, is another myth.

 

You can change your load order when you want because the game is prepared for made it.

You can add and remove any mod in any moment BUT follow the un-instalation instructions.

For example, i add Rigmor of Bruma and Rigmor of Cyrodil to that savegame near 2 months after start it and play it completelly and remove it and clean my save with ReSaver and nothing happend.

My game and my savegame are perfectly fine.

 

Another myth is that you MUST clean yours masters but i NEVER made it.

Clean the masters was a obligatory operation in the years 2012 and 2013 because Beth modify the game and delete some records and any mod with reference to deleted records made CTD and you must clean the masters and undelete that references.

But the game not change in 5 years and all the mods with reference to deleted records has been patched and today is totally unnecesay clean the masters, is another myth.

[worik]

12 hours ago, GenioMaestro said:

Yes, not doubt it. All the corruption problems in the game are caused exclusively when save or load.

That's interesting ?

What I always thought was: Once the save game is corrupt all following are corrupt as well and there is just the way back till the last coherent save.

 

Now, your observations sound like:

• I can make a save at 15 minutes playtime and somehow it is corrupt (but I won't know it right now, because I keep playing. I will learn later, when I CTD or try to load from that)
• I keep on playing, I make another save at 30 minutes playtime and it stands a chance that it is NOT corrupt?

 

Edit: for the record

I play a game that is 13 month old in real time, L38 (because I tuned the leveling seriously down), 300+ real mods in ~130 visible+merged esps, updated mods, shifted them between merges, 1500-something saves. ~29MB, problem-free.

But I guess it works because I am probably paranoid to keep it clean and tidy

[GenioMaestro]

4 hours ago, worik said:

That's interesting ?

What I always thought was: Once the save game is corrupt all following are corrupt as well and there is just the way back till the last coherent save.

 

Now, your observations sound like:

• I can make a save at 15 minutes playtime and somehow it is corrupt (but I won't know it right now, because I keep playing. I will learn later, when I CTD or try to load from that)
• I keep on playing, I make another save at 30 minutes playtime and it stands a chance that it is NOT corrupt?

Open the spoiler for read more info.

Spoiler

Any database, like Oracle or SQL Server, and any profesional program like like PS or 3dStudio, have a total and asurely reability of 100% when save and load. NEVER can have a problem loading or writing a file. NEVER.

 

But when we add plugins to PS or 3dStudio, if  these plugins are free and not are totally profesional and guaraticed, we can start with problems because can have a fail. The reability when load and writing files MUST BE 100% but not always is 100% and some times is enougth with 99% or 99.5% or 99.9% because the developers, normally, only solve evident problems that can be repeated.

If a specific plugins have a specific problem that only happend in very strange circunstances to a very small group of users in random situations that can not be repeated is near imposible solve the problem.

 

The same happend when talk about Skyrim. The base game store their file, SKSE store their file and normally Crash Fixes intercep the process of store and load. Where is the problem??? When happend the problem??? WHY happend???? Can you repeat it for reproduce it??? If not, the developers can NOT debug it and can NOT solve the problem.

 

I think Skyrim have a 100% sure guaranty when load a savegame. If the savegame have that specific info the game load it perfectly. Is easy debug it because you can examine the save and look if all the thinks stored in the savegame are in memory. Really, i NEVER have a diferent result when load the SAME savegame.

 

But when store the savegame the things change and probably only have near a 99.9% of guaranty.

Because you can save the game in ANY moment and not ALL moments are good for save the game.

Maybe, the reability, can down to 99.5% depending of the exact moment and depending about the things that you have in the screen and around you. Maybe, the size of the savegame, the mods instaled and the things that made each mod can lower that reability.

 

I think that ALL the corruption problems in Skyrim are generated when SAVE the game.

If you have problems when LOAD a specific save is because THAT savegame is corrupt.

 

Of course, you can have a BAD savegame and you NEVER notice it because you NEVER load THAT save.

Normally, I save a lot and only load the last save while I make a lot of intermediate saves and not load all of them.

Is imposible know if any of my intermediate saves are corrupt until i load it and play with it.

And when i have problems with a save i simply delete it and load another.

 

If you load a BAD savegame and save it again, of course, the new save can be corrupt.

For that exist the golden rule: If the game make strange things and not works as expected NEVER save the game. Load another save and repeat.

[galgat]

  I agree scaring people with that golden rule is over kill, nobody always or "never do this" or "never do that", we all reach a point where we remove a mod, add a mod, and or clean our saves. 

 

That of course is a good reason to make a safe save before adding a new mod. Also a good reason to have multiple profiles, and keep one profile completely safe, and never add or subtract from it for a main game play through, and from it you can grab a clean save for another Profiles that you wish to test a mod in.

[no_way]

Tried using but it opens the command prompt and then immediately closes, any tips to get it working?