Corrupted Saves

[Centrophy]

 

 

So I'm looking through a savefile with tesvesse and I'm searching for strcount. Can't seem to locate it. Did you guys find that out with that tool or did you just use a hexeditor and the uesp info to find it? I'm trying to see how close I am to the FF FF limit on a new mod setup.

 

 

Expand "Global Data table 3" and click on Papyrus. You don't need to expand it. strcount is "Strings", the first entry. 

 

 

Since strcount is a (presumably unsigned) 16 bit integer field, valid values for it are 0 through 65,535. As you can see that save is some way off becoming corrupt but it's not clear just yet what sort of "cushion" you would need to have a full playthrough...

A few posts back vyper said he starts new games with a strcount of 55,000 and hits the limit at about 100 hours of play time. It likely depends largely on the mods as well just how quickly it grows. 

 

 

Thanks. I'm at 63, 437 in only an hour in. It's easy to see why it gets corrupted so easily with SexLab mods. Every animation has an entry, creature anims seem to have multiple "duplicates." Example: Daedra_Hugging A1_S1 - Daedra_Hugging A2_S4. That's 8 entries just for one animation. Estrus has loads of entries as well. If you use RaceMenu + Bodyslide, every slider has an entry for bodytype and tint mask. I hope we can get some more attention focused on this problem so we can raise that limit.

[Monoman1]

They're not duplicates. They're the different stages of the animation. Different animations can have different number of stages. Some might have 4, some might have 5 or 6. 

Animations and complex MCMs are two things that seem to have a big impact. Anything ending in OID is likely an MCM component but nothing says it HAS to end in OID to be a MCM component. It's just a convention. So OIDs are just obvious MCM components. A single MCM slider requires the definition of several variables.

 

Is this also a question of persistence? I vaguely remember reading somewhere that MCMs are persistent so obviously the variables are persistent. Could the variables be made non-persistent? Dump them to an external storage and release them from the strcount pool... ?

They're not something that changes very often anyway....

 

Or could arrays be used instead? 

[sheson]

They're not duplicates. They're the different stages of the animation. Different animations can have different number of stages. Some might have 4, some might have 5 or 6. 

Animations and complex MCMs are two things that seem to have a big impact. Anything ending in OID is likely an MCM component but nothing says it HAS to end in OID to be a MCM component. It's just a convention. So OIDs are just obvious MCM components. A single MCM slider requires the definition of several variables.

 

Is this also a question of persistence? I vaguely remember reading somewhere that MCMs are persistent so obviously the variables are persistent. Could the variables be made non-persistent? Dump them to an external storage and release them from the strcount pool... ?

They're not something that changes very often anyway....

 

Or could arrays be used instead? 

 

In this context the term "string" covers anything "text". For example, it includes the script names (without .pex), the names of variables regardless of the type of variable and naturally the values of string variables or string arrays. It is only the last two which tend to use them all up a lot. As far as I can tell it only saves strings that are currently in use one way or the other.

 

Further testing should be done to see if setting a string variable to "" 'releases' whatever the value was before, if it is not needed anymore.

Because of persistent data, experienced papyrus scripters also know to 'release' ObjectReferences by setting them to None when they are not needed anymore. It may or may no work.

 

In cases where lots of strings need to be saved for later use I can suggest tools like PapyrusUtil. Works for DynDOLOD

[Monoman1]

Jeez, script names too? That's a hefty chunk of that field gone straight away. 7000-ish for me Honestly, whoever made the call to make it 16 bit ought to be loaded into a cannon and fired towards the sun. 

 

Edit: That actually goes a way to explaining why captured dreams and maria eden have a big impact on this issue. CD has over 800 scripts, ME - 600. I'm sure there's a variable or 2 there as well but I'd imagine most of them are not 'active'. 

Don't get me wrong though, they're both fantastic mods. Not the authors fault skyrim is crapping out. 

[LukeDuke]

What's Skyrim needs it's a x64 bit update, so there would be no need for ENBoost/SKSE memory patches and also it would use more CPU cores and memory resources like Fallout 4..

But I don't think Bethesda will make it, maybe at later time with ''HD'' revision for PS4/XOne/PC.

 

P.S. I'm so glad that I found this thread, now I know why my saves was corrupted so many times! 

[markdf]

What's Skyrim needs it's a x64 bit update, so there would be no need for ENBoost/SKSE memory patches and also it would use more CPU cores and memory resources like Fallout 4..

But I don't think Bethesda will make it, maybe at later time with ''HD'' revision for PS4/XOne/PC.

 

P.S. I'm so glad that I found this thread, now I know why my saves was corrupted so many times! 

 

Yeah, Bethesda isn't going to do that. Even if they make a new release, just think about what your mod list would look like if it were a hypothetical XBox One special edition of Skyrim. Just the vanilla game with DLCs, bug fixes, patches, and graphical enhancements. No extra quest mods, no overhauls, etc. That's all just PC stuff. The 16bit strCount field is perfectly fine for all possible console needs, which is probably why it never occurred to Bethesda that this would be an issue.

 

If you've tried installing Morrowind, you're probably familiar with all the patchers for the executable; they fix bugs in the code, change memory usage, optimize certain sections of code, and so forth. They actually edit the EXE file. Unless the SKSE team can pull a rabbit out of their hat and fix this in SKSE, a patcher like that is the only solution.

 

That, or just uninstall some mods. :-)

[Centrophy]

 

What's Skyrim needs it's a x64 bit update, so there would be no need for ENBoost/SKSE memory patches and also it would use more CPU cores and memory resources like Fallout 4..

But I don't think Bethesda will make it, maybe at later time with ''HD'' revision for PS4/XOne/PC.

 

P.S. I'm so glad that I found this thread, now I know why my saves was corrupted so many times! 

 

Yeah, Bethesda isn't going to do that. Even if they make a new release, just think about what your mod list would look like if it were a hypothetical XBox One special edition of Skyrim. Just the vanilla game with DLCs, bug fixes, patches, and graphical enhancements. No extra quest mods, no overhauls, etc. That's all just PC stuff. The 16bit strCount field is perfectly fine for all possible console needs, which is probably why it never occurred to Bethesda that this would be an issue.

 

If you've tried installing Morrowind, you're probably familiar with all the patchers for the executable; they fix bugs in the code, change memory usage, optimize certain sections of code, and so forth. They actually edit the EXE file. Unless the SKSE team can pull a rabbit out of their hat and fix this in SKSE, a patcher like that is the only solution.

 

That, or just uninstall some mods. :-)

 

 

Even with the patchers it still didn't fix many issues with the Morrowind engine and some heavy mods. Hopefully the SKSE team isn't retired from messing with Skyrim now that FO4 is out. Did they ever get back to you?

[sheson]

Jeez, script names too? That's a hefty chunk of that field gone straight away. 7000-ish for me Honestly, whoever made the call to make it 16 bit ought to be loaded into a cannon and fired towards the sun. 

 

Edit: That actually goes a way to explaining why captured dreams and maria eden have a big impact on this issue. CD has over 800 scripts, ME - 600. I'm sure there's a variable or 2 there as well but I'd imagine most of them are not 'active'. 

Don't get me wrong though, they're both fantastic mods. Not the authors fault skyrim is crapping out. 

 

Just Vanilla, DLC and USLEEP right from start uses over 28,000 strings. Yes 16 bit counter was incredibly short sighted. Especially since every other thing in the save is basically 32 bit, more or less.

 

We are just starting to get the word out about this. It will take some time to trickle down the channels. In the coming weeks we will hopefully lean a few things about how to write scripts that work around the issue. I am going to check a few ideas along the way while stress testing my new beta scripts I posted earlier.

 

I think it should be possible to read the save data and count which scripts specifically uses how many strings. That would help identifying candidates that would benefit from a work around. Seems like a good job for the save game tool guys to look into.

[crusher]

.. maybe use a virtual save structure (similar to MO's virtual file structure), to pack everything into Papyrus Utility (32 bit). I'm probably way outside the box..

[markdf]

Even with the patchers it still didn't fix many issues with the Morrowind engine and some heavy mods. Hopefully the SKSE team isn't retired from messing with Skyrim now that FO4 is out. Did they ever get back to you?

 

They did. Their concern was that the strCount field may be 16 bits in the engine itself, which would mean that a lot of variables and function calls would need to be changed, not just one field of the save file format.

 

For the record, I disagree. I think the evidence suggests that the string table size is 32 bits internally, and that the whole table is being successfully written -- just with a truncated strCount. But they're the experts and I'm not, so take my opinion with a grain of salt.

 

We are just starting to get the word out about this. It will take some time to trickle down the channels. In the coming weeks we will hopefully lean a few things about how to write scripts that work around the issue. I am going to check a few ideas along the way while stress testing my new beta scripts I posted earlier.

 

I think the starting point for most mods will be trimming down the MCMs, and putting as many options as possible into INI files or JSON files (using PapyrusUtil, JContainers, or JaxonzConsolePlugin).

 

The biggest difference will be finding ways to reduce numbers of scripts. Interesting NPCs uses 3400 strings before even leaving qasmoke. It has 6800 scripts, most of which have at least one property. There could potentially be HUGE gains to be made there.

 

Like all of those auto-generated dialogue scripts that do nothing except advance a quest stage. Isn't there any other way to do that? Like without a script? I don't mess with the dialogue system much, but I feel like we could get rid of some of those scripts.

[Centrophy]

 

Even with the patchers it still didn't fix many issues with the Morrowind engine and some heavy mods. Hopefully the SKSE team isn't retired from messing with Skyrim now that FO4 is out. Did they ever get back to you?

 

They did. Their concern was that the strCount field may be 16 bits in the engine itself, which would mean that a lot of variables and function calls would need to be changed, not just one field of the save file format.

 

For the record, I disagree. I think the evidence suggests that the string table size is 32 bits internally, and that the whole table is being successfully written -- just with a truncated strCount. But they're the experts and I'm not, so take my opinion with a grain of salt.

 

We are just starting to get the word out about this. It will take some time to trickle down the channels. In the coming weeks we will hopefully lean a few things about how to write scripts that work around the issue. I am going to check a few ideas along the way while stress testing my new beta scripts I posted earlier.

 

I think the starting point for most mods will be trimming down the MCMs, and putting as many options as possible into INI files or JSON files (using PapyrusUtil, JContainers, or JaxonzConsolePlugin).

 

The biggest difference will be finding ways to reduce numbers of scripts. Interesting NPCs uses 3400 strings before even leaving qasmoke. It has 6800 scripts, most of which have at least one property. There could potentially be HUGE gains to be made there.

 

Like all of those auto-generated dialogue scripts that do nothing except advance a quest stage. Isn't there any other way to do that? Like without a script? I don't mess with the dialogue system much, but I feel like we could get rid of some of those scripts.

 

 

Hopefully they'll look into it and maybe release a beta for trials. Did they mention if they were going to look into it or not? I like the json idea in the meantime but I'm pretty ignorant when it comes to that.

[BigOnes69]

As per earlier posts I just upgraded my Video card to an ASUS GTX970 Strix and my current character is not suffering from the corrupted saves papyrus overload at this time. I will continue to play to see where not when the corruption starts to occur again. I will also reload a couple of past characters that were consistently suffering from the save corruption and Papyrus overload too see if it changes the bar. This supports what I was saying before and the tweaks that can be made to raise the bar for longer characters lives before the corruption. Hopefully as you say they will be willing to put the time into an older game to give some more life for I dont think they will overcome the limitation entirely.

[sheson]

 

They did. Their concern was that the strCount field may be 16 bits in the engine itself, which would mean that a lot of variables and function calls would need to be changed, not just one field of the save file format.

 

For the record, I disagree. I think the evidence suggests that the string table size is 32 bits internally, and that the whole table is being successfully written -- just with a truncated strCount. But they're the experts and I'm not, so take my opinion with a grain of salt.

 

I would expect the game to CTD if string pointers start to wrap and point at the wrong strings at run time. Evidently the game is fine at run time as far as I can tell from the posts.

[Centrophy]

 

 

They did. Their concern was that the strCount field may be 16 bits in the engine itself, which would mean that a lot of variables and function calls would need to be changed, not just one field of the save file format.

 

For the record, I disagree. I think the evidence suggests that the string table size is 32 bits internally, and that the whole table is being successfully written -- just with a truncated strCount. But they're the experts and I'm not, so take my opinion with a grain of salt.

 

I would expect the game to CTD if string pointers start to wrap and point at the wrong strings at run time. Evidently the game is fine at run time as far as I can tell from the posts.

 

 

Not necessarily. Perhaps in game the number of strings can go as high as needed but the actual save file itself can only hold 16 bits worth.

[Monoman1]

 

 

They did. Their concern was that the strCount field may be 16 bits in the engine itself, which would mean that a lot of variables and function calls would need to be changed, not just one field of the save file format.

 

For the record, I disagree. I think the evidence suggests that the string table size is 32 bits internally, and that the whole table is being successfully written -- just with a truncated strCount. But they're the experts and I'm not, so take my opinion with a grain of salt.

 

I would expect the game to CTD if string pointers start to wrap and point at the wrong strings at run time. Evidently the game is fine at run time as far as I can tell from the posts.

 

Game ctds when loading these saves but not while the game is already running. ie: load good save. Corruption occurs at some point. Create new bad save. Game continues running. So it seems the engine is capable but the save file format is not. Like you say. If the pointer had wrapped around  to 00 while in game I'd imagine CTD or at the very least, some serious weirdness would start to occur. Which in my experience it doesn't 

[Nepro]

Any tips for what in general causes save corruption?

 

Flexcreator followed up with me over on the nexus forums. He believes that it is an issue with Skyrim's underlying engine; it writes the header incorrectly when the amount of data to be written gets too high. I'm inclined to agree. The only consistent thing about the crashes is that they seem to start when there is a lot of data to write, be it scripts or script instances or MCM data or forms.

 

 

So to recap this issue:
1) The headers of saves are being corrupted in a way that is unrecoverable.
2) It only seem to occur after Sexlab registers animations.
--2a) It is more likely to occur when Sexlab has registered many animations.
--2b) It is certain to occur if DynDoLod is installed alongside Sexlab.
--2c) It is more likely to occur when many scripted mods are installed, regardless of whether those scripts are running or not. For example, removing Wet and Cold doesn't help, but removing Sisters of Dibella or Captured Dreams does help. It got really bad for me when I installed Falskaar and Wyrmtooth, even in a brand new fame where I hadn't left the Alternate Start dungon.
3) Rebuilding Sexlab sometimes solves the problem, but it returns as soon as more Sexlab animations are registered -- such as enabling creature animations.

This all paints a confusing picture. Registering an animation doesn't create additional scripts or script instances, and it uses a trivial amount of memory (as I understand it, anyway). All it does is allocate a little bit of memory in a standard C++ library down in SKSE, and create a few entries in the MCM.

I would literally kill for a chance to debug this issue properly, with Bethesda's source code in front of me and an IDE that let me step through the relevant code. :-)

 

Does adding mods few by few help? Btw you were talking about no limit on ESMs. Wrye Bash can make any esp an esm plugin.

[Monoman1]

Not really. Have a read of the thread from post (Edit: sorry, 74) and onward.

Doesn't matter if you add 100 mods in one go and get corruption. If you add the same 100 mods one by one you will still get bad saves. You're more likely to find at what point saves become bad but since the field we're talking about grows as the games go on, if you were to add mods right up to the point before when your saves become bad they will become bad after some time of playing unfortunately. 

[Earen]

BSFixedString (its SKSE implementation) doesn't release its strings. It might be not an issue, IF the counter doesn't get saved. I remember I was asking the team about this, http://forums.bethsoft.com/topic/1512976-wipz-skyrim-script-extender-skse/page-7?do=findComment&comment=23931061 .  That's why I began use modified one

 

Plain english version: currently SKSE and most of plugins don't release strings, each time  a function implemented by SKSE or some plugin returns new string ( aka BSFixedString("some-new-string") ) the counter gets increased, but it won't be decreased later. Since most of plugins don't introduce new strings it's not an issue (?).  IMO, amount of strings just reflects how much huge your setup is.

[Para]

I have the problem with this kind of corrupted saves from time to time too. It was really bad as I tried to install estrus and isle of mara. But it happens during scripted game events like battle of whiterun, too.

I'm also not able to make an working save during the first battle with alduin. I'm uninstalling some sexlab mods now trying to get a stable game for the battle.

[BlackArk]

AHH! so this was my problem. I found out it was DynDOLOD and removed it way last year but I did not know why it would cause it and now it is captured dreams which I just installed recently. 

 

Removed a few mods and I am at a safer 63k rather than the almost to the limit 65k Then went back up to 64k >.<

 

I removed captured dreams and swapped out dawnguard sentry plus with the no script no mcm menu version.

 

Need to find other things I can remove.

 

I am considering removing nsap

 

And that also explains the SLAL save corruption issue that some people had I guess.

[Guest]

BSFixedString (its SKSE implementation) doesn't release its strings. It might be not an issue, IF the counter doesn't get saved. I remember I was asking the team about this, http://forums.bethsoft.com/topic/1512976-wipz-skyrim-script-extender-skse/page-7?do=findComment&comment=23931061 .  That's why I began use modified one

 

Plain english version: currently SKSE and most of plugins don't release strings, each time  a function implemented by SKSE or some plugin returns new string ( aka BSFixedString("some-new-string") ) the counter gets increased, but it won't be decreased later. Since most of plugins don't introduce new strings it's not an issue (?).  IMO, amount of strings just reflects how much huge your setup is.

 

Actually pretty much all mods that have some scripts inside will introduce new string.

 

Not a big problem, but you can find strings also from mods that are uninstalled.

 

Check with a simple mod thatt add a papyrus script with

String test = "ThIs Is A New String 0192837465"

 

Load the mod, be sure the script is run, save, uninstall the mod.

 

Then do another mod with"

 

String whallabahalla = "this IS a NEW string 0192837465"

 

 

And check the result.

You will get the capitalization of the previous mod.

[Veladarius]

 

The biggest difference will be finding ways to reduce numbers of scripts. Interesting NPCs uses 3400 strings before even leaving qasmoke. It has 6800 scripts, most of which have at least one property. There could potentially be HUGE gains to be made there.

 

Like all of those auto-generated dialogue scripts that do nothing except advance a quest stage. Isn't there any other way to do that? Like without a script? I don't mess with the dialogue system much, but I feel like we could get rid of some of those scripts.

 

 

There is no way to do anything in dialogue or a scene without a script, all dialogue will do is dialogue and nothing else and scenes can only do dialogue or perform AI packages. Believe me, if there was a way to not have to do a script in every piece of dialogue or scene where something needs done I would do it as it is a tedious pain in the ass to do.

[Nepro]

 

Even with the patchers it still didn't fix many issues with the Morrowind engine and some heavy mods. Hopefully the SKSE team isn't retired from messing with Skyrim now that FO4 is out. Did they ever get back to you?

 

They did. Their concern was that the strCount field may be 16 bits in the engine itself, which would mean that a lot of variables and function calls would need to be changed, not just one field of the save file format.

 

For the record, I disagree. I think the evidence suggests that the string table size is 32 bits internally, and that the whole table is being successfully written -- just with a truncated strCount. But they're the experts and I'm not, so take my opinion with a grain of salt.

 

We are just starting to get the word out about this. It will take some time to trickle down the channels. In the coming weeks we will hopefully lean a few things about how to write scripts that work around the issue. I am going to check a few ideas along the way while stress testing my new beta scripts I posted earlier.

 

I think the starting point for most mods will be trimming down the MCMs, and putting as many options as possible into INI files or JSON files (using PapyrusUtil, JContainers, or JaxonzConsolePlugin).

 

The biggest difference will be finding ways to reduce numbers of scripts. Interesting NPCs uses 3400 strings before even leaving qasmoke. It has 6800 scripts, most of which have at least one property. There could potentially be HUGE gains to be made there.

 

Like all of those auto-generated dialogue scripts that do nothing except advance a quest stage. Isn't there any other way to do that? Like without a script? I don't mess with the dialogue system much, but I feel like we could get rid of some of those scripts.

 

 

Quoted you just so you get pop up. I don't know if this solves anything but I suffered too from the CTD with heavy mod setup on save load especially after I enabled more anims or enabled Creature Anims in SL. The solution for me was to turn down the number of cores being used by Skyrim. I have 12 cores in total. I toned down the Skyrim's core usage to 8. Now if I go and try to load a save with all cores running I will get instant CTD but with cores number decreased to 8 the game loads and is stable. I guess YMMV.

 

I also tested threading support offered by ENBoost. This too solved my CTD on save load but it greatly impacts performance.

[D_ManXX2]

i did not know about cores where do you change the core settings for skyrim ??

[Monoman1]

Ok stop. This is not ctd on load. Not the same issue.

These saves will never ever ever load in skyrim or any other program. I don't mean to be rude but there are 1,000,001 threads about ctd on load. If you load your save in the save game cleaner (http://www.nexusmods.com/skyrim/mods/52363/?) and it doesn't immediately crash then you do NOT have this issue.

 

Let's keep this thread for strcount > ffff.

 

So even dialog adds strings (sounds stupid when I say it)? I noticed a block of strings from apropos. It would explain CDs and interesting npcs impact on the problem. And, at the end of the day, how poor having 16 bit limit is for a "RPG" :sighface: