Corrupted Saves

[vyper00]

As far as I can see, the bulk of the String data is, as you say, script variables. I guess they're not exactly static, seeing as scripts can get attached to actors, objects, etc.

 

I'll just post some data what I've checked so far:

 

On a clean start save (only with official DLC) I have around 27k strings.

With all my mods (255+, merged ones etc.) I get for a new start like 55k

100 hours into the save I'm now around 65k, but I also added some mods during that time.

Perhaps keeping it at like 60k will be enough that fluctuation isn't too high... need to test that out.

 

What is ironic, perhaps: pretty much the only count the .ess file keeps as uint16 is the stringCount, everything else is uint32, like scriptCount, when it should be obvious that there will be much more strings in the save than scripts, if I understand this save format correctly.

 

 

[Monoman1]

Yea, that is odd: 

 

strCount uint16

scriptCount uint32

 

So we can have 4,294,967,296 scripts

but only 65,536 variables?

 

eh, what Bethesda? Surely that should be the other way around....

 

[markdf]

 

Looking at how the .ess file is laid out ( see http://www.uesp.net/wiki/Tes5Mod:Save_File_Format/Papyrus ),

the first thing after the header is strCount and strings. Since strCount is saved as "uint16", the maximum possible value is 65535,

which in the save file itself would be saved as hex value right at the beginning after the header, for me it's like

header: 04 00,

then strCount: DC FF .

You can see this in TESVESSE.

 

The problem is, what happens after it reaches FF FF. I looked in a corrupt file and it says 19 00, which would mean a strCount of 25.

So I guess Skyrim just decides to start counting again from 00 00 after FF FF. Therefore, since in reality there is a

lot more Strings there, but the file only counts 25, the game can't read the save correctly, same goes for the usual save editors.

 

So I guess a good question here is: what variables get serialized to that section? Is it all named variables? Just the heap? Just the stack? Does it include properties? If we knew exactly what was being serialized to that section, mods could be patched to use variables more efficiently. For example, does a ten element array count as a one entry, or ten?

[Monoman1]

Unfortunately, at this point we need the more experienced modders to take a look at this issue. Most mod updates are going to add more variables and with more and more animations being released  I think it's going to become a more common problem. At the very least it would be nice if there was an easy way to find out which mod all variables belongs to so we could make a call on which mod would be best to remove. Some are easily identified due to the mod authors convention (eg. dcur_ = Cursed loot). Most are not. 

[Centrophy]

A lot of good information here. I wasn't aware of that other save tool. Sadly, there is no fix yet. Right now, it's probably best to limit sexlab or sex related mods to 5 per profile. As I stated in another thread I'm currently running a sexlab only test profile in which only sex mods and their prerequisites are enabled. So far there has been no save corruption but I did have to disable a few like milk mod economy, untamed, etc to even get the game to load on clicking new game.

[4nk8r]

Does this string limit have anything correlation with form count?  I've finally managed to get to 9.4MB papyrus size with no corruption, and I'm sitting somewhere in the 50k range of form count. 

[markdf]

Does this string limit have anything correlation with form count?  I've finally managed to get to 9.4MB papyrus size with no corruption, and I'm sitting somewhere in the 50k range of form count. 

 

No, form count can be CRAZY high. Mine floats around 90k sometimes. This *seems* to be about named variables (and maybe properties) in scripts.

[Monoman1]

Papyrus size seems to include other things besides strcount. I've seen saves with 140,000 and I'm pretty sure I've seen closer to 200,000 form count.

 

There are no duplicate variables in strcount that I've seen to account for script instances attached to npcs or objects etc or account for why a game gradually goes bad. So to me, it looks like with a new game only variables of functions that have been called reside here. As the game progresses and more functions are called for the first time strcount grows. There might not be any 'garbage collection' going on.

 

But since there aren't any duplicates they don't seem to be variables that store information. They just seem to be variable definitions.

[markdf]

I've been going through my string section, trying to identify which mods add the most variables.

 

• When Vampires Attack and Run For Your Lives have surprisingly high amounts of variables.
• Legacy of the Dragonborn adds more than anything else, by far.
• Violens adds a lot.
• Frostfall and Campfire add quite a few between them.
• Immersive College of Winterhold adds a huge amount.

 

[Centrophy]

 

I've been going through my string section, trying to identify which mods add the most variables.

 

• When Vampires Attack and Run For Your Lives have surprisingly high amounts of variables.
• Legacy of the Dragonborn adds more than anything else, by far.
• Violens adds a lot.
• Frostfall and Campfire add quite a few between them.
• Immersive College of Winterhold adds a huge amount.

 

 

Well 2 out of 5 ain't bad. I always run Violens for player killmove immunity, turning off ranged killmoves, and turning off the slowdown effect. I also just like Immersive College of Winterhold. When Vampires Attack and Run For Your Lives have been superseded by Immersive Citizens.

[markdf]

What's frustrating about this bug is that it would be such an easy fix for Bethesda -- just change that one field to be uint32. Someone could knock this out in an hour, most of which would be testing the fix.

 

The kind of people who make cracks and patchers for games could probably do this too. Disassembling executables and making one or two tiny changes is their whole thing! Morrowind had a whole bunch of exe patchers for fixing various bugs and performance bottlenecks.

 

For the rest of us, all we can do is spread the word, and encourage modmakers to use variables and properties more sparingly.

[Monoman1]

Sad but true. 

Just had a look at the disassembled exe there. Suffice it to say I haven't got a fucking clue what I'm looking at and I did a little assembly back in school. Unfortunately compilers can optimize the hell out of source code making reading the decompiled version much more difficult. I think, like you said, Bethesda could fix this really easily with access to the source code. 

If it were made known to them what do you think the chances of them fixing it would be. What, 4 or 5 years after the last update. Pretty damn slim I would say. 

[markdf]

I did some more tests. The worst offenders I've found for using up the string block are:

• Interesting NPCs -- approximately 3400.
• Legacy of the Dragonborn -- a little under 4000.
• Sexlab -- approximately 450 just for being there.
• Sexlab -- approximately 1400 to initialize it.
• Sexlab -- approximately 550 to activate creature animations.
• Sexlab -- approximately 12 strings per animation registered (2000 to activate 170 animations).

 

These results were obtained without ever leaving QASmoke. Some mods may be be much higher once they activate.

[Monoman1]

Easy to see how quick it adds up. And many functions probably haven't been called yet, so it's not a maximum, or at least that's my theory. 

Any results for DynDoLod? I'm curious since it's such a small mod script wise. It seems at odds with what we think. 

 

Edit: You might try saving in qasmoke. Then outside somewhere (exterior) then back inside again after. Just to see if it decreases. If I remember rightly, corruption was much more common when saving in an exterior cell. I'd check it myself but I haven't got the DynDO output generated on my laptop...

[Ollie]

Okay, you guys are golden. 

Just disabled Interesting NPCs, and now registering SL animations (SL base + creatures + NSAP + FunnyBiz pack + Estrus = 310+) no longer causes saves to go nuts. SaveTool opens them fine and doesn't crash. Before that just registering creatures would break it.

I'll put some hours in playthrough to see if the problem will pop up later on in the game.

[sheson]

Easy to see how quick it adds up. And many functions probably haven't been called yet, so it's not a maximum, or at least that's my theory. 

Any results for DynDoLod? I'm curious since it's such a small mod script wise. It seems at odds with what we think. 

 

Edit: You might try saving in qasmoke. Then outside somewhere (exterior) then back inside again after. Just to see if it decreases. If I remember rightly, corruption was much more common when saving in an exterior cell. I'd check it myself but I haven't got the DynDO output generated on my laptop...

 

Based on this info that markdf pointed out to me, I went ahead and updated the papyrus scripts for DynDOLOD to use as few strings as possible. Comparing the total amount of strings before/after in the save files after a speedrun shows a couple thousand less strings in the table actually. I figured every bit helps for busy load orders...

 

To update an existing save, first deactivate DynDOLOD, wait for deactivation message, go into interior, check MCM DynDOLOD is still deactivated, save, exit game. Disable DynDOLOD .esp and remove old scripts. Start game, load last save, wait a bit, save again. Replace old scripts with the updated BETA papyrus scripts, enable DynDOLOD.esp, start game, load the last save, go outside. DynDOLOD should initialize by itself.

[markdf]

Easy to see how quick it adds up. And many functions probably haven't been called yet, so it's not a maximum, or at least that's my theory. 

Any results for DynDoLod? I'm curious since it's such a small mod script wise. It seems at odds with what we think. 

 

Edit: You might try saving in qasmoke. Then outside somewhere (exterior) then back inside again after. Just to see if it decreases. If I remember rightly, corruption was much more common when saving in an exterior cell. I'd check it myself but I haven't got the DynDO output generated on my laptop...

 

It seems to be complex. Interesting NPCs grabs its 3400 without ever leaving QASmoke, but I know from playing the game that the strings count goes up as you play (and occasionally drops back down).

 

So some of it is one-time costs, and some of it is dynamic.

 

DynDoLod seems to max out at 80 strings, which I agree is at odds with how strongly it gets associated with this particular form of savegame corruption. There may be some weird set of conditions where it starts to make thousands of strings though, possibly as a result of interactions with other mods like footprints or realshelter or something.

[markdf]

 

Based on this info that markdf pointed out to me, I went ahead and updated the papyrus scripts for DynDOLOD to use as few strings as possible. Comparing the total amount of strings before/after in the save files after a speedrun shows a couple thousand less strings in the table actually. I figured every bit helps for busy load orders...

To update an existing save, first deactivate DynDOLOD, wait for deactivation message, go into interior, check MCM DynDOLOD is still deactivated, save, exit game. Disable DynDOLOD .esp and remove old scripts. Start game, load last save, wait a bit, save again. Replace old scripts with the updated BETA papyrus scripts, enable DynDOLOD.esp, start game, load the last save, go outside. DynDOLOD should initialize by itself.

 

Seems to be down to 63 strings. :-)

 

Next step: get every single Skyrim modder on earth to do the same thing...

[Monoman1]

 

Easy to see how quick it adds up. And many functions probably haven't been called yet, so it's not a maximum, or at least that's my theory. 

Any results for DynDoLod? I'm curious since it's such a small mod script wise. It seems at odds with what we think. 

 

Edit: You might try saving in qasmoke. Then outside somewhere (exterior) then back inside again after. Just to see if it decreases. If I remember rightly, corruption was much more common when saving in an exterior cell. I'd check it myself but I haven't got the DynDO output generated on my laptop...

 

Based on this info that markdf pointed out to me, I went ahead and updated the papyrus scripts for DynDOLOD to use as few strings as possible. Comparing the total amount of strings before/after in the save files after a speedrun shows a couple thousand less strings in the table actually. I figured every bit helps for busy load orders...

 

To update an existing save, first deactivate DynDOLOD, wait for deactivation message, go into interior, check MCM DynDOLOD is still deactivated, save, exit game. Disable DynDOLOD .esp and remove old scripts. Start game, load last save, wait a bit, save again. Replace old scripts with the updated BETA papyrus scripts, enable DynDOLOD.esp, start game, load the last save, go outside. DynDOLOD should initialize by itself.

 

Cheers sheson, you're a star. I'll check it out as soon as RL stops kicking me in the teeth. Hopefully other authors will follow suit. 

 

Any thoughts on the possibility of fixing the issue at root, ie- changing the strcount field from 16 bit to 32 bit ?

[markdf]

Any thoughts on the possibility of fixing the issue at root, ie- changing the strcount field from 16 bit to 32 bit ?

 

I emailed the SKSE team about it. Not much else to do about that, unless you know someone who is good at decompiling software and patching it. The kind of person who makes cracks and memory patches for games, that kind of thing.

 

Edit: I also posted about this on the SLAL support thread, since they're liable to hear complaints about this issue from time to time. I would message Ashal about it, but I feel like Ashal already gets enough complaints about things that aren't actually Sexlab's fault. :-)

[crusher]

maybe someone will just hack the FO4 executable to work with Skyrim. Yeah, fantasizing is my speciality.  

 

or, create an exporter for bsa, esp; from Skyrim to FO4 -- sell copies of FO4 by providing an upgrade to Skyrim -- makes too much sense for the client base to be practical for the developer. Though if the heavy lifting were provided by the community, as is typically the case in after-release content, might make sense. 

[sheson]

 

Any thoughts on the possibility of fixing the issue at root, ie- changing the strcount field from 16 bit to 32 bit ?

 

I emailed the SKSE team about it. Not much else to do about that, unless you know someone who is good at decompiling software and patching it. The kind of person who makes cracks and memory patches for games, that kind of thing.

 

 

I really have no desire to look at the code again

The SKSE team should be much more at home in the executable, considering they have been doing this for decades. Maybe they could just add a new data type which saves to the co-save instead.

 

Doing changes for DynDOLOD is simple enough since it is contained in itself and already uses PapyrusUtil jsons for external data as much as possible anyways.

[Monoman1]

Cool, appreciate the input. Hopefully the guys at SKSE haven't gone into Skyrim retirement

[Centrophy]

So I'm looking through a savefile with tesvesse and I'm searching for strcount. Can't seem to locate it. Did you guys find that out with that tool or did you just use a hexeditor and the uesp info to find it? I'm trying to see how close I am to the FF FF limit on a new mod setup.

 

[Monoman1]

 

So I'm looking through a savefile with tesvesse and I'm searching for strcount. Can't seem to locate it. Did you guys find that out with that tool or did you just use a hexeditor and the uesp info to find it? I'm trying to see how close I am to the FF FF limit on a new mod setup.

 

 

Expand "Global Data table 3" and click on Papyrus. You don't need to expand it. strcount is "Strings", the first entry. 

 

 

Since strcount is a (presumably unsigned) 16 bit integer field, valid values for it are 0 through 65,535. As you can see that save is some way off becoming corrupt but it's not clear just yet what sort of "cushion" you would need to have a full playthrough...

A few posts back vyper said he starts new games with a strcount of 55,000 and hits the limit at about 100 hours of play time. It likely depends largely on the mods as well just how quickly it grows.