Corrupted Saves

[markdf]

UPDATE:

 

This issue has been determined by vyper00 to be caused by an overflow of the strings table in the savefile. The strCount field which indicates how many string are in the table is 16 bits (unsigned), so the maximum value is 65,535. When the numbers of strings exceeds 65535, the game continues to run without issue except for numerous errors in the Papyrus logs and savegames that are unreadable . Even Script Scalpel and SaveTool can't load them, they are COMPLETELY unrecoverable.

 

The TESV ESS tool can show you how many strings are in a working savegame; look in global data table 3, in the papyrus subsection.

 

----

 

So where do all these string come from?

 

The vanilla game + DLCs and USLEEP patch require 28,000 strings.

The vanilla game + DLCs and USLEEP and a full S.T.E.P. install requires 32,000 strings.

 

Here is a sample of a few mods that increase the string count substantially:

• Interesting NPCs (3400)
• Legacy of the Dragonborn (4000)
• Campfire and Frostfall (1000)

Sexlab increases the string count enormously:

• 450 before it's been activated
• 1400 more to initialize it
• 500 for creature animations
• ~12 per additional animation

---

 

This has already been reported to the developers of SKSE and ENB, and on the Bethesda, Nexus forums and STEP forums. Don't hold your breath for a patch, this is a somewhat obscure issue that only affects a minority of players who have particularly heavy load orders. The best thing you can do is watch for people posting about CTDs that match this issue, and referring them to one of the threads about it.

 

EpitomyOfShyness has compiled a structured document which includes links to other threads about this issue. It is a community resource and contains a lot of information that this post does not. Check it out. It's a superb resource if you are struggling with this issue.

 

At this time, the only solution is to remove mods with lots of scripts and limit the number of sexlab animations you install. The number of strings increases continually as you play, so try to keep it below 50,000 when you're starting out. Keep in mind that the fully patched vanilla game starts out at 28,000, and a full STEP:Extended game starts out at 32,000.

 

There is an unofficial/unsupported patch for sexlab that reduces the number of strings used by animations. It requires JContainers 3.3alpha. It is not compatible with existing savegames. Like seriously. If you try to use it with an existing savegame, the savegame will jump out of the screen and slap you to death. Don't try it. Anyway, the link is at the bottom of this post. If you install it, don't go asking for support on the main Sexlab threads because it is completely unofficial and unsupported.

 

---

NOTABLE STRING COUNTS

 

All values are approximate. Most were measured in QASmoke and may be higher during actual gameplay. Values reported by myself, Centrophy, and malilie.  EpitomyOfShyness has additional info available (link).

• 2000 - Sexlab Framework
•   550 - Sexlab Creature Animations
•     12 - Sexlab (per animation)
•     63 - DynDoLod
•   330 - Civil War Overhaul
• 3200 - Interesting NPCs
• 3900 - Legacy of the Dragonborn
• 1200 - Frostfall & Campfire
• 2700 - Ultimate Deadly Encounters (Sands of Time)
•   170 - Genesis Dynamic Spawns
•     28 - Revenge of the Enemies
•       0 - Inconsequential NPCs
•   830 - SPERG
•   560 - Skyrim Immersive Creatures
•   790 - iNeed
•   170 - Open Cities
•     93 - Expanded Towns and Cities
•     00 - OBIS
•     41 - Moonpath to Elswyr
•   110 - Real Names
•   326 - Arissa the Wandering Rogue
•       0 - When Vampires Attack
•       0 - Run for Your Lives
• 1300 - Helgen Reborn
• 4000 - S.T.E.P. Extended
• 1800 - Anna NPCs
•   670 - Dev Aveza
•   630 - Familiar Faces
•   670 - Falskaar
•   930 - Holidays
•     12 - Ethereal Elven Overhaul
•     35 - Book Covers of Skyrim
•     76 - Gildergreen Regrown
•     79 - Hidden Hideouts
•   120 - Immersive Patrols
•   120 - Inns and Taverns
•   110 - No Snow Under the Roof
•     84 - SkyFalls + SkyMills
•     36 - Skyrim Sewers
•   210 - Skyrim Monster Mod
•   150 - Sneak Tools
•   360 - Sofia
•   110 - The Choice is Yours - Improved Dialogue Options
•     29 - Travelers of Skyrim
•     82 - Unlimited Bookshelves
•   310 - Violens
•   350 - Wyrmstooth
•   370 - Moon & Star
•   130 - Huge bunch of merged hair mods
•   120 - TK Children
•     55 - Diverse Guards

 

Thanks for reading.

Sexlab Strings Patch 2016-04-30.7z

[Guest]

May I ask you which version of SexLab are you using?

[markdf]

I'm running 1.61b.

[Ashal]

Try uninstalling Captured Dreams.

 

Can't speak as to the reason, but I know personally that I always have broken saves every time I've tried that mod, and I've specifically narrowed it down to Captured Dreams specifically as being the one mod that fixes my save game problems 100% of the time after disabling it.

[markdf]

Some details I forgot:

 

1. The only other thing I've ever found that caused this kind corrupted save is DynDoLod, when it's running alongside heavily mod load.

 

2. It's not due to running out of RAM, VRAM, Papyrus heap memory, or Papyrus stack memory.  I checked them all.

 

3. There seems to be a connection to having an MCM. If I disable a mod with an MCM, I can add two or three mods without an MCM. I'm not sure if that applies to non Sexlab mods with an MCM or not. But it's still triggered by registering Sexlab animations.

[Comrade AethyrMipe]

Try uninstalling Captured Dreams.

 

Can't speak as to the reason, but I know personally that I always have broken saves every time I've tried that mod, and I've specifically narrowed it down to Captured Dreams specifically as being the one mod that fixes my save game problems 100% of the time after disabling it.

Was that with a recent version? I'm curious because I've never had save corruption issues with it

[markdf]

Try uninstalling Captured Dreams.

 

Can't speak as to the reason, but I know personally that I always have broken saves every time I've tried that mod, and I've specifically narrowed it down to Captured Dreams specifically as being the one mod that fixes my save game problems 100% of the time after disabling it.

 

No change, although I'm uninstalling Captured Dreams just in case.

 

[Ashal]

 

Try uninstalling Captured Dreams.

 

Can't speak as to the reason, but I know personally that I always have broken saves every time I've tried that mod, and I've specifically narrowed it down to Captured Dreams specifically as being the one mod that fixes my save game problems 100% of the time after disabling it.

Was that with a recent version? I'm curious because I've never had save corruption issues with it

 

 

 

Recent version of sexlab? No such thing for me really, I live in between versions and only ever use the version  I'm developing. It was specifically between both of the developments of 1.59c to 1.60, and 1.60 to 1.61. Both 1.60 and 1.61 would've been out months earlier than normal, had constant crashes on load not basically ruined my desire to ever test things in the update. It wasn't until the develop period between 1.61hf2 and 1.61 though that I finally threw my hands up in the air and spent almost 2 days of full time work narrowing it down specifically to Captured Dreams.

 

Whether it's specifically a problem with Captured Dreams I don't know, could be a bad interaction with another mod that's to blame, I didn't use to have a problem with it, but I can 100% replicate the issue with my Skyrim by simply re-enabling it.

 

It's also important to note this is NOT save corruption, if a save is corrupted you know it because Skyrim specifically tells you directly that "this save is corrupted" when you try to load it. CTD when loading a save isn't corruption, it's a bad mod install that can potentially be fixed. In my case with Captured Dreams, it wasn't ALWAYS crashing when loading to desktop, but it would take me a good 3-5 attempts at loading a save before I could actually get into the game without crashing, which is why it frustratingly delayed 1.60/1.61 so long.

[markdf]

 

It's also important to note this is NOT save corruption, if a save is corrupted you know it because Skyrim specifically tells you directly that "this save is corrupted" when you try to load it. CTD when loading a save isn't corruption, it's a bad mod install that can potentially be fixed. In my case with Captured Dreams, it wasn't ALWAYS crashing when loading to desktop, but it would take me a good 3-5 attempts at loading a save before I could actually get into the game without crashing, which is why it frustratingly delayed 1.60/1.61 so long.

 

My issue is definitely save corruption. I mean the file is so trashed that it makes Script Scalpel lock up and declare that there are 1.8 script definitions. It makes SaveTool crash instantly.

[markdf]

I did some more tests. Removing non-sexlab mods with MCMs does seem to help.

 

So I now have a theory about what is happening. There is some upper limit on how many MCM elements can be safely included in a save. Registering animations creates many more elements in the Sexlab MCM (and in the Aroused Creatures MCM). Sexlab mods usually have an MCM, which is why they seem to exacerbate the problem.

 

Once the limit on MCM elements is exceeded, trying to save results in parts of the save file being overwritten. Like a buffer overflow error.

[Ashal]

My issue is definitely save corruption. I mean the file is so trashed that it makes Script Scalpel lock up and declare that there are 1.8 script definitions. It makes SaveTool crash instantly.

 

What you describe is script scalpel, or your computer,  being bad at handling a specific scenario in your save, not corruption. Save corruptions are clear with a blatant message in Skyrim when loading a game. Though it is without question an invaluable tool for save fixing, it is not the final arbiter in save corruption. I prefer Scalpel myself like you're using, but  the other save game cleaner, whose name I can't remember, is much more user friendly in genreal and may provide different results.

 

 

I did some more tests. Removing non-sexlab mods with MCMs does seem to help.

 

So I now have a theory about what is happening. There is some upper limit on how many MCM elements can be safely included in a save. Registering animations creates many more elements in the Sexlab MCM (and in the Aroused Creatures MCM). Sexlab mods usually have an MCM, which is why they seem to exacerbate the problem.

 

Once the limit on MCM elements is exceeded, trying to save results in parts of the save file being overwritten. Like a buffer overflow error.

 

None of anything you just said has anything to do with anything in sexlab 1.60 or newer (currently 1.61b) 

 

The MCM limit is 128. SexLab paginates animations once they reach this limit and has been very successfully at just that. Had it not been, there would be thousands more people yelling about this issue than just yourself, considering SLAL and other animation packs make it very easy to surpass this limit and many users are regularly using them without problem. Creature and other normal animations also don't count towards the same supposed limit, they are saved and handled by entirely separate entities. 

 

Ultimately, the advice I'd give after looking at your papyrus log is simple. Your save is largely fucked up. It has all the markers of a game that has seen tons of mods installed, then uninstalled, and/or other mods that weren't installed properly, but kept anyway over multiple sessions.

 

You might be able to salvage the save if you can figure out what your white whale is like I did with Captured Dreams, enabling you to load and make a fresh save after a successful load that can be appropriately cleaned. 

 

In any case, NONE of what sexlab framework specifically does even remotely describes or starts the problem you've had. You're problem is with an individual sexlab plugin other than sexlab.sem, or general unrelated mixed mod install issues.

[markdf]

 

My issue is definitely save corruption. I mean the file is so trashed that it makes Script Scalpel lock up and declare that there are 1.8 script definitions. It makes SaveTool crash instantly.

 

What you describe is script scalpel, or your computer,  being bad at handling a specific scenario in your save, not corruption. Save corruptions are clear with a blatant message in Skyrim when loading a game. Though it is without question an invaluable tool for save fixing, it is not the final arbiter in save corruption. I prefer Scalpel myself like you're using, but  the other save game cleaner, whose name I can't remember, is much more user friendly in genreal and may provide different results.

 

The other one is called SaveTool, and it crashes almost instantly when loading these saves.

 

 

None of anything you just said has anything to do with anything in sexlab 1.60 or newer (currently 1.61b)

 

That is clear to me now. My theory is that it's a problem with how the game engine serializes all those SKI_ConfigBase scripts. Sexlab just happens to have a big, complex MCM.

 

What convinced me is that I got to a point where I couldn't make new saves anymore, but removing any mod with an MCM fixed things. Hardly enough to prove it to anyone else though... At some point I'm going to try to build a test mod that will demonstrate this problem more convincingly.

 

I appreciate your feedback. :-)

[BigOnes69]

Ashal is correct in that its a mod, but I dont think its limited to captured dreams. I noticed with save tool the amount of active and orphan  scripts would increase with each save and the size of the file would increase. With 1.60 Sexlab I have to monitor my saves and clean them constantly so as to run smoothly. I will try 1.61 latest version to see if anything happens with this.I uninstalled all of the older mods even slowly introducing the latest mods slowly but this continued happening although on a less frequent occurrence. It happens constantly when large combats occur making me think it has something to do with one of my rape or slave mods randomly activating other mods scripts.

[izumonoryu]

I've had the same issue, where registering the anims results in all saves from then on being corrupt, crashing Save Cleaner. However, cleaning up the SL system after registration (rebuild) appears to avoid this issue. Need to check if I have all the animations in.

 

This is on SL1.61b.

[markdf]

I've had the same issue, where registering the anims results in all saves from then on being corrupt, crashing Save Cleaner. However, cleaning up the SL system after registration (rebuild) appears to avoid this issue. Need to check if I have all the animations in.

 

This is on SL1.61b.

Does the issue stay away as you proceed through the game? How long have you tested it for? That's pretty great if it works! :-D

[4nk8r]

@markdf: You're not alone.  I have encountered this "save corruption" last night.  Took a few hours of tinkering to narrow it down to SLAL animation registration as the tipping point.  If I register more than about 100 additional animations to what I already have (some arbitrary amount...not sure), I get the apparent save corruption and SaveTool's inability to load the save for editing.

 

I did find another tool that would load the save without a crash.  TESV ESS.  It's not nearly as user friendly as SaveTool, so I've yet to work out how to repair a save just yet.  So far, I've noted that in games that won't load, when viewing [ Global data table 3 ] / [ 2 Papyrus ], the tool reports a "Range Check Error".  For example:

 

Global data table 3
+1 Temp Effects
-2 Papyrus
-- Type: 1001 [Papyrus]
-- Length: 8601876 (Range Check Error on this value.  Gets reset to 1032 after clicking OK. Length in a good save: 8414050)

-- Data (hex view of data values.)

 

 

+ represents unexpanded element
- expanded element
-- subelements of expanded element

 

 

After saving the savefile with just this change/reset, it still won't load, but resetting the value results in a save file that is radically smaller...from 11MB down to 5MB.  The visible "Data" in the hex viewer after this gets reset is gibberish.  On a healthier save, the visible data are game variables, like "EasmarchHoldLocation_var", etc.

 

 

Edit: Apparently the Length and Data don't always get reset after experiencing the "Range Check Error".  The error only happens when you actually click on the "2 Papyrus" element text, not the +/- expansion or sub-elements.  Resets happen only randomly.  Data is still normal looking (like a healthy save) when the Length isn't being reset.  I've also had Length get reset to 0 and Data was completely blank instead of gibberish.

[izumonoryu]

 

I've had the same issue, where registering the anims results in all saves from then on being corrupt, crashing Save Cleaner. However, cleaning up the SL system after registration (rebuild) appears to avoid this issue. Need to check if I have all the animations in.

 

This is on SL1.61b.

Does the issue stay away as you proceed through the game? How long have you tested it for? That's pretty great if it works! :-D

 

 

The first time it happened, I did the SL cleanup thing and it was pretty stable - I got in quite a few hours and the saves were loading.

 

I tried it again today, loading up some of the new packs - and it started crashing SaveTool again. So I tested the rebuild Sexlab method. It failed the first time, and I reset and reapplied the JSON settings from the SLAnim MCM, and it stabilized the save (tested in SaveTool).

 

So I went in and re-registered the animations from SLAnim to do a second check, and sure enough new saves were borked again. Reran the cleanup in SexLab and saves back to normal. Reapplied JSON settings and saves still normal.

 

Bottomline: pretty sure that the application, or mere installation of the animations borks saves, at least for me. The Sexlab cleanup function MAY work, or some combination of it and the SLAnimLoader Reapply JSON to Registered Animations function.

 

People should keep SaveTool on hand and test it out...for me, if SaveTool can't load it, Skyrim won't either, even through the load starting save + load problem save method, and THAT to me is an indication of a corrupted save.

 

[markdf]

From that data in TESV ESS, it seems like you're running up against some limit in Skyrim's ability to serialize data. At 8.4 million bytes you're okay, and at 8.6 million you're not. When I get home from work I'll try it out and see if I get similar numbers.

 

This lines up with my theory that it's the MCMs that are the problem. MCMs stay in memory, and every single text field, list entry, and checkbox takes up scripting resources and has to be serialized when you save. Every SLAL animation has an entry in SLAL's MCM, and when you register it gets an entry in Sexlab's MCM, possibly an entry in Aroused Creature's MCM (if you use it), possibly an entry in Sexlab Cumshot's MCM (if you use it), and so on.

 

DynDoLod can cause this too -- and what does it have? A huge dynamic MCM with lots and lots of entries.

[EikaForge]

To give a wider reference point of ESS data, my save started giving out of range errors at 10506672 length, reset to 10495642 and became loadable, albeit it either zeroed or fucked up most mcm settings.

 

Some good/loadable saves have the lengths of: 10515001, 10495642(Not reset like the one above), 8963460(Very early game, several mod addons ago.)

Some bad saves have the lengths of: 10506672, 1065455, 10618481

[Guest]

so far what I've done is gotten rid of captured dreams and some other mods that I wasn't using right now and only installed a smaller slal pack and even then
I didn't register all the animations at once. Now my save isn't corrupt so far, going to see what happens if i try the bigger pack (funnybiz one) and install captured dreams. :7
(I really hope I can use the funnybiz animations, they are pretty damn cool :7)

[markdf]

so far what I've done is gotten rid of captured dreams and some other mods that I wasn't using right now and only installed a smaller slal pack and even then

I didn't register all the animations at once. Now my save isn't corrupt so far, going to see what happens if i try the bigger pack (funnybiz one) and install captured dreams. :7

(I really hope I can use the funnybiz animations, they are pretty damn cool :7)

 

I know, right? It's hard to resist the temptation to add one more mod, and then one more, to push it right to the edge of where the corrupting issue appears.

[Guest]

 

so far what I've done is gotten rid of captured dreams and some other mods that I wasn't using right now and only installed a smaller slal pack and even then

I didn't register all the animations at once. Now my save isn't corrupt so far, going to see what happens if i try the bigger pack (funnybiz one) and install captured dreams. :7

(I really hope I can use the funnybiz animations, they are pretty damn cool :7)

 

I know, right? It's hard to resist the temptation to add one more mod, and then one more, to push it right to the edge of where the corrupting issue appears.

 

You just described what I feel when I mod my game perfectly, finally I get a stable game and I just feel like I could add one more, then one becomes another two and then suddenly I wrecked my game and have to start over.

[Guest]

That is pretty normal, .

The only advice is: do a save just before adding a new mod, and if it will screw up your game, revert back to it.

[markdf]

I asked flexcreator (the creator of Script Scalpel) to take a look at one of these corrupted saves. He confirmed that the header is corrupted.

[markdf]

So to recap this issue:

1) The headers of saves are being corrupted in a way that is unrecoverable.

2) It only seem to occur after Sexlab registers animations.

--2a) It is more likely to occur when Sexlab has registered many animations.

--2b) It is certain to occur if DynDoLod is installed alongside Sexlab.

--2c) It is more likely to occur when many scripted mods are installed, regardless of whether those scripts are running or not. For example, removing Wet and Cold doesn't help, but removing Sisters of Dibella or Captured Dreams does help. It got really bad for me when I installed Falskaar and Wyrmtooth, even in a brand new fame where I hadn't left the Alternate Start dungon.

3) Rebuilding Sexlab sometimes solves the problem, but it returns as soon as more Sexlab animations are registered -- such as enabling creature animations.

 

This all paints a confusing picture. Registering an animation doesn't create additional scripts or script instances, and it uses a trivial amount of memory (as I understand it, anyway). All it does is allocate a little bit of memory in a standard C++ library down in SKSE, and create a few entries in the MCM.

 

I would literally kill for a chance to debug this issue properly, with Bethesda's source code in front of me and an IDE that let me step through the relevant code. :-)