Corrupted Saves

[Earen]

 

 

New question: Would it be possible to patch SkyUI to use jcontainers? Would it be worthwhile?

Someone on nexus wanted to describe MCM UI layout using json files. I think that this is more that just patching, but this would actually save a lot of work when creating new MCMs, defining event handlers. If someone wants to start this, I think I would elaborate. Though it won't save much strings' space

 

Really? I think it would save a lot of strings. Auto-generated MCMs (like SLAL and big parts of SexlabFramework) don't seem to use up any strings at all. I think JSON MCMs sounds like a grand idea.

 

Well, I don't know how much strings they may use. OID property names consume some amount of space. Though new way of defining event handles inside states helps eliminate that. Sometimes. I.e. in some cases OIDs shouldn't be used at all.

 

It was almost the first time I opened SKI_ConfigMenu. 128*3 strings per MCM. But they remove them in CloseConfig

MCMs don't cause much trouble. And doing some MCM json ski-fi no-one will use might be bad idea

[BlackArk]

Ahh, I added more chesko mods after reducing the strings by removing other mods that I do not use much anymore, the updated wearable lanterns and simply knock and I am getting closer to my save game imploding again. At 64054

 

 

Animation Slots Patch

Before activating sexlab              : 29792
After activating sexlab               : 30102 (+310)
After enabling creature animations    : 30346 (+244)
After registering 278 SLAL animations : 30855 (+509, 1.8/anim)

If anyone wants to test out my patch for sexlab, here it is. It should bring the string usage way, way down.

 

Do NOT use this with any savegame that you care about! If you want to try it in an existing save, you should completely uninstall Sexlab, clean your save, and then reinstall with the patch.

 

I decided to give that a shot in an existing save just for the hell of it and to brought it down to 59887 and then 59596

 

So it was a reduction of 4167/4458. Much better. I am going to do a dirty installation test it in a current save that I care a little bit about that I have back ups of. Wish me luck xD

Okay so I did not notice any major issues with doing a dirty installation of the patch other than the game hanging a little when I save.

 

Now to try quick clean save installation where I just use the save cleaner and do a quick mod edit and clean out specifically sexlab.esm and see what happens.

Well that broke cursed loot making it think I have mods that edit children tags? Reinstalling cursed loot fixed it though and strings are at 59308 after loading up all the mcm menus and registering all the animation mods again. An improvement of 4746. Impressive. Though the game hangs when saving. Better than my game imploding again I guess.

 

Hmm, run for your lives and vampires attack add 0 strings? And to think I removed them and put in immersive citizens.

Dyndolod is fixed? I am tempted to install it again.

[Monoman1]

Sheson posted beta scripts for DynDoLod in this thread a while back. 

I don't think the nexus mod has had those added yet. 

 

Oh and btw. Removing mods mid game won't necessarily remove all the strings it created. See the nexus post. Uninstalling Interesting NPC left about 800 strings.

[BlackArk]

Sheson posted beta scripts for DynDoLod in this thread a while back. 

I don't think the nexus mod has had those added yet. 

 

Oh and btw. Removing mods mid game won't necessarily remove all the strings it created. See the nexus post. Uninstalling Interesting NPC left about 800 strings.

 

Oh yeah I know. I managed to reduce it to about 62k strings from just uninstalling mods and cleaning the save. It was captured dreams and some other mods I forget. The save corruption was imminent and already happening to some of my saves at that point so reducing and having leftover strings is better than losing my save completely.

[malilie]

 

Hmm, run for your lives and vampires attack add 0 strings? And to think I removed them and put in immersive citizens.

Dyndolod is fixed? I am tempted to install it again.

 

Someone earlier in the thread than my tests had reported them as 'having more strings than I would have thought' without posting details.  The 0 number comes from a new game with no playtime from my test.  They may be the kind of mod that increases string count as you go along.

I'm afraid I'm not nearly familiar enough with papyrus to help coming up with solutions.  But I have the time and the interest to help test things if someone needs it.  Just send a PM if I can help. 

[Monoman1]

Yea I don't know how accurate the list is as playing could add more strings. 

DynDoLod was reported to add 63 strings here, I'd presume that was a new game. 

 

A guy on the nexus recovered 20k strings by uninstalling DynDoLod (He wasn't using the beta scripts). Just shows how tricky this problem is. 

 

In other news DavidCobb on the nexus is looking at uploading some TesEdit scripts that rename script properties. Problem is I don't think TesEdit has an API a phyton script (or anything else) could use. 

 

Should be an interesting experiment though. Even if we could make string optimized versions of string heavy mature (not being developed anymore) mods individually it would help. Hopefully we can swap script property names without papyrus having a shit attack. 

[Monoman1]

xEdit scripts for property manipulation have been uploaded. 

https://github.com/DavidJCobb/xedit-tesv-scripts

Credit goes to DavidCobb over on the nexus. 

Looks interesting. 

[malilie]

Okay, a bit of data on string growth.  I have a current playthrough I started with this thread with roughly 350 mods or so.  In the LAL cell the save had 51,055 strings.  Now, 87 hours in with no mods removed but several added (although none that added many strings as I checked as I added them) I'm sitting at 55,700 strings.

 

So, one test with a single modlist the growth was 53.4 strings per hour.

[Mud]

In the same vein, just a few more stats on growth. (Play time is the "date" given on each save file as in "001.34.56" or similar, is that right? This is written under that assumption.)

 

I've been playing a burgeoning save file that felt as though it was ready to implode on itself from the start, before finally collapsing due to this issue around the level 40 range. My save file made 1.5 hours after the beginning of the game clocks in at a precarious 64,024 strings, meaning I'm already dangerously close to corruption between mods like SexLab, Legacy of the Dragonborn, Interesting NPCs, and more. At some point, I did remove a couple of mods, among them SexLab Hormones, while also updating some like going from SL 1.60 to the latest and putting in the new ZAP. The saves started corrupting while I was working through Meridia's temple approximately 150 hours later. Checking a save file made a handful of hours afterward, as I'd deleted many older saves between then and learning about this issue, I see the save has 65,470 strings. I persevered with these saves for an additional 23 hours, plagued by corruption that got increasingly bad, and checking the last working save I had before throwing in the towel sees it has 65,456 strings. So it would appear string count can fluctuate rapidly by a minimum of 80 strings at any moment, as I'd taken to saving multiple times in the same location just in the hopes of getting one to work, and sometimes one would come out valid and sometimes they'd be corrupt.

[merryMalfunctioning]

In the same vein, just a few more stats on growth. (Play time is the "date" given on each save file as in "001.34.56" or similar, is that right? This is written under that assumption.)

 

I've been playing a burgeoning save file that felt as though it was ready to implode on itself from the start, before finally collapsing due to this issue around the level 40 range. My save file made 1.5 hours after the beginning of the game clocks in at a precarious 64,024 strings, meaning I'm already dangerously close to corruption between mods like SexLab, Legacy of the Dragonborn, Interesting NPCs, and more. At some point, I did remove a couple of mods, among them SexLab Hormones, while also updating some like going from SL 1.60 to the latest and putting in the new ZAP. The saves started corrupting while I was working through Meridia's temple approximately 150 hours later. Checking a save file made a handful of hours afterward, as I'd deleted many older saves between then and learning about this issue, I see the save has 65,470 strings. I persevered with these saves for an additional 23 hours, plagued by corruption that got increasingly bad, and checking the last working save I had before throwing in the towel sees it has 65,456 strings. So it would appear string count can fluctuate rapidly by a minimum of 80 strings at any moment, as I'd taken to saving multiple times in the same location just in the hopes of getting one to work, and sometimes one would come out valid and sometimes they'd be corrupt.

 

I can't believe you made it as far as you did! Maybe try uninstalling Interesting NPCs. It's a fantastic mod, but also a huge string hog.

[Mud]

I saw that, but unfortunately I've played this game so many times I simply absolutely need those mods that add new faces and content to justify revisiting Skyrim. Although that same mindset is what's led me to have Wyrmstooth and Falskaar installed for ages and end up playing through them a grand total of zero times...

 

There's also the part where Interesting NPCs is part of my NPC visual patch made using the NPC Visual Transfer Tool, and cutting that out means having to make that thing again entirely from scratch.

 

Definitely trying to see what I can cut out while planning my next file, though. Already pulled out Moon and Star, which I finally managed to play through on the corrupted file. Wondering if I should go back to SkyMoMod, after seeing it has 350 strings less than Immersive Creatures. But of course, this installation also in turn requires a few new mods, so... hopefully I don't render all my gains for naught with the additions!

[Monoman1]

About string growth:

My games don't seem to grow by all that much, relatively to other people. To people that are seeing 5,000+ increase in strcount: Are you using the older DynDoLod scripts? Are you measuring before or after animation registration? 

 

My last attempts at a play through. Just after LAL cell, MCMs configured, 16mins in:

Strings: 63,147

Scripts: 6,864

Script Instances: 87,593

 

90 hours later:

Strings: 65,236

Scripts: 7,094

Script Instances: 161,312

 

Judging by "scripts" I either added some mods in that time or maybe just mod updates. 

 

String increase: 2089

Strings per hour: 23. 

 

I've a theory that script instances are keeping variable strings in the string table which is why the string table grows slowly. Example: 

 

Script 1 has variables: Variable1, Variable2, Variable3.

Script 2 has variables: Variable3, Variable4, Variable5. 

 

A new instance of script 1 is created. It is the first one of its kind to be instanced:

Variable1, Variable2, Variable3 are added to the string table.

 

Another instance of script 1 is created. No strings are added to the string table as they already exist. 

Some time later a new instance of script 2 is created. It is the first instance of script 2 to be created:

Variable4 and Variable5 are added to the string table. Variable3 already existed. There's bound to be a certain amount of overlap of variable names between scripts. 

 

Does this theory hold water? I would presume that at some point script instances eventually complete. So if the 2 instances of script 1 were to complete Variable1 and Variable2 would be removed from the string table. But it's probably more likely that more instances of script 1 would be created before the first two instances finished up => strCount increases...

 

Sound plausible? 

 

----------------------------

 

About save file string table vs. game engine string table: 

I recently created a new version of the save breaking mod. It adds 99,000 string variables to the string table. I installed it on to a save with an existing string count of 45,608. 

 

So:

65,535 - 45,608 = 19,927 strings are needed to fill the string table to 0xFFFF. Which leaves:

99,000 - 19,927 = 79,073 strings to wrap around and overwrite the string table again entirely. 

 

In theory if the game engine string table were to be overwritten like this it should produce all sorts of weirdness in game and papyrus spam to the logs. Does this then prove that the engine is capable of a string count higher than 16 bits and that it is just the sub-routine that reads/writes the save file that uses 16 bits? 

 

People should give it a go themselves to see if they experience any weirdness or log spam. 

Note that if you are not getting debug.notification spam every 5 seconds or so from the mod in the top left corner of the screen it's likely the scripts have ended up on the suspended stack and are not actually running. Not sure why that is happening. 

[EpitomyofShyness]

Hey all, I've been following this pretty close since I'm a heavy modder and I don't want to wind up 100 hrs into a game and unable to make a save. 

 

I compiled a list of known spreadsheets that cover which mods affect the string count, as well as links to all the forum discussions on this issue; https://docs.google.com/document/d/1HY4b4L3tJuYPxJazEc8XeF-q3WXI11u76hlKKleQel8/edit?usp=sharing

 

I also made my own spreadsheet which can be edited by anyone, it is a compilation of every mod and how it affects string count that I could find; https://docs.google.com/spreadsheets/d/1iIpwNT75ZQ5eejRgrSLt5kySrAKtTW1n7Q7IgtNy3NE/edit?usp=sharing

 

Thank you all for all the research and investigating you've done. You guys are awesome.

[Mud]

Hey all, I've been following this pretty close since I'm a heavy modder and I don't want to wind up 100 hrs into a game and unable to make a save. 

 

I compiled a list of known spreadsheets that cover which mods affect the string count, as well as links to all the forum discussions on this issue; https://docs.google.com/document/d/1HY4b4L3tJuYPxJazEc8XeF-q3WXI11u76hlKKleQel8/edit?usp=sharing

 

I also made my own spreadsheet which can be edited by anyone, it is a compilation of every mod and how it affects string count that I could find; https://docs.google.com/spreadsheets/d/1iIpwNT75ZQ5eejRgrSLt5kySrAKtTW1n7Q7IgtNy3NE/edit?usp=sharing

 

Thank you all for all the research and investigating you've done. You guys are awesome.

What's this optimized version of Wyrmstooth mentioned in the sheet? It seems to have cut down on its string usage by a great deal.

[EpitomyofShyness]

 

Hey all, I've been following this pretty close since I'm a heavy modder and I don't want to wind up 100 hrs into a game and unable to make a save. 

 

I compiled a list of known spreadsheets that cover which mods affect the string count, as well as links to all the forum discussions on this issue; https://docs.google.com/document/d/1HY4b4L3tJuYPxJazEc8XeF-q3WXI11u76hlKKleQel8/edit?usp=sharing

 

I also made my own spreadsheet which can be edited by anyone, it is a compilation of every mod and how it affects string count that I could find; https://docs.google.com/spreadsheets/d/1iIpwNT75ZQ5eejRgrSLt5kySrAKtTW1n7Q7IgtNy3NE/edit?usp=sharing

 

Thank you all for all the research and investigating you've done. You guys are awesome.

What's this optimized version of Wyrmstooth mentioned in the sheet? It seems to have cut down on its string usage by a great deal.

 

 

Oops that looks like a mistake on my part. I'll fix it asap.

[merryMalfunctioning]

Hey all, I've been following this pretty close since I'm a heavy modder and I don't want to wind up 100 hrs into a game and unable to make a save. 

 

I compiled a list of known spreadsheets that cover which mods affect the string count, as well as links to all the forum discussions on this issue; https://docs.google.com/document/d/1HY4b4L3tJuYPxJazEc8XeF-q3WXI11u76hlKKleQel8/edit?usp=sharing

 

I also made my own spreadsheet which can be edited by anyone, it is a compilation of every mod and how it affects string count that I could find; https://docs.google.com/spreadsheets/d/1iIpwNT75ZQ5eejRgrSLt5kySrAKtTW1n7Q7IgtNy3NE/edit?usp=sharing

 

Thank you all for all the research and investigating you've done. You guys are awesome.

 

Thanks, I put links to your documents into the OP.

[EpitomyofShyness]

 

Thanks, I put links to your documents into the OP.

 

 

 I am so honored!

 

But mostly just glad to be of some small help. 

[Mud]

Animation Slots Patch

Before activating sexlab              : 29792
After activating sexlab               : 30102 (+310)
After enabling creature animations    : 30346 (+244)
After registering 278 SLAL animations : 30855 (+509, 1.8/anim)

If anyone wants to test out my patch for sexlab, here it is. It should bring the string usage way, way down.

 

Do NOT use this with any savegame that you care about! If you want to try it in an existing save, you should completely uninstall Sexlab, clean your save, and then reinstall with the patch.

Hmm... so what is this patch supposed to be comprised of? Testing it out on a new game, all it seems to do is make SL stop working. While watching my console for debug info during the installation, I see it spit a lot of fatal errors, the animation registration is slowed to an unbearable crawl, and I can't seem to access anything to do with SL's animations, such as its Toggle Animations menu. It's not reducing my string count either, comparing the strcount for the new files I made with and without the patch actually has the patched save holding more strings. Should I check for any particular installation requirements or do something else?

[Monoman1]

The patch requires jcontainers 3.3

[sidfu]

readying this imo probaly only way to get past this issue would to have a save system seprate from the skyrim save take over saveing.

[Monoman1]

Really quiet in here lately.

 

@sidfu

I think (could be wrong) something like that would also require modifying the binaries as well anyway since you'd need access to more than papyrus can provide.

 

@markdf

Any word on the papyrus util patch mark?

Not rushing, just curious.

[sidfu]

Really quiet in here lately.

 

@sidfu

I think (could be wrong) something like that would also require modifying the binaries as well anyway since you'd need access to more than papyrus can provide.

 

@markdf

Any word on the papyrus util patch mark?

Not rushing, just curious.

 

well the hard part even if we find a way to offload the stuff to other programs its getting the mod authurs to do it.

 

to many mod makers are of this mentalty "its never happened to me so its not a problem".

if we can atleast get something that willl force mcm to save stuff to jcontainers will will save alot of string space. i just get this feeling that the guys who make the mcm mod not gonna help any.

 

ive did alot of testing( thank god for MO) wither starting with differnt builds of differnt starting ammounts of strings. from whati found if u start with anything over 55k u looking at around 20 hours before teh issue comes up. another thing i found is mods that use this tend to dont have much of mcm get saved into the save.

 

probly best rigth now is to focus of getting the mcm seetings moved from being strings in the save first. on my builds if could cut out just the mcm in the saved would cut down the sstrings probally by a good 6k or more.

[EpitomyofShyness]

 

 from whati found if u start with anything over 55k u looking at around 20 hours before teh issue comes up. 

 

That is highly concerning. For those of us interested in 100+ hr saves what would you guestimate is a string count to aim for? Was the string rate of increase slowing down at all when you got towards the end of those 20 hrs using the 55k starting file? 

[sidfu]

right now im getting a 50kish one to try. so i would start at 50k probaly go give u headroom. now this all depends on your mods. u might beable to play for hundreds of hours with your mod setup where others wont. all u can do is test it.

 

i wouold recommend probaly like the other guy load up as usual grab a house and ride around skyrim for 2 hours. probaly good to visit every city run thru them and if after 2 hours your count looks good( also probaly good to have saves every 10min t ocompare)

 

i would rather spend 2 hours testing than to spend 20+ and fail.

[Monoman1]

If you can ride around skyrim on a house your game is definitely fucked

Format your hard disk and start again.