Help test fix for save game string table overflow

[Guest]

 

Any difference in script latency from having a much larger string table?

 

 

I did not test that. My test was only to check after a save/reload with a bunch of strings that all strings had the right value.

Keep in mind that strings are indexed, so getting one of 10,000, or one or 1,000,000 has pretty much the same time. (A string is a pointer, so it does not really matter.)

On saving, my save takes less than one second, so for me it is hard to check if the performance goes down.

 

BTW, I am really interested in the logic you do to "clean up" the strings. What you are actually doing? (I did not reverse engineered your code...)

[markdf]

BTW, I am really interested in the logic you do to "clean up" the strings. What you are actually doing? (I did not reverse engineered your code...)

 

I change the names of the scripts' Variables so that every script uses the same set of names. The majority of the string table is the names of Variables -- particularly the autovariables generated for Properties with the AUTO flag. They are safe to change because they are only ever used in the script that defined them and -- due to an optimization in Papyrus -- any scripts that extend it.

 

The one exception is variables with the CONDITIONAL flag -- they can be accessed from ESPs, so I leave them alone.

 

I did not test that. My test was only to check after a save/reload with a bunch of strings that all strings had the right value.

Keep in mind that strings are indexed, so getting one of 10,000, or one or 1,000,000 has pretty much the same time. (A string is a pointer, so it does not really matter.)

On saving, my save takes less than one second, so for me it is hard to check if the performance goes down.

 

I was more thinking of scripts starting and stopping; how much time gets used up merging scripts' string tables with the main string table? And do the strings ever get removed? Do they get reference-counted or something? Or do they occupy the string table forever, simply by virtue of existing?

 

I would dearly love to crawl around the Skyrim source code....

[markdf]

I can't even get SaveTool to recompile using the same IDE that it was created in.  
I can't even tell why it's not working. It creates an executable, everything seems fine, but it crashes as soon as it loads ANY savegame. The only thing I can think of is that it's some 64-bit vs 32-bit windows thing.

 

At this point I think it would almost be easier to make my own savegame cleaner.

[Guest]

I can't even get SaveTool to recompile using the same IDE that it was created in.  

I can't even tell why it's not working. It creates an executable, everything seems fine, but it crashes as soon as it loads ANY savegame. The only thing I can think of is that it's some 64-bit vs 32-bit windows thing.

 

At this point I think it would almost be easier to make my own savegame cleaner.

 

Knowing your knowledge of the Skyrim save game, and you development skills, I will mildly encourage to build a new tool for save games.

[h38fh2mf]

Thank you all for testing. Released with V8 of CrashFix mod, default is enabled!

[Dave0000]

hey just found this... gonna check it out ... i started to get ctd on load saved game... lets see if that helps!

[Mookeylama]

 

I can't even get SaveTool to recompile using the same IDE that it was created in.  

I can't even tell why it's not working. It creates an executable, everything seems fine, but it crashes as soon as it loads ANY savegame. The only thing I can think of is that it's some 64-bit vs 32-bit windows thing.

 

At this point I think it would almost be easier to make my own savegame cleaner.

 

Knowing your knowledge of the Skyrim save game, and you development skills, I will mildly encourage to build a new tool for save games.

 

indeed! a new save cleaner is about all we're missing from this process of dealing with the string bug. the Restringer keeps getting better to.

 

H38f and Markdf really need to set up some way to donate. went to the Crash Fixes nexus site and no donate button there

 

[Dave0000]

its amazing ... you sir are a NEO of skyrim ... i have verry buggy game game but even with 200 mods ...legends of dragonborn... etc... i have NOT crashed at game load now... only sometimes infinite loading like stuff... but its my setup i believe so not related.

 

wonderfull!

[slutofshades]

Since getting back into Skyrim recently, save corruption has been a pain in my ass. You... are a saint. I loaded a bunch of string heavy mods just to check.. purposefully trying to break my game without using a save breaker... Can confirm a load order that would normally result in a New game save being corrupt ((too many strings))... works perfectly. Even with Savecleaner not loading it.

 

I love you, and markdf and.... everyone that has been looking into this problem.

[TamaChan]

its amazing ... you sir are a NEO of skyrim ... i have verry buggy game game but even with 200 mods ...legends of dragonborn... etc... i have NOT crashed at game load now... only sometimes infinite loading like stuff... but its my setup i believe so not related.

 

wonderfull!

I noticed you posting on my thread too. If using slal is causing big problems for you let me know. I can help you delete it from your data files.

[Mookeylama]

I can't even get SaveTool to recompile using the same IDE that it was created in.  

I can't even tell why it's not working. It creates an executable, everything seems fine, but it crashes as soon as it loads ANY savegame. The only thing I can think of is that it's some 64-bit vs 32-bit windows thing.

 

At this point I think it would almost be easier to make my own savegame cleaner.

hey Mark, wondering if you are working on a save cleaner that'll read these new format saves?

[markdf]

I haven't started writing code yet, but yes I'm getting ready to start on that prject. Work is just taking up a lot of time.

[Monoman1]

Anyone seeing any problems when over the limit? 

 

Seeing a lot of these kinds of errors in the log. I think it only happens when over the limit: 

 

 

[05/08/2016 - 01:07:38PM] warning: Unable to get type ::OutfitList_var referenced by the save game. Objects of this type will not be loaded.

[05/08/2016 - 01:07:38PM] Cannot open store for class "::callForHelpMsg_var", missing file?

[05/08/2016 - 01:07:38PM] warning: Unable to get type ::callForHelpMsg_var referenced by the save game. Objects of this type will not be loaded.

[05/08/2016 - 01:07:38PM] Cannot open store for class "::GenderedCreatures_var", missing file?

[05/08/2016 - 01:07:38PM] warning: Unable to get type ::GenderedCreatures_var referenced by the save game. Objects of this type will not be loaded.

[05/08/2016 - 01:07:38PM] Cannot open store for class "::PuzzleDoorKeyholes_var", missing file?

[05/08/2016 - 01:07:38PM] warning: Unable to get type ::PuzzleDoorKeyholes_var referenced by the save game. Objects of this type will not be loaded.

[05/08/2016 - 01:07:38PM] Cannot open store for class "::Sanguine_var", missing file?

[05/08/2016 - 01:07:38PM] warning: Unable to get type ::Sanguine_var referenced by the save game. Objects of this type will not be loaded.

[05/08/2016 - 01:07:38PM] Cannot open store for class "::OpenLockProjectile_var", missing file?

[05/08/2016 - 01:07:38PM] warning: Unable to get type ::OpenLockProjectile_var referenced by the save game. Objects of this type will not be loaded.

[05/08/2016 - 01:07:38PM] warning: Variable ::Alias_Forsworn1_var on script playerbookshelfcontainerscript loaded from save not found within the actual object. This variable will be skipped.

[05/08/2016 - 01:07:38PM] warning: Variable ::Alias_Forsworn2_var on script playerbookshelfcontainerscript loaded from save not found within the actual object. This variable will be skipped.

[05/08/2016 - 01:07:38PM] warning: Variable ::Alias_Forsworn3_var on script playerbookshelfcontainerscript loaded from save not found within the actual object. This variable will be skipped.

[05/08/2016 - 01:07:38PM] warning: Variable ::Alias_Forsworn4_var on script playerbookshelfcontainerscript loaded from save not found within the actual object. This variable will be skipped.

[05/08/2016 - 01:07:38PM] warning: Variable ::Alias_Forsworn5_var on script playerbookshelfcontainerscript loaded from save not found within the actual object. This variable will be skipped.

It may be an issue with my load order though. I'll wait for someone else to confirm anything.

I've also had skyui mess up completely and attempt to reregister every MCM again and other mods complaining. Like frostfall detecting an older version of campires. Only seems to happen when reloading. 

[h38fh2mf]

Looks like issue with the string table. Did this start right after going over the limit?

[Dave0000]

 

its amazing ... you sir are a NEO of skyrim ... i have verry buggy game game but even with 200 mods ...legends of dragonborn... etc... i have NOT crashed at game load now... only sometimes infinite loading like stuff... but its my setup i believe so not related.

 

wonderfull!

I noticed you posting on my thread too. If using slal is causing big problems for you let me know. I can help you delete it from your data files.

 

 

thx but slal itself is not causing problems i gues... and i am using mod organizer so i easy can delete it. but slal packs are not felxible enouth atm. so for example to have lesbian dildo animations and some object related animations i still need to use nsap too. thats why i have 4500 animations according to Fnis. so maybe thats what prob is. but still since i am using crash fix plugin i have no ctds anymore not ingame and not onload. so i gues i can live with slal and nsap at the same time.

 

[TamaChan]

 

 

its amazing ... you sir are a NEO of skyrim ... i have verry buggy game game but even with 200 mods ...legends of dragonborn... etc... i have NOT crashed at game load now... only sometimes infinite loading like stuff... but its my setup i believe so not related.

 

wonderfull!

I noticed you posting on my thread too. If using slal is causing big problems for you let me know. I can help you delete it from your data files.

 

 

thx but slal itself is not causing problems i gues... and i am using mod organizer so i easy can delete it. but slal packs are not felxible enouth atm. so for example to have lesbian dildo animations and some object related animations i still need to use nsap too. thats why i have 4500 animations according to Fnis. so maybe thats what prob is. but still since i am using crash fix plugin i have no ctds anymore not ingame and not onload. so i gues i can live with slal and nsap at the same time.

 

 

Sorry, I was tired when I typed that. I meant to put nsap, anyway. I meant that, if you know you won't use the animations, you can delete them from nsap, or any slal pack. I only use about 30 of the sl animations in nsap. Those 30 total about 100-150. I delete the other about 800 animations. There are no negative side effects that I have noticed, it's just a lot of work. I do this with my slal packs too. Total I have about 9000 animations, but at least a couple thousand of those don't get read by fnis. And true, I was having the same problem which is why I grabbed the crash fix plugin, but if you are someone that uses save tools, then the crash fix plugin is only a band-aid, not a permanent solution. 

 

edit: and if you are talking about plugin space, I am doing some testing right now, I am pretty sure that the plugins for nsap's objects, leito's objects, and the universal slal anim. object plugin can be merged. Still doing more testing though, so can't say for sure.

[Dave0000]

 

 

 

its amazing ... you sir are a NEO of skyrim ... i have verry buggy game game but even with 200 mods ...legends of dragonborn... etc... i have NOT crashed at game load now... only sometimes infinite loading like stuff... but its my setup i believe so not related.

 

wonderfull!

I noticed you posting on my thread too. If using slal is causing big problems for you let me know. I can help you delete it from your data files.

 

 

thx but slal itself is not causing problems i gues... and i am using mod organizer so i easy can delete it. but slal packs are not felxible enouth atm. so for example to have lesbian dildo animations and some object related animations i still need to use nsap too. thats why i have 4500 animations according to Fnis. so maybe thats what prob is. but still since i am using crash fix plugin i have no ctds anymore not ingame and not onload. so i gues i can live with slal and nsap at the same time.

 

 

Sorry, I was tired when I typed that. I meant to put nsap, anyway. I meant that, if you know you won't use the animations, you can delete them from nsap, or any slal pack. I only use about 30 of the sl animations in nsap. Those 30 total about 100-150. I delete the other about 800 animations. There are no negative side effects that I have noticed, it's just a lot of work. I do this with my slal packs too. Total I have about 9000 animations, but at least a couple thousand of those don't get read by fnis. And true, I was having the same problem which is why I grabbed the crash fix plugin, but if you are someone that uses save tools, then the crash fix plugin is only a band-aid, not a permanent solution. 

 

edit: and if you are talking about plugin space, I am doing some testing right now, I am pretty sure that the plugins for nsap's objects, leito's objects, and the universal slal anim. object plugin can be merged. Still doing more testing though, so can't say for sure.

 

 

would be nice to know how exactly on the easy way to get rid of unwanted animations from nsap and for example anubs pack from which i only use 3 animations at the moment

 

[TamaChan]

 

-snip-

 

Sure, a little busy right now, but when I get the chance, I'll make another thread, with the info there. It's really easy though. Personally I keep backups of everything so I can either move the hkx files or just delete them, but the idea is, you look at the json that comes with the pack(although some modders are really organized with their naming formats, so you might not need to) and look at the names of the positions. The positions you want, you put the related hkx files aside, then delete everything that remains(or move them somewhere else for backup), and move the files you do want back. Later I can write down a much more thurough tutorial on it.

[Dave0000]

 

 

-snip-

 

Sure, a little busy right now, but when I get the chance, I'll make another thread, with the info there. It's really easy though. Personally I keep backups of everything so I can either move the hkx files or just delete them, but the idea is, you look at the json that comes with the pack(although some modders are really organized with their naming formats, so you might not need to) and look at the names of the positions. The positions you want, you put the related hkx files aside, then delete everything that remains(or move them somewhere else for backup), and move the files you do want back. Later I can write down a much more thurough tutorial on it.

 

thanks

[Monoman1]

Looks like issue with the string table. Did this start right after going over the limit?

 

A new game for me would have started over the limit to begin with. I'm beginning to think the further you go over the limit the more pronounced the problem becomes as more strings are pushed into the expanded space and until it hits some critical variables like campfires version variable you only get log errors. 

 

When I start a new game I save as I go through the MCMs and initialize/configure mods. Went back into the game there and: 

loaded the first save I made and seems to have loaded no problem (in game at least)

loaded the second save -> again, seemingly no problem. 

Loaded the third save and I start getting debug.messagebox errors from a lot of different mods and the papyrus log explodes. 

 

I may have added more mods as I can't remember my load order exactly. 

 

Massive papyrus log ahoy: Papyrus.0.log.7z

Search for "Reverting game" to separate save loadings. 

 

Here is the corresponding crash fixed save log: 

 

 

08 May 15:32:40 Read from storage #1: 65535

08 May 15:32:40 Read from storage #2: 67359

 

08 May 15:33:21 Read from storage #1: 65535

08 May 15:33:21 Read from storage #2: 71151

 

08 May 15:33:50 Read from storage #1: 65535

08 May 15:33:50 Read from storage #2: 75388

 

I'd wait for someone else to confirm first though as I'm only rebuilding my load order at the moment. 

[h38fh2mf]

I'll try to see if I can recreate the problem.

[markdf]

My guess, based on the description of the problem and the papyrus log posted, is that this is related to Skyrim's autovariable optimization.

 

If you're not familiar with it, what happens is that Properties with the AUTO flag have a variable generated for them -- an autovariable or autovar. Within that script, any attempt to access the property actually accesses the autovar directly.

 

The "optimization" is that subclasses of that script ALSO directly access the autovar instead of the property. And so do the getVMQuestVariable and getVMObjectVariable functions in CreationKit.

 

My guess is that one or both of those pathways use a 16 bit index into the string table.

 

It seems clear that MOST string indices are already 32 bit, otherwise Crash Fixes v8 wouldn't have worked at all. But this bug indicates that there are at least a few 16 bit indices left.

 

That's my guess anyway, based on my own misadventures with string tables.

[Monoman1]

I've been trying to recreate it with a simpler load order + the save breaker mods. Be careful if using the strcount save breaker. The variables versions have a habit of ending up on the suspended stack so they won't increase the string table. 

[Monoman1]

Doubled the number of properties in the save breaker to 80,000. Still can't seem to break frostfall on a smaller load order. 

08 May 17:39:53 Read from storage #1: 65535

08 May 17:39:53 Read from storage #2: 116002

 

Don't know what the difference is...

[markdf]

Which version of save breaker is it? The one that makes thousands of local variables, or the one that has thousands of properties?

Nevermind, you already stated.

What order are the loading in? Maybe try putting save breaker at the top of your load order so that its strings get loaded before Frostfall's.