Help test fix for save game string table overflow

[markdf]

I've started work on a new save editor. So far I can display the screenshot from the savegame... :-)

[h38fh2mf]

I don't know what the cause is. The pointer indices definitely don't get messed up, tried save / load and all the index still point to correct string (checked manually). It also doesn't seem to be a mishandling of 16 bit value because if script says that some random wrong variable name was not found on script and I check the corresponding value (either + 0x10000 or & 0xFFFF) then the variable name still doesn't make sense for that script. Also it's not only >= 65536 or < 65536 that gets messed up, sometimes variable with index 2300 is reported as error and sometimes 72000 so the index doesn't seem to affect it. Another thing that might cause it is if the save file format is incorrectly read, however this doesn't seem to be the case either because it only fails to read papyrus data, if save game was messed up it wouldn't be able to read anything after that and report as corrupt.

 

Edit: when I use the (variable) corruptor I don't get any errors at all even if it's loaded first in load order. when I use (property) corruptor I get loads of errors, what's up with that?

[Monoman1]

Are the scripts for the variable version ending up on the suspended stack? The variables versions do that sometime.

[h38fh2mf]

Perhaps, but the string count is still high so it doesn't matter where they are? Property version: 75k strings and big problems, variable version: 95k no problems. With variable version the scripts continue to run after (other scripts too), with property version it fails to completely load the papyrus section - that's where the errors come from (during loading), variable version has no errors during loading and the usual stuff after. I'm not sure why

[Mookeylama]

Perhaps, but the string count is still high so it doesn't matter where they are? Property version: 75k strings and big problems, variable version: 95k no problems. With variable version the scripts continue to run after (other scripts too), with property version it fails to completely load the papyrus section - that's where the errors come from (during loading), variable version has no errors during loading and the usual stuff after. I'm not sure why

so are these problems that might occur during normal gameplay, or errors just caused by these "script breaker" things?

basically i was going to continue my current playthru with v8 of Crash fix. or would it be better to wait and see how this new string predicament develops??

 

[h38fh2mf]

I'm not sure, you can disable the string count fix in the INI if you wish, then the save would just become corrupt instead of you not knowing that the scripts are messing up.

[Monoman1]

Perhaps, but the string count is still high so it doesn't matter where they are? Property version: 75k strings and big problems, variable version: 95k no problems. With variable version the scripts continue to run after (other scripts too), with property version it fails to completely load the papyrus section - that's where the errors come from (during loading), variable version has no errors during loading and the usual stuff after. I'm not sure why

Hmm. In my experience when the scripts ended up on the suspended stack the string table wasn't affected so saves weren't corrupt. 

[markdf]

I've been writing a new savegame editor, and it has made me concerned about whether the new savefile format can work. All of the string indices in the save format are 16 bits. That includes script names, script types, variable types, variable names, etc. None of them can point to any string past 65535. I think the save format needs a lot more changes for this to work.

[Dave0000]

i am using this fixes and restringer and the heavy loaded game with 200+ mods gives me now ~15 frames more then usual and all stuff is much more quickier... i just say it works some how

[h38fh2mf]

Yes all pointers to string table in save file are 16 bit, the patch addresses all them since game uses one method to write all pointers.

[markdf]

Yes all pointers to string table in save file are 16 bit, the patch addresses all them since game uses one method to write all pointers.

 

So every string table index in the save file format becomes 32 bit? Is that only true when the string table exceeds 65536? I'm making my new save editor, and I want to build in compatibility for your new format.

[h38fh2mf]

The format is this:

 

Old function:

writeValue(stream, value)
{
stream.writeUInt16(value);
}

readValue(stream)
{
return stream.ReadUInt16();
}

New function:

writeValue(stream, value)
{
if(value > 65520) // 0xFFF0
{
stream.WriteUInt16(0xFFFF);
stream.WriteUInt32(value);
}
else
stream.WriteUInt16(value);
}

readValue(stream)
{
result = stream.readUInt16();
if(result == 0xFFFF)
result = stream.readUInt32();
return result;
}

This is for both the size of string table and also the pointer index.

[markdf]

Thanks!

[Mookeylama]

so this string fix is sorta dead? shame as it woulda been invaluable to games in progress.

[Ollie]

All right, it works and I have to say one thing: you guys are GOLDEN. Thank you for what you do.

[h38fh2mf]

Did more research, it seems to write two strings on the same index, and only happens sometimes. Not sure why, example:

Write: [006673 / 73878] "dbrecurringquestscript"
Write: [006673 / 73878] "NoRedress"

In the same save writes 6673 index for string "dbrecurringquestscript" and then also writes index 6673 for string "NoRedress", when the save is loaded then both strings will be read as "NoRedress", very few strings are mangled this way (about 1% in that save), looking further for cause.

[h38fh2mf]

Ok nevermind, fixed, releasing update soon.

[Mookeylama]

wait. seriously?? it's fixed?! weeee! i would so looove to continue with my current save.

 

i hate to say it, because then i have to acknowledge it...but i was recently diagnosed w/ cancer and i'm afraid i'll be to sick from chemo, or worse, to start another playthru for awhile

[h38fh2mf]

That sucks.

 

It should be fixed, did some tests although can't say for sure, since last time I thought it was fixed too.

[Mookeylama]

That sucks.

 

It should be fixed, did some tests although can't say for sure, since last time I thought it was fixed too.

 

thanks. yeah it sucks

 

maybe Monoman will give it some testing. i still gotta get some merges and other unrelated crashes in order before i can move forward.

and hopefully Markdf will continue w/ his save game leaner since we'll need that

[markdf]

 

That sucks.

 

It should be fixed, did some tests although can't say for sure, since last time I thought it was fixed too.

 

thanks. yeah it sucks

 

maybe Monoman will give it some testing. i still gotta get some merges and other unrelated crashes in order before i can move forward.

and hopefully Markdf will continue w/ his save game leaner since we'll need that

 

On that note, I could use a few sample of savegames that require cleaning, including notes on what kinds of specific things to be looking for -- so far my tool can read savegames, but I haven't added logic for detecting errors yet.

[Monoman1]

Would love to give the new version a test drive but won't be able to until next week. Stupid work. 

[Monoman1]

So I finally had a chance to try out the new version this morning. 

Looks like the issues that the previous version had are gone. No missing variables when loading save reported in the papyrus log that I could see. Started a new game and this time had no problem initializing mods in the MCMs and configuring them. Managed to play Skyrim for 3 or 4 hours straight without a single crash which is damn near unprecedented in my setup. 

Massive kudos to you and mark for kicking the crap out of this problem

[h38fh2mf]

Cool, thanks for testing.

[galgat]

  This is all great stuff you all !

 

Really waiting in the wings for your save editer Markdf

 

 

I have just sense the latest animation SLAL and NonSexlabAnimationPack3.1 started to get corruption. The only way I have avoided it is to save in a smaller cell, and keep my Mods very low on each profile.

 

for instance I can run Deviously cursed loot, and SLAL and NonSexlabAnimationPack3.1,  but if I try to install Estrus, even with out the ESP ( just for the resources ), and Estrus Chaurus v4.20a, I will start to get corrupt save files, almost every time, which is sort of sucky .

 

   So I am, very hopeful for some save editor function for this project, as to often one need to clean the saves of mods you decide to dump, or update.

 

  Thx U all for your Hard work.