Corrupted Saves

[Monoman1]

Would this help? Patches the Papyrus Compiler to only run in 32-bit mode.

http://www.nexusmods.com/skyrim/mods/65578/?tab=1&navtag=http%3A%2F%2Fwww.nexusmods.com%2Fskyrim%2Fajax%2Fmoddescription%2F%3Fid%3D65578%26preview%3D&pUp=1

Nope. It helps run the Script compiler under mod organizer because MO can not yet run 64 bit applications. But it's got nothing to do with the save file format.

 

The problem here isn't the scripts. It's the 'container' used to hold them is the problem. Ie the save.

[BigOnes69]

 

Would this help? Patches the Papyrus Compiler to only run in 32-bit mode.

http://www.nexusmods.com/skyrim/mods/65578/?tab=1&navtag=http%3A%2F%2Fwww.nexusmods.com%2Fskyrim%2Fajax%2Fmoddescription%2F%3Fid%3D65578%26preview%3D&pUp=1

Nope. It helps run the Script compiler under mod organizer because MO can not yet run 64 bit applications. But it's got nothing to do with the save file format.

 

The problem here isn't the scripts. It's the 'container' used to hold them is the problem. Ie the save.

 

 

I am not saying this effects saves but that if uou can lock the compiler into 32 bit mode can you not do something similar with the save format. A compiler for the saves,  Or for that matter create an add on emulator and spoof the save format into saving the strings in a separate file that all it does is recognize the higher count. I dont know but all kinds of emulators have been created over the years for various programs and games.

[endersgame]

Ugh. I've managed to trim my game down to about 48K, but I'm still reluctant to start a new play-through, especially since I've no idea the rate the strings will grow. At the current rate, it looks like its growing by about 100 every 10 min...

[markdf]

Ugh. I've managed to trim my game down to about 48K, but I'm still reluctant to start a new play-through, especially since I've no idea the rate the strings will grow. At the current rate, it looks like its growing by about 100 every 10 min...

 

A lot of that is one-time increases. Some variables and properties aren't initialized, some scripts aren't loaded yet, and so on. The increases should taper off as you play... unless you start uninstalling mods and adding new ones, and whatnot.

 

[endersgame]

 

Ugh. I've managed to trim my game down to about 48K, but I'm still reluctant to start a new play-through, especially since I've no idea the rate the strings will grow. At the current rate, it looks like its growing by about 100 every 10 min...

 

A lot of that is one-time increases. Some variables and properties aren't initialized, some scripts aren't loaded yet, and so on. The increases should taper off as you play... unless you start uninstalling mods and adding new ones, and whatnot.

 

 

 

Yeah, that was my gut feeling. Probably will play the save for a few hours to see if the rate changes. 

[domp]

So any news from skse team or beth developers?

[Guest]

So any news from skse team or beth developers?

 

Bethesda development abandoned Skyrim a while ago. They are busy on the current selling games.

About SKSE team, they are still active, but their priority is F4SE right now.

[Monoman1]

We've discovered this at pretty much the worst time possible really. I must be the only one who can't "get into" fallout 4 at all and I played the first two a lot. Played 12 hours of 4 and haven't felt the need to return.

 

Anyone know any good sources for assembler coding?

[markdf]

We've discovered this at pretty much the worst time possible really. I must be the only one who can't "get into" fallout 4 at all and I played the first two a lot. Played 12 hours of 4 and haven't felt the need to return.

 

Anyone know any good sources for assembler coding?

 

I suspect LOTS of people are waiting to play Fallout 4, so that Bethesda can finish patching it, for the unofficial patch, for the scripting extension, for ENBoost, and all the other good stuff that we take for granted with Skyrim.

 

Raising awareness of this issue is the best thing we can do, short of going out and learning how to trace your way assembly code.

[Monoman1]

Hope you're right. 

 

Btw Mark can you fix the OP. Wasn't actually me that figured it out. It was vyper00 back in post 74. 

[Monoman1]

Guys, I've posted up on the nexus forum in a bid to expand awareness of this issue. 

 

http://forums.nexusmods.com/index.php?/topic/3924850-corrupt-saves-strcount-0xffff-ctd-on-load/

 

If I've left something out let me know. 

 

Edit: I've also PMed Boris on it, so don't everyone start PMing him. I'd say the amount of shit in his inbox would make you sick. 

[markdf]

Hope you're right. 

 

Btw Mark can you fix the OP. Wasn't actually me that figured it out. It was vyper00 back in post 74.

 

How on Earth did I mix that up?

 

Guys, I've posted up on the nexus forum in a bid to expand awareness of this issue. 

 

http://forums.nexusmods.com/index.php?/topic/3924850-corrupt-saves-strcount-0xffff-ctd-on-load/

 

If I've left something out let me know. 

 

Edit: I've also PMed Boris on it, so don't everyone start PMing him. I'd say the amount of shit in his inbox would make you sick.

I added a link to that post and a note that Boris has already been contacted.

[Centrophy]

Guys, I've posted up on the nexus forum in a bid to expand awareness of this issue. 

 

http://forums.nexusmods.com/index.php?/topic/3924850-corrupt-saves-strcount-0xffff-ctd-on-load/

 

If I've left something out let me know. 

 

Edit: I've also PMed Boris on it, so don't everyone start PMing him. I'd say the amount of shit in his inbox would make you sick. 

 

Immediately there's a "It doesn't happen to me, debunked!" post. I swear, the internet brings out the worst [in] people.

[Monoman1]

Hmm. I've stumbled on to something interesting. 

 

In an effort to provide the doubters with easily reproducible proof of this issue I've created a small mod to break your saves, yay!

 

The mod simply declares 60,000 integers and holds the function in a loop for 30,000 seconds so the variables are persistent and not just discarded once the function completes. I was unable to declare the full 60,000 variables in a single script. The script either ended up on the suspended stack or did not seem to run at all with all the variables in one script. So I created 3 scripts with 20,000 declarations each.

 

Now the interesting part. 

Initially I had the script variables named the same across all 3 scripts. 

 

Eg:

Script 1 had a variable named strCountTestVariable0

Script 2 had a variable named strCountTestVariable0

Script 3 had a variable named strCountTestVariable0

etc

 

Number of total strings created by the mod = 20,000 => Save NOT corrupt. 

 

I changed each script to use unique variable names.

Eg:

Script 1 had a variable named strCountTest1Variable0

Script 2 had a variable named strCountTest2Variable0

Script 3 had a variable named strCountTest3Variable0

etc

 

This time it looks like I got the desired result. Save becomes corrupt and we can presume 60,000 strings were created. 

 

How this is interesting:

If mod authors were to use a common set of variable names it seems strCount would be reduced accordingly. It would be very far from ideal for mod authors though. 

 

StrCount Save Breaker.7z

To break your saves:

1. Install mod on vanilla or any load order. 60,000 string definitions should break any load order. 

2. Coc qasmoke in the console at the main menu. Or load your favorite save. Should break anything assuming the scripts actually run. 

3. Save your game and join the broken save club, :*(

 

[markdf]

Hmm. I've stumbled on to something interesting.

 

In an effort to provide the doubters with easily reproducible proof of this issue I've created a small mod to break your saves, yay!

 

 

Variables usually aren't the problem; I doubt there are ever more than a hundred variables in use at a time (not counting your save-breaker). Script properties are what take up most of the space. A mod like Interesting NPCs has thousands of scripts; most only have one or two properties, often with the same names (playerref usually). But internally, I suspect the script's name is part of the property's name, so it's a new string. The script itself may also get a string to store its name. There's no way around this except to eliminate the property or the entire script. And I'm told that for dialogue scripts (which are the vast majority of scripts), that's impossible. So reusing variable names wouldn't accomplish much.

 

You could try your experiment again using 20,000 properties instead of 20,000 variables, and confirm (or who knows, maybe disprove) what I've said.

 

What I'd like to try and do is make a patch for Sexlab to bring down the string count for animations, from 12 down to maybe 2 or 3. It takes about 2000 strings to activate sexlab, and at least half of that is for animations.

[Monoman1]

 

Hmm. I've stumbled on to something interesting.

 

In an effort to provide the doubters with easily reproducible proof of this issue I've created a small mod to break your saves, yay!

 

Variables usually aren't the problem; I doubt there are ever more than a hundred variables in use at a time (not counting your save-breaker). Script properties are what take up most of the space. A mod like Interesting NPCs has thousands of scripts; most only have one or two properties, often with the same names (playerref usually). But internally, I suspect the script's name is part of the property's name, so it's a new string. The script itself may also get a string to store its name. There's no way around this except to eliminate the property or the entire script. And I'm told that for dialogue scripts (which are the vast majority of scripts), that's impossible. So reusing variable names wouldn't accomplish much.

 

You could try your experiment again using 20,000 properties instead of 20,000 variables, and confirm (or who knows, maybe disprove) what I've said.

 

What I'd like to try and do is make a patch for Sexlab to bring down the string count for animations, from 12 down to maybe 2 or 3. It takes about 2000 strings to activate sexlab, and at least half of that is for animations.

 

That is true. 

For demonstration purposes however, the outcome is the same - strCount > 0xFFFF. 

 

I will look into making a properties version though isn't it the same concept. Name the properties the same.... ? Aren't they identified by their formID rather than the string assigned to their name? Genuine question as I've not used the CK for some time now. 

 

Edit:

For example. Instead of naming animations:

 

Leito_Amazing_animation_1_Stage_1

&

FunneyBusiness_Amazing_Animation_1_Stage_1

 

just have

Amazing_Animation_1_Stage_1

Amazing_Animation_1_Stage_1

 

 

Edit 2:

Creating a properties version may take more time. There seems to be a limit of around 1023 when opening the properties screen of the CK (Error while reloading script). And strings don't seem to translate to the game either unless this window can open. 

[Monoman1]

Script properties version of the strCount breaker: 

StrCount Save Breaker (Properties Edition).7z

 

Uses 40 scripts attached to 1 quest adding 1000ish properties each for a total of 40,000 strings. Should be enough to break most saves unless maybe your'e not running the official DLC. Note there is absolutely nothing happening in the scripts - zero looping or processing. Just properties that are not even filled. 

[markdf]

Script properties version of the strCount breaker: 

StrCount Save Breaker (Properties Edition).7z

 

Uses 40 scripts attached to 1 quest adding 1000ish properties each for a total of 40,000 strings. Should be enough to break most saves unless maybe your'e not running the official DLC. Note there is absolutely nothing happening in the scripts - zero looping or processing. Just properties that are not even filled.

Nice job, works exactly as advertised. :-)

[markdf]

Script properties version of the strCount breaker: 

StrCount Save Breaker (Properties Edition).7z

 

Uses 40 scripts attached to 1 quest adding 1000ish properties each for a total of 40,000 strings. Should be enough to break most saves unless maybe your'e not running the official DLC. Note there is absolutely nothing happening in the scripts - zero looping or processing. Just properties that are not even filled.

I tried my own version, but instead of having lots of properties I only used three properties in two scripts. But using the Papyrus 2D array trick, those three properties formed a 128x128 array of strings, which increased the string count by a little over 16390. So 128 x 128 + 6.

 

16384 : a 128x128 array

        2 : two script names

        3 : three properties

        1 : mysterious extra string

------------

16390 : total

 

So these tests prove that the following things increase the string count:

• Properties, even properties with the same names.
• Local variables in running scripts.
• Strings, even when they are stored in arrays.

 

My test also showed that a script's properties only count once, even if you create hundreds of instances of it. If every instance of my subarray had its properties counted, the increase in the string count would have been much higher.

 

--

I did a second test where I made an array of JArrays to contain my strings, and that version used only 6 strings instead of 16390. So putting stuff into JContainers definitely works.

 

Now we need to petition Ashal to modify a few of the Sexlab scripts to use JArrays instead of regular string arrays. :-D

[Monoman1]

Hmm, ok I just changed the scripts in the properties version of the strcount breaker to use the same property names in each script. Our results are not the same...

It only created around 1000 strings for me. What's the difference? Maybe there should be different mods per script or quests? 

 

strCount Save Breaker (Script Properties Version. Same names in each script).7z

 

Edit: Just a mad thought that popped into my head (Getting way ahead of myself). If it's the case that using the same property names reduces string count how viable would it be to create a compiler for the compiler? - A compiler that changes properties to use a set of common names (property1, property2 etc) and then runs it through the papyrus compiler. Filling those properties is a problem though. You would want to keep separate development version of your mod and a compiled version. Headache. 

 

Edit 2: 

Just trying to make sense of the string table here. 

It looks like anything starting with :: is either a variable or a script property. 

and anything without :: is either a script name, fragment name or some built in skyrim function. The question I have is why some scripts are listed and not others. There are definitely some fragments in there but nowhere near the number you would expect. Interesting NPCs has over 6000 fragments beginning with tif_*.psc. Quest isn't active maybe?

[markdf]

Edit: Just a mad thought that popped into my head (Getting way ahead of myself). If it's the case that using the same property names reduces string count how viable would it be to create a compiler for the compiler? - A compiler that changes properties to use a set of common names (property1, property2 etc) and then runs it through the papyrus compiler. Filling those properties is a problem though. You would want to keep separate development version of your mod and a compiled version. Headache.

 

It's sometimes called a precompiler. But it doesn't need to be anything fancy, a Python script could do the job. It would probably never be something that mod developers use; more likely it would be a tool that people like ourselves who are having these issues could use ourselves. Do you know Python?

[Monoman1]

Nope. I know a little bit of C++, java, papyrus but not enough of any one of them to make me an expert. 

[markdf]

Nope. I know a little bit of C++, java, papyrus but not enough of any one of them to make me an expert.

It would be good to try and find a volunteer to research this.

 

I'm working on my sexlab patch. I've got the per-animation string cost down from 12 to 1.7, and I'm pretty sure that I can get it down to 0.7. Once I'm finished I'll post it here for people to test and send a copy to the SexlabFramework developers in case they want to incorporate the fix.

 

The results seem really promising so far. :-)

[Monoman1]

That's music to my ears (eyes)

[FoxFingers]

Edit 2: 

Just trying to make sense of the string table here. 

It looks like anything starting with :: is either a variable or a script property. 

and anything without :: is either a script name, fragment name or some built in skyrim function. The question I have is why some scripts are listed and not others. There are definitely some fragments in there but nowhere near the number you would expect. Interesting NPCs has over 6000 fragments beginning with tif_*.psc. Quest isn't active maybe?

 

As far as I can tell, just by looking at my own stuff in the table, a Script Name only shows up in that table if it is used as a property.

 

That is, your generic setstage type script on dialogue does not seem to show up in there as far as I can tell.

 

Quest Aliases

Message Variables

Quest variables

Script Properties

Script Function Names

 

Those can show up in there.

 

It seems like it might be possible to alleviate for a Modder by trying to use the same name over and over.  But really, the reason you give things unique names is to know what it is and what you are doing with it.

 

Imagine you could only use the phrase "New Folder" in Windows Explorer and then try and keep track of what's in all the "New Folder"(s) in all the directory structures everywhere.  That's a heavy burden and not particularly realistic.

 

That said, being aware of the issue, I know I could do some optimization in my own Mod but it would require a complete rewrite, new testing, and would not be something anyone could use outside of a brand new game.