PapyrusUtil LE/SE/AE

[h38fh2mf]

You take over OP and whole mod it's easier that way for both of us.

[darkconsole]

am i misunderstanding how things like FloatListSet() work? i have a timer that ticks every ten seconds doing:

FloatListSet(who,"name",iter,(FloatListGet(who,"name",iter)+0.01))

and each loop it is showing as the initial value and not incremented as i was expecting. the weird thing is i thought it was working for a while. i was wondering if im hitting some sort of concurrency bug or if i am just stupid.

 

i've tried on 2.3 and the 2.4 that was posted a few posts up for testing. its broke in both, so i have to just be bad.

[Ashal]

Nothing sticks out at me as to why it wouldn't work just looking at your pseudo-code example, would have to see the actual script to give you any real answer. The problem is likely with the actual implementation of the line and not that one line itself.

[darkconsole]

	a = 0
	While a < gcount
		gvalue = (FloatListGet(who,"sgoGemData",a) + (diff / 2))
		If gvalue > 6
			gvalue = 6
		EndIf

		FloatListSet(who,"sgoGemData",a,gvalue)
		gweight += gvalue

		a += 1
	EndWhile

what i think was happening is that if FloatListGet, and if iter does not exist, the fact it returns 0 makes it impossible to tell if its 0 because it didn't exist or 0 because it was 0, and FloatListSet with an invalid iter doesn't make it exist. that is what i THINK happened. i ended up deleting the entire dataset i had stored already and let the mod create it from the beginning and i was getting my increments again. i also rewrote the code to use FloatListCount instead of keeping my own counter.

 

so i think i was incrementing a 0 that meant it did not exist, and then trying to store it to a place that still did not exist.

[Ashal]

ListSet/Get expect the iterator to already exist, if it doesn't it can't set/get a value there. 

[darkconsole]

right. and i think that was my problem, i think somehow mycounter got out of sync with reality so i was reading iterators that were far out of bounds. the problem is the return value of 0 for errors.

would be it possible to get another argument, errorCode=whatever for these things? for example, my mod only deals with positive numbers, so ListSet(..., errorCode=-1) would allow me to decide how to flag an actual problem = unless papyrus allows "None" for primitive types like int and bool. then None would be a very valid invalid return code for all the gets.

 

sorry if this is all daft, i am a backend web developer. my life is frameworks and error codes, hehe. maybe not best suited for game development and lowest framerate impact >_>

 

[update] so i see the flat value methods have that.

float function GetFloatValue(Form obj, string key, float missing = 0.0) global native

but not the list values methods.

float function FloatListGet(Form obj, string key, int index) global native

*puppy dog eyes*

[Ashal]

Adding another argument onto an existing function would break backwards compatibility for any mods currently using the functions. 

 

Just do the smart/safe thing and pre-validate your data before trying to using it.

[darkconsole]

adding a new one shouldn't break it. removing one would.

float function FloatListGet(Form obj, string key, int index, float missing = 0.0) global native

should not break anything. default behaviour maintained, new usability and error handling feature added. oh and now it would be consistant with GetFloatValue. i see only win here.

 

also, progress has to be made some day.

[TnTTnT]

New Sexlab1.58b use Papyrusutil2.3 and some user including me has problem that the game won't start.  Luckily Ashal still support 2.2 in his code so I can use 2.2 with Sexlab 1.58b to start the game normally.

 

Is there any suggest method to get the game run with Papyrusutil2.3 for someone like me?

 

Still use old PC with WindowsXP SP3 myself. 

[Srende]

adding a new one shouldn't break it. removing one would.

float function FloatListGet(Form obj, string key, int index, float missing = 0.0) global native

should not break anything. default behaviour maintained, new usability and error handling feature added. oh and now it would be consistant with GetFloatValue. i see only win here.

 

also, progress has to be made some day.

The compiler adds the default parameters on every function call if they're not specified. So, all mods compiled with an older version would pass one argument less than needed, breaking the compatibility.

[Ashal]

adding a new one shouldn't break it. removing one would.

float function FloatListGet(Form obj, string key, int index, float missing = 0.0) global native

should not break anything. default behaviour maintained, new usability and error handling feature added. oh and now it would be consistant with GetFloatValue. i see only win here.

 

also, progress has to be made some day.

 

 

It would break it, it forces every mod that uses it to recompile to the new source scripts, breaking any mods on the older version and vice versa. 

 

It also wouldn't solve your problem. They already return default values if they can't retrieve, your problem would still remain with the fact that you aren't initializing the list or validating the size before trying to loop through it.

[Ashal]

New Sexlab1.58b use Papyrusutil2.3 and some user including me has problem that the game won't start.  Luckily Ashal still support 2.2 in his code so I can use 2.2 with Sexlab 1.58b to start the game normally.

 

Is there any suggest method to get the game run with Papyrusutil2.3 for someone like me?

 

Still use old PC with WindowsXP SP3 myself. 

 

http://www.loverslab.com/topic/18708-sexlab-framework-development/?p=866829

[darkconsole]

 

It also wouldn't solve your problem. They already return default values if they can't retrieve, your problem would still remain with the fact that you aren't initializing the list or validating the size before trying to loop through it.

 

 

it would solve my problem. in this case, i would say missing = -1, since 0 is a valid number to math against. i'd have saved a few hours of bashing my head against this 0+0.1 = 0 problem if i had seen my -1 come across.  granted i fixed my code. now im just arguing for new features and consistency across the functions.

 

each developer would be able to choose the invalid code just like they can for GetWhateverValue now.

[Ashal]

They can accomplish exactly this by simply doing index < FloatListCount(obj, "key") before trying to pull values in the first place. Aside from breaking backwards compatibility, moving error codes into the arguments of the Get does nothing but move where the if condition is performed (before vs after), promoting bad practices.

 

It's simply a better practice to validate your input before trying to use it, rather than using invalid inputs and then checking the result.

[h38fh2mf]

Like Ashal says there is already a way to check if value exists and is valid for every type. For list it's "index >= 0 && index < list.count" and for any normal value it's HasFloatValue(form, "test")

 

No need to change existing functions.

 

If you want the "missing" argument so much you can implement this on your end. For example if you have values from 0 to 6 then make values from 1 to 7 and when getting value make GetValue - 1, this way you will have -1 if value is error and 0 to 6 any other way.

[Aleanne]

Hi there,

 

I'm having a strange issue. I'm not sure it's related to PapyrusUtil, but it would seems so.

 

I installed PapyrusUtil24 on 07/24. I didn't had any problem I could relate to the 23 version, but it's new and shiny, so it must be better, right ?

 

Today, I noticed that my skse save games were rapidly growing in size. In fact, they seem to double, but not at every save.

 

 

 

26/07/2014  15:06            18 039 Save 7022 - Gwen  Abandoned Prison  00.27.56.skse

26/07/2014  15:08            20 162 Save 7022 - Gwen  Abandoned Prison  00.29.36.skse

26/07/2014  15:08            20 162 Save 7023 - Gwen  Druadach Redoubt Cave  00.30.19.skse

26/07/2014  15:23            21 665 Save 7022 - Gwen  Druadach Redoubt Cave  00.45.15.skse

26/07/2014  16:09           251 301 Save 7022 - Gwen  Druadach Redoubt Cave  01.15.34.skse

26/07/2014  16:14           251 331 Save 7022 - Gwen  Druadach Redoubt Cave  01.20.04.skse

26/07/2014  16:18           251 323 Save 7022 - Gwen  Druadach Redoubt Cave  01.23.44.skse

26/07/2014  16:23           472 379 Save 7022 - Gwen  Druadach Redoubt Cave  01.28.04.skse

26/07/2014  16:29           472 939 Save 7022 - Gwen  Druadach Redoubt Cave  01.33.49.skse

26/07/2014  16:36           473 379 Save 7022 - Gwen  Druadach Redoubt Cave  01.40.37.skse

26/07/2014  18:04           469 575 Save 7022 - Gwen  Skyrim  02.05.03.skse

26/07/2014  18:16           469 567 Save 7022 - Gwen  Skyrim  02.17.17.skse

26/07/2014  18:21           475 459 Save 7022 - Gwen  Skyrim  02.22.01.skse

26/07/2014  18:31           475 451 Save 7022 - Gwen  Druadach Redoubt Cave  02.31.56.skse

26/07/2014  20:15           471 663 Save 7022 - Gwen  Druadach Redoubt Cave  02.35.37.skse

26/07/2014  20:41         1 369 898 Save 7022 - Gwen  Druadach Redoubt Cave  02.57.34.skse

27/07/2014  00:07         1 372 077 Save 7022 - Gwen  Skyrim  03.08.37.skse

27/07/2014  00:09            19 279 Save 7022 - Gwen  Skyrim  03.11.01.skse -> Probably a failed save, corresponding ess is named ess.tmp
and I had two crashes on save these last few days.

27/07/2014  00:18         1 374 075 Save 7022 - Gwen  Harmugstahl  03.12.55.skse

27/07/2014  00:28         4 063 428 Save 7022 - Gwen  Editor Smoke Test Cell  03.13.27.skse

27/07/2014  00:51         1 376 147 Save 7022 - Gwen  Darkshade  03.13.41.skse

27/07/2014  00:54         2 720 770 Save 7022 - Gwen  Darkshade  03.14.57.skse

27/07/2014  01:27         1 374 067 Save 7022 - Gwen  Skyrim  03.16.21.skse

27/07/2014  02:17         2 717 066 Save 7022 - Gwen  Harmugstahl  03.16.29.skse

27/07/2014  02:22         2 717 042 Save 7022 - Gwen  Skyrim  03.20.59.skse

27/07/2014  02:54         2 719 007 Save 7022 - Gwen  Druadach Redoubt Cave  03.37.54.skse

27/07/2014  03:05         2 719 007 Save 7022 - Gwen  Druadach Redoubt Cave  03.48.27.skse

27/07/2014  09:21         5 414 156 Save 7022 - Gwen  Darkshade  03.16.41.skse

27/07/2014  09:30         5 414 924 Save 7022 - Gwen  Skyrim  03.25.10.skse

27/07/2014  13:57         8 097 827 Save 7022 - Gwen  Druadach Redoubt Cave  04.14.29.skse

27/07/2014  14:03         8 098 245 Save 7022 - Gwen  Druadach Redoubt Cave  04.20.15.skse

27/07/2014  14:13         8 098 730 Save 7022 - Gwen  Druadach Redoubt Cave  04.30.21.skse

27/07/2014  14:21         8 098 730 Save 7022 - Gwen  Druadach Redoubt Cave  04.38.57.skse

27/07/2014  14:31        16 176 917 Save 7022 - Gwen  Druadach Redoubt Cave  04.39.44.skse

27/07/2014  14:42        16 176 868 Save 7022 - Gwen  Druadach Redoubt Cave  04.50.45.skse
 
Next save done after reverting to PapyrusUtils contained in SL 1.58b 

27/07/2014  15:29            25 482 Save 7022 - Gwen  Druadach Redoubt Cave  04.51.01.skse

 

 

 

While looking into the save with an hex editor, I noticed this :

 

 

 

 

 

 

The bigger the number after "debuganimlist", the bigger the save is.

Uninstalling papyrusutilv24.zip (that would be SKSE\Plugins\StorageUtil.dll, scripts\MiscUtil.pex and scripts\StorageUtil.pex) and reverting to the components of SL 1.58b shrinked the skse save to a normal sive (20 kb or so).

 

What do you think ?

[Ashal]

Reverting to the old version would shrink your skse save because it invalidates any saved data from the previous version, effectively completely emptying any storage data you had.

 

My guess would be you have a mod installed that's not working properly with the new version and and instead of properly handling the error is repeatedly trying again and again, either failing to finish or properly see that it did in fact succeed the first time. 

 

Play with the new version, and once it bloats again send the .ess and .skse save files, as well as the StorageUtilDev.log file, which can be found in the root Skyrim folder.

[h38fh2mf]

There is some difference in string ListFind. In old implementation case insensitive search was performed but in new version case sensitive std::find is used so it's possible the script uses some sort of find and never finds then adds infinitely.

[Ashal]

In the rewrite version all keys are searched and stored as all lowercase, so find should be safe to use rather than the previous for loop + _strcmpi(), since keys and search values should match their lowercase equivalent via find. I haven't tested it with non-english characters however, there may be an issue there in relation to that change, I don't expect that to be much of an issue if it is though, and it probably would've been an issue _strcmpi() as well in that case.

 

Regardless though, if DebugAnimList is the one you suspect of bloating, than your install of 2.4 didn't go right. DebugAnimList is a string list used by PapyrusUtil 2.3 and older's debug mode in the ObjectUtil animation override system. In the test 2.4 rewrite version it's removed and no longer used in any way to the point that the word DebugAnimList doesn't appear anywhere in the source code anymore. So in a properly updated version it should be impossible to be increased at all, unless you have a lingering script instance still using the older dll file.

 

the version I posted for 2.4 is still considered alpha, I'm still in the process of rewriting and finalizing various things, It's been fine in my general test oriented gameplay, but I can't account for everything and still wouldn't recommend using it for general gameplay unless you're willing to potentially risk losing a save file.

 

In any case, I'm almost done finalizing it and will likely release an official update in the next couple days, quickly followed by a SexLab update that fixes scaling and makes use of the new StorageUtil features I've added/changed.

[h38fh2mf]

List find uses values to find index. Values are not converted to lower case in your implementation (as they shouldn't) so, I don't remember syntax but:

stringlist.add("Abc")

int index = stringlist.find("abc")
index is -1 because "Abc" != "abc"

_stricmp does case insensitive compare and std::find does case sensitive because values have to match exactly.

Same with stringlist.add if allowDuplicate is false then we will have both "Abc" and "abc" in list because they are not equal with std::find

This may be the difference that is causing infinite add.

Also debuganimlist is only used in debug build of old storage util, I don't know why someone would even be using this.
 

#if _DEBUG
if(curAnimStr)
{
std::string saveAnim = curAnimStr;
i_StringListAdd(NULL, NULL, BSFixedString("debugAnimList"), BSFixedString(saveAnim.c_str()), false);
}
#endif

Only possible way maybe if you have both DLL files? new and old in SKSE plugins folder.

 

[Ashal]

List find uses values to find index. Values are not converted to lower case in your implementation (as they shouldn't) so, I don't remember syntax but:

 

Ahh good point, though it shouldn't matter for int/float/form, I should probably switch the find and remove functions over to using loops instead of find specifically for the sake of strings. 

 

On a semi related though, I'm also considering changing the form storage system from form id's to strings of "baseFormID|modname.esp" instead, I have aboslutely no idea how the form load order functions all work though, hex and logical and/or shifting are all far beyond my grasp currently. Is there anyway to remove the load order number from the form id so it's assumed to be load order 00, then just using the current esp/esm name from the string to add on the current load order for the form. Basically replicating exactly how the papyrus Game.GetFormFromFile() works with base form id and mod file name.

 

In any case though, for it to be the source of an infinite loop the mod would honestly have to be doing something incredibly stupid. Not that it's not something it shouldn't account for though, I'll update the final version to use loops instead of find for find and remove functions.

 

This particular case though is definitely an issue of a botched install, the simply should be no way for it to still be calling the debuganimlist if they have a properly installed StorageUtil.dll. I'd suggest mainly waiting for the final version of the rewrite rather than using the test version I posted, but otherwise trying to repeat the bloat on a new game or killing active script instances on the save with a save script cleaner tool to insure the new versions of things take precedence after reinstalling the update, ensuring proper installation.

 

 

Just came across a major bug both in the current test version and all previous versions of PapyrusUtil, any string with 1027+ characters will crash the game when used with MiscUtil.PrintConsole(), noticed it after spending a solid 5 hours thinking the test implementation of my new list slice functions where causing it, only to angrily realize the 5 hours banging my head on it were wasted and the problem was actually with the debug log output of the slice to the console. I have it truncating PrintConsole() strings over 1000 characters now. 

[h38fh2mf]

Lets say you have full form ID in UInt32 fullForm to get load order into a UInt32 do this:

UInt32 loadOrder = fullForm >> 24;

and to get just the base form ID do this:

UInt32 baseForm = fullForm & 0x00FFFFFF;

And yes I think you should make the string search value with case insensitive, I always had some problems in papyrus where case randomly changed or wasn't what I wanted it to be.

 

std::find is basically same as iterating through all elements until find or end so it should not cause any slowdown either way other than casting to upper or however you plan to implement it.

 

I think the debug anim list issue may have been because the new DLL kept loading the old StorageUtil data into new format but the old data was not cleared or reappeared so every time the save game was loaded it reloaded the old data and kept doing that? That's why the size doubled? I dunno

 

[Aleanne]

Thanks guys.

 

That's strange. The character was created  after the upgrade to v24. Since it dumped the debugAnimList and since it doesn't exists in v23 and in v24, I must have run v22 all along. All installs are made with Wrye Bash. I know that dll are a sensitive point for Bash, so maybe something went wrong at some point.

 

In the last save I made (using v22), I'm having debugAnimList  in my skse save. It's ok, I guess. Only 169 items.

I upgraded to v24 (avoiding v23 because it does not seems stable, from what I heard). Double checked, I only have one StorageUtil.dll, 07/23/2014 5:55pm, 586 kb.

Loaded the last save, created a new save, debugAnimList  still there but with the same number than the one which was in the previous save, guess it's normal, no invalidation since I upgraded, data was persisted even if not used anymore.

 

I'll post here if something goes south again. Thanks again for the help

[b3lisario]

Hi,

 

I was testing SOS and I found a couple of problems with the papyrusutil v24.

I had installed a "papyrusutilv24.zip" dated of 24/07/2014. Then I saw there was another archive "PapyrusUtil_24.zip" in this post this post http://www.loverslab.com/topic/23713-papyrusutil/?p=864505 and also reproduced the problems with it

 

Issue #1

 

I have a skse co-save file generated using v23.

- Installed v24

- Load the savegame

No problems reading, though I did not check all the stored data. This is the skse.log relevant part:

 

 

loading co-save

Loading mod list:

(0 -> 0) Skyrim.esm

(1 -> 1) Update.esm

(2 -> 2) Unofficial Skyrim Patch.esp

(3 -> 3) Schlongs of Skyrim - Core.esm

(4 -> 4) Strap.esp

(5 -> 5) Aradia Kato Outfit.esp

(6 -> 6) Hooves.esp

(7 -> 7) RaceMenu.esp

(8 -> 8) Alternate Start - Live Another Life.esp

(9 -> 9) Brawl Bugs CE.esp

(10 -> 10) SkyUI.esp

(11 -> 11) TestCustomArmor.esp

(12 -> 12) SOS - B3lisario UNP Addon.esp

(13 -> 17) Schlongs of Skyrim.esp

(14 -> 19) SOS - Smurf Average Addon.esp

(15 -> 18) SOS - VectorPlexus Muscular Addon.esp

(16 -> 16) SOS - VectorPlexus Regular Addon.esp

(17 -> 14) SOSRaceMenu.esp

(18 -> 13) TestAlias.esp

Loading menu open/close event registrations...

Loading key input event registrations...

Loading control input event registrations...

Loading mod callback event registrations...

Loading crosshair ref event registrations...

Loading camera event registrations...

Loading actor action event registrations...

dispatch message (3) to plugin listeners

no listeners registered

cleared save path

 

 

- Quicksave

- Exit

At this point the new skse file is like corrupted. It seems there is random stuff between the storageutil data.

 

I'm attaching an archive with:

v23.skse: the original skse co-save

v24_2.skse: the new skse co-save

a PapyrusUtilDev.log

 

Didn't see any error in skse.log

 

co-saves.7z

 

 

 

EDIT 1:

What I thought of the skse co-save was wrong. I figured out the random stuff I am seeing is normal, it's something from the SKSE, like separator chars or so. I got confused because previous versions of storageutil hadn't these chars.

 

I'm still having issues with v24 though. While testing random stuff in SOS the skse data was lost, but I was using at that point the first v24 version and I think I can't reproduce that loss with the current v24 version, or the conditions changed, whatever.

 

Another problem I saw trying to reproduce the previous issue, each time I load the savegame the stored info is multiplied.

Steps:

- new game with v23, SOS stores some info

- save, exit

- install v24

- load game

- load the same game again, without saving or exiting

Stored info is multiplied

 

I will try to reproduce the original problem.

 

EDIT 2:

It looks like the problem is while resolving the mod load order and some of the data is invalidated.

Steps:

- new game with v23, SOS stores some info

- save, exit (this produces the attached v23.skse)

- install v24, load game, save, exit (v241)

- load game, save, exit (v242)

There is a stored form in "SOS_Addons", from mod in position 6, the id is 100666722 = 6000D62. After saving a couple of times with v24, that form becomes 134221154 = 8000D62 withouth changing the load order. Also 8000D62 points to the 9th mod and I just had 8 at this moment.

 

co-saves-v23-v241-v242.7z

 

Issue #2

 

papyrus errors

 

 

[07/28/2014 - 10:31:58PM] error: Native static function FileFormListInsert could find no matching static function on linked type StorageUtil. Function will not be bound.

[07/28/2014 - 10:31:58PM] error: Native static function FileStringListInsert could find no matching static function on linked type StorageUtil. Function will not be bound.

[07/28/2014 - 10:31:58PM] error: Native static function FileFloatListInsert could find no matching static function on linked type StorageUtil. Function will not be bound.

[07/28/2014 - 10:31:58PM] error: Native static function FileIntListInsert could find no matching static function on linked type StorageUtil. Function will not be bound.

[07/28/2014 - 10:31:58PM] error: Native static function FileFormListSet does not match existing signature on linked type StorageUtil. Function will not be bound.

[07/28/2014 - 10:31:58PM] error: Native static function FileSetFormValue does not match existing signature on linked type StorageUtil. Function will not be bound.

 

 

Thanks in advance

[Expired6978]

Plugin co-save data is wrapped by plugin-like records, data corruption can only occur for the said plugin, other plugin data should be unaffected. As for the Papyrus errors, those occur when the pex signatures don't match the internally registered signatures. The first 5 are from the function not existing at all, and last two are from a signature change (signature referring to the function parameters).