PapyrusUtil LE/SE/AE

[b3lisario]

SOS uses formlists to dinamically store diverse stuff like actors or armors.

I'm seeing formlists have serious limitations (slowness and lack of .RemoveAt() function), so I'm considering StorageUtil.

What surprised me the most is the speed, I am doing tests with 100 items, StorageUtil writes or reads 100 times faster than using a formlist lol

 

I have some questions:

Can I store forms from other mods? For example an actor provided by a follower mod.

Is there a size limitation? Could I save like 10 lists with 1000 items each?

Let's say one of the stored forms is None because it is an armor from a mod no longer present. Could I then remove it safely? For example:

While i < count
  Armor a = StorageUtil.FormListGet(f, "k", i) as Armor
  If !a
    StorageUtil.FormListRemoveAt(f, "k", i)
  Else
    ...
  EndIf
  i += 1
While

Thanks in advance

[h38fh2mf]

Yes all forms can be saved. If it doesn't exist later it will become none in the list but not removed automatically. If you save a value on a form itself that doesn't exist later then these values will be removed when saving game.

 

There is no size limitation other than RAM. I recently fixed a bug that caused saving game to take about a minute. What was wrong? a list had 29 million entries in it due to an error which is now fixed of course but the game itself was still running fine. There is some limit because the file wasn't saved properly due to having too big. But the limit is a ridiculous value that I don't believe anyone will ever find out unless there is another mistake that adds millions of stuff. 10 lists with 1000 items is not a lot at all this mod was meant to do just that. I can post the format for saving this stuff if you wish so you can calculate exactly how many bytes it would take in a save game to save the amount of values you want. But all my calculations with the worst case scenario put the size less than 50 KB for 1000 integers on all different objects. I don't expect anyone to notice any save game file size changes due to this.

 

All of the functions are safe to use, if you try to remove stuff from invalid indexes or get values that aren't there then no errors are produced, in the example of removeat if you try to remove a value out of index or the list didn't exist then false is returned, true is returned if you did remove the value successfully.

 

Hmm I thought of a possible bug that could happen while writing this answer. I will post a fix.

 

But to answer the question directly yes you can get / remove / removeat forms that have become none.

[h38fh2mf]

You can't use removeat the way you edited in the loop. You either increase iterator (i += 1) or remove. Because removing something from a list will make i point to the next entry already.

[DeepBlueFrog]

A few suggestions for future updates of PapyrusUtil if you are looking for ideas:

 

- MathUtil - Standardized min, max, range functions as an extension of Math. I have seen those implemented with different names and syntax in multiple mods. It would be good to have a set of function already there.

 

- BodyUtil - I am thinking of turning some of the code I have for body and color changes into a library. My first attempts didn't work so well but I didn't have time to push through it. I can give you the code if you are interested. It includes utility functions to change breast, butt, belly nodes when available, weight change with built in check for mounted and refresh of the model, HSL to RGB conversions, application of color to selected parts (lips, eyelashes, dirt, ... ). 

 

I had another idea but it escapes me for now...

[h38fh2mf]

Maybe, I'll think about it. I was really trying to keep in this library only functions that couldn't be possible in papyrus normally.

If you're going to remove stuff from a list I usually do it this way:

int i = StorageUtil.FormListCount(f, "k")
while(i > 0)
i -= 1
Armor abc = StorageUtil.FormListGet(f, "k", i) as Armor
if(!abc)
StorageUtil.FormListRemoveAt(f, "k", i)
endif
endwhile

If you do it backwards then it doesn't matter if you remove stuff because index will always be valid.

 

Edit: I see you are doing some other stuff there too, but if you're only thinking about removing invalid forms from a list you can do this too:

StorageUtil.FormListRemove(f, "k", none, true) ; remove all instances of "none" from a list and return integer how many was removed

[b3lisario]

Thanks for the clarifications.

 

I'm seeing a lot of potential here. Let me see if I'm understanding this. With StorageUtil being faster and easier to use than arrays or formlists, I don't see the benefit of keep using them for storage. Right?

[h38fh2mf]

Yes I agree. StorageUtil lists are faster, dynamically sized and accessible from all mods, only thing to consider perhaps is that when your mod is uninstalled the data remains in the save game unless you save data on a form that is created by your plugin, for example if you save all data on your mod's quest then this data will be deleted when the quest no longer exists. This is probably what you want but if you want to save data over uninstall / reinstall / update then you can also save it globally and nothing bad will happen.

[b3lisario]

I'm having a weird issue, looks like an OnInit event is called twice when I use StorageUtil. May it be possible?

 

Scriptname TestStorageScript extends ReferenceAlias

Armor Property ArmorIronCuirass Auto
Armor Property ArmorIronGauntlets Auto

Event OnInit()
  Debug.Notification("count: " + StorageUtil.FormListCount(ArmorIronCuirass, "k"))
  StorageUtil.FormListAdd(ArmorIronCuirass, "k", ArmorIronGauntlets, false)
EndEvent

It's a script attached to a reference alias. I start a new game via coc in the menu screen, and I'm seeing:

count: 0

count: 1

 

If I change the line

Debug.Notification("count: " + StorageUtil.FormListCount(ArmorIronCuirass, "k"))

for

Debug.Notification("count: ")

and start a new game then I see just one notification, not two

 

I don't know what could be the issue. Both properties are filled. It's a new vanilla game, without other mods. I attach a simple mod that reproduces the problem.

TestStorage.7z

[h38fh2mf]

The quest doesn't have run-once flag.

 

See this and look at the run once line:

http://www.creationkit.com/Quest_Data_Tab

[b3lisario]

Ugh that was unexpected lol. Thanks for the help.

 

What I was trying to reproduce until I found the "OnInit issue" is another problem when using scripts or quests as objects where to save the info.

I can add items, but they don't persist after I save & reload. It works if I use an Actor form pointing to the player.

 

For example I have this func in the main quest script:

Scriptname TestStorageQuestScript extends Quest  

Actor Property PlayerRef Auto
Quest Property TestStorageQuest Auto

Function NotifyCount()
  Debug.Notification(StorageUtil.FormListCount(self, "k") + "," + StorageUtil.FormListCount(TestStorageQuest, "k") + "," + StorageUtil.FormListCount(PlayerRef, "k"))
EndFunction

I can add stuff in another script and it seems to work fine:

StorageUtil.FormListAdd(questScript, "k", equipped, false) ; points to the main quest script
StorageUtil.FormListAdd(testStorageQuest, "k", equipped, false) ; points to the main quest
StorageUtil.FormListAdd(PlayerRef, "k", equipped, false) ; player

questScript.NotifyCount() ; shows 1,1,1

But then I quicksave & reload and, in a 3rd script:

Event OnPlayerLoadGame()
  questScript.NotifyCount() ; shows 0,0,1 .... 
EndEvent

I'm attaching another test mod. I hope I haven't made any noob mistake this time...

TestStorage.7z

[h38fh2mf]

I installed this mod you attached, when I first start it equipped and said 1,1,1 then I quicksave and quickload it said 1,1,1 then I exited game, started up again and loaded quicksave still 1,1,1

 

Edit: now I tried to change load order of this mod and still 1,1,1

[b3lisario]

wtf why is all so complicated lol

 

Yes I tested again and it works...sometimes... at least for me.

It works if I use an existing save:

- load existing save, ok (it shows 1,1,1)

- quicksave & reload, ok

 

But it won't if I start a new game and then quicksave & reload:

- start new game, ok

- quicksave & reload, fail (it shows 0,0,1)

 

And, even weirder:

- load existing save, ok

- quicksave & reload, ok

- exit to menu, start new game, ok

- quicksave & reload, ok (wtf?)

 

I start new games by typing "coc qasmoke" in the menu screen. I guess it must be that. BRB gonna start the game in the traditional way...cart ride time

[h38fh2mf]

Try starting with live another life it's fast start.

 

I did read somewhere I think that quest starting is delayed when you start new game? So that quests are started when you get hands unbind or some other time I don't remember.

[b3lisario]

Ok I could reproduce the issue with a traditional game start.

 

Run Skyrim, new game, enjoy cart ride, reload the first autosave. It outputs 0,0,1 for me.

 

Had to change the Debug.Notification to Debug.Trace because the initial cutscene was hiding the 1st notif.

[h38fh2mf]

Try seeing what happens if you delay the code you have in OnInit until your character gets hands freed.

[b3lisario]

I did read somewhere I think that quest starting is delayed when you start new game? So that quests are started when you get hands unbind or some other time I don't remember.

Hmmmm but the quest is a valid object while the game is running for the first time. StorageUtil writes and reads fine using the quest, until you save and reload.

 

Anyway this is not a big issue for me if I can use the player form to save the stuff. Maybe it's just something wrong in my installation, or a game or skse issue and is not related to StorageUtil. I don't want to bother you more with this.

 

On a different topic, is there an elegant way of detecting if this plugin is installed, maybe to throw a warning? I tried SKSE.GetPluginVersion("StorageUtil"), or "PapyrusUtil"

[h38fh2mf]

Don't save on player. It is good to save on quest because if your mod is uninstalled and you save on player the data will remain on there forever but if you save on quest then the data is removed because the quest would no longer exist on uninstall assuming of course that this library will remain installed. If you do decide to save on player then you should take into account that on reinstall of this mod the data may already exist and you shouldn't initialize again but use the old data.

 

Yes you can use

int function GetVersion() global native

in script "PapyrusUtil"

 

It will return 0 if mod is not installed due to function not existing and > 0 if it is installed. Version 1.6 would return 16

 

You can also create a dummy form to save data on if the quest somehow is causing problems. This form would be deleted when you uninstall mod and so the data will as well.

 

Edit: actually you are right it is strange that it would not save load properly immediately because a form is still a form even if it's not initialized. Can you try to do the same on some other form that your mod adds? For example save all data on the base armor form the one that had enchantment. Not the object reference but the armor base item. See if it saves properly immediately or not, If yes then something is special with quests if not then there is a bug and I must fix.

[b3lisario]

Try seeing what happens if you delay the code you have in OnInit until your character gets hands freed.

Didn't work

Changed the equipitem for additem so I can trigger the script manually

new game via coc whiterundragonsreach

exit the building to breath fresh air

equip the item ---> 1,1,1

quicksave and reload ---> 0,0,1

Later I will try with the live another life mod.

 

Ok I see the problems when using the player.

But then I can use just vanilla stuff like playerRef or Game.GetFormFromFile(0x12e46, "Skyrim.esm"). That, or not use StorageUtil if I check it is a new game, not sure how.

I tried using a dummy form I made (a formlist). Didn't work after a new game, i tried getting the ref via script property and also GetFormFromFile()

[h38fh2mf]

Ok I think I know what the problem is. Will attempt to fix can you test if I post a new DLL file here?

[b3lisario]

Sure

[h38fh2mf]

Ok, great. I changed how a few things are loaded and initialized this should fix new game problems. This is quite a big bug it's good that you found it.

StorageUtil_test.zip

[b3lisario]

Still doesn't work

And maybe I found a new issue related to the allowDuplicate parameter, it puts the same item twice.

 

This is what am I using to store:

- quest script

- quest

- player ref

- a formlist added to the mod

- the same formlist using GetFormFromFile(0x12c6, "TestStorage.esp")

- vanilla ref GetFormFromFile(0x12e46, "Skyrim.esm") ArmorIronGauntlets

 

What am I doing:

- Run game and coc qasmoke in menu screen

- Equip the Test item that is added to inventory --> 1,1,1,1,1,1 ok

- quicksave & reload --> 0,0,1,0,0,1 (fail unless vanilla?)

- unequip & equip the Test item --> 1,1,2,1,1,2 (it is allowing duplicates)

- quicksave & reload again --> 1,1,2,1,1,2 ok

TestStorage.7z

[h38fh2mf]

No it's not duplicate problem. The form ID is not calculated properly when you start new game. This is the problem I was trying to fix but I was apparently unsuccessful. I will try more things.

[h38fh2mf]

Ok I found problem, when you open game and click new game immediately then SKSE does not send ANY information at all indicating something has happened. This is a bit of a problem but I think I found a workaround, so try if it works now.

StorageUtil_test2.zip

[b3lisario]

Yes now it works.

Thanks for the fix!