PapyrusUtil LE/SE/AE

[b3lisario]

Update:

 

I figured out the random stuff I am seeing is normal, it's something from the SKSE, like separator chars or so. I got confused because previous versions of storageutil hadn't these chars.

 

I've updated my previous post with new findings.

[b3lisario]

Looking at the commented code in Serialization_Save (StorageData.cpp) I figured out another problem.

 

Steps:

- Start a new game using v24, some data is stored

- save and quit

- load last savegame, the stored data is lost at this point

 

I think that's because Serialization_Load doesn't get called when starting a new game, so nobody is calling LoadModInfo() until the first game load.

[Ashal]

A lots changed since since that first source release of v24, I'll try and release an updated version later tonight.

 

When troubleshooting it though, take a look at PapyrusUtilDev.log in the skyrim folder, a lot of what the plugin is doing gets printed out there so it's easier to track where things are failing.

[Ashal]

Here's updated testing version:

 

Source Code:papyrusutilv24_source.zip

SKSE Plugin: <removed attachment, see newer version>

 

Everything should be functional with the exception of the GetNth debug functions on StorageUtil, which I still haven't gotten around to and am debating removing them completely in favor of using unordered_map for the minor performance gain (which due to it's unordered-ness would make GetNth infeasible).

 

I originally developed the rewrite using boost::unordered_map in place of the previous versions std::map, which made implementing the GetNth debug functions impossible. The performance gains of boost::unordered_map over std::map seem fairly significant at certain length. I have absolutely no idea at what point that map length makes it noticeably worthwhile however.

 

I've rewritten decent amounts of how the main storage system works, mostly just applying some of the stuff I've learned since then through the process of writing it the first time.

 

Aside from the various rewrites, the main thing of note is that I've added in special cases for StringListFind(), StringListHas(), StringListRemove(), and StringListAdd(allowDuplicate = true) to deal with case insensitive searches as h38fh2mf suggested.

 

I also messed around with changing the form storage over to strings of "baseid|modname.esp" and/or std::pair(UInt32, std::string) it worked in general testing, but backwards compatibility with old saves still using unsigned int's for forms and the separate storage system for animation/package overrides complicated things beyond what I felt like dealing with at the moment.

 

If I were to change it over in a later version however, would that in any way affect the assembly implementation of DecideAnimReplace(int) and DecidePackage(int, int) seeing as their relevant arguments would need to be changed to strings to compensate?

 

The Slice functions I've added are now non functional compared to the previous release, fuck my life. I'm not sure what exactly has changed to cause it, hours of banging my head on my keyboard hasn't revealed much of real use beyond the suggestion that I yell "fuck it" and revert the change back to how it was when it worked. As far as I'm aware the only real change is the location of the "slicing" has moved from being a Papyrus native function in StorageUtil.cpp.

 

There was a point at which they were functional after the move, they broke in the same way at first but was somehow fixed and then broken again after that. I haven't a clue what's wrong, the Papyrus Log complains that "warning: Replacing native function IntListSlice on unlinked object StorageUtil, and the signature is changing."

 

I'm at a complete and total loss at this point, Google has proven fruitless since this is Papyrus error specifically related to SKSE plugin development; and the SKSE team seems to actively discourage public discussion issues using the plugin API... My assumption is that it has something to do with the array argument, or more specifically maybe the added template function I've added to be able to compile using TESForm arrays, which SKSE oddly doesn't include itself. I used the same implementation for VMArray<TESForm*> usage in SexLabUtil.dll, maybe they are conflicting in some way?

 

TL;DR Here is new test version, Fixed string list case insensitive searches as per h38fh2mf's suggestion. Should all work except list slicing and debug_getNth functions. The fucks up with VMArrays, and how about them boost::unordered_map's?

[b3lisario]

I think this particular problem still persists

http://www.loverslab.com/topic/23713-papyrusutil/?p=872448

because the code that would fix it is still commented.

 

I can't test right now the new version, so I am not sure if that problem is fixed somewhere else in the code. Since you didn't mention anything about the issue I guess you didn't try to reproduce it.

[Ashal]

I think this particular problem still persists

http://www.loverslab.com/topic/23713-papyrusutil/?p=872448

because the code that would fix it is still commented.

 

I can't test right now the new version, so I am not sure if that problem is fixed somewhere else in the code. Since you didn't mention anything about the issue I guess you didn't try to reproduce it.

 

I don't doubt it's an issue. After you mentioned it I threw a Lib::LoadModInfo() into the main.cpp's SKSEPlugin_Load() to load the mods at start to compensate. It resulted in an infinite loop or some sort of error however, as the result was the plugin completely fails to load unless I comment it out. I haven't dug into the issue much beyond that. I'll figure out a solution for new saves before the final release though. My assumption is that the plugin loading happens before SKSE loads any load order info; in which case moving it to the revert callback should fix it.

 

I can better integrate it into one of the callbacks, or completely remove it's necessity even, if I can get through implementing the form to string storage system instead of the current form to UInt32 system, since that'd make the mod load order saving/loading completely unnecessary.

[h38fh2mf]

You can still use GetNth with unordered map. The order is not important only that GetNth(0) != GetNth(1) so that all objects in the map can be enumerated. However all of these debug functions could just be replaced by adding one or two functions which print all saved data to file or console for easy inspection by mod authors and you.

 

You could load mod info when saving, because that's when you would be using it and you can't save unless mods are loaded? Or I'm not sure how you get the mod name of a form.

[b3lisario]

@Ashal

 

I did a bit of testing using this version http://www.loverslab.com/topic/23713-papyrusutil/?p=872918

I can confirm the new game issue still happens.

I think loading an old savegame file is now worse than it was before. Previously it looked like the reading was OK and it failed after a couple of saves, now the data is missing right after loading the old savegame.

 

Please let me know if you want to focus in fixing these issues that I reported, or if you can't reproduce them with the steps I provided. There is a PapyrusUtilDev.log in one of the archives I attached, which have 0 downloads at this moment.

I won't do more testing until then.

 

I am aware of the work you have put in this, and that maybe you might be busy with some other stuff than refactoring PapyrusUtil.

I spent my yesterday's free time trying to reproduce and report the issues the best I could.

I'm sorry if I sound like a jerk, but I feel like if I need to insist or convince anybody about these problems, I'll rather spend my time on making SOS not using StorageUtil.

 

Thanks for your work on this

[Ashal]

Did some more testing going back and forth to v23 and v24 as well as messing with load orders, and think I was able to replicate the issue and track it down.

 

This version should fix the forms unexpectedly changing, but will not fix it for any save on the previous version of v24. Should also fix the new game issue.

 

<removed attachment, see newer version>

 

[b3lisario]

Hi again,

 

With this new version I only could reproduce the "new game" issue. Fortunately it's very easy to reproduce. This is what I'm doing

- Start a new game, OnInit saves some info

- Quicksave and exit game

- The .skse has the info that was previously stored, but there is no list of installed plugins (I guess the mod info is not yet accessible when that LoadModInfo() is called).

 

The papyrusUtilDev.log:

 

 

Loading

Storage Saving...

Removed Objects: 0

Storage Saving...

Removed Objects: 0

 

 

 

The savegamefile:

quicksave.7z

 

From my own experience SKSE doesn't offer an easy way to run code just when starting a new game. Not even with the new messaging system added in 1.7.1. I think the available options run too early, before the game reads the load order.

I guess that's why there was a call to LoadModInfo on serialization_save when a variable is not initialized.

 

I am also seeing dozens of papyrus errors like "error: Native static function IntListSlice could find no matching static function on linked type StorageUtil. Function will not be bound". This might be normal since I am using the scripts from v23.

[Aleanne]

Creatures in SexLab were broken with StorageUtil from post #229.

With an activation via Submit, this test was failing (surrendering was refused) :

bool Function SentientNPC(Actor Target)

    If (_SLConfig.VictimRapeActive && SexlabConfig.bAllowCreatures && Sexlab.AllowedCreature(Target.GetRace()) && (Target.GetRace() == Werewolf || Target.GetRace() == Draugr || Target.GetRace() == DraugrMagic || Target.GetRace() == Falmer || Target.GetRace() == Troll || Target.GetRace() == FrostTroll || Target.GetRace() == DLC1Gargoyle || Target.GetRace() == DLC1GargoyleVariantBoss || Target.GetRace() == DLC1GargoyleVariantGreen || Target.GetRace() == DLC1VampireBeast || Target.GetRace() == SkeletonArmor || Target.GetRace() == DLC1SoulCairnKeeper || Target.GetRace() == DLC1SoulCairnSkeletonArmor || Target.GetRace() == DLC2Seeker || Target.GetRace() == DLC2AshSpawn || Target.GetRace() == Giant))

        return True

    Else

        return False

    EndIf

EndFunction

because Sexlab.AllowedCreature(Target.GetRace()) was false, even if the animations were checked. I could trace this back to this in SexLabUtils (but no real proof, I didn't debugged this part)

bool function HasRace(Race RaceRef) global

    return StorageUtil.StringListFind(GetConfig(), "SexLabCreatures", MiscUtil.GetRaceEditorID(RaceRef)) != -1

endFunction

So I thought, maybe this version of StorageUtil fails to retrieve the animation parameters from the creatures. So I installed the StorageUtil from post #234. Rebooted Sexlab (rebuild and clean) and the creatures started to work again. The first and second tries failed, but it worked after that. Still, very minimal tests. But it worked. I guess you're on the good path

[Ashal]

Hi again,

 

With this new version I only could reproduce the "new game" issue. Fortunately it's very easy to reproduce. This is what I'm doing

- Start a new game, OnInit saves some info

- Quicksave and exit game

- The .skse has the info that was previously stored, but there is no list of installed plugins (I guess the mod info is not yet accessible when that LoadModInfo() is called).

 

The papyrusUtilDev.log:

 

 

Loading

Storage Saving...

Removed Objects: 0

Storage Saving...

Removed Objects: 0

 

 

 

The savegamefile:

quicksave.7z

 

From my own experience SKSE doesn't offer an easy way to run code just when starting a new game. Not even with the new messaging system added in 1.7.1. I think the available options run too early, before the game reads the load order.

I guess that's why there was a call to LoadModInfo on serialization_save when a variable is not initialized.

 

I am also seeing dozens of papyrus errors like "error: Native static function IntListSlice could find no matching static function on linked type StorageUtil. Function will not be bound". This might be normal since I am using the scripts from v23.

 

 

This version definitly fixes it, or at the very least I'm no longer able to reproduce it myself. 

 

StorageUtil.zip

 

Also fixed the list slicing functions, an issue with loading custom JSON files, and did some reorganizing of the code (for the 4th time)

 

Everything needed to be functional for a final release should be good to go as far as I know. If nobody has any more complaints or bugs to report I'll probably just do some cleanup and put in a few more utility functions such as a function to set the size of list, creating/filling a list with default values, and such.

[Ashal]

Creatures in SexLab were broken with StorageUtil from post #229.

With an activation via Submit, this test was failing (surrendering was refused) :

 

 

It's working fine for me on the current version I just posted above, my guess would be that while testing the version posted in #229 you lost the valid creature list along the way, with the creature list emptied all creatures wouldn't work. Rebuilding SexLab's settings from the MCM option should fix it in that case.

[Aleanne]

 

Creatures in SexLab were broken with StorageUtil from post #229.

With an activation via Submit, this test was failing (surrendering was refused) :

 

 

It's working fine for me on the current version I just posted above, my guess would be that while testing the version posted in #229 you lost the valid creature list along the way, with the creature list emptied all creatures wouldn't work. Rebuilding SexLab's settings from the MCM option should fix it in that case.

 

 

Some work, some don't. I think that some of these are custom races and are filtered by submit. When I removed the filter, I got this, then CTD. But the next tries worked, so I'm a bit of a loss here. Last StorageUtil, new game.

 

[07/31/2014 - 09:43:12PM] Error: Cannot cast from None to sslBaseAnimation[]

stack:

    [sexLabQuestAnimations (0C0639DF)].sslanimationslots.GetList() - "sslAnimationSlots.psc" Line ?

    [sexLabQuestAnimations (0C0639DF)].sslanimationslots.GetByTags() - "sslAnimationSlots.psc" Line ?

    [sexLabQuestFramework (0C000D62)].sexlabframework.GetAnimationsByTag() - "SexLabFramework.psc" Line ?

    [zadQuest (1500F624)].zadlibs.CheckForBoundAnims() - "zadLibs.psc" Line ?

    [zadQuest (1500F624)].zadbq00.Maintenance() - "zadBQ00.psc" Line ?

    [alias Player on quest zadQuest (1500F624)].zadPlayerScript.OnPlayerLoadGame() - "zadPlayerScript.psc" Line ?

[07/31/2014 - 09:43:23PM] error: Native static function FileFormListSet does not match existing signature on linked type StorageUtil. Function will not be bound.

[07/31/2014 - 09:43:23PM] error: Native static function FileSetFormValue does not match existing signature on linked type StorageUtil. Function will not be bound.

Wasn't able to replicate it. If I'm speaking nonsense, just tell me. Don't wanna clog up the thread.

 

[b3lisario]

The version from #237 works fine for me.

[Ashal]

The version from #237 works fine for me.

 

Any additional function requests you have before I go on finalizing everything?

[Ashal]

Finalized release version, will give it some time for people and myself to run it through it's paces before adding it to the OP.

 

SKSE plugin: papyrusutilv25.zip

Source: papyrusutilv25_source.zip

 

• Re-added GetNth() debug functions for non files - the only missing functions from previous version now are Import/ExportFile and the debug file functions. Both of which are should be safe to just leave missing.
• Added StorageUtil and JsonUtil ListResize() functions to set a specified list to a given size with given filler value
• Added StorageUtil and JsonUtil ListCopy() functions to set a specified list to match the contents of a given Papyrus array

 

Aside from the final testing the the only thing missing is adding documentation for the new functions.

 

Going to start working on a new version of SexLab now to make use of some of the new functions.

[germanicus]

Thanks ashal. Both files solved the problem. No CTDs on loading and the menu appeared.

[easykill]

You should make a git repo for this.

[zensunniwanderer]

Just wanted to say thank you for version 2.5

 

Prior to your releasing it I was unable to use submit due to CTD. Now it's working!

[Ashal]

Cleaned up some stuff, added documentation for the new stuff that needed it, and released an official version of 2.5. It's in OP now and the version now currently in use by SexLab 1.59 beta 2.

[zaira]

Thanks for your cool library!

 

I have some questions about FormLists :

• Are they slower, better or faster than the native Papyrus FormLists? Currently I'm using native FormLists to store/restore actor clothes.
• What about native form arrays - are they slower, better or faster than your FormLists?
• If I import an utils formlist from a json file containing forms that are no longer present - what will happen?

 

Some notes about ActorUtils:

I get a reproducible CTD when I add a Package Override with a package the belongs to a different quest (Travel Package - containing an alias). No logs.

 

Some feature requests:

• Function to close any menu - not only user menus - for example to provide an action button in the MCM menu
• Execute console command - not only a bat
• Copy / convert function between native FormLists and Utils FormLists
• Function to add all forms from an utils formlist into an object container (AddItem)
• Function to add & equip all forms from an utils formlist (AddItem + EquipItem)
• Function to copy / move all forms from an object container into an utils formlist
• Function to copy / move all forms from an actor equiped container into an utils formlist
• ForEach function - user can provide a function that will be called for each item of an utils formlist

[Ashal]

 

Thanks for your cool library!

 

I have some questions about FormLists :

• Are they slower, better or faster than the native Papyrus FormLists? Currently I'm using native FormLists to store/restore actor clothes.
• What about native form arrays - are they slower, better or faster than your FormLists?
• If I import an utils formlist from a json file containing forms that are no longer present - what will happen?

 

• They are measurably faster than native FormLists
• In terms of straight read/write to an existing native form[] array, they are slower. Nothing will ever be faster than simply accessing a simple local variable. However, unlike the Util's FormLists, native papyrus form[] arrays lack any ability to dynamically change size, are limited to 128 elements, and exist only on the script you set them in.
• They will load into the game as a "none", the json file would be unaffected and keep it's value should the form be present again; so long as you don't overwrite the json value.

[zaira]

Maybe it is faster to use invisible containers and transfer a big bunch of items between these containers via RemoveAllItems() ?

[b3lisario]

Some feature requests:

• Function to close any menu - not only user menus - for example to provide an action button in the MCM menu

This can be already done. For example to close the MCM menu:

Function CloseMCM()
    UI.Invoke("Journal Menu", "_root.QuestJournalFader.Menu_mc.ConfigPanelClose") ; mcm
    UI.Invoke("Journal Menu", "_root.QuestJournalFader.Menu_mc.CloseMenu") ; quest journal
EndFunction

The command might be different for other menus.