PapyrusUtil LE/SE/AE

[Srende]

Srende I looked over package code and it seems ok. Do you think you can get the memory address where game exception occurred?

 

How can I get that? I'm not getting any error messages during the crash.

[h38fh2mf]

If you are using HDT I think it saves in a log or something.

 

Here it is as an example in a windows error report:

http://cdn.howtogeek.com/wp-content/uploads/2009/12/sshot20091219235718.png

 

The bottom "Exception offset".

 

I don't think skyrim creates this type of error report though.

[Srende]

Yes, I actually just realised I have the hdt mem dump & log.

Here's the log:

 

 

hdtSkyrimMemPatch
[07/06/14 13:48:45]INFO: Queue OK
[07/06/14 13:48:45]INFO: SKSEPlugin_Load
[07/06/14 13:50:58]ERROR: Fatal error occured
[07/06/14 13:50:58]ERROR: Code : 0xc0000005
[07/06/14 13:50:58]ERROR: Flag : 0x00000000
[07/06/14 13:50:58]ERROR: Module : D:\Games\Steam\steamapps\common\skyrim\Data\SKSE\Plugins\StorageUtil.dll
[07/06/14 13:50:58]ERROR: Address : 0x54676eac
[07/06/14 13:50:58]ERROR: Module Address : 0x54630000
[07/06/14 13:50:58]ERROR: AccessViolation, try to read 0x00000000 failed
[07/06/14 13:50:58]ERROR: Call Stack (Skyrim ignored the frame pointer register (ebp) so it may not correct) :
[07/06/14 13:50:58]ERROR:      : 0x54676eac
[07/06/14 13:50:58]ERROR:      : 0x54650d83
[07/06/14 13:50:58]ERROR:     TESV : 0x00c46e61
[07/06/14 13:50:58]ERROR:     TESV : 0x00f5d656
[07/06/14 13:50:58]ERROR: Thread local Havok memory heap:
[07/06/14 13:50:58]ERROR:     Temp memory allocator:0x01bb1c94
[07/06/14 13:50:58]ERROR:     Temp memory allocated:-1
[07/06/14 13:50:58]ERROR:     Temp memory in used:-1
[07/06/14 13:50:58]ERROR:     Temp memory available:0
[07/06/14 13:50:58]ERROR:     Temp memory total available:-1
[07/06/14 13:50:58]ERROR:     Temp memory largest block:-1
[07/06/14 13:50:58]ERROR:     Heap memory allocator:0x01bb1c94
[07/06/14 13:50:58]ERROR:     Heap memory allocated:-1
[07/06/14 13:50:58]ERROR:     Heap memory in used:-1
[07/06/14 13:50:58]ERROR:     Heap memory available:0
[07/06/14 13:50:58]ERROR:     Heap memory total available:-1
[07/06/14 13:50:58]ERROR:     Heap memory largest block:-1
[07/06/14 13:50:58]ERROR:     Debug memory allocator:0x01bb1c94
[07/06/14 13:50:58]ERROR:     Debug memory allocated:-1
[07/06/14 13:50:58]ERROR:     Debug memory in used:-1
[07/06/14 13:50:58]ERROR:     Debug memory available:0
[07/06/14 13:50:58]ERROR:     Debug memory total available:-1
[07/06/14 13:50:58]ERROR:     Debug memory largest block:-1
[07/06/14 13:50:58]ERROR:     Solver memory allocator:0x01bb1c94
[07/06/14 13:50:58]ERROR:     Solver memory allocated:-1
[07/06/14 13:50:58]ERROR:     Solver memory in used:-1
[07/06/14 13:50:58]ERROR:     Solver memory available:0
[07/06/14 13:50:58]ERROR:     Solver memory total available:-1
[07/06/14 13:50:58]ERROR:     Solver memory largest block:-1
[07/06/14 13:50:58]ERROR: Minidump saved in hdtSkyrimMemPatch.dmp
[07/06/14 13:50:58]ERROR: Plugin is trying to save game
[07/06/14 13:50:58]ERROR: Game saved

 

[h38fh2mf]

Can you upload the dmp file? It's difficult to say where crash happens because it's in a function that's used almost everywhere.

[Srende]

Sure.

Edited July 6, 2014 by Srende

[h38fh2mf]

Hmm I can't open this for some reason.

 

If error happens in the function you said then it's this line:

for(std::vector<int>::iterator itr3 = itr2->second.begin(); itr3 != itr2->second.end(); )
{
    if(packForm == *itr3)
    {
        itr3 = itr2->second.erase(itr3); // <<<<<<<<<<<<<<<<<<<<<<<<<<
        count++;
    }
    else
        itr3++;
}

But I don't see anything wrong with it.

 

[Srende]

With further testing, I just commented out that one call to RemoveAllPackageOverride and it doesn't crash. However, later on I actually do the same exact call, with the same paremeters and it didn't crash on that. The difference I can think of is that on the second call the same actors should have another package override set that's shown in the snippet I posted. I then remove both of those packages right after each other a while later, with the same function.

 

I should be able to remove the first call without any effect actually on my mod, as packages added later should take precedence(?), and just remove them both at the same time. But, I guess that doesn't really solve the actual problem, and why it worked on earlier versions

[Ashal]

JSON might corrupt the same way, I think corruption happens if game crashes during file saving or some OS file access problem. Maybe workaround would be to save to a new file, then when saving is done, delete old file and rename new file to old file. If game has crashed in between then nothing really is lost.

 

 

The json files are written to and read from at runtime when it's needed, not during save/load. So if the corruption happens during a save/load that crashes the the game it wouldn't be affected by it.

[h38fh2mf]

Ah right. And JSON file might be easier for you as well to handle.

[Ashal]

I've started working on organizing and cleaning up the source code, namely pulling everything unnecessary out of main.cpp and separating it into individual files for StorageUtil/ActorUtil/MiscUtil/PapyrusUtil, mimicking how it's done by SKSE's papyrus functions. Also getting rid of most of the the constantly repeated code and duplicated functions for int/float/string/form and replacing them with templated versions so it's significantly easier to edit and maintain.

 

So far I've got the basic Get/Set/Unset/HasValue() functions converted to templates.

[h38fh2mf]

Ah nice, can you maintain the source in SexLab git and update this post with new versions?

[Rabblerouser]

There seems to be a small, but growing number of users having the game hang on startup. Steam avatar keeps the border green and you have to go into Task Manager and force-quit TESV.exe.

 

With some beginner sleuthing, it seems it's something to do with a different StorageUtil in the new SL 1.58. The old 1.57 works smooth as silk when ran from Mod Organizer.

 

 

I noticed some speak of JContainers earlier in this thread.. should ask if you're going to make a seperate, Windows XP-compatible version like JContainers did as well?

 

I'm no expert. I'm not a coder, you guys are. And I apologize if I'm butting into the conversation here, but simply trying to throw ideas about that might work.

[Ashal]

Ah nice, can you maintain the source in SexLab git and update this post with new versions?

 

I'll upload the source once I'm finished so you can take a look at it before I slap it over the existing official upload, can set up a Git repo as well if necessary.

 

My cleaning up the source code has at this point more or less turned into a rewrite of plugin, for the storage stuff anyway, I don't understand assembly in the slightest so all the stuff going on with package and animation overrides will be left mostly untouched beyond adapting it to new storage setup. Got all the flat values and lists functions fully rewritten, tested, and functional all using a templated class.

[Ashal]

Anybody aware or actively using StorageUtil's current JSON system besides me in SexLab? 

 

I've got a JSON system in place in my rewrite, one that works as a more direct storage rather than as a middleman between external files and the main data storage of StorageUtil, so if I can safely just ignore having to add in support for the existing ExportFile and ImportFile functions and just adapt SexLab's current use of it to the new system I have ready, it would save me a huge headache and day or two of work.

 

the new system essentially being doing just

 

StorageUtil.JsonGetFloat("mydata.json", "key")

 

working pretty much the same as the existing storageutil.save functions, only with the added ability of being able to specify separate files.

 

 

[DeepBlueFrog]

Anybody aware or actively using StorageUtil's current JSON system besides me in SexLab? 

 

I've got a JSON system in place in my rewrite, one that works as a more direct storage rather than as a middleman between external files and the main data storage of StorageUtil, so if I can safely just ignore having to add in support for the existing ExportFile and ImportFile functions and just adapt SexLab's current use of it to the new system I have ready, it would save me a huge headache and day or two of work.

 

the new system essentially being doing just

 

StorageUtil.JsonGetFloat("mydata.json", "key")

 

working pretty much the same as the existing storageutil.save functions, only with the added ability of being able to specify separate files.

 

I think Radiant Prostitution uses a JSON file for configuration of its quest parameters (I can't confirm for sure.. I am away from my gaming PC for the day).

[DeepBlueFrog]

Question for PapyrusUtil experts.

 

Is there a way to access StorageUtil directly using condition functions?

 

http://www.creationkit.com/Condition_Functions

 

The way I currently use to do this is to have a dummy topic dialogue with the target NPC during which I set global variables from storageUtils values so that I can test for those global variables in further dialogue topics with that NPC.

 

it's a little clunky to say the least.

 

I would rather have a direct call as one of the conditions on the topic itself.

[Srende]

Is there a way to access StorageUtil directly using condition functions?

You could probably use the condition function that lets you look into script properties (getvmscriptvariable or getvmquestvariable)

and then in the script:

scriptname myscript extends quest conditional
 
int Property puCondition conditional
    function get()
        return StorageUtil.GetIntValue(none, "key")
    endfunction
 
   function set(int newVal)
       StorageUtil.SetIntValue(none, "key", newVal)
   endFunction
EndProperty

Though haven't tested that.

 

Edit: Turns out it won't work. Compiler just gives me an error "Flag conditional is not allowed on properties", though that's slightly wrong: I'm using it on an auto property elsewhere and it works. But apparently not here.

[Ashal]

I've got all StorageUtil and MiscUtil functions finished + plus the new JsonUtil script.

 

Working on package and animation overrides now, I just finished getting a working messy version of package overrides implemented, just need to figure  out how best to clean it up now. Most of that should carry over into animation overrides as well, so shouldn't be too far off from finishing everything. 

 

Each data map is now also stored inside it's own serialization record in SKSE rather than a single large 'DATA' record. This way they serialize/unserialize to/from much smaller strings to help avoid any problems, and if one record's serialization string corrupts in some way the other storage maps are unaffected. I've left in conversion code for loading 2.3 and existing 'DATA' record to keep backwards compatible.

 

Did a few preliminary benchmarks comparing StorageUtil 2.3's performance to the new version I've rewrote thus far, new version has consistently performed slightly faster in a read/write test, and the new JsonUtil functions for external file storage work nearly as fast.

 

I've got a couple new functions I'd like to implement as well. The main one being an array slice utility for quickly getting a list of values out of StorageUtil into a Papyrus array. 

; // StorageUtil.psc Function:
function StringListSlice(form formKey, string stringKey, string[] output, int startIndex = 0) global native 

; // Usage:
string[] slice = new string[10]
StorageUtil.StringListSlice(none, "mykey", slice, 4)

Would place the first 10 elements of the given StorageUtil list starting from the 4th index. It would fill the array until either the papyrus array or StorageUtil list runs out of elements to place into the array.

 

 

As I am new at C++ however, I'm convinced I'm making some horrible pointer mistake somewhere, though I haven't noticed any major issues yet in my testing, I haven't done any sort of extensive play sessions or testing yet. So once I finish the initial version I'd much appreciate anybody more familiar with C++ giving it a good look over. 

 

Should hopefully be done in the next couple days.

 

 

I think Radiant Prostitution uses a JSON file for configuration of its quest parameters (I can't confirm for sure.. I am away from my gaming PC for the day).

 

 

Radiant Prostitution is using JContainers for json storage as far as I know, so shouldn't be affected. 

 

 

[Srende]

Hmm, any luck trying to resolve why 2.3 seems to crash in place where 2.2 didn't? Apparently that one place wasn't the only one where DH is crashing for some with 2.3 and there really is no way to work around that in papyrus.

 

Although, related to that, do you have a version of your modifications that could be tested Ashal? I can't say I'm that familiar with C++, but could (and would like to) test it to see if it has any effect on the problems I've seen with 2.3.

[Ashal]

It's a complete rewrite for the most part, anything not rewritten is moved and cleaned up to match the newer code. It also 100% compiles under VS 2008 and it switches all the thread locking juggled around between 2.2 and 2.3 to use the exact same thread locking method used by SKSE internally; which should be the primary thing that fixes any issue people had with either 2.2 or 2.3.

 

I'm more or less sticking to the same standard/style used by SKSE's source code as much as possible. I did the same with SexLabUtil.dll as well, which was added in SexLab 1.58, so if you can run that without problem this new version of StorageUtil.dll shouldn't give any issues either.

 

Or the rewrites could make everything implode and 10x worse, I have no idea. I'm fairly confident it should be fixed though.

[Srende]

Heh, thought something like that might be the case, thanks.

[Ashal]

Finished and tested everything I consider completely necessary for new version of PapyrusUtil, so here's a quick test version I could use some testers and feedback on. 

 

PapyrusUtil_24.zip

 

Couple notes and changes:

 

• I have not done any extensive usage testing!
  • I have tested and confirmed the core functions of all previous functionality works, such as setting/getting/unsetting variables, package and animation overrides. The more specialized functions of each, such as "InsertAt", "RemoveAt", "ListSort" and such are largely untested, but should  be functional as far as I know, from a purely theoretical perspective. 
  • I've generally only tested from smaller data sets from new saves, old saves that have been using StorageUtil extensively for a long time may or may not crash the game when attempting to load, I don't know. 
  • I can however confirm general loading a save with 8 hours playtime, with StorageUtil used throughout, loaded and converted to the new version fine. As did new saves.
• I literally did not know anything beyond printing "Hello World" in C++ a month or two ago, rewriting this has for the majority of time spent been a lesson in banging my head on the desk in anger and Googling for solutions, tutorials, examples, and best practice guidelines. But despite all that Googling, I am a complete and utter noob in this world, I've lived solely in a world of cushey interpreted programming languages beforehand.
  • In other words if the source code is horrifying or I'm making some terrible newbie mistake with pointers/reference or other memory related shit, tell me.
  • That said; I'm fairly confident it's sufficient and a major improvement over the previous PapyrusUtil versions where the source was all shoved into a single main.cpp. Entirely possible that's a completely false sense of confidence and everything is atrocious. 
• Added new JsonUtil script, allows you to read/write from properly formatted json formatted files.
  • External files are created/updated whenever the player saves/loads their game, so changes made by a script will not immediately reflect in external file until after the player saves. Changes to any related storage data however is reflected immediately, it's just the writing to the external file that's delayed, continued pulling of variables from the file pre-save will still give the up to date data.
• Package overrides currently completely ignore the priority argument.
  • Debating whether or not this is worth re-implementing, I figure that the people making use of it all in general set the priority at 100, so it would just use the last added in 90% of cases anyway, making the priority more or less redundant. 
• I have not implemented various StorageUtil debug_ functions yet, namely just anything dealing with pulling a specific index out.
• I have not implemented the previous versions ExportFile/ImportFile functions for saving json files.
  • Not sure if I will or not
  • A JSON <-> StorageUtil function would probably be useful for some, so may add that and ExportFile/ImportFile will likely be a short jump to implement from that.
• Should fix any crashing on Skyrim start or corrupt saves people have had with past newer versions of PapyrusUtil.
• I still want to add a couple new utility functions before final release such as a papyrus array to StorageUtil slicing function

Here's the source code for anybody who is willing to give it a look over for me, which would be much appreciated. 

 

PapyrusUtil__24_Source.zip

 

In order to use the project file, the only alteration you should need to make is changing the path to Boost headers.

[Srende]

Nice, I'll need to test that.

 

Though I am currently using the package override priority combined with package conditions to determine which one of them should be active. The conditions do overlap and might change back and forth, leaving it to the priority to determine the running package on those cases. So I'd still like to see it work.

[h38fh2mf]

I looked at some of the code and it looks good. Much more easy to understand and edit. I put all into 1 file because I had no idea this mod would become so big it started out just as a way to use TFC console command if I remember correct, then I added all the other stuff over time.

[Ashal]

Here's another version of the final plugin: papyrusutilv24.zip

 

I added the list slice function mentioned in one of my previous posts, package and animation overrides are all fully functional, and "de-natived" the functions in the papyrus scripts that are no longer or currently not implemented so they won't print out a bunch of debug errors when used.

 

The only ones currently not implemented are the index debug functions, which I'm unsure if anybody is using in final code, I'd assume they shouldn't be considering they are debug functions. And almost Export/ImportFile, which as far as I know I was the only one who ever used it, and I plan on adapting the next version of SexLab to use the new JsonUtil script.

 

If that's all okay with you to go ahead and use as the official release, feel free. I can take over the OP as well if you'd prefer to just fully hand off the reigns, whatever you prefer, it is your mod.

 

In addition to the list slicing function and JsonUtil, I've also added a few minor functions.

 

• MiscUtil.SetFreeCameraState(bool enable, float speed = 10.0)
  • For setting the TFC state directly as either on/off, because I found myself having to do "if (Game.GetCameraState() == 3) MiscUtil.ToggleFreeCamera()" a lot in SexLab.
• ​MiscUtil.GetActorRaceEditorID(Actor actorRef)
  • ​Again because I'm lazy and found myself having to do "MiscUtil.GetRaceEditorID(ActorRef.GetLeveledActorBase().GetRace())" a lot, this does it all just off actor.
• ​The StorageUtil.FileXXXXX functions now includes Form verisons of each functions, and saves to a json file instead.
  • I assume the Form functions were skipped originally since there was no mod list for the files to get new load orders off of for form id's, so the external file saves forms as strings instead with the format of "xxxxxx|modname.esp", so it can look up the mod's load order when pulling values.
• Added bool StorageUtil.TypeListHas(Form obj, string key, Type value) functions,
  • Again because I'm lazy and found myself doing "StorageUtil.TypeListFind() != -1" a lot.