PapyrusUtil LE/SE/AE

[h38fh2mf]

Ok I'll try cleaning up source a bit and release something. It's better this way anyway, I'm not actively working on it now and it could be useful if someone wants to improve it.

[h38fh2mf]

Ok it was too much work to make code readable and clean up so I just post it here as it is, maybe it will be of some help. It's one cpp file with 8000 lines of code, I didn't expect the project to become so big.

 

Requires http://jsoncpp.sourceforge.net/ and also 1.6 SKSE source files, open the project file with a text editor to see where to place the project in relation to those required files.

 

If you have older version than VS 2013 then just create a new multi threaded DLL project and add the files manually. Open project file with text editor to see which files to add.

plugin_example2.zip

[Ashal]

And any idea on the vcomp100.dll or corrupted save stuff?

[h38fh2mf]

I'm not sure, these vcomp dll issues started when I switched to VS 2013 but at the same time I included some new header files too so it could be that. Corrupted save and 0 version I would think is because the mod failed to load for some reason, this can happen with missing DLLs, not starting SKSE in admin mode or few other reasons.

[b3lisario]

About vcomp: I guess it's the omp thing (#pragma omp critical etc).

No idea how to avoid the dependency. Maybe including the entire omp stuff in the dll somehow?

 

http://msdn.microsoft.com/en-us/library/0h7x01y0.aspx

http://www.dependencywalker.com/

[h38fh2mf]

No I have used OpenMP for many versions, it's definitely not that. You can replace the #pragma omp critical with any other thing that ensures thread safety, I didn't find a good small library for it that can be included in the DLL. I think the problems started when I tried to make case insensitive substring search algorithm. So maybe one of the headers that was required for it to work added dependency.

[b3lisario]

Well I'm sorry to disagree but I'm pretty sure it's the omp thing LOL

I don't know since which version papyrusUtil needs vcomp, at least version 1.9 does.

 

I did a quick test using the gcc tools. Got a cpp file that grabbed from wikipedia (http://en.wikipedia.org/wiki/OpenMP)

 

A simple compilation (g++ -Wall -fopenmp test.cpp) generates an executable with 4 dependencies: cygwin1.dll, cygstdc++-6.dll, kernel32.dll and cyggomp-1.dll (I guess this is the omp implementation for cygwin, as vcomp is the omp implementation for visual c)

 

Adding the -static flag (g++ -Wall -fopenmp -static test.cpp) produces a bigger executable with only 2 dependencies: cygwin1.dll and kernel32.dll

 

I don't know if this is valid for visual studio. Actually I have little experience working with C++.

 

EDIT: "Visual C++ does not support static linking of the OpenMP runtime" from http://msdn.microsoft.com/es-es/magazine/cc163717(en-us).aspx

[DeepBlueFrog]

Is it possible to save a Race? as in... the race of the player when you start a new game (for future detection of transformation into werewolf or vampire).

 

 

[b3lisario]

Is it possible to save a Race? as in... the race of the player when you start a new game (for future detection of transformation into werewolf or vampire).

Yes Race extends Form, so you can use StorageUtil.FormListAdd()

[h38fh2mf]

Hmm yes maybe you are right, I don't have seem to put it in changelog. I thought it was around version 1.5

 

C++ isn't really my favorite language so I'm not good at it, perhaps someone here can fix it.

[Ashal]

Maybe a dumb question, I come from a web programming background where thread issues are largely a non issue:

 

Is omp even really needed? Papyrus already thread locks scripts, which would make doing it from the plugin with omp somewhat superfluous, considering only one thread can ever access StorageUtil.pex at a time.

[h38fh2mf]

Yes but actor package update is in another thread. Right now I think this is the only reason it's necessary.

[b3lisario]

I did a bit of research.

In case the lock system is needed, this might work:

HANDLE ghMutex;

ghMutex = CreateMutex( NULL, FALSE, NULL);

WaitForSingleObject(ghMutex, INFINITE);
// do critical stuff
ReleaseMutex(ghMutex);

[h38fh2mf]

If you are going to use Winapi then http://msdn.microsoft.com/en-us/library/windows/desktop/ms682608%28v=vs.85%29.aspx is better to use. Might be best in this case because it doesn't add any additional dependency other than kernel32.dll but that's already used anyway I think.

[Ashal]

as far as I can tell JContainers just uses a simple spinlock, (source: https://github.com/SilverIce/JContainers/blob/master/JContainers/src/spinlock.h )

 

I'm incapable of performing any such update myself, so if b3lisaro is also unable to as well and h38fh2mf is unwilling to update it anymore, maybe the author of JContainers, Earendil, would be willing to apply the needed patch.

[Earen]

as far as I can tell JContainers just uses a simple spinlock, (source: https://github.com/SilverIce/JContainers/blob/master/JContainers/src/spinlock.h )

 

I'm incapable of performing any such update myself, so if b3lisaro is also unable to as well and h38fh2mf is unwilling to update it anymore, maybe the author of JContainers, Earendil, would be willing to apply the needed patch.

 

don't you had a wish to learn c++?

https://gist.github.com/SilverIce/8922765605aef4102902 - small instruction for those who will decide to use spinlock

[Ashal]

 

as far as I can tell JContainers just uses a simple spinlock, (source: https://github.com/SilverIce/JContainers/blob/master/JContainers/src/spinlock.h )

 

I'm incapable of performing any such update myself, so if b3lisaro is also unable to as well and h38fh2mf is unwilling to update it anymore, maybe the author of JContainers, Earendil, would be willing to apply the needed patch.

 

don't you had a wish to learn c++?

https://gist.github.com/SilverIce/8922765605aef4102902 - small instruction for those who will decide to use spinlock

 

 

I added the changes you list, however it becomes unable to compile under anything other than VS2013, VS2010 and older don't have mutex, and VS2012 gives version conflict errors with the JSON library StorageUtil uses.

 

Anyway around this? requiring the VS2013 redistributable is sort of an unreasonable requirement for just a plugin in my opinion, and it's still recent enough that there's not a reasonable assumption everybody would have it already. Considering SKSE and Skyrim use VS2008 that would be the ideal platform to compile for.

[h38fh2mf]

Try something like this:

#include <windows.h>

CRITICAL_SECTION cs;

void DoStuff()
{
EnterCriticalSection(&cs);
// here is thread safe
LeaveCriticalSection(&cs);
}

It shouldn't add any dependency.

[b3lisario]

About the corrupted storageUtil.save file. A possible workaround.

 

I am not sure what's the purpose of this file, I saw it stores data from some arrays that I don't use (SOS mod), so I think my StorageUtil.save always has a bunch of zeroes.

 

Users report that computer crashes for whatever reason and then storageUtil.save instead of a bunch of zeroes has a bunch of NULLs. The NULLs cause the game not to load properlly, I guess StorageUtil crashes trying to read the file.

 

So the real issue is why the file stores NULLs instead of zeroes. I have no idea:

- The code that writes to file seems OK, I don't know how it could output NULLs unless the memory is wrong.

- Could it be a I/O issue? Ascii code for NULL is 0, but it doesn't look like a crash during I/O because the space characters are OK.

 

A possible workaround could be to detect if the file has a NULL character and, in this case, assume each NULL is a zero.

[h38fh2mf]

Hmm I'm not sure, can you upload a broken storageutil save file here?

 

The file saves data that is independent of save game so all saves can share data, for example configure mod so that all save games share same configuration, or with a bit of scripting make a shared stash like in Diablo.

 

I'm not sure it can be I/O issue because it would throw exception and fail instead of writing badly. I think it must be a problem in my code, some scenario that is overlooked perhaps. Computer would have to crash exactly while it is writing, this seems very unlikely unless a mod had an error and the file is huge.

[b3lisario]

Ah that kind of data. I'm not aware of any mod uses that feature.

 

Here is a corrupted save, CTD was while doing a quicksave

http://www.loverslab.com/topic/26358-storageutilsave-corrupted/?p=713032

 

I was wrong before, the space characters are also NULLs. The file has 11 NULL chars (instead of 6 zeroes and 5 spaces, I guess).

[h38fh2mf]

Hmm I have no idea what would cause this. If the file is corrupt there isn't anything to do with it anyway, just delete the file and let plugin create new file itself but you would lose all data that was in the file. I don't think a check is necessary for NULLs because this is a freak accident, if it happens multiple times then we should search for the problem.

[b3lisario]

Yeah I totally agree that the optimal thing to do is fix whatever is causing the issue.

 

Deleting the file fixes that. The main problem is that is very hard to diagnose. Users are playing, they got CTD, and then Skyrim doesn't work anymore.

 

When the .save is damaged, Skyrim crashes at the start, when SKSE and StorageUtil begin to do their job. Is not like a common CTD issue. Users may think is not a mod issue because they were playing just a minute ago and now the game refuses to load. There is no clue about what could be wrong. Normal users don't know about that .save file.

 

They eventually may figure out it's a SKSE or PapyrusUtil issue, but because reinstalling those mods doesn't fix the problem, it's not likely they find the .save file. You need to wipe your Skyrim installation, or selectively delete folders until you can figure out the offending file.

 

The worst case scenario is with MO, where the save file goes to the "overwrite" folder. In that case not even doing a complete Skyrim reinstall fixes the issue.

 

And I think all this problems could be avoided just checking the integrity of that file before filling the maps. I think is not really that the file is corrupted, is just that it has the wrong data, for whatever reason.

[gooser]

I've lost count of how many times I've said to remove the "StorageUtil.save" file on the various forums...

 

Very annoying indeed.

[nutluck]

Well I often try and help people so this is good information to have. Now if I see people with that problem I can suggest deleting that file. Hopefully you guys can track down the cause of it and get it fixed.