Jcontainers skse64 upgrade attempt.

[pugwash99]

As we all know jcontainers is an skse plugin that many of our favourite mods rely on.

 

I've been making attempts to upgrade it to skse64 but first I wanted to make sure I was  building the original oldrim version correctly. It took me a while to do it but I finally managed it Wednesday morning after making several mistakes along the way. As a test to confirm it was correct  I played on Oldrim for a couple of hours with the version I built and registered sexlab animations and used mods that relied on it with no issues appearing - no ctd and no issues with animations - it registers in skse, no errors in the Jcontainers.log file

 

So why have I created this thread, it's so that if anyone else wants to try they can get a head start as my knowledge of C++ is limited and the issues with upgrading to 64bit may well be beyond my current talents. The issues stem from the version of the compiler it was written for there's nothing wrong with code at the time but Microsoft have made changes that have broken it when attempting to link it to the current skse64.

 

Ok the technical details now (read excuses). JC v3.3A was written with VC120 - Visual Studio 2013 as was the last version of skse. The current skse64 is written for VC140 - Visual Studio 2015 / Visual Studio 2017. When switching JC to vc140 it breaks several routines in it's main codebase and in it's dependencies that are absolutely necessary for it to function, downgrading skse64 to vc120 breaks several objects in the same way. I'll keep trying to workout how to fix it but google searches can only take you so far  especially when the errors aren't exactly helpful in finding the cause.

 

If you're not a programmer the rest will probably mean little to nothing.

 

So to get a working build environment for skse oldrim version.

 

Required Software:

Cmake available from https://cmake.org/

Python available from https://www.python.org/ - install with path options

Visual Studio 2013 (with latest update)

Visual Studio 2015 or 2017 (with latest update)

 

After installation reboot as the environment variables need to update.

 

Code requirements

Jcontainers V3.3 domain - https://github.com/SilverIce/JContainers/releases

Jansson 2.7 - http://www.digip.org/jansson/releases/

Boost 1.5.9 - http://www.boost.org/users/history/version_1_59_0.html

Googletest - https://github.com/google/googletest

Flatbuffers - https://github.com/google/flatbuffers

luajlt - https://github.com/LuaJIT/LuaJIT

 

Unzip jcontainers and then unzip the others into their respective places in the 'dep' folder.  There are a couple of things you'll need to do to get it all to compile in VS.

 

Boost:

 

If vs2013 and vs2015 / vs2017 are installed you'll need to make some changes to the boost library to get it to compile. First open a Visual Studio command prompt and go to the 'dep\boost' folder and run 'bootstrap.bat' this will create the build program. Next edit the build_libraries.bat file and add 'toolset=msvc-12.0' without the quotes after b2. Now run it. If all goes well you should see lots of lines of text with vc120 in them. If you see vc140 then something went wrong. All of this is where I initially went wrong - Silverice added some code which isn't in the original release that is used by JC so downloading the prebuilt libraries is a complete waste of time. The newly compiled libraries end up in 'dep\boost\boost_lib\lib'

 

Jansson:

Jansson is missing jansson_private_config.h file that's required to compile it in Visual Studio. After installing cmake find cmake-gui and select the jansson folder using 'Browse Source', select a folder to build in, we'll need it later. Click on configure and select the visual studio 12 2013. At this point the file we're missing will be build and placed in the 'private_include' folder you selected to build to. Copy jansson_private_config.h and place it with the rest of the source code.

 

Now we're in a position to load the jcontainers project. You must have vs 2013 installed (from my experience) but you can open the project in vs 2015 or vs2017, I'd suggest 2015 just to save time and personally I prefer the editor but when asked if you want to update the projects select 'NO', they should still have the visual studio 2013 tag in the solution explorer.

 

This will build to folder under \program files (x86) to change it open the properties of the 'post-build' project - in 'build events' - 'pre-build event' change the path in the command line.

 

At this point you should be able to do a build on the solution and get a copy of the oldrim version of jcontainers.

 

The upgrade begins

 

First step - backup the project - zip, .7z, .rar or plain old copy & paste but keep a backup of the working solution.

 

To get a skse64 version we need to change all the projects over to vc140 -  sadly this is where it all starts going wrong for me.

 

Boost needs recompiling again in a visual studio 2015 command prompt, so undo the editing we did before (or just delete the folder and drop in the uncompiled code), two files will need changing.

 

basic_binary_iprimitive.hpp line 117 and basic_binary_oprimitive.hpp line 112 - I believe these two routines, 'load' and 'save', are the code Silverice added which broke the compatibility to the original precompiled boost and why we have to do all this. They currently have 'BOOST_ARCHIVE_OR_WARCHIVE(void)' declarations but this doesn't work in vc140 so I changed them to 'BOOST_ARCHIVE_OR_WARCHIVE void' run the 'build_libraries.bat' routine again but with toolset=msvc-14.0 instead. Lots of lines of white text flash by with vc140 in the file names and you should end up with 7 new lib files. I'm not convinced changing those two lines is the correct action and it might even be the cause of my problems but changing them to anything else and it doesn't compile.

 

In solution explorer open the properties of each project and under 'General' change the platform toolset to 'Visual Studio 2015 (v140)' to match skse64.

 

Congratulations you've now reached where I am. With no code open I think it will show 3 warnings and 9 or more build errors related to std::map or form_ref related to code, you'll also get some compiler errors that can cleared with the additions to a few extras to the 'C/C++ - All options - Preprocessor definitions' all of these are errors are caused by the changes made by Microsoft in the vc140 platform tools.

 

Normally I'm a .Net programmer where you can take something written in VS2010 .Net 2.0 and use it in VS2017 .Net 4.7.1 with no changes or even recompiling, so suddenly finding myself in the middle of a sea of C++ errors that make little sense is bit new when the code worked perfectly 2 minutes before. 

 

As an alternative solution I tried downgrading skse64 to the vc120 platform - similar errors to the above as it's been fixed for vc140 but broken for vc120.

 

Ok, that's the story so far - two weeks of head scratching and a lot of swearing to get stuck right before the finale. If I can work out how to fix the issues then switch to 64bit it should then be a relatively simple procedure to switch over to the skse64 code - probably.

[mangalo]

I'm afraid I can't be of any help here but just wanted to say I appreciate your efforts and hope this topic gets the attention it deserves

[sunhawken]

Your best bet would be going on reddit and also talking to users who were able to already convert they plugins , I only have python coding EXP. 

[DykkenTomb]

No.. Go to the discord servers and talk with individuals there. It's a real type chat scenario and will help you much more quickly. 

[pugwash99]

I didn't help that the next day while moving the build to my desktop the psu decided to kill itself with a load bang and took out the fuse in the power cable and tripped all the breakers. I've only just got it working again with a new psu, thankfully nothing else appears to have been damaged. I did try the original psu in a socket but it trips the breakers immediately.

 

I'm hoping that comparing the changes between skse and skse64 will guide me towards a solution as it has gone through a similar issue.

 

I've got a feeling that most if not all the issues stem from 1 or 2 changes which C++ announces as other problems as there are several posts about deleted functions which are actually something else. I know that most of the dependent libraries compile in vc140 with no issues and I think almost all the issues come from the form_ref object.

[javier_himura]

Check this fork in Github, there all the errors with the migration of the code to VS2012, SKSE64 and the laterst version of boost are fixed

https://github.com/ryobg/JContainers

[pugwash99]

Yeah I saw it the other day as I got included in a thread on the original github library.

 

https://github.com/ryobg/JContainers/tree/port64

 

I think ryobg with your contributions are doing a far far better job of it than I could ever do given how far it's progressed. As I said I'm not a C++ programmer, I do code but it's obvious to me they've far more experience than I in C++.

 

One thing I did manage to fix was the post build command line for python where it goes to the hard coded folder but the thread is closed 

 

Quote

This will build to folder under \program files (x86) to change it open the properties of the 'post-build' project - in 'build events' - 'pre-build event' change the path in the command line.

 

as each release has the MO folder coded in that command line of the python. So I might download it and give it try.

 

<i should really try to read user names in posts >

 

 

[javier_himura]

2 hours ago, pugwash99 said:

Yeah I saw it the other day as I got included in a thread on the original github library.

 

https://github.com/ryobg/JContainers/tree/port64

 

I think ryobg with your contributions are doing a far far better job of it than I could ever do given how far it's progressed. As I said I'm not a C++ programmer, I do code but it's obvious to me they've far more experience than I in C++.

 

Ey, don't worry, it was the same for me,i am a C# developer. I have been triying to compile JContainers for weeks without sucess. I resolved the problem of the native function but it was unable to fix the compiling errors after switching from VS2013 to VS2015 or VS2017. But ryobg is more experience in C++ and he was able to resolve the problem and the upgrade to the last version of boost

 

2 hours ago, pugwash99 said:

One thing I did manage to fix was the post build command line for python where it goes to the hard coded folder but the thread is closed 

You can open a new issue in ryobg Github and tell him there how to fix it. What i did was more rudimentary, i created the hardcoded folder and the compiling generate correctly the zip with JContainers x64 ready to be distribute, now it just need to be tested.

[Hylysi]

I still have a compile error referring to std::conditional while compiling JContainers from the port64 tree.

Does this not occur for anyone else or what did you change to solve this error?
 

Spoiler

1>------ Build started: Project: JContainers, Configuration: Debug x64 ------
1>tes_api_3.cpp
1>D:\JC\JContainers\src\reflection/tes_binding.h(263): error C2976: 'std::conditional': too few template arguments
1>C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.12.25827\include\xtr1common(65): note: see declaration of 'std::conditional'
1>D:\JC\JContainers\src\api_3/tes_object.h(37): note: see reference to class template instantiation 'reflection::binding::state_proxy<T>::magick<collections::object_base *tes_api_3::tes_object::retain(collections::tes_context &,tes_api_3::tes_object::ref,const char *)>' being compiled
1>        with
1>        [
1>            T=collections::object_base *(__cdecl *)(collections::tes_context &,tes_api_3::tes_object::ref,const char *)
1>        ]
1>D:\JC\JContainers\src\reflection/tes_binding.h(263): error C2955: 'std::conditional': use of class template requires template argument list
1>C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.12.25827\include\xtr1common(65): note: see declaration of 'std::conditional'
1>D:\JC\JContainers\src\reflection/tes_binding.h(193): error C2976: 'std::conditional': too few template arguments
1>C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.12.25827\include\xtr1common(65): note: see declaration of 'std::conditional'
1>D:\JC\JContainers\src\api_3/tes_jcontainers.h(17): note: see reference to class template instantiation 'reflection::binding::proxy<T>::magick<bool tes_api_3::tes_jcontainers::__isInstalled(void)>' being compiled
1>        with
1>        [
1>            T=bool (__cdecl *)(void)
1>        ]
1>D:\JC\JContainers\src\reflection/tes_binding.h(193): error C2955: 'std::conditional': use of class template requires template argument list
1>C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.12.25827\include\xtr1common(65): note: see declaration of 'std::conditional'
1>reflection.cpp
1>D:\JC\JContainers\src\reflection/tes_binding.h(193): error C2976: 'std::conditional': too few template arguments
1>C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.12.25827\include\xtr1common(65): note: see declaration of 'std::conditional'
1>src\reflection\detail\reflection.cpp(87): note: see reference to class template instantiation 'reflection::binding::proxy<T>::magick<void reflection::reflection___Test::TestBody::test_class::nothing(void)>' being compiled
1>        with
1>        [
1>            T=void (__cdecl *)(void)
1>        ]
1>D:\JC\JContainers\src\reflection/tes_binding.h(193): error C2955: 'std::conditional': use of class template requires template argument list
1>C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.12.25827\include\xtr1common(65): note: see declaration of 'std::conditional'
1>Generating Code...
1>Done building project "JContainers.vcxproj" -- FAILED.
========== Build: 0 succeeded, 1 failed, 6 up-to-date, 0 skipped ==========

 

 

[russianmind]

Really appreciate your effort in all this!

 

Wish I knew a thing about coding to help.

[Pfiffy]

11 hours ago, javier_himura said:

Ey, don't worry, it was the same for me,i am a C# developer. I have been triying to compile JContainers for weeks without sucess. I resolved the problem of the native function but it was unable to fix the compiling errors after switching from VS2013 to VS2015 or VS2017. But ryobg is more experience in C++ and he was able to resolve the problem and the upgrade to the last version of boost

 

You can open a new issue in ryobg Github and tell him there how to fix it. What i did was more rudimentary, i created the hardcoded folder and the compiling generate correctly the zip with JContainers x64 ready to be distribute, now it just need to be tested.

I'm not into coding, but I would like to help with testing.

[javier_himura]

11 hours ago, hylysi said:

I still have a compile error referring to std::conditional while compiling JContainers from the port64 tree.

Does this not occur for anyone else or what did you change to solve this error?
 

  Reveal hidden contents

1>------ Build started: Project: JContainers, Configuration: Debug x64 ------
1>tes_api_3.cpp
1>D:\JC\JContainers\src\reflection/tes_binding.h(263): error C2976: 'std::conditional': too few template arguments
1>C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.12.25827\include\xtr1common(65): note: see declaration of 'std::conditional'
1>D:\JC\JContainers\src\api_3/tes_object.h(37): note: see reference to class template instantiation 'reflection::binding::state_proxy<T>::magick<collections::object_base *tes_api_3::tes_object::retain(collections::tes_context &,tes_api_3::tes_object::ref,const char *)>' being compiled
1>        with
1>        [
1>            T=collections::object_base *(__cdecl *)(collections::tes_context &,tes_api_3::tes_object::ref,const char *)
1>        ]
1>D:\JC\JContainers\src\reflection/tes_binding.h(263): error C2955: 'std::conditional': use of class template requires template argument list
1>C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.12.25827\include\xtr1common(65): note: see declaration of 'std::conditional'
1>D:\JC\JContainers\src\reflection/tes_binding.h(193): error C2976: 'std::conditional': too few template arguments
1>C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.12.25827\include\xtr1common(65): note: see declaration of 'std::conditional'
1>D:\JC\JContainers\src\api_3/tes_jcontainers.h(17): note: see reference to class template instantiation 'reflection::binding::proxy<T>::magick<bool tes_api_3::tes_jcontainers::__isInstalled(void)>' being compiled
1>        with
1>        [
1>            T=bool (__cdecl *)(void)
1>        ]
1>D:\JC\JContainers\src\reflection/tes_binding.h(193): error C2955: 'std::conditional': use of class template requires template argument list
1>C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.12.25827\include\xtr1common(65): note: see declaration of 'std::conditional'
1>reflection.cpp
1>D:\JC\JContainers\src\reflection/tes_binding.h(193): error C2976: 'std::conditional': too few template arguments
1>C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.12.25827\include\xtr1common(65): note: see declaration of 'std::conditional'
1>src\reflection\detail\reflection.cpp(87): note: see reference to class template instantiation 'reflection::binding::proxy<T>::magick<void reflection::reflection___Test::TestBody::test_class::nothing(void)>' being compiled
1>        with
1>        [
1>            T=void (__cdecl *)(void)
1>        ]
1>D:\JC\JContainers\src\reflection/tes_binding.h(193): error C2955: 'std::conditional': use of class template requires template argument list
1>C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.12.25827\include\xtr1common(65): note: see declaration of 'std::conditional'
1>Generating Code...
1>Done building project "JContainers.vcxproj" -- FAILED.
========== Build: 0 succeeded, 1 failed, 6 up-to-date, 0 skipped ==========

 

 

Those errors do not happen in my PC, but i compiled in Release, not in Debug, and building boost with the files of boost 1.66, not with the ones from the original branch. Also check that you have downloaded all the externals repositories

Also the first time i opened the solution VS2017 told me to download Windows 10 SDK
 

[pugwash99]

I've just built a copy, still using 2015 at home so I switched the compile to v140_xp removed the 5.02 requirement in the linker for the post built. Switched it to release 64bit and built.

 

Several warnings, switched some of the int references to size_t so that it scales with the processor. Got a build copy which I'll try in moment.

 

update: skse log shows it has loaded the dll but ctd before getting to the loading menu.

[Hylysi]

8 hours ago, javier_himura said:

Those errors do not happen in my PC, but i compiled in Release, not in Debug, and building boost with the files of boost 1.66, not with the ones from the original branch. Also check that you have downloaded all the externals repositories

Also the first time i opened the solution VS2017 told me to download Windows 10 SDK
 

Interesting I must have done something wrong along the way.

 

I'm compiling with VS2017 on Windows 10, I've downloaded and placed everything from the original post in their corresponding Jcontainers-port64 sub directories too (save for boost which is 1.66 instead of 1.59.) Boost compiles with no issues as well as every other project as well. I've tried changing from debug to release and the entire project is set to build as x64.

[pugwash99]

I'm compiling on Windows 7, changed all the projects to vc140, not vc140_xp. and retargeted the projects to 8.1. Also not seeing those errors. 

 

What I am getting are warnings about size_t to int. I actually fixed those the first time I compiled but I was getting a ctd when the menu starts to load so I undid the changes but still getting the ctd at the same point. The dll is definitely loading, tried removing / disabling all my mods and load with only the standard mods but still getting the ctd, nothing in the log files.

 

SKSE64.log compared between with or without jc, is missing some lines after the reading translations.

 

I've found the jcontainers log file, it's still going to the 'my games\skyrim\skse' folder, it must be hard coded.

 

JContainers 3.3.64
plugin loaded
Registering functions started
'Registering functions' throws 'boost::filesystem::directory_iterator::construct: The system cannot find the path specified: "Y:\steamlibrary\steamapps\common\Skyrim Special Edition\Data\SKSE\Plugins\JCData/Domains/"' of type 'class boost::filesystem::filesystem_error'
Registering functions finished in 0.005000 sec
 

The build doesn't create the folder, I just added it myself as I'd created a rar for NMM to install and it's missing. Adding the \Domains\ folder fixes the crash and I'm at the game menu.

 

skse_callbacks.cpp has the oldrim folder hard coded on line 121

 

edit:

 

ok got into game, loaded a save and so far all going well. Loaded the radiant prostitution config page and tried to check for quests - game hung with the hidden debug button.

[javier_himura]

I will make a pull request changing the hardcoded oldrim folders to use the folders from skyrim special edition instead.

About the Domain folder i checked the oldrim version and the mod author added an empy Domain folder to the zip that is in nexus. The original source code does not create that folder when the zip is created, for now anyone who get JContainers from the source will need to create the folder

 

EDIT: Both folder problems fixed

[pugwash99]

OK, got a debug session in skyrim running. I built a debug version of jcontainers and installed it using NMM, set the command line in the debugging section of the project properties to the skrim.exe with the option to attach to existing process. Used NMM to start skyrim using skse and then started a VS debug session.

 

Getting an a Priveleged instruction exception in string.h line 53.

 

    string_ref& operator = (const char* ref) { assign(ref); return *this; }

 

This is when loading a cleaned save with radiant prostitution. It's ref variable is 'Data/MF_RP_Config.json' file, the RP configuration json file.

 

DEFINE_MEMBER_FN(ctor, string_ref *, 0x00A511C0, const char * buf);
DEFINE_MEMBER_FN(Set, string_ref *, 0x00A51210, const char * buf);
DEFINE_MEMBER_FN(Release, void, 0x00A511B0);

 

I believe these three lines are causing the issue as the 0x00A51xxx numbers are for skyrim, they should match the ones in gametypes.h which for SE are

 

        DEFINE_MEMBER_FN(ctor, string_ref *, 0x00C28280, const char * buf);
        DEFINE_MEMBER_FN(Set, string_ref *, 0x00C283F0, const char * buf);
        DEFINE_MEMBER_FN(Release, void, 0x00C283D0);

 

after recompiling I've gotten past the loading issue and no crash ctd when loading the clean save. From the RP config page telling it to search for quests doesn't ctd but it isn't registering them as quests.

[kinanizer]

I've been watching your progress on github, really glad for your efforts!

[pugwash99]

Looks like the debug release has some issues, RP doesn't register the quests and I've been gettting ctd's. Trying to start a ;home  delivery' mission and it says jcontainers isn't installed.

 

edit: debug release was fine I think it was the attached debugger causing the problem.

 

However, installing a release build and the quest count updates and RP has just begun a 'home delivery' mission  

 

hmm, reloading a save after completing it resulted in a ctd with the amount of erros in the skse log I'd say it's either corrupting the save game  or the load function is working as the skse save file has the data in it.

 

 

[pugwash99]

Well that's annoying, the ctd I was getting and the corrupted save which I thought was happening might not actually be a problem. I think it's because I replicated the original folder stucture of jcontainers a bit too well as the dummy file in the domains folder might be the issue.

 

Without it I can save and load with no issues. With and I get a ctd on every load of save that has jcontainers data in the skse save file.

 

So yeah, atm it looks like it's working.

[infiniteone]

Wow, you guys sure chewed that up quick!  Thanks!  I wish I knew C#, maybe someday.

[pugwash99]

I can't take credit with the code as the most I did was find a couple of issues with oldrim references, a missing folder and being too efficient by copying the originals mods file structures causing a skse save file problem (I think it's the old dummy file causing the problem but it might also be causing something else to go wrong without it).

 

It really needs some more testing to confirm it's 100% but Radiant Prostitution appears to be working correctly.

[blueballbob]

You going to go ahead and publish it when your done testing?   I think the original authors still responding, and looked like hed be ok with it on reddit.

[pugwash99]

I would imagine so, where is another matter. Ryobg and javier_himura did the hard work converting it so I'd rather let them decide where and when.

[General Neondaze]

Could someone confirm that I have installed JContainers correctly?

 

I downloaded the zip from GitHub (clone/download) and installed using MO2. As there were a lot of folders I selected the folder labelled JContainers.

 

No conflicts have been caused loading and saving. I specifically wanted Jcontainers to be able to use SlaveTats. Opening the MMC reports an error about a JSon API version.

 

Have I actually installed correctly or have I missed something along the way.

 

I also really appreciate all the work that has been carried out to add this functionality to SSE. :-)