HDT SMP for SSE?

[Yanme]

1 hour ago, Kalessin42 said:

Really, the ones at fault for why this stuff breaks on updates is Microsoft.

I wouldn't say Microsoft is at fault here. Regardless of any differences between POSIX and Windows shared libraries (which I wouldn't know about), the Sykrim executable was never linked against SMP/SKSE in the first place. Instead, SMP relies on directly using memory addresses for manipulating the executable. Offsets/addresses will always change on update, regardless if Windows, Unix etc.

[Yinkle]

Correct me if I'm wrong but once skse 2.0.8 (guessing the version) comes out and authors start updating mods that rely on version specific skse, any new features/fixes on those mods from here on out will no longer support older versions of skse.

 

So even if we never update the game to keep smp on the go we will eventually be crippling what we can do mod-wise.

[degenernate]

4 hours ago, Kalessin42 said:

Really, the ones at fault for why this stuff breaks on updates is Microsoft.

Wow, interesting! Thanks for advancing my knowledge of this under-the-hood stuff. Everything you said makes sense, but I'm still curious - if, early on in Skyrim's development, Todd Howard decreed from on high that any mod requiring a DLL would only ever have to be compiled once and would work forever on any further patched engine versions, would such a thing be theoretically possible under Windows? From my very inexperienced perspective, this feels like something that could be accomplished, but would require so much low-level fuckery it was never considered worth the effort. One way or another, it seems pretty unreasonable to expect Bethesda to even attempt refactoring how DLLs work. This mess with SMP is just an annoying symptom of the ecosystem modders and Bethesda inhabit, and no one is really the bad guy here.

 

But hey, if you recompile a new version of Skyrim, you immediately know what all the new memory addresses are, right? If Bethesda isn't even willing to just release that list to people like the SKSE devs to make updating dead easy, if Bethesda's stance is basically "fixed a typo in CC lol start up your hex editors yet again and go fish, assholes", doesn't it suggest an overall corporate stance that isn't prioritizing mod authors' needs?

[Yinkle]

5 hours ago, Kalessin42 said:

Really, the ones at fault for why this stuff breaks on updates is Microsoft.

Thanks for this clear explanation, much appreciated and I totally agree about the hdt thing, the fact that he has kept working on it as long as he has is amazing. Of course the same goes for racemenu etc

[Kalessin42]

4 hours ago, Yanme said:

I wouldn't say Microsoft is at fault here. Regardless of any differences between POSIX and Windows shared libraries (which I wouldn't know about), the Sykrim executable was never linked against SMP/SKSE in the first place. Instead, SMP relies on directly using memory addresses for manipulating the executable. Offsets/addresses will always change on update, regardless if Windows, Unix etc.

Well, if that's how they're hooking in SKSE, then that's that much worse, though I would expect that any SKSE plugins would then be loaded in the normal manner, which would be affected by Microsoft's design decisions with dlls. Regardless, the nature of dlls on Windows generally makes it so that when one thing is rebuilt, everything has to be rebuilt, which is incredibly annoying to developers. There are plenty of things that Microsoft has gotten right, but shared libraries is not one of them.

1 hour ago, degenernate said:

Wow, interesting! Thanks for advancing my knowledge of this under-the-hood stuff. Everything you said makes sense, but I'm still curious - if, early on in Skyrim's development, Todd Howard decreed from on high that any mod requiring a DLL would only ever have to be compiled once and would work forever on any further patched engine versions, would such a thing be theoretically possible under Windows? From my very inexperienced perspective, this feels like something that could be accomplished, but would require so much low-level fuckery it was never considered worth the effort. One way or another, it seems pretty unreasonable to expect Bethesda to even attempt refactoring how DLLs work. This mess with SMP is just an annoying symptom of the ecosystem modders and Bethesda inhabit, and no one is really the bad guy here.

 

But hey, if you recompile a new version of Skyrim, you immediately know what all the new memory addresses are, right? If Bethesda isn't even willing to just release that list to people like the SKSE devs to make updating dead easy, if Bethesda's stance is basically "fixed a typo in CC lol start up your hex editors yet again and go fish, assholes", doesn't it suggest an overall corporate stance that isn't prioritizing mod authors' needs?

Well, if a dll is dynamically loaded by the program, then as long as anything that the dll itself depends on isn't changed, then it should continue to work (that's what you typically get with plugins), but if the program links against it rather than dynamically loading it, then you're screwed. However, that sort of dll isn't something that would be done by modders. If they were doing dlls, it would be plugins, and the place that a dll plugin would normally be screwed is if the API it links against is changed. And that normally gets left alone if possible, because otherwise, it breaks all plugins for any software that uses plugins. But sometimes, changes are required, and every dll plugin has to be updated.

 

However, while what I described for dlls is correct in general, it sounds like the way that SKSE hooks into the game is a bit different (since it's starting the game rather than the game loading it). Rather, they're essentially hacking the game using hardcoded addresses in the binary, and that risks changing any time that the binary is rebuilt. So, the dll design problem is not the main reason that SKSE breaks, and to avoid having SKSE broken by updates, what would probably be required would be Bethesda building it in rather than it being effectively hacked in by folks in the community. However, if the SKSE plugin dlls need to be rebuilt and not just SKSE itself, then that is almost certainly due to the general problems that dlls have with not being able to handle changes to anything they depend on. But since I haven't studied SKSE in detail, I don't know what exactly what they're up to. I'm very familiar with the issues with dlls in general but not very familiar with the ins and outs of SKSE's internals.

[degenernate]

30 minutes ago, Kalessin42 said:

However, while what I described for dlls is correct in general, it sounds like the way that SKSE hooks into the game is a bit different (since it's starting the game rather than the game loading it). Rather, they're essentially hacking the game using hardcoded addresses in the binary, and that risks changing any time that the binary is rebuilt. So, the dll design problem is not the main reason that SKSE breaks, and to avoid having SKSE broken by updates, what would probably be required would be Bethesda building it in rather than it being effectively hacked in by folks in the community.

Gotcha. Hanlon's Razor applies everywhere here - Microsoft definitely didn't do a stellar job with its implementation of shared libraries, but that's just Microsoft being short-sighted and dumb, not actively evil; at some point, Bethesda probably would have been happy to integrate SKSE (though not SMP), but the SKSE team's employment clauses prevented it; Beth could also be doing more to make authoring mods less painful, but at some point you run up against the tech stack you're precariously built upon and QoL improvements aren't worth the man-hours; it could also be waaay easier to be an end-user of Skyrim mods, but that probably looks something like Bethesda buying out Nexus and completely rethinking how mods are deployed; etc. etc.

 

Anyway, thanks for the trip down the rabbit hole. We can now return to our regularly scheduled programming, which is basically just reiterating that there isn't any malice at work anywhere in here, but it's still going to be a pretty dick move if Hydrogen got bored with SMP but refuses to release the source code, won't hand the project over to a trusted peer, and can't be bothered to recompile future updates (including for SkyrimVR!) There's an unspoken social contract inherent in developing a mod that tons of other mods rely on, and that's not how you uphold it. Still, PE's source got released when work on that officially halted, so we know Hydrogen isn't anti-OSS, he/she is just sitting on this large codebase with a lingering obligation to continue development but no time/motivation to do so. All it would take is the right person to show up to take over - I'd gladly spend dozens of hours figuring out how to be the steward of SMP, but I'm nobody to Hydrogen.

 

Do we have any reports of further contact with Hydrogen on this topic? @realclone??

[Vyxenne]

20 hours ago, codylowey said:

the only thing that updating "breaks" is SKSE.  So, either Bethesda knows about SKSE and that updates break it, or they don't.

They not only know about it, they promised to stop doing it- that what the new *.cc files are supposedly all about. They're just breaking their own promise, which is what made me ask if it could be a deliberate attack to suppress the modding community to make getting paid mods via CC look more attractive.

 

20 hours ago, codylowey said:

If they know about SKSE, and know enough about it to know that updating the SE runtime "breaks" SKSE, then they also know enough to know that SKSE's devs always update SKSE within a couple of days to make it compatible and unbreak everything again.

Well, that's not quite the end of the Havok (pun intended) Bethesda wreaks with their updates... most if not all dll-based mods have to be updated as well after SKSE64 is updated- Stuff like SkyHUD, Racemenu, SMP-Physics, etc.

 

20 hours ago, codylowey said:

 isn't it really a decision Hydrogen made all by himself to have SMP be reliant on a hard-coded SE version number to work?  Couldn't he, if he had wanted to, simply added a "anything newer than version number x.xx.xxxx" check instead?

As I understand it, no- any and all dll-based mods must be updated for the specific game runtime AND the version of SKSE64 (if the mod is SKSE-dependent) that matches that particular game version. Mods like SkyUI, which are NOT dll-based but are merely SKSE64-dependent, are as you described- they work with any version of SKSE64 >= 2.0.x

 

That's why Hydro quitting on SMP is disastrous for those of us who use it currently as well as those using CBP Physics (or no physics at all) while waiting for HDT-SMP-SE to be debugged, "finished" and posted somewhere more accessible (to Westerners like us) than 9damao-dot-com. That's why people are hoping that Hydro will release the source code if he's not going to work on SMP-SE any more, so others can pick it up and finish it, and update it when Bethesda breaks everything again.

[Vyxenne]

19 hours ago, perrymwt said:

22 hours ago, Vyxenne said:

When TES VI drops in 17E 598, if it's still the Gamebryo engine I think I'll pass on it. To me, SSE is the pinnacle of Gamebryo engine development. If TES VI doesn't have in-built physics (like Tomb Raider 2013 does with Lara's hair) and full control over character creation like Skyrim has I'll just skip it and keep on with SSE.

I think I have to disagree with you on this point for a couple reasons.

 

1) There will be thousands of man-years of modders' work and knowledge base just going to be thrown away.  Everything would have to be relearnt and rebuilt, starting with basic import/export plugins for 3D programs and tools like Nifskope.

2) There is no guarantee that the new engine will be mod-able, at least as far as the models and animations are concerned.  With DA:I, Bioware simply refused to let any info on their engine get out by claiming it was too complicated for ordinary people to understand and therefore mod.  With DA:O, at least there was some info that modders could use.  Check out the number of body mods for DA:O and DA:I.  As for Bethesda, who knows what the suits will decide.

 

I hope you find these objections reasonable.

You make some very good points in favor of keeping the basic Gamebryo engine around for TES VI.  The ability to mod the game (and the availability of large numbers of mods) is crucial to my enjoyment of the game. I never played DA-O but I got DA-I at launch but ended up quitting it after maybe 50 hours because it got boring (very grindy) and there were no hair, face, body or outfit mods for it back then.

 

Maybe they can find ways to make it more flexible and realistic than they did in FO4- I really am astonished at the gawky walk-run animations and severely limited character face and body options in their latest game (FO4), which should have been "way better than Skyrim" but isn't even close to half as good as Skyrim. For some odd reason, all attempts I have made to create a truly good-looking character (even with many beauty mods installed) all turn out to look like variations of the default face, which is to say plain and frumpy- lipstick on a pig. But my biggest issue with FO4 is the walk-run animations- they are the worst I have ever seen in a modern game, and nobody has made better ones that I have been able to find. Yes, there are some that are labeled as better female walk-run, but they really aren't anything like the ones I have in Skyrim. I want my FO4 characters to look and move like this:

Spoiler

 

 

[Yinkle]

8 minutes ago, Vyxenne said:

That's why Hydro quitting on SMP is disastrous

I suspect the disastrousness of this to be quite a bit more far reaching. Once folk know about it they will quit modding for skyrim entirely if they are making mods that use smp in favour of another game. I know I would.

[kazuya482]

The abandonment of development for a stable physics mod has been enough for me to go back to LE.  Damn shame, but I've grown too accustomed to capes and cloth with actual physics to go backwards.  I'll have to live with the lower stability threshold .

[pugwash99]

22 hours ago, codylowey said:

Really, isn't it really a decision Hydrogen made all by himself to have SMP be reliant on a hard-coded SE version number to work?  Couldn't he, if he had wanted to, simply added a "anything newer than version number x.xx.xxxx" check instead?

That depends on the code involved, there are certain pointers that reference game code and memory locations that change after updates - that;s why we have to wait for skse. But because most of the mods use skse code they have to be recompiled against the new version to allow them to use the new locations. The two mods I have looked at - JContainers and SOS both use skse code rather than linking to the skse dll,

 

There are some ways around this but it would require those variables to be published in the skse dll's, then the mods could pick them up, it might already do this but I've only skipped through some of the code. Changes to memory structures could cause memory leaks or random ctd.

 

One way around that would be to publish all the data structures in a  C# library and recompile mods with the CLR option turned on so that the MS C++ can access them. When the code changes in the C# library, which would have to be published with skse, the C++ code would use the new types immediately without updating their own code. Never having done this I can't confirm that this would happen with a 100% certainty but it would reduce the need of recompiling mods unless there's a big change.

 

[full_inu]

Does 9damao have official announcement about stopping development?

Thread, hydrogensaysHDT's commentary... etc

[Rex89]

On 9/2/2018 at 2:23 PM, Vyxenne said:

They not only know about it, they promised to stop doing it- that what the new *.cc files are supposedly all about. They're just breaking their own promise, which is what made me ask if it could be a deliberate attack to suppress the modding community to make getting paid mods via CC look more attractive.

technically this only happens when they add a new frame work to the game that a CC mod uses, but the plus side is when they add a new frame work the modding community can use it.

so its like a double edged sword.

I've seen the game update only cc files a few times so technically they haven't  broken there own promise but instead they only reduce the amount of updates needed.

i don't see the problem of have a once in awhile update for skyrim SE compared to fallout 4s once a month updates.

i mean the last update for skyrim SE was on 3 April 2018 before this one came out, i say that pretty tolerable, same goes for fallout 4 it only update's once a month witch is predictable and also tolerable at lest its not once a week.

[full_inu]

We have SKSE64 with all source code, and nobody managed to make a quick fix before SKSE Team will make an official update.

Did we really have a people, that can continue development of HDT-SMP?

_{really makes you think.jpg}

[Vyxenne]

3 hours ago, Barragan said:

We have SKSE64 with all source code, and nobody managed to make a quick fix before SKSE Team will make an official update.

Did we really have a people, that can continue development of HDT-SMP?

_{really makes you think.jpg}

You say that as if having the source code was the only requirement. Nobody has updated SKSE64 because it would not be legal for anyone to do that even though the source code is available.

 

Drawing the conclusion that nobody can do something just because nobody has done it is invalid logic- a fallacy.

[Tooneyman23]

1 hour ago, Vyxenne said:

You say that as if having the source code was the only requirement. Nobody has updated SKSE64 because it would not be legal for anyone to do that even though the source code is available.

 

Drawing the conclusion that nobody can do something just because nobody has done it is invalid logic- a fallacy.

Plus it always takes the team time to update. ? 

[full_inu]

2 hours ago, Vyxenne said:

You say that as if having the source code was the only requirement. Nobody has updated SKSE64 because it would not be legal for anyone to do that even though the source code is available.

 

Drawing the conclusion that nobody can do something just because nobody has done it is invalid logic- a fallacy.

Uh, maybe my logic was really wrong this time :>)

[Grey Cloud]

2 hours ago, Vyxenne said:

a fallacy.

Given the context, a phallusy surely?

[tasairis]

9 hours ago, Barragan said:

We have SKSE64 with all source code, and nobody managed to make a quick fix before SKSE Team will make an official update.

Did we really have a people, that can continue development of HDT-SMP?

_{really makes you think.jpg}

Actually no, we don't have all the source. Some of the more sensitive parts are closed-source, and that's the main reason why the project hasn't been DMCAed. (IIRC there was some private communication between the SKSE team and a Bethesda dev relating to this.) What we, the community, have access to are the "headers" and other components necessary for mod authors to create their own DLL-based mods.

[joenothing12]

I can second tasairis.  before skse was released for sse I looked at it pretty closely.  What's there isn't enough to compile skse, it's enough to compile DLL's that use functions in SKSE. There was no point looking further, i haven't done any reverse engineering in two or three decades.  It would be a monumental task to recreate their work.

When they did announce they were working on it they were not interested in help finding the (addresses of) function calls they were hooking into, in light of a potential DMCA threat it makes sense. 

By comparison smp is something the community might, perhaps, maybe be able to move forward on.  It's a rather specialized skill set even among programmers but there's free and open APIs for bullet physics at: https://github.com/bulletphysics/bullet3

It's possible, just damn hard.

[Tooneyman23]

1 hour ago, joenothing12 said:

I can second tasairis.  before skse was released for sse I looked at it pretty closely.  What's there isn't enough to compile skse, it's enough to compile DLL's that use functions in SKSE. There was no point looking further, i haven't done any reverse engineering in two or three decades.  It would be a monumental task to recreate their work.

When they did announce they were working on it they were not interested in help finding the (addresses of) function calls they were hooking into, in light of a potential DMCA threat it makes sense. 

By comparison smp is something the community might, perhaps, maybe be able to move forward on.  It's a rather specialized skill set even among programmers but there's free and open APIs for bullet physics at: https://github.com/bulletphysics/bullet3

It's possible, just damn hard.

Funny when I keep seeing and hearing the words damn hard. I makes me think there is someone out there where this is easy. 

[Vyxenne]

19 hours ago, Grey Cloud said:

21 hours ago, Vyxenne said:

a fallacy.

Given the context, a phallusy surely?

[Grey Cloud]

Everyone's a critic. ?

[Gromilla]

??@Vyxenne  ???

 

[full_inu]

I knew it...

SKSE64 was updated just now. SMP no more works!