Need to find what specifically causes a crash in an esp

[AlphaAndOmega]

After some changes to the load order the game started to ctd on launch with some weird data in the crash log. Managed to figure out what esp is responsible: Requiem for the indifferent.esp. It's a balance patch that is redone every time the load order changes that conforms keywords and actor stats to the norms of the Requiem mod. This means that's not really possible to just not use it and, well, it has only decided to act up now. Since I did not change anything about the patcher itself, the problem lies in the things that are patched. Thus, I need help reading the crash logs tofigure out at least the type of the mod that's responsible and then go from there. Or I'm wrong and the issue is to be approached differently.

 

I'm moderately confused about the type of the crash though. Immediate ctd is usually the result of missing masters or skyrim version mismatch, but MO is silent about these.

 

Here is an example of the logs crash-2024-04-07-06-19-22.logPapyrus.0.log.

 

Either way, any input is much appreciated.

[traison]

It tried to dereference RAX which at the time of the crash was 0x0 (a null pointer). The stack was read wrong for some reason, that is not a valid stack. It is possible something altered the value of RSP when it shouldn't have, its not a generic purpose register, at least not traditionally. We can not trust that the value in RCX is involved, especially since its value is about to get overwritten once RIP returns.

 

This crash log is essentially useless. There is nothing actionable here besides the callstack offsets, and getting a debugger into SkyrimSE.exe is easier said than done. Considering however where it crashed, one might assume this could be a game data issue. Since it seems you already (somehow) determined that a plugin is involved, the only reasonable next step I see is to open your load order as-is in xEdit and go through everything in "Requiem for the indifferent.esp", paying close attention to missing or deleted record fields and edits that conflict with other mods (overwriting or overwritten).

 

Edit: I have never seen the stack read wrong. I feel like this is not the fault of CrashLoggerSSE as this is not exactly a tricky thing to get right. However, since you're on 1.5.97 you'll have multiple options for CrashLoggerSSE versions. You could try a different version, just to see if you get different results. Producing crash logs like these is not a straight-forward process afterall, maybe something went wrong internally. I use this version on 1.5.97: CrashLogger-59818-1-0-0-4-1653384565

Edited April 7, 2024 by traison

[AlphaAndOmega]

1 hour ago, traison said:

I use this version on 1.5.97: CrashLogger-59818-1-0-0-4-1653384565

Tried that one, can't say that it's significantly differentcrash-2024-04-07-10-43-08.log.

1 hour ago, traison said:

Since it seems you already (somehow) determined that a plugin is involved, the only reasonable next step I see is to open your load order as-is in xEdit and go through everything in "Requiem for the indifferent.esp", paying close attention to missing or deleted record fields and edits that conflict with other mods (overwriting or overwritten).

While doing the initial debugging and trying to retrace the steps taken I've tried loading the game without said plugin and it worked without an issue. Due to the nature of the plugin, it's meant to overwrite everything it touches, but looking into missing/deleted records (How could it happen though, since the plugin is made on the spot?) is a valid point. I'll go boot SSEEdit and have a look. 

 

That being said, how do I properly identify a broken record?

 

Edit: Since LOOT is usually the one that's telling me about ITMs, deleted references and all that, I've decided to go there and check. It seems to be adamant about there being nothing wrong with the plugin.

Edited April 7, 2024 by AlphaAndOmega

[traison]

41 minutes ago, AlphaAndOmega said:

How could it happen though, since the plugin is made on the spot?

 

I'm not familiar with these generated patches, but there's been a few cases where bashed patches for instance have caused crashes like these - though the crash logs weren't broken in those. Personally I would guess that the plugin generation framworks these use aren't flawless. Maybe they miss a field somewhere, or write a bit wrong somewhere else - like I said, not familiar with these.

 

41 minutes ago, AlphaAndOmega said:

That being said, how do I properly identify a broken record?

 

Back in Skyrim LE, a (relatively speaking) somewhat common null pointer crash caused by plugins could be found in armor mods where the author forgot to set an inventory preview/icon value for one of the items. Opening the inventory, or scrolling over the item in the inventory would cause the game to crash with a null pointer reference.

 

So what is a broken record then? A field that is missing where Skyrim didn't expect it to be missing - i.e. a field that Skyrim doesn't check the value of before using it. Is there a list of these? Doubt it. Since I don't know the internal workings of the entire game engine (who does?) I would focus on records that this patch alters that were also altered by some other mod, or records altered by this patch that again get overridden by another mod. The nature of this patch plugin, since you said its automatically generated, may be such that it already incorporates all edits made by previous mods, and its probably loaded last meaning there's nothing else to override it. In this case I'm not sure how to proceed - I'd probably have to see this issue for myself.

 

The sledgehammer approach to this issue could be to remove edits made by the patch plugin one-by-one until the issue goes away. By drag-n-dropping between the columns in xEdit you can do this relatively quickly. I suppose it depends on how massive this plugin is whether or not this is practical.

 

Edit: I'm about to grab a sort of memory snapshot from my 1.5.97 and see if I can find anything at those callstack offsets. No promises though but I'll post again if I find something. I'm mostly interested in what happened to the RSP register.

Edited April 7, 2024 by traison

[AlphaAndOmega]

18 minutes ago, traison said:

The sledgehammer approach to this issue could be to remove edits made by the patch plugin one-by-one until the issue goes away. By drag-n-dropping between the columns in xEdit you can do this relatively quickly. I suppose it depends on how massive this plugin is whether or not this is practical.

I threw the plugin into xEdits autocleaner, it did find and delete several things, but it did not improve the situation overall. I then asked xEdit to check for errors and it returned me 200+ results, making it practically unusable. Thus, still at square one.

Manual is moderately out of question, the plugin is huge as a result of the bloated load order of mine (and it touches a significant portion of it).

 

I'll try to go though its masters and look into those that I've updated recently. Maybe I'll find the issue this way.

Edited April 7, 2024 by AlphaAndOmega

[traison]

Right so looking at this thing in the debugger (keep in mind that this is on my modded setup) I immediately see something interesting. Just before the faulting offset at 05E1F22 there's a call instruction:

call <skyrimse.TESObjectREFR::SetObjectReference_140296520>

 

And in this "function" we have this:

call <skyrimse.BGSDestructibleObjectForm::IsDestructible>

 

...as well as this unconditional jump:

jmp <skyrimse.TESForm::SetDestructible>

 

So, is there a flag like "IsDestructible" or anything like this in an object reference? Does Google have anything to say about this? Since the player object was mentioned in the RCX register, how about that record (form id 0x14)?

 

 

Edit: It also looks like RCX is used as one of the parameters for the function containing the faulting offset. The player character is quite likely involved.

 

Edit again: SetObjectReference seems to be specifically called on an Actor. Again pointing to the player character.

Edited April 7, 2024 by traison

[AlphaAndOmega]

12 minutes ago, traison said:

The player character is quite likely involved.

I did see mentions of it, but I don't know how to approach it, since it isn't really telling me what's wrong with it. I would've understood it if it crashed on save load, but it's the game that's trying to start, the player character doesn't even exist yet (from my understanding, at least).

 

Edit: This, however, means that the problem is inherent to the character itself, thus creating an alibi for all the armour mods. That's something, at least.

Edited April 7, 2024 by AlphaAndOmega

[traison]

Another pointer to the player is near offset 05B6E47:

 

mov ecx,7

call <skyrimse.LookupFormById_140194230>

 

It's passing the ActorBase id of the player (0x7) to LookupFormById. Again pointing heavily at the player actor being involved.

 

I think that's your answer: Something to do with destruction flags/data/fields on the player actor or the player actor's base. Cross-reference this with what the plugin is changing and you should find it.

[traison]

11 minutes ago, AlphaAndOmega said:

I did see mentions of it, but I don't know how to approach it, since it isn't really telling me what's wrong with it.

 

I'm going through the Skyrim.esm here, found this in form id 0x7:

 

[AlphaAndOmega]

11 minutes ago, traison said:

 

I'm going through the Skyrim.esm here, found this in form id 0x7:

 

Well it's unchanged, by the looks of it. Can't say that I understand the meaning of the flag, but the plugin does nothing here.

 

 

Edit: What the plugin does is that it adds a few perks and actor effects. I don't see something similar to the points that were previously discussed.

Edited April 7, 2024 by AlphaAndOmega

[traison]

19 minutes ago, AlphaAndOmega said:

the player character doesn't even exist yet (from my understanding, at least).

 

The player character is defined in SkyrimSE.exe of all places. Even if you deleted Skyrim.esm you'd still have a player. Also, you can coc out of the main menu and run around as a generic male nord without ever actually clicking the New Game button. The player exists far sooner than one might imagine.

 

... you don't have anything altering the SkryimSE.exe do you? 🤔

[AlphaAndOmega]

8 minutes ago, traison said:

... you don't have anything altering the SkryimSE.exe do you? 🤔

I don't thinks so. If there was something like that then things would be off even without the plugin loaded, but it behaves properly. And it's done within the isolated structure of MO2. the exe should be safe.

Edited April 7, 2024 by AlphaAndOmega

[traison]

If we assume it was finished with the IsDestructibe/SetDestructible thing and forget about that, and instead look at what would have come after the crash (which occured after SetDestructible). Note that this is mostly speculation, since I'm working with a memory snapshot:

 

Shortly after the crash it would have:

call <skyrimse.PlayerCharacter::sub_1405FB980>

 

...which leads to:

call <skyrimse.ActorProcess::GetCharacterController_14067BDF0>

 

...and eventually there's a check for what I would assume a dead or misconfigured player, which would result in the following error message being dumped somewhere:

"ERROR: Health value is 0 on the Player. Fix '%s' with the editor."

 

I.e. this is very early setup of the player character.

 

Edit: Clarification: Whether or not the error message would have been printed is not clear, and that's not the point here.

Edited April 7, 2024 by traison

[traison]

This may be early enough for the debugger traps to not be active yet. It might be possible to get a debugger in here without tripping them, even if you're using MO2 which complicates things.

 

Is a remote desktop session out of the question for you? Hit me up with a PM if this sounds reasonable.

[AlphaAndOmega]

37 minutes ago, traison said:

 

 

I.e. this is very early setup of the player character.

 

In order to check it in laboratory conditions I've disabled the whole load order apart from the main mod and its requirements (Skyrim, dlcs and USLEEP) and the plugin. I've regenerated the plugin and the game has launched normally. It puts the problem in a curious point where it's not the patcher that's inherently broken, but it's the fact that when it makes a patch for an unidentified mod (that does not cause a problem by itself) it breaks the startup. This moves us from one weird plugin to 132 more normal ones where one (hopefully only one) is responsible for the mess. Guess it's searching by halves from here.

Edited April 7, 2024 by AlphaAndOmega

[AlphaAndOmega]

After spending a "go fuck yourself" amount of time sorting through the entire load order, the culprits ended up being Honed Metal and Being a Cow 2.0.9. They both caused the same error with the patch applied. Version 2.0.8 works fine though.

 

That's rather anticlimactic, but it is what it is. The reason why the crash log was that weird remains unknown.

Edited April 7, 2024 by AlphaAndOmega