Random hard freezes

[Strager]

Game runs pretty smooth until it freezes entirely. Can't figure out any pattern, it seems random. Any help here?

Papyrus.0.log loadorder.txt

[traison]

Unfortunately the only way I know of to debug a freeze is to attach a debugger and figure out where its stuck. Process Explorer from Sysinternals (Microsoft) may also be helpful in revealing which thread it may be stuck on.

 

Typically what people end up doing in these situations is the good ol' "remove mods until the problem goes away" -tactic.

Edited October 8, 2023 by traison

[anjenthedog]

and fwiw, if you have to follow the method that most of typically use because we don't have traison's skills, remember to use the "last installed first uninstalled" method to reduce the likelihood of load-order-reshuffle issues that can result in even worse problems than a random crash.

 

thank you, btw, @traison, for sharing the link for that debugger. Much appreciated. I'm not as savvy as you, and I'm very rusty, but I'm going to give that a try soon.

 

one oddball question though, if you don't mind,  do I need both the snapshot AND the similarly named "PDB" file?

(yes, the answer might seem obvious to the knowledgeable, but it's not to me)

 

 

Edited October 8, 2023 by anjenthedog
details

[traison]

The pdb is a file that contains debug information for an application. It is extremely useful if you happen to get one for the application you're debugging, as it contains the debug symbols (function and class names among other things) basically making the assembly code readable to a human. I managed to get my hands on a SkyrimSE.pdb file a while back from this forum; someone dumped it in a thread. I don't know where it came from but it feels like contraband - something I'm not supposed to have. You probably don't need that specific pdb file from the x64dbg downloads, as you most likely won't be debugging/developing x64dbg itself.

 

As for figuring out freezes. Its not something I do often (luckily), but maybe I can share some ideas and tips:

1. Most applications these days are multithreaded.
   1. If the UI is frozen, the application may still be doing things in other threads. This is why you sometimes hear music and sounds from Skryim, even if the screen is frozen.
   2. A data processing thread of an application may be frozen, even if the UI isn't.
2. Thread pools add an additional layer of complexity on top of the existing multithreaded architecture.
   1. Code is split up into tasks and the thread pool manager distributes these out to whichever logical thread is currently available. You may have 30 tasks, taking turns on 3 actual threads. Whether Skyrim uses this or not, I do not know.
3. An application may freeze in many different ways.
   1. A loop that never completes. From the application's perspective it's alive and running, however it may appear frozen because it hasn't redrawn the screen in a while.
   2. A thread deadlock: 2 threads that are waiting for eachother. These should be the simplest of all kinds of freezes to detect.
   3. Suspended threads. Probably not something you will see in production applications.

In Process Explorer I'd look for:

1. The thread that's using the most cpu cycles. This could reveal a never ending loop.
2. A thread that isn't suspended, but isn't using any cpu cycles either. This could reveal a thread that's waiting on a mutex (or similar).
3. Context switches may reveal something, however I'm not sure if a freeze would cause an unusually high or low context switch count. Also you'd need a baseline to compare to which may not be available.

Additional challenges:

 

DRM and anti-tamper. By default, SkyrimSE.exe will crash either instantly or a few seconds after a debugger is attached to it, depending on where its at in the code right now. To get a snapshot, I use pssuspend.exe from Sysinternals (Microsoft) to suspend SkyrimSE.exe before attaching the debugger. This means you won't see the code progressing, you only get a snapshot of a single moment, a single state.

 

There is a way around the DRM obviously but I won't go into that here. I will say however that MO2 and its usvfs makes this more complex. If you intend to keep everything in usermode code you'd ideally either run with Vortex or no mod manager at all when debugging Skyrim. For ring 0 solutions you probably got more options, as this will sit below the usvfs, however at this point I'd recommend using a virtual machine instead of your actual setup. My understanding of the reason why Skyrim crashes when it detects a debugger is that the debugger trap points RIP to a hidden thread (created and hidden by one of Steam's dlls) and execution can not continue at a memory address that does not appear to exist - so down it goes.

 

After all that, what do you get? Say you identified the frozen thread, and you got a snapshot or even managed to get the debugger into live code, you now have assembly code in front of you - potentially millions of lines of it flying past your screen every second. It can be daunting, or rather, if you ask me it very much is. Some of these debugging jobs I do sometimes take me a week or more to figure out. Even if its something simple like "oh this is opening a file". I typically do not spend nearly that amount of time on the crashing issues on this forum - maybe an hour max. But this is basically where the "real work" starts, because now you not only need to figure out what it's doing (why its stuck) and then how to not get it stuck again (how to fix it).

 

Edit: The situation gets even "better" when you're in the OP's shoes with random freezes. The first thing I do when facing a more severe technical problem, be it a freeze or a bluescreen, is to learn how to reproduce it on demand - learn to break it. If you have to wait a day to "try again" you will almost certainly get nowhere. This may not always be possible and I know how frustrating it is to work on a random problem - I've fought with such a problem for 100+ days before. It was a great learning experience and I can attribute most of the knowledge I have of internal workings of Windows and computers in general to working on that problem. But I can honestly also say that it took me about half a year to get over the mild PTSD I got from it. Every minute dreading the thought "what if I didn't fix it, what if it comes back?".

 

Edit again: Have I scared you enough already? Deleting your entire mod setup and starting over is looking like a great idea suddenly huh? ☠️

Edited October 8, 2023 by traison

[anjenthedog]

3 hours ago, traison said:

I managed to get my hands on a SkyrimSE.pdb file a while back from this forum; someone dumped it in a thread. I don't know where it came from but it feels like contraband - something I'm not supposed to have. You probably don't need that specific pdb file from the x64dbg downloads, as you most likely won't be debugging/developing x64dbg itself.

Lol on the first, and thanks on the second. I downloaded it but will simply keep it in archive for now.

 

Thanks also for the added info. Way more than I asked, so thanks again and once more for good measure 

 

ps> scared? nah... I'm too dumb to be scared  (I already am suffering from periodic freezes. I'm obstinate enough that I just mutter some vulgarities both at the problem and at myself for not having made another save every few minutes to reduce progress loss *when it inevitably happens, and reload)

 

Edited October 8, 2023 by anjenthedog