guide How to debug CTD without OSR Online Crash Dump Analysis

[Brisl]

Hello, dear people,

Thanks for your time to have a look at my guide for all those who already had a CTD in Skyrim and want to have the SKSE logs analyzed but can't analyze the *.dmp files on OSR http://www.osronline.com/page.cfm%5ename=Analyze.htm anymore because you can read the following message:

--------------------------------

OSR Instant Crash Online Analyze

 

Immediate !Analyze -v: OSR's Instant Online Crash Analysis

 

Effectively immediately, support for our Instant Online Crash Analyzer has been withdrawn.

As you know, the OSR Online website has been retired... our developer blogs have moved to our corporate web site, and the NTDEV, NTFSD, and WINDBG lists were migrated to our Community forum. Now, it's time for the Instant Online Crash Analyzer to fade go into the retirement it so richly deserves.

Does this leave you in a bind? Hate the fact that the Instant Online Crash Dump Analyzer has gone away? Feel free to write to us and complain! Write to CrashDumpAnalyzer@osr.com and tell us what you miss and why. Who knows... Dan might decide to bring this feature back!

 

------------------------------

For all these, I have here a great guide with which you can perform your analysis locally on your computer to check through your *.dmp files when Crash Fix and Papyrus again only execute adress and dubious numbers.

First of all you have to activate logging for SKSE.

This works by clicking on ..\Steam\SteamApps\common\Skyrim\Data\SKSE for all NMM users or worse ( Vortex ) or for the users of ModOrganizer2 who followed the guide of GamerPoet ( https://www.youtube.com/watch?v=MbxIu1XUxUE ) in the path ...\Mod Organizer 2\mods\SKSE - Scripts\ and add the folder skse to it and create or open the file skse.ini and enter this line in it:

[Debug]

WriteMinidumps=1

Now all you have to do is wait until your game crashes and you're already in the path: C:\Users\yourname\Documents\My Games\Skyrim\SKSE\Crashdumps have a *.dmp file.

As I just wrote, OSR has decided to discontinue their Instant Online Crash Analyze and retire. Well you could now look for an alternative and you will surely find something after hours of googling, but you can do it yourself anytime with this steps.

First download the Debugging Tools for Windows direct:

Best here: Debugging Tools for Windows Direct Download

I took the Debugging Tools for Windows (x86) version 6.12.2.633 and it worked fine (under Windows 10 too).

Once you have done this and installed it you should go to the path:

C:\Program Files (x86)\Debugging Tools for Windows (x86) in there you can find the tools.

Especially important is the windbg.exe

Now you have to do the following:

Stage 1: Making sure you have installed.NET Framework on your computer

 

Stage 2: Associating .dmp files with WinDBG

 

In order for you to be able to read and analyze the .dmp files your computer creates, you need to first associate .dmp files with WinDBG. In order to do so, you need to:

If you are using Windows 8 or later, right-click on the Start Menu to open the WinX Menu and click on Command Prompt (Admin). If you are using an older version of Windows, open the Start Menu, search for “cmd”, right-click on the search result named cmd and click on Run as administrator. This will launch an elevated Command Prompt.

Type the following into the elevated Command Prompt and press Enter:

cd c:\Program Files (x86)\Debugging Tools for Windows (x86)

Note: If the installation location for your instance of WinDBG is different, replace everything in front of cd in the command line above with the actual installation location of WinDBG in your case.

Next, type the following into the elevated Command Prompt and press Enter:

 

windbg.exe –IA

 

If all went well, a new WinDBG window containing a dialog box confirming the association of your computer’s .dmp files with WinDBG will appear. If such a confirmation box appears, you can go ahead and close both WinDBG and the elevated Command Prompt.

Stage 4: Configuring the Symbol Path for WinDBG

 

In order to read the binaries in a .dmp file, WinDBG uses symbols which it needs to have on hand whenever you require it to read and analyze a .dmp file. The symbol path is the directory on your computer where WinDBG stores all of its downloaded symbols. While you are free to turn any location on your computer’s hard drive into the symbol path for your installation of WinDBG, this is an extremely crucial and fragile stage, which is why it is recommended that you simply use the default location. Here’s how you can configure the symbol path for WinDBG:

Launch WinDBG.exe by opening the Start Menu and click the shortcut or start it from your install path c:\Program Files (x86)\Debugging Tools for Windows (x86).

 

When WinDBG launches, click on File > Symbol File Path.

Type the following into the Symbol Search Path box and click on OK:

SRV*C:\SymCache*http://msdl.microsoft.com/download/symbols

 

 

This will instruct WinDBG to create a new folder named SymCache in Local Disk C and download new symbols and save them to this folder. You can replace C:\SymCache in the text above with any directory of your choosing where you wish to have WinDBG store its symbols.

Click on File > Save WorkSpace. This will save the new Symbol Path that you have configured.

Close WinDBG by clicking on File > Exit.

 

NOW YOU ARE READY TO ANALYZE YOUR SKSE *.DMP FILES ON YOUR COMPUTER.

 

What you need to do:

Open the folder C:\Users\yourname\Documents\My Games\Skyrim\SKSE\Crashdumps and doubleclick your dumpfile.

 

WinDgb will open and you see a window with a text like:

Loading Dump File [C:\Users\Win10\Documents\My Games\Skyrim\SKSE\Crashdumps\2018-10-31_11.20.09.dmp]
User Mini Dump File: Only registers, stack and portions of memory are available

WARNING: Minidump contains unknown stream type 0x15
WARNING: Minidump contains unknown stream type 0x16

 

Now in the bottom of the field is a white long line beside a number that looks like: 0:046> ( as example, can be different and depends on the *dmp file).

At this line  you can enter a command and in this field simple write the command:

!analyze –v

 

Now your *dmp file analyze start at your Computer.

WinDgb will analze your *.dmp file and with much luck your can see maybe whats mod or issue caused your crash

… for more informations about the Analyze see the Guide here from h38fh2mf at https://www.loverslab.com/topic/46913-how-to-debug-ctd/

Sorry for my terrible english guys but maybe its help further for CTD Analyze. Of anyone can explain it better or have Questions about it, no problem, ask me, i am open for critics

 

See you and Greetz Brisl

 

 

 

 

 

 

 

 

 

 

 

[KoolHndLuke]

Good info! I think I tried to analyze my skse logs about once. After that I figured out easier ways to find conflicts or problems in my load order by using TESVEdit or just the old fashioned turn them off and on again in groups until you find the culprit(s). Takes time to do it this way, but it's the way I prefer.

[CharlieFoxxx]

Use !analyze -v to get detailed debugging information.

*** WARNING: Unable to verify checksum for hdtPhysicsExtensions.dll
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for hdtPhysicsExtensions.dll - 
*** WARNING: Unable to verify checksum for JContainers.dll
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for JContainers.dll - 
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for nvd3dum.dll - 
Failed calling InternetOpenUrl, GLE=12029
Probably caused by : memory_corruption ( memory_corruption!TESV )

Followup: MachineOwner

 

 

this is all thats given to me

[Thost]

Stuck at the windbg.exe -IA line. It won't recognize it as an "executable" so I can't go further.

[Grey Cloud]

33 minutes ago, Thost said:

Stuck at the windbg.exe -IA line. It won't recognize it as an "executable" so I can't go further.

I'll alert the media.

[Guest]

On 6/5/2019 at 9:00 PM, Thost said:

Stuck at the windbg.exe -IA line. It won't recognize it as an "executable" so I can't go further.

I had the same problem. I literally copy&pasted that line from OP and only then it would work for some reason.

[Mister X]

For Windows 10 I offer the official Windows Store Link for the WinDBG Preview:

https://www.microsoft.com/store/p/windbg/9pgjgd53tn86

 

That's enough to debug all your .dmp files and if .dmp isn't related to an executable, it automatically will be linked to WinDBG on installation.

[ShenGo]

On ‎7‎/‎7‎/‎2019 at 11:38 AM, Mister X said:

For Windows 10 I offer the official Windows Store Link for the WinDBG Preview:

https://www.microsoft.com/store/p/windbg/9pgjgd53tn86

 

That's enough to debug all your .dmp files and if .dmp isn't related to an executable, it automatically will be linked to WinDBG on installation.

I'm pretty lost. When I try to open a dmp with the application in the first post, I get "%1 is not a valid win32 application" (or something like that). The app you linked to WILL open the files. But I don't know how to make sense of it. D:

 

Edit: Sorry for the necro, but this is all I've found on the subject of analyzing crash dumps since OSR isn't available. I'm pretty desperate.

[Guest]

20 minutes ago, ShenGo said:

I'm pretty lost. When I try to open a dmp with the application in the first post, I get "%1 is not a valid win32 application" (or something like that). The app you linked to WILL open the files. But I don't know how to make sense of it. D:

 

Edit: Sorry for the necro, but this is all I've found on the subject of analyzing crash dumps since OSR isn't available. I'm pretty desperate.

Read the first part of this post.

[ShenGo]

3 hours ago, Hawk9969 said:

Read the first part of this post.

So basically, I'll never be able to find out what causes a crash. 

[Guest]

17 minutes ago, ShenGo said:

So basically, I'll never be able to find out what causes a crash. 

You can try attaching the dump and I'll take a look. Can't promise I'll have a solution, but if the code around your crash is clear enough to me within a quick analysis or the crashing address is known to me, I'll tell you what might be causing your crash.

Mind you, debugging optimized code without symbols requires some reverse engineering (which is the case for Skyrim), and this is not something simple (complexity depends on the level of optimization and what the code actually does), even for experienced low-level programmers. Stack and heap corruption issues are extremely hard and time consuming to reverse engineer back to its source.

[ShenGo]

2 hours ago, Hawk9969 said:

You can try attaching the dump and I'll take a look. Can't promise I'll have a solution, but if the code around your crash is clear enough to me within a quick analysis or the crashing address is known to me, I'll tell you what might be causing your crash.

Mind you, debugging optimized code without symbols requires some reverse engineering (which is the case for Skyrim), and this is not something simple (complexity depends on the level of optimization and what the code actually does), even for experienced low-level programmers. Stack and heap corruption issues are extremely hard and time consuming to reverse engineer back to its source.

I would IMMENSELY appreciate it. For the most part, I can play without issue, but occasionally (often repeatedly in certain settlements, and upon arriving in Solstheim), the sound will intermittently cut out for a second or two, then I'll CTD, and I'm worried I won't be able to complete some quests because of it. I'll attach a file in the morning, and thank you so much!

[ShenGo]

12 hours ago, Hawk9969 said:

You can try attaching the dump and I'll take a look. Can't promise I'll have a solution, but if the code around your crash is clear enough to me within a quick analysis or the crashing address is known to me, I'll tell you what might be causing your crash.

Mind you, debugging optimized code without symbols requires some reverse engineering (which is the case for Skyrim), and this is not something simple (complexity depends on the level of optimization and what the code actually does), even for experienced low-level programmers. Stack and heap corruption issues are extremely hard and time consuming to reverse engineer back to its source.

2020-06-22_15.33.09.dmp

[Guest]

3 hours ago, ShenGo said:

2020-06-22_15.33.09.dmp 104.97 kB · 0 downloads

Crash happens at member function TESV.exe+262ED0, which takes a pointer as its only argument.

The crash happens because in your case, the caller is calling it with a null pointer.

00662ED0 53                   push        ebx  // Save current content of ebx into the stack
00662ED1 55                   push        ebp  // Save current content of ebp into the stack
00662ED2 8B 6C 24 0C          mov         ebp,dword ptr [esp+0Ch]  // Store function argument into ebp
00662ED6 8B 45 34             mov         eax,dword ptr [ebp+34h]  // CRASH: dereference function argument, which in your case is a null pointer (EBP == 0)

Attached a debugger to my game to see what calls this function and it seems to be function TESV.exe+263410.

The interesting thing is that Crash Fixes has a hook as the first and last instruction for this function.

TESV.exe+263410 - E9 0F290077           - jmp CrashFixPlugin.dll+5D24
TESV.exe+26344E - E9 12290077           - jmp CrashFixPlugin.dll+5D65

These hooks are part of Crash Fixes' FixUnsafeEffectList().

; Info: Active effect list of actors is not thread safe but accessed and modified from multiple threads, this will add mutex to each actor's
;       effect list.
FixUnsafeEffectList=1

My question is, do you have Crash Fixes 12 installed and this fix enabled?

[ShenGo]

2 hours ago, Hawk9969 said:

Crash happens at member function TESV.exe+262ED0, which takes a pointer as its only argument.

The crash happens because in your case, the caller is calling it with a null pointer.

00662ED0 53                   push        ebx  // Save current content of ebx into the stack
00662ED1 55                   push        ebp  // Save current content of ebp into the stack
00662ED2 8B 6C 24 0C          mov         ebp,dword ptr [esp+0Ch]  // Store function argument into ebp
00662ED6 8B 45 34             mov         eax,dword ptr [ebp+34h]  // CRASH: dereference function argument, which in your case is a null pointer (EBP == 0)

Attached a debugger to my game to see what calls this function and it seems to be function TESV.exe+263410.

The interesting thing is that Crash Fixes has a hook as the first and last instruction for this function.

TESV.exe+263410 - E9 0F290077           - jmp CrashFixPlugin.dll+5D24
TESV.exe+26344E - E9 12290077           - jmp CrashFixPlugin.dll+5D65

These hooks are part of Crash Fixes' FixUnsafeEffectList().

; Info: Active effect list of actors is not thread safe but accessed and modified from multiple threads, this will add mutex to each actor's
;       effect list.
FixUnsafeEffectList=1

My question is, do you have Crash Fixes 12 installed and this fix enabled?

Yes, I do have Crash Fixes v12 installed and enabled. I thought it was odd that it wasn't giving me any kind of error messages for these types of crashes when it had usually given me messages so reliably before. Such as the 'foot pk' message when I had too many animations, the 'virtual function call' message, and things like that. It's not giving me any messages now.... Should this be set to 0 instead of 1, then?

[Guest]

1 hour ago, ShenGo said:

Yes, I do have Crash Fixes v12 installed and enabled. I thought it was odd that it wasn't giving me any kind of error messages for these types of crashes when it had usually given me messages so reliably before. Such as the 'foot pk' message when I had too many animations, the 'virtual function call' message, and things like that. It's not giving me any messages now.... Should this be set to 0 instead of 1, then?

No, it should be 1 (true) and there won't be a message if the crash was not expected (unhandled exception).

Your crash might be caused by having a bad effect on an actor, either because of a bad mod or a bad save.

Try the attached fix. I've made the function return false if the pointer argument is NULL.

 

You can follow the same instructions from here on how to enable it. 

 

Crash Fix Test.CT

[ShenGo]

17 minutes ago, Hawk9969 said:

No, it should be 1 (true) and there won't be a message if the crash was not expected (unhandled exception).

Your crash might be caused by having a bad effect on an actor, either because of a bad mod or a bad save.

Try the attached fix. I've made the function return false if the pointer argument is NULL.

 

You can follow the same instructions from here on how to enable it. 

 

Crash Fix Test.CT 822 B · 1 download

Is 'Cheat Engine' a Skyrim tool? I googled it, but found a lot of different types of results, so I'm not sure which is the appropriate application... Sorry for my ignorance!

[Guest]

9 minutes ago, ShenGo said:

Is 'Cheat Engine' a Skyrim tool? I googled it, but found a lot of different types of results, so I'm not sure which is the appropriate application... Sorry for my ignorance!

https://www.cheatengine.org

[ShenGo]

15 hours ago, Hawk9969 said:

https://www.cheatengine.org

Thank you! I haven't had a chance to try it yet, been pretty busy... Can you tell me, though, what you mean by a bad 'effect'? Could it be a magical effect? A visual effect? A script that's been applied to an actor? Bad HDT or weights on armor? Or is there just no way to know?

[neverend22]

Hello!Who can tell a person far from programming to a person what is my mistake?

2020-02-02_13.25.50.dmp

[Guest]

7 hours ago, ShenGo said:

Thank you! I haven't had a chance to try it yet, been pretty busy... Can you tell me, though, what you mean by a bad 'effect'? Could it be a magical effect? A visual effect? A script that's been applied to an actor? Bad HDT or weights on armor? Or is there just no way to know?

The crash happens at a function that is called by a function accessing/modifying active magic effects. I didn't debug it further to try to understand what the crashing function does.

 

5 hours ago, neverend22 said:

Hello!Who can tell a person far from programming to a person what is my mistake?

2020-02-02_13.25.50.dmp 89.51 kB · 0 downloads

00D6C819 0F BF 04 41          movsx       eax,word ptr [ecx+eax*2]  

Crash happens because it's dereferencing ECX and ECX is a null pointer in your case.

 

I can't get my game to execute the function where your crash is happening, so I'll need more context of what triggers your crash.

From a preliminary look, this function aligns the stack to 16 bytes so it can use the movaps instruction.

ECX is the dereference plus offset 0xC of the first argument.

mov ecx,[ebp+08]
mov ecx,[ecx+0C]

Looks like to be a data copying function that takes 4 arguments.

EBP being the frame pointer.

ebp+00 = original ebp

ebp+04 = function return address

ebp+08 = first argument

ebp+0C = second argument

ebp+10 = third argument

ebp+14 = fourth argument

[ShenGo]

On 6/23/2020 at 7:00 PM, Hawk9969 said:

The crash happens at a function that is called by a function accessing/modifying active magic effects. I didn't debug it further to try to understand what the crashing function does.

Thank you, that's at least a good clue.

 

 

Edit: Okay, I'm finally getting a chance to try this. So far, I still got a crash within about 15 seconds of entering Riften... I don't know, maybe Riften crashes for a different reason? It could take some time before I know if this works, since sometimes I can play for hours before crashing.

 

IF this does work, will I need to run Cheat Engine every time I load the game?

 

Edit 2: That was fast, lol. I was paused for a few seconds, and as soon as I unpaused, I got the sound cut, then CTD again. It had been playing for a few minutes before that. Shame there's no way to know exactly which mod is the cause. I can't think of any off the top of my head that apply magic effects to anyone but my PC, that have been active longer than a couple days.

[neverend22]

On 6/24/2020 at 3:00 AM, Hawk9969 said:

The crash happens at a function that is called by a function accessing/modifying active magic effects. I didn't debug it further to try to understand what the crashing function does.

 

00D6C819 0F BF 04 41          movsx       eax,word ptr [ecx+eax*2]  

Crash happens because it's dereferencing ECX and ECX is a null pointer in your case.

 

I can't get my game to execute the function where your crash is happening, so I'll need more context of what triggers your crash.

From a preliminary look, this function aligns the stack to 16 bytes so it can use the movaps instruction.

ECX is the dereference plus offset 0xC of the first argument.

mov ecx,[ebp+08]
mov ecx,[ecx+0C]

Looks like to be a data copying function that takes 4 arguments.

EBP being the frame pointer.

ebp+00 = original ebp

ebp+04 = function return address

ebp+08 = first argument

ebp+0C = second argument

ebp+10 = third argument

ebp+14 = fourth argument

Hello ^_^ When i loud mod SSME - Skyrim Startup Memory Editor this help me for no more CDT, but fullovers start be invisbl -_-

[ShenGo]

Ugh, so frustrating. I thought I might have accidentally changed my load order or something else that could be causing these issues, so I started a fresh new game for testing, and went straight to Riften. I hung out there for half an hour last night with no issues (in my main save, it crashes within the minute). Then today, I loaded that same, new game, and crashed twice in Riften within 15 minutes. I looked through my mod list to see what might be trying to apply bad magic effects, and disabled everything possible in the MCMs (everything that can't be disabled via MCM was added to my old game AFTER the crash issues were already prevalent), but the crashes don't stop... And the fix doesn't seem to... Well, fix it. ?

 

AND this new game is exhibiting issues the old game didn't have. Pressing 'K' to select an actor for SL scene adjustment instead terminates the SL scene. And a lot of NPCs suddenly have the SOS undies while dressed (in armor flagged as revealing), which was never the case in my old game.

 

And now I need feel bad for resurrecting this thread, bringing in a lot of others looking for help. I'd only hoped to be able to decipher the dumps myself. I'm so terribly sorry...

[Tester123643]

And what can you do when your game crashes without creating a dmp at all?

Every time I enter the cell around the tower before bleak falls barrow the game ctd's apparently without the game itself knowing anything about it. no .dmp, no clue inside the papyrus log or any other log...