Skyrim SE Stuttering.

[phillout]

2 hours ago, GenioMaestro said:

You are another of that users that think that, in each Stack Dump, the game automatically terminate all the scripts exposed in the Stack Dump making the game totally unstable and that derive in a broken save game. Please, read my blog and my post, download my tools and learn how works the game.

 

That not have any relation. Any of my test mods can force the game to write one Stack Dump per minute in the papyrus log and my game, Skyrim Legenday Edition, no have any problem and no make CTD.

 

 

Dude, the Papyrus VM doesn't have an unlimited pool of resources, it can execute only a limited amount of scripts per every frame, and this is defined by the INI settings, something you could read in the official documentation. It can't process absolutely everything in a limited amount of time because CPU performance is, doh, limited as well. And if you throw too much at Papyrus, it simply can't process it. It has to terminate the scripts. And yes, it restarts them - over and over again. And then it has to start other scripts, triggered by new events. And those get terminated again to avoid a stack overflow. And will be restarted again, with older scripts that have been terminated and restarted before. That's how save files with 1000s of running script instances get born.

 

All of it due to the simple fact that you apparently don't understand - computer resources are limited, and Papyrus is slow. So after you throw at it more than it could chew, shit hits the fan.  It can take short bursts of scripts running at the start of the game, for example - but it can't take a sustained script overload.

 

And yes, I took a look at your blog, you measure Papyrus performance on simple script and claim that it can withstand 1000s of script running w/o a problem. The problem is - not all scripts are as simple as your tests. It's not only a number of scripts that counts, you know.

[GenioMaestro]

7 hours ago, phillout said:

Dude, the Papyrus VM doesn't have an unlimited pool of resources, it can execute only a limited amount of scripts per every frame, and this is defined by the INI settings, something you could read in the official documentation. It can't process absolutely everything in a limited amount of time because CPU performance is, doh, limited as well. And if you throw too much at Papyrus, it simply can't process it. It has to terminate the scripts. And yes, it restarts them - over and over again. And then it has to start other scripts, triggered by new events. And those get terminated again to avoid a stack overflow. And will be restarted again, with older scripts that have been terminated and restarted before. That's how save files with 1000s of running script instances get born.

The only limit of the Script Engine is the maximun available memory of the game, 3.1 GB in LE, your total phisical RAM in SE. The Script Engine not have any other limit and, really, can cumulate thousands and thousands of scripts.

The problems with the scripts that we have for years are always caused by the memory management of the game and the pattetic memory blocks. Every time the memory blocks are totally consumed the game make CTD. If the game can not alocate a block of the necesary size make CTD. Every script need memory and, as the memory management as bad, every time a script request a small bit of memory, if you your game is not correctly configured, the game make CTD.

 

Of course, the Papyrus Engine is executed for a maximun time specified in the INI files. But that is maximun time. That not mean that the Papyrus Engine is executing for that entire time. Only mean that specific scripts can be executed for that entire time. Read the web page of the CK carreful:

However most of the time the VM won't take this entire time slice and increasing the value will have no effect.

 

Under normal circunstances, none script is executed for the entire time of 1.2 ms and all script are suspended before the Papyrus Engine consume all the asigned time. You can verify it whit my test tools.

Open your Skyrim.ini and put the parameters of the fUpdateBudgetMS and fExtraTaskletBudgetMS in the aberrant value of 800 ms. Yes, make that, not worry.

Start a New Game and look if you have any problem. Not?

Launch my ScriptTest and execute as many cell scan as you want. You have any problem. Not?

Launch my ScriptTest and execute some mathematical test. You have any problem. YES... the framerate down.

 

That mean that the Papyrus Engine can run 3200 cell scans, all the same time, in less than 1.2 ms. Because, really, the scripts are not executed constanly for the asigned time of 1.2 ms. The scripts, really, are executing only a few lines in some nano seconds and are automatically suspended until the next frame because the script is waiting for external data.

But when you run my mathematical test the script can not be suspended and my mathematical scripts can consume all the asigned time of 800 ms and that can have a severe impact in the framerate.

Make yours owns test and learn how really works the game that you play for years.

 

Aditionally, a script never can be terminated or eliminated or droped or forzible ended in any way.

The Papyrus Engine always execute all the scripts no matter how many script launch the mod.

No matter if the action is a good intentional request to execute 100 script, 1000 or 3200 scripts.

The Papyrus Engine can not know if that 3200 scripts from my mod are good or bad and execute all of them whitout lossing one sinlge script. In the same way, is a script go crazy and launch 3200 bad script the Papyrus Engine go to execute all of them, one after another, no matter how many minutes can need for execute it.

 

The Stack Dump is a simple alert from the Papyrus Engine that say:

"Suspended stack count is over our warning threshold, dumping stacks:"

That mean that the Papyrus Engine has get a lot of new work and show that alert=warning in the log for alert the developer about a posible problem. That not mean that the script are "terminated or eliminated or droped or forzible ended" because the Stack Dump is a simple WARNING whitout any consecuence.

 

7 hours ago, phillout said:

It can take short bursts of scripts running at the start of the game, for example - but it can't take a sustained script overload.

 

And yes, I took a look at your blog, you measure Papyrus performance on simple script and claim that it can withstand 1000s of script running w/o a problem. The problem is - not all scripts are as simple as your tests. It's not only a number of scripts that counts, you know.

If you think that my test run small scripts that can not give any problem to the game take a look to the source code. 

	while runTest == 1
		if Test_active == 1
			cell Player_cell = PlayerREF.getParentCell()
[...]
		endif		
		if Test_active == 2
			;CPU code 
			Form[] buffer = new Form[128]
[...]
		endif
		
		if Test_active == 3
			while(runTest == 1)
				;;;;;;; this REALLY slowdown the fps
				Game.GetPlayer().QueueNiNodeUpdate()
				;;;;;;; this REALLY slowdown the fps
			endwhile
		endif
	endWhile

You see the line     while runTest == 1 ???

That mean that every one of my test run a constant while -> loop clycle that is eternal, whitout end, until you stopt the test.

If have 3200 scripts running constantly whitout stop and whitout end not overcharge the Script Engine, how a normal mod can overcharge it?

Maybe you think that a normal mod can run 3200 scripts in a constant and eternal while->loop whitout end?

Maybe the diference is in WHAT make the script and you can run the mathematical test for verify it.

My game support up to 200 mathematical test before I notice the test is running. 

Run 200 simultaneous mathematical script, all at the same time, whitout end and whitout stop, not give any problem to the game.

Maybe you think that a normal mod can run 200 special scripts that can not be suspended in a constant and permanent way?

Of course, NOT, none normal mod can make that and none normal mod can give problems to the script engine.

Only a bad mod with a bad script that go crazy can cause problems.

Then, you can execute all the mods with scripts, all at the same time, whitout any problem.

[phillout]

On 9/26/2019 at 5:20 AM, GenioMaestro said:

The only limit of the Script Engine is the maximun available memory of the game, 3.1 GB in LE, your total phisical RAM in SE. The Script Engine not have any other limit and, really, can cumulate thousands and thousands of scripts.

The problems with the scripts that we have for years are always caused by the memory management of the game and the pattetic memory blocks.

Loading script isn't enough, you have to actually execute them. And it takes CPU time -those small fractions of time the game can dedicate to Papyrus between rendering frames.

 

The fact that the engine never gives up on executing scripts is the exact reason why the game may hoard 10000s of scripts in the save game - because the game simply can not execute them and they continue piling up.

 

> If have 3200 scripts running constantly whitout stop and whitout end not overcharge the Script Engine, how a normal mod can overcharge it?

 

You have 3200 scripts running for the whole time. But add this to OnHit() events, add a lot of NPCs fighting and see what happens next. Why? Because the script threads will continue piling up, and after the total cumulative time needed to process them exceeds the one the game engine can dedicate to script processing - kaboom. The game can not process all scripts anymore, and they just continue piling up, dumping stacks etc, doing nothing but trashing your game. Just add the same code to onHit handler, start spawning 10s of Imperials and Stormcloaks outside of Whiterun and see what happens next - yep, the same stuff we observe here. Oh, is your code suddenly "broken" now?

 

Post edited by gregathit

Reason: Removed insult to other forum member

[Tiress]

I will quote my old post here, it might be helpful:

Quote

Another interesting quote from SmkViper, it's from a FO4 FAQ though:

Quote

The only thing "overloading" the script system will ever do will be to slow down all
script processing. You could theoretically run your system out of memory at some point
but the game would be unplayable long before then.

This one is from an old Bethesda forum:

Quote

Q: Do you know if it's possible for Papyrus to get so overloaded that it just 'drops' scripts, failing to fire them entirely? I get sparse reports from certain people that various scripts in my mod just 100% never go off for them. The entire rest of the mod is fine (quest stages, events, NPCs, etc.) but that one script doesn't fire and it breaks everything. I have them look into their papyrus logs and they're usually pretty messy.
 
Do you think they have a lot of other mods with update blocks or something? (Or any suggestions I could give them?)

 

A: No, it will not "drop" function call requests or running threads. If it is overloaded things will take a while to process, but it should never lose anything. (Ruling out any bugs I'm unaware of, of course)

What CAN happen, however, is a script could get locked down for a long length of time, preventing any function calls to said script from running because they cannot obtain the lock. For example, if a quest script went into a loop with no latent function and no outside calls it would prevent anyone from calling SetStage on that quest until it exited the loop (assuming the loop isn't infinite).

If you're getting reports like that, I suggest asking your users to type in "DPS" in the console to dump everything running to the log, and ask for that log. It should tell you everything that is running and it should be clear if something is stuck (though admittedly it might be hard to know for sure if the thing that is stuck is a script you don't have the source to)

 

[phillout]

10 hours ago, Tiress said:

I will quote my old post here, it might be helpful:

 

> No, it will not "drop" function call requests or running threads. If it is overloaded things will take a while to process, but it should never lose anything.

 

And this is the problem. A system under stress (being at the point it can't process everything thrown at it) should be able to discard some tasks to be able to handle the load properly, otherwise you've got tasks piling up, that's how DDoS work, for example.

 

In a well-designed software system there is always a way to gracefully handle performance degradation. Speaking of Papyrus, the system could start discarding scripts the game can live without, like those Defeat event handlers, once it can't process all scripts anymore and dumps stacks, signalling that the stack overflow is imminent. This would need a flag on a script, like "discardable", like "if you can't execute me - that's fine".

 

 

edit: I'm sorry if I had lost my nerve in this discussion a bit, but I've got enough of it having to explain simple things to young programmers at work recently, and having to do the same here was a bit too much for me, an old coot.

[Myst42]

On 9/26/2019 at 7:18 AM, phillout said:

The fact that the engine never gives up on executing scripts is the exact reason why the game may hoard 10000s of scripts in the save game - because the game simply can not execute them and they continue piling up.

This part is wrong.

Can confirm from extensive tests and investigation over this last year, that the game does not "terminate" or "stop" scripts when accumulation happened.

It stores pending calls, and puts them in queue. I can't remember the CK link exactly, but it said exactly that. All scripts get executed no matter what. They dont get "discarded" just because they failed to get executed in time. It's not the first time someone makes this wrong assumption for some reason.

 

Now, is the queued script storing capacity limited? Who knows. Maybe. But one thing is sure, and that is that Genio's tests were designed to make the game explode on script accumulation, and they go far beyond any normal mod to do that. Even the "heaviest of scripted" mods cannot generate such levels of accumulation at the same ratio. With a "heavy" mod, we can have script lag that goes away in a couple of minutes while with the test, it can take more than 20 to get rid of it. With a bad mod... well that's another concept. The important part is that no matter how aberrant the accumulation gets, all scripts get run, and that's precisely the problem. Excess of calls does not get discarded, it get queued and generates lag.

The real problem with script accumulation, and bad mods, is when it becomes progressive and does not decrease. It generates script lag and save bloating, but for that, you need an actual problem in a mod that actually spams events and doesnt stop doing so, ever.

On 9/26/2019 at 7:18 AM, phillout said:

But add this to OnHit() events, add a lot of NPCs fighting and see what happens next

Now, this is exactly how I've been testing these concepts. both OnHit and OnMagicEffectApply, as well as OnItemAdded sometimes, are VERY spamy events. Without a proper failsafe, these are the most likely candidates to generate script accumulation. OnMagicEffectApply being by far the most dangerous of them all. However, I've witnessed MASSIVE loads of stack dumps go away simply because a complicated battle ended. It may take a minute, it may take 10 minutes, but the script lag/accumulation crisis ends. And the log keeps executing every last call even if it wasn't done in time.

On a script crisis you can cast a simple spell such as AddItemMenu spell, and it will not work... until those 10 minutes of script lag pass. Look at the log and it's a 20MB abomination with millions of lines of stack dump, but in the end, the function ran.

This is not to say script lag is not a serious problem, because it is. But the game is a lot better equipped to handle it than people normally assume.

And again... The only real danger here is not on script accumulation per-se, but on the mods that create it.

It can be a badly coded mod, or a mod that is conflicting with another, or a mod that for some reason, is not working properly, but if a mod allows event spam, accumulation will happen, and sometimes, the installation does not allow said event spam to decrease over time, and when that happens, the game will most definitely explode. But this is not because " a stack dump got generated" it's because the game was unable to recover from the increasing accumulation, it's not because it suddenly stopped executing scripts, it's because it has too many scripts that keep getting executed and never stop coming

 

 

I have gone from knowing nothing to making some serious things in scripting in only a couple of years. I have seen my game get "corrupted" by save bloat many times over. I have read numerous guides on stability and finally have had the chance, and knowledge to test myself why the game generates lag, and now, I have "heavy scripted" mods working like a charm, logs that never go over 4MB in long game sessions and never over 2 in short ones. Every now and then I see a mini stack dump if someone uses concentration spells or something, but other than that, my game is the most stable thing regarding script performance. Oh yeah, and my fps rate is quite shitty indeed, so I should have twice the problems most users ever have simply because of that, yet somehow, I dont. Not since I decided to listen and learn for myself.

 

EDIT:

On 9/26/2019 at 9:05 AM, phillout said:

And this is the problem. A system under stress (being at the point it can't process everything thrown at it) should be able to discard some tasks to be able to handle the load properly, otherwise you've got tasks piling up, that's how DDoS work, for example.

Anyway, after seeing this, I wonder if I wrote all that long test for nothing, since this makes it seem you are properly aware that papyrus does not "drop" scripts with pending execution, or the ones that are currently being executed for the matter.

*shrugs

[Church the Cat]

I downloaded ReSaver and saved the game once the stutter started, and this is what I got. What should I do next? Thanks, for all the help thus far.

[Church the Cat]

 

I've also done some more trimming to my load order, trying to find the culprit.

[Church the Cat]

Here's my most recent save files, for whom they may concern.

Save20_7A18B674_0_486A6F726C692053756D6D65722D426C6F6F6D_WhiterunWorld_000116_20190927163554_12_1.skse Save20_7A18B674_0_486A6F726C692053756D6D65722D426C6F6F6D_WhiterunWorld_000116_20190927163554_12_1.ess

[phillout]

13 hours ago, Myst42 said:

This part is wrong.

Can confirm from extensive tests and investigation over this last year, that the game does not "terminate" or "stop" scripts when accumulation happened.

It stores pending calls, and puts them in queue. I can't remember the CK link exactly, but it said exactly that. All scripts get executed no matter what. They dont get "discarded" just because they failed to get executed in time. It's not the first time someone makes this wrong assumption for some reason.

 

It does, it's right there, in the official manual, no need to guess anymore.

 

Quote

WARNING: this setting is for stack size not heap size, so there is no reason for setting this to a huge value even though it is possible. Increasing iMaxAllocatedMemoryBytes to values much larger than default can cause stack thrashing (stack buffer overflows), intermittent game stuttering, erratic game behavior and CTDs. Stack thrashing will produce stack dumps in the Papyrus log, similar to the example below

Once you've got a stack overflow, you can't rely on the stack state anymore, you've got to throw away the results and start over again. This is what Papyrus does. And while from the state integrity's PoV this may be a good idea, all those CPU cycles already spent on Papyrus processing are now wasted.

 

And you can only have so much of them = FPS * fUpdateBudgetMS, meaning 72ms of every second in an optimal case when you have 60 fps. If your FPS drops, you've got even less CPU time every second dedicated to Papyrus VM running.

 

Quote

The real problem with script accumulation, and bad mods, is when it becomes progressive and does not decrease.

That's right, that's what I'm talking about, thank you for noticing it.

 

Short-time stresses can be handled by Papyrus, but if you've got lots of NPCs fighting in every cell, and all of them are handled by onHit() handlers, and all those handlers do process a lot of parameters (well, just take a look), you've got exactly that - a constant flow of scripted events Papyrus simply can't handle because of the sustained flow of said events.

 

Papyrus can handle a lot of events in short bursts (meaning it will eventually process them), but it simply lacks a capability to process a heavy constant load. Being a binary interpreter it's fucking slow already, not exactly Java VM, plus the fact that it runs only for the 1/14 of the second every second (at best, dropping to 1/30 of a second once you hit 30 fps low barrier) doesn't make it faster either.

 

Maybe if OP would go to some cave or qasmoke and let Papyrus run for an hour of real-life time, those scripts would finally have finished their execution and gone from the queue. That provided Papyrus is really 100% flawless, even if slow. And then, after some time do it again, and again, so the game engine had the time to process all the scripts. But in a typical real-world usage scenario it simply doesn't happen.

 

This discussion would be much more meaningful if someone would point out what exactly is wrong with Defeat, making it a "bad mod", how the "NPC vs NPC" code could have been improved. Until then I just reserve my opinion, calling it a day, I mean - the mod just tries doing too fucking much. Look at all those settings on NPC vs NPC page, random events etc. And then it starts a frickin' scripted scene for every rape event that lasts eternity and does fucking too much as well. This is literally a definition of clusterfuck, if anyone needed an illustration how it looks.

 

IOW, if you have mods adding a lot of NPCs in combat - just don't enable that option. This is simply more than the game could handle. Someone prove me wrong please and make a patch for Defeat, go ahead.

[Tiress]

Hmm, 1 active script an no suspended stacks. Something is telling me it won't be script related..

[phillout]

35 minutes ago, Church the Cat said:

 

I downloaded ReSaver and saved the game once the stutter started, and this is what I got. What should I do next? Thanks, for all the help thus far.

Just disable NPC vs NPC, type in "coc qasmoke" in console, and then let the computer run for a half an hour - just leave it alone. Then save, exit and check the # of script instances running, if it goes back to something manageable.

 

And either never enable that option again, or do not install mods that add a lot of NPCs.

[Church the Cat]

12 minutes ago, phillout said:

Just disable NPC vs NPC, type in "coc qasmoke" in console, and then let the computer run for a half an hour - just leave it alone. Then save, exit and check the # of script instances running, if it goes back to something manageable.

 

And either never enable that option again, or do not install mods that add a lot of NPCs.

I've already been playing with NPC vs. NPC disabled for a while now, but I'll try the half-hour in qasmoke method. Thanks, for the reply.

[Church the Cat]

Half-hour, standing in qasmoke. Looks normal.

No noticeable change here, that I can see.

[phillout]

59 minutes ago, Church the Cat said:

Half-hour, standing in qasmoke. Looks normal.

 

No noticeable change here, that I can see.

 

Your memory footprint is just 2GB now, that's quite a notable change from what you had before, 60GB.

 

Does your game still stutter? If it still does, @Tiress is right, I guess - scripts aren't to blame anymore. You've got 1 active script running, the number of instances (the total one) is quite normal for a modded game (with Ordinator and AFT). From what I see, your game should be fine from now on.

 

If it's still behaves badly - post your ini files.

[Church the Cat]

It still stutters, it happens about half the time during gameplay. Sometimes it plays fine for hours, sometimes it starts stuttering. The most recent one from earlier was a new character that I started in Whiterun. Haven't even ventured out of town, yet. I'll share my ini files, below.

 

SkyrimPrefs.ini Skyrim.ini

[phillout]

Your INI files are okay too.

 

Does it stutter when you play a game w/o a single mod installed? Just start a game w/o any single mod, without SKSE, type in "coc WhiterunPlainsDistrict01" while you're in the main menu (this will start a new game with a default Skyrim character, male Nord). Will it still stutter?

 

1) If yes, then you've got a problem outside of the game. It could be anything, like a faulty hardware. Or one of your master files got corrupted.

2) If not, it's some mod to blame. Start disabling them one by one - I mean mods, not ESPs, because in theory it could be anything again - textures, meshes, SKSE plugins.

[Church the Cat]

1 hour ago, phillout said:

Your INI files are okay too.

 

Does it stutter when you play a game w/o a single mod installed? Just start a game w/o any single mod, without SKSE, type in "coc WhiterunPlainsDistrict01" while you're in the main menu (this will start a new game with a default Skyrim character, male Nord). Will it still stutter?

 

1) If yes, then you've got a problem outside of the game. It could be anything, like a faulty hardware. Or one of your master files got corrupted.

2) If not, it's some mod to blame. Start disabling them one by one - I mean mods, not ESPs, because in theory it could be anything again - textures, meshes, SKSE plugins.

Tried this, but it crashes immediately after arriving in Whiterun. No active mods, deleted all texture replacers, using normal Skyrim SE launcher, instead of SKSE.

[phillout]

1 hour ago, Church the Cat said:

Tried this, but it crashes immediately after arriving in Whiterun. No active mods, deleted all texture replacers, using normal Skyrim SE launcher, instead of SKSE.

Well, read my post again then, #1. Verify local files in Steam and try again. If it still crashes - something is wrong with your hardware or maybe drivers. Vanilla, non-modded, game crashing in Whiterun immediately isn't something common. A typical issue could be CPU/GPU overheating. Open the case while PC is on and check if all fans are spinning, remove the dust if you've got a lot of it (vacuum cleaner is okay, but I clean my PC at least once per year using an industrial compressor, the one used in car workshops). IIRC give your PC a full hardware sweep. You won't believe how much PITA a DIMM can cause if just slightly misplaced, and this is really hard to pinpoint.

[Church the Cat]

Started running the game in compatibility mode this past week, seems to be running okay now.

 

Thanks, for all the advice guys. ?