Any Advice on multithreading in papyrus (Skyrim AE)

[dullman]

Hello I ended writing my script for that manage economy of my mod and gone to drink some tea and not because it was a success but another thing just to kill time until my script stops running (I tested for 20 customers which takes around 10 seconds to complete, whilst the target was something around 5 sec for 100 customers) so basically I began to search for answer about multithreading and came to ModEvents, since it seems can resolve some calculation time for me I mean I had 3 partial steps that should be run with nearly the same parameters for every customer but you know what I read about multithreading in skyrim (or generally) that I had a problem because I need the only one source of truth for threads (means data) I mean whilst I had maybe more complicated calculation on arrays but also had saved those result into memory by StorageUtil and thought about that how much faster is reading a FloatArr from StorageUtil than adding for example values of array by native function? If not by large margin I would not gain much performance until I can get any idea how to resolve this problem.

 

My first idea is create a same copy of memory for each thread but it comes with managing it when the value changes in one of threads, the only thing I can think is threadManager only setValue in the case when all threads stopped working, until that he just cached arriving data.

 

My second idea is based on that if the StorageUtil is fast enough I can create an external data Store for threads so that they can access it when it's possible.

 

So anyone can tell me your thoughts on this topic or maybe other architecture for multithreaded mod based heavily on memory and calculations?

[traison]

54 minutes ago, dullman said:

how much faster is reading a FloatArr from StorageUtil than adding for example values of array by native function?

 

This is a pointles question when you're dealing with a language as slow as Papyrus. Both of those most likely result in a c/c++ implementation of some kind, but when the language calling those native functions is a billion times slower (probably not an exaggeration) it's irrelevant which one is a few cpu cycles faster than the other.

 

Anyhow, if I deciphered your wall of text devoid of any punctuation correctly, what you're asking for is a way to make data thread-safe? While Papyrus does do some form of threading, its sort of lacking all the typical control methods usually associated with that, like mutexes and other wait handles. The only thread-safe way to emulate a wait handle in Papyrus is with script states (not quest stages) as far as I know. However I wouldn't implement this on a per-variable basis - there's easier ways to brick the Papyrus engine than this.

 

I think there may be a better way to approach your problem, but you'd have to better explain what you're trying to do. Like for instance, how are you implementing threads? What is this situation where data storage and/or retrieval is a problem?

[dullman]

13 minutes ago, traison said:

 

This is a pointles question when you're dealing with a language as slow as Papyrus. Both of those most likely result in a c/c++ implementation of some kind, but when the language calling those native functions is a billion times slower (probably not an exaggeration) it's irrelevant which one is a few cpu cycles faster than the other.

 

Anyhow, if I deciphered your wall of text devoid of any punctuation correctly, what you're asking for is a way to make data thread-safe? While Papyrus does do some form of threading, its sort of lacking all the typical control methods usually associated with that, like mutexes and other wait handles. The only thread-safe way to emulate a wait handle in Papyrus is with script states (not quest stages) as far as I know. However I wouldn't implement this on a per-variable basis - there's easier ways to brick the Papyrus engine than this.

 

I think there may be a better way to approach your problem, but you'd have to better explain what you're trying to do. Like for instance, how are you implementing threads? What is this situation where data storage and/or retrieval is a problem?

 

Sorry for my text being a wall.

 

But to answer I create an algorithm for calculation supply/demand on market

 

Quote

; Describe Algorithm

; 1. Calculate a supply for milk on market (means just get the values from storage since it would only change when actor is milked)

; 2. Calculate how much money each customer is willing to spend on milk

; 3. Calculate that how much demand is for milk for each customer (how much milk each actor wants)

; 4. Calculate planned shopping by each customer (it only planned it should be usually different from actually bought milk)

; 5. Calculate the market change in value for milk per race (If all wants nord milk then it will rise a price for this kind of milk whilst other will drop)

; 6. Calculate the actual shopping on market per actor and immediately update supply

; 7. Decide if there is over or under supply (To Be Done)

; 8. Calculate how much of gold is a cut of PC (depends on provided milk by her/him and her/his followers)

; 9. In future if under supply becomes norm there might be new independent cows, and for over supply player might be tasked to search for new rich customers or new markets

 

So basically I need to efficiently implement it, and currently I know that I can't do that in single thread (or maybe function).

 

As for resolve I thought about two ways one cut everything into independent threads which will share Storage as I tested self for parent and children will be shared for StorageUtil at least.

 

The second thing I could do, is just to do a calculation for single actor during different timeframes (means one actor every half a game day, another actor every 2/3 game day etc), so that we do calculation slowly and only make a final calculation when PC withdraw money (or check the bill  ). As the result we get much less time consuming single operation.

[traison]

1. Consider faking the supply value. You don't want to calculate some value from each actor in a hold/area. You don't even want to attempt to count them. Instead, if you want to bind your data to some "reality" do some background research on the population in each hold and calcualte a per day production multiplier. Falkreath maybe produces less as everyone there seems to be either dead or depressed. Solitude might be more productive, being the shopping center and capital of Skyrim.
2. Consider implementing market demand either per hold (9 variables per product in a quest), or as a global value (a single variable per product in a quest). Doing it per actor is not a good idea. Just like the latest Call of Duty title won't render the world behind the nearest building, you shouldn't be considering what Ysolda is doing while the player is in Markarth. You can simulate it, but never keep a reference to the actor.
3. Consider implementing demand per customer like SLA does arousal: A cloak and faction membership where faction rank = demand/willingness to buy. Again, don't consider everyone, just the ones in the immediate surroundings. Don't depend on the faction membership to stick - the game may get rid of it, and your code should accomodate that.
4. Same as demand (3), no?
5. Simple, demand divided by supply. If demand is 10, and supply is 8, the price is higher: 1.25x
6. Simulate it. Add random chance and simulated events to cause fluctuations in the market. This is not the real world, you can scam the system as much as you want.
7. Same as 5.

To put it short, you need a Quest, a few factions and an OnUpdate loop.

[JimUpdating]

agreed on the 9-10 variables.

Unless the character is on some rampage (combat or pregnancy) then the population for a hold shouldn't change significantly. The only loss of immersion would be if the quests with plagues (death consumes all) has been activated and some holds have been reduced. https://www.nexusmods.com/skyrimspecialedition/mods/41910

 

Consider Slaverun / Paradise Halls / SL Fame / Gossip. They all work on a value per hold.

At most, see if there is an event call for a crime (murder) in a hold, but then there is the choice of whether to go for reputation (the PC is both the criminal and the supplier) or for population (-1%, which probably won't be visible).

 

A reason to go to an individual would be if a particular NPC needs a particular type of product, to improve an acceptance for trade in a city or just for additional NPC involvement. In which case I'd be looking at a way of setting up radiant quests and having either a letter/courier delivery, or some suitably modified Missives board. 

[Fotogen]

Quest is a sort of a thread. If you want 5 threads, you create 5 quests.

Papyrus is a "glue" language. You don't do any kind of heavy lifting in Papyrus.

Skyrim is a realtime ARPG. So lots of action (you kill monsters and stuff), it runs in realtime(I hate lag) and it has some role play mechanics. Its not designed for "economy simulation".

Plus, anything that is not in the same cell as Player, doesn't even exists. Its not loaded or it was unloaded. If you try to "use" it, you can get a CTD.

 

I suggest you don't do heavy "math". Fake it. Call a couple of Utility.RandomFloat(), and thats it. But only "a couple". So, 1, 2, maybe 3. 4 or 5 may be too much for Skyrim.

[traison]

7 minutes ago, Fotogen said:

Call a couple of Utility.RandomFloat(), and thats it. But only "a couple". So, 1, 2, maybe 3. 4 or 5 may be too much for Skyrim.

 

To clarify, this is because by default the random functions are latent, meaning before they return they have to wait for the next frame. There's a mod that detaches them from the render loop as it were, but I can't remember the name of that mod right now. Quite certain I have it installed myself.

[dullman]

28 minutes ago, Fotogen said:

Quest is a sort of a thread. If you want 5 threads, you create 5 quests.

Papyrus is a "glue" language. You don't do any kind of heavy lifting in Papyrus.

Skyrim is a realtime ARPG. So lots of action (you kill monsters and stuff), it runs in realtime(I hate lag) and it has some role play mechanics. Its not designed for "economy simulation".

Plus, anything that is not in the same cell as Player, doesn't even exists. Its not loaded or it was unloaded. If you try to "use" it, you can get a CTD.

 

I suggest you don't do heavy "math". Fake it. Call a couple of Utility.RandomFloat(), and thats it. But only "a couple". So, 1, 2, maybe 3. 4 or 5 may be too much for Skyrim.

 

I ask you to be sure about bolded text, If I call from actor who is not present can I get a crash? Even if I don't touch anything besides displayName() (or actorBase) and gold can get a CTD, I thought at least reference to this actor should exists somewhere.

 

 

As for your suggestions I would my more complicated economy came to my mind after thinking how to bound it to more quest/dialogue oriented approach. 

What I mean is to get a new customer it should be done only by quest (radiant one). The same if we want change customer preference to something more convenience for us or maybe just increase amount of bought milk by him. 

Basically the only way, besides debugging, the changes should be done by radiant quest, also the demand will decay may decay, so we should check this regularly at our customers to be sure it's not happening (of course it also quest)

 

So generic population might be good way to do that but basically I feel like it will lose the purpose of generating new quests.

As for why economy is so complicated I also considered an option with single person market meaning all those calculation should be done when we negotiate with customer directly without any "global broker" (Of course simplified thing since basically we negotiate directly and with single entity) so all global demands should be abandoned directly in this case. 

And I think above approach is something that you should more find agreeable for you.

 

So in the end I wrote the economy engine for milk rather to get and see what I can do with papyrus, and saw nothing too much complicated.

As for basic functionality of producing milk is something that I would use to experiment with multithreading since I think it will be a worth lesson for me.

 

Also besides the the radiant quest there will be more "main" looking quest like increasing your level, getting permission for trade or just pay back your debt (magically generated by mod  ) by selling milk but those I consider when I add more plugins from me since basically I'm considering something like slaverun but not so much bdsm oriented.

 

ps. To be sure that I mention this all actor variables are being kept on form (script) bounded with quest, not on any actor instance, the only need that I want to have an actorRef is to get his/her displayName (and make it's unique) and keep it as unique key on form so it would be easy to clean it or reset data etc.

[Fotogen]

2 hours ago, dullman said:

I ask you to be sure about bolded text, If I call from actor who is not present can I get a crash? Even if I don't touch anything besides displayName() (or actorBase) and gold can get a CTD, I thought at least reference to this actor should exists somewhere.

Yes and no. Thats the problem. Some calls will work, some will crash. But I don't know which ones are "bad".

 

I do know, that "actor.Is3DLoaded()" is a good enough check for more or less all things. But don't quote me. Its Skyrim. You can never be 100% sure. 

 

So "loaded" NPC works, "not loaded" maybe.

 

By the way. I would not be surprised if displayName would work, but "get gold" not. NPC always has a name. But Im not sure if they have inventory (for gold).

Edited January 25 by Fotogen

[libertyordeath]

On 1/25/2024 at 1:47 PM, traison said:

 

...but when the language calling those native functions is a billion times slower (probably not an exaggeration)... it's irrelevant which one is a few cpu cycles faster than the other.

I mean, according to Papyrustweaks in vanilla skyrim it runs at 100 ops per "frame". Depending on what frame means (i'll assume a screenframe), if we multiply this by 60 (for 60FPS), then that's a processing speed of 6 kilohertz, NOT megahertz. Or about 300 times slower than a Commodore 64 from the 1980's. The only saving grace (that i know of) is that it's supposedly multithreaded, meaning it can execute multiple papyrus scripts at the same time.

 

And that's just raw ops/sec, completely disregarding efficiency. For example, back in the Oblivious days i discovered that Papyrus parses every line in an IF-THEN-ELSE block, regardless of if the condition is true or false. The only way to skip unneccessary code was the return statement. Back then, you could speed up some functions 20-40 times by simply checking if there's nothing to do at the top of the function, and if yes stop execution with a return statement. Not sure if the Skyrim version of Papyrus is still just as dumb.

Edited January 27 by libertyordeath