Mantella + Skyrim can be quite an experience

[aspirine2]

On 1/6/2025 at 8:54 PM, LordDragon said:

Hi! What I did was, that I used runpod.io to run Moistral. The OpenRouter will not show that model for some reason so you cannot use that.

 

In Runpod, I created a koboldcpp server with moistral as the model and used something like A6000 as GPU. Then when the solution is up an running, you can connect to that interface that the Runpod.io now hosts.

Steps (I assume you have some credits in runpod.io):
- Go to Pods --> Deploy

- Select specific GPU (cost and VRAM, like A40 with 48Gb of VRAM)
- Select Pod template "KoboldCpp - Official Template - Text and Image" and when this you have it selected, click "Edit Template".

- A popup appears named "Pod Template Overrides". Here, change the value of "KCPP_MODEL" to your preferred model (e.g. for Moistral v3 11B you could enter https://huggingface.co/TheDrummer/Moistral-11B-v3-GGUF/resolve/main/Moistral-11B-v3-Q4_K_M.gguf
- In the KCPP_ARGS you can increase the context a bit (Depending on the GPU you selected and the model that is used) like: --contextsize 16384
- Select you pricing, I use either On-Demand or Spot (Spot my disappear if someone pays more). and then in the bottom "Deploy On-demand"
- Next, the pod is being set up. Open it up and then press "Connect" and select "HTTP Service [Port 5001]". You can select it even it is not ready yet (the model you chose is downloaded, that takes some time)
- When the system is ready, the connection to HTTP Service should open up a Kobold interface. Here, copy the address of the link - it is in format SOMETHINGSOMETHING-5001.proxy.runpod.net.
- This address will be the endpoint which you will set in the Mantella. Note that you need to give it in the formal format, which means adding the port and the v1 for the API address. For example, if you address was zaq12wsxcde3-5001.proxy.runpod.net, your endpoint is https://zaq12wsxcde3-5001.proxy.runpod.net:443/v1
- That's about it in from the LLM perspective - set this endpoint in the Mantella config and you can, if you like, tune the parameters but that is not mandatory.

❗ REMEMBER TO DELETE THE POD WHEN YOU HAVE FINISHED GAMING, OTHERWISE IT WILL CONSUME YOUR CREDITS!

that's a very complicated method. i'm afraid to even try it. here is what i get when i edit the prompt. i even don't know how to connect it to my game.

 

Found variable '{Pamela}' in prompt which is not part of the allowed variables player_name, player_description, player_equipment, game, name, names, names_w_player, bio, bios, trust, equipment, location, weather, time, time_group, language, conversation_summary, conversation_summaries or actions