Hey folks! I'm pleased to announce that I have begun the execution of a brand new AI Platform, SkyrimNet. This is an entirely new platform that focuses on speed, reliability, performance, and ease of use.
Project Philosophy
- The project should follow modern software engineering best practices.
- AI content should never break vanilla Skyrim content. The two should seamlessly coexist where possible, and the AI content should degrade contextually where appropriate to avoid breaking the base game.
- The project should provide a stable experience that is free of crashes, freezes, and other problems. The project is built with a suite of tests that run against every build in order to reduce the chance of these issues making it out to users.
- The project should be as performant as possible, and avoid patterns where we have to do heavy in-papyrus script processing.
- Ease of Setup: The project should be essentially plug and play like any other Skyrim mod, aside from configuring the remote TTS and the LLM calls.
- As much as possible should be done inside the DLL for both ease of setup, speed, and performance. At present, every component except TTS and the LLM call are done inside the Skyrim process. This means that there is no requirement for an external server, or the WSL at all. This also means that there is no need to sync game state externally, and all data that we expose to the LLM can be real-time with no staleness. This also opens up leveraging spatial / positioning information, the state of executing animations, and so forth.
- The roleplay quality of NPC's is paramount. We should ensure that the prompts we're sending are high quality, configurable, and free of formatting mistakes / slop.
- The mod should try to provide a cohesive experience. CHIM's logic for determining which NPC responds is based on: Target in Crosshairs -> Closest Target -> Narrator. SkyrimNet instead utilizes a more context aware system for determining which actor (if any) is the most relevant to respond given the situation. Additionally, this same logic applies to the continuation of conversations (Like CHIM's rechat), and to the response to interesting events.
- The core of the mod will be completely SFW. Replicating MinAI content will be done in the future via extensions.
- The project should provide an easy to use API that allows for powerful AI integrations purely from Papyrus.
- The project starts from an "AI First" perspective, and then filter back to the specific targets. It is designed from the ground up with the assumption that every NPC is an agent, and does not have the concept of "adding or removing" an NPC from AI. Instead, the project limits which NPC are eligible to react in a given situation (Allowing for follower only gameplay, or however the user wants to use it).
- The mod should be compatible with as many things as is reasonably possible. CHIM/MinAI and SkyrimNet are SaveGame compatible. Both can be used together in the same save without issue.
- Narration should be a well supported (OPTIONAL) part of gameplay. I personally like it and thinks it adds a lot, but recognize that not everyone will agree. If using narration (Configurable), it will be a well supported part of the game. See the videos for what this currently looks like.
- The project should have native support for any number of disembodied voices (Player thoughts, narrator, etc) with proper context isolation between them. Player thoughts will be a natively supported feature rather than being hacked in.
- Text used in prompts from any source should never be hardcoded. This means that everything sent to the LLM is configurable via the Inja templating, and that we can provide translated templates in the future for supporting different languages.
- As much as possible, the system should be designed in such a way that the core functionality is divorced from the specific game this is integrated with. This will allow the project to be ported to other games in the future.
- The prompts used should not require JSON formatted responses where possible in order to maintain compatibility with simpler LLM's, and provide superior streaming capabilities.
Current Status: In Development.
At present, I have a number of major systems built out, and what I do have demonstrates the feasibility of the approach. At present, integration is functional for:
- Prompt Templating: The project utilizes [Inja](https://github.com/pantor/inja) templates for prompt generation. These are stored on-disk, and allow for the customization of literally every interaction with the LLM. The project detects changes to these and hot-reloads them without requiring a restart of Skyrim.
- Config Management: The project utilizes Yaml based configuration files for every option we expose to users. These likewise hot-reload when changed.
- LLM: Currently supports OpenRouter, and anything compatible with the same api. The project allows for the configuration of different llm's on a per-situation basis. Use a smart model for general dialogue, a fast model for combat, an uncensored model for nsfw, and so forth. There are several "meta" requests that SkyrimNet currently utilizes for performing tasks such as mood evaluation, scene evaluation (Who advances the scene the most by talking next?), and so forth.
- Speech To Text: Using Whisper out of the Skyrim process directly, with no external call.
- Text To Speech: Currently supports XTTS and Zonos. Zonos supports automatic voice cloning currently, xtts does not. Supports splicing between different audio types within the same response, and utilizes a more efficient and performant means of utilizing TTS that results in far lower response times. You can expect Zonos to be faster than XTTS was, and XTTS to be faster than even Melo. Player input correctly interrupts ongoing conversations smoothly.
- Persistence: The project is wired up to a local SQLite database, with in-memory caching in front of it. The database is synced to on save, and loaded from on load. I don't currently have support for rewinding / soft deleting events on reload, coming soon. Given that we are operating out of the same process as Skyrim, we don't need to store anything that represents game state (gender, race, etc). As a result of this + the in memory cache, I expect this database to be performant enough to not have any impact on the player's game, even over long playthroughs. If this fails to scale, I can migrate to an external DB in the future as needed.
- Context: At present, all dialogue (ai or vanilla) are detected, and persisted to the local database. Context isolation is stronger in general. There should be much fewer context leaks between situations and conversations.
There are still major components remaining to be built:
- Profile Management
- Papyrus Bindings: This project will allow Papyrus scripts to register decorators for resolving in-game variables for into things that can be used in prompts (For resolving things like SunHelm needs, or adult integrations). When these variables are needed for prompt generation, SkyrimNet will call the registered Papyrus function in order to retrieve the data on-demand, instead of in-game scripts constantly polling and pushing it. This will result in far superior performance with much lower script utilization.
- Some form of UI for writing and customizing Inja Templates, and managing Settings is needed.
- Spatial Audio Playback
- Skyrim Game Plugin (Everything in the videos is purely out of the DLL with no skyrim plugin - we will require one to finish the implementation for things like text input)
- The Game Master system (Central system that decides things like the current sentiment of actors, the best actor to respond to progress the scene, etc) needs to be fleshed out and expanded further. This concept has a lot of potential to improve gameplay.
- I need to write out proper high quality prompts.
- And much more.
Next Steps:
- I will continue to maintain MinAI, though will stop active feature development on it moving forward in order to focus on the new project.
- I will continue to utilize both CHIM and MinAI personally in my own games, and will continue occasionally contributing bugfixes to CHIM, though will cease active development for the project.
- I have started my own Discord Server where I will be posting updates about the project, and providing alpha/beta access in the future. The link can be found here. I hope to see you there, and look forward to pursuing next gen AI integrations with you.
I am posting videos showcasing the development here:
