Artificial intelligence in Skyrim state of the art (GPT, Voice, Stable diffusion, waifu, chatGPT)

[sarahpetersonsarahpeterson]

Current closed source stack:

head space:

3d vert deep variational  convolutional auto encoder >1b param model for latent modelling/sampling of the entire Nexus/loverslab waifu space:

>OPERATIONAL

>all possible waifus faces have been modelled, all waifus are belong to us. 

>*some fine facial contours are lost, no model for generating corresponding normal map (3d to 2d autoencoding in normal map space)

 

Garments:

functionally infinite outfit combinations with nested lists, hotkey mapped, trillions upon trillions of 256k textures in 8k containers for hundreds of trillions of billions of outfit permutations

>OPERATIONAL

>needs hand built meshes and morphs*

 

Voice models:

Tortoise voice modelling

>development stage

>needs gpt3/stable diffusion integration 

>needs someone (the guy who made xedit please) to make bindings or external/python etc for writing diaglog lines to xedit records.

>*need this first or alternate chat system reading from live database etc, then can integrate chatbot/chatgpt 

 

 

Tint/masks: 

stable diffusion with hand made visual prompting/guiding, automated batch 4k upscale for infinite tint masks.

>OPERATIONAL

 

Comment:

Given the progress in the field so far and recently, and skyrims modding progress:

The community should focus 100% on developing a system to programmatically write to skyrims diaglog format for npcs. Similar to all other records which can be scripted in xedit. 

 

 

Video archives of progress/works of art:

 

vertex model:

 

masks:

 

https://www.youtube.com/channel/UC5H4yOoUwSfzO0vsCGnI9TQ

 

https://www.instagram.com/sarahpetersonsarahpeterson/

 

narrative possibilites:

 

 

**all ai models are not released yet, in development/sandboxing

 

*this is open source tool used for the back end of voice model:

https://github.com/neonbjb/tortoise-tts

[Yinkle]

The 1st of April is still a few weeks away.

[sarahpetersonsarahpeterson]

On 2/17/2023 at 8:06 PM, Yinkle said:

The 1st of April is still a few weeks away.

The model outputs are on display in the youtube videos.

And recently here too:

https://www.instagram.com/p/CoxqVdUvJzJ/

 

Attached is one recent model iteration showing perturbations through the latent space of all nexus/loverslab waifus. 

 

You can see artifacts introduced by the compression dimension in the deep convolutional auto encoder for tight vertex groupings around the ears/lips. 

 

 

 

 

 

 

 

 

Edited February 20, 2023 by sarahpetersonsarahpeterson

[blodtanner]

I'm working on "real time" tts for this. This will require beefy hardware, so don't expect this to run on your 1050. You can train all of your own voices etc.

 

 

Worst quality takes about 5 seconds to generate.

Best quality takes about 3 minutes. 

[sarahpetersonsarahpeterson]

3 hours ago, blodtanner said:

I'm working on "real time" tts for this. This will require beefy hardware, so don't expect this to run on your 1050. You can train all of your own voices etc.

 

 

Worst quality takes about 5 seconds to generate.

Best quality takes about 3 minutes. 

This is great, but you haven't solved the main problem outlined in the OP:

 

There is no way except to manually click for several days to add large amounts of voice dialog to npcs. 

 

 

[blodtanner]

That's pretty easy to do with the tool I'm using / extending.  You can just have a file with all the sentences and which model to use, and let it run over night. Like;
<model>,<sentence>

 

Which voices are you using?

[sarahpetersonsarahpeterson]

19 hours ago, blodtanner said:

That's pretty easy to do with the tool I'm using / extending.  You can just have a file with all the sentences and which model to use, and let it run over night. Like;
<model>,<sentence>

 

Which voices are you using?

How are you adding new diaglog lines to a custom npc? It takes several hours for just a few lines with the CK. 

 

Obviously just overwriting existing responses/conversations and their audio is trivial and takes minutes.

 

You need to be able to programmatically write dialog trees/ responses etc into the esp for a particular npc and populate AND link the audio file to text line in the ESP too. 

 

Then we can GPT to fill the trees and tortoise to voice them.