OpenFaceSwap Tutorial

[Joaku]

Tutorial for creating DeepFakes with The Klub models.

 

OpenFaceSwap Download:

 

 

https://drive.google.com/file/d/15P5NURf5sGqM9NX5QkJcSUPnWL7_PdKW/view?usp=sharing

 

 

There are a lot of other software out there, but for the purposes of this tutorial, we'll just be using

this slightly old version. It's simpler and faster than a lot of other programs. Once you're familiar with

the basics, you can always try out some of the more powerful stuff ?

 

Alright, first things first: If you never tried this kind of software before, you might run into a problem with your CUDA settings/cores. It's basically obscure Lovecraftian nonsense, and I have zero idea how any of it works, but you should be able to google your way out of it fairly easily. If you DO get completely stuck on that one, I'll try to see if I can figure it out. 

 

Your interface should look like this: 

 

 

 

You'll notice the save/load icons in the top-left corner. These are for saving and loading configs. It used to be, that this was one of the only programs with that option, but now some of the other ones have created these kind of interfaces as well. I'm sticking with this for now, because it's quick and easy. You can always try the more complicated ones on your own. 

 

 

 

Most of these categories are pretty self explanatory, but notice the SECOND set of video/images/faces row? These are for if you wanna make videos on the fly. For our purposes, we're just gonna be using "FACES B".

 

Familiarize yourself with the difference between the blue, arrow-shaped buttons on the LEFT, and the folder/search/settings buttons on the RIGHT. The blue arrows are for starting the process, the folder buttons are for choosing a folder path, and the search button is for checking the content of the path. 

 

Setting up a folder structure is key to using this easily and quickly. With the right kind of setup, you will only have to use the save/load buttons at the top, to bring up all the paths and folders you need for a specific model. I will cover how I've set up mine in the next post. 

[Joaku]

Setup

 

(Link to program in first post)

 

[Whenever I talk about "alignments" or "aligned", I'm referring to the face images that has been ripped from a video.]

 

For a smooth experience, I recommend setting up your folders in a way that is both easy to navigate, and that you are comfortable using. I like separate folders for separate types of files. In my main OpenFaceSwap folder, I have three folders that look like this: 

 

 

 

 

My "0Configs" folder holds all of my saved setups. The "Align" folder has all of my extracted faces, separated into folders for each model. The "Model" folder holds all of my AI models. It also contains three folders called "0_Original". This is where you can REALLY speed things up, but I'll get to that later.

 

You will notice I have a "0" in front of some folders. This is to make sure they appear at the top of a load order, so I can always access them without having to scroll down ?.

 

With this setup, I can easily find and access the files I need. When loading a config file, it will look something like this: 

 

 

 

As you can see, I have both the video, images, aligned faces etc. ready. From here, all I need to do is press "MODEL", to start the training process. I also saved the folder for my outputs here, so I can simply press "SWAPS" to start the merging process. 

 

You'll also notice, that I already have a path for the "FACES B". This is because I've already ripped the faces I want to use, and stored them in the aforementioned "Align" folder. 

 

Again, I cannot stress this enough: create a folder structure that you can easily navigate, because you can quickly end up with A LOT of files!

 

In the next post, I will show a very simple example of how to rip faces from a video, to start your collection of reference photos. 

[Joaku]

Extracting images

 

(Link to program in first post)

 

Extracting faces is a simple process, but it can take a long time, and might require some clean-up. It's important that you DO NOT use SPACES or NUMBERS in any of your files. It can VERY quickly mess up the system, or entirely prevent it from running. Stick with letters and underscores. (We'll be using the numbers for another trick later).

 

In this example, I have a short, 8 second clip of Jane Douglas (my apologies in advance, miss Douglas). 

 

 

 

As you can see, I currently have my IMAGES folder, and my ALIGNED folder in my main folder. It's just for now - we'll delete these later. You can extract directly to whichever folders you please. This is just for clarity. 

 

Having selected my path to the video, and having an IMAGES folder ready, I simply push "IMAGES A", and the extraction process will begin. (In the Settings tap, you can select between JPEGs or PNGs. PNGs take longer, and I don't notice a ton of difference in quality.)

 

 

 

You should now have a folder that looks something like this, containing the extracted images from the video. TIP: If you want/need to check long or high resolution videos, you can set it to only extract one image per second. It's a much faster way to check if models match up, and see different facial expressions and angles. 

 

Having extracted the images from the video, I can now push the "FACES A" button, to extract the FACES from the IMAGES. This process can take quite a while, depending on how many images you have. The process may appear to have stalled, but give it at least a few minutes, before you shut it down, even if it looks like nothing is happening.

 

 

 

 

And here we have the most important part of the process: the aligned faces we'll be using to swap out. 

 

IMPORTANT: You'll notice a file called "Alignments.json" in the top-left corner. This file contains instructions as to where the IMAGES are, and where the FACES fit on the IMAGES. For our INPUT files (ie the faces we want to apply to a video), we don't need this file - the program will figure it out on it's own. But for our OUTPUT files (ie the images we want to alter), it's EXTREMELY important that you don't delete or move this file. The folder structure also comes into play here, but I'll cover that later.

 

For now, I would suggest playing around with extracting some files. Once you have some aligned face images extracted, you can freely copy them from the "AlignJaneDouglas" folder, into whatever folder you have in your "Align" folder. You can copy all the files you want in there, just remember to name your videos different things, as it will pick up the name for the image files from the video!

 

Any questions? 

[Joaku]

Basic Training

 

(Link to program in first post)

 

When you have built a bit of a reference catalogue of images, you're ready to start the actual training. 

 

I have changed my folder structure to how I'm used to working: 

 

 

 

 

I have a short video captured in The Klub with FRAPS (https://fraps.com/download.php), "3DJaneDouglas.wmv"

 

I have moved all the ALIGNED FACES to a sub-folder in my "0Align" folder. I have named it something that makes it easy to spot, as I'm browsing through it: "JaneDouglasHighRez". 

 

In my "0Model" folder, I have my actual AI model, also named for easy access. I will get to Models in a second. 

 

I have created folders for all the files I want to work on: Images, Faces and the merged files, which we will just call "Merges" from now on. If you haven't designated a folder, it will create its own, and it can quickly get chaotic. 

 

These are folders that I expect to change and work on often, which is why I have them out in my main install folder. I will record a lot of different videos, make tests and generally need to swap them out. These should not be considered stationary files, so I like to keep them out where I can browse them. If a different folder structure works better for you, create whichever directory you feel like. 

 

As you can see, the files I am currently looking to change are all in the "A" column of the UI. The "Faces B" tab has been pathed to my face folder in "0Align". 

 

Now to the Models! 

 

The Settings tab, next to the "Model" button allows you to choose a few different types: 

 

 

 

The model type (the "Trainer" option, currently set to "Original") determines what kind of AI you want to do the work for you. These are highly dependent on your hardware! If you don't have the fanciest of machines, try the "Low Memory" model. I personally prefer the "Original" type, for the fastest, most accurate results. However, doing DeepFakes with The Klub is very different from doing it with a live model, so the Low Memory model might give perfectly acceptable results as well. 

 

The other options you see here are fairly simple (except for "Allow Growth" - I will get to that in a later post). Your Batch Size essentially determines how many faces it compares at a time. I have tried messing around with this, but I don't see much of a difference. A higher number of Batches might create a more stable image, but it will also require a more powerful machine. "Save Interval" deals with how often the model with "save" itself, but more importantly, how often you'll get an update to the on the Preview screen.

 

I am going to upload a pre-trained model at the end of this post, and explain how I created it, but I HIGHLY recommend that you try setting up an empty folder for the model, and run it from the ground up. Watching how it changes and morphs will give you a sense of what the AI "sees" as it's crunching numbers. It takes a long time, and it's boring as hell, but it will help you understand how the program works (and how it doesn't.)

 

 Here's an example of how a new (left), and a "Trained" (right) model looks: 

 

 

 

Make a couple of test runs: try different clips, lengths and models. This is where you can get a true sense of how the software works. At any time you feel comfortable with it, try pressing the "SWAP" button. This will start the actual process of merging you images into the final result. Checking it along the way is important. You can quickly loose a lot of time on a model that doesn't quite work. 

 

I have uploaded a Trained model for download, that you can use to cut down training time significantly. I created this by training about 60.000 face images at random, to give it a base line for what a human face looks like. Instead of creating a new model every time, you can simply copy/paste this folder, rename it to whatever you like, and use it as a starting point. It especially helps in terms of seeing if The Klub model looks right.

 

Extract it like you would any other folder. Remember that the Model files (decoders/encoder) is not reliant on folder structure, so you can mess around with these as much as you like. If you think maybe you messed it up (which can happen), simply delete the Model files, copy/paste the files from the Original file, and try again. With a base model like this, you don't have to worry about starting over from scratch. 

 

0Original Model: https://drive.google.com/file/d/1etdhc7-aBulI7My4O92BLKRQZR0UWq55/view?usp=share_link

 

Try different combinations and check your work regularly. You're not nearly done, but this is where you can see what you're actually doing! 

 

In the next post, I'll cover different methods for training, including how to use "Allow Growth". 

Edited October 29, 2022 by Joaku