VAE is a partial update to Stable Diffusion 1.4 or 1.5 models that will make rendering eyes better. I will explain what VAE is, what you can expect, where you can get it, and how to install and use it.
VAE stands for variational autoencoder. It is part of the neural network model that encodes and decodes the images to and from the smaller latent space, so that computation can be faster.
Do I need a VAE
You don’t need to install a VAE file to run Stable Diffusion—any models you use, whether v1, v2 or custom, already have a default VAE.
When people say downloading and using a VAE, they refer to using an improved version of it. This happens when the model trainer further fine-tunes the VAE part of the model with additional data. Instead of releasing a whole new model, which is a big file, they release only the tiny part that has been updated.
You don’t need to use a VAE if you are happy with the result you are getting. E.g., you are already using face restoration like CodeFormer to fix eyes.
You should use a VAE if you are in the camp of taking all the little improvements you can get. You only need to go through the trouble of setting it up once. After that, the art creation workflow stays the same.
How to use VAE
Download
Currently, there are two improved versions of VAE released by Stability. Below are direct download links.
Install
This install instruction applies to AUTOMATIC1111 GUI. Place the downloaded VAE files in the directory
stable-diffusion-webui/models/VAE
For Linux and Mac OS
For your convenience, run the commands below in Linux or Mac OS under stable-diffusion-webui’s directory downloads and installs the VAE files.
wget https://huggingface.co/stabilityai/sd-vae-ft-ema-original/resolve/main/vae-ft-ema-560000-ema-pruned.ckpt -O models/VAE/vae-ft-ema-560000-ema-pruned.ckpt wget https://huggingface.co/stabilityai/sd-vae-ft-mse-original/resolve/main/vae-ft-mse-840000-ema-pruned.ckpt -O models/VAE/vae-ft-mse-840000-ema-pruned.ckpt
Use
To use a VAE in AUTOMATIC1111 GUI, go to the Settings tab and click the Stabe Diffusion section on the left.
And find a section called SD VAE. In the dropdown menu, select the VAE file you want to use.
Press the big red Apply Settings button on top. You should see the message
Settings: sd_vae applied
in the Setting tab when the loading is successful.
Other options in the dropdown menu are:
- None: Use the original VAE that comes with the model.
- Auto: see this post for behavior. I don’t recommend beginners use Auto since it is easy to confuse which VAE is used.
Pro tip: If you cannot find a setting, click Show All Pages on the left. All settings will be shown on a single page. Use Ctrl-F to find the setting.