Submitted by jayalammar t3_xvje2n in MachineLearning
Hi r/MachineLearning,
​
Here's a visual description of how Stable Diffusion works, with over 30 original images covering diffusion models, latent diffusion models, CLIP and how it's trained, and more.
https://jalammar.github.io/illustrated-stable-diffusion/
I appreciate all corrections and feedback.
Much appreciated <3
This is amazing. I have been following your work for years, you do a tremendous service for the entire community/industry!
I agree, this is great work!
I would like to take this opportunity to thank you for you article on transformers. It helped tremendously.
Thank you for sharing this
Minor typo: "ransom latents tensor".
For the diffusion model the "noise" is actually (approximately) integrated over, is that correct?
This is such a great visual description of stable diffusion. I love thinking of it like "This is what a sequence of gradually noisier images looks like" , then flipping the sequence around and saying "this is what a natural image generated from noise looks like" and using that as training
Typo:
>The trained noise predictor can take a noisy image, and athe number of the denoising step, and is able to predict a slice of noise.
Whenever I see "diffusers", in my mind I read it as "diff users" and no one is going to stop me.
This is amazing! Thanks for sharing.
When you say that OpenClip can potentially replace the CLIP model, the rest doesn't need to be retrained does it? Is the CLIP model trained jointly with the diffusion Unet and autoencoder?
New Stable Diffusion models have to be trained to utilize the OpenCLIP model. That's because many components in the attention/resnet layer are trained to deal with the representations learned by CLIP. Swapping it out for OpenCLIP would be disruptive.
In that training process, however, OpenCLIP can be frozen just like how CLIP was frozen in the training of Stable Diffusion / LDM.
I'm very new/ ignorant, yet I immediately saved this reddit. I already can tell this is going to be awesome.
Thank you for sharing this
Thanks Jay for doing this!!
I'd love a bit more information on the text conditioning steps.
Definitely, the best illustrated article out there.
I have thinking if this model is possible use in opposite, I consider that can be powerful for describe images and useful by blind people.
About time! Your illustrated transformers helped is what helped me learn them and I got this present just when I wanted to know how stable diffusion works. Thank you!
>(The actual complete prompt is here)
Should there be a link in this part? For me it's just plain text.
My bad, you're right. It's "paradise cosmic beach by vladimir volegov and raphael lacoste". I arbitrarily picked an image from https://lexica.art/.
Yet I open up the codebase and I still can't understand shit. What are we getting out of the model when we run inference on it? It's not an image, it looks like some sort of "bag of imagery". We have a sampler that is sampling this bag. How does this work exactly? I hate these high level explanations, they don't explain anything. No one can read this article and reimplement Stable Diffusion. I look at the different samplers implemented in k-diffusion and I am left mystified.
Sorry if I come off aggressive, not the intention! Your explanation on transformers is truly amazing and this one is great as well. I'm just tired of reading these overly simplified explanations targeted at 'mom and dad'; These little arrows and grids don't mean anything to me if you don't relate them to the code. Stable Diffusion has nothing to do with maths and statistics, it is a programmed behavior. Imagine if we explained how to implement a raycaster purely theoretically with pictograms. F*** that! A minimal implementation of a raycaster with heavy documentation, and pictograms on the side if you want, is infinitely more useful.
I may not be a master statistician, but as a programmer if you explain each line one by one I should be able to truly grasp what is happening. Print the tensors, show me exactly what they look like in text, then you can map the text to images. If someone actually explained these implementations, we could unlock a whole new pool of talent contributing to the field. This does not help anyone understand how SD works, it only helps to pretend like I do.
​
This might be closer to what you're looking for: https://huggingface.co/blog/annotated-diffusion
Fantastic work. As a laymen I am almost starting to understand much of this. Almost.
In the section titled "How Clip is trained", are the captions correct? The first appears to have a typo and the FC caption seems jumbled.
Thank you!
This caption?
>Larger/better language models have a significant effect on the quality of image generation models. Source: Google Imagen paper by Saharia et. al.. Figure A.5.
What's the issue?
Sorry if I wasn't clear. I meant the captions on the first graphic
You have a figure with 3 images, a pagota, an eagle, and a Far Cry screen shot. The first and third captions appear to have a mistake.
The pagota caption:
"Photo pour Japanese pagoda....."
The Far Cry caption:
"Far Cry 4 concept art is the reason why it 39 s a beautiful game VG247. Black bedroom furniture...... "
Oh, okay, I understand you now. These are actual examples from the dataset. These were the captions of these images in the LAION Aesthetic dataset. https://huggingface.co/datasets/ChristophSchuhmann/improved_aesthetics_6.5plus
Have you seen any of the results from ADC biologically integrated chips? Diffuse mode offers an explanation on how a model will sort according to a scale of set values dependent on weights. Following a path with weights along the edges, hidden variables included, lead to destinations previously overlooked by a default, biological focused learning. Focused learning hijacks your attention to the initial focal point, pinging or reinforcing the same thing over and over down a fruitless path. Without a focal input is without discrimination. Calorically exhausting and a task without a task manager. But it works, very well.
“It is impossible to be fully immersed in a world with no depth.” -forever yours, truly and sadly” -2D
Would you like some mayo with your word salad?
DigThatData t1_ir1j32u wrote
For those who don't recognize the name/url: OP authored a similarly titled tutorial on transformers that is widely considered one of the best introductions to the topic.
Keep up the good work Jay, thanks for the great content :)