Diffusion: How Image Generators Actually Draw

Diffusion models generate images by learning to reverse a process that gradually destroys an image with Gaussian noise (random static whose values follow a bell curve). A neural network trained to predict the noise at each corruption level can then start from pure noise and remove it step by step until a picture appears, guided by a text prompt. This article takes apart Stable Diffusion 1.x, the classic open model from 2022. It works in a compressed latent space, with a U-Net denoiser (a network that shrinks the image step by step and grows it back, with shortcuts linking the matching sizes) conditioned on the outputs of CLIP's (Contrastive Language-Image Pre-training) text encoder, plus samplers that trade speed for quality. Newer generators keep the idea but change the parts; the section "What changed after Stable Diffusion 1.x?" covers them.

The model was never told what a cat is. It was only trained to remove noise from images at different corruption levels. To do that well it had to learn internal features for edges, textures, objects and layouts, and researchers can read those features back out of it: features taken from inside Stable Diffusion find matching points between different photos better than DINO and OpenCLIP features on a hard benchmark, with no extra training (Tang et al. 2023, Emergent Correspondence from Image Diffusion). From that one skill, removing noise, comes the ability to draw almost any scene.

Think of a sculptor who starts with a rough block of marble and chips away at it, guided by a written description, until a statue appears. At every step the sculptor only decides which chips to remove next, but choosing well means knowing a great deal about what statues look like. Diffusion models work the same way: the marble is random noise, the chiseling is step-by-step denoising, the sculptor's skill is a neural network, and the description comes from a text encoder such as CLIP's.

In our photo app, we might use this to generate images from a description, or to let users edit their photos with a text prompt (see "How do you edit a photo with diffusion?" below). Understanding the denoising process is the key to controlling quality and speed.

What is the forward diffusion process?

The forward process is a Markov chain (each step depends only on the step before it) that adds Gaussian noise in small steps. Starting from a clean image $x_0$, each step $t$ samples $x_t$ from a Gaussian centered on a slightly shrunk copy of $x_{t-1}$, with variance $\beta_t$ (Ho et al. 2020, Denoising Diffusion Probabilistic Models). The schedule $\{\beta_t\}$ is chosen so that after many steps the distribution of $x_T$ is nearly standard normal, whatever the original image. DDPM uses 1,000 steps, with $\beta_t$ rising linearly from 0.0001 to 0.02. The process has a convenient shortcut. Define $\alpha_t = 1 - \beta_t$ and $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$. Then you can sample $x_t$ directly from $x_0$ in one shot:

In words: the noisy image is a faded copy of the clean image plus a measured scoop of random static. The later the step, the less image and the more static.

In code, assuming alpha_bar holds the cumulative products:

def forward_diffusion_sample(x0, t, alpha_bar):
    noise = torch.randn_like(x0)
    xt = alpha_bar[t].sqrt() * x0 + (1 - alpha_bar[t]).sqrt() * noise
    return xt, noise

This closed form is the backbone of training. It lets us create noisy versions of any image at any timestep without simulating the whole chain, and it gives us the exact noise $\epsilon$ that was added. The network will later be asked to predict that noise from $x_t$.

How does the reverse process learn to denoise?

The generative model defines a reverse Markov chain $p_\theta(x_{t-1} \mid x_t)$ that starts at $x_T \sim \mathcal{N}(0, I)$ and removes noise step by step. Each reverse step is modeled as a Gaussian whose mean $\mu_\theta(x_t, t)$ is predicted by a neural network. That choice works because, when the steps $\beta_t$ are small, the true reverse step is close to Gaussian. The reverse step that also knows the clean image, $q(x_{t-1} \mid x_t, x_0)$, is exactly Gaussian for any step size, and its mean can be written in terms of $x_0$ and the noise $\epsilon$ that produced $x_t$. By training the network to predict $\epsilon$, we can plug that prediction into the formula for this mean and get $\mu_\theta$ (Ho et al. 2020).

The training objective is a simplified variational bound. It reduces to a mean squared error between the true noise $\epsilon$ and the network's prediction $\epsilon_\theta(x_t, t)$:

In words: show the network a noisy image and its step number, ask it to guess the noise, and penalize the squared error of the guess.

This is equivalent to denoising score matching: the network learns to approximate the score $\nabla_x \log p_t(x)$, the direction that would make the noisy image more likely, up to a scaling factor (Song et al. 2021, Score-Based Generative Modeling through Stochastic Differential Equations). The network's input is never a clean image during training. It only sees noisy versions and learns how much noise was added.

Some implementations use a $v$-prediction parameterization. Define $v = a_t \epsilon - \sigma_t x_0$ with $a_t = \sqrt{\bar{\alpha}_t}$ and $\sigma_t = \sqrt{1 - \bar{\alpha}_t}$. The network predicts $v_\theta(x_t, t)$, and you can recover both $\epsilon$ and $x_0$ algebraically. Salimans and Ho introduced it because it keeps the implied prediction of the clean image stable even at the noisiest steps, and found it the most stable option they tried (2022, Progressive Distillation for Fast Sampling of Diffusion Models).

Why operate in latent space?

Running diffusion on full-resolution pixels is expensive. A 512×512 RGB image has 786,432 numbers, and the denoiser would have to process all of them at every step. Stable Diffusion sidesteps this by running diffusion in the latent space of a pretrained autoencoder (Rombach et al. 2022, High-Resolution Image Synthesis with Latent Diffusion Models). An autoencoder is a learned compressor and decompressor. Its encoder squeezes an image into a much smaller latent tensor (4×64×64 for a 512×512 image, a downsampling factor of 8, per the Stable Diffusion v1 model card) while keeping its structure and meaning, and its decoder rebuilds the image. Diffusion then works on these 16,384 numbers instead of the pixels. That compressed space is what "latent space" means here.

That is 48 times fewer numbers for the denoiser to handle at every step. The autoencoder is trained separately, with reconstruction and perceptual losses plus an adversarial loss from a patch-based discriminator (a second network that tries to tell real images from rebuilt ones). The variant Stable Diffusion uses is a VAE (variational autoencoder), but with only a very weak KL penalty pulling its latents toward a standard Gaussian, weighted by about one millionth, so that reconstructions stay sharp. Because that pull is so weak, the latents do not have unit variance, and Stable Diffusion multiplies them by a fixed 0.18215 before diffusion and divides by it before decoding. In our photo app, we use the pretrained autoencoder from Stable Diffusion, so the latent space is fixed and we only need to run the denoiser.

How does text conditioning work?

The denoiser inside a latent diffusion model is conditioned on text through cross-attention. In Stable Diffusion 1.x, the text encoder is the text transformer of CLIP ViT-L/14, kept frozen. It turns the prompt into a sequence of 77 token vectors of 768 numbers each, and the model card says this non-pooled output is what feeds the U-Net through cross-attention. (The original latent diffusion paper's own text-to-image model used a BERT tokenizer and a transformer trained together with the model; the CLIP encoder came with the Stable Diffusion release.) In every cross-attention layer, the latent image features act as queries and the token vectors act as keys and values, so each spatial location can attend to the words that matter for it at each timestep.

This is not the single CLIP vector from Part 1. Stable Diffusion uses the whole sequence of per-token outputs, and the U-Net learns how words map to image regions through cross-attention during its own training. That is why encoders trained only on text work as well or better: the Imagen team found that T5, a language model trained on text alone, is surprisingly effective, and that making the text encoder bigger helped more than making the image model bigger (Saharia et al. 2022, Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding). When the denoiser sees the token "cat", the cross-attention maps highlight regions that should hold cat-like features, and the denoising step strengthens those features while removing noise elsewhere.

How does sampling actually produce an image?

Sampling is the loop that turns $z_T \sim \mathcal{N}(0, I)$ into a clean latent $z_0$. At each step, the sampler picks a timestep $t$, feeds $z_t$ and $t$ into the U-Net to get a noise prediction $\hat{\epsilon}$, and then computes $z_{t-1}$ using a specific update rule. The choice of sampler sets the rule, the number of steps, and whether randomness is injected.

The original sampler from Denoising Diffusion Probabilistic Models walks the full Markov chain of 1,000 steps and adds fresh noise at each one. DDIM keeps the same trained network but samples deterministically and can skip steps; its authors report high quality samples 10 to 50 times faster in wall-clock time (Song et al. 2021, Denoising Diffusion Implicit Models). DDIM and its successors treat sampling as following a smooth path, an ODE (ordinary differential equation) that you can walk in big or small steps.

Higher-order ODE solvers walk that path more accurately per step. DPM-Solver produces high quality samples in about 10 to 20 network evaluations (Lu et al. 2022, DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps). Its version for guided sampling, DPM-Solver++, is the "DPM++" in many sampler menus; its authors report high quality guided samples in 15 to 20 steps, where DDIM generally needs 100 to 250 (Lu et al. 2022, DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models).

A minimal DDIM loop looks like this (simplified):

def ddim_step(z, t, t_prev, model, alpha_bar):
    eps = model(z, t)
    alpha_t = alpha_bar[t]
    alpha_prev = alpha_bar[t_prev]
    # Predict x0
    x0_pred = (z - (1 - alpha_t).sqrt() * eps) / alpha_t.sqrt()
    # Direction pointing to z
    dir_zt = (1 - alpha_prev).sqrt() * eps
    z_prev = alpha_prev.sqrt() * x0_pred + dir_zt
    return z_prev

In practice, you iterate over a decreasing sequence of timesteps, and many tools let you choose how those steps are spaced; the Karras schedule, a common option, spends shorter steps at the low noise end, where fine detail is settled (Karras et al. 2022, Elucidating the Design Space of Diffusion-Based Generative Models). For our photo app's text-to-image feature, we use DPM++ 2M with 25 steps as a good balance of speed and quality.

What is classifier-free guidance and why does it matter?

Classifier-free guidance (CFG) strengthens prompt adherence by mixing the conditional and unconditional noise predictions during sampling (Ho & Salimans 2022, Classifier-Free Diffusion Guidance). One network learns both modes because the caption is dropped at random during training; Stable Diffusion v1 dropped it 10% of the time. At inference you compute both predictions at every step, so CFG costs two network passes per step, and you combine them:

where $c$ is the text embedding, $\emptyset$ is the embedding of an empty prompt, and $s$ is the guidance scale. In words: start from the prediction without the prompt, then move $s$ times as far as the prompt pulls. With $s = 1$ you get the plain conditional prediction, with no guidance at all; $s > 1$ pushes past it toward the prompt. The standard Stable Diffusion pipeline in Hugging Face's diffusers library uses this form and defaults to 7.5. Ho and Salimans write the same rule with a weight $w = s - 1$, so in their paper $w = 0$ means no guidance. Check which convention a paper or tool uses before copying its numbers.

Higher $s$ makes the image match the prompt more literally but can oversaturate colors and add artifacts. The Imagen authors traced this to a mismatch between training and sampling: at high guidance, the model's prediction of the clean image goes outside the range of real pixel values, so they added dynamic thresholding to pull it back at every step. In our app, we set $s = 7.5$ as a default that balances fidelity and naturalness.

What changed after Stable Diffusion 1.x?

Everything above describes the 2022 design, and it is still the clearest way to learn diffusion. Three changes since then matter most.

The U-Net gave way to transformers. A Diffusion Transformer, or DiT, cuts the latent into patches, treats each patch as a token, and runs a transformer instead of a U-Net. Peebles and Xie found that DiTs given more compute (deeper, wider, or with more tokens) kept getting better, which is why the design scaled (2022, Scalable Diffusion Models with Transformers). The video models in Part 4 build on this idea.

The training target got straighter. Flow matching (Lipman et al. 2022, Flow Matching for Generative Modeling) and rectified flow (Liu et al. 2022, Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow) train the network to predict a velocity: the direction from noise toward the image along a path that is as straight as possible. Straight paths can be followed in fewer, larger steps. Stable Diffusion 3 combined both ideas in a rectified flow model whose transformer keeps separate weights for image and text tokens and lets information flow both ways between them (Esser et al. 2024, Scaling Rectified Flow Transformers for High-Resolution Image Synthesis).

The text encoders got bigger. Stable Diffusion 3 reads the prompt with three encoders: two CLIP models and T5. Black Forest Labs' FLUX.1 (August 2024) is a 12 billion parameter model built from diffusion transformer blocks and trained with flow matching (Announcing Black Forest Labs), with T5 and CLIP text encoders in its reference code. FLUX.2 (November 2025) couples a rectified flow transformer with a 24 billion parameter Mistral vision-language model that reads the prompt (FLUX.2: Frontier Visual Intelligence).

Not every image generator is a diffusion model. OpenAI's Addendum to GPT-4o System Card: Native image generation (March 2025) says that, unlike DALL·E, 4o image generation is an autoregressive model built into GPT-4o. Google's Imagen 4 Model Card (May 2025) calls latent diffusion "the de facto standard approach for modern image and video models". Many vendors do not publish how their newest image models work, so as of October 2026 you often cannot tell from the outside which method is behind a product.

Where does diffusion go wrong?

Diffusion models fail in recognizable ways, and knowing them helps you set expectations and write better prompts:

• Hands and fine structure. Generated hands often have the wrong number of fingers or odd shapes, because hands deform and hide each other in so many ways in training photos (Lu et al. 2023, HandRefiner: Refining Malformed Hands in Generated Images by Diffusion-based Conditional Inpainting).
• Readable text. Older models scrambled the letters in signs and labels; the DALL·E 2 paper reports that its model struggles to produce coherent text (Ramesh et al. 2022, Hierarchical Text-Conditional Image Generation with CLIP Latents). Newer models with large text encoders spell much better.
• Missing objects and swapped attributes. Ask for "a red cube on a blue ball" and the colors may swap, or one object may vanish. Chefer et al. found Stable Diffusion sometimes leaves out a subject entirely and fails to bind colors to the right objects (2023, Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models). Counting is hard too: the T2I-CompBench++ benchmark tests attribute binding, object relationships and generative numeracy as separate weaknesses (Huang et al. 2023, T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generation).
• Memorized training images. Carlini et al. extracted over a thousand training examples from state-of-the-art diffusion models, from photos of individual people to trademarked logos (2023, Extracting Training Data from Diffusion Models).
• Oversaturation at high guidance, as described above.

How do you edit a photo with diffusion?

The same denoiser that draws from noise can edit a photo you already have. In image to image editing, you add noise to your photo part of the way, then denoise it with a new prompt; the more noise you add, the more the result can change. That is the idea behind SDEdit (Meng et al. 2021, SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations), and it is the "strength" slider in many tools. Inpainting changes only a masked region: RePaint, for example, keeps the known pixels from your photo at every step and lets the model generate only inside the mask (Lugmayr et al. 2022, RePaint: Inpainting using Denoising Diffusion Probabilistic Models). In our photo app, "replace the gray sky with a sunset" is an inpainting job on the sky.

Stable Diffusion 1.x reference

Frequently Asked Questions

Q: Why does increasing the guidance scale sometimes produce oversaturated or distorted images? Guidance pushes the prediction past anything the model produced in training. The Imagen authors found that at high guidance the predicted clean image falls outside the range of real pixel values, which shows up as saturated, unnatural colors. Their fix, dynamic thresholding, pulls those predictions back inside the range at every step.

Q: Can I use any text encoder, or does it have to be CLIP? Any encoder that produces a sequence of token vectors can work, but the denoiser's cross-attention layers learn to read one specific encoder during training, so swapping encoders means retraining. CLIP was a natural choice in 2022 because it was trained on images and captions, but text-only encoders work as well or better: Imagen used T5, Stable Diffusion 3 combines two CLIP models with T5, and FLUX.2 uses a vision-language model.

Q: What is the difference between DDIM and DPM++? DDIM is a first-order solver for the probability flow ODE: it uses one network evaluation per step and takes a straight step each time. DPM-Solver++ uses higher-order steps that follow the curve of the path more closely, and its multistep variant, the "2M" in sampler menus, reuses the previous step's prediction instead of spending extra evaluations. Its authors report high quality guided samples in 15 to 20 steps, where DDIM generally needs 100 to 250.

Q: How do I choose the number of sampling steps? Start with the sampler's recommended minimum (around 20 for DPM++ 2M). Increase the steps if you see blurriness or unfinished detail. Returns diminish quickly: 25 to 30 steps is usually enough, and going beyond 50 rarely helps.

Q: Why does the same seed and prompt produce different images with different samplers? Each sampler approximates the reverse process differently. Deterministic samplers like DDIM follow a specific ODE path, while stochastic ones inject noise. Even among deterministic solvers, the discretization error and the step schedule change the path through latent space, leading to different final latents.

Test yourself

Your photo app's text-to-image feature uses a 20-step DDIM sampler, and users report blurry, low-detail results. You suspect the sampler is not converging. What would you investigate and change?

Answer: First, switch to a higher-order solver such as DPM++ 2M with the Karras schedule, which converges in fewer steps, and raise the step count to 30 or 40 to give the solver more room; the DPM-Solver++ authors report that DDIM generally needs 100 to 250 steps under strong guidance, so 20 may simply be too few. Check that the decoder is the right autoencoder for this model's latents; a mismatched decoder can blur the image. Verify the latent scaling: Stable Diffusion's latents must be divided by 0.18215 before decoding, and skipping that step garbles the output. If the problem persists, inspect the noise schedule: one that spends too few steps at low noise can leave residual noise that the decoder turns into blur. Finally, make sure the guidance scale is not set too high, since excessive guidance damages image fidelity.

Next: Video Models: Why Motion Is So Much Harder. It adds time to everything here, running the same denoising over whole clips.

If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.

Sources

• Denoising Diffusion Probabilistic Models (Ho et al. 2020)
• Score-Based Generative Modeling through Stochastic Differential Equations (Song et al. 2021)
• High-Resolution Image Synthesis with Latent Diffusion Models (Rombach et al. 2022)
• Stable Diffusion v1 model card (CompVis)
• Classifier-Free Diffusion Guidance (Ho & Salimans 2022)
• Progressive Distillation for Fast Sampling of Diffusion Models (Salimans & Ho 2022, v-prediction)
• Denoising Diffusion Implicit Models (Song et al. 2021)
• DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps (Lu et al. 2022)
• DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models (Lu et al. 2022)
• Elucidating the Design Space of Diffusion-Based Generative Models (Karras et al. 2022)
• Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (Saharia et al. 2022, Imagen)
• Emergent Correspondence from Image Diffusion (Tang et al. 2023)
• Scalable Diffusion Models with Transformers (Peebles & Xie 2022)
• Flow Matching for Generative Modeling (Lipman et al. 2022)
• Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow (Liu et al. 2022)
• Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (Esser et al. 2024, Stable Diffusion 3)
• Announcing Black Forest Labs and FLUX.2: Frontier Visual Intelligence (Black Forest Labs)
• Addendum to GPT-4o System Card: Native image generation (OpenAI, March 2025)
• Imagen 4 Model Card (Google DeepMind, May 2025)
• HandRefiner: Refining Malformed Hands in Generated Images by Diffusion-based Conditional Inpainting (Lu et al. 2023)
• Hierarchical Text-Conditional Image Generation with CLIP Latents (Ramesh et al. 2022)
• Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models (Chefer et al. 2023)
• T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generation (Huang et al. 2023)
• Extracting Training Data from Diffusion Models (Carlini et al. 2023)
• SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations (Meng et al. 2021)
• RePaint: Inpainting using Denoising Diffusion Probabilistic Models (Lugmayr et al. 2022)
• reference code