Diffusion: How Image Generators Actually Draw

Diffusion models generate images by learning to reverse a process that gradually destroys an image with Gaussian noise. A neural network trained to predict the noise at each corruption level can then start from pure noise and iteratively denoise it into a picture, guided by a text prompt. Modern systems like Stable Diffusion perform this in a compressed latent space using a U-Net conditioned on CLIP (Contrastive Language-Image Pre-training) embeddings and specialized samplers that trade speed for quality.

The same model that draws a photorealistic cat has no internal representation of a cat. It only knows how to remove noise from images at different corruption levels. Yet from that single skill, the ability to generate any scene emerges.

In our photo app, we might use this to generate synthetic images from captions to augment search results or to let users edit photos with text prompts. Understanding the denoising process is key to controlling quality and speed.

What is the forward diffusion process?

Think of a sculptor who starts with a block of marble and chips away at it, guided by a written description, until a statue appears. The sculptor does not know what a statue is. They only know how to remove marble chips at each step to reveal the form implied by the description. Diffusion models work the same way: the marble is random noise, the chiseling is iterative denoising, the sculptor’s skill is a neural network, and the description comes from a text encoder like CLIP.

The forward process is a Markov chain that adds Gaussian noise in small steps. Starting from a clean image $x_0$, each step $t$ samples $x_t$ from a Gaussian centered on a slightly scaled version of $x_{t-1}$ with variance $\beta_t$ Ho et al. 2020. The schedule ${\beta_t}$ is chosen so that after many steps (often 1000) the distribution of $x_T$ is nearly standard normal, regardless of the original image. This process is analytically tractable. Define $\alpha_t = 1 - \beta_t$ and $\bar{\alpha}t = \prod{s=1}^t \alpha_s$. Then you can sample $x_t$ directly from $x_0$ in one shot:

$$x_t = \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1 - \bar{\alpha}_t},\epsilon, \quad \epsilon \sim \mathcal{N}(0, I).$$

In code, assuming alpha_bar holds the cumulative products:

def forward_diffusion_sample(x0, t, alpha_bar):
    noise = torch.randn_like(x0)
    xt = alpha_bar[t].sqrt() * x0 + (1 - alpha_bar[t]).sqrt() * noise
    return xt, noise

This closed form is the backbone of training. It lets us create noisy versions of any image at any timestep without simulating the whole chain, and it gives us the exact noise $\epsilon$ that was added. The network will later be asked to predict that noise from $x_t$.

How does the reverse process learn to denoise?

The generative model defines a reverse Markov chain $p_\theta(x_{t-1} \mid x_t)$ that starts at $x_T \sim \mathcal{N}(0, I)$ and iteratively removes noise. Each reverse step is modeled as a Gaussian whose mean $\mu_\theta(x_t, t)$ is predicted by a neural network. The true reverse conditional $q(x_{t-1} \mid x_t, x_0)$ is also Gaussian (for small $\beta_t$), and its mean can be expressed in terms of $x_0$ and the noise $\epsilon$ that produced $x_t$. By training the network to predict $\epsilon$, we can plug that prediction into the formula for the true posterior mean and get $\mu_\theta$ Ho et al. 2020.

The training objective is a simplified variational bound. It reduces to a mean squared error between the true noise $\epsilon$ and the network’s prediction $\epsilon_\theta(x_t, t)$:

$$\mathcal{L}\text{simple} = \mathbb{E}{t, x_0, \epsilon}\left[|\epsilon - \epsilon_\theta(x_t, t)|^2\right].$$

This is equivalent to denoising score matching: the network learns to approximate the score $\nabla_x \log p_t(x)$ up to a scaling factor Song et al. 2021. The network never sees a clean image during training. It only ever sees noisy versions and learns how much noise was added.

Some implementations use a $v$-prediction parameterization. Define $v = \alpha_t \epsilon - \sigma_t x_0$ with $\alpha_t = \sqrt{\bar{\alpha}_t}$ and $\sigma_t = \sqrt{1 - \bar{\alpha}t}$. The network predicts $v\theta(x_t, t)$, and you can recover both $\epsilon$ and $x_0$ algebraically. This balances gradient magnitudes across noise levels and often improves stability Salimans & Ho 2022.

Why operate in latent space?

Running diffusion on full-resolution pixels is expensive. A 512×512 RGB image has 786,432 dimensions, and the U-Net would need to process that at every step. Stable Diffusion sidesteps this by moving diffusion into the latent space of a pretrained variational autoencoder (VAE) Rombach et al. 2022. The VAE encoder compresses an image to a much smaller latent tensor (e.g., 4×64×64 for a 512×512 image) while preserving semantic and structural information. Diffusion then operates on this 16,384-dimensional space instead of the pixel space. After denoising, the VAE decoder maps the latent back to an image.

This compression cuts compute by roughly an order of magnitude. The VAE is trained separately to reconstruct images with a mix of L1 and perceptual losses, plus a KL regularizer to keep the latent distribution close to a standard Gaussian. In our photo app, we use a pretrained VAE from Stable Diffusion so that the latent space is fixed and we only need to train or run the U-Net.

How does text conditioning work?

The U-Net inside a latent diffusion model is conditioned on text via cross-attention. The text encoder (CLIP ViT-L/14 in Stable Diffusion) converts the prompt into a sequence of token embeddings, each a 768-dimensional vector Rombach et al. 2022. These embeddings are fed into every cross-attention layer of the U-Net, where they act as keys and values while the latent features serve as queries. This lets the network attend to relevant words at each spatial location and timestep.

Because you already know CLIP from Part 1, recall that its embeddings align text and image semantics. The U-Net inherits that alignment. When the denoiser sees the token “cat,” the cross-attention maps will highlight regions that should contain cat-like features, and the denoising step will preserve or enhance those features while removing noise elsewhere.

How does sampling actually produce an image?

Sampling is the iterative loop that turns $z_T \sim \mathcal{N}(0, I)$ into a clean latent $z_0$. At each step, the sampler picks a timestep $t$, feeds $z_t$ and $t$ into the U-Net to get a noise prediction $\hat{\epsilon}$, and then computes $z_{t-1}$ using a specific update rule. The choice of sampler determines the rule, the number of steps, and whether randomness is injected.

The original DDPM sampler uses the full Markov chain with 1000 steps and adds fresh noise at each step. DDIM Song et al. 2021a is a deterministic alternative that can skip steps, producing coherent images in as few as 20-50 iterations. Higher-order ODE solvers like DPM++ Lu et al. 2022 converge even faster by approximating the reverse probability flow ODE with adaptive step sizes.

A minimal DDIM loop looks like this (simplified):

def ddim_step(z, t, t_prev, model, alpha_bar):
    eps = model(z, t)
    alpha_t = alpha_bar[t]
    alpha_prev = alpha_bar[t_prev]
    # Predict x0
    x0_pred = (z - (1 - alpha_t).sqrt() * eps) / alpha_t.sqrt()
    # Direction pointing to z
    dir_zt = (1 - alpha_prev).sqrt() * eps
    z_prev = alpha_prev.sqrt() * x0_pred + dir_zt
    return z_prev

In practice, you iterate over a decreasing sequence of timesteps, often using a schedule that concentrates steps where the score changes fastest (e.g., near $t=0$). For our photo app’s text-to-image feature, we use DPM++ 2M with 25 steps as a good speed-quality tradeoff.

What is classifier-free guidance and why does it matter?

Classifier-free guidance (CFG) strengthens prompt adherence by mixing the conditional and unconditional noise predictions during sampling Ho & Salimans 2022. The network is trained to denoise both with and without text conditioning. At inference, you compute a guided prediction:

$$\hat{\epsilon}\text{guided} = \epsilon\theta(z_t, t, c) + w \bigl(\epsilon_\theta(z_t, t, c) - \epsilon_\theta(z_t, t, \emptyset)\bigr),$$

where $c$ is the text embedding, $\emptyset$ is a null embedding, and $w$ is the guidance scale. When $w > 1$, the model is pushed away from the unconditional trajectory and toward the conditional one. Typical values are 7-9.

Higher $w$ makes the image match the prompt more literally but can oversaturate colors and introduce artifacts because it effectively sharpens the score field beyond what the model was trained on. In our app, we set $w=7.5$ as a default that balances fidelity and naturalness.

Quick Reference

Frequently Asked Questions

Q: Why does increasing the guidance scale sometimes produce oversaturated or distorted images? CFG extrapolates the score beyond the training distribution. The model was trained on a mix of conditional and unconditional denoising, and a high $w$ effectively pushes the trajectory into regions where the score estimate is less reliable. This can amplify high-frequency artifacts and shift color statistics.

Q: Can I use any text encoder, or does it have to be CLIP? You can use any encoder that produces a fixed-length sequence of embeddings, but the U-Net’s cross-attention layers were trained to align with that specific encoder’s representation space. Swapping encoders without retraining will break the conditioning. CLIP is popular because its contrastive training aligns well with image semantics.

Q: What is the difference between DDIM and DPM++? DDIM is a first-order deterministic solver for the probability flow ODE. DPM++ is a family of higher-order solvers that use additional function evaluations per step to achieve lower error. DPM++ 2M, for example, converges in roughly half the steps of DDIM for the same perceptual quality.

Q: How do I choose the number of sampling steps? Start with the sampler’s recommended minimum (e.g., 20 for DPM++ 2M). Increase steps if you see blurriness or incomplete convergence. Diminishing returns set in quickly. For most use cases, 25-30 steps is enough. Going beyond 50 rarely helps unless you need pixel-perfect reproducibility.

Q: Why does the same seed and prompt produce different images with different samplers? Each sampler approximates the reverse process differently. Deterministic samplers like DDIM follow a specific ODE trajectory, while stochastic ones inject noise. Even among deterministic solvers, the discretization error and step schedule change the path through latent space, leading to different final latents.

Test yourself

Your photo app’s text-to-image feature uses a 20-step DDIM sampler and users report blurry, low-detail results. You suspect the sampler is not converging. What would you investigate and change?

Answer: First, switch to a higher-order solver like DPM++ 2M Karras, which converges faster and handles the noise schedule better. Increase steps to 30-40 to give the solver more budget. Check that the VAE decoder is the correct one for the latent distribution; a mismatched decoder can introduce blur. Verify that the latent is properly scaled before decoding (some implementations require rescaling by the VAE’s standard deviation). If the problem persists, inspect the noise schedule. A schedule that spends too few steps at low noise levels can leave residual noise that the decoder interprets as blur. Finally, ensure the CFG scale is not too high, as excessive guidance can suppress fine details.

If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.

Next, we will look at how these same diffusion models can be adapted for image editing, inpainting, outpainting, and instruction-based modifications, by conditioning on masks and existing image latents.

Sources

• Denoising Diffusion Probabilistic Models, Ho et al. 2020
• Score-Based Generative Modeling through Stochastic Differential Equations, Song et al. 2021
• High-Resolution Image Synthesis with Latent Diffusion Models, Rombach et al. 2022
• Classifier-Free Diffusion Guidance, Ho & Salimans 2022
• Progressive Distillation for Fast Sampling of Diffusion Models, Salimans & Ho 2022 (v-prediction)
• Denoising Diffusion Implicit Models, Song et al. 2021a
• DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling, Lu et al. 2022