Diffusion Models

Purpose

Diffusion models are easiest to understand if you separate three layers:

1. a fixed forward process that gradually destroys data with noise
2. a learned reverse process that denoises
3. a training objective that makes denoising equivalent to a tractable regression problem

The DDPM paper is the canonical reference for that setup.

flowchart LR
    x0["x0: data"] -- "q, fixed noising" --> x1["x1"]
    x1 -- "q" --> mid["..."]
    mid -- "q" --> xT["xT: pure noise"]
    xT -. "p, learned denoising" .-> mid
    mid -. "p" .-> x1
    x1 -. "p" .-> x0

The solid direction is fixed and has no parameters. All learning happens in the dotted direction.

Forward Process

Start from clean data $x_0$. Define a Markov chain:

with Gaussian step

where ${\beta }_t$ is a variance schedule.

Let

Then DDPM uses the closed form

which yields the useful sampling identity

This is why training can sample arbitrary timesteps directly instead of simulating the entire noising chain every time.

Variational Objective

The paper starts from the usual variational bound on negative log likelihood:

That objective can be decomposed into timestep-wise terms. The paper’s key contribution is not only writing down the bound. It is showing how to parameterize the reverse process so training becomes simple and stable.

Reverse Process and Noise Prediction

The reverse model is

parameterized as a Gaussian with learned mean. DDPM shows that a particularly convenient approach is to predict the noise $ϵ$ instead:

This leads to the simplified training objective

That is the practical heart of the method. The model is trained as a noise regressor, not as a direct pixel generator.

Why training is cheap even though sampling is slow

The closed form $x_t=\sqrt{{\bar{\alpha }}_t}{x}_0+\sqrt{1-{\bar{\alpha }}_t}ϵ$ means a training step samples a random $t$, produces $x_t$ from $x_0$ in one shot, and regresses the known $ϵ$ — no simulation of the noising chain, no sequential dependence between timesteps. Every minibatch trains the denoiser at randomly scattered noise levels simultaneously. The $T$-step sequential cost only appears at sampling time.

Sampling Algorithm

The paper’s Algorithm 2 is the reverse-time sampler:

1. start from $x_T\sim \mathcal{N}(0,I)$
2. for $t=T,\dots ,1$, predict noise with ${ϵ}_{\theta }(x_t,t)$
3. form the mean of $p_{\theta }(x_{t-1}\mid x_t)$
4. add Gaussian noise when $t>1$

The extracted update in the paper is

with $z\sim \mathcal{N}(0,I)$ for $t>1$.

This explains why diffusion generation is slow. One sample requires many denoising steps.

Why U-Nets Became the Default Backbone

Image diffusion models need:

• local spatial processing
• large receptive fields
• high-resolution detail preservation
• conditioning on timestep $t$

U-Nets already solve most of that. They combine:

• a contracting path
• an expanding path
• skip connections across scales

So diffusion models inherited a backbone that was already strong for dense image-to-image problems.

Results from the DDPM Paper

The paper reports:

• Inception score 9.46 on unconditional CIFAR-10
• FID 3.17 on unconditional CIFAR-10
• sample quality on 256x256 LSUN similar to ProgressiveGAN

The paper also argues that diffusion admits a progressive lossy decompression view, which is a useful intuition: early reverse steps recover coarse structure, later ones add detail.

NumPy Schedule and Noising Step

import numpy as np
 
def make_schedule(T, beta_start=1e-4, beta_end=2e-2):
    betas = np.linspace(beta_start, beta_end, T, dtype=np.float32)
    alphas = 1.0 - betas
    alpha_bars = np.cumprod(alphas)
    return betas, alphas, alpha_bars
 
def q_sample(x0, t, alpha_bars, noise):
    return np.sqrt(alpha_bars[t]) * x0 + np.sqrt(1.0 - alpha_bars[t]) * noise
 
def predict_x0_from_eps(xt, t, eps_hat, alpha_bars):
    return (xt - np.sqrt(1.0 - alpha_bars[t]) * eps_hat) / np.sqrt(alpha_bars[t])

PyTorch Noise-Predictor Sketch

import math
import torch
import torch.nn as nn
from einops import rearrange
 
class TimeEmbedding(nn.Module):
    def __init__(self, dim: int):
        super().__init__()
        self.dim = dim
 
    def forward(self, t: torch.Tensor) -> torch.Tensor:
        half = self.dim // 2
        freqs = torch.exp(-math.log(10000) * torch.arange(half, device=t.device) / half)
        args = t[:, None].float() * freqs[None]
        return torch.cat([args.sin(), args.cos()], dim=-1)
 
class ConvBlock(nn.Module):
    def __init__(self, cin: int, cout: int, tdim: int):
        super().__init__()
        self.conv1 = nn.Conv2d(cin, cout, 3, padding=1)
        self.conv2 = nn.Conv2d(cout, cout, 3, padding=1)
        self.time_proj = nn.Linear(tdim, cout)
        self.act = nn.SiLU()
 
    def forward(self, x, t_emb):
        h = self.act(self.conv1(x))
        h = h + rearrange(self.time_proj(t_emb), "b c -> b c 1 1")
        h = self.act(self.conv2(h))
        return h
 
class TinyDenoiser(nn.Module):
    def __init__(self, channels: int = 64, tdim: int = 128):
        super().__init__()
        self.time = TimeEmbedding(tdim)
        self.in_block = ConvBlock(3, channels, tdim)
        self.out = nn.Conv2d(channels, 3, 1)
 
    def forward(self, x_t: torch.Tensor, t: torch.Tensor) -> torch.Tensor:
        t_emb = self.time(t)
        h = self.in_block(x_t, t_emb)
        return self.out(h)

The point here is the parameterization. The model takes $(x_t,t)$ and predicts the noise.

Related Notes

• Convolutional Neural Networks
• Modeling, Architecture, and Data

Sources

• Ho, Jain, and Abbeel (2020), Denoising Diffusion Probabilistic Models
• Ronneberger, Fischer, and Brox (2015), U-Net
• einops documentation