A diffusion model is trained on a job that sounds useless: take a photograph, mix in a measured amount of static, and guess which static was added. Repeat a few hundred million times and you own a machine that can start from pure noise and subtract its way to a picture. Here you can drive a real noise schedule in both directions, then train a small diffusion model β forward pass and backpropagation, written out in this page β until it sculpts a shape out of random points.
The reverse sweep replays the same schedule backwards using the noise we already know we added. Nothing here is predicting anything β it is the destination the next section's network has to reach without being told the answer.
Pictures are too big to train on in a browser tab, so this works in two dimensions instead, where the whole idea fits on screen. The target is a cloud of points in a shape. A small neural network β two hidden layers, 4,994 weights, trained by backpropagation written out below this page's controls β learns one thing only: given a noisy point and the timestep it came from, which way was the noise pushing? Then it starts from pure Gaussian points and undoes its own answer, over and over.
The training data. The network never sees this directly β only noisy versions of it.
Sampled with the weights straight after random initialisation. A shapeless blob, as it should be.
The same sampler, the same starting noise, after training. Only the weights changed.
Loss 1.00 is what you get for answering βno noise at allβ every time, because the noise has variance 1. Anything below that is real knowledge about the shape.
Each denoising step is a straight-line guess at a curved path. Take few, long steps and you overshoot; take many short ones and you stay on the curve. Same trained network, same starting noise, only the number of steps changes β which is exactly why a picture generated at 4 steps looks smeared and the same seed at 50 steps looks clean.
Nothing above knows about words. Turning a denoiser into something you can talk to takes three extra pieces: a way to turn text into numbers that live in the same space as pictures, a way to let those numbers reach into every denoising step, and a dial for how hard to insist. The diagrams here are drawn to explain the wiring β they are not running a text model.
CLIP trains two encoders at once β one for images, one for captions β and pushes the matching pair together while pushing every other pair in the batch apart. OpenAI trained it on 400 million imageβtext pairs with 32,768 pairs in each batch, so each correct pairing was scored against 32,767 wrong ones at a time. Afterwards a sentence and a photograph are points in one shared space, and "close together" means "these are about the same thing".
Stable Diffusion turns your prompt into 77 token vectors of 768 numbers each. Inside the U-Net, every position in the latent image asks a question of all 77 of them and pulls back a weighted blend. At 64Γ64 latent resolution that is 4,096 positions Γ 77 tokens = 315,392 attention weights per head, per layer, recomputed at every one of the 50 steps. The line weights drawn below are illustrative, not measured.
The model is run twice at every step β once with your prompt and once with an empty one β and the difference between the two predictions is the part that is because of the prompt. Guidance scale multiplies that difference before it is applied. The arithmetic below is exactly the formula that runs inside the model; the arrows are two-dimensional stand-ins for a prediction that really has 16,384 numbers in it.
Denoising 512Γ512 pixels directly means carrying 786,432 numbers through every step. Stable Diffusion does not. A separately trained autoencoder squeezes the image into a 64Γ64 grid of 4 channels β 16,384 numbers, a 48-fold reduction β and the entire diffusion process happens in there. Only at the very end does the decoder turn the finished latent back into pixels. That single choice is why the model runs on a laptop graphics card instead of a data centre.