The network below is real. Its forward pass and its backpropagation are written out in plain JavaScript on this page — no library, no pre-trained weights, nothing precomputed. It starts out knowing nothing at all. You hand it points in two colours, and it adjusts a few hundred numbers, over and over, until it stops getting them wrong.
Both curves start near 0.69 — the loss of a network guessing at random.
Every line is one weight. Thickness is magnitude, colour is sign. Nothing else about the network changes during training — no rules are written, no examples are stored.
Real: the weights, the forward pass, the gradients and the training loop are all computed in your browser, in roughly a hundred lines of JavaScript in this page's source. No answers are stored anywhere. Reset the weights and it will find a slightly different boundary, because it starts from different random numbers.
A frontier model differs in scale, not in kind. It runs the same two steps — a forward pass, then a nudge to every weight against the gradient — across something like 10¹¹ to 10¹² weights rather than the — here, for months, on tens of thousands of chips. Published estimates put a GPT-4-class training run near 2 × 10²⁵ arithmetic operations; this page has done 0 so far. What twenty neurons can show you is the mechanism. It cannot tell you how a large language model behaves, and you should not let anyone claim otherwise.
Strip the network down to a single weight and the whole method fits on one curve. The height is how wrong the network is. The slope under the ball tells you which way is downhill and how steep it is. Take a step against the slope, sized by the learning rate, and repeat. That is all backpropagation does — it just works out the slope for several hundred weights at once. Drag the ball anywhere on the curve, then step.
This curve is a made-up function of one variable, chosen to have two valleys. It is not the loss surface of the network above — that one has hundreds of dimensions and cannot be drawn.