A language model has one job: given the text so far, put a probability on every token that could come next. Everything else — answering, translating, writing code — is that one operation run over and over, each output fed back in as input. This page does the whole loop in front of you with a real byte-pair tokeniser, real co-occurrence vectors, real attention arithmetic and a real probability model, all trained in your browser on the small corpus at the bottom of the page.
Not words — tokens. A byte-pair encoder starts from single characters and repeatedly merges the commonest adjacent pair, so frequent words end up whole and rare ones stay in pieces. Drag the BPE merges slider to zero and watch every word fall apart into letters. The ▁ marks a space before the token. Click any token to follow it through the rest of the page.
Each token gets a list of numbers. Here those numbers are counts of what the token appears next to in the corpus, which is the honest ancestor of a learned embedding: words used in similar places end up with similar vectors. The map is those vectors flattened to two dimensions — so distance on it really does mean "used similarly".
Before predicting, the model lets every token look at the ones before it and decide how much each one matters. The grid is that decision: row = the token doing the looking, column = what it looks at, brightness = how much weight it gave. The upper right is dark because a token cannot see the future — that mask is the whole reason the model can be trained on ordinary text.
The output is a probability for every token in the vocabulary. Temperature reshapes that list before a sample is drawn: near zero it always takes the most likely token and the text goes flat and repetitive; above about 1.2 it starts picking tail options and the text falls apart. Top-k throws away everything outside the k best first.
Real: the byte-pair tokeniser is trained on the corpus below when the page loads, and its merges are the actual commonest pairs. The vectors are real co-occurrence statistics. The attention weights are a real scaled dot-product softmax with a real causal mask. The next-token probabilities come from a real smoothed trigram model, and the sampling maths — temperature, top-k — is exactly what a production model does.
Not real: there is no neural network here and nothing was learned by gradient descent. A trigram model looks two tokens back; a transformer looks at thousands and has hundreds of billions of weights to decide what matters. That difference is why this page produces plausible-sounding nonsense and a frontier model does not. The mechanism is the same shape. The scale is not, and the scale is the entire story.
A model can only ever reproduce the statistics of what it was shown. This one was shown a few hundred sentences about the topics in these labs — so it will happily tell you the sea floor is made of basalt, and has no idea who you are.