How a Language Model Works: Tokens, Next-Word Prediction, Embeddings and Attention
A tiny language model you can poke: see text split into tokens (words, characters or BPE pieces), how a model scores the next word, what temperature, top-k and top-p do, how words become vectors, and how attention mixes words. Paste your own text. Private: nothing is uploaded.
Use your own text
Text becomes tokens
A model never sees letters or words, only numbered pieces called tokens. Type anything and choose how to cut it up.
Predict the next word
The model counted which words follow which in the practice text. Type a start, and it scores every word it knows as the next one.
Words become points: embeddings
Words that appear in similar company get similar vectors. Here every common word of the practice text is squeezed into 8 numbers (from how often words occur near each other) and drawn using its first two. Click a word to see its closest neighbours.
Attention: how words look at each other
Each word asks a question (Q), every word offers a label (K), and the better they match, the more of that word's content (V) is mixed in. The result is a table: row = the word that is looking, column = the word it looks at.
How a language model works, with a tiny one you can poke
Large language models predict the next token. Everything else (answering, summarising, coding) comes from doing that billions of times with a model that has learned patterns from an enormous amount of text. This page builds the smallest honest version of each ingredient so you can see it work.
Questions
What is a token?
A piece of text with a number. Whole words are simple but cannot handle new words; single characters never fail but make very long sequences. Byte-pair encoding (BPE) starts from characters and repeatedly glues together the most frequent neighbouring pair, so common words become one token and rare words are built from pieces. The "word pieces" view learns its pieces from the practice text.
What do temperature, top-k and top-p do?
The model gives every word a probability. Temperature sharpens (below 1) or flattens (above 1) those probabilities before one is drawn at random: 0 always takes the most likely word, high values get adventurous and eventually nonsense. Top-k only allows the k most likely words, and top-p only the smallest group of words whose probabilities add up to p.
Why does this toy repeat itself and wander?
It only remembers the last one to three words and has read a few hundred. Real models look at thousands of earlier tokens, have read much of the internet, and replace the counting table with a neural network that generalises, but sampling works the same way.
What is an embedding?
A list of numbers that stands for a word, where similar words get similar lists. Here they come from counting which words appear near each other (positive pointwise mutual information, then keeping the 8 strongest directions with SVD). Chatbots learn theirs while training.
What is attention?
The mixing step of a transformer: softmax(Q·Kᵀ/√d)·V. The weights here use random, untrained matrices, so the pattern is not meaningful language: it shows the mechanism. Choose Similarity of word vectors to see attention when the weights simply compare real word vectors. A chatbot hides "look backwards only" so a word cannot peek at the future.
Is it private?
Yes. Everything is computed in your browser and nothing is sent anywhere. The three practice texts were written for this page.