# How LLMs Work, Explained Visually: Tokens, Attention, Training, Temperature | Koso Learn

> A visual guide to how large language models actually work: next-token prediction, tokens, embeddings, attention, the transformer stack, pretraining and RLHF, temperature, the context window, and why models hallucinate. Every concept drawn as a diagram.

[](/)

[Services](/#services)[AI OS](/ai-os)[Training](/training)[Education](/education)[Agents](/agents)[Learn](/learn)

…

[Book a call ](/book)

1. [Home](/)
2. [Learn](/learn)
3. How LLMs Work

Chapter 01 · Foundations

# The language engine.

ChatGPT, Claude and Gemini are all wrapped around the same core machine: a language engine. This chapter opens it up. What it does, what it's made of, how it was trained, and where its strange failure modes come from. No math required; every idea is drawn.

9 sections · about 25 minutes · a diagram in every section

In this chapter

1. [01One job: predict the next token](#prediction)
2. [02Tokens: how text becomes numbers](#tokens)
3. [03Embeddings: meaning becomes a map](#embeddings)
4. [04Attention: every token reads every other](#attention)
5. [05The transformer stack](#stack)
6. [06Training: from internet to assistant](#training)
7. [07Temperature: the dial on the dice](#sampling)
8. [08The context window: working memory](#context)
9. [09Hallucination: confident, not correct](#hallucination)

Chapters

[01 · How LLMs Work](/learn/how-llms-work)[02 · Prompt Engineering](/learn/prompt-engineering)[03 · AI Agents](/learn/ai-agents)

In this chapter
1. [01One job: predict the next token](#prediction)
2. [02Tokens: how text becomes numbers](#tokens)
3. [03Embeddings: meaning becomes a map](#embeddings)
4. [04Attention: every token reads every other](#attention)
5. [05The transformer stack](#stack)
6. [06Training: from internet to assistant](#training)
7. [07Temperature: the dial on the dice](#sampling)
8. [08The context window: working memory](#context)
9. [09Hallucination: confident, not correct](#hallucination)

01.1

## One job: predict the next token

Strip away the chat interface and an LLM does exactly one thing: **given some text, score every possible next token by how likely it is to come next**. It doesn't “answer questions”, it continues text, one token at a time, feeding each choice back in and predicting again. Run that loop a few hundred times and you get a paragraph.

ThecapitalofFranceisthe prompt so farthe enginea probability machine“what token comes next?”, every option scoredParis87%the5%Lyon3%located2%…3%append the winner (“Paris”), feed the longer text back in, and predict again, one token at a time

Swipe the diagram sideways to see it all. The whole trick, in one picture. The engine never really answers a question, it repeatedly picks a likely next token, appends it, and goes again. Every essay, poem and code file comes out of this loop.

Everything surprising about these models lives inside that loop. Trained on enough of humanity's writing, “predict what comes next” quietly forces the model to absorb grammar, facts, logic and style, because you can't predict the next word of a physics proof or a Python file without modeling some of the physics or the Python. The intelligence is a side effect of the prediction task.

Why this matters in practice

The engine always completes the pattern you start. A sloppy, vague prompt is a pattern too, and it gets completed sloppily. That single fact is the foundation of all of chapter 2.

01.2

## Tokens: how text becomes numbers

The engine never sees letters or words. A **tokenizer** first chops your text into pieces from a fixed vocabulary (roughly 100,000 entries) and hands the model a list of integers. Common words are one token; rare words get assembled from sub-word fragments.

your texta stringtokenizerfixed vocabulary, \~100k entriesKoso42817▁builds9114▁un517believ31456able481▁AI15837▁agents21967what the model actually receives: a list of integersrare words are built from pieces, since the model has never seen “unbelievable” as one unit

Swipe the diagram sideways to see it all. “Koso builds unbelievable AI agents” becomes seven tokens. Common words get one token each; the rare word is assembled from sub-word pieces (un + believ + able). The ▁ marks a leading space, since spaces are part of tokens too.

Tokens explain a family of famous quirks. Models are billed and limited **per token** (a rough rule: one token ≈ ¾ of an English word). Arithmetic is shaky partly because “1234” may be one token while “1235” is two, since numbers aren't digits to the model. And counting the letter **r** in “strawberry” is genuinely hard when the model sees the token, not the letters inside it.

01.3

## Embeddings: meaning becomes a map

Each token id then becomes an **embedding**: a long list of numbers, thousands of dimensions, that acts as coordinates in a space of meaning. The model learns these coordinates during training, and it learns them so that **similar meanings land close together**.

the map of meaning (2 of \~4,096 dimensions)dogcatpuppykittenpets, close togetherinvoiceledgertaxfinance, far awaykingmanqueenwomanking − man + woman ≈ queen“dog” =0.12\-0.870.441.30\-0.050.61⋮one point = one list of numbers

Swipe the diagram sideways to see it all. Left: a 2-D shadow of the real thing (models use thousands of dimensions). Similar meanings sit close together, and _directions_ mean something: the man→woman arrow is the same arrow as king→queen. Right: what one token really is inside the model, a long list of numbers.

This is the model's real native language: not words but geometry. “Similar” becomes “nearby”, and analogies become directions you can travel. It's also a tool you can use directly. Semantic search, recommendations and the retrieval half of RAG all work by embedding your documents and finding what's **near** the question, even when no words overlap.

01.4

## Attention: every token reads every other

A word's meaning depends on its neighbours. “Bank” means one thing next to “river” and another next to “loan”. **Attention** is the mechanism that handles this: every token looks at every other token in the window, scores how relevant each one is to itself, and blends in information from the winners.

Therobotpickeduptheballbecauseitwaslight“it”: which earlier token do I mean?attention weights for “it”ball0.62robot0.24light0.08rest0.06

Swipe the diagram sideways to see it all. To decide what “it” means, the model lets that token _look at_ every earlier token and weigh how relevant each one is. “Ball” wins, so “it was light” gets read as the ball being light. Swap the ending to “it was strong” and the weights shift to “robot”. That re-weighing is attention, and it happens for every token, in every layer, in parallel.

Attention is the breakthrough that made modern LLMs possible (the 2017 paper that introduced the architecture is literally titled **“Attention Is All You Need”**). Because every token attends to every other in parallel, models can hold long-range connections: a pronoun linked to its noun forty words back, a bug linked to the variable declared two screens up, all while running fast on GPUs.

01.5

## The transformer stack

Now assemble the machine. A **transformer** is a stack of identical blocks, each doing two things: attention (tokens exchange information with each other) and a feed-forward layer (each token gets processed independently, and this is where most of the stored “knowledge” lives). Stack that block 30 to 100+ times and put a prediction head on the end.

tokensembedids → vectorsattentionfeed-forwardmix, then thinkattentionfeed-forwardmix, then think⋯attentionfeed-forwardmix, then thinkthe same block, repeated ×30–100, is where the billions of weights liveprediction headvector → scoresnexttokenodds

Swipe the diagram sideways to see it all. The architecture is almost boring. The same block, attention (tokens exchange information) followed by a feed-forward layer (each token gets processed on its own), gets stamped 30 to 100+ times. Meaning gets refined layer by layer: early layers handle grammar, later layers hold concepts. All of a model's “knowledge” lives in the billions of weights inside these blocks.

Meaning is refined layer by layer: early layers resolve spelling and grammar, middle layers track entities and relationships, late layers hold abstract concepts and task intent. A model's parameter count (the “70B” in a model name) is mostly the weights inside these repeated blocks. More blocks and wider layers mean more capacity, more cost, and more latency for every single token generated.

01.6

## Training: from internet to assistant

Where do the weights come from? Three stages, each on the same engine:

* →**Pretraining.** Show the model trillions of tokens of text and endlessly grade its next-token guesses. Months of GPU time, the expensive part, and the source of its raw capability.
* →**Fine-tuning.** Continue training on curated examples of the behaviour you want: question in, helpful answer out.
* →**RLHF** (reinforcement learning from human feedback). Humans rank candidate outputs, and the model is nudged toward the ones people prefer.

?startrandom weights1 · pretraininglearns the languagethe internettrillions of tokens · months of GPUs2 · fine-tuninglearns the jobcurated examplesgood question → good answer pairs3 · RLHFlearns the mannershuman rankings“this answer beats that one”assistantthe model you chat withsame engine throughout, each stage only adjusts the weights

Swipe the diagram sideways to see it all. Three passes over the same engine. Pretraining teaches the _language_ (predict the next token across trillions of words). Fine-tuning teaches the _job_ (here is how an assistant answers). RLHF teaches the _manners_ (humans rank outputs; the model is nudged toward the ones people prefer). Chat models feel helpful because of stages two and three. The raw pretrained model would just continue your text.

This pipeline explains model personality. The raw pretrained model is a pure continuation machine. Ask it a question and it might continue with **more questions**, because that's what forum pages look like.

The helpfulness, the refusals, the tone: those come from stages two and three. It also explains the knowledge cutoff. The model knows nothing after its training data ends, unless you put newer facts into its context window yourself.

01.7

## Temperature: the dial on the dice

The engine outputs probabilities; something still has to **pick**. Always taking the top token makes output deterministic but flat and repetitive. So we sample instead, and **temperature** is the dial on that dice roll.

temperature 0.1Paris96%Lyon3%the1%same answer every run, good for facts, code, extractiontemperature 1.0Paris42%Lyon20%the13%lovely9%Narnia6%top-p 0.9 cuts the tail below this linevaried and surprising, good for ideas, drafts, names

Swipe the diagram sideways to see it all. Same engine, same probabilities. The dial only changes how the dice are thrown. Temperature near 0 sharpens the distribution: repeatable, safe, dull. Higher temperature flattens it: varied and creative, but the tail (where the nonsense lives) gets real chances. Top-p is the safety scissors: sample only from the smallest set of tokens covering, say, 90% of the probability, and cut the rest.

This is why the same prompt gives different answers on different runs, and it's a setting you control in every API:

* →**Low (0–0.3).** Extraction, classification, code, anything you'll parse with software.
* →**Higher (0.7–1.0).** Brainstorming and drafts, where you want the model to surprise you.

Top-p caps how far into the tail the dice can reach. A pipeline that mixes both is common: research or extraction steps run cold, writing steps run a bit warmer.

01.8

## The context window: working memory

The engine has no memory between calls. What it has is a **context window**: the maximum number of tokens it can attend over in one go. Your system prompt, the whole conversation so far, and any pasted files all compete for the same window.

old turnspasted docsystem promptturn 12turn 13turn 14your questionthe context window: all the model can “see”fell out, gone for the model8k ≈ short story · 128k ≈ novel · 1M ≈ small library, and attention cost grows with length

Swipe the diagram sideways to see it all. The window is everything the model can see _right now_: system prompt, conversation, pasted files, all of it counted in tokens. Nothing outside it exists for the model. When a chat “forgets” your instructions from an hour ago, nothing broke: those tokens slid out of the window. Rough sizes: 8k tokens ≈ a short story, 128k ≈ a novel, 1M ≈ a small library.

Every “the AI forgot my instructions” story is this picture. Chat apps replay the conversation into the window on every turn; when it overflows, something old gets dropped or summarized, and the model genuinely no longer knows it. Long context isn't free either. Attention over more tokens costs more compute, and models attend most reliably to the start and end of the window, so where you put things matters.

01.9

## Hallucination: confident, not correct

Ask the engine about something it has no data on, and it does the only thing it knows how to do: **continue the pattern plausibly**. Legal citations that look exactly like legal citations. API methods that should exist but don't. It's delivered with the same fluent confidence as real facts, because fluency is what got optimized, not truth.

the question“Cite the 1985 Koso ruling”lane 1 · ungroundedengineno sources in viewfluent answernames a case, a judge, a pageconfident, inventedthe ruling neverexistedlane 2 · groundedretrieve firstsearches your document storeengine + sourcesfacts sit in the windowcited, or “not found”every claim pointsat a real source

Swipe the diagram sideways to see it all. The engine's job is “continue this text plausibly”, not “say true things”. So when it has no relevant facts, it completes the pattern anyway, and fluently. That's a hallucination. The fix isn't scolding the model; it's changing the plumbing: retrieve real documents into the context window first, ask it to cite them, and let “not found” be an acceptable answer. That plumbing is RAG, and chapter 3 builds it in full.

Hallucination is an engineering problem, so it has engineering fixes:

* →Put the relevant facts **into the window** before asking (retrieval).
* →Require citations, so every claim is checkable.
* →Make “I couldn't find it” an explicitly acceptable output.
* →Run temperature low for factual work.

None of this makes the engine truthful. It makes the system around the engine trustworthy, which is the mindset shift this whole guide builds toward: you don't fix the model, you design the system.

[← Back toThe Learn indexAll chapters, and how to work through them.](/learn)[Next chapterPrompt EngineeringThe techniques that reliably change model output: prompt anatomy, few-shot examples, chain of thought, self-consistency, structured output, the system-prompt stack, grounding and a debugging loop.](/learn/prompt-engineering)

A custom AI development agency. Bespoke agents, copilots and products engineered for teams that want a moat, not a subscription.

Ideanab Private Ltd is a company registered in England and Wales, no. [16344048](https://find-and-update.company-information.service.gov.uk/company/16344048). Registered office: 124 City Road, London, EC1V 2NX, United Kingdom.

### Company

* [Services](/#services)
* [Custom AI Agency](/custom-ai-agency)
* [AI Product Development](/ai-product-development)
* [AI OS](/ai-os)
* [AI Engineering](/ai-engineering-company)
* [AI Consulting](/ai-consulting-company)
* [AI Consulting Services](/artificial-intelligence-consulting-services)
* [Training](/training)
* [Education](/education)
* [Free AI Agents](/agents)
* [Learn AI](/learn)
* [Fractional CAIO](/caio)
* [About Us](/about)
* [Careers](/careers)

### Contact

* [Contact us](/contact)
* [Book a call](/book)
* [LinkedIn](https://www.linkedin.com/company/koso)
* [Instagram](https://www.instagram.com/okaashish)

### Legal

* [Terms of Service](/terms)
* [Privacy Policy](/privacy)
* [Cookie Policy](/cookies)
* [Acceptable Use Policy](/acceptable-use)

© 2026 Koso, a brand of Ideanab Private Ltd. All rights reserved.

[Terms of Service](/terms)[Privacy Policy](/privacy)[Cookie Policy](/cookies)[Acceptable Use Policy](/acceptable-use)

Crafted with care · Built to compound