Conversations become tokens

Training

The idea

There is no separate chat machinery. A dialogue is flattened into one token stream: special tokens mark where a user turn starts, where it ends, where the assistant begins - and the model is trained, then run, as plain next-token prediction over that document (Karpathy shows the encoding at ~1:01-1:20).

That reframe settles two things at once: why an assistant can be built purely from finetuning data (it’s just more documents, in costume), and why the protocol tokens matter - they are the walls between voices.

The story, animated

Two speech bubbles collapse into one token stream - watch the walls between voices form.

Where it sits

Worth having first: Tokens · Base models

Unlocks: Instruction tuning & RLHF

Source-checked against Karpathy, “Deep Dive into LLMs like ChatGPT” (2025), ch. 1:01-1:20.