Base models
Training
The idea
Section titled “The idea”What falls out of pretraining is not an assistant. It is an internet-document simulator: hand it any opening and it continues in the register it learned - a poem begets stanzas, a forum flame begets flames. Ask it a question and it may answer, continue with three more questions, or write the rest of the exam (Karpathy probes a Llama base model live at ~0:42-0:59).
The base model demoed here belongs to this lineage. Its actual 2023 argument: a 13B LLaMA, trained longer on more public data, beat the 175B GPT-3 outright - open data beats proprietary scale, not just “smaller is fine if you compensate.”
Everything assistant-like is fitted on afterwards. Prompt a base model into a chat format and it will role-play the transcript - which is the whole trick the next two pages build into a product.
Two live behaviours make the point unforgettable. Heavily repeated documents come back nearly verbatim - Wikipedia is oversampled during training, so a famous article can be recited, memorisation showing through (ch. 0:52-0:54). And primed with text dated past its training cutoff, the model doesn’t stop - it confidently continues a future it never saw, because continuing documents is the only thing it does (ch. 0:54-0:55).
Try it
Section titled “Try it”Where it sits
Section titled “Where it sits”Worth having first: Pretraining · Sampling
Unlocks: Conversations become tokens · Instruction tuning & RLHF
Source-checked against Karpathy, “Deep Dive into LLMs like ChatGPT” (2025), ch. 0:42-0:59.