Instruction tuning & RLHF

Training · 2022

The idea

A pretrained model predicts; it does not converse. Instruction tuning and RLHF reshape it around what people actually want from it.

The mechanism this page’s title promises: SFT is the same next-token objective as pretraining, pointed at a curated set of human-written conversations - InstructGPT (2022) published the recipe (ch. 1:00-1:12; how those conversations become a token stream is its own page).

PaperTraining language models to follow instructions with human feedbackOuyang, Wu, Jiang et al. · OpenAI · 2022arXiv:2203.02155 →

The paper’s own headline number is worth having, because it’s exactly the kind of concrete, checkable figure this site otherwise insists on: in human evaluations, a 1.3B-parameter InstructGPT was preferred over the 175B GPT-3 it was built from - 100x fewer parameters, still preferred, because SFT + RLHF changed what the model was trying to do, not how big it was.

The limits caveat that matters most: an SFT answer is an imitation of what a contracted labeler, following an instruction document, would have written. Talking to the model is talking to a simulation of that labeler - not to research, and not to the internet (ch. 1:17-1:18).

Verdict

What it fixedA raw next-token predictor becomes an assistant that follows instructions.

What it still costsTuning toward preferred answers can shave off capability and variety - alignment is not free.

The story, animated

Same predictor, same objective - only the training diet changes, and that’s what turns it into an assistant.

Where it sits

Worth having first: Pretraining

Unlocks: What it can’t remember

Source-checked against Karpathy, “Deep Dive into LLMs like ChatGPT” (2025).