RL & reasoning models

Training · 2025

The idea

Imitating experts (supervised finetuning) copies the form of good answers; reinforcement learning optimises the outcome. Give the model thousands of maths and code problems whose answers can be checked, let it sample many attempts, and reward the ones that scored. What emerges was never in any demonstration: long chains of thought - trying, checking, backtracking - because spreading the work across many tokens is what wins the reward. That is the reasoning-model moment (DeepSeek-R1, early 2025; Karpathy’s talk walks it at ~2:14-2:42).

Not the same claim as “chain-of-thought prompting.” Wei et al. (2022) found that a frozen, already-trained model reasons better if a prompt shows a few worked examples - no weights change. What this page teaches is a different, later result: train the weights with RL against checkable answers, and long reasoning chains emerge as a side effect of maximising reward, unprompted. Both are true; they are three years and one training step apart.

PaperDeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language ModelsShao, Wang, Zhu et al. · DeepSeek-AI · 2024arXiv:2402.03300 →

The RL algorithm behind this - and behind DeepSeek-R1 a year later - is GRPO (Group Relative Policy Optimization), introduced in this paper. Its one real simplification over standard RLHF’s PPO: no separate value/critic model. Sample a group of attempts at the same problem, use the group’s own average reward as the baseline, and reinforce whichever attempts beat it - cheaper, and it’s what makes sampling “thousands of attempts” per problem affordable.

For everything that can’t be checked - taste, helpfulness, tone - RLHF trains a reward model to stand in for human judgment and optimises against that. It works, briefly: the judge is a lossy simulation of people, and a strong optimiser finds inputs that score high while getting worse. Runs are stopped early for exactly this reason (~2:48-3:09).

Verdict

What it fixedPractice on problems with checkable answers: the model samples many attempts, keeps what scored, and discovers long chains of thought no human demonstration taught it.

What it still costsOnly works where an answer can be verified. Everywhere else RLHF substitutes a learned judge - and a judge can be gamed, so those runs must stop early.

The story, animated

Sample many attempts, keep what scored - then watch a learned judge get gamed.

Where it sits

Worth having first: Instruction tuning & RLHF · The next-token objective · Sampling

Source-checked against Karpathy, “Deep Dive into LLMs like ChatGPT” (2025), chapters 2:07-3:09.