ORPO
Post-training & alignment · 2024
The idea
Section titled “The idea”DPO still needs a separately instruction-tuned model to start from, and a frozen reference copy to stay close to during training. ORPO folds preference alignment into ordinary supervised fine-tuning itself - one model, one training run, a small odds-ratio penalty away from the rejected response.
Verdict
Section titled “Verdict”What it fixedOne odds-ratio loss does SFT and preference alignment together, in a single pass, with no reference model at all.
What it still costsNewer and less battle-tested than DPO; the odds-ratio penalty is a specific design choice, not a universal optimum.
Where it sits
Section titled “Where it sits”Worth having first: DPO