Self-Instruct
Post-training & alignment · 2022
The idea
Section titled “The idea”Instruction tuning needs thousands of curated (instruction, response) pairs, normally written by paid human labelers. Self-Instruct starts from just 175 seed tasks, has the model generate new instructions, inputs and outputs itself, filters out the bad ones automatically, and fine-tunes on what’s left - within 5 points of a model trained on real human-written instruction data.
Verdict
Section titled “Verdict”What it fixedBootstrap a whole instruction-tuning dataset from a handful of seed examples - the model writes its own new tasks, inputs, and outputs, then gets fine-tuned on the filtered result.
What it still costsQuality caps out below real human-written data; a model can't teach itself something it doesn't already know.
Where it sits
Section titled “Where it sits”Worth having first: Instruction tuning & RLHF