Lecture 7: Synthetic Data and Modern Post-training Methods
Nathan Lambert
Course on RLHF and post-training. Chapter 12 on Synthetic Data.
Why synthetic data took over
When the first models were trained with RLHF, human data was the only way to get high-quality responses and reliable feedback. As models got better, that assumption broke down fast.
- Cheaper, faster iteration – synthetic data lowered the price of a post-training experiment, opening the field to everyone who was priced out of human-data pipelines. The time-to-collect it is also far faster, enabling the RSI arguments we hear today.
- A capability threshold – synthetic data in post-training only worked once GPT-4-class models arrived. Llama 2 and GPT-3.5-Turbo were not reliable enough to generate or supervise data; the LLM-as-a-judge ability emerged in the GPT-3.5 → GPT-4 jump.
- The center of gravity of post-training – today, leading models need synthetic data to reach the frontier. Distillation is the general word for how to transfer capabilities from a stronger model to a student.
This lecture
We survey how synthetic data has replaced or expanded much of the post-training pipeline – then derive on-policy distillation from scratch as the technical core.
Definition: Synthetic data is any training data used for an AI model that is generated in some part from another AI model. This is very broad!
- The roles of synthetic data
- General distillation with synthetic data
- The path to on-policy distillation (the technical core)
- AI feedback & Constitutional AI
- Rubrics – prompt-specific AI feedback
Lecture 7: Where it sits
- Introduction
- Key Related Works
- Training Overview
- Instruction Tuning
- Reward Models
- Reinforcement Learning
- Reasoning
- Direct Alignment
- Rejection Sampling
- What are Preferences
- Preference Data
- Synthetic Data & CAI
- Tool Use
- Over-optimization
- Regularization
- Evaluation
- Product & Character
- A. Definitions
- B. Style & Information
- C. Practical Issues
Recall: where synthetic data sits in a pipeline
The post-training pipeline is many moving parts:
- Collect / generate prompts
- Generate completions for SFT
- Collect preferences for RLHF
- Score rewards for RLVR
- Filtering, cleaning, all of the above
Synthetic data from other language models now feed every box: writing prompts from seeds, generating completions, labeling preferences, and verifying answers for RL.
We will cover a few training methods that emerged around these ideas too.
Part 1: The roles of synthetic data
What synthetic data is used for
“Synthetic data” in modern post-training spans the whole pipeline – a single model is reused for many roles:
- Generate new prompts from seed examples (Wang et al., 2023)
- Modify / expand existing prompts
- Generate completions to prompts (Beeching et al., 2024)
- Provide AI feedback to create preference data (Cui et al., 2023)
- Filter completions for quality (Li et al., 2024)
- Verify answers as rewards for RL
The balance between human and synthetic data
Synthetic data has not replaced human data uniformly across the pipeline.
- Instruction data (SFT): synthetic has largely won – distillation beats most human writers at scale.
- Preference data (RLHF): mixed – academic work shows it performs comparably, yet frontier labs treat human preference data as a competitive moat.
- Evaluation: LLM-as-a-judge scales scoring cheaply, but benchmarks and ground-truth labels still need human grounding/correlation.
Human data curation is heavily involved at determining the frontier of models and seeding initial progress, then synthetic data is used to scale it.
Around the launch of ChatGPT, human data was a central driver of progress.
Model collapse: an outdated worry
A common criticism: repeatedly training on a model’s own generations can narrow the effective training distribution (Shumailov et al., 2024).
The argument follows as: As diversity drops, rare facts and styles are underrepresented and small mistakes compound across iterations.
Model collapse: an outdated worry
But collapse is mostly a failure of unfiltered, single-model, self-training loops. In practice it is avoided by:
- mixing in real / human data,
- using diverse teachers,
- deduplication,
- strong quality filters.
Evidence suggests synthetic data can – and should – be used at scale without the catastrophic regressions of the strongest collapse story (Gerstgrasser et al., 2024) (Feng et al., 2024).
Canonical, early synthetic datasets and their scale
A few datasets defined each era: UltraFeedback (Cui et al., 2023) (kickstarted the DPO revolution), Stanford Alpaca (early chat SFT), Tülu 3 (Lambert et al., 2024) (pre RLVR SFT set), and OpenThoughts 3 (Guha et al., 2025) (general reasoning set).
| Prompts | Tokens (approx.) | |
|---|---|---|
| Stanford Alpaca (2023) | 52K | ~10M |
| Tülu 3 (2024) | ~1M | ~500M |
| OpenThoughts 3 (2025) | 1.2M | ~10B |
Two meanings of “distillation”
Technical (Knowledge Distillation): train a smaller student to match a stronger teacher’s full output distribution – soft labels, not one-hot targets (Hinton et al., 2015). More on this later in the lecture.
Colloquial (today’s usage): “train a weaker model on the outputs of a stronger model.” It’s what people mean when they say something like “Chinese models distill from GPT/Claude.”
Aside: distillation on Interconnects this year
- How much does distillation really matter for Chinese LLMs? (Feb 2026) - how much of Chinese open models’ gains actually trace to distilling frontier APIs.
- The distillation panic (May 2026) - why reacting quickly to frontier lab’s fear-based messaging on distillation could be bad for the broader AI ecosystem.
Distillation 1: Classic knowledge distillation
Distillation 2: The synthetic-data generation pipeline
Yes, this is intentionally very simple as a diagram.
Early on this was simple: train on a strong model’s API outputs.
Reasoning models made it harder – you want the reasoning traces, and closed APIs stopped returning them. Distillation shifted to:
- More aggressive jailbreaking to recover hidden traces (controversial).
- Open models more often – having the weights lets you get the full trace.
Part 2: The path to on-policy distillation (OPD)
Setup and notation
Knowledge distillation, introduced by Hinton, Vinyals & Dean (Hinton et al., 2015), in its original form uses soft labels – the full distribution over next tokens – rather than the one-hot target of next-token prediction. To apply it to autoregressive LMs, decompose the loss per token.
Let:
- s – the source prompt
- u = (u_1, \ldots, u_J) – a complete output sequence from the teacher
- \mathcal{V} – the output vocabulary (tokenizer)
- q – the teacher’s next-token distribution, p – the student’s
Notation discipline: u is a teacher trajectory; we reserve a for the student-sampled completion in the on-policy / RL notation later. This u-vs-a split is the whole point of “on-policy.”
Word-level (per-token) distillation
Standard teacher-student distillation for an LLM (Kim & Rush, 2016) – the classic Hinton soft-label idea applied at every token position:
At each position, this matches the teacher’s full next-token distribution over the entire vocabulary on a pre-existing training corpus \mathcal{V} – soft labels, not the one-hot target. The inner \sum_{k=1}^{|\mathcal{V}|} runs over every possible next token.
Matching a distribution over every token sounds expensive, but it is tractable: it is just |\mathcal{V}| probabilities per position – the same O(J\,|\mathcal{V}|) cost as ordinary cross-entropy. Matching over whole sequences is the hard part (next slide).
Read “word-level” as per-token over the tokenizer.
Sequence-level distillation
Word-level KD gives soft per-token distributions. The goal of sequence-level distillation from the paper is to be apply to apply this to data that the teacher generated, providing fresh training data/signal, and improving performance!
(WORD-KD is the baseline in the Kim & Rush paper.)
Sequence-level distillation
Sequence-level KD instead approximates the teacher’s distribution over whole sequences \mathcal{U} – an intractable sum over exponentially many sequences – by its mode: the teacher generates one high-probability output \hat{u} = \mathrm{BeamSearch}_q(s) and the student trains on it with plain NLL:
Aside: A first wave of offline-KD models were classifiers – DistilBERT (Sanh et al., 2019) and TinyBERT (Jiao et al., 2020) – pairing offline distillation with other LM advances. Not sequence distillation because these are embedding models, but building on related momentum in the area.
Finding a connection between SEQ-KD and SFT
To start, recalle the cross-entropy of a teacher q and student p – the same form as the KD losses:
Add and subtract the teacher’s own log-probabilities, \sum_z q(z)\log q(z) – does not involve p at all:
The last two sums share the weight q(z), so they fold into one log-ratio (log rules):
Finding a connection between SEQ-KD and SFT
We just showed cross-entropy decomposes into the teacher’s entropy plus a KL:
H(q) depends only on the fixed teacher, so minimizing cross-entropy is minimizing the forward KL:
So sequence-level KD reduces to SFT on the teacher’s generated text – “offline KD,” generations produced often ahead of time by a teacher model.
Exposure bias in offline KD: the train / test mismatch
Offline KD samples teacher trajectories u \sim \pi_T and matches per-token (here q = \pi_T, p = \pi_\theta):
But at test time the student rolls out under its own policy (\ell_{\mathrm{task}}(s, a) is the loss function for that test-time domain):
Since \pi_T \neq \pi_\theta, training and test prefixes come from different state distributions – exposure bias is the propensity for the student to accumulate errors (Arora et al., 2022) (Song & Zheng, 2026).
The DAgger analogy: compounding error
On-policy distillation connects to imitation learning: DAgger trains an agent on its own rollouts, with an oracle (teacher) labeling the action it should have taken (Ross et al., 2011).
Suppose the student matches the teacher within per-step error \epsilon on teacher-induced states:
The supervised imitation-learning analysis (Ross et al., 2011) shows that the expected loss accumulated along a length-L trajectory sampled from the student can scale quadratically in L (Song & Zheng, 2026):
For LLMs this is more of an analogy – token losses are distributional (KL), not 0-1 action disagreement (indicator function in the first equation on this slide returns 0 or 1).
From off-policy to on-policy
Sampling from the student rather than the teacher minimizes a lot of the distributional errors we’ve covered.
- In offline KD, a single suboptimal token can nudge the student generation slightly out-of-distribution; the model, never having seen that token in training, is more likely to err again.
- On-policy distillation iteratively samples from the student and supervises it with the teacher at its own visited states.
- Under DAgger’s interactive analysis, this drops compounding from O(\epsilon L^2) to O(\epsilon L) (Ross et al., 2011).
- MiniLLM introduced a reverse-KL objective inside a policy-gradient frame (Gu et al., 2024); concurrent work connected on-policy KD to imitation learning (Agarwal et al., 2024) (this paper is closer to the modern distillation form used today).
The on-policy distillation objective
Let a = (a_1, \ldots, a_L) be a completion sampled from the student \pi_\theta(\cdot \mid s), with token-level state s_t = (s, a_{<t}). The teacher \pi_T is fixed:
This is now in the sampling / expectation framework of Chapter 6 (policy gradients) – a natural bridge to modern RL training infrastructure that alternates generate-and-update.
Sampling from the student is also what flips the KL direction we minimize (forward D_{\mathrm{KL}}(\pi_T \,\|\, \pi_\theta) → reverse D_{\mathrm{KL}}(\pi_\theta \,\|\, \pi_T) – next slide).
Forward vs. reverse KL
Sampling completions from the student is what puts \pi_\theta on the left of the KL – which flips its direction (estimating the KL and its direction relies on which distribution you sample from):
Offline KD / SFT (forward KL) – the expectation is over the teacher, z \sim \pi_T (off-policy: a fixed teacher dataset):
Mass-covering – weighted by teacher mass: wherever the teacher has mass and \pi_\theta \to 0, the log-ratio blows up, so the student must cover everything the teacher might say.
On-policy distillation (reverse KL) – the expectation is over the student, z \sim \pi_\theta (on-policy: you sample the model you’re training):
Mode-seeking – weighted by the student’s own mass: penalized only where it puts probability the teacher dislikes, so it collapses onto the teacher’s modes. (Why reverse KL is often better: Chapter 15.)
KD as an RL advantage
Recent implementations take the KD distance directly as a reward: substitute the negative per-token reverse-KL contribution as the advantage (Lu & Thinking Machines Lab, 2025). The reverse KL at state s_t is an expectation over student-sampled tokens:
In practice you never sum over the vocabulary: the single sampled token is an unbiased estimate \hat{D}_{\mathrm{KL}} of that KL (Schulman, 2020), and the per-token advantage is its negative:
KD as an RL advantage
In practice you never sum over the vocabulary: the single sampled token is an unbiased estimate \hat{D}_{\mathrm{KL}} of that KL (Schulman, 2020), and the per-token advantage is its negative:
- Tokens more likely for the teacher → positive advantage; less likely → negative.
- Same form, opposite KL: \log \pi_T - \log \pi_\theta looks like a forward-KL term – what makes it reverse KL is sampling a_t \sim \pi_\theta (the student), not the sign.
- The teacher log-prob gap is dense, token-level feedback – potentially richer than a sparse verifiable reward or a single scalar reward-model score.
- This layers into modern RL machinery – e.g. add it alongside GRPO’s group-level normalization for more complex reward shaping.
Multi-teacher on-policy distillation (MOPD)
Use several teachers – domain specialists (math, code) or earlier checkpoints – each with a per-prompt mixture weight w_k(s) (with \sum_k w_k(s) = 1) (LLM-Core Xiaomi, 2025):
At scale, this lets a growing org divide labor: many groups train expert teachers that later distill into one final student (DeepSeek-V4-Pro (DeepSeek-AI, 2026), MiMo-V2-Flash (LLM-Core Xiaomi, 2025)).
For more, see the conversation I had with Finbarr Timbers that maps MOPD across the 2026 frontier recipes.
Self-distillation: pushing the frontier
At the absolute frontier there is no stronger model to distill from. On-Policy Self-Distillation (OPSD) sidesteps this: the teacher is the same model conditioned on privileged information – a hint the student model won’t have at inference (Zhao et al., 2026). The self-distillation gradients will teach the model that tokens after the hint were a mistake, absorbing the lesson with an OPD-style loss.
Cursor’s Composer 2.5 (from Kimi K2.5) trained this way (Team, 2026) (highly recommend watching this video!):
- A judge reviews RL trajectories against a list of common bugs.
- On a bug, it inserts a hint into the sequence – privileged information the model wouldn’t see at test time.
- The model takes a KD loss toward its own hinted continuation, learning to reach it unaided.
- A hint in token space is enough to self-correct – how to structure that signal is an active area (“privileged information”) (Penaloza et al., 2026).
On-policy distillation is becoming very popular
A resurgence of teacher-student KD has accompanied the shift toward reasoning and agentic models. Leading models trained with new forms of knowledge distillation include:
- Qwen3 (Alibaba) (Yang et al., 2025)
- MiMo-V2-Flash (Xiaomi) (LLM-Core Xiaomi, 2025) (introduced MOPD)
- DeepSeek-V4-Pro (DeepSeek-AI, 2026)
One caveat: per-token KD needs the student and teacher to share a tokenizer – unusual among post-training methods, and part of why it thrives inside a lab’s own model family.
From offline KD to self-distillation
From offline KD to self-distillation
From offline KD to self-distillation
Part 3: AI feedback & Constitutional AI
Reinforcement learning from ai feedback (RLAIF)
Soon after RLHF took off, RL from AI Feedback (RLAIF) emerged – using AIs to approximate the human-data step, starting with pairwise preferences (Lee et al., 2023) (Sharma et al., 2024) (Castricato et al., 2024).
After initial debates if this worked well, eventually it became the default. Cost was one of the obvious advantages (estimates):
- One piece of human preference data: $1 – $10+ per prompt.
- AI feedback (e.g. GPT-4o): < $0.01 per prompt.
Human labor cost is roughly flat; model price-per-performance keeps dropping. This opened RLHF experimentation to a population previously priced out.
The bias-variance tradeoff
I’ve heard a colloquial rule of thumb in early RLHF v RLAIF debates. There’s an intuitive nature to it.
Human data – high-noise, low-bias.
Harder to collect and filter, but when wrangled it gives a very reliable signal.
Synthetic preference data – low-noise, high-bias.
Easier to start with, but can carry tricky second-order effects that are systematically baked into the data.
Balancing human and AI feedback
No clear literature on the ultimate limits between human and AI preference data. Some context includes:
- Early RLAIF literature claimed AI feedback could fully replace human data – especially on chat tasks (Lee et al., 2023) (Cui et al., 2023).
- Later work is more nuanced: on broader evaluations (incl. reasoning), the best mix routes hard data points to humans while sending most to AI (Miranda et al., 2025) (Xu et al., 2025).
- No study has mapped the human/AI balance across all domains, highlighting a general limitation of open academic work to wrangle high-quality human data.
- Industry reality: human preference data is still treated as a substantial moat. Could be more from prompt distributions, implicit feedback, etc. Human data still is used.
Building specialized judge models
If you’re using substantial AI feedback or LLM-as-a-judge evaluations, the question arises as to if you should have a specialized model for that purpose. The question is – how well do they work?
- Some research is done understand the performance of LLMs in these feedback domains. Results include how LLMs are inconsistent evaluators (Wang et al., 2024) and show self-preference bias – they favor their own generations (Panickssery et al., 2024).
- Dedicated judge / critic models exist – Prometheus (Kim et al., 2024), Prometheus 2 (Kim et al., 2024), and others – but are not widely adopted in documented post-training recipes.
- Equilibirum: Frontier models are already trained hard for judging, so you rarely need your own – unless your task has private data not on the public internet.
CAI: the earliest large-scale synthetic RLHF data
Constitutional AI (CAI) – Anthropic’s post-training method for the Claude models – is the earliest documented, large-scale use of synthetic data for RLHF (Bai et al., 2022). CAI refers to a specific set of techniques for their early Claude models, and definitely has shifted substantially (though Anthropic still uses a constitution – yes, confusing).
The term RLAIF was coined in this paper as well, prompting confusion on the relation of the two.
CAI is the example that kickstarted the broader field of RLAIF. CAI ⊂ RLAIF.
CAI generates synthetic data in two ways – one for instructions, one for preferences. The well-known and more influential part of it is preferences.
Constitutional AI: The original diagram
Stage 1: critique and revise → SFT data
A constitution \mathcal{C} is a human-written set of principles (e.g. “Is the answer encouraging violence?”, “Is the answer truthful?”).
The model repeatedly samples a principle c_i \in \mathcal{C} and revises its latest output y^i to the prompt x to align with c_i:
The model is then fine-tuned on the refined dataset. These critique methods are also used broadly for data filtering and synthetic-data generation.
Stage 2: AI preference labels → RLAIF
Construct preferences by giving a feedback model:
- a prompt x,
- a subset of principles \{c_0, \ldots, c_n\},
- two completions y_0, y_1 labeled (A) / (B).
The model selects which answer is higher quality and more aligned with the principle. Then RLHF proceeds as normal – hence RLAIF.
Aside: Also linked to literature like generative reward models and progression in the LLM-as-a-judge field. See Chapter 5 / Lecture 2.
- Earlier: prompt with
The answer is:and read which of A / B has higher token probability. - Modern: a generative reward model explains its reasoning, then selects (Mahan et al., 2024) (cf. principle-guided reward models (Sun et al., 2024)).
Part 4: Rubrics – prompt-specific AI feedback
Why did rubrics become popular?
Rubrics became a popular tool for scaling RL on the long-tail of domains. They’re also used to help with domain-specific evaluations and any other place domain expertise needs to be “trained into” the models.
- A way to extend ideas from RL with verifiable rewards (Chapter 7) to tasks without clearly verifiable answers.
- Write nearly-verifiable criteria for a prompt, generate multiple answers, and RL-update toward the best ones.
- Emerged in late 2024 → 2025 as LLM judges and synthetic-data practices matured. Also likely a function of making RL more broadly accesible to frontier post-training.
- Already delivering gains in scientific reasoning and factuality (Gunjal et al., 2025) (Viswanathan et al., 2025) (Rezaei et al., 2025) (Liu et al., 2025).
A rubric example
For a prompt with no single right answer, score against tagged criteria (Liu et al., 2025). An example, abbreviated rubric is below:
Prompt: As a museum curator, suggest five obscure artifacts for a
"Mysteries of the Ancient World" exhibit ...
Rubric:
1. Includes exactly five distinct artifacts. [Hard Rule]
2. Each from a different culture and time period. [Hard Rule]
3. Brief description of each artifact's significance. [Hard Rule]
6. Communicates clearly and is well-organized. [Principle]
8. Uses engaging language that stimulates curiosity. [Principle]
[Hard Rule] = atomic, must-pass checklist items; [Principle] = softer quality criteria. The tags encode priority (numbers also work). Subcomponents of a rubric contribute to the score.
Per-prompt generation via a meta-prompt
Rubrics are generated per prompt to ensure quality and robustness to over-optimization. E.g. a per-domain base rubric, refined per-prompt by a supervising LM (Gunjal et al., 2025).
You are an expert rubric writer for science questions ...
Choose 7-20 rubric items based on question complexity.
Each item: title (2-4 words), description (category prefix +
what to look for), weight.
- Essential : critical facts; omission invalidates the answer (1-5)
- Important : key reasoning / completeness (1-5)
- Optional : nice-to-have depth or style (1-5)
- Pitfall : common mistakes to penalize (-1,-2)
Output: a JSON array of {title, description, weight}.
(Truncated – real meta-prompts are long and tuned to the training setup.)
Where rubrics are going
Rubric-based RL is a frontier of AI-feedback-driven training, expanding beyond its early uses:
- Advanced instruction-following (He et al., 2025)
- Deep research agents (Shao et al., 2025)
- Evaluating research agents (Sharma et al., 2025)
- Long-form generation with structured checklists (Ruan et al., 2025)
Very general tool!
Conclusions
Synthetic data is the single most common tool used by researchers (other than building infra) to make great models
The techniques surveyed here will continue to grow in complexity, and it’s super fun to see. When I started writing this book, it was still a struggle to set up some synthetic data workflows! Knifecuts can happen, but overall it’s a well-known workflow now.
There are many, minute open questions on how best to do this, but it often is a domain specific reflection of the technial problem at hand.
Course outline
- Introduction & Training Overview – Chapters 1-3
- IFT, Reward Models, Rejection Sampling – Chapters 4, 5, 9
- RL Theory – Chapter 6 (Part 1)
- RL Implementation & Practice – Chapter 6 (Part 2)
- Reasoning – Chapter 7
- Direct Alignment Algorithms – Chapter 8
- Synethic Data – Chapter 12
- Preferences & Preference Data – Chapters 10/11
…