Lecture 10: Regularization Tools and The Mechanics of SFT vs. RL
Nathan Lambert
Course on RLHF and post-training. Chapter 15.
How much should we let the model change to earn more reward?
Recall Lecture 9: the x-axis was KL
This lecture
Over-optimization is unavoidable – regularization is how we keep it under control.
The spine of this lecture: in RL post-training, nearly every objective points the KL in the same direction.
- The explicit KL penalty -- the workhorse, and how to measure it
- Implicit regularization -- why RL forgets less than SFT
- Other tools -- pretraining gradients, NLL terms, reward margins
What “off the rails” looks like
Without regularization, strong optimizers push language models into:
- Fluent-looking reasoning with extremely incorrect answers
- Repeated text and excessive special characters
- Language switching mid-generation – recall Lecture 5: R1-Zero mixed languages while reasoning, and labs added language-consistency rewards to stop it
Regularization is the difference between healthy RL training and these failure modes.
Part 1: The explicit KL penalty
The workhorse: a KL penalty to the reference
- KL control predates LLMs: dialogue agents (Jaques et al., 2017), then fine-tuning pretrained models (Jaques et al., 2020). Lecture 3 covered the per-token mechanics (\tilde{r}_t = -\beta\,\text{KL}_t).
- Note the direction: this penalty is a reverse KL – estimated by sampling from the policy and scoring against the reference. It punishes the policy for putting mass where the reference would not.
- “KL distance” is the optimization distance spent (colloquially – KL is not a true metric). The x-axis of the over-optimization curves is this quantity.
Measuring KL in practice
Sampling from P turns the definition into an expectation:
- Practitioners watch the KL curve during training – a very large KL usually means a bug or a broken model.
- Lower-variance estimators (k_1, k_2, k_3) came up in Q&A 2 (Schulman, 2020).
# sample from the policy
tokens = model.generate(inputs)
# score sampled tokens under both models
logprobs = log_softmax(model.forward(tokens).logits)
ref_logprobs = log_softmax(ref_model.forward(tokens).logits)
# log-probs of the tokens actually generated
token_lp = gather(logprobs, tokens)
ref_token_lp = gather(ref_logprobs, tokens)
# sequence-level difference approximates KL
kl_approx = token_lp.sum(-1) - ref_token_lp.sum(-1)
Part 2: Implicit regularization
Even with no penalty: “SFT memorizes, RL generalizes”
Controlled study: post-train on one task, evaluate under a rule shift (Chu et al., 2025).
- GeneralPoints: reach 24 from four cards; shift the face-card rule (train: J/Q/K = 10; test: 11/12/13).
- V-IRL: visual navigation; shift from absolute (north/east) to relative (left/right) directions.
On V-IRL, RL improves out-of-distribution accuracy 80.8% → 91.8%. SFT collapses it 80.8% → 1.3% – destroying spatial reasoning the base model already had.
RL-based post-training carries implicit regularization from its on-policy structure alone.
SFT and RL are the two directions of KL
Recall the frame from Lecture 7 (on-policy distillation):
Forward KL – supervised fine-tuning:
Samples come from the target (a fixed dataset). Mass-covering: wherever the target has mass and \pi_\theta \to 0, the loss blows up – the model must spread to cover everything.
Reverse KL – reinforcement learning:
Samples come from the policy itself. Mode-seeking: only penalized where it places mass, so it concentrates on high-reward modes.
Chapter 12 promised the why reverse KL is better in chapter 15 – here it is.
SFT is forward KL
Same gradients, same minimum: minimizing the SFT loss is minimizing forward KL to the data distribution.
RL is reverse KL
Start from the KL-regularized objective (the one our RL trainers optimize):
Dividing by -\beta and normalizing the reward-tilted reference with Z(x) turns the objective into a single KL – minimized exactly when \pi equals the tilted distribution. Lecture 6 walked this same path to the optimal policy (the starting point of DPO):
RL is reverse KL
Now expand the reverse KL to \pi_\star, substituting \log \pi_\star = \log \pi_{\text{ref}} - \log Z(x) + \tfrac{1}{\beta} r(x,y):
SFT minimizes forward KL to the data; RL minimizes reverse KL to the reward-tilted reference.
Which direction should forget less?
The naive read: forward KL is mass-covering, so SFT should preserve every mode – while mode-seeking RL should collapse onto one and forget the rest.
Sequential fine-tuning experiments say otherwise.
Which direction should forget less?
The opposite holds. That intuition assumes a unimodal policy – LLMs are multimodal.
RL’s razor
“Among the many high-reward solutions for a new task, on-policy methods such as RL are inherently biased toward solutions that remain closer to the original policy in KL divergence.” (Shenfeld et al., 2026)
- Forgetting tracks KL drift: \text{Forgetting} \approx f\!\left(\mathbb{E}_{x \sim \tau}\!\left[D_{\mathrm{KL}}\!\left(\pi_0 \,\|\, \pi\right)\right]\right) with R^2 = 0.96 – measured on the new task’s data. A cheap forgetting predictor.
- The ablation: on-policy data fully accounts for the difference – negative gradients have no discernible effect.
Part 3: Other regularization tools
Other regularization in the wild
- Pretraining gradients (InstructGPT): add \gamma\, \mathbb{E}_{x \sim \mathcal{D}_{\text{pretrain}}}\left[\log \pi_{\text{RL}}(x)\right] to the objective, “to fix the performance regressions on public NLP datasets” (Ouyang et al., 2022).
- NLL alongside DPO: \mathcal{L}_{\text{DPO+NLL}} = \mathcal{L}_{\text{DPO}} + \alpha\, \mathcal{L}_{\text{NLL}} keeps the chosen text high-likelihood in absolute terms, not just relatively better (Pang et al., 2024).
- Margin loss for reward models (Llama 2): -\log \sigma\!\left(r_{\theta}(y_c) - r_{\theta}(y_r) - m(y_c, y_r)\right), where the margin m comes from annotator rating deltas – the Likert scales from Lecture 8 (Touvron et al., 2023).
Most of these are scaffolding: added to stabilize one setup, simplified away in the next model generation.
Recap: keeping the policy close
Every RL-side objective in this lecture points the KL the same way: sampled from the model, scored against a target.
- Explicit -- the reverse-KL penalty to the reference (the workhorse)
- Implicit -- on-policy sampling alone biases RL toward KL-minimal solutions
- Auxiliary -- pretraining gradients, NLL terms, reward margins
Takeaways
- The KL penalty is the explicit control: a reverse KL, estimated on the policy’s own samples. Watch the KL curve during training.
- Even with no penalty, on-policy RL is implicitly regularized – SFT memorizes, RL generalizes.
- Forgetting tracks KL drift (RL’s razor), and on-policy data – not negative gradients – is why RL forgets less.
- Most other regularizers are scaffolding: they stabilize one setup and disappear in the next generation.
Where to go deeper
Regularization is where RLHF practice earns its stability – these are the references people reach for when debugging real runs.
- RL's Razor -- why on-policy forgets less.
- SFT Memorizes, RL Generalizes -- the OOD study.
- Approximating KL Divergence -- the k1/k2/k3 estimators.
- Book chapter 15.
The course so far
- Prerequisites review
- Overview (ch. 1-3)
- IFT, Reward Models & Rejection Sampling (ch. 4, 5, 9)
- RL: Motivation & Math (ch. 6)
- RL: Implementation & Practice (ch. 6)
- The Rise of Reasoning Models (ch. 7)
- Direct Preference Optimization (ch. 8)
- Synthetic Data & Modern Post-training (ch. 12)
- Preferences & Preference Data (ch. 10-11)
- Over-Optimization & RLHF’s Bad Reputation (ch. 14, app. B)
- Regularization Tools & Understanding How Post-Training Changes Models (ch. 15) – today
- Evaluation (ch. 16) – next (tentative)