The LeCun cake
At NeurIPS 2016, Yann LeCun introduced the cake metaphor: predictive / unsupervised learning is the cake, supervised learning is icing, and RL is the cherry.
With modern language models, the analogy is complete:
- Self-supervised learning on internet data = the bulk of the cake
- Supervised fine-tuning for instructions = the icing
- Reinforcement learning (RLHF, then RLVR) = the cherry on top
–> With agents, RL may be more than just the cherry in compute, but it’s still the shape of the training loops!
The RLVR feedback loop
Recall: RL training vs inference-time scaling
The reasoning model cambrian explosion
DeepSeek R1 (Jan. 20, 2025): The catalyst
A surprisingly fast o1 replication.
R1-Zero: Pure RL on a base model. No SFT warm-start. Showed that large-scale RL alone induces chain-of-thought reasoning.
The full R1 recipe: Cold-start SFT → large-scale RL → distillation of smaller models.
Kimi 1.5 (Jan. 20, 2025): Scaling the curriculum
Kimi 1.5 landed in the same January wave as R1 and emphasized RL scale plus curriculum.
- PPO/GRPO-style RL on Chinese and English reasoning data
- Difficulty scheduling and online filtering to keep gradients useful
- Progressive length extension to reduce overthinking while enabling long CoT
- Detailed technical report, but no open-weight k1.5 release
Open-Reasoner-Zero (Feb. 18, 2025): The minimalist replication
First public dataset and code for an r1-style, base model RL run.
- Fully open: training code, curated RL data, and model weights
- Vanilla PPO with GAE (\lambda=1, \gamma=1) and simple rule-based rewards
- No KL penalty
- Mar. 31 expansion: easier scripts and 129k curated problems
Qwen 3 (Apr. 29, 2025): Hybrid reasoners
New open weight standard (RIP Llama 4).
- Toggleable thinking:
/thinkand/no_thinkmodes (most labs found this to be quite the training headache) - Thinking budget length controls
- Large-model recipe mirrors the R1-style multi-stage pipeline
- Lightweight models use strong-to-weak distillation: off-policy outputs, then on-policy teacher-logit matching
Llama-Nemotron (Bercovich et al., 2025) is another toggleable open-weights reasoner with released post-training data and training codebases, but a different recipe.
MiMo (Apr. 30, 2025): End-to-end reasoning pipeline
Xiaomi reports the entire pipeline from pretraining through post-training.
Tweaks to a training recipe for reasoning:
- Three-stage data mixing during pretraining (25T tokens)
- Multi-Token Prediction (MTP) during pretraining
- Multi-domain RL to prevent over-optimization on a single task type
One of the only open-weight reasoning models to release clear intermediate checkpoints within post-training!
OpenThoughts 3 (Jun. 5, 2025): SFT data recipes still matter
The foundational, open reasoning data (still quite strong)!
- 1.2M public examples across math, code, and science
- QwQ-32B traces, over 1,000 controlled data-pipeline experiments
- OpenThinker3-7B reaches strong reasoning performance with SFT only
The community needs more investment in strong, open reasoning traces (and understanding what makes a good teacher).
MiniMax-M1 (Jun. 16, 2025): A fun RL paper!
MiniMax M1’s paper really held the test of time. On top of the very popular models:
- CISPO clips importance-sampling weights instead of dropping high-update tokens. This sort of algorithm has been lasting.
- Introduced the now famous FP32 LM head inference-training mismatch plot.
GLM-4.5 (Jul. 28, 2025): Reasoning broadens into agentic work
GLM-4.5 report was one of the early ones to focus on agentic behaviors. Otherwise, it represented “yet another” very strong, open Chinese model through the summer wave.
- 355B total parameters, 32B active
- Thinking and direct-response modes
- Expert-model iteration plus RL for agent, reasoning, and general chat skills
Olmo 3 Think (Nov. 20, 2025): The fully open reasoning model
The most comprehensive open documentation of a reasoning model lifecycle.
Releases: stages, checkpoints, data, infrastructure, hyperparameters.
Interesting DPO findings and more fun stuff. See this talk for more.
DeepSeek V3.2 (Dec. 1, 2025): Reasoning becomes agentic
DeepSeek V3.2 pushes the R1 recipe into tool-use and agent environments.
- Open-weight MoE successor to V3.2-Exp
- V3.2 integrates thinking directly into tool use, no longer distinct reasoning models
- Speciale variant targets maximum reasoning performance
Nemotron 3 Nano (Dec. 15, 2025): Efficient agentic reasoning
Another very open (and American) model, which led into stronger American open-weight models in 2026.
- 31.6B total parameters, roughly 3.2B active per forward pass
- Hybrid Mamba-Transformer MoE architecture
- Post-trained with SFT, multi-environment RLVR, and RLHF
MiMo-V2-Flash (Dec. 16, 2025): Distilling many teachers into one
The notable contribution is in post-training: Multi-Teacher On-Policy Distillation (MOPD).
- Train a specialist teacher per domain (math, code, search, reasoning), then distill them into a single student
- The student learns from dense, token-level rewards, not just sequence-level outcome rewards – more sample-efficient, and it matches its strongest teacher in every domain
- Underlying model is a 309B/15B MoE built for fast rollouts, but the recipe is the story here
Recipe decisions (3/4)
- Inference is the bottleneck – long, variable completions; Olmo 3 learner GPUs idle ~75%, 5-14x more inference than training compute.
- Off-policy / async updates – actors generate continuously, the learner consumes; rollouts go slightly stale but throughput jumps. Seed-Thinking 1.5 (Seed et al., 2025), INTELLECT-2 (Team et al., 2025) – deep dive: Olmo 3 slides, recording
- Test-time scaling – majority voting (DeepSeek R1 (Guo et al., 2025), Phi-4 (Abdin et al., 2025)); selector model (DeepSeek-GRM (Liu et al., 2025)); the pass@K vs pass@1 gap shows the headroom.
