A short introduction to RLHF and post-training focused on language models.
Reinforcement learning from human feedback (RLHF) has become an important technical and storytelling tool to deploy the latest machine learning systems. In this book, we hope to give a gentle introduction to the core methods for people with some level of quantitative background. The book starts with the origins of RLHF – both in recent literature and in a convergence of disparate fields of science in economics, philosophy, and optimal control. We then set the stage with definitions, problem formulation, data collection, and other common math used in the literature. The core of the book details every optimization stage in using RLHF, from starting with instruction tuning to training a reward model and finally all of rejection sampling, reinforcement learning, and direct alignment algorithms. The book concludes with advanced topics – understudied research questions in synthetic data and evaluation – and open questions for the field.
Last built: 02 March 2026
February 2026: v2 content: direct alignment chapter, new diagrams, RL cheatsheet, appendices, search bar, Kindle support, editor fixes.
January 2026: Major chapter reorganization to match Manning book structure; code examples library; old URLs redirect to new locations.
December 2025: Working on v2 of the book based on editors feedback! Do check back for updates!
November 2025: Manning preorder available.
July 2025: Add tool use chapter (see PR)
June 2025: v1.1. Lots of RLVR/reasoning improvements (see PR)
April 2025: Finish v0; overoptimization, open questions, etc.; evaluation section; RLHF x Product research, improving website, reasoning section.
March 2025: Improving policy gradient section; finish DPO, major cleaning; start DPO chapter, improve intro.
February 2025: Improve SEO, add IFT chapter; RM additions, preference data, policy gradient finalization; PPO and GAE; added changelog, revamped introduction.
I would like to thank the following people who helped me directly with this project: Costa Huang, (and of course Claude). Indirect shout-outs go to Ross Taylor, Hamish Ivison, John Schulman, Valentina Pyatkin, Daniel Han, Shane Gu, Joanne Jang, LJ Miranda, and others in my RL sphere.
Additionally, thank you to the contributors on GitHub who helped improve this project.
If you found this useful for your research, please cite it!
@book{rlhf2026lambert,
author = {Nathan Lambert},
title = {Reinforcement Learning from Human Feedback},
year = {2026},
publisher = {Online},
url = {https://rlhfbook.com}
}