# RLHF Book

> A free online book and course on reinforcement learning from human feedback (RLHF), reward models, preference tuning, RLVR, and post-training language models.

The RLHF Book by Nathan Lambert explains how modern language models are post-trained, from instruction tuning and preference data to reward modeling, policy optimization, direct alignment, reasoning training, evaluation, and product behavior.

Use the chapter links for canonical web pages. Use `llms-full.txt` for a single concatenated Markdown context file generated from the public chapter sources.

## Core Resources

- [Full-text LLM context](https://rlhfbook.com/llms-full.txt): Concatenated Markdown for all public book chapters, excluding the references build utility page.
- [Book homepage](https://rlhfbook.com/): The canonical web version of the RLHF Book.
- [Course](https://rlhfbook.com/course): Free lectures and course material on RLHF and post-training.
- [Model library](https://rlhfbook.com/library): Model completion comparisons across supervised finetuning, RLHF, DPO, and related post-training stages.
- [RL cheatsheet](https://rlhfbook.com/rl-cheatsheet): One-page reference for PPO, GRPO, RLOO, REINFORCE, DPO, and related RLHF methods.

## Chapters

- [Introduction](https://rlhfbook.com/c/01-introduction): A first-principles introduction to RLHF, what it changes in language models, and how it became part of modern post-training.
- [A Tiny History of RLHF](https://rlhfbook.com/c/02-related-works): Key papers and historical milestones behind RLHF, reward modeling, preference learning, and post-training language models.
- [Training Overview](https://rlhfbook.com/c/03-training-overview): A high-level map of modern post-training recipes, including instruction tuning, RLHF, RLVR, and direct alignment methods.
- [Instruction Fine-Tuning](https://rlhfbook.com/c/04-instruction-tuning): How instruction tuning turns base language models into usable assistants and sets up later RLHF and post-training stages.
- [Reward Modeling](https://rlhfbook.com/c/05-reward-models): How reward models are trained from preference data and used as the learned objective in RLHF post-training pipelines.
- [Reinforcement Learning](https://rlhfbook.com/c/06-policy-gradients): Policy gradient methods for RLHF and LLM post-training, including PPO, REINFORCE, RLOO, GRPO, and implementation details.
- [Reasoning and Inference-Time Scaling](https://rlhfbook.com/c/07-reasoning): Reasoning training and inference-time scaling in post-training, including RLVR and thinking models.
- [Direct-Alignment Algorithms](https://rlhfbook.com/c/08-direct-alignment): Direct alignment algorithms such as DPO that optimize preference objectives without an explicit reward model or RL loop.
- [Rejection Sampling](https://rlhfbook.com/c/09-rejection-sampling): Rejection sampling and best-of-n methods for improving post-trained language models with reward or preference signals.
- [The Nature of Preferences](https://rlhfbook.com/c/10-preferences): A conceptual chapter on human preferences, preference learning, and why RLHF can optimize hard-to-specify behaviors.
- [Preference Data](https://rlhfbook.com/c/11-preference-data): How preference datasets are designed, collected, filtered, and used for RLHF and post-training language models.
- [Synthetic Data & Distillation](https://rlhfbook.com/c/12-synthetic-data): Synthetic data, distillation, Constitutional AI, and AI feedback methods used throughout modern post-training.
- [Tool Use and Function Calling](https://rlhfbook.com/c/13-tools): Tool use and function calling as post-training targets for building more capable language model products and agents.
- [Over-Optimization](https://rlhfbook.com/c/14-over-optimization): How RLHF over-optimization and reward hacking happen, and why they matter for post-training and alignment.
- [Regularization](https://rlhfbook.com/c/15-regularization): Regularization methods that keep RLHF and post-training updates useful without degrading the base model.
- [Evaluation](https://rlhfbook.com/c/16-evaluation): Evaluation methods for measuring RLHF, post-training, reward models, open-ended generation, and model behavior.
- [Crafting Model Character and Products](https://rlhfbook.com/c/17-product): How RLHF and post-training shape model character, product behavior, UX, and deployed AI systems.
- [Appendix A: Definitions](https://rlhfbook.com/c/appendix-a-definitions): Definitions and background for RLHF, reinforcement learning, language models, and post-training terminology.
- [Appendix B: Beyond "Just Style"](https://rlhfbook.com/c/appendix-b-style): Style, calibration, and information presentation issues that RLHF and post-training shape in language models.
- [Appendix C: Practical Issues](https://rlhfbook.com/c/appendix-c-practical): Practical advice for running RLHF and post-training experiments, from engineering constraints to debugging.

## Optional

- [PDF](https://rlhfbook.com/book.pdf): Printable book build.
- [EPUB](https://rlhfbook.com/book.epub): Ebook build.
- [GitHub source](https://github.com/natolambert/rlhf-book): Source repository for the book and course.
- [ArXiv paper](https://arxiv.org/abs/2504.12501): Paper version of the RLHF Book.
- [Manning book page](https://www.manning.com/books/reinforcement-learning-from-human-feedback): Publisher page for Reinforcement Learning from Human Feedback.