A short introduction to RLHF and post-training focused on language models.
Nathan Lambert
Course
A full course accompanying the book with added resources and other lectures I've given.
The slide decks are built by me with the help of the latest frontier models, in my Markdown-based slide tool, Colloquium. PRs are welcome. Please extend, modify, and use these slides for your courses if you are an educator.
This course is roughly aimed at early AI PhD or master's students, but it is designed to be accessible to anyone willing to put in the work.
You do not need prior reinforcement learning or language modeling background to start. A motivated learner who studies hard — leaning on today's top LLMs as a tutor to unpack unfamiliar math, code, and jargon — can follow the entire course. I encourage you to go down rabbit holes, skip or reorder videos, and chase what excites you.
If you prefer the traditional coursework path, the usual background is the basics of language modeling / NLP plus basic machine learning (e.g. an intro to AI course and an intro to ML course).
Conversation 2: From academic research to frontier practice (w/ Scott Geng)
We discuss what it takes to land a well-grounded academic result into a near-frontier model. Scott and I worked together on DPO for Olmo 3, and how building models is much more than the idea, it's making it fit in a more complex puzzle.
Other lectures I've given closely related to the book content. The newest ones are beginning to include material not covered in the course. I hope to keep this up to date over the next few years!
RL Systems, Score Centering, and Modern RL
Cornell CS 4789 guest lecture · Nov 2026 · Chapter 6 · Async RL, training-inference mismatch, and score centering
Various presentations over my last few years working in post-training, grouped by corresponding chapter. These are talks I gave while building the book, and may be more research-oriented, but serve as great additional context to the course.
Outside material that I've personally used for going deeper on reinforcement learning and language models. The books in particular are wonderful complements.
If you found this useful for your research, please cite it!
For the web and arXiv version:
@misc{lambert2025reinforcementlearninghumanfeedback,
title = {Reinforcement Learning from Human Feedback},
author = {Nathan Lambert},
year = {2025},
eprint = {2504.12501},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2504.12501}
}
For the Manning edition:
@book{lambert2026reinforcement,
author = {Nathan Lambert},
title = {Reinforcement Learning from Human Feedback: Alignment and post-training of {LLMs}},
year = {2026},
publisher = {Manning Publications},
isbn = {9781633434301},
url = {https://www.manning.com/books/reinforcement-learning-from-human-feedback}
}