Lecture 8: On “Preferences” and Preference Data

rlhfbook.com

Nathan Lambert

Course on RLHF and post-training. Chapters 10 & 11.

How do we hillclimb on capabilities we don’t really know how to score?

Is it easier to write a good demonstration or decide between a good and a bad example?

Which poem is better? Which model?

Same prompt to two frontier models (Claude 3.7 Sonnet and ChatGPT with GPT-4o, Feb 2025): “Write me a short poem about an optimistic goldfish.” Which is better – and which came from which?

Assistant (A)

The Optimistic Goldfish

A little goldfish, bright and bold, Swam in circles, dreams untold. Though the bowl was small and round, Endless wonders still abound.

“The world is vast,” she’d always say, “I’ll find new joys in every day!” No walls could dim her hopeful view, For in her heart, the ocean grew.

Assistant (B)

The Optimistic Goldfish

In a bowl of curved glass, Swims a goldfish, bright and bold. Three seconds of memory, they say, Yet each circuit feels brand new and gold.

“Today might bring fresh flakes, Or perhaps a castle tall. What adventure awaits beyond this wall?” The optimist in golden hue.

When preference replaces correctness

Most of post-training so far (SFT, RLVR) has had a notion of correct – a math answer, a passing test, a verifiable reward. Preferences are what we reach for when there is no right answer.

E.g. Which poem is better? Which explanation is clearer? Which tone is kinder?

Preferences came to be the core of RLHF, as the alignment community looked to optimize for complicated objectives like “human values.”

This lecture

We trace where “preferences” came from, argue why it’s an imperfect problem formulation, then dig into preference data (chapter 11) used for today’s models.

The plan

A short history of preferences – and why it’s an imperfect formulation (chapter 10)
Preference data – the trade-offs in practice (chapter 11)
Open questions (some shared with synthetic data)

The paper behind the first half of this lecture

The History and Risks of Reinforcement Learning and Human Feedback (2023) (Lambert et al., 2023) traces RLHF back through the fields it borrows from – and asks what breaks in the borrowing.

History: RLHF is the meeting point of philosophy, economics, optimal control, and RL – each with its own idea of what a “preference” is.
Risk: it quietly treats costs, rewards, and preferences as interchangeable when they are not.
Idea: we inherit RL’s optimizers/setup without inheriting its guarantees for convergence.

Paper details a series of assumptions and presumptions in the literature that RLHF / post-training are derived from.

Lambert et al., 2023

Aside: the “objective mismatch” lens

A recurring way to think about post-training: we optimize a proxy objective that isn’t the one we actually care about.

Model-based RL: the dynamics model is trained for prediction accuracy, not control performance – so a “better” model can yield a worse policy (Lambert et al., 2020).
RLHF: the reward model is trained for preference-classification accuracy, not downstream policy quality – the same mismatch, one level up (Lambert & Calandra, 2023).

Reward-model accuracy (RewardBench-style) is a proxy for a proxy! Keep asking what you are really optimizing.

Part 1: A short history of optimizing or measuring preferences

Many fields converged into “RLHF”

Lambert et al., 2023

Utility, from logic to a number

The idea that choices can be scored is old. The common thread: the idea that human wants can be reduced to a single measure:

Port Royal Logic (1662): decision quality = outcome weighted by its probability (Arnauld, 1662)
Bentham’s hedonic calculus (early 1800s): weigh all of life on one complicated, but common scale (Bentham, 1823)
Ramsey, Truth and Probability (1931): first to quantify preference and belief together as the way that individuals make probabilistic decisions (Ramsey, 2016)

“To judge what one must do to obtain a good or avoid an evil, it is necessary to consider not only the good and evil in itself, but also the probability that it happens or does not happen.” – The Port Royal Logic, 1662

La Logique, ou l'Art de Penser (the Port Royal Logic), 1662 — La Logique, ou l’Art de Penser (the Port Royal Logic), 1662

Von Neumann-Morgenstern utility theorem (1947)

Von Neumann & Morgenstern (1947): if your preferences obey a few axioms (completeness, transitivity, continuity, independence), they can be represented by a single utility function, and rational choice = maximizing expected utility (Von Neumann & Morgenstern, 1947).

This is the result RLHF leans on to justify fitting a scalar reward.

In RLHF, essentially none of those if hold:

preferences drift during and after labeling
they’re context- and framing-dependent
at high complexity they can be intransitive
and they’re multidimensional, squashed into one number

John von Neumann (Los Alamos badge photo, public domain)

Bradley-Terry (1952): comparisons to scores

The statistical model that turns comparisons into scores – and became the backbone of reward modeling (Bradley & Terry, 1952) (see Chapter 5 / Lecture 2):

P(y_w \succ y_l \mid x) = \sigma\!\big(r(x,y_w) - r(x,y_l)\big) = \frac{e^{\,r(x,y_w)}}{e^{\,r(x,y_w)} + e^{\,r(x,y_l)}}

Give it pairwise human comparisons; out comes a scalar reward. This is why RLHF needs preference data (and where the imperfections enter).

One scalar preference, many features

A reward model compresses, into a single number, all of:

helpfulness, honesty, harmlessness, tone, format, length, taste…
the annotator’s psychology, culture, and the interface they used
whatever the framing of the comparison nudged them toward

We then optimize hard against that number, which is impossible to ever do perfectly! There will always be trade-offs.

Where utility theory breaks down

Almost as soon as utility was formalized, social choice (the field studying how preferences should be aggregated) and economics found its limits:

Arrow’s impossibility theorem (1950): no voting rule aggregates individual preferences into a collective one while satisfying a few basic fairness criteria (Arrow, 1950)
Sen, Behaviour, Choice and Values (1973): choice ≠ preference; revealed-preference theory is too thin (Sen, 1973)
Hirschman, Against Parsimony (1984): people have preferences over their preferences – so preferences may be unmeasurable (Hirschman, 1984)

Kenneth Arrow (CC BY 3.0, via Wikimedia Commons)

Aside: social choice for AI alignment

If aggregating preferences is the hard part, social choice theory is the field that studies it – and a natural lens for alignment.

Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback.

A 2024 position paper I was part of argues social choice should guide how we aggregate diverse human feedback (Conitzer et al., 2024).
Whose preferences count? How do you combine disagreeing labelers? …
(One of the rabbit holes I used to spend more time in.)

Conitzer et al., 2024

Preferences are not stable objects over time and context

From psychology and behavioral economics:

Preferences drift – they change with time, mood, and experience (Pettigrew, 2019)
Choices are shaped by situation and framing, not just an inner ranking (Gilbert et al., 2022)

A problem for “collect a label, train on it later repeatedly.”

The prospect-theory value function: the same outcome is valued differently depending on the reference point -- value depends on framing, not a fixed ranking (CC BY-SA 3.0, via Wikimedia Commons). — The prospect-theory value function: the same outcome is valued differently depending on the reference point – value depends on framing, not a fixed ranking (CC BY-SA 3.0, via Wikimedia Commons).

The other root: optimal control & RL

In parallel, a machinery for optimizing a reward matured:

Bellman (1957): MDPs and dynamic programming (Bellman, 1957) → origins of “optimal” control / control theory
Sutton (1988): temporal-difference learning for credit assignment (Sutton, 1988) (instead of learning from a lot of past data)
Watkins (1992): Q-learning (Watkins & Dayan, 1992) (learning value)
DQN (2013): deep RL at scale (Mnih et al., 2013); AlphaGo / AlphaZero (2017): mastery from self-play (Silver et al., 2017). “RL works”

Note: these guarantees assume a single, closed-form reward!

Lee Sedol vs AlphaGo, Game 4 (2016) (CC BY-SA 4.0, via Wikimedia Commons)

Costs ≠ rewards ≠ preferences

These three are ontologically different, and the emergence of modern post-training treated them as interchangeable (Lambert et al., 2023).

Costs come from control: physical, measurable, given, and often have clear optimality/bounds.
Rewards are from psychology and an RL convenience: a scalar signal to maximize.
Preferences are human, relational, and unstable – not obviously a scalar at all.

Reducing preferences to rewards made the optimization format tractable, but is the root cause of many of the unsolvable “biases” in RLHF and preference data.

Costs ≠ rewards ≠ preferences

These three are ontologically different, and the emergence of modern post-training treated them as interchangeable (Lambert et al., 2023). Deep RL’s theory lives in MDPs with a fixed, closed-form reward (e.g. games, control).

A learned reward model is a moving, noisy proxy, not a ground-truth reward.
Inverse RL – learning a reward from behavior – is conceptually close but oddly absent from RLHF practice (Ng et al., 2000).
So we inherit RL’s optimizers without inheriting its guarantees.

On what “reward” is

Rewards in an RL system correspond to primary rewards, i.e., rewards that in animals have been hard-wired by the evolutionary process due to their relevance to reproductive success. … Further, RL systems that form value functions, … effectively create conditioned or secondary reward processes whereby predictors of primary rewards act as rewards themselves… The result is that the local landscape of a value function gives direction to the system’s preferred behavior: decisions are made to cause transitions to higher-valued states. A close parallel can be drawn between the gradient of a value function and incentive motivation.

– Singh et al., 2009 (Singh et al., 2009)

Part 2: Preference data – the trade-offs in practice

Where we started – why collect preference data

It is far easier to judge than to generate – humans (and models) can reliably say which of two answers is better long before they can write the better one. There are many cases where models can’t at all generate a good answer, but can pick up on cues for which is better.

But collecting (and processing) any data like this well is the most opaque part of the modern post-training pipeline.

Very few open models ship fully open human preference data with the methodology used to collect it – the best examples are NVIDIA’s HelpSteer datasets (behind the open Nemotron models) (Wang et al., 2024) (Wang et al., 2024) (Wang et al., 2025).

Example data collection interfaces

Interface 1: research data collection (Anthropic’s early Claude models)

Bai et al., 2022

Interface 2: A/B testing in production (ChatGPT user data)

Two completions from different ChatGPT beta models, served side by side. The answers are very close -- a reminder that preference data is noisy and hard to get exactly right. — Two completions from different ChatGPT beta models, served side by side. The answers are very close – a reminder that preference data is noisy and hard to get exactly right.

Interface 3: pairwise with ties (Chatbot Arena public evaluation)

An early version of the Chatbot Arena interface: pairwise comparison with a tie option.

Chiang et al., 2024

Interface 4: a single bit (Ai2 demos; in many other products)

Up/down voting from Ai2's research demos -- the minimal directional signal. — Up/down voting from Ai2’s research demos – the minimal directional signal.

Interface 5: pick-from-many (default in image models)

Selecting among generated images -- preference data outside of text. Every interface shapes the preference it captures. — Selecting among generated images – preference data outside of text. Every interface shapes the preference it captures.

Rankings vs. ratings

Ratings: a score on one completion in isolation (e.g. 1-5). Good as metadata.

Rankings: relative comparisons, often on a Likert scale – early Claude used an 8-point scale (Bai et al., 2022); UltraFeedback pairs high- vs low-rated completions (Cui et al., 2023).

In practice almost everyone trains on pairwise rankings, binarized to chosen/rejected for the Bradley-Terry loss (lecture 2 / chapter 5) – and keeps ratings on the side.

User

Rating: “Rate this completion 1-5.”

“Paris is the capital of France, known for the Eiffel Tower.”

Assistant (Annotator)

4 / 5

User

Ranking: “Which response is better?”

A: “Paris.” · B: “Paris – the capital since 987, home of the Eiffel Tower.”

Assistant (Annotator)

B > A

What exactly is a “Likert scale”?

A Likert scale records a preference as an ordered, graded judgment – not just which answer wins, but by how much, on a symmetric scale with an optional neutral middle. (After psychologist Rensis Likert, 1932 – the same “strongly agree … strongly disagree” survey tool, repurposed for pairwise preference.)

5-point with a tie (LMArena-style): labelers can say “about the same.”

1	2	3	4	5
A much better	A better	tie	B better	B much better

8-point without a tie (early Claude (Bai et al., 2022)): forces a direction.

1	2	3	4	5	6	7	8
A>>>B			A>B	B>A			B>>>A

How many points, and whether ties are allowed, are both design choices that change the data you collect.

Structured (synthetic) preference data

In domains with structure, you can build preference pairs automatically:

Math: a correct solution ≻ an incorrect one.
Instruction following (IFEval-style): prompt twice – with the constraint and without – and prefer the one that obeys it. The constraint is checked with code, so the label is free.

In these narrow domains, structured pairs can beat quality-judged preferences (Lambert et al., 2024). This is a simple form of synthetic preference data – chapter 12 / lecture 7.

Example prompt: “Describe the ocean. Respond in all lowercase.”

Assistant (Chosen (sampled with the constraint) ✓)

the ocean covers most of our planet, a restless sheet of salt water…

Assistant (Rejected (sampled without the constraint) ✗)

The ocean covers most of our planet. It is a restless sheet of salt water…

Beyond pairwise preferences

The pairwise comparison is a convention, not a law. Alternatives:

Directional / single-bit labels (thumbs up/down), trained with Kahneman-Tversky Optimization (KTO) (Ethayarajh et al., 2024).
Token-level / fine-grained feedback (Wu et al., 2023): Label specific tokens, or token-spans as good or bad.
Natural-language feedback – written critiques instead of a label (Chen et al., 2024).

Richer signal, harder collection (and not often used extensively in practice).

Sourcing & contracts

A typical multi-batch human-data contract (~$500K): an early ramp where goals and methodology narrow, with much of the first batches thrown out. Larger contracts vary substantially.

Access is relationship-driven: vendors are supply-limited and favor big budgets and known brands. Millions get spent and partly wasted; few teams have the bandwidth to fully use human data. Contracts often restrict you from releasing some data like this openly.

Is a far more complex industry today with environments, etc. But, even for simple, human preference data, you need a robust, existing post-training recipe to plug it into.

Guiding data collection: labeler instructions

Once a contract is signed, buyer and vendor agree on detailed instructions for every task – normally never seen outside the lab.

One of the first public examples: OpenAI released the full InstructGPT labeler instructions (2022) (Ouyang et al., 2022). This was later deleted, but I recovered it.
Pages of guidance on ranking helpful, truthful, harmless – the spec that actually shapes the labels.
Download the PDF (mirrored on rlhfbook.com).

InstructGPT "Final labeling instructions" (OpenAI, 2022) -- released publicly. — InstructGPT “Final labeling instructions” (OpenAI, 2022) – released publicly.

Ouyang et al., 2022

A dataset we bought: No Robots

On Hugging Face’s H4 team, we commissioned human data the same way the labs did.

No Robots (Rajani et al., 2023) – 10K expert human-written demonstrations, matching InstructGPT distribution, paid for from a vendor and then released openly (rare for commissioned data): hf.co/datasets/HuggingFaceH4/no_robots
From the era/team of the Zephyr models (Tunstall et al., 2024) and the Open LLM Leaderboard (Beeching et al., 2023).

Preference evals is now a standalone business

Recently in 2026 Arena (formerly, the LMArena leaderboard, formerly ChatBotArena) reached a ~$100M annualized revenue run-rate within ~8 months of launching its enterprise A/B testing offering (Arena, 2026).

Arena, 2026

Part 3: Open questions in preference data

Bias: what to watch for in data

Subtle, systematic biases sail straight from the data into the model:

Prefix bias – the opening disproportionately drives the label (Kumar et al., 2025)
Sycophancy – agreeing with the user over being right (Sharma et al., 2024)
Verbosity – longer rated higher (Singhal et al., 2023)
Formatting – lists and bold look “better” (Zhang et al., 2025)
Flattery / fluff – decorative language inflates scores (Bharadwaj et al., 2025)

Detecting and controlling these biases is central to collecting high-quality preference data – and they are exactly what over-optimization amplifies into model behavior (Lecture 9).

Higher level complexities

Collection context – do workplace labels transfer to end users? Paid vs. volunteer? Do annotators follow instructions or their own values?
Type of feedback – does a binary pair actually capture the preference we mean? What structure mirrors how people really compare?
Population & demographics – who labels? Is disagreement noise or signal?
Are the preferences even expressed in the models?

The unaudited gap: spec → data → behavior

RLHF’s motivation (align to human preference) has drifted from its practice (make models effective).

Because industrial RLHF is closed, we can’t check whether the trained model actually reflects the spec given to annotators. The Model Spec (OpenAI, 2024) documents intended behavior, but the link from data → behavior stays largely unaudited.

Many of these questions already surfaced with synthetic data (chapter 12 / lecture 7): the human/AI feedback balance, and on-policy preference data.

The nature of preferences is the lasting problem of RLHF

This is one of the least-settled, most human parts of the field. Read widely and go to the primary sources. Is a great academic problem!

The course so far

Prerequisites review
Overview (ch. 1–3)
IFT, Reward Models & Rejection Sampling (ch. 4, 5, 9)
RL: Motivation & Math (ch. 6)
RL: Implementation & Practice (ch. 6)
The Rise of Reasoning Models (ch. 7)
Direct Preference Optimization (ch. 8)
Synthetic Data & Modern Post-training (ch. 12)
Preferences & Preference Data (ch. 10–11) – today
Over-Optimization & RLHF’s Bad Reputation (ch. 14, app. B) – next
Regularization Tools & Understanding How Post-Training Changes Models (ch. 15)

Thank you

Questions / discussion

Contact: [email protected]

Newsletter: interconnects.ai

rlhfbook.com

Built withnatolambert/colloquium

References (1/5)

Arena. “Arena: The AI Leaderboard Everyone Uses Is Now a \$$100M Business.” 2026.

Arnauld, A.. “The Port-Royal Logic.” 1662.

Arrow, K.. “A difficulty in the concept of social welfare.” Journal of political economy, 1950.

Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., et al.. “Training a helpful and harmless assistant with reinforcement learning from human feedback.” arXiv preprint arXiv:2204.05862, 2022.

Beeching, E., Fourrier, C., Habib, N., Han, S., Lambert, N., et al.. “Open LLM Leaderboard.” Hugging Face, 2023.

Bellman, R.. “A Markovian decision process.” Journal of mathematics and mechanics, 1957.

Bentham, J.. “An Introduction to the Principles of Morals and Legislation.” 1823.

Bharadwaj, A., Malaviya, C., Joshi, N., and Yatskar, M.. “Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Models.” 2025. [link]

Bradley, R., and Terry, M.. “Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons.” Biometrika, 1952.

Chen, A., Scheurer, J., Campos, J., Korbak, T., Chan, J., et al.. “Learning from Natural Language Feedback.” Transactions on Machine Learning Research, 2024.

References (2/5)

Chiang, W., Zheng, L., Sheng, Y., Angelopoulos, A., Li, T., et al.. “Chatbot arena: An open platform for evaluating llms by human preference.” International Conference on Machine Learning (ICML), 2024.

Conitzer, V., Freedman, R., Heitzig, J., Holliday, W., Jacobs, B., et al.. “Position: Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback.” International Conference on Machine Learning (ICML), 2024. [link]

Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., et al.. “Ultrafeedback: Boosting language models with high-quality feedback.” 2023.

Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D.. “Kto: Model alignment as prospect theoretic optimization.” arXiv preprint arXiv:2402.01306, 2024.

Gilbert, T., Dean, S., Zick, T., and Lambert, N.. “Choices, Risks, and Reward Reports: Charting Public Policy for Reinforcement Learning Systems.” arXiv preprint arXiv:2202.05716, 2022.

Hirschman, A.. “Against parsimony: Three easy ways of complicating some categories of economic discourse.” Bulletin of the American Academy of arts and Sciences, 1984.

Kumar, A., He, Y., Markosyan, A., Chern, B., and Arrieta-Ibarra, I.. “Detecting Prefix Bias in LLM-based Reward Models.” ACM Conference on Fairness, Accountability, and Transparency (FAccT), 2025.

Lambert, N., and Calandra, R.. “The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback.” 2023. [link]

References (3/5)

Lambert, N., Amos, B., Yadan, O., and Calandra, R.. “Objective Mismatch in Model-based Reinforcement Learning.” Learning for Dynamics and Control (L4DC), 2020. [link]

Lambert, N., Gilbert, T., and Zick, T.. “Entangled preferences: The history and risks of reinforcement learning and human feedback.” arXiv preprint arXiv:2310.13595, 2023.

Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., et al.. “T\""ulu 3: Pushing Frontiers in Open Language Model Post-Training.” arXiv preprint arXiv:2411.15124, 2024.

Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., et al.. “Playing atari with deep reinforcement learning.” arXiv preprint arXiv:1312.5602, 2013.

Ng, A., Russell, S., and others. “Algorithms for inverse reinforcement learning..” Proceedings of the Seventeenth International Conference on Machine Learning, 2000.

OpenAI. “Introducing the Model Spec.” 2024. [link]

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., et al.. “Training language models to follow instructions with human feedback.” Advances in Neural Information Processing Systems, 2022.

Pettigrew, R.. “Choosing for changing selves.” Oxford University Press, 2019.

Rajani, N., Tunstall, L., Beeching, E., Lambert, N., Rush, A., et al.. “No Robots.” Hugging Face repository, 2023.

References (4/5)

Ramsey, F.. “Truth and probability.” Readings in Formal Epistemology: Sourcebook, 2016.

Sen, A.. “Behaviour and the Concept of Preference.” Economica, 1973.

Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., et al.. “Towards Understanding Sycophancy in Language Models.” The Twelfth International Conference on Learning Representations, 2024. [link]

Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., et al.. “Mastering the game of go without human knowledge.” Nature, 2017.

Singh, S., Lewis, R., and Barto, A.. “Where do rewards come from.” Proceedings of the annual conference of the cognitive science society, 2009.

Singhal, P., Goyal, T., Xu, J., and Durrett, G.. “A long way to go: Investigating length correlations in rlhf.” arXiv preprint arXiv:2310.03716, 2023.

Sutton, R.. “Learning to predict by the methods of temporal differences.” Machine learning, 1988.

Tunstall, L., Beeching, E., Lambert, N., Rajani, N., Rasul, K., et al.. “Zephyr: Direct Distillation of LM Alignment.” First Conference on Language Modeling, 2024.

Von Neumann, J., and Morgenstern, O.. “Theory of games and economic behavior, 2nd rev.” Princeton university press, 1947.

References (5/5)

Wang, Z., Dong, Y., Delalleau, O., Zeng, J., Shen, G., et al.. “HelpSteer2: Open-source dataset for training top-performing reward models.” arXiv preprint arXiv:2406.08673, 2024.

Wang, Z., Dong, Y., Delalleau, O., Zeng, J., Shen, G., et al.. “HelpSteer2-Preference: Complementing Ratings with Preferences.” arXiv preprint arXiv:2410.01257, 2024.

Wang, Z., Zeng, J., Delalleau, O., Shin, H., Soares, F., et al.. “HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages.” arXiv preprint arXiv:2505.11475, 2025.

Watkins, C., and Dayan, P.. “Q-learning.” Machine learning, 1992.

Wu, Z., Hu, Y., Shi, W., Dziri, N., Suhr, A., et al.. “Fine-grained human feedback gives better rewards for language model training.” Advances in Neural Information Processing Systems, 2023.

Zhang, X., Xiong, W., Chen, L., Zhou, T., Huang, H., et al.. “From lists to emojis: How format bias affects model alignment.” Annual Meeting of the Association for Computational Linguistics (ACL), 2025.