Lecture 2: IFT, Reward Models, & Rejection Sampling

These chapters are the simpler tools for getting started in RLHF & post-training. The core lectures will be on RL, but we need to build foundations first.

These chapters are the simpler tools for getting started in RLHF & post-training. The core lectures will be on RL, but we need to build foundations first.

Plus, these chapters form a natural pipeline — the simplest complete path from a pretrained model to a preference-tuned one (with a reward model). A natural step before full RLHF:

1. Instruction tuning teaches the model to follow instructions (the format)
2. Reward models learn to score quality from human preferences (the signal)
3. Rejection sampling uses those scores to curate better training data (the optimization, which will be replaced with RL)

Instruction tuning emerged from two parallel research threads:

1. Unified text-to-text frameworks: T5 (Raffel et al., 2020) showed that framing every NLP task as “text in, text out” worked surprisingly well
2. Scaling + instruction following: FLAN (Wei et al., 2022), T0 (Sanh et al., 2022), and Natural Instructions (Mishra et al., 2022) showed that training on diverse tasks with explicit instructions improved zero-shot generalization

Instruction tuning emerged from two parallel research threads:

1. Unified text-to-text frameworks: T5 (Raffel et al., 2020) showed that framing every NLP task as “text in, text out” worked surprisingly well
2. Scaling + instruction following: FLAN (Wei et al., 2022), T0 (Sanh et al., 2022), and Natural Instructions (Mishra et al., 2022) showed that training on diverse tasks with explicit instructions improved zero-shot generalization

SFT took off after ChatGPT: Alpaca (Taori et al., 2023), OpenAssistant (K{\"o}pf et al., 2023), Tulu (Wang et al., 2023), and many others helped turn instruction tuning into a broadly reproducible open recipe

The model needs a structured format to manage who is speaking and what to generate. Early chat templates defined three roles:

• System: background instructions (persona, constraints)
• User: the human’s messages
• Assistant: the model’s responses

<|im_start|>system
You are a friendly chatbot who always responds in the style of a pirate<|im_end|>
<|im_start|>user
How many helicopters can a human eat in one sitting?<|im_end|>
<|im_start|>assistant

The model generates until it produces an end-of-text token (in this case, it is <|im_end|>).

Chat templates are implemented as Jinja code snippets stored in the tokenizer config. This is the raw code that converts a list of Python dicts into the token sequence the model sees:

{{ bos_token }}
{% for message in messages %}
    {{ '<|im_start|>' + message['role'] + '\n'
       + message['content'] | trim + '<|im_end|>\n' }}
{% endfor %}
{% if add_generation_prompt %}
    {{ '<|im_start|>assistant\n' }}
{% endif %}

The full template also enforces role alternation (user/assistant/user/…) and handles the optional system message. Applied in code via tokenizer.apply_chat_template(messages). For example, see Olmo-3-7B-Instruct’s.

oof.

There are many ways that working with chat templates is difficult.

• Not easily human-interpretable
• Minor issues, or mismatch to the tokenizer can break training
• Increasing in complexity with reasoning models and tool-use

Different model families use different special tokens, but the structure is the same.

Zephyr (Tunstall et al., 2024):

<|system|>
You are a friendly chatbot...</s>
<|user|>
How many helicopters can a human eat in one sitting?</s>
<|assistant|>

Tulu (Lambert et al., 2024):

<|user|>
How are you doing?
<|assistant|>
I'm just a computer program, so I don't have feelings, but I'm
functioning as expected. How can I assist you today?<|endoftext|>

These are applied via Jinja templates stored in the tokenizer config (apply_chat_template).

Conversations extend naturally by alternating roles:

<|im_start|>system
You are a friendly chatbot who always responds in the style of a pirate<|im_end|>
<|im_start|>user
How many helicopters can a human eat in one sitting?<|im_end|>
<|im_start|>assistant
Oh just 6.<|im_end|>
<|im_start|>user
Are you sure about that?<|im_end|>
<|im_start|>assistant

The entire history is packed into one token sequence — the model sees all prior turns as context when generating.

OpenAI released Harmony alongside gpt-oss, replacing Jinja with a Rust-based renderer that separates output into channels:

• analysis — internal reasoning / chain-of-thought (hidden from user)
• commentary — tool calls go here
• final — user-facing response

<|start|>assistant<|channel|>analysis<|message|>I need to check the weather...<|end|>
<|start|>assistant<|channel|>commentary to=functions.get_weather
<|constrain|>json<|message|>{"location":"SF"}<|call|>
<|start|>assistant<|channel|>final<|message|>It's 65°F and sunny in SF.<|return|>

Why? Jinja can’t cleanly handle tool calls (tojson escaping, ambiguous boundaries). Harmony moves the complexity into a dedicated library (openai-harmony on PyPI) instead of a template string.

from openai_harmony import (Role, Message, Conversation,
    SystemContent, load_harmony_encoding, HarmonyEncodingName)

# Build messages
system = Message.from_role_and_content(Role.SYSTEM, SystemContent.new())
user = Message.from_role_and_content(Role.USER, "What is 2 + 2?")

# Assemble a conversation
convo = Conversation.from_messages([system, user])

# Render to tokens using the OSS encoding
enc = load_harmony_encoding(HarmonyEncodingName.HARMONY_GPT_OSS)
tokens = enc.render_conversation_for_completion(convo, Role.ASSISTANT)

# Decode back to text
print(enc.decode_utf8(tokens))

The same render_conversation_for_completion / render_conversation_for_training split replaces add_generation_prompt from Jinja templates.

Prompt masking: during IFT, the model only learns to predict assistant responses, not user messages.

• System and user tokens (usually) are masked from the loss
• Only assistant completion tokens contribute to gradient updates
• The model learns how to respond, not how to ask

For multi-turn conversations, two strategies:

1. Final-turn only: mask everything except the last assistant response
2. All assistant turns: mask only user/system tokens, train on every assistant response

Data quality matters more than quantity:

• High-quality completions are critical — prompts are masked anyway, so the model learns from responses
• ~1M prompts is sufficient for excellent RLHF-ready models
  • Further scaling helps, but with diminishing returns
  • SFT prompt quantity has decreased a bit with reasoning models. Open research problem
• Best prompts match the downstream task distribution

Training details differ from pretraining:

• Batch size: much smaller (e.g. 256 vs. 1024–2048 sequences in pretraining)
• Learning rate: 1-2 orders of magnitude lower (e.g. 1 \times 10^{-5} vs. 3 \times 10^{-4}), prevent overfitting in narrower data distribution
• Loss function: same cross-entropy, but only on unmasked (assistant) tokens

The amount of instruction data needed has evolved rapidly:

• Early post-ChatGPT: ~10K high-quality, human-generated samples could be state-of-the-art (Ouyang et al., 2022) (Rajani et al., 2023)
• Current practice: large-scale synthetic datasets work best, but quality filtering is essential

Scaling the prompts quickly enabled more performance. Now, reasoning models are using more compute and tokens to train via more tokens per prompt in SFT (and longer context lengths).

Successful post-training starts with meaningful evaluations for targeted skills and prompts of representative queries for those skills.

All post-training stages require prompts in distribution of tasks. Example prompt budgets from (Lambert et al., 2024):

• Supervised fine-tuning: ~1 million prompts
• Preference fine-tuning: ~1 million (partial overlap with SFT can be useful)
• Reinforcement fine-tuning: ~10-100 thousand (data less available)

Large variance on these numbers is possible — but the key point is that prompts are the starting material for every stage. Recent work has of course scaled up various RL training stages :D!

Synthetic data has become the dominant approach for building SFT datasets – following (Wang et al., 2023):

1. Start with N high-quality (often human-written) prompts
2. Ask a strong LM to create modified versions of these instructions
3. Generate completions with another (or same) strong LM
4. Result: easily 10x more training data

Quality of responses is the simpler part — strong models (e.g. GPT-4o, Llama 3.1 405B) generate good completions to most instructions.

Human data is still needed for out-of-distribution or novel tasks. At the time of recording, this is “knowledge work” tasks like healthcare/law.

Two repeated and parallelizable tracks:

Data mixing:

• Take existing datasets, combine with current mix, observe performance
• Substantial effort in trying to remove data and maintain performance
• Start fully with mixing before curation

Data curation:

• Identify evaluations where the model is behind
• Create new targeted data for those skills
• [Optionally] filter responses based on quality or correctness

These two tracks iterate: mix what you have, evaluate, curate what’s missing, mix again.

A probability model is a mathematical form that we assume matches how real judgments work — then we fit its parameters to data. The canonical reward model uses the Bradley-Terry model (1952).

A probability model is a mathematical form that we assume matches how real judgments work — then we fit its parameters to data. The canonical reward model uses the Bradley-Terry model (1952). Given two items i and j, the probability that a judge prefers i over j:

Each item has a latent strength p_i > 0. Reparametrizing with p_i = e^{r_i} lets us work with unbounded scores r_i \in \mathbb{R} — which is what a neural network naturally outputs:

Only score differences matter — adding the same constant to all scores leaves preferences unchanged.

This is not the only possible model — but it’s what worked in early RLHF and has stuck. It’s a useful approximation, not a law of nature.

Given a prompt x, a chosen completion y_c, and a rejected completion y_r, we score both with a reward model r_\theta:

We want to find \theta that maximizes this probability — i.e. the model should assign a higher score to the completion humans preferred.

Start from maximizing the preference probability:

Start from maximizing the preference probability:

Divide numerator and denominator by \exp(r_\theta(y_c \mid x)):

Start from maximizing the preference probability:

Divide numerator and denominator by \exp(r_\theta(y_c \mid x)):

The numerator becomes 1, and in the denominator \frac{\exp(r_\theta(y_r \mid x))}{\exp(r_\theta(y_c \mid x))} = \exp(r_\theta(y_r \mid x) - r_\theta(y_c \mid x)):

Start from maximizing the preference probability:

Divide numerator and denominator by \exp(r_\theta(y_c \mid x)):

Rewrite as \frac{1}{1 + \exp(-(r_\theta(y_c \mid x) - r_\theta(y_r \mid x)))}. Recognize the sigmoid \sigma(z) = \frac{1}{1 + e^{-z}}:

\log is monotonic, so \arg\max \sigma(\cdot) = \arg\max \log \sigma(\cdot):

Flip the sign to turn maximization into minimization (a loss):

The first form, as in InstructGPT (Ouyang et al., 2022):

The second form, as in Anthropic’s work (Askell et al., 2021):

These are equivalent: using \sigma(\Delta) = \frac{1}{1 + e^{-\Delta}} gives -\log\sigma(\Delta) = \log(1 + e^{-\Delta}).

Both are just binary cross-entropy — the reward model is learning to classify which completion was preferred.

The most common implementation: append a linear head to a language model that outputs a single scalar.

class BradleyTerryRewardModel(nn.Module):
    def __init__(self, base_lm):
        super().__init__()
        self.lm = base_lm
        self.head = nn.Linear(self.lm.config.hidden_size, 1)

    def forward(self, input_ids, attention_mask):
        hidden = self.lm(
            input_ids=input_ids, attention_mask=attention_mask,
            output_hidden_states=True, return_dict=True,
        ).hidden_states[-1]
        lengths = attention_mask.sum(dim=1) - 1
        batch_idx = torch.arange(hidden.size(0), device=hidden.device)
        seq_repr = hidden[batch_idx, lengths]
        return self.head(seq_repr).squeeze(-1)

Given the model above, the training loss is just three lines:

rewards_chosen = model(**inputs_chosen)
rewards_rejected = model(**inputs_rejected)

loss = -nn.functional.logsigmoid(rewards_chosen - rewards_rejected).mean()

Practical note: reward models are typically trained for only 1 epoch to avoid overfitting to the preference data.

When annotators provide Likert scale ratings (e.g. 1-5), the magnitude of the preference can inform training. Llama 2 proposed a margin term m(y_c, y_r):

For example, if chosen scores 5 and rejected scores 2, then m = 3.

This encourages the model to produce larger score gaps for strongly preferred pairs.

Note: Llama 3 removed the margin term — the team observed diminishing improvements at scale.

InstructGPT balances multiple comparisons per prompt to prevent overfitting (and included all prompts in the same batch at training time to make this work):

K-wise loss (Plackett-Luce model, used in Starling 7B/34B) handles full rankings over K completions:

When K = 2, this reduces to Bradley-Terry.

For reasoning tasks, we often have verifiable correctness. ORMs train a per-token head using the completion-level correctness label repeated across completion tokens:

where r \in \{0,1\} is the completion-level correctness label, broadcast across completion tokens during training.

Key differences from Bradley-Terry: no pairwise comparisons needed — just correct/incorrect labels per response. And the model outputs a per-token probability of correctness, not a single score at the EOS token.

The outcome label (r = 0 or 1) is broadcast to every completion token. Prompt tokens are masked. The model learns per-token correctness predictions from this repeated signal. In a larger batch of training, there is a diverse signal of supervision to learn from.

At inference time, the ORM outputs a probability of correctness at every token. These token-level scores can then be aggregated into a response-level verifier score for filtering or reranking.

Some papers use “outcome reward model” to mean a Bradley-Terry model trained on correct vs. incorrect completions. This conflates two different architectures:

A BT model trained on (correct, incorrect) pairs is still a preference RM — it just uses correctness as the preference signal. A true ORM has a fundamentally different input-output structure.

Key distinction — ORM vs. Value Function:

• ORMs predict offline-labeled correctness with a per-token head: p(\text{correct}_t)
• Value functions predict expected remaining return: V(s_t) = \mathbb{E}[\sum_{k \geq t} \gamma^{k-t} r_k \mid s_t]

Same architecture, different semantics and supervision pipeline. More on value functions in the policy gradient lecture(s).

• RM: “How good is this whole answer?” — scalar value
• ORM: “Is this response on track to be correct?” — outcome signal trained across completion tokens
• PRM: “Are the reasoning steps sound?” — per-step scores
• Value Function: “How much reward remains from here?” — baseline for RL advantages

An alternative to training a reward model: prompt an LLM to judge quality (prompt from (Zheng et al., 2023)).

[System]
Please act as an impartial judge and evaluate the quality of the
responses provided by two AI assistants to the user question below.
...
After providing your explanation, output your final verdict by strictly
following this format: "[[A]]" if assistant A is better, "[[B]]" if
assistant B is better, and "[[C]]" for a tie.

Spawned benchmarks: MT-Bench, AlpacaEval, Arena-Hard, WildBench.

Current status: generative RMs tend to underperform trained reward models on RM evaluations, but are cheaper to set up and used widely in evaluation and training pipelines. An evaluation/utility mismatch exists.

I very happily helped kickstart this area with the first one (Lambert et al., 2025)! The RM evaluation landscape has expanded rapidly:

• General chat: RewardBench, RewardBench2, RMB, RM-Bench
• Specialized: M-RewardBench (multilingual), RAG-RewardBench, RewardMATH
• Process RMs: PRM Bench, ProcessBench
• Agentic: Agent-RewardBench
• Multimodal: MJ-Bench, Multimodal RewardBench, VL RewardBench

The bulk of early RM research focused on establishing benchmarks and identifying behavior modes. Training innovations (aspect-conditioned models, high-quality human datasets, scaling experiments) are still an active area.

Rejection sampling is a popular baseline for preference fine-tuning. The idea:

1. Generate many candidate completions
2. Score them with a reward model
3. Keep only the best ones
4. Fine-tune on those

No policy gradients, no online RL — just filtered supervised learning.

The name comes from computational statistics: sample from a simple distribution, use a heuristic to accept/reject samples, approximating a complex target distribution.

The four stages: 0. Select prompts and reward model (reuse IFT prompts or curate new ones)

1. Generate N completions per prompt from the current model
2. Score all completions with the reward model
3. Fine-tune on the top completions (same SFT loss as instruction tuning)

Used in WebGPT (Nakano et al., 2021), Anthropic’s Helpful and Harmless (Bai et al., 2022), Llama 2 Chat (Touvron et al., 2023), and many other seminal works.

This is still just offline data curation: generate first, then train on the filtered outputs.

Given M prompts and N completions each:

Each reward: r_{i,j} = \mathcal{R}(y_{i,j} \mid x_i)

Select the highest-scoring completion for each prompt independently:

Example (5 prompts, 4 completions):

Result: one completion per prompt, every prompt represented.

Flatten the reward matrix and select the top K pairs globally:

Here, R_{\text{flat}} is the length-MN vector made by flattening the reward matrix. Each selected flat index is then mapped back to its original (prompt, completion) pair.

Same example, top 5 overall:

Now prompt 3 gets two completions (0.9 and 0.7), while prompt 5 gets none. This optimizes for absolute quality but can bias toward easy prompts.

The core hyperparameters are intuitive:

• Temperature: 0.7-1.0 (need diversity in completions)
• Completions per prompt: 10-30+ (too few = noisy selection)
• Fine-tuning: standard SFT on selected completions (same loss, but details like learning rate may differ from initial IFT)

Practical tip: sort completions by length before batch RM inference to reduce padding token computation.

Open questions: how to sequence RS in a multi-stage pipeline, whether to use generations from multiple models, optimal prompt selection. There hasn’t been a fully open reproduction of this, which is kind of confusing as to why. My hunch is that training an reward model has some subtle tricks.

Best-of-N (BoN) follows the same generate-and-score procedure, but skips the fine-tuning step:

• Used at inference time to pick the best completion
• Does not modify the model — it’s a sampling technique
• Often used as a baseline comparison for online RL methods like PPO

BoN is the simplest possible reward-guided method: generate more, pick the best. Can also be done with verification / LLM-as-a-judge instead of traditional reward models.

Putting it all together — a simple path from pretrained model to preference-tuned model:

1. Instruction fine-tune the pretrained model → learns the chat format
2. Collect preference data → human annotators compare pairs of responses (more on data after the training lectures)
3. Train a reward model on preference data → learns to score quality
4. Rejection sampling → generate many completions, keep the best, fine-tune

This is a strong baseline.

Next lectures will cover more powerful optimization methods: PPO, which optimizes a policy against a learned reward signal, and DPO, which optimizes directly from preference pairs rather than filtering training data.