Q&A 1: RLHF course student questions

Careful: two different meanings.

• Reward granularity: final-answer reward vs step/token rewards (PRM, KL, format rewards) (Lightman et al., 2024).
• Loss granularity: average by sequence vs average by active tokens; DAPO uses token-level policy-gradient loss (Yu & others, 2025).

# Sequence-normalized: each completion has equal weight
loss = ((pg_loss * mask).sum(dim=1) / mask.sum(dim=1)).mean()

# Token-normalized: each generated token has equal weight
loss = (pg_loss * mask).sum() / mask.sum()

Why token-level can help:

• local signals: KL, format rewards, PRMs, partial credit
• long-CoT RL: more gradient on the tokens actually sampled
• batching: normalize by active tokens, not examples
• DAPO: token-level loss + Clip-Higher + dynamic sampling for reasoning RL (Yu & others, 2025)

Counterpoint: if the reward is only final-answer correctness, sequence-level advantage is still natural; GSPO pushes the ratio back to sequence level (Zheng et al., 2025).

• \pi(\cdot \mid s_t) is vocab-shaped: [B, T, V]
• \pi(a_t \mid s_t) is gathered at sampled tokens: [B, T]
• \rho_t is scalar per token; rollout tensor is [B, T]

# logits/logprobs over vocab: [B, T, V]
old_logps_all = old_policy.log_softmax(logits_old, dim=-1)
new_logps_all = policy.log_softmax(logits_new, dim=-1)

# sampled token ids: [B, T]
actions = input_ids[:, 1:]

# gather only the probabilities of the sampled tokens: [B, T]
old_logps = old_logps_all.gather(-1, actions.unsqueeze(-1)).squeeze(-1)
new_logps = new_logps_all.gather(-1, actions.unsqueeze(-1)).squeeze(-1)

# importance ratio per sampled token: [B, T]
rho = torch.exp(new_logps - old_logps)

# PPO minimizes a negative policy loss
pg_loss_unclipped = -advantages * ratio
pg_loss_clipped = -advantages * torch.clamp(
    ratio, 1.0 - eps, 1.0 + eps
)

# choose the more conservative / worse loss
pg_loss = torch.max(pg_loss_unclipped, pg_loss_clipped)

torch.max matters because PPO is written as a loss to minimize.

It selects the update that gives less improvement after clipping.

# value clipping: do not let critic jump too far
v_clip = old_values + torch.clamp(
    v_pred - old_values, -vf_eps, vf_eps
)

vf_unclipped = 0.5 * (v_pred - targets) ** 2
vf_clipped = 0.5 * (v_clip - targets) ** 2

# optimize the worse value error
vf_loss = torch.max(vf_unclipped, vf_clipped)

Policy clipping limits policy drift.

Value clipping limits critic drift.

Same pattern: compare unclipped vs clipped, train on the worse one.