RLHF's exponential penalty on disagreeing users is a preference-data sourcing problem

What changed, and when

On 11 September 2026 Kazusato Oko, Annie Ulichney, Nika Haghtalab (UC Berkeley) and Han Bao (Institute of Statistical Mathematics and Tohoku University) posted Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner (arXiv:2609.12651v1), in the ICML 2026 proceedings, PMLR 306.

The quantity under analysis is distortion, borrowed from social choice: the multiplicative gap between the average user utility that a mechanism achieves and the highest average utility achievable. Gölz et al. (2025) showed that some distortion is unavoidable, with an algorithm-independent lower bound of Omega(beta), and that RLHF specifically suffers an exponential lower bound, e to the Omega(beta), because the Bradley-Terry likelihood is nonlinear. Nash Learning from Human Feedback attains the Omega(beta) optimum; RLHF appeared not to.

This paper re-derives that with reward clipping and finds the exponential was a consequence of an implicit assumption: unbounded mismatch between mu, the distribution that generates the preference data, and pi-ref, the KL reference policy. Introduce B as a bound on the log density ratio between them and the picture changes shape. In the representative regime with a constant KL budget, distortion is Theta-tilde(beta B + beta): linear in beta, linear in B, with matching upper and lower bounds. When mu equals pi-ref, B is 0 and RLHF hits the optimal O(beta).

The evidence table

The bounds first, as the paper's own Table 1 states them, assuming a constant KL budget tau. Here beta is the Bradley-Terry temperature, B the maximum log density ratio between the preference-data distribution and the reference policy, and m the number of alternatives in the social-choice setting.

Reproduced from Table 1 of arXiv:2609.12651v1. The asymptotic notation is the paper's; hidden constants are not stated.

Then the measured quantities. This is a theory paper with a short experiments section, and everything below comes from it.

Values from arXiv:2609.12651v1, Section 6 and Appendices D.1 to D.3. We report the Figure 3 endpoints as the paper describes them; the figure gives no printed endpoint value, so we quote the axis range rather than a number.

The reward-scale measurement is the part that stops this being purely theoretical. Beta upper bounds the maximum difference between rewards, so a reward model whose pairwise differences reach 108.8 is operating deep in the nonlinear regime of the Bradley-Terry likelihood, which is exactly where distortion is non-negligible. Two widely used open reward models, four times apart in scale, are both there.

B is decided by who generated the completions, not by who ranked them

The mismatch does not come from the annotators. It comes from the sampling step before them. You collect completions from some model, humans rank them, a Bradley-Terry reward is fitted, and the policy is optimised under a KL constraint to a reference. If the completions came from the same checkpoint as the reference, the two distributions are the same object and B is 0. If they came from a previous-generation model, a different vendor's model, or a mixed pool assembled months earlier, B is whatever it is, and nothing downstream measures it.

The paper's Appendix D.3 is the practical contribution hiding in an appendix. It notes that B is normally impossible to pin down, then finds a case where it is not: OLMo 3 publishes both its DPO training data and its intermediate checkpoints, and part of that data was generated by OLMo 2. Treating the pre-DPO checkpoint as pi-ref and OLMo 2 7B Instruct as mu, the authors measure the log-likelihood difference on the completions actually used in training. It reaches roughly 100 at the completion level, and up to 4.729 per token after length normalisation, with a minimum of -3.449.

Those two figures bracket the honest answer. The completion-level maximum treats each completion as an alternative and gives B of about 100, which the authors themselves call possibly too pessimistic. The per-token figure of 4.729 is the conservative reading. Either way it is not 0, and the bound is linear in it.

Read the two curves for their shape only. Both bounds carry unknown constants, and we set them to 1 to draw anything at all. What the picture shows is the structural claim: the earlier result grew with the exponent of beta, and if beta is in the range the open reward models suggest, that is an unusable bound. The new one grows with beta times a factor you can shrink by changing where your data comes from.

What it means for anyone buying or collecting preference data

Where the completions came from is a specification, not a detail. A preference dataset is normally described by its prompts, its annotator pool, its rubric, and its pair count. This analysis says the generating model belongs in that list, because it sets B, and B multiplies the utility gap. Two datasets identical in every other respect are not equivalent if one was generated by the model you are about to tune and the other by something else.

On-policy collection has a price and now it has a number attached. Sampling fresh completions from your current checkpoint each round is more expensive than buying a static preference corpus once: you pay inference, you cannot amortise annotation across model versions, and every new checkpoint invalidates the old pairs. What you buy for it is B = 0 and the algorithm-independent optimum. Whether that trade is worth it depends on your beta, which you can estimate from your own reward model's pairwise differences in an afternoon.

If you must use off-policy data, fine-tune toward it first. That is the paper's second recommendation, and it is the cheaper of the two: supervised fine-tuning on data close to mu before RLHF moves pi-ref toward mu, which shrinks B directly. This gives a concrete reason for a step many pipelines already perform for vaguer reasons.

Vendors could publish B, and none do. The measurement in Appendix D.3 needs three things: the completions, the model that generated them, and the checkpoint the buyer will regularise against. The first two are the vendor's to disclose. This is the same disclosure gap we found in label noise estimates for HH-RLHF: the properties that decide whether a preference corpus is usable are properties of how it was produced, and they are the ones least often written down. Our judgement: naming the generating model and its sampling settings is a low-cost disclosure that would let any buyer compute this themselves.

The headline is good news, with a caveat. RLHF is not structurally unsuited to a population that disagrees. But the paper's own framing is careful: the effect of mismatch persists even under a small KL budget, and it is ruled out as exponential only unless the mismatch is extreme. A B of about 100 on a real open training set is not obviously far from extreme.

Check it yourself

The reward-scale measurement is the one you can repeat on your own stack today, and it is the one that tells you whether any of this matters for you. Both models and both datasets are public.

open https://arxiv.org/html/2609.12651v1        # Table 1, Section 6, Appendices D.1 to D.3

# the reward models and preference sets measured in Section 6.1
# Skywork-Reward-V2-Llama-3.1-8B  +  Skywork-Reward-Preference-80K-v0.1
# UltraRM-13B                     +  UltraFeedback (best vs worst of 4 completions)
# procedure: sample 5,000 (prompt, chosen, rejected) triples, score each completion,
# take |r(chosen|prompt) - r(rejected|prompt)|, plot the cumulative distribution.
# reported maxima: 108.8 (Skywork), 25.4 (UltraRM)

# the mismatch estimate of Appendix D.3, on models whose data and checkpoints are public
# mu      = allenai/OLMo-2-1124-7B-Instruct        (generated the flan-subset completions)
# pi_ref  = allenai/Olmo-3-7B-Think, pre-DPO checkpoint
# measure log mu(x) - log pi_ref(x) on those completions; drop top and bottom 5% by length
# reported: about 100 at completion level; 4.729 max and -3.449 min per token

python3 - <<'PY'
import math
for label, B in [("on-policy", 0.0), ("OLMo 3, per token", 4.729),
                 ("paper's synthetic", 11.5), ("OLMo 3, per completion", 100.0)]:
    print("%-24s B=%6.3f  bound multiplier (1+B) = %7.1fx" % (label, B, 1+B))
print("shape at beta=10: exp(beta)=%.0f vs (1+4.729)*beta=%.1f" % (math.exp(10), (1+4.729)*10))
PY
# on-policy                B= 0.000  bound multiplier (1+B) =     1.0x
# OLMo 3, per token        B= 4.729  bound multiplier (1+B) =     5.7x
# paper's synthetic        B=11.500  bound multiplier (1+B) =    12.5x
# OLMo 3, per completion   B=100.000 bound multiplier (1+B) =   101.0x
# shape at beta=10: exp(beta)=22026 vs (1+4.729)*beta=47.3

Two honest limits on that snippet. The multiplier is the bound's dependence on B with the hidden constants and logarithmic factors dropped, so it is a ratio between bounds, not a predicted ratio between measured utilities. And the exp-versus-linear comparison sets both unknown constants to 1, which is a drawing convention, not a result. The falsifiable content is in the reward-scale measurement, which is a number you can reproduce exactly.

What would prove this wrong

The theory is proved with matching upper and lower bounds, so the place it can fail is the bridge from the bound to practice: whether B, as defined on log density ratios, is the quantity that actually governs utility loss in a real fine-tune, and whether the reward-scale proxy for beta is meaningful given optimisation error. The authors flag the second themselves.

A dated prediction: by 31 December 2027, a published controlled comparison holding prompts, annotators and pair count fixed while varying only the generating model will report a measurable average-utility gap between on-policy and off-policy preference data on a pluralistic evaluation. If such a study runs and finds no gap beyond noise, the practical reading here is wrong, whatever the bounds say.

Sources

1. Oko, K., Ulichney, A., Haghtalab, N., Bao, H. Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner. arXiv:2609.12651v1, 11 September 2026. ICML 2026, PMLR 306. Table 1, Theorems 3.1, 4.1 and 5.1, Section 6, Appendices D.1 to D.3; HTML version.
2. Liu, C., et al. Skywork-Reward-V2-Llama-3.1-8B and Skywork-Reward-Preference-80K-v0.1, the reward model and preference set measured in Section 6.1.
3. Cui, G., et al. UltraRM-13B and UltraFeedback, the second reward model and preference set measured in Section 6.1.
4. AllenAI. OLMo 3 and OLMo 2 7B Instruct, the open models whose published DPO data and intermediate checkpoints make the mismatch estimate in Appendix D.3 possible.
5. BLOMEGA. Two estimates of label noise in HH-RLHF, 1.77% and 38.97%.
6. BLOMEGA. Data annotation research: the latest.