Reward models pick a confident wrong answer over a hedged wrong one up to 99% of the time, and a near-free fix exists

What changed, and when

Length and sycophancy bias in reward models are known problems, documented at scale in Secrets of RLHF in Large Language Models Part II: Reward Modeling (Wang, Zheng, Chen et al., arXiv:2401.06080, 2024). Most fixes to date have worked by changing the training data, reweighting examples, or adjusting the RL objective.

On 6 February 2026 Daniel Fein, Max Lamparth, Violet Xiang, Mykel J. Kochenderfer and Nick Haber posted One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models (arXiv:2603.03291); the camera-ready version appeared at ICML 2026 on 31 May 2026 (PMLR vol. 306, pp. 29862 to 29896). Their stated affiliations are not listed on the arXiv page. They tested five reward models, DeBERTa, two Skywork-Qwen variants (0.6B and 8B), AllenAI-Llama-8B and Skywork-Llama-8B, and measured five kinds of bias: length, sycophancy, overconfidence against hedged answers, answer-position, and a newly identified bias toward a specific generator model's writing style. Instead of retraining or reweighting data, they look inside each model's activations for a single linear direction that encodes a given bias and subtract it out before the reward head reads the representation, a technique they call mechanistic reward shaping.

The evidence table

Numbers are transcribed from arXiv:2603.03291v2, Table 7 (Appendix C.2) for the uncertainty bias, and Table 2 for RewardBench2.

Read the other way, at baseline these five models prefer the confidently stated wrong answer 73.3%, 98.2%, 96.7%, 94.4% and 99.3% of the time respectively, on items where both answers are equally wrong. After the intervention, three of the five (Skywork-Qwen-8B, AllenAI-Llama-8B, Skywork-Llama-8B) cross past the 50% midpoint into mildly preferring the hedged answer; DeBERTa and Skywork-Qwen-0.6B land just under it.

The uncertainty fix specifically costs nothing measurable on general reward quality, 70.1% before and after. The other two biases tested less cleanly: on answer-position, the standard deviation of accuracy across multiple-choice positions A to D ranges from 0.8 to 27.7 points across models and tasks at baseline, falling to 0.9 to 16.0 after nulling, a reduction but not a clean fix for every model. Sycophancy resists the method outright: on GSM8K-MC, nulling moved DeBERTa's cave rate (lower is better) from 86.9% to 78.5%, an improvement, but its help-when-wrong rate (higher is better) fell from 61.1% to 24.6%, a large drop; for AllenAI-Llama-8B on the same task, both measures got worse after nulling (cave rate 75.6% to 88.7%, help rate 96.3% to 27.6%). The paper's own text describes the sycophancy results as inconsistent.

How "nulling" works, and where it stops working

The method builds a probe direction in the reward model's activation space from contrastive pairs of responses that differ mainly in the bias dimension being targeted, for example the same answer stated with and without hedging language. That direction is then projected out of the hidden state before the reward head scores it, so the model's reward no longer varies with that specific feature. The authors classify a bias as low-complexity when this works cleanly, meaning the direction is linearly separable from the signal the reward model actually needs, which is what they find for length, position and uncertainty. They classify a bias as high-complexity when it resists this, which is what they find for sycophancy: the feature that drives sycophantic agreement appears to overlap with features the reward model needs for legitimately agreeing with a correct correction, so removing the direction removes both.

What it means for anyone training or buying a reward model

Not every reward-model bias costs the same to fix. Length, position and uncertainty bias responded to a cheap, post-hoc, linear intervention with almost no measured cost to RewardBench2 accuracy. Sycophancy did not. If you are auditing a vendor's reward model, ask which category each known bias in it falls into, because "we fixed our biases" can mean a near-free patch for one kind and an unsolved problem for another, inside the same model.

A bias fix measured on a static benchmark is not the same claim as a bias fix that survives training. This is not a hypothetical caution, it already happened once to a related bias from the same lab. A follow-up paper, Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressure (Lamparth, Fein, Haupt, Hussing, Kochenderfer, arXiv:2605.27996, 27 May 2026), found that nulling a length bias during live GRPO training shortened responses as intended, but shifted optimization pressure onto confidence calibration instead, producing more overconfident, less factually accurate free-form answers. The uncertainty fix featured here was validated on a static audit distribution, the same kind of test the length fix passed before it failed under live training.

Track more than one bias at once. The substitution paper's core argument is that successful mitigation, substitution, and overcorrection look identical when you only measure the one bias you targeted. A data-quality program built on reward models should monitor length, position, uncertainty and sycophancy together, not pick whichever one made headlines.

Check it yourself

Code for the nulling method is public at github.com/drfein/OneBiasAfterAnother (no license file was visible in the repository at the time of writing), with example experiment configs. The baseline confident-wrong preference rates are the complement of the table above, a quick check:

python3 -c "
vals=[26.7,1.8,3.3,5.6,0.7]
print([round(100-v,1) for v in vals])"
# [73.3, 98.2, 96.7, 94.4, 99.3]

RewardBench2 and the five reward models tested are all public. The full tables, including the sycophancy and position-bias breakdowns by dataset, are in the HTML version at arxiv.org/html/2603.03291v2, Tables 2, 7, 9, 10 and 13.

What would prove this wrong

The length-bias fix from this same method already failed a stricter test, a live RLHF training loop, after passing the static-audit test featured here. The uncertainty fix has, as far as we found, only been validated on the static audit. If an independent live-training test of the uncertainty-nulling intervention, run the same way the length-bias substitution study ran its test, finds no comparable shift of optimization pressure onto a correlated proxy by 31 December 2027, the uncertainty fix would be the first of this paper's interventions confirmed safe under training rather than only under audit. If it does show substitution, "near-free" would describe the audit result only, not the fix.

Sources

1. Fein, D., Lamparth, M., Xiang, V., Kochenderfer, M. J., Haber, N. One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models. arXiv:2603.03291v2, ICML 2026, PMLR vol. 306; HTML version.
2. Lamparth, M., Fein, D., Haupt, A., Hussing, M., Kochenderfer, M. J. Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressure. arXiv:2605.27996, 27 May 2026.
3. Wang, B. et al. Secrets of RLHF in Large Language Models Part II: Reward Modeling. arXiv:2401.06080, 2024.
4. Code: github.com/drfein/OneBiasAfterAnother.
5. BLOMEGA. HH-RLHF preference label noise and Data annotation research: the latest.