Pooling RLHF raters into one calibrator leaves a fixable 8.58% of RMSE on the table

What changed, and when

On 25 June 2026, Arnav Raj (Department of Computer Science and Engineering, Indian Institute of Technology Delhi) posted PEBS: Per-rater Empirical-Bayes Shrinkage for RLHF Reward-Model Calibration (arXiv:2606.27578v1).

RLHF reward models are trained on preferences pooled from thousands of annotators who rate completions on a shared numeric scale, commonly 0 to 100. Different annotators use that scale differently: some lenient, some strict, some compressing their ratings into a narrow band, some spreading them wide. The standard fix is a single global affine calibrator, (alpha-pop, beta-pop), fit by pooled ordinary least squares across every rater at once. That calibrator is, by construction, a compromise that fits the average rater and nobody in particular.

PEBS (Per-rater Empirical-Bayes Shrinkage) instead fits a per-rater affine calibrator on a held-out slice of that rater's own ratings, then shrinks the per-rater slope and intercept toward the population mean using a closed-form Morris-James-Stein empirical-Bayes weight, the classical shrinkage estimator from Robbins (1956) and James and Stein. The shrinkage weight is larger for raters with fewer ratings (who would otherwise overfit) and smaller for raters with many ratings (who can support their own calibrator). No part of the underlying reward model is retrained.

The evidence table

Reproduced from Table 1 of arXiv:2606.27578v1, Section 3.1. Held-out score-prediction RMSE for N=1,394 users with 5-fold cross-validation on a 7B base reward model, PRISM Alignment corpus. 95% CI on the 8.58% gain: [7.59%, 9.42%], 4,000-replicate cluster bootstrap by user, excluding zero.

The "+9.66%" and "+18.24%" entries are stated as RMSE reduction (improvement), matching the paper's own sign convention in Section 3.4; arXiv:2606.27578v1, abstract and forest-plot discussion.

Why shrinkage, not a calibrator per rater, is what works

Fitting a separate OLS calibrator for each of 1,394 raters sounds like it should beat one calibrator shared by all of them, and it does, on the training objective. It does worse on held-out RMSE, because a rater who only rated a handful of pairs gives their personal calibrator almost nothing to fit on, so that calibrator's slope and intercept are noise dressed up as personalization. Table 1 shows this directly: naive per-user OLS (23.73) is worse than PEBS (23.33), even though both start from the same per-rater fits.

Only one of 1,394 raters ends up with a marginally negative shrunk slope, close to the theoretical minimum the paper derives for how far shrinkage can distort a slope. The paper verifies that bound empirically rather than treating it as given.

What it means for anyone running an RLHF pipeline on pooled annotators

The gain is a held-out RMSE number, not yet a win-rate number. The quantity that most directly enters RLHF's DPO or PPO loss is the pairwise Bradley-Terry negative log-likelihood, which the paper reports improves by 5.7% relative (paired-t p<10^-7), tail-concentrated rather than uniform: the median per-pair improvement is near zero and the gain is carried by a minority of users with atypical calibration on hard pairs. PEBS does not change pair accuracy at all, by the paper's own Proposition 1; what it changes is the magnitude of the score gap that downstream losses weight by.

That magnitude matters because DPO and PPO consume raw scores, not just ranks. DPO's loss is a sigmoid of a score difference; PPO's advantage normalization and multi-attribute score aggregation both operate on raw scalars. Per-rater scale heterogeneity distorts a sum of raw scalars in a way it does not distort a ranking, which is why a calibration fix that leaves pair accuracy untouched can still change training behavior. The paper's own PPO probe shows the uncorrected reward collapsing at KL ≥ 1.0 while the PEBS-shrunk arm holds.

The cost is close to zero, which is the real argument for adopting it. PEBS is a closed-form, post-hoc estimator: it needs a held-out slice of each rater's own ratings and no retraining of the reward model. For any pipeline that already pools ratings from a large, heterogeneous annotator base, a near-zero-cost correction is a different proposition than a method that requires a new round of RLHF.

Our judgement, specific to data provenance: this result depends entirely on preserving stable per-annotator identifiers through the pipeline, which many commercial annotation vendors strip out before delivery. A buyer who cannot tell which ratings came from the same rater cannot run PEBS at all, no matter how good the method is. Annotator-identity preservation, done consensually, is itself a dataset property worth specifying in a licensing deal.

Check it yourself

open https://arxiv.org/abs/2606.27578                           # paper, Table 1, Algorithm 1
open https://github.com/deadsmash07/pebs-pluralistic            # code and fitted calibrators, paper's footnote 1
open https://huggingface.co/datasets/HannahRoseKirk/prism-alignment  # PRISM Alignment corpus

# what to check without rerunning anything:
# Table 1, Section 3.1: no-calibration RMSE 27.13, pop-slope 25.52 (baseline),
#   per-user OLS 23.73 (-7.02%), EB-shrunk PEBS 23.33 (-8.58%)
# Section 3.1: 95% CI on the 8.58% gain is [7.59%, 9.42%], 4,000-replicate
#   cluster bootstrap by user, excludes zero
# Section 3.2: per-user paired effect size d=0.542 (95% CI [0.491, 0.607]);
#   BT-NLL improves 5.7% relative (paired-t p<10^-7), tail-concentrated

python3 - <<'PY'
arms = {"no calibration": 27.13, "population slope": 25.52,
        "per-user OLS (naive)": 23.73, "EB-shrunk (PEBS)": 23.33}
baseline = arms["population slope"]
for name, rmse in arms.items():
    pct = 100 * (rmse - baseline) / baseline
    print(f"{name:22s} RMSE={rmse:6.2f}  vs pop-slope: {pct:+6.2f}%")
PY
# no calibration        RMSE= 27.13  vs pop-slope:  +6.32%
# population slope      RMSE= 25.52  vs pop-slope:  +0.00%
# per-user OLS (naive)  RMSE= 23.73  vs pop-slope:  -7.02%
# EB-shrunk (PEBS)      RMSE= 23.33  vs pop-slope:  -8.58%

What would prove this wrong

The RMSE result replicates across PRISM, PluriHarms and HelpSteer2, which is more than a single-dataset anecdote, but all three evaluations come from the same author and the same codebase. The result that would most change the practical read is the downstream one: whether PEBS's calibration fix changes final policy behavior, not just held-out RMSE and a single PPO probe on one policy and one reward model.

A dated prediction: by 31 December 2027, an independent group running a full RLHF fine-tune with and without PEBS-style per-rater shrinkage, holding the reward model and training data fixed, will report a measurable difference in final policy win rate against a held-out human preference set, not just in reward-model RMSE. If such a study finds no measurable win-rate difference despite a clear RMSE gain, the calibration fix would be shown to not propagate through training in a way that matters, contrary to what the BT-NLL and PPO-probe results here suggest.

Sources

1. Raj, A. (Indian Institute of Technology Delhi). PEBS: Per-rater Empirical-Bayes Shrinkage for RLHF Reward-Model Calibration. arXiv:2606.27578v1, 25 June 2026. Pluralistic Alignment Workshop, ICML 2026, Seoul. Table 1, Algorithm 1, Sections 2.3, 3.1 to 3.3.
2. Kirk, H.R., et al. PRISM Alignment corpus, the public RLHF dataset with stable per-annotator identifiers used for the primary result.
3. BLOMEGA. RLHF's exponential penalty on disagreeing users is a preference-data sourcing problem.
4. BLOMEGA. Two estimates of label noise in HH-RLHF.