RGPO Nearly Triples RLHF's Tie Rate, 12.27% to 29.03% on MultiPref, and Its Baseline Rule Is Nowhere in the Paper

What did the paper claim, and when?

On 7 July 2026, Jingyi Huang, Ruohan Zong, Yujun Feng, Liran Ma, Lanyu Shang and Yang Zhang posted Reliability-Aware LLM Alignment from Inconsistent Human Feedback (arXiv:2607.20515, cs.AI), proposing Reliability-Guided Preference Optimization, RGPO. Huang, Feng, Ma and Zhang are at Miami University, Zong is at the University of Illinois Urbana-Champaign, and Shang is at Loyola Marymount University. Code and configuration went up the same window at github.com/GenieHuang/RGPO, where the README describes it as an ICML 2026 paper. We could not independently confirm that acceptance on a public ICML 2026 program listing, so we treat it here as the authors' own claim, not a verified fact.

The method treats every DPO preference pair as the output of a noisy voting process rather than ground truth. For each comparison, a Dawid-Skene style EM model gives every annotator j a 3-by-3 confusion matrix over {prefer A, tie, prefer B}, estimates a class prior, and alternates an E-step that computes the posterior probability of each latent true label per comparison with an M-step that re-fits the confusion matrices from those posteriors. An annotator's reliability score, q_j, is the mean of the diagonal of their converged confusion matrix: how often their vote matches the model's best guess at the truth. The latent label is the posterior's argmax (its MAP estimate); if that argmax is a tie, the pair is dropped from training. Surviving pairs get a per-comparison consistency weight: the binary entropy of the vote mass across directions, rescaled through 0.5 x (tanh(s/1.0) + 1), so a comparison every reliable annotator agreed on gets close to full weight and one with a near-even split gets close to zero. That weight multiplies the ordinary DPO, SimPO or IPO loss term for that pair.

Two datasets carry the experiments. MultiPref (Miranda et al., ACL 2025) supplies 10,461 comparisons with four votes each, 227 named annotators, 41,844 raw annotations. HelpSteer2-Disagreement is the authors' own construction from NVIDIA's public nvidia/HelpSteer2 release (Wang et al., NeurIPS 2024 Datasets and Benchmarks Track): they take the dataset's disagreements split, which lists every individual annotator's 0-4 helpfulness score per single response, group responses by prompt, keep only prompts with exactly two responses, and pair each annotator's score for response 1 against the same annotator's score for response 2. The paper reports 11,824 such pairs from up to 6 annotators, since pairing two 3-to-5-rater responses by list position yields at most 6 usable slots per prompt. Llama-3-8B-Instruct and Qwen2.5-7B-Instruct are each fine-tuned with DPO, SimPO and IPO, with and without RGPO, and scored on AlpacaEval 2 (raw and length-controlled win rate against gpt-4-1106-preview) and Arena-Hard (against gpt-4-0314).

How much does reliability weighting change the labels?

The paper's own Table 3 is the headline. Every number below is quoted directly from it, both datasets, both methods.

r_A / r_B are the paper's own column names for the two directions; on MultiPref "first" is completion_a and on HelpSteer2-Disagreement it is response_1, by the authors' own preprocessing code (see the mechanism section below). All counts from arXiv:2607.20515, Table 3.

Four of the eight model/dataset/benchmark cells we checked in the paper's Table 1 move the other direction when RGPO is added. Qwen2.5-7B-Instruct trained with DPO on MultiPref scores 78.20% on Arena-Hard without RGPO and 77.10% with it. The same model with IPO on MultiPref goes from 81.30% to 80.90%. Neither change is marked significant in the paper, but neither is the headline framing, which leads with the AlpacaEval 2 length-controlled column, where RGPO helps in every cell we checked and the gain is flagged significant (p < 0.05) in three of four.

Why does weighting by reliability produce more ties?

A uniform ensemble (a plain majority or sum of votes, every annotator weighted the same) always produces a decisive label unless the vote is perfectly split, because it has no concept of an annotator being unreliable, only of a vote existing. Modeling reliability changes what "unresolved" means: a 3-1 split from four annotators with reliability scores clustered near 0.5 (close to a coin flip in the model's own confusion-matrix estimate) can land with a lower-confidence posterior than a 2-2 split from annotators the model has learned to trust asymmetrically. The paper's own reliability distribution supports this: MultiPref's mean q_j is 0.646, but Figure 3 shows a long tail below 0.4, and those low-reliability annotators' votes get discounted enough, in aggregate, to flip a pair's MAP label from decisive to tied far more often than a simple vote-count would predict.

This is also why the method's main claimed benefit and its main cost are the same lever. Dropping ambiguous pairs removes supervision the authors argue was teaching the model to overfit noise, but it also shrinks the training set by 17.5 to 20.2 percentage points of comparisons (29.03% minus 12.27% on MultiPref, 31.90% minus 11.71% on HelpSteer2), none of which the paper replaces with anything.

What is the undocumented baseline rule, and does it hold up?

This section is ours, computed by BLOMEGA on 24 October 2026. Section 4.1 of the paper states only "we employ uniform ensemble for baseline preference construction" and Section 3.1 says comparisons with a tie outcome are "filtered out for both baselines and RGPO-enhanced models," with no equation or vote-counting rule given for either. We rebuilt both training sets from scratch using the authors' own public preprocessing notebook (data/data_preprocess.ipynb in the GitHub repo) run against the live Hugging Face releases, then tested candidate aggregation rules against the reported Table 3 baseline counts.

The data matched first. Downloading allenai/multipref's parquet gives exactly 10,461 comparisons, 227 annotators and 41,844 votes, matching the paper's own stated sizes. Downloading nvidia/HelpSteer2's disagreements split and grouping its 23,652 single-response annotation rows by prompt gives 11,825 unique prompts, of which 11,824 have exactly two responses (one prompt has four and is dropped, exactly as the notebook's warning branch specifies), each paired across up to 6 shared annotator slots. Both the "11,824" and the "6 annotators" the paper states match our rebuild exactly.

The rule did not match on the first try. We tested summing each comparison's votes, weighted and unweighted, on normal annotators only, experts only, and both combined, with ties assigned by three different thresholds. The one that fit: encode each vote on the paper's own preference-degree scale (clearly-better = 2, slightly-better = 1, tie = 0, with sign by side), sum the votes for a comparison, and call the comparison a tie exactly when that sum equals zero. On HelpSteer2, where the per-vote "degree" is simply one annotator's helpfulness score for response 1 minus the same annotator's score for response 2, this rule reproduces the paper's reported baseline counts bit for bit: 4,904 pairs with a positive sum, 1,384 at exactly zero, 5,536 negative, matching 41.47% / 11.71% / 46.82% to the integer.

On MultiPref the same rule gets close but not exact: 3,021 comparisons summed positive, 6,120 negative, 1,320 exactly zero, which is 12.62% ties against the paper's reported 12.27%, a gap of 37 comparisons out of 10,461. The win/loss split only lines up with the paper's reported 59.51% / 28.22% once the two sides are swapped (58.50% ours against 59.51% theirs for the larger side, 28.88% against 28.22% for the smaller), which tells us the direction convention differs somewhere in the pipeline we could not pin down from the public code, but the magnitudes and the near-exact tie rate are strong evidence the summing rule itself is right. We flag the 0.35-point, 37-comparison discrepancy rather than paper over it: something about MultiPref's case, not HelpSteer2's, keeps us from an exact match.

One number the paper does not report, because it had no reason to: MultiPref's raw annotator votes carry a real, substantial side preference, independent of RGPO. Across all 41,844 individual votes, 10,997 (26.28%) are a direct tie, 11,104 (26.53%) favor completion_a, and 19,743 (47.18%) favor completion_b, nearly double A's share. That skew, not anything RGPO does, is most of why the "uniform ensemble" baseline resolves almost 88% of comparisons decisively in one direction or the other. And the two datasets' raw per-vote tie rates land within two hundredths of a point of each other despite using completely different rating instruments: 26.28% on MultiPref's five-category scale, 26.26% on the 40,365 helpfulness-difference votes we derived from HelpSteer2-Disagreement. We have no mechanism to offer for why; we note it because it held up when we checked, not because we can explain it.

What this means if you build or train on preference data

Budget for the dropped third, not just the weighted two-thirds. RGPO's gain comes with 17.5 to 20.2 points of the training set removed outright once a reliability model decides what counts as resolved. Decide up front whether dropped pairs get re-annotated, replaced, or simply accepted as a smaller dataset, because the paper tests none of those three.

State your aggregation rule in the paper, not just in the code. We spent more time reconstructing RGPO's "uniform ensemble" baseline than checking its headline claim, because the rule exists only implicitly in a function the public repository does not yet contain. A reviewer without the time to rebuild the dataset from the preprocessing notebook has no way to check whether 12.27% is right.

Check your raw-vote skew before trusting any aggregate. MultiPref's roughly 2-to-1 vote skew toward completion_b is not something RGPO or the earlier BA-DPO analysis flags, and it shapes every majority-vote baseline built on this corpus. If response slots are not truly randomized, any method's baseline inherits that skew before it inherits anything about annotator quality.

Pick your benchmark before you pick your method. RGPO's gain is consistent on length-controlled AlpacaEval 2 and inconsistent on Arena-Hard, including two outright losses in the four cells we checked. Run your own target metric before committing training compute, rather than assuming a win generalizes across evaluators the way it does across datasets.

Annotator identity, once anonymized to an index, cannot be un-anonymized. HelpSteer2-Disagreement's "annotatorID" is a 0-to-5 slot position within each prompt's annotation list, not a real identity. A claim about a specific annotator's reliability on this dataset is a claim about a list position that may belong to a different person on every other prompt.

Check it yourself

Both datasets are public. This reproduces our MultiPref and HelpSteer2 counts, and the sign-of-sum rule that matched HelpSteer2's baseline exactly, in about a minute.

mkdir -p recount && cd recount
curl -sL -o mp.parquet \
  https://huggingface.co/api/datasets/allenai/multipref/parquet/default/train/0.parquet
curl -sL -o disagreements.jsonl.gz \
  https://huggingface.co/datasets/nvidia/HelpSteer2/resolve/main/disagreements/disagreements.jsonl.gz
gunzip -k disagreements.jsonl.gz
curl -s https://raw.githubusercontent.com/GenieHuang/RGPO/main/data/data_preprocess.ipynb -o preprocess.ipynb

python3 - <<'PY'
import pandas as pd, json
from collections import defaultdict

# --- MultiPref: sign-of-sum reconstruction ---
ENC = {"A-is-clearly-better": 2, "A-is-slightly-better": 1, "Tie": 0,
       "B-is-slightly-better": -1, "B-is-clearly-better": -2}
d = pd.read_parquet("mp.parquet")
na = nb = nt = 0
for r in d.itertuples():
    votes = [ENC[a["overall_pref"]] for anns in
             (r.normal_worker_annotations, r.expert_worker_annotations) for a in anns]
    s = sum(votes)
    na += s > 0; nb += s < 0; nt += s == 0
print("MultiPref", len(d), na, nt, nb)
# 10461 3021 1320 6120  -> tie 12.62% vs paper's reported 12.27%

# --- HelpSteer2-Disagreement: rebuild the authors' own pairing, then sign-of-sum ---
rows = [json.loads(l) for l in open("disagreements.jsonl")]
groups = defaultdict(list)
for r in rows: groups[r["prompt"]].append(r)
na = nb = nt = 0
for prompt, resp in groups.items():
    if len(resp) != 2: continue
    r1, r2 = resp
    n = min(len(r1["helpfulness"]), len(r2["helpfulness"]))
    s = sum(r1["helpfulness"][i] - r2["helpfulness"][i] for i in range(n))
    na += s > 0; nb += s < 0; nt += s == 0
print("HelpSteer2", na, nt, nb)
# 4904 1384 5536 -> matches arXiv:2607.20515 Table 3 exactly
PY

Fields that matter: MultiPref's normal_worker_annotations and expert_worker_annotations each carry a 2-entry list with an overall_pref string per vote. HelpSteer2's disagreements split carries helpfulness as a list of 3 to 5 integers, one per annotator, in whatever order the original annotation pipeline recorded them, which is why pairing by list index (not by an actual annotator ID) is the authors' own method, not an approximation we introduced.

What would prove this wrong

Our reconstruction of the baseline rule is falsifiable against the authors' own eventual code. The GitHub repository's data/train/ outputs are referenced in the README but not yet committed; when they are, or when any replication publishes the exact function used to build Table 3, we expect it to match sign-of-sum-with-zero-as-tie on the encoding we used. If the published function instead uses a different threshold, a confidence-weighted sum, or drops certain annotation types we included, our reconstruction is wrong on the method even though it happens to land close on the numbers, and we would want this piece corrected. We expect that code, or a clear denial that our rule is correct, to be available by 30 April 2027, a reasonable window past any ICML 2026 camera-ready deadline.

The paper's own central claim has a sharper test. If reliability weighting is capturing genuine annotator noise and not an artifact of these two specific datasets' annotation pipelines, then a third preference dataset with real, non-anonymized annotator identities and repeated same-annotator comparisons, such as PRISM, should show RGPO's estimated per-annotator reliability q_j correlating with that annotator's own split-half consistency, computed independently the way the "RLHF May Not Reflect Genuine Preferences" study measured it on the same corpus (see our prior note on that paper). A correlation below roughly 0.3 would say RGPO's reliability score is fitting something specific to MultiPref and HelpSteer2's annotation setup rather than a general property of annotator quality.

Sources

1. Huang, J., Zong, R., Feng, Y., Ma, L., Shang, L., Zhang, Y. Reliability-Aware LLM Alignment from Inconsistent Human Feedback. arXiv:2607.20515v1, 7 July 2026. Sections 3.1, 4.1 and 4.4, Tables 1 and 3, Figure 3.
2. GenieHuang/RGPO. Source code, training scripts and preprocessing notebook, including data/data_preprocess.ipynb. Accessed 24 October 2026.
3. Allen Institute for AI. allenai/multipref on Hugging Face, ODC-BY. Downloaded 24 October 2026.
4. NVIDIA. nvidia/HelpSteer2 on Hugging Face, CC-BY-4.0, disagreements and preference configs added 1 October 2024. Downloaded 24 October 2026.
5. Miranda, L. J. V. et al. Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback. ACL 2025. The MultiPref paper.
6. Wang, Z. et al. HelpSteer2: Open-source dataset for training top-performing reward models. NeurIPS 2024 Datasets and Benchmarks Track.
7. Ghafouri, A. et al. RLHF May Not Reflect Genuine Preferences: Annotator Inconsistency and Downstream Effects. ICML 2026 (spotlight), arXiv:2604.03238.
8. BLOMEGA. BA-DPO's per-annotator length bias recount on MultiPref. 29 September 2026.
9. BLOMEGA. RLHF preference labels flip for 18.6% of prompts once inconsistent raters are filtered.