Swapping annotator pools moves zero models on a 6-way leaderboard, 86% on a 10-way one

What changed, and when

On 17 August 2026, Anik Jha posted Whose Gold? Annotator-Pool Disagreement Is Large at the Item Level, and Hidden by Small Leaderboards (arXiv:2608.15980), submitted to the HAIC workshop at NeurIPS 2026. It is a single-author workshop submission, not yet peer-reviewed at a main conference, and that status matters for how much weight to put on it below.

The paper re-examines two existing preference datasets that were each labeled twice by different annotator populations: MultiPref (Miranda et al., 2024, six models, 10,461 pairwise comparisons, rated twice by crowdworkers and twice by screened domain experts) and MT-Bench's human judgments (Zheng et al., 2023, 3,355 votes, with 292 item cells rated both by the benchmark's own authors and by separately recruited experts). The question is not whether experts and crowds agree on average, it is whether the specific items where they disagree change which model wins, and whether that effect depends on how many models are being compared.

The evidence tables

To isolate pool disagreement from ordinary tie-breaking noise, the paper restricts its primary analysis to items where each pool is internally unanimous (both ratings on the same side), so no tie-breaking convention is needed: 2,885 of 10,461 MultiPref comparisons (27.6%), and 246 of 292 MT-Bench cells.

Source: arXiv:2608.15980, Sections 3 to 4. On MultiPref, the two experts assigned to the same item disagree with each other 50.1% of the time, a separate number from the expert-vs-crowd pool contrast above.

Yet on both corpora, ranking six models by expert-majority labels versus crowd-majority labels gives the identical order, Kendall tau 1.00, zero of six displaced. The crowd-minus-expert shift in each model's win rate ranges from −2.1 to +2.8 percentage points (SD 1.9pp), small relative to the observed gaps between adjacent models on the leaderboard (11.1, 2.9, 2.8, 0.8 and 2.5 percentage points). Only the tightest pair, 0.8pp apart, had meaningful swap risk: a 38% chance of reversing order under the item-level bootstrap. Resampling items directly (rather than assuming a fixed shift) displaces at least one of the six models in 28% of bootstrap resamples, close to a parametric estimate of 28.1%.

The paper then asks what happens to that same measured perturbation as the leaderboard grows and models cluster more tightly, which is what real model arenas look like.

Source: arXiv:2608.15980, leaderboard-size analysis, uniform spacing with the measured win-rate perturbation held fixed. The paper reports only these five data points and notes that real model arenas cluster scores more tightly than uniform spacing, so these are lower bounds on real displacement risk.

Why a stable leaderboard is not proof the gold labels agree

The mechanism is arithmetic, not a flaw in either annotator pool. A leaderboard ranks models by their win-rate gaps, and whether a gap survives a given amount of label noise depends only on how wide that gap is relative to the noise, not on whether the noise itself is small. Six models spread across the typical range of win rates tend to have a few wide gaps (11.1 and 2.9 percentage points in this data) and at most one narrow one, so most of the ranking is immune to a perturbation of 1.9pp standard deviation. Add more models into the same win-rate range and the gaps between neighbors shrink toward that same 1.9pp, at which point the noise that was invisible at six models starts flipping adjacent pairs routinely.

This is a different claim from "crowd and expert annotators disagree." Dawid-Skene-style aggregation and larger annotator panels address disagreement on a single item. What this paper measures is what a fixed amount of per-item disagreement does to a downstream ranking as the ranking's own resolution increases, which no amount of per-item aggregation fixes, because the aggregation already ran before the leaderboard was built.

The paper also tests three local open-weight judges (Qwen3.6-35B-A3B, Qwen3.6-27B, Gemma-4-26B-A4B) on the pool-split MultiPref items and finds all three agree more with the crowd majority than the expert majority: 39.5% to 46.3% match to crowd versus 32.7% to 34.0% match to expert, with the rest matching neither. Under the paper's primary estimator the gap shrinks to 2.2 to 4.3 percentage points and two of three confidence intervals touch zero, so this is a real but modest lean, not a clean substitution of one pool for the other. A separate, unrelated finding in the same paper: switching a judge's output format from free JSON to a single constrained token changed its own self-agreement to 47.2% on 500 items, recovered to 94.6% by prefilling the JSON scaffold and reading logits instead, a reminder that judge-measurement choices can move numbers as much as the annotator-pool question this paper is about.

What it means for anyone running an eval or a QA gate

A stable leaderboard is evidence about your leaderboard's size and spacing, not about your labels. Treating "my ranking doesn't change when I re-annotate with a different pool" as proof the gold labels are reliable conflates two different quantities: per-item label agreement and aggregate rank stability. This paper's numbers show they can point in opposite directions at once, on the same data.

Per-item labels are not safe to use directly at any leaderboard size tested. The 23.6% to 30.5% item-level disagreement never goes away, it is simply absorbed differently depending on how many things are being ranked. A team using MultiPref-style labels to train a reward model, rather than to rank a small leaderboard, is exposed to the full item-level disagreement rate regardless of how stable any downstream leaderboard looks.

Growing an arena from 6 models to 20 changes the risk profile even if nothing else changes. A model-comparison product that scales up its roster should expect displacement risk to rise sharply well before it reaches 20 entries, purely from compressed spacing, with no change to annotation quality required to explain it.

Our judgement: this is a single-author workshop submission (HAIC @ NeurIPS 2026), not a peer-reviewed venue paper, and it reuses two existing public datasets rather than collecting new dual-pool labels. The leaderboard-size table has five points, not a dense curve, and the paper's own note that real arenas cluster more tightly than its uniform-spacing assumption means real displacement risk could be higher, not lower, than these numbers suggest. The LLM-judge-favors-crowd finding is directional with confidence intervals that partly touch zero, treat it as a lean, not a settled result.

Check it yourself

MultiPref and MT-Bench's human judgments are both public. The comparison that matters is the leaderboard-size table, not the headline Kendall tau alone.

open https://arxiv.org/abs/2608.15980          # abstract, author, submission date, HAIC workshop note
open https://arxiv.org/html/2608.15980v1       # full text: item-level tables, leaderboard-size analysis, LLM-judge section

python3 - <<'PY'
# recompute the headline contrast directly from the paper's reported numbers
multipref_items, multipref_differ, multipref_reverse = 2885, 0.236, 0.092
mtbench_items, mtbench_differ, mtbench_reverse = 246, 0.305, 0.085
print("MultiPref: %d items, majority differs on %.1f%%, reverses on %.1f%%" %
      (multipref_items, multipref_differ * 100, multipref_reverse * 100))
print("MT-Bench: %d items, majority differs on %.1f%%, reverses on %.1f%%" %
      (mtbench_items, mtbench_differ * 100, mtbench_reverse * 100))

displacement = {6: 0.28, 10: 0.86, 20: 0.9997, 30: 1.00, 50: 1.00}
for k, p in displacement.items():
    print("K=%d models: P(at least one displaced) = %.2f%%" % (k, p * 100))
PY
# MultiPref: 2885 items, majority differs on 23.6%, reverses on 9.2%
# MT-Bench: 246 items, majority differs on 30.5%, reverses on 8.5%
# K=6 models: P(at least one displaced) = 28.00%
# K=10 models: P(at least one displaced) = 86.00%
# K=20 models: P(at least one displaced) = 99.97%
# K=30 models: P(at least one displaced) = 100.00%
# K=50 models: P(at least one displaced) = 100.00%

The item-level percentages and the leaderboard-size probabilities are both the paper's own reported numbers, not re-derived estimates. What is worth checking independently is whether your own leaderboard's adjacent gaps look more like the 6-model case (wide, mostly safe) or the 20-model case (narrow, mostly unsafe), since that is what determines which half of this paper's results applies to you.

What would prove this wrong

The paper's leaderboard-size result assumes uniform spacing with a fixed perturbation replayed across sizes, a simulation built on top of one real 6-model measurement, not five independently measured leaderboards. If real arenas with 10, 20 and 50 models, measured directly rather than simulated, show meaningfully lower displacement than this uniform-spacing model predicts, because real score clustering differs from the assumption, the specific percentages here would not hold even though the underlying mechanism (narrower gaps survive less fixed noise) would still be correct.

A dated prediction: by 31 December 2027, someone, plausibly the author, a conference reviewer, or an independent group, will run this same dual-pool relabeling on a live model arena with a real 20 or more model roster and report the measured displacement rate rather than the simulated one. If that measured rate comes in well under 50%, this paper's leaderboard-size extrapolation is wrong even if its 6-model result stands.

Sources

1. Jha, A. Whose Gold? Annotator-Pool Disagreement Is Large at the Item Level, and Hidden by Small Leaderboards. arXiv:2608.15980, 17 August 2026, submitted to the HAIC workshop, NeurIPS 2026. HTML version.
2. Miranda, L. et al. MultiPref dataset, 2024, as re-analyzed in arXiv:2608.15980 Section 3.
3. Zheng, L. et al. MT-Bench human judgments, 2023, as re-analyzed in arXiv:2608.15980 Section 3.
4. BLOMEGA. LLM annotators, kappa near zero.
5. BLOMEGA. Data annotation research: the latest.