An LLM judge matches the majority label, but predicts human disagreement worse than 3 voters

What changed, and when

On 1 September 2026 Sebastian Steindl, Nikos Voskarides, Alberto Gasparin and Diego Marcheggiani posted Post-hoc Alignment of LLM-judges to Human Judgment Distribution (arXiv:2609.01073, v1). Most LLM-as-judge validation compares a model's output to an aggregated label: the majority vote or the mean rating. This paper also scores the model against the unaggregated human judgment distribution, the soft label, on five datasets chosen for different sources of disagreement.

The datasets are SummEval (news summaries, three expert ratings per item), TopicalChat (dialogue responses, three crowd ratings), ChaosNLI (high-disagreement SNLI items re-annotated by 100 crowd workers each), DynaSent round 2 (adversarial sentiment), and Anecdotes (Reddit "who is in the wrong" posts, restricted to items with 15 votes). DynaSent and Anecdotes are capped at 1,500 items each, sampled as 500 per disagreement tercile by label entropy. Every result uses a 20/80 train/test split and 20 runs. The main judge is Claude-4-Sonnet, with GPT-OSS-120B and Qwen3-32B in an appendix; the authors report the backbone does not change the conclusions.

Soft-label error is reported as distribution calibration error (DistCE) and Jensen-Shannon distance, lower is better. The human reference in Table 5 samples 20% of each item's annotations (minimum one) and scores that sample against the full distribution. That is what gives these numbers a human scale: 20 annotators on ChaosNLI, 3 votes on Anecdotes.

The evidence table

Soft-label DistCE, lower is better. Base judge is Claude-4-Sonnet prompted with soft-label in-context examples (the paper's SLP-SE setting). Values are from Tables 3 and 5 of arXiv:2609.01073v1; the gap-closed column is ours.

The hard-label picture is different. Macro F1 against the majority label, Claude-4-Sonnet against a sampled human, by disagreement tercile (Table 1): ChaosNLI 0.99 vs 0.93 (low), 0.81 vs 0.74 (medium), 0.61 vs 0.53 (high); Anecdotes 0.48 vs 0.50, 0.48 vs 0.42, 0.34 vs 0.35; DynaSent 0.95 vs 1.00, 0.81 vs 0.80, 0.40 vs 0.43. On the rating tasks the average Kendall correlation is 0.480 for the LLM and 0.542 for humans on SummEval, and 0.636 against 0.559 on TopicalChat (Table 2). By the majority label, the judge looks human-level. By the distribution, it looks like a small panel at best.

Two readings of the table the paper does not spell out. First, the authors write that NAPHA "approximates human performance on the Anecdotes and DynaSent datasets". With predicted entropy classes, the deployable setting, Anecdotes goes from 0.299 to 0.272 against a human 0.165; the approximation holds only with oracle classes (0.172). Second, on DynaSent the base judge already beats the human reference, because the reference there is a single rating, and a single rating is a poor estimate of a distribution.

On Anecdotes, the judge trails 3 voters most on the items voters agree about

The per-tercile numbers carry the least intuitive result. On Anecdotes the gap between the LLM and 3 human votes is 0.182 on low-disagreement items, 0.141 on medium and 0.088 on high. On ChaosNLI the LLM's error is 2.0x the human sample on low-disagreement items, 2.8x on medium and 3.3x on high. The judge is not simply bad at hard cases. When humans nearly all agree, it still spreads probability across labels they did not choose; when humans split, a small human sample is itself noisy, and the judge's relative gap narrows on Anecdotes. Treat the ChaosNLI ratios and the Anecdotes differences as different measures: we computed both from the paper's tables, and they point in different directions on the high tercile.

NAPHA is built around that stratification. It predicts an entropy class for each item and routes the judge's soft label through an alignment model trained for that class. The authors report that with predicted classes, alignment on the low-entropy class gets worse than the base judge, and the effect disappears with oracle classes. Their own mitigation is to skip NAPHA on items classified as low entropy. The bottleneck is the entropy classifier, which the oracle column quantifies.

What it means for annotation and evaluation pipelines

Majority-vote validation hides the failure. If you validated an LLM judge by agreement with aggregated labels, you validated the part it does well. For any task where disagreement is signal (toxicity, ethics, helpfulness, sentiment with irony) the check that matters is distributional. The paper's human reference gives a practical yardstick: on these datasets the judge is worth somewhere between one rater (DynaSent) and fewer than three voters (Anecdotes), and well short of 20 crowd workers (ChaosNLI).

Keep collecting multiple annotations per item. Our judgement: the case for single-annotator labelling plus an LLM tie-breaker is weakest exactly where teams use it, on subjective items. You cannot measure a judge's soft-label error without human soft labels, and the paper's limitations section notes three of its five datasets have fewer than six annotators per item because larger panels are rare and expensive.

Judge validation studies rarely collect disagreement at all. ServiceNow's AgentJudgeBench (arXiv:2608.26623, 27 August 2026) finds six LLM judges converge to a 77% to 82% alignment band on hard tool-calling queries without ground truth. Its human check is 120 records scored by one annotator each, 92.7% agreement with the programmatic scorer, and the authors state they cannot report inter-annotator agreement. That is the norm this paper pushes against.

Budget for an entropy estimate. The oracle column says most of NAPHA's remaining gap is knowing which items are contested. A small multi-annotator pilot on your own data produces that signal; the rest of the corpus can then be routed. This matches how we run human-in-the-loop annotation: model first, humans where the model's confidence and the humans' agreement diverge.

Check it yourself

All numbers are in the arXiv HTML rendering. Table 3 has the dataset-level soft-label results, Table 5 the human sample, Tables 4, 6 and 7 the per-tercile values for Anecdotes, ChaosNLI and DynaSent, and Table 1 the hard-label F1.

open https://arxiv.org/html/2609.01073v1

# gap closed and ratios used above
python3 - <<'PY'
rows = {  # human sample, base judge, NAPHA predicted, NAPHA oracle
  "ChaosNLI":   (0.070, 0.200, 0.174, 0.153),
  "Anecdotes":  (0.165, 0.299, 0.272, 0.172),
  "SummEval":   (0.264, 0.371, 0.303, 0.263),
  "TopicalChat":(0.238, 0.374, 0.363, 0.312),
}
for k,(h,b,p,o) in rows.items():
    print(f"{k:12} predicted {100*(b-p)/(b-h):.0f}%  oracle {100*(b-o)/(b-h):.0f}%")
tercile = {"low": (0.077,0.038), "med": (0.206,0.073), "high": (0.316,0.097)}
print({k: round(l/h,1) for k,(l,h) in tercile.items()})   # ChaosNLI LLM/human
PY
# ChaosNLI     predicted 20%  oracle 36%
# Anecdotes    predicted 20%  oracle 95%
# SummEval     predicted 64%  oracle 101%
# TopicalChat  predicted 8%  oracle 46%
# {'low': 2.0, 'med': 2.8, 'high': 3.3}

ChaosNLI itself, with all 100 labels per item, is public at github.com/easonnie/ChaosNLI, so the entropy terciles can be rebuilt. The paper links no code. Its SummEval and TopicalChat soft-label rows are reported as averages over several rating dimensions with only three ratings per item behind them, so we treat those two rows as less directly comparable than ChaosNLI and Anecdotes.

What would prove this wrong

The claim rests on one closed-weights backbone for the main tables and on a human reference built by subsampling. It would be wrong if a replication with a different sampling rule (for example 5 fixed annotators per ChaosNLI item) put the human error above the LLM's 0.200, or if a newer judge prompted the same way scored below 0.100 on ChaosNLI without any alignment step.

A dated prediction: by 31 March 2027, no published LLM-as-judge result on ChaosNLI soft labels, prompt-only and without training on ChaosNLI annotations, will report DistCE at or below the 0.070 human-sample figure from this paper. One such result falsifies it.

Sources

1. Steindl, S., Voskarides, N., Gasparin, A., Marcheggiani, D. Post-hoc Alignment of LLM-judges to Human Judgment Distribution. arXiv:2609.01073v1, 1 September 2026. Tables 1 to 7; HTML version.
2. Verma, A., Saha, A. K., Subramanian, S., Aluru, S. H. AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling. arXiv:2608.26623v1, 27 August 2026. Abstract and Appendix G.
3. Nie, Y., Zhou, X., Bansal, M. What Can We Learn from Collective Human Opinions on Natural Language Inference Data? EMNLP 2020. Data: github.com/easonnie/ChaosNLI.
4. BLOMEGA. Data annotation research: the latest.