Half the "human" in an LLM-judge hardness study is SummEval's crowd: with experts only, consistency drops from 50.7% to 23.6% human-hard

What arXiv:2610.02877 claims, as posted on 2 October 2026

Longwei Cong, Sonja Hahn, Sebastian Gombert, Leon Camus, Fabian Zehner, Hendrik Drachsler and Ulf Kroehne (DIPF Leibniz Institute for Research and Information in Education, with affiliations at ZIB, Goethe University Frankfurt and Chemnitz University of Technology) posted Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty as v1 in cs.CL. The arXiv comment says it is accepted at AACL-IJCNLP 2026. The work is funded through the Leibniz project ALwAI (T163/2024).

The method fits a rating-scale Many-Facet Rasch Model (MFRM) separately to human and LLM ratings of the 1,600 SummEval summaries (100 CNN/DailyMail articles, 16 systems) on coherence, consistency, fluency and relevance. "Residual hardness" for a summary-dimension unit is the mean absolute difference between observed and model-expected ratings over a panel's raters. Each unit is then called hard if it sits above its panel's overall median.

The LLM panel is 17 models from 0.8B to 14B parameters (Qwen3.5, Phi-3 and Phi-4, Llama 3.1 and 3.2, Gemma 3, Ministral 3, OpenChat 3.5), greedy decoding, one prompt. The human panel is all eight SummEval annotation positions: three experts and five crowd workers, which the paper describes as "anonymized rater IDs" (Section 3.1).

Headline results: latent quality aligns at Spearman 0.371 between panels, residual hardness at 0.089 over 6,400 units (Table 7). Consistency shows an LLM-hard shift of +23.9 points, coherence a human-hard shift of -22.5 (Table 2). A logistic model predicts human-easy/LLM-hard units at AUROC 0.631 (Table 4). The paper links no code, no LLM ratings and no fitted parameters.

Does the public SummEval file reproduce the paper's human numbers?

Yes, closely enough that we are confident the paper used this exact file with list position as rater identity. Our own MFRM (adjacent-category logit, weak L2, L-BFGS in SciPy rather than the paper's AdamW in PyTorch) lands on the same fit statistic and the same hard-rates. The rows below the line are new.

Paper numbers from arXiv:2610.02877v1. BLOMEGA rows computed on 5 October 2026 from model_annotations.aligned.jsonl (1,600 lines, 5,839,062 bytes, SHA-256 f0d4166e...d3d6ab) linked from the Yale-LILY/SummEval README. Hard means above the panel's overall median residual hardness, the paper's definition.

Two things stand out in the chart. Our 8-position refit sits on top of the paper's human bars (57.3, 50.7, 48.9, 43.0 against 57.3, 50.8, 48.9, 43.0). And the expert-only refit is a different shape altogether: coherence 76.4% and relevance 72.7% hard, consistency 23.6% and fluency 27.3%. The experts' ranking of dimensions is close to one minus their share of 5/5 ratings (74.1, 13.2, 16.9 and 73.1%), which is what a median split on residuals does when one dimension sits at the ceiling.

The paper did run an expert-only check (Section 4.1, Table 7): quality alignment rises to 0.688 against the 7 judges of 7B or more, residual alignment stays at 0.092, and the consistency gap "remains +26.2" points. It reports no per-dimension rates for that comparison. If our expert fit matches theirs, the implied consistency LLM-hard rate for the larger judges is about 49.8% (23.6 + 26.2), down from 74.7% for the full panel. That is our inference, not a reported number.

Why the 8-rater human panel mostly measures the crowd

An MFRM gives every rater one severity parameter and assumes each rater is the same person across all 1,600 summaries. That holds for the three expert positions. For the five crowd positions the file gives no worker IDs; the SummEval authors confirmed on GitHub (issue 41, 2 March 2023) only that each annotation dictionary is one person across the four dimensions. Whatever crowd worker sits in slot 3 for summary 1 need not be the one in slot 3 for summary 2, so "rater severity" for those five slots is an average over unknown people.

The crowd ratings also carry almost no expert signal. Per summary, the mean of the five crowd ratings correlates with the mean of the three experts at Spearman 0.039 on coherence, -0.010 on consistency, 0.051 on fluency and 0.091 on relevance. On the 1,306 consistency units where all three experts gave 5, the crowd gives 5 only 28.1% of the time, with a mean of 3.82. In 807 of 1,600 rows, all five crowd workers submitted the identical four-dimension vector, which inflates crowd-only alpha to a near-uniform 0.478 to 0.495 on every dimension. Those 807 rows all come from the 12 older systems (62 to 79 rows each); the four pretrained-model systems (M17 T5, M20 GPT-2, M22 BART, M23 PEGASUS) have none.

Put five such positions next to three experts and the residuals are dominated by expert-crowd disagreement. Our 8-position human hardness correlates with the spread of the three experts at only 0.053 (coherence) to 0.207 (consistency); the expert-only hardness correlates at 0.573 to 0.665. On the 1,306 consistency units the experts unanimously rate 5/5, the 8-position model still labels 46.6% "human-hard". The expert-only model labels 9.2%.

Our judgement on the headline claims: the low human-LLM hardness correlation (0.089) is partly built in, because a human hardness signal that is mostly crowd noise cannot correlate strongly with anything. The paper's expert-only check (0.092) shows weak alignment survives the swap, which is the stronger evidence and deserves to be the main result. The coherence "human-hard shift" looks most fragile: experts disagree on coherence (pairwise exact agreement 34.6%, 44.9% of units spread by 2 or more points), so with experts only, coherence becomes 76.4% human-hard, and the gap against the LLM panel's 34.8% would likely grow, but it would then describe genuine expert ambiguity rather than crowd noise.

Is "LLM-hard consistency" a ceiling effect? Partly, and a constant judge shows how much

Consistency on SummEval is a near-binary task for experts: 4,168 of 4,800 expert ratings are 5, all three experts give 5 on 81.6% of summaries, and on extractive systems the share of 5s is 98.6% (abstractive 82.9%). A panel that agrees with the experts would give nearly every unit a 5, the fitted model would expect a 5, and residuals would be close to zero. Expert interval alpha on consistency is the highest of the four dimensions at 0.899, but nominal alpha is 0.535 and ordinal 0.796, because almost all the variance sits in a small tail of factual errors.

That makes a trivial baseline informative. A judge that always answers 5 on consistency matches 86.8% of individual expert ratings and 89.9% of expert majority labels, with mean absolute error 0.34 against the expert mean. One expert predicting the rounded mean of the other two reaches 90.6% exact and 0.131 MAE. On fluency the constant judge matches 83.1% and 88.7%; on coherence only 25.9% and 31.1%.

The paper does not report raw LLM agreement per dimension, so we cannot say whether its judges beat the constant. What the numbers do say: on consistency, an LLM panel that spreads its ratings across 3, 4 and 5 for factually clean summaries will show large residuals against its own fitted model on the same units where humans (experts) show none. That is a real behavioural gap, consistent with the paper's factual-verification reading (Section 5), but it is a calibration gap on a saturated scale rather than evidence that particular summaries are hard to verify. Our judgement: the paper's SHAP features for consistency (digit count, sentence count, negation count, system type) plausibly track the same split that drives the ceiling, since only 1.4% of expert consistency ratings on extractive summaries are below 5 against 17.1% on abstractive ones.

A consistency check from the paper's own tables: with Table 2's rates, independent hardness labels would put about 588 consistency units in the human-easy/LLM-hard quadrant; Table 4 reports 571. Across all four dimensions independence predicts 1,605 and Table 4 reports 1,494. The quadrant the RQ3 classifier learns is, to within 7%, the size two unrelated median splits would produce.

What this means if you validate an LLM judge against human ratings

Audit the human panel before you fit anything to it. SummEval's crowd columns correlate near zero with its experts and half the rows hold five identical vectors. Any psychometric model that pools them treats that as rater behaviour. The SummEval paper itself (Fabbri et al., TACL 2021) evaluated metrics against the expert annotations; follow-up work that pools all eight positions inherits a panel whose majority is weakly tied to the construct.

Require stable rater IDs for any rater-facet model. MFRM severity terms assume one person per column. If your annotation tool does not log a persistent anonymised worker ID per rating, collect it now; without it, rater-level fit and misfit diagnostics (the paper's Appendix A.2.5) are not interpretable for crowd data.

Report a constant baseline per dimension. On saturated dimensions such as SummEval consistency (86.8% 5s) and fluency (83.1%), raw agreement above 85% is free. Score the judge on the minority class: recall on the units experts flag below 5, which here is 294 of 1,600 consistency units without a unanimous 5.

Route by dimension, but with expert-only references. The paper's practical advice, to send consistency cases to source-grounded checks or human review, holds up under the expert-only refit: experts find consistency the easiest dimension (23.6% hard) while the LLM panel finds it the hardest. For coherence, the routing signal is the opposite of what a crowd-pooled panel suggests: it is the experts who disagree, so extra human votes there buy less than a sharper rubric.

Ask for the LLM ratings. The paper's 17 x 6,400 rating matrix would let anyone rerun the comparison against experts only and check judges against the always-5 floor. It was generated locally with greedy decoding, so publishing it costs little.

Check it yourself

One public file, about 5.8 MB, no LLM calls. The analysis ran in 13 seconds on a laptop.

# human annotations linked from github.com/Yale-LILY/SummEval README ("Human annotations")
curl -s -o ann.jsonl https://storage.googleapis.com/sfr-summarization-repo-research/model_annotations.aligned.jsonl
shasum -a 256 ann.jsonl     # f0d4166e0cdeb439b387c4449634b067b7e9f721b9ef4c2b722f6b7550d3d6ab
# fields per line: id, decoded, expert_annotations[3], turker_annotations[5], references, model_id, filepath
# each annotation: {coherence, consistency, fluency, relevance} in 1..5; no rater ID field

python3 - <<'PY'
import json, numpy as np
from scipy.stats import spearmanr
R=[json.loads(l) for l in open("ann.jsonl")]
D=["coherence","consistency","fluency","relevance"]
E=np.array([[[a[d] for d in D] for a in r["expert_annotations"]] for r in R])
T=np.array([[[a[d] for d in D] for a in r["turker_annotations"]] for r in R])
print("expert share of 5:", (E==5).mean((0,1)).round(3))
print("crowd share of 5:", (T==5).mean((0,1)).round(3))
print("crowd vs expert rho:", [round(spearmanr(T[:,:,k].mean(1),E[:,:,k].mean(1)).statistic,3) for k in range(4)])
print("rows with 5 identical crowd vectors:", sum((t==t[0]).all() for t in T))
PY
# expert share of 5: [0.259 0.868 0.831 0.269]
# crowd share of 5: [0.237 0.279 0.201 0.284]
# crowd vs expert rho: [0.039, -0.01, 0.051, 0.091]
# rows with 5 identical crowd vectors: 807

The MFRM refit: rating-scale adjacent-category model with summary quality, rater severity (one per list position), dimension severity and four ordered thresholds, maximum likelihood with L2 penalty 0.001, centred facets, fitted with scipy.optimize.minimize(method="L-BFGS-B") on the analytic gradient (270 iterations for 8 positions, 237 for experts only). Residual hardness is the mean absolute residual across the panel's raters per summary-dimension unit; hard means above the panel median over all 6,400 units. Extractive systems are M0, M1, M2 and M5, matching the SummEval README; with that coding Table 3's human row reproduces exactly. Release check: the arXiv abstract page and PDF contain no repository link, and GitHub repository searches for "residual judging difficulty", "summeval rasch" and "residual hardness" returned nothing from the authors on 5 October 2026. The Hugging Face copy mteb/summeval holds only per-summary expert means (100 rows, lists of 16), so it cannot support a rater-facet model.

What would prove this wrong

Our claim is that the paper's human residual hardness is driven mainly by SummEval's crowd positions, and that against experts alone the dimension pattern is governed by rating saturation. Two outcomes would contradict it.

First, if the authors release their LLM rating matrix and, refitting against the 3 experts only, the consistency LLM-hard minus human-hard gap shrinks below +10 points for the full 17-judge panel, then the crowd was masking agreement rather than disagreement and our reading is wrong. We predict it will exceed +40 points (74.7 against 23.6, give or take the refit).

Second, a dated prediction: by 30 September 2027, any study that reruns residual-hardness alignment on SummEval with experts only and a judge panel including at least one model above 30B parameters will report a consistency median-split human-hard rate below 30%, whatever it reports for the LLM side. A rate above 40% would mean our refit does not generalise.

Sources

1. Cong, L., Hahn, S., Gombert, S., Camus, L., Zehner, F., Drachsler, H., Kroehne, U. Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty. arXiv:2610.02877v1, 2 October 2026 (accepted at AACL-IJCNLP 2026). Sections 3.1, 4.1, 4.2, Tables 1 to 4, 7 to 10, Appendix A.
2. Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., Radev, D. SummEval: Re-evaluating Summarization Evaluation. TACL 9:391 to 409, 2021.
3. Yale-LILY. SummEval repository and README, human annotations file model_annotations.aligned.jsonl. Accessed 5 October 2026.
4. Yale-LILY/SummEval issue 41, "Identifying individual annotators", answer by Alex Fabbri, 2 March 2023.
5. MTEB. mteb/summeval on Hugging Face, last modified 3 May 2025. Accessed 5 October 2026.
6. Eckes, T. Introduction to Many-Facet Rasch Measurement. Peter Lang, 2015.
7. BLOMEGA. One 32-model judge panel, two answers: worth 4.24 humans, or worth 2.30.
8. BLOMEGA. An LLM judge matches the majority label, but predicts human disagreement worse than 3 voters.

FAQ

Does arXiv:2610.02877 show LLM judges find factual consistency harder than humans do?

It shows a gap against a mixed panel. Table 2 reports 74.7% of consistency units LLM-hard against 50.8% human-hard, using 3 SummEval experts plus 5 crowd positions as the human panel. We refit that human model from the public file and matched it (50.7%). Fitting the 3 experts alone, the consistency human-hard rate is 23.6%, so the gap against experts is larger in hard-rate terms, but 86.8% of expert consistency ratings are 5/5, which makes low residuals for humans close to automatic.

How reliable are the SummEval crowd annotations?

Weak as a measurement of the expert construct. In the public model_annotations.aligned.jsonl, the per-summary mean of the 5 crowd ratings correlates with the expert mean at Spearman 0.039 (coherence), -0.010 (consistency), 0.051 (fluency) and 0.091 (relevance). In 807 of 1,600 rows the five crowd rating vectors are identical across all four dimensions, all from the 12 older systems. The file carries no worker IDs.

How does an always-5 judge score on SummEval consistency?

It matches 86.8% of individual expert consistency ratings and 89.9% of expert majority labels, with mean absolute error 0.34 against the expert mean. One expert predicting the rounded mean of the other two matches 90.6%. Any LLM judge agreement figure on consistency should be read against that floor; arXiv:2610.02877 reports no raw agreement per dimension.