Twenty-one LLM judges from ten providers were tested across 118 runs and roughly 541,000 individual judgments on MT-Bench, JudgeBench and RewardBench. On MT-Bench, every judge's exact-match agreement with the reference labels beat its Cohen's kappa by 33.8 to 41.3 percentage points, a gap the authors call kappa deflation. The same judges' rankings were not stable across benchmarks: Llama 3.3 70B ranked 5th of 21 on MT-Bench and 19th on JudgeBench, a 14-position drop.
The paper is "Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias," by Justin D. Norman, Michael U. Rivera and D. Alex Hughes, posted to arXiv on 17 June 2026 as arXiv:2606.19544. The authors describe it as the largest systematic LLM-as-judge evaluation published to date, and the scale backs that up: 21 judges spanning OpenAI, Google, Anthropic, Meta, Alibaba, Mistral, MiniMax, DeepSeek, Moonshot and Zhipu, evaluated on three separate benchmarks (MT-Bench's 2,391 pairwise comparisons, JudgeBench, and RewardBench) across 118 runs, producing about 541,000 individual judgments.
The core argument is a measurement one. Teams that adopt an LLM as a judge, whether to grade annotator output, score a reward model, or QA a pre-labeling pass, usually validate it by checking how often the judge's label matches a reference label: percent agreement. Percent agreement does not subtract out the agreement you would get from chance alone, especially when most items in a dataset share a lopsided label distribution (for instance, when one response in a pair is usually the better one). Cohen's kappa corrects for that. The paper runs both metrics on the same 21 judges and the same roughly 541,000 judgments, and reports how far apart they land.
| Judge | Benchmark | Exact-match agreement | Cohen's kappa | Kappa deflation (points) | Source |
|---|---|---|---|---|---|
| Gemini 3.1 Pro | MT-Bench | 84.9% | 0.511 | 33.8 | Norman et al. 2026, Table 2 |
| GPT-4o-mini | MT-Bench | 80.9% | 0.396 | 41.3 | Norman et al. 2026, Table 2 |
| GPT-4o-mini | JudgeBench | 67.6% | 0.325 | 35.1 | Norman et al. 2026, Table 2 |
| GPT-4o-mini | RewardBench | 83.1% | 0.622 | 20.9 | Norman et al. 2026, Table 2 |
| Cohort (21 judges) | MT-Bench | not reported | not reported | 33.8 to 41.3 (mean 38.6) | Norman et al. 2026, §4.1 |
| Cohort (21 judges) | JudgeBench | not reported | not reported | 8.1 to 38.5 (mean 23.7) | Norman et al. 2026, §4.1 / Table 2 |
| Cohort (21 judges) | RewardBench | not reported | not reported | 5.9 to 21.3 (mean 10.4) | Norman et al. 2026, §4.1 / Table 2 |
GPT-4o-mini sits at the high end of the MT-Bench gap in that table: 80.9% exact-match agreement looks respectable on its own, but a kappa of 0.396 is the range conventionally labeled "fair" agreement, not "substantial." Applying the standard kappa identity, κ = (Po − Pe) / (1 − Pe), to GPT-4o-mini's own reported numbers (Po = 0.809, κ = 0.396) implies a chance-level agreement rate (Pe) of about 68.4%, meaning more than two-thirds of its raw agreement score would have happened even if the judge's labels were uncorrelated with the task beyond matching the base rate.
The mechanism is arithmetic, not mysterious, but it is easy to skip when a vendor report only shows one number. If a label distribution is skewed (most pairwise comparisons in MT-Bench have a response that is usually preferred), two independent raters who both lean toward the majority label will agree often purely by chance, inflating raw percent agreement. Kappa subtracts that chance component before crediting the judge.
The paper's second mechanism is what it calls the consistency-bias paradox: a judge can agree with itself almost perfectly on repeated runs while still leaning toward one answer position regardless of content. Qwen 3 8B had a test-retest reliability of 0.992 on MT-Bench and a position-bias score of 0.192. Gemini 2.5 Flash had test-retest of 0.988 and position bias of 0.125. Both numbers describe the same judge: it will give you the same verdict almost every time you ask it again (reassuring), and it is still measurably more likely to pick whichever response appears in one position over the other, independent of content (not reassuring). Across the 17-judge cohort the paper measured for consistency, mean test-retest reliability was 0.944 on MT-Bench and 0.913 on JudgeBench, well above the 0.95 threshold the authors flag as the point where position bias needs checking rather than assumed away.
Not reliably. The paper ranks all 21 judges separately on each of the three benchmarks and finds the order is not stable. Llama 3.3 70B ranked 5th on MT-Bench and 19th on JudgeBench, a drop the abstract, introduction, conclusion and the paper's own Figure 2 caption all describe as 14 positions (worth flagging: the paper's Table 5 and Section 4.3 instead list a JudgeBench rank of 20, a 15-position drop; the authors do not reconcile the two in the text, so a reader checking the primary source should expect that small internal inconsistency). Minimax M2.7 moved the other direction, from 16th on MT-Bench to 5th on JudgeBench, by the figures the authors repeat most often. Across the full cohort, only Gemini 3.1 Pro and Claude Opus 4.6 are reported to hold a top-three position on all three benchmarks (though Table 5 separately lists Opus 4.6 at 4th on RewardBench, another internal mismatch worth checking against the original tables before citing a specific rank).
If your QA step, pre-labeling audit, or reward-model check reports only percent agreement against a small golden set, that number is probably closer to the exact-match column in the table above than to the kappa column, and the gap between them on this paper's data ran 5.9 to 41.3 percentage points depending on the benchmark. Three decisions follow directly from the data rather than from general caution.
First, report Cohen's kappa or Krippendorff's alpha alongside any percent-agreement figure before deciding a judge is production-ready, and treat the chance-corrected number as the one that governs the decision, not the raw one. Second, do not transfer a judge's reputation from one benchmark or vendor demo to your own task: the rank reordering here, up to a 14-position swing for a single model between two general-purpose benchmarks, means a judge that is 5th-best on a public leaderboard is not guaranteed to be anywhere near 5th-best on your labeling task. Third, a judge passing a self-consistency check (the same answer on repeated runs) is not the same evidence as a judge being unbiased: Qwen 3 8B and Gemini 2.5 Flash both cleared 0.95 test-retest reliability while also clearing the 0.10 position-bias line in the same evaluation.
The paper and its numbers are public. To check the central claim without taking this write-up's word for it:
# Read the paper directly (HTML renders the tables; the PDF text is compressed)
open https://arxiv.org/abs/2606.19544
open https://arxiv.org/html/2606.19544
# Tables to check against the figures in this article:
# Table 2 and Table 4: exact-match agreement, Cohen's kappa, and kappa deflation per judge, per benchmark
# Table 3: MT-Bench position bias and verbosity bias per judge
# Table 5: cross-benchmark rank per judge
# Table 6: test-retest, self-consistency and flip rate for the 17-judge consistency cohort
# To reproduce the deflation math on your own judge-vs-human labels:
python3 -c "
from sklearn.metrics import cohen_kappa_score
# y_ref: human reference labels, y_judge: your LLM judge's labels, same order
exact_match = (y_ref == y_judge).mean()
kappa = cohen_kappa_score(y_ref, y_judge)
print('exact match:', exact_match, 'kappa:', kappa, 'deflation (points):', (exact_match - kappa) * 100)
"
The claim here is specific enough to fail. If, by 31 December 2027, a replication at comparable scale (at least 100,000 judgments, at least three benchmark families spanning preference-style and correctness-style labels, at least 15 judges from at least five providers) finds a mean MT-Bench-style kappa deflation under 10 percentage points, and finds that per-model rank shift across benchmark families stays under 5 positions for every judge tested, the pattern reported here would not hold for the next generation of judge models, and the "validate per task, report kappa, do not transfer rankings" guidance in this article would need to be dropped as a general rule rather than treated as still current.