The 96.5% LLM-judge accuracy is 100% by construction on 429 of 600 items; where a human set the label, 52 of 60 judges lose to always saying "incorrect"

What arXiv:2610.05094 measured, and what it released on 4 October 2026

Laurène Vaugrante and Thilo Hagendorff (AI Safety Research Group, University of Stuttgart) posted How Much Do LLM-as-a-Judge Design Choices Matter? as v1 on 4 October 2026, under CC BY 4.0. It crosses three design choices over 10 reasoning models: gpt-oss 120b, GPT-5.5, GPT-5.6 Terra, Gemini 3.7 Flash, DeepSeek V4 Pro, Claude Haiku 4.5, Claude Sonnet 5, Kimi K3, Qwen 3.8 27B and GLM-5.3.

The rating task has Grok 4.1 generate 600 sentences per dimension. 501 sentiment and 521 toxicity sentences form the evaluation split; Prolific raters scored each on a 1-7 scale (10 per sentiment item, 15 per toxicity item, Krippendorff alpha 0.86 and 0.68, paid £9.95 an hour on average). Each of 120 judge designs per dimension crosses 3 prompt detail levels (base, short in-context, long in-context), 4 output scales (1-7, -3..3, 0-1, 0-100, all rescaled to 1-7) and the 10 models.

The classification task takes 600 text-only, non-multiple-choice questions from Humanity's Last Exam (HLE), answered by Gemini 3.1 Pro at temperature 0.7. 60 judge designs (10 models, 3 prompt levels, low or default reasoning effort) see the question, the reference answer and the response, and return correct or incorrect. Headline results: 96.5% mean accuracy (92.7% to 98.3%), leniency (the share of errors that are false positives) dropping from 81.3% to 52.4% with the long prompt, and a 56.1-point leniency gap between models.

Everything is on OSF project sra9e, public, created 23 September 2026 and last modified 1 October 2026: 14 files covering benchmarks, per-annotator ratings, every judge output, prompts and statistics. That release is what made the checks below possible. Two caveats from the API on 6 October 2026: the node has no licence set (node_license is null), and its description still carries an earlier title, "Choosing an LLM-as-a-judge".

Do the released files reproduce the paper, and what do they add?

They reproduce it. Rescaling the released judge scores to 1-7 and comparing with annotator_mean_score gives the paper's numbers to two decimals. The rows marked BLOMEGA are new.

Paper numbers from arXiv:2610.05094v1. BLOMEGA rows computed on 6 October 2026 from the OSF files sentiment_judge_ratings.xlsx, toxicity_judge_ratings.xlsx, *_benchmark_with_annotator_detail.xlsx and classification_judge_assessments.xlsx. The one-error gap (1,266 vs 1,267) sits in the near-agreement subset and comes from 26 judge cells we could not parse as true or false.

The accuracy split is the main finding. On the 429 items where all 60 designs agreed, the verdict became the label and every judge scores 100%. On the 124 near-agreement items the 90%+ majority became the label, so mean accuracy there (96.5%) is the mean agreement rate (0.965 in the file) restated. Of the 1,266 errors we count, 0 fall on unanimous items, 262 on near-agreement items and 1,004 on the 47 human-resolved items: 79.3% of all errors on 7.8% of the data.

On those 47 items the human resolved 36 as incorrect (76.6%). A judge that always says "incorrect" scores 76.6% there; the 60 designs average 64.4%, and only 8 beat the constant. The best, gpt-oss 120b with the long prompt at both effort levels, reaches 87.2%. Gemini 3.7 Flash, grading answers from a sibling Gemini model, scores 33.7% on these items.

Why a 96.5% accuracy is close to guaranteed by the labelling rule

Section 3.3.2 states the rule plainly: an item's label is the majority verdict whenever at least 90% of the 60 designs agree, and only the remainder is resolved by hand. The paper's limitations section acknowledges that this "weakens our accuracy-linked findings". What it does not quantify is how much of the headline is fixed in advance.

The arithmetic is short. Unanimous items contribute 429 correct verdicts per design. Near-agreement items contribute at least 90% by definition, so at least 111.6 of 124. Even if every judge got all 47 contested items wrong, mean accuracy would be (429 + 111.6) / 600 = 90.1%. At coin-flip on the contested items it would be 94.0%. The reported 96.5% sits 6.4 points above a floor the judges set for themselves, and the judges' spread (92.7% to 98.3%) is mostly a ranking of how often each design sides with the majority.

The base rate matters too. The answer key is 43.8% correct, so a constant judge scores 43.8% or 56.2% on the full set, and the full-set gap between real judges and constants is large. That part is genuine: on the 429 unanimous items, 60 different designs from 10 providers agreed on 162 correct and 267 incorrect answers, which is evidence of something real. Our judgement: the honest headline is "judges agree with each other on 92% of HLE items, and on the contested 8% they are worse than a constant", and a reader cannot see the second half from the abstract.

The same rule shapes leniency. On contested items the human chose "incorrect" 76.6% of the time, while judges said "correct" more freely, so most contested-item errors are false positives (78.2% leniency on that subset in Table 13). Leniency is a ratio of errors, and with few errors it swings. GPT-5.5 makes 0 errors on near-agreement items under two prompts, so its leniency there is undefined; DeepSeek V4 Pro with the base prompt makes 1, so its leniency is 100%.

What the 28.9-point and 56.1-point leniency gaps are in absolute errors

Converted from shares of errors to rates per judgment, the leniency effects shrink and show their cost. Each prompt level covers 20 designs, or 12,000 judgments.

The long prompt's 28.9-point leniency drop is 81 fewer false positives bought with 150 more false negatives across 12,000 judgments: total errors rise from 406 to 475. Whether that trade is good depends on what a false "correct" costs you, and the paper does not frame it that way. The 56.1-point model gap (97.2% for Gemini 3.7 Flash against 41.1% for GPT-5.6 Terra) is a 4.0-point gap in false-positive rate, 5.78% against 1.81%. Gemini 3.7 Flash is wrong in the lenient direction 208 times and the strict direction 6 times.

Is model identity really the dominant source of variance?

The paper never runs a variance decomposition; "dominant" rests on average pairwise differences. Its own Table 12 already cuts against the claim for ratings: averaged over both benchmarks, the text reports impact of 0.13 points for scale, 0.11 for model and 0.10 for prompt detail, and for toxicity alone scale averages 0.19 against 0.15 for model, with maxima of 0.93 against 0.53.

The design is fully crossed, so main-effect shares are well defined. We computed per-design bias and MAE from the released ratings and split the between-design sum of squares. Model leads for sentiment bias (37.7%, scale 13.3%, prompt 8.9%, interactions 40.0%). For toxicity bias, scale leads at 44.9% and model is second at 20.6%. Removing the -3..3 scale lifts the model share to 49.6% for sentiment and 50.0% for toxicity, which is the condition under which the abstract's claim is true.

For per-item error, model is never first: it explains 11.7% of MAE variance for sentiment (scale 51.0%) and 11.4% for toxicity (prompt 35.5%, scale 34.8%). In the classification task the claim holds strongly: model explains 56.0% of accuracy variance, 51.8% of leniency variance and 87.6% of false-positive-rate variance, against 0.0% to 0.3% for reasoning effort. The scale-only shift on toxicity exceeds 0.3 points in 18 of the 30 model and prompt combinations; for sentiment its maximum is 0.24 and it never exceeds 0.3.

How many Prolific raters is one judge worth on the rating task?

The 0.11-point headline is the absolute value of a signed mean over roughly 500 items, so over- and under-ratings cancel. A constant that outputs the human mean for every sentence has bias 0.000 and would beat all 240 designs on that metric, with a per-item MAE of 1.64 (sentiment) and 1.73 (toxicity). All 240 designs beat the constant on MAE, so the judges carry real signal, but bias alone cannot certify a judge. The pooled per-item MAE, 0.49, is 4.3 times the 0.11.

To put MAE on a human scale we held out a random 5 sentiment raters (7 for toxicity) as the reference, averaged k of the remaining raters, and repeated over 200 splits. One sentiment rater misses the held-out mean by 0.608 points, three by 0.425, four by 0.391; the median judge design scores 0.407. For toxicity, one rater misses by 0.979, three by 0.638, four by 0.581; the median judge scores 0.623. On both tasks the median design is worth between 3 and 4 raters; the best design (Gemini 3.7 Flash, short in-context prompt, 0-1 scale) reaches 0.323 on sentiment, better than 5 raters. That matches a pattern we measured on ChaosNLI in One 32-model judge panel, two answers, and it is the finding in this paper we would act on.

One data note: for 31 of the 521 toxicity items, the stored annotator_mean_score differs from the mean of that row's 15 rater columns, by up to 0.217 points. Sentiment has no such mismatch. The paper's toxicity numbers use the stored mean; the cause (perhaps excluded responses) is not documented in the release.

What this means if you validate an LLM judge for annotation work

Never grade a judge against a key the judges wrote. If consensus labels are needed for scale, report accuracy only on the independently labelled slice, and size that slice for the confidence you need. Here 47 items give a 95% interval of roughly plus or minus 14 points on a single design's accuracy; the comparison that matters (64.4% against a 76.6% constant) only appears there.

Report constant baselines on every subset. The always-"incorrect" judge scores 56.2% on the whole set and 76.6% on the contested items. For ratings, report per-item MAE next to a constant-mean baseline (1.64 and 1.73 here); a signed-bias summary such as 0.11 is satisfied by the constant.

Convert leniency to per-judgment rates and price them. A 28.9-point leniency drop that turns 81 false positives into 150 false negatives is a bad trade for a grading pipeline that only flags failures, and a good one for a pipeline that must never pass a wrong answer. Decide which error you pay for, then choose prompt and model.

Avoid centred scales for harm ratings. The -3..3 scale moves toxicity bias by more than 0.3 points in 18 of 30 combinations and up to 0.93. Use 1-7 with anchors or 0-100; both had mean toxicity |bias| of 0.13 and 0.11.

Budget one judge as about three raters, and keep humans on the contested tail. On these two rating tasks the median judge sits between 3 and 4 Prolific raters. On grading hard answers, the items where judges split are the items where they fail. For BLOLabel-style pipelines that means routing low-agreement items to people rather than adding judges, and keeping a human-labelled audit set that judges never touch. The paper's own recommendation to aggregate 2 to 3 models (35% to 37% lower bias) points the same way.

Check it yourself

The full OSF tree is about 29 MB; skip classification_raw_results_detail.xlsx (26.8 MB) and the rest is 3 MB. Needs pandas and openpyxl.

# list files (OSF API, checked 6 October 2026)
curl -s https://api.osf.io/v2/nodes/sra9e/files/osfstorage/
curl -s https://api.osf.io/v2/nodes/sra9e/ | python3 -c "import json,sys;print(json.load(sys.stdin)['data']['attributes']['node_license'])"   # None

# classification: subset accuracy and the constant baseline
curl -sL https://osf.io/download/6abea159d5015ea6d039da87/ -o cls.xlsx
python3 - <<'PY'
import pandas as pd
d=pd.read_excel("cls.xlsx")
f=lambda v:{"TRUE":1,"FALSE":0,1:1,0:0}.get(v.strip().upper() if isinstance(v,str) else v)
y=d.answer_gemini31pro_IS_CORRECT.map(f)
J=d[[c for c in d if c.startswith("gemini31pro__")]].apply(lambda s:s.map(f))
for s,g in d.groupby("classification_source").groups.items():
    acc=(J.loc[g].eq(y[g],axis=0)).mean()
    print(s,len(g),round(acc.mean(),3),"always-incorrect",round(1-y[g].mean(),3),"beat it",(acc>1-y[g].mean()).sum())
print("full-set correct share",y.mean())
PY
# agreement 429 0.999 always-incorrect 0.622 beat it 60   (0.999: unparseable cells count as misses)
# human 47 0.643 always-incorrect 0.766 beat it 8
# nearly agree 124 0.964 always-incorrect 0.274 beat it 60
# full-set correct share 0.438

# ratings: bias, MAE and the constant (rescale -3..3: x+4; 0-1: 1+6x; 0-100: 1+6x/100)
curl -sL https://osf.io/download/6abe8982465696b6ed8bfc29/ -o tox.xlsx
python3 - <<'PY'
import pandas as pd,re
r=pd.read_excel("tox.xlsx"); h=r.annotator_mean_score
f={None:lambda x:x,"3":lambda x:x+4,"1":lambda x:1+6*x,"100":lambda x:1+6*x/100}
B=[];E=[]
for c in r.columns[2:]:
    s=re.match(r"(base_prompt|short_incontext|long_incontext)(_(3|1|100))?_",c).group(3)
    x=f[s](pd.to_numeric(r[c],errors="coerce")); B.append(abs((x-h).mean())); E.append((x-h).abs().mean())
print(sum(B)/len(B),sum(E)/len(E),(h-h.mean()).abs().mean())
PY
# 0.161 0.609 1.732

Column names: rating files hold question, annotator_mean_score and one column per design named <prompt>[_3|_1|_100]_<model> (no suffix means 1-7). The classification file holds answer_gemini31pro_IS_CORRECT, classification_source (agreement, nearly agree, human), agreement_rate and 60 columns named gemini31pro__<prompt>__<model>[_lowreas]; verdicts mix strings and integers. The variance shares are main-effect sums of squares over the 120 (or 60) per-design values. The k-rater curve uses seed 0 and 200 splits. HLE itself is on Hugging Face as cais/hle, MIT licence, auto-gated.

What would prove this wrong

Our claim is that the classification headline measures inter-judge agreement, and that on independently labelled hard items most judge designs do no better than a constant. A full human labelling of the 553 consensus items would test it. If a human audit of a random 100 of those items finds the consensus label wrong on fewer than 2, the 96.5% is close to what an independent key would give and our objection is mostly formal. If it finds 5 or more wrong, the accuracy figure is inflated by at least that margin.

A dated prediction: by 31 March 2027, any revision or follow-up from these authors that labels all 600 HLE items independently of the judges will report mean judge accuracy below 93%, and leniency on the formerly unanimous items above 0%. If the independently labelled mean stays at or above 96%, we were wrong about how much the labelling rule supplied.

Sources

1. Vaugrante, L., Hagendorff, T. How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models. arXiv:2610.05094v1, 4 October 2026, CC BY 4.0. Sections 3.1 to 3.3, 4.1, 4.2, 5.1, Tables 1, 2, 7, 8, 12, 13, 14, 15, 16, 17.
2. Vaugrante, L., Hagendorff, T. Comparing_LLMjudges, OSF project sra9e (benchmarks, judge results, prompts, statistics). Created 23 September 2026, modified 1 October 2026, no licence set. Accessed 6 October 2026.
3. Phan, L., Gatti, A., Han, Z., Li, N., et al. Humanity's Last Exam. arXiv:2501.14249, 2025. Dataset cais/hle (MIT, gated), checked 6 October 2026.
4. Verga, P., et al. Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models. arXiv:2404.18796, 2024.
5. BLOMEGA. One 32-model judge panel, two answers: worth 4.24 humans, or worth 2.30.
6. BLOMEGA. An LLM judge matches the majority label, but predicts human disagreement worse than 3 voters.

Questions readers ask about this study

Is the 96.5% LLM-judge accuracy in arXiv:2610.05094 an independent measurement?

Mostly not. The labels for 553 of the 600 HLE answers were set by the 60 judge designs themselves: unanimous verdicts (429 items) and 90%+ majorities (124 items) became the answer key. Judges score 100% and 96.5% on those subsets by construction, and mean accuracy cannot fall below 90.1% whatever they do on the rest. On the 47 items a human resolved, the 60 designs average 64.4%, and 52 of them score below the 76.6% of a judge that always says incorrect.

Does a 0.11-point mean bias mean an LLM judge rates items like humans do?

No. The 0.11 is the absolute value of a signed average over about 500 items, so errors cancel. A constant that outputs the human mean has bias 0.000 and beats all 240 designs on that metric, with per-item error of 1.64 and 1.73 points. The judges' per-item MAE is 0.37 (sentiment) and 0.61 (toxicity), and the median judge lands between the average of 3 and 4 Prolific raters against a held-out human reference.

Which LLM-judge design choice matters most for ratings?

It depends on task and metric. For toxicity bias the rating scale explains 44.9% of between-design variance and the model 20.6%; the -3..3 scale alone shifts bias by up to 0.93 points. Model identity leads for sentiment bias (37.7%), for toxicity bias once the -3..3 scale is dropped (50.0%), and for false-positive rate in the classification task (87.6%).