Dropping the riskiest half of LLM-judge verdicts, by a label-free score, lifts accuracy 12.2 points

What changed, and when

On 30 September 2026, Yang Ba, Michelle V. Mancenido and Rong Pan, of Arizona State University's School of Computing and Augmented Intelligence, posted Can We Trust This Evaluation? Instance-Level Reliability of LLM-as-a-Judge (arXiv:2610.14029). The paper targets a specific gap in how LLM judges get used as annotators or QA gates in a pipeline: aggregate agreement numbers (a judge agrees with humans X% of the time) say nothing about which specific verdict, right now, in front of you, is one of the ones in the disagreeing fraction.

Their method, SCUR (Structured Cross-fitted Uncertainty Risk), is a per-item risk score computed entirely from the judge panel's own behavior: how much the panel disagrees with itself, how much each judge's stated confidence matches what a cross-fitted model of the panel would have predicted, and how much the verdict shifts under answer-order swaps or small rubric changes. None of these require a gold label.

The evidence table

SCUR is compared against four uncertainty baselines, predictive entropy (PE), vote entropy (VE), a Dawid-Skene-style entropy (DS entropy) and maximum softmax probability (MSP), on five benchmarks: Chatbot-Arena, HaluEval, MMLU, LLMBar and LLMEval2. Each cell reports AUROC (higher is better, at distinguishing correct from incorrect majority-vote verdicts) and Err@50 (the error rate among the riskiest half the score identifies, lower is better).

Source: arXiv:2610.14029, per-benchmark results table. Each cell is AUROC / Err@50; bold marks SCUR. SCUR has the best AUROC on all five benchmarks and the best (lowest) Err@50 on all five.

Averaged across the five benchmarks: PE, the strongest single baseline, reaches mean AUROC 0.673 and Err@50 0.174. SCUR reaches mean AUROC 0.702 and Err@50 0.147. The paper also reports a combined area metric, 1−AURC, where SCUR scores 0.841 against 0.818 for PE.

How a risk score gets computed with no ground truth

SCUR starts from a latent measurement model that decomposes each judge's verdict log-odds into item, judge and perturbation effects, then cross-fits the panel: each judge's prediction is estimated from the rest of the panel, not from itself, which stops a judge's own overconfidence from inflating its own reliability score. Three signals come out of this: preference uncertainty (U), the binary entropy of the mean cross-fitted probability; measurement cross-entropy (C), how much a judge's own stated confidence mismatches what the cross-fitted model expected from it; and perturbation sensitivity (S), how much the verdict moves when the answer order is swapped or the rubric wording is changed, split into a part shared across judges and a part specific to one judge.

Each signal is converted to a percentile against an unlabeled reference split, so the final score is scale-free across tasks. The two sensitivity percentiles are combined by taking their maximum, meaning an item is flagged as risky if it is sensitive to order or to rubric wording, either is enough. All three normalized signals are then fused with a complement-based geometric mean into the final SCUR score, which ranks items for automatic acceptance or routing to human review.

What it means for teams using LLM judges as a QA gate

A risk score is a triage tool, not a replacement for ground truth. SCUR does not tell you a verdict is correct, it tells you which verdicts are more likely to be wrong relative to the rest of the batch, so a team can route a fixed review budget to the highest-risk fraction instead of auditing a random sample or everything.

The gain compounds with how much review budget you have. The jump from 4.6 points at 80% retention to 12.2 points at 50% retention is not linear, which matches the intuition that the riskiest items are disproportionately wrong: pulling out the worst half removes much more than half the errors.

Confidence alone (MSP) is the weakest of the four baselines on four of five benchmarks. A judge's own stated confidence is the signal many pipelines already have for free, and it is outperformed by predictive entropy, a Dawid-Skene-style entropy, and SCUR on nearly every benchmark tested, including the full reversal on LLMBar where MSP's AUROC (0.498) is at chance.

Our judgement: this is one paper's benchmark suite, and SCUR's AUROC advantage over the strongest baseline (PE) is real but modest on four of five benchmarks (0.005 to 0.031 points), with the clearest separation on LLMBar (0.031) and the smallest on HaluEval (0.004). The retention-accuracy gains (4.6pp, 12.2pp) are averages across five benchmarks and five aggregation rules, not a single clean number, so the paper does not report whether any individual benchmark underperforms that average.

Check it yourself

All five benchmarks (Chatbot-Arena, HaluEval, MMLU, LLMBar, LLMEval2) are public. The comparison that matters is the per-benchmark table, not the mean alone, since SCUR's margin varies benchmark to benchmark.

open https://arxiv.org/abs/2610.14029          # abstract, authors, submission date
open https://arxiv.org/html/2610.14029         # full text: per-benchmark table, SCUR method section, retention-accuracy results

python3 - <<'PY'
# recompute the mean AUROC and Err@50 directly from the paper's per-benchmark table
rows = {
    "PE":  [.674, .728, .787, .623, .553],
    "VE":  [.625, .667, .744, .585, .523],
    "DS":  [.673, .674, .730, .640, .546],
    "MSP": [.647, .718, .693, .498, .603],
    "SCUR":[.696, .732, .794, .654, .632],
}
for method, vals in rows.items():
    print("%-5s mean AUROC = %.3f" % (method, sum(vals) / len(vals)))
PY
# PE    mean AUROC = 0.673
# VE    mean AUROC = 0.629
# DS    mean AUROC = 0.653
# MSP   mean AUROC = 0.632
# SCUR  mean AUROC = 0.702

These means match the paper's own stated averages for PE (0.673) and SCUR (0.702); VE, DS entropy and MSP means are this article's own arithmetic from the paper's per-benchmark table, not separately stated by the authors, and are flagged as such above.

What would prove this wrong

SCUR's AUROC advantage over predictive entropy is small on most benchmarks (as little as 0.004), and the paper tests five benchmarks chosen by the authors rather than a held-out benchmark suite selected by someone else. A judge panel setup that differs meaningfully from the paper's (a single judge instead of a panel, a judge family whose confidence calibration differs from the ones tested) could shrink or erase the advantage, since cross-fitting and perturbation sensitivity both assume a multi-judge panel to begin with.

A dated prediction: by 31 December 2027, an independent team deploying an LLM-judge QA gate in production will report whether routing by a SCUR-style risk score measurably reduces the error rate of accepted verdicts compared to routing by predictive entropy alone, on a benchmark not in this paper's five. If the gap closes to statistical noise outside these five benchmarks, the practical case for the added complexity of cross-fitting and perturbation testing weakens even if the AUROC numbers here are accurate.

Sources

1. Ba, Y., Mancenido, M. V., Pan, R. Can We Trust This Evaluation? Instance-Level Reliability of LLM-as-a-Judge. arXiv:2610.14029, 30 September 2026, Arizona State University. HTML version.
2. BLOMEGA. LLM judge sparse overlap and wrong decisions.
3. BLOMEGA. Judge-panel effective human size on ChaosNLI.
4. BLOMEGA. Data annotation research: the latest.