The 25% wrong-decision rate for LLM judges at 5% overlap is 33% once ten trivially rejected judges leave the average

What did the Adobe paper measure, and when?

On 25 September 2026 Junxuan Li, Arko Mukherjee (Adobe) and Soumyabrata Pal (Adobe Research) posted LLM Judge Validation Under Sparse Overlap: From Inference to Design (arXiv:2609.31857v1, cs.AI). It formats itself as a NeurIPS submission, checklist included.

The setting is the one most teams are actually in. The judge labels all n items for almost nothing, one primary human labels all of them too, and every further human labels only a fraction ρ of the corpus. The judge is then accepted by the leave-one-out rule used by Calderon et al. (2025) and others: hold out each human in turn, compare judge-vs-pool agreement against human-vs-pool agreement with a tolerance ε = 0.05, and accept when at least half of the K comparisons favour the judge (Eqs. 5 and 6).

Three claims carry the paper. Sampling variance of every agreement score scales as γ_F/m, with the constant blowing up for Cohen's κ and Krippendorff's α under label skew but staying bounded for raw observed agreement (Theorem 1). A Gaussian recipe gives the overlap needed to certify one judge or rank several (Eq. 7, Theorem 2). A zero-cost stratified allocation, where every secondary human labels the same proportional sample drawn by the primary annotator's labels (STRAT), halves false rejections when the strata carry signal (Section 3, Table 3). Ten judges are tested: five inherited from Calderon et al. and five new ones (GPT-5.4, GPT-5.2, GPT-4o, Claude Opus 4.5, Claude Sonnet 4.5, 28,822 new labels) on WAX, CeBaB aspects, CeBaB stars and SummEval.

Where do the 25% and 65% come from?

Neither headline number is synthetic. Both are averages over the four real evaluation matrices at ρ = 0.05. The 25% is the Random row of Table 3, mean wrong-decision rate over 40 judge-benchmark pairs. The 65% is the Mean column of Table 2, probability that the top-ranked judge on the sparse matrix is not the top judge on the dense one. We rebuilt both from the per-judge Appendix Tables 13, 14 and 15 and the SummEval note in Appendix C.3.1, and they reconcile to the third decimal.

Table 3 defines FR over 20 pass judges (ω ≥ 0.60), FA over 16 reject judges (ω < 0.50) and WDR over all 40. The arithmetic check: (20 × 0.290 + 4 × 0.597 + 16 × 0.112) / 40 = 0.2495. Ten of the 16 reject judges are SummEval judges with 0% error, so FA over the other six is 16 × 0.112 / 6 = 0.299.

SummEval is not a weak benchmark for this purpose, it is an easy one: with 3 humans, 6,400 dimension-level items and every one of the ten judges well below the human ceiling (dense ω = 0), no sampling noise at 5% overlap is large enough to flip a verdict. It adds ten guaranteed zeros to the average. The paper states this fact in Appendix C.3.1 and still reports the pooled mean in the abstract. Our judgement: the pooled figure is correct arithmetic and an optimistic headline. A practitioner whose judge sits anywhere near the human ceiling should read 33% to 46%.

Why is a judge just below the bar harder to catch than one just above it?

We wrote our own Monte Carlo of the synthetic setup in Section 4.1, Appendix B.2 and Table 7: n = 500 items, K = 4 humans at accuracy p_h = 0.85, L = 2 balanced labels, symmetric noise, ε = 0.05, primary annotator dense, the other three Bernoulli(ρ), the dense-matrix decision as reference. It matches Figure 2 (uniform, Random, observed agreement) within 4 percentage points in every cell: 10.8% against the paper's 12% for p_j = 0.85 at ρ = 0.05, 33.7% against 33% for p_j = 0.70, 2.2% against 3% for p_j = 0.90. The largest gap is p_j = 0.60 at ρ = 0.05, ours 10.9% against 7%, which is about 2.5 standard errors at the paper's 300 trials. So the synthetic grid is reproducible from the text alone. What the synthetic grid does not do is produce 25%: its worst p̂_o cell is 33% and most cells are under 12%. The 25% belongs to the real benchmarks.

The same simulation exposes something Figure 2 shows and the paper does not discuss. Under this noise model p_j = 0.85 sits 0.05 above the acceptance line and p_j = 0.70 sits 0.055 below it, almost the same distance, yet at ρ = 0.05 the paper reports 12% and 33% wrong decisions for them. The cause is the aggregation rule, not the noise.

With K = 4 and "accept iff ω ≥ 0.5", a tie of 2 votes to 2 accepts. A good judge has to lose three of its four per-rater comparisons to be rejected. A bad judge only has to win two to be approved. Sampling noise at small m therefore helps bad judges more than it hurts good ones. In our runs at m = 25 items per secondary annotator, a judge 0.05 above the bar is mishandled 9.2% of the time and a judge 0.05 below it 37.0% of the time, a four-fold gap at equal distance. The paper's Table 3 shows the real-data version once SummEval is set aside: false approvals run at 29.9%, as high as false rejections at 29.0%, even though the paper's text calls FA rates "lower (7 to 17%)".

This also explains the STRAT trade-off the paper reports in Section 4.4. STRAT corrects an upward bias in the random-spread human ceiling. A lower, unbiased ceiling is easier for any judge to clear, so false rejections fall (29.0% to 14.2%) and false approvals rise. Across the six non-trivial reject judges the rise is from 29.9% to 45.8% by our recomputation; on WAX, Mistral-v0.3, a judge with dense ω = 0, goes from 7.0% to 33.0% false approval (Table 14).

How many co-labelled items does a judge validation need?

The paper's planning answer is ρ ≥ 0.25 with STRAT for non-borderline judges, and its Gaussian recipe (Theorem 2 ii) bounds the per-rater false-rejection rate. Neither tells you the item count for a target wrong-decision rate on the whole decision, or anything about false approval, which Theorem 2 does not cover. So we computed it. For each signed margin (judge-vs-human agreement gap plus ε), we found the smallest per-secondary-annotator overlap m after which the wrong-decision rate stays at or below 5% on a grid from 10 to 2,000 items, 2,000 trials per cell.

Setup: K = 4 humans at p_h = 0.85, L = 2 balanced labels, ε = 0.05, n = 2,000 (so m = 2,000 is the dense matrix), exactly m items per secondary human, truth = population decision. Margin = 0.7 p_j − 0.545 under this noise model. The recipe column applies m = 1.96² σ₀² / margin² with σ₀² the per-item variance of [judge = r1] − [rk = r1]; applying it to negative margins is our extension. Grid resolution means each m is an upper bound to within one grid step.

Budget for the side you fear. At a margin of 0.05, 75 shared items protect a good judge but a bad judge needs 300. At n = 500 those are ρ = 0.15 and ρ = 0.60. The paper's ρ ≥ 0.25 rule covers the first and not the second. If a false approval costs more than a false rejection, which is the usual case for a judge about to replace human review, plan from the orange bars.

Count items, not percentages. Table 17 of the paper shows reliability tracks m = ρn rather than ρ. SummEval's near-perfect ranking at ρ = 0.05 is 320 items per rater, WAX's is about 12. A "5% overlap" policy means very different things on those two corpora.

The Eq. 7 recipe is conservative for pass verdicts. It asks for 392 items at margin +0.05 where the simulated whole-decision rule needs 75, because it bounds each per-rater comparison at 2.5% while the 2-of-4 rule tolerates two failed comparisons. Use it as a ceiling, not a target. It is not a bound on false approval at all.

Consider changing the rule, not only the budget. The asymmetry comes from ties going to accept at even K. The paper's own Appendix E.2 suggests raising the bar to ω ≥ 0.6; with K = 4 that means 3 of 4 votes, which moves the asymmetry to the other side. With odd K (3 or 5) a strict majority removes the tie entirely. We did not simulate these variants; that is the next check.

Treat ω near 0.5 as inconclusive. Here we agree with the paper outright. Its four borderline judges stay at 30% to 44% wrong under STRAT even at ρ = 0.50 (Table 4), and our ±0.02 rows need 800 to 1,600 items. Related notes: how many humans a judge panel is worth on ChaosNLI and LLM annotators that agree with each other and nobody else.

Check it yourself

No code is released. The NeurIPS checklist answers [Yes] to open access (question 5), but its own justification reads: "this submission does not provide an anonymized code release or a single bundled reproduction package." Question 8 refers to scripts in code/experiments/ that are not linked anywhere. The arXiv abstract page carries no repository link, and a GitHub search on 30 September 2026 for the title and the arXiv id returned 0 repositories. So the checks below use only the paper's text.

curl -sL https://arxiv.org/pdf/2609.31857 -o p.pdf && pdftotext -layout p.pdf p.txt
grep -n "Mean FR" -A 22 p.txt        # Table 3
grep -n "Table 13\|Table 14\|Table 15" p.txt   # per-judge rates
curl -s "https://api.github.com/search/repositories?q=2609.31857" | grep total_count   # 0 on 2026-09-30

# 1) Recompute the headline from per-judge rows at rho = 0.05 (Random, Strat)
python3 - <<'PY'
FR = {"stars":[(.130,.037),(.113,.020),(.153,.053),(.233,.067),(.270,.113),(.247,.143),(.287,.140),(.423,.177),(.503,.330)],
      "wax":[(.370,.033),(.553,.180),(.637,.473),(.540,.290),(.583,.313),(.783,.543),(.733,.507)],
      "asp":[(.097,.090),(.163,.090),(.147,.073),(.203,.067),(.230,.157),(.213,.117),(.217,.200),(.367,.380)]}
FA = {"stars":[(.433,.567)], "wax":[(.127,.457),(.157,.517),(.070,.330)], "asp":[(.643,.727),(.363,.150)],
      "summeval":[(0,0)]*10}          # App. C.3.1: all 10 SummEval judges dense-reject, 0% wrong
rows = [(b,r) for d in (FR,FA) for b,v in d.items() for r in v]
for i,name in ((0,"Random"),(1,"Strat")):
    m = lambda xs: sum(xs)/len(xs)
    print(name, "WDR40=%.3f" % m([r[i] for b,r in rows]),
          "WDR30=%.3f" % m([r[i] for b,r in rows if b!="summeval"]),
          "FA6=%.3f" % m([r[i] for b,v in FA.items() if b!="summeval" for r in v]))
PY
# Random WDR40=0.250 WDR30=0.333 FA6=0.299
# Strat  WDR40=0.184 WDR30=0.245 FA6=0.458

# 2) Monte Carlo of the leave-one-out rule (App. B.2 noise model, Eqs. 3-6)
python3 - <<'PY'
import numpy as np
def decide(A, D, eps=0.05):                 # A: labels, last col = judge; D: human mask
    K = D.shape[2]; v = 0
    for k in range(K):
        nj = nh = den = 0
        for q in range(K):
            if q == k: continue
            w = D[:,:,k]*D[:,:,q]; den = den + w.sum(1)
            nj = nj + (w*(A[:,:,K]==A[:,:,q])).sum(1); nh = nh + (w*(A[:,:,k]==A[:,:,q])).sum(1)
        den = np.maximum(den,1); v = v + (nj/den >= nh/den - eps)
    return v/K >= 0.5                       # accept iff omega >= 0.5 (2 of 4 is enough)
def run(pj, rho, n=500, K=4, ph=0.85, T=2000, seed=0):
    r = np.random.default_rng(seed); y = r.integers(0,2,(T,n))
    A = np.where(r.random((T,n,K+1)) < np.array([ph]*K+[pj]), y[...,None], 1-y[...,None])
    D = np.ones((T,n,K)); dense = decide(A, D)
    D[:,:,1:] = r.random((T,n,K-1)) < rho   # r1 dense, others Bernoulli(rho)
    return 100*np.mean(decide(A, D) != dense)
for pj in (0.95, 0.90, 0.85, 0.70, 0.60):
    print(pj, [round(float(run(pj, rho)),1) for rho in (0.05, 0.10, 0.25, 0.50)])
PY
# 0.95 [0.3, 0.0, 0.0, 0.0]     paper Fig. 2: 0 0 0 0
# 0.9  [2.2, 0.4, 0.0, 0.0]                   3 1 0 0
# 0.85 [10.8, 7.2, 2.2, 0.6]                  12 7 4 1
# 0.7  [33.7, 23.2, 9.4, 2.9]                 33 24 7 3
# 0.6  [10.9, 2.4, 0.2, 0.0]                  7 2 0 0

The 65% needs no simulation: (0.670 + 0.887 + 0.875 + 0.177) / 4 = 0.652 from Table 2, and 0.811 without SummEval. For the planning table, set every secondary human to exactly m items (a random permutation per trial), use n = 2,000 and compare against the population decision, margin > 0 means accept. The only parameter we had to choose is the trial count: the paper uses 300, we used 2,000 per cell for the planning table.

What would prove this wrong

Our asymmetry claim is specific: under the paper's own rule (K = 4, accept at ω ≥ 0.5), a judge at a given distance below the bar needs roughly four times the overlap of one at the same distance above it (300 against 75 items at 0.05). If the authors release the code/experiments/ scripts and, run on the Appendix B.2 settings, they show wrong-decision rates for p_j = 0.70 and p_j = 0.85 that differ by less than a factor of 2 at ρ = 0.05, our simulation has the rule wrong and this note should be corrected. Figure 2 already reports 33% against 12%, which is why we expect it to hold.

A dated prediction: by 30 June 2027, any published replication of this pipeline on a benchmark with at least five reject judges that sit within 0.1 of the human ceiling will report mean false approval above 20% under random allocation at 5% overlap, and higher under STRAT than under random. If it comes in under 15% for random, the 29.9% we recomputed from six judges was a small-sample accident of WAX and CeBaB, and the paper's "FA rates are lower" reading was right.

The synthetic reproduction is falsified if the released code gives any Figure 2 cell for observed agreement, uniform prevalence, random design more than 5 percentage points from the paper's printed value, which would mean the figure and the described generator disagree.

Sources

1. Li, J., Mukherjee, A., Pal, S. LLM Judge Validation Under Sparse Overlap: From Inference to Design. arXiv:2609.31857v1, 25 September 2026. Eqs. 3 to 7, Theorems 1 and 2, Figure 2, Tables 1 to 5, Appendix B.2, Tables 7, 13 to 17, Appendix C.3.1 and E.2, NeurIPS checklist questions 4, 5 and 8.
2. Calderon, N., Reichart, R., Dror, R. The Alternative Annotator Test for LLM-as-a-Judge. ACL 2025, pages 16051 to 16081. Source of the leave-one-out rule and five of the ten judges.
3. Fabbri, A. et al. SummEval: Re-evaluating Summarization Evaluation. TACL, 2021.
4. Abraham, E. D. et al. CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behavior. NeurIPS, 2022.
5. Codella, N. et al. Skin Lesion Analysis Toward Melanoma Detection: ISBI 2017 challenge hosted by ISIC. ISBI 2018. Source of the WAX / lesion matrices.
6. GitHub repository search API, q=2609.31857, queried 30 September 2026: 0 results.
7. BLOMEGA. Judge panel effective human size on ChaosNLI.
8. BLOMEGA. When LLM annotators agree with each other and with nobody else.