One 32-model judge panel, two answers: worth 4.24 humans, or worth 2.30

What changed, and when

On 18 September 2026 Chao Li, Yingying Yu and Yunfeng Li posted How Many Humans Is a Judge Panel Worth? (arXiv:2609.21277v1, 18 pages, 10 figures, 8 tables). The vote archive is public at github.com/Chao1208/chaosnli-judge-votes, snapshot 5da92bb.

The setup is deliberately boring, which is what makes the result usable. Thirty-two models from 10 provider families each label 1,000 items per task at temperature 0, one user message, no system message, label-only instruction. The tasks are MNLI-m, SNLI and alpha-NLI drawn from ChaosNLI, where every item carries 100 human labels rather than one adjudicated gold. Items are sampled approximately equally across human-entropy terciles with seed 42. That gives 96,000 baseline records, of which 95,994 parse, with 6 failures at 0.00625 percent retained as flagged placeholders.

Most judge-panel work asks whether the panel agrees with a gold label. This one asks a different question: how many independent draws from the empirical human label distribution would produce the same statistic as this panel. The answer turns out to depend entirely on which statistic you pick, and the two obvious picks disagree by nearly a factor of two.

The evidence table

The headline numbers are the paper's Table 3, for the full 32-judge panels on each task. Read the last row as a warning label: nu_H and nu_MSE are answers to different questions, not two estimates of one quantity.

All values from arXiv:2609.21277v1. Percentages in the last two rows are as printed in the paper. Majority vote exceeds the best individual model only on SNLI.

Note the row most people would have quoted. Majority-vote agreement with the human mode is 68.6, 86.5 and 93.0 percent, and the paper is explicit that the Table 3 measurements do not predict it: agreement uses a thresholded decision, while the distributional error uses unthresholded panel frequencies. Three plausible summaries of one panel, all defensible, all answering different questions.

Where the two numbers separate

The split is not a modelling choice made late in the analysis. It is baked into what each statistic retains. Both branches start from the same residual: a judge's one-hot label minus h, the item's empirical 100-label frequency vector. The spectral branch normalises each judge's residual, builds a Gram matrix with unit diagonal and takes its participation ratio. Normalising is what removes member energy, which is how far a judge sits from the human distribution on average.

The error branch never normalises. It averages raw label frequencies across the panel and measures the squared distance to h. The paper's Proposition 1 makes the relationship exact: error is the energy scale times a sum of eigenvalues weighted by how well the averaging direction aligns with each eigenvector, while the participation ratio depends only on the sum of squared eigenvalues. Orientation and energy are exactly the information the spectral summary discards.

This is not a technicality about hidden constants. The authors give a realisable four-judge construction with equal member energies, zero mean residuals and nonnegative correlations, in which raising the participation ratio from 2 to 16/7 raises the error from 1/4 to 5/16. Larger spectral diversity, worse distribution recovery, with no pathology involved.

It shows up in the real pool too. Within-size Spearman correlation between participation ratio and negative error runs 0.960 to 0.977 on MNLI-m, 0.711 to 0.867 on SNLI, and only 0.223 to 0.419 on alpha-NLI. Scaling error by mean member energy lifts the alpha-NLI correlations to 0.937 to 0.960; at panel size 16 the jump is from 0.338 to 0.937. In other words, the ranking mismatch is mostly about how far individual members sit from the human distribution, not about how differently they disagree with each other.

Single member additions can move the two objectives in opposite directions and keep doing so on both halves of the item set. Of the 54,348 four-to-five additions per dataset, 2,116 on MNLI-m (3.89 percent), 309 on SNLI (0.57 percent) and 1,127 on alpha-NLI (2.07 percent) conflict stably with both relative changes at least 1 percent. The frequency falls with panel size and reaches zero for all three tasks at 31 to 32.

Keeping the disagreement roughly doubles the panel

This is the part with the most direct consequence for how you store annotations. The model answers are held fixed. Only the anchor the residuals are measured against changes: a uniform label vector, the human mode label, or the full 100-label distribution h.

Against a uniform anchor the panel measures 2.75, 2.57 and 2.33. Against the mode, which is what a dataset with one adjudicated gold label per item gives you, it measures 2.76, 4.32 and 3.94. Against the full distribution it measures 4.24, 6.46 and 6.50. The mean squared inner product between judges falls in step, from 0.412 to 0.425 to 0.212 on MNLI-m and from 0.531 to 0.284 to 0.128 on SNLI.

Our reading, stated as a judgement: a corpus that keeps only the adjudicated label is measurably less able to tell you what your judge panel is doing, and the size of that loss is about a factor of 1.5 on two of the three tasks here. The disagreement is not noise you cleaned up. It is the reference the audit needs.

How many humans you need for that reference is bounded too. Estimating the anchor from M labels drawn without replacement from each item's 100-label pool gives a median nu_H of 3.35, 4.36 and 4.32 at M equal to 5, and 3.70, 5.10 and 5.12 at M equal to 10, approaching the full-reference values of 4.24, 6.46 and 6.50 as M rises. The paper is careful that this does not validate deploying with five human labels, because the simulations still draw from the full distribution.

What this changes for anyone buying or running a judge panel

A vendor claim of "our panel replaces N annotators" is incomplete without the target. On these three tasks the same votes support 4.24 or 2.30 with equal honesty. Ask which statistic was matched, on which reference, over which items. If the answer is agreement with a single gold label, that is a third quantity again, and the paper shows it is not predicted by either of the other two.

Buy panel members for distance to the human distribution, not for variety. The alpha-NLI result is the practical one: a panel can look diverse in residual direction while its members all sit far from h. Reporting mean member error energy alongside any diversity number costs nothing and is what separated a correlation of 0.338 from one of 0.937 at panel size 16.

Adding a 17th model is mostly theatre. Stable conflicting additions fall from 3.89 percent at four-to-five to 0.02 percent at 16-to-17 on MNLI-m and to zero at 31-to-32 on all three tasks, and the median effective size rises with diminishing gains. The gap that matters is between 32 nominal and roughly 2 to 6.5 effective, and it does not close by buying more models from the same pool.

Equal-weight averaging keeps a large shared component. The consensus direction holds 43.8 percent of centred residual variance on MNLI-m and 33.7 percent on SNLI. Whatever the panel shares survives averaging by construction, which is why variance reduction flattens out well before the nominal size is reached.

Reordering the options does not buy independence. Replacing one judge at a time with its option-reordered responses, 86 of 91 model-dataset combinations moved further than the 97.5th percentile of an independent-transition reference with matched conditional flip rates, and the median absolute-change ratios were 6.57, 4.91 and 2.22. The reference predicts a negative mean change for every combination; the observed change was positive in 37 cases. Presentation shuffling is not a cheap substitute for a genuinely different judge, though the authors note cached responses prevent isolating an order effect.

Keep the raw per-annotator votes. This connects to the same collection-side point we found in soft labels and human disagreement: the audit above is only possible because ChaosNLI published 100 labels per item. A pipeline that adjudicates at collection time destroys the reference before anyone can use it.

Check it yourself

The vote archive is public and the human counts come from ChaosNLI under its original terms. The cheapest reproduction is the one that matters most: confirm that the two effective sizes separate on your own panel.

open https://arxiv.org/abs/2609.21277          # Tables 3, 4 and 5, Sections 3.2 to 4.9
git clone https://github.com/Chao1208/chaosnli-judge-votes   # snapshot 5da92bb

# what is in the archive: 96,000 baseline records (32 judges x 1,000 items x 3 tasks)
# plus 122,000 presentation-order records, with requested model identifiers and
# failure indicators. Item text and the 100 human counts are NOT redistributed:
# get ChaosNLI separately and join on dataset + item id.

# the two statistics, in the paper's own notation
#   residual        r_ai = onehot(label_ai) - h_i
#   spectral        C_ab = tr(B_ab) / sqrt(tr(B_aa) tr(B_bb));  PR = k / (1 + (k-1) qbar)
#                   nu_H  = value of m whose simulated PR_0(m) matches PR_obs
#   distributional  E     = mean_i || mean_a onehot(label_ai) - h_i ||^2
#                   nu_MSE = J / E,  J = mean_i (1 - ||h_i||^2)

python3 - <<'PY'
pairs = [("MNLI-m", 4.24, 2.30, 0.19691, 43.8),
         ("SNLI",   6.46, 3.75, 0.09068, 33.7),
         ("aNLI",   6.50, 3.44, 0.04825, 35.9)]
for t, nh, nm, E, g in pairs:
    print("%-7s nu_H=%.2f  nu_MSE=%.2f  ratio=%.2f  E=%.5f  gamma_co=%.1f%%  "
          "share of nominal 32: %.1f%% vs %.1f%%"
          % (t, nh, nm, nh/nm, E, g, 100*nh/32, 100*nm/32))
PY
# MNLI-m  nu_H=4.24  nu_MSE=2.30  ratio=1.84  E=0.19691  gamma_co=43.8%  share of nominal 32: 13.3% vs 7.2%
# SNLI    nu_H=6.46  nu_MSE=3.75  ratio=1.72  E=0.09068  gamma_co=33.7%  share of nominal 32: 20.2% vs 11.7%
# aNLI    nu_H=6.50  nu_MSE=3.44  ratio=1.89  E=0.04825  gamma_co=35.9%  share of nominal 32: 20.3% vs 10.8%

Two limits on that snippet, both the paper's own. The ratios describe these full panels and are not calibration factors you can carry to another pool. And the paper does not ship executable reproduction of every reported result: its supplement is a reproduction map identifying available records, result fields and missing inputs, with the calibration curves for nu_H saved rather than regenerated.

What would prove this wrong

The spectral identity is proved, so the place this fails is the bridge from the audit to practice. The load-bearing empirical claim is that the two targets separate by a factor close to two on a real pool, and that the separation is a property of judge panels rather than of these three categorical NLI tasks and this one set of 32 models.

A dated prediction: by 31 December 2027, a replication on a non-NLI task with published per-annotator human labels, at least 20 human labels per item and a panel drawn from a different model pool, will report nu_H and nu_MSE at least 1.4 times apart. If a replication of that shape reports the two within 10 percent of each other, the separation is an artefact of ChaosNLI's label geometry and the practical advice above should be withdrawn.

A second, cheaper falsifier: the authors' own alpha-NLI instability. Cross-half rank correlation of the participation ratio is 0.370 to 0.554 on alpha-NLI against 0.915 to 0.942 on MNLI-m. If a larger item set makes alpha-NLI rankings as stable as MNLI-m's, then item sampling, not the measurement target, explains most of the disagreement we have treated as structural.

Sources

1. Li, C., Yu, Y., Li, Y. How Many Humans Is a Judge Panel Worth?. arXiv:2609.21277v1, 18 September 2026. Tables 2 to 5, Sections 3.2 to 4.9, Limitations. HTML version.
2. Li, C. chaosnli-judge-votes, snapshot 5da92bb. The 96,000 baseline and 122,000 presentation-order judge records used in the audit.
3. Nie, Y., Zhou, X., Bansal, M. What Can We Learn from Collective Human Opinions on Natural Language Inference Data? EMNLP 2020. ChaosNLI, the source of the 100 human labels per item.
4. Kish, L. Survey Sampling. Wiley, 1965. The design-effect form the consensus-share statistic reduces to.
5. BLOMEGA. LLM judges, soft labels and human disagreement.
6. BLOMEGA. Data annotation research: the latest.