Editing the rubric clauses an LLM judge panel disagrees on raises consensus 0.069, raises agreement with humans by 0
A preregistered study from CEO Business School tested whether rewriting the rubric clauses an LLM judge panel disagrees on most makes the panel more reliable. On SummEval, it raised inter-judge Krippendorff's alpha by 0.069 (95% CI 0.046 to 0.092) over the original rubric. Agreement with expert human ratings moved by -0.007 (95% CI -0.037 to 0.021), no detectable change. A control that edited an equal number of random, uncontested clauses the same way reached higher human agreement than targeting the contested ones, on all three benchmarks tested.
What changed, and when
On 9 October 2026, Sangdo Han and Moonsoo Kim (CEO Business School) posted Agreement Without Alignment: Label-Free Rubric Clarification Increases LLM-Judge Consensus but Not Agreement with Humans (arXiv:2610.13303v1). LLM judge panels are increasingly used to grade open-ended output, and when panel members disagree, the usual diagnosis is an unclear rubric. The paper tests the natural fix directly: find the sub-criteria judges disagree on most, using only judge outputs, and have an editor model rewrite just those, without ever reading a human label.
The procedure decomposes each rubric into sub-criteria (75 total across three benchmarks), has a 5-judge, 3-provider panel probe every development item against each sub-criterion, ranks sub-criteria by cross-judge disagreement rate, and rewrites up to 5 of the most contested per dimension with an editor model (Claude Opus 5.5). A separate stage that would move text-checkable criteria to deterministic code checkers mapped none of the 75 sub-criteria to a checker, so it never activated in this evaluation. Four hypotheses, the selection rule, and the statistical tests were committed to version control before any evaluation call, under a sealed manifest (SHA-256 prefix 3f49c9c4).
The test ran 129,770 judge requests (25,954 per judge) across three benchmarks with existing human labels: SummEval (news summaries, 4 dimensions, expert mean ratings), the HealthBench meta-evaluation (medical conversations, 34 official criteria, physician majority verdicts), and USR Topical-Chat (knowledge-grounded dialogue, 6 qualities, annotator mean ratings). It cost US$236.69 at batch API prices and completed in 23.95 hours.
The evidence table
The three preregistered primary comparisons per dataset: the edited ("Full") rubric against the original, the edited rubric against a random-clause control edited the same way, and whether either change moved agreement with human ratings. An alpha difference only counts as an effect if it is at least 0.05 and its interval excludes zero; that threshold does not apply to the human-agreement metrics.
| Dataset | Comparison | Difference | 95% CI | Effect? |
|---|---|---|---|---|
| SummEval | Full − Original, inter-judge α | +0.069 | 0.046 to 0.092 | Yes |
| SummEval | Full − Original, human agreement (τ) | −0.007 | −0.037 to 0.021 | No |
| SummEval | Full − Random clause, inter-judge α | +0.074 | 0.047 to 0.097 | Yes |
| HealthBench | Full − Original, inter-judge α | −0.012 | −0.031 to 0.007 | No |
| HealthBench | Full − Original, human agreement (F1) | −0.023 | −0.043 to −0.003 | No (Holm p=0.155) |
| USR | Full − Original, inter-judge α | +0.027 | 0.007 to 0.047 | No (below 0.05) |
Reproduced from Table 3 of arXiv:2610.13303v1, Section 5.2. Differences are equally weighted means over dimensions, paired cluster bootstrap, Holm-adjusted p-values over 9 tests.
The mechanism: judges can converge on an axis humans don't weight
The paper's reading, consistent with prior work it cites (Mukherjee et al., 2026; Song et al., 2026), is that a panel of LLM judges can agree with each other while converging on a shared reading of a clause that is internally consistent but not the one human annotators hold. A label-free editor has no way to tell which of several plausible readings of an ambiguous clause the human annotation guidelines actually intended, so it can resolve the ambiguity in a direction that raises consensus without raising correctness.
A concrete case from the paper's pre-flight checks illustrates the failure mode directly: HealthBench criterion C31 passes a response only if it does not ask the user follow-up questions. Two judges marked a response consisting only of follow-up questions as meeting that criterion, the opposite of the intended reading, and one judge's own stated reasoning correctly identified the problem immediately before emitting the wrong verdict. That is exactly the kind of clause-level ambiguity, here in mapping a negated condition to an output value, the procedure is designed to catch, and in this instance it did not change the underlying misreading, only who shared it.
What it means for anyone running an LLM judge panel
Rising inter-judge agreement is not evidence a rubric fix worked. Teams that monitor panel consensus as a proxy for rubric quality should stop treating it as sufficient. The paper's own matched control shows that editing clauses judges did not flag as contested, with the same editor and example rule, produced almost as much of the consensus gain (0.074 vs 0.069) as targeting the actually-contested ones, and reached higher human agreement in every point estimate.
A one-shot rewrite with no disagreement signal at all did no harm and sometimes did best. On HealthBench, the plain rewrite condition produced the highest alpha (0.389) and the highest macro F1 against physicians (0.689) of any edited condition, better than either the contested-clause edit or the random-clause control. Simply asking an editor to tighten a rubric's wording, without any targeting logic, was not worse than targeting by disagreement, and on this benchmark it was better.
Multi-provider judge panels need format checks per output type. The paper's own pipeline caught a real operational bug: one provider's batch transport silently failed to enforce a JSON response schema on nested probe outputs (0 of 20 test outputs conformed) while enforcing it correctly on a different documented field (20 of 20). A panel that assumes schema enforcement is uniform across providers can silently corrupt a fraction of its data.
Our judgement, specific to vendor-QA pipelines that rely on multi-model judge panels: treat inter-judge consensus and human alignment as two separate metrics with two separate acceptance gates, never one proxy for the other, and keep a small human-labeled holdout specifically to catch the case where an edit raises the first while leaving the second flat or negative. This sits alongside our review of how many humans a judge panel is actually worth: a panel that agrees with itself more is not the same claim as a panel worth more human-equivalent judgment.
Check it yourself
open https://arxiv.org/abs/2610.13303 # paper, Tables 1-6, Section 5, Appendix A-E
# code, frozen rubrics, item identifiers and judge scores: release promised in the paper
# what to check without rerunning anything:
# Table 3 (Section 5.2): the 9 preregistered primary comparisons, difference / CI / Holm p
# SummEval Full-Original alpha: +0.069 [0.046, 0.092], p<0.001
# SummEval Full-Original human (tau): -0.007 [-0.037, 0.021], not significant
# SummEval Full-Random clause alpha: +0.074 [0.047, 0.097], p<0.001
# Table 4 (Section 5.2): absolute alpha and human agreement for all 6 rubric conditions
# note Random clause reaches the HIGHEST ensemble human tau on SummEval (0.516)
# and USR (0.748) of any condition, original included
# Section 5.1: 129,770 total judge requests, 23.95 hours, $236.69 at batch prices
# Appendix C: the batch-transport schema-enforcement failure for one provider
python3 - <<'PY'
# SummEval Table 4: alpha and human tau for original vs. full vs. random-clause
rows = {
"Original": (0.577, 0.501),
"Full": (0.646, 0.494),
"Random clause":(0.572, 0.516),
}
for name, (alpha, tau) in rows.items():
print(f"{name:14s} alpha {alpha:.3f} human tau {tau:.3f}")
PY
# Original alpha 0.577 human tau 0.501
# Full alpha 0.646 human tau 0.494 (alpha up, tau essentially flat)
# Random clause alpha 0.572 human tau 0.516 (alpha flat, tau is the HIGHEST of the three)What would prove this wrong
This is a result on one editor model (Claude Opus 5.5), a specific 5-judge panel skewed toward one provider (3 of 5 judges are Claude models), and three English-language benchmarks. The authors flag this confound themselves: provider, model size and reasoning effort are not separated in their panel composition.
A dated prediction: by 30 June 2027, a published replication using a panel balanced across five different providers, with an editor model from a sixth, will still show the central pattern: inter-judge consensus rising from targeted clause edits without a corresponding, statistically detectable rise in agreement with human expert ratings. If a balanced-panel replication shows both consensus and human agreement rising together by a comparable margin, this prediction is wrong, and the original result would look more like an artifact of editor-judge family overlap than a general property of label-free rubric repair.
Sources
- Han, S., Kim, M. (CEO Business School). Agreement Without Alignment: Label-Free Rubric Clarification Increases LLM-Judge Consensus but Not Agreement with Humans. arXiv:2610.13303v1, 9 October 2026. Tables 1-6; Sections 3-7.
- BLOMEGA. One 32-model judge panel, worth 4.24 humans or 2.30.
- BLOMEGA. An LLM translation judge scored wrong-dialect Arabic as a perfect match, every human scored it zero.
- BLOMEGA. Data annotation research: the latest.