BayesJudge's expert vs crowd split on SummEval falls out of an 8x8 kappa matrix, and the served code does not run
BayesJudge (arXiv:2610.11116, posted 8 October 2026, accepted at NeurIPS 2026) says its Bayesian layer separates SummEval experts from crowdworkers "without rater metadata". We rebuilt the paper's exact 1,000 pairs and found the two groups agree at Cohen kappa 0.007, so a one-line clustering finds the split, and predicting a withheld expert from its two cluster peers scores 68.5% against BayesJudge's 65.3%, with no LLM calls.
The anonymous code release, checked on 9 October 2026, does not run as served: the evaluator the scripts call is missing five of its methods, and the GPT-4o-mini judgments behind every SummEval number are not included.
What arXiv:2610.11116 claims, as of 8 October 2026
Jiahao Zhang and Pengbin Feng (equal contribution; South China Normal University and University of Southern California), with Chunlei Meng (Fudan), Hang He (East China Normal University) and Tianyong Hao (South China Normal University, co-corresponding), posted BayesJudge: Uncertainty-Aware Bayesian Meta-Evaluation of Human and LLM Judgments as v1 on 8 October 2026 (stat.AP). The footer places it in the NeurIPS 2026 Evaluations and Datasets Track.
The model treats each pairwise comparison as a latent verdict over A, B or tie. Each human rater gets a 3x3 confusion matrix with a Dirichlet prior, each LLM judge gets a position-bias coefficient beta, and an assumed-density sequential Monte Carlo filter updates both online. Two protocol choices carry the identifiability argument: tie-open labels, and asking every LLM judge twice with the order swapped (AB and BA).
The real-data test is SummEval (Fabbri et al., 2021): 100 CNN/DailyMail articles, 16 systems, 3 expert and 5 crowd ratings per summary on four 1-to-5 dimensions. The authors average each rater's four scores, compare the 16 summaries of an article pairwise, and call a tie only on exact equality (epsilon 1e-6). That gives 12,000 pairs; the paper uses 1,000 with an 80/20 split (Appendix B.2, Table 4). Headline results: a "rater-type reversal" in leave-one-rater-out prediction (Table 2), crowd confusion matrices that "lock onto the tie diagonal" (Appendix I.2, Figure 14), posterior entropy correlated with human vote entropy at Spearman 0.404 (Figure 6), and LLM order bias for three judges (Table 13). The NeurIPS checklist answers Yes to open access to data and code.
Which BayesJudge numbers survive a rebuild of the same 1,000 pairs?
First we had to find the 1,000 pairs. Taking the first 1,000 of the converter's output gives 3,355 A, 3,503 B and 1,142 ties, which does not match Table 4. Running the repo's own convert_summeval_to_pair.py on the public SummEval file and then random.Random(42).sample(pairs, 1000) reproduces Table 4 (3,231 / 3,594 / 1,175) and Table 5 (478 expert ties, 697 crowd ties) to the count. Seed 42 is also the README's default. Every BLOMEGA row below uses that set, with the first 800 as train and the last 200 as test.
| Quantity | Paper | BLOMEGA | Source |
|---|---|---|---|
| Labels A / B / tie in the 1,000 pairs | 3,231 / 3,594 / 1,175 | 3,231 / 3,594 / 1,175 | Paper Table 4; BLOMEGA rebuild, seed 42 |
| Tie rate, experts vs crowd | 15.93% vs 13.94% | 15.93% vs 13.94% | Paper Table 5; BLOMEGA |
| Crowd tie mass in posterior confusion | 0.44 to 0.54 on the tie diagonal | raw tie rate 13.5% to 14.8% per crowd slot | Paper App. I.2, Fig. 14; BLOMEGA |
| Mean Cohen kappa: expert-expert / crowd-crowd / expert-crowd | not reported | 0.402 / 0.408 / 0.007 | BLOMEGA |
| Expert majority equals crowd majority (825 pairs with a clear majority on both sides) | not reported | 43.9% (chance 42.6%) | BLOMEGA |
| Summaries with all 5 crowd score vectors identical | not reported | 807 of 1,600 (experts: 46) | BLOMEGA, model_annotations.aligned.jsonl |
| Held-out expert / crowd accuracy, majority of other 7 | 44.8% / 64.7% | 44.2% / 63.6% | Paper Table 2; BLOMEGA rerun on test 200 |
| Held-out expert / crowd accuracy, BayesJudge full | 65.3% / 42.3% | not rerunnable | Paper Table 2 |
| Held-out expert / crowd accuracy, kappa cluster + peer majority | not reported | 68.5% / 68.7% | BLOMEGA |
| Crowd ECE: BayesJudge vs MACE | 0.042 vs 0.035 | n/a | Paper Table 2 (text calls 0.042 the lowest) |
| Spearman with 8-rater vote entropy | 0.404 (posterior entropy) | 0.546 (Dawid-Skene), 0.553 (score gap), 0.617 (crowd vote) | Paper Fig. 6; BLOMEGA |
| Swap pattern back/front among decisive pairs: GPT-4o-mini, DeepSeek-V4-Flash, Claude-Haiku-4.5 | 273/27, 68/38, 794/1 | sign test p 2e-52, 0.005, 8e-237 | Paper Table 13; BLOMEGA arithmetic |
Paper numbers from arXiv:2610.11116v1. BLOMEGA numbers computed on 9 October 2026 from SummEval's model_annotations.aligned.jsonl (1,600 rows) and the paired copy with article text, using the repo's converter. We made no LLM calls; the swap row only re-tests the paper's own counts.
Two rows in the paper's own tables contradict its prose. Section 5.3 says the full model reaches "the lowest crowdworker calibration error (ECE = 0.042)"; MACE in the same Table 2 has 0.035. And the full model's crowd accuracy, 42.3%, is 22.4 points below plain majority vote in the same table. The summary in Section 5.4 explains this as the model "refusing to exploit mechanical tie heuristics", but the raw labels show crowd slots tie less often than experts, not more.
Our rerun of the majority-of-seven baseline gives 44.2% on experts and 63.6% on crowd, within 0.6 and 1.1 points of the paper's 44.8% and 64.7%, so the test split matches. The kappa-cluster baseline then scores 68.5% on experts and 68.7% on crowd. It beats BayesJudge's expert headline by 3.2 points and its crowd accuracy by 26.4 points, and it matches MACE (69.1%) and BayesJudge without the LLM (68.9%) on crowd. Ties between peer votes are broken by the withheld rater's most frequent label on the training pairs.
Why a kappa matrix already finds the experts and the crowd
The "expert and crowdworker behaviour signatures" in Figure 14 are a restatement of a block structure that is visible before any model is fitted. Inside the expert block kappa runs 0.358 to 0.462; inside the crowd block 0.371 to 0.437; across the blocks every cell sits between -0.015 and 0.041. Average-linkage clustering on one minus kappa returns {expert_1, expert_2, expert_3} and {turker_1 ... turker_5}, and does so from the 800 training pairs alone. Any method with per-rater parameters will separate two groups that share no signal.
The crowd block is uniform for a mechanical reason. SummEval's file holds five crowd score vectors per summary, and in 807 of 1,600 summaries all five are identical. When both summaries in a pair carry such rows, all five crowd labels are the same label by construction. That happens on 224 of the 1,000 pairs, where crowd-crowd kappa is exactly 1.0. On the 229 pairs where neither summary has identical rows, crowd-crowd kappa is 0.055, while expert-expert kappa on the same pairs is 0.441. Identical rows also inflate ties: the crowd tie rate is 22.8% on the duplicated pairs against 14.0% on the rest, which is our best explanation for the tie-heavy crowd matrices in Figure 14. We flagged the 807 rows in an earlier note on residual hardness; the pair-level effect on a confusion-matrix model is new here.
The crowd "raters" are also not people. The released file lists five crowd annotations per summary with no worker IDs, so turker_1 is the first list position. A per-slot confusion matrix averages over whoever filled that position, which makes the paper's reading of turker_1 as "an outlier with inverted A/B preferences" hard to interpret as rater behaviour. The paper's own Limitations section concedes that "annotators are not consistent across samples". In our counts, slot turker_1 chooses A 43.1% and B 43.4% of the time, close to the other four slots.
Expert and crowd judgements are close to independent. Their summary-level mean scores correlate at Spearman 0.097 over 1,600 summaries, and on pairs with a clear majority in both groups, the expert majority equals the crowd majority 43.9% of the time against 42.6% expected by chance. This is why "the LLM channel captures expert judgment" (59.5% without humans) while the human channel predicts crowd: with 5 of 7 remaining voters coming from a group that does not track the experts, any pooled human estimate is a crowd estimate. A model that knows the grouping, or finds it by kappa, does not need the LLM to predict experts.
Does posterior entropy track human disagreement better than the labels themselves?
No. Figure 6 reports Spearman 0.404 between BayesJudge's posterior entropy and the vote entropy of the eight human labels, over all 1,000 pairs. The posterior is computed from those same eight labels (on the 800 training pairs they also updated the parameters), so the right comparison is other functions of the same labels.
A plain three-class Dawid-Skene posterior over the eight slots, with no LLM, reaches 0.546 (0.477 on the 200 test pairs alone). The negative absolute gap between the two summaries' mean human scores reaches 0.553. Crowd-only vote entropy reaches 0.617; expert-only reaches 0.385. The informative number is the last one: expert vote entropy and crowd vote entropy, which share no labels, correlate at 0.031 (p 0.32). Disagreement inside one group does not predict disagreement inside the other, so "human disagreement" on this set mostly measures the expert-crowd gap plus crowd duplication. Dawid-Skene shows the same non-monotone quintiles the paper describes for Table 3: mean human entropy 0.364, 0.650, 0.728, 0.891, then 0.843 in the top quintile.
The order-bias result does hold, and it is visible without the Bayesian layer. Table 13's own counts give GPT-4o-mini 273 back-position wins to 27 front among decisive swaps, DeepSeek-V4-Flash 68 to 38 and Claude-Haiku-4.5 794 to 1. A sign test on those counts gives p of 2e-52, 0.005 and 8e-237, and the ranking matches the fitted betas (-0.57, -0.11, -2.35). One caveat: the paper describes DeepSeek-V4-Flash as "position-invariant" with a credible band including zero, yet its raw 68-to-38 split is significant at 0.5%. We did not rerun the LLM calls.
What this means if you aggregate human and LLM labels
Print the rater-by-rater kappa matrix before fitting anything. On SummEval it shows the whole "unsupervised rater typing" result in 28 numbers. If your panel splits into blocks with near-zero cross agreement, you have two constructs, not one construct with noise, and a confusion-matrix model will fit them as two noise types. Decide which construct you want as gold before fusing.
Check for duplicated annotation rows. Five identical score vectors on half the summaries is a collection artifact, whether copied defaults, a template or a pipeline join. It produces perfect agreement on a quarter of pairs and inflates ties. A filter that flags items where all workers submit the same full vector costs one line and changes every per-rater statistic downstream.
Treat list positions as positions. If your vendor delivers crowd labels without worker IDs, per-rater confusion matrices are per-slot averages. Ask for stable pseudonymous worker IDs at collection time; we do this by default on BLOLabel because every rater model, from MACE to BayesJudge, needs them.
Benchmark uncertainty signals against label-only baselines. A posterior entropy that correlates 0.404 with vote entropy is weaker than the label-gap baseline at 0.553. For routing items to adjudication, the cheaper signals are at least as good here. Our judgement: the value a Bayesian layer could add is in the LLM swap channel and in calibration, and the paper's Table 2 does not show it beating MACE on calibration.
Keep the swap protocol. The paired AB/BA design is cheap and its raw counts already show order bias for all three judges. Report the back/front counts next to any fitted bias term so readers can see the evidence without the model. For more on how judge panels behave when labels are sparse, see our note on sparse overlap and wrong decisions.
Check it yourself
The release check needs only curl. The anonymous service exposes a file list and a zip:
# release, checked 2026-10-09
curl -s https://anonymous.4open.science/api/repo/BayesJudge-0879/options
# lastUpdateDate 2026-05-10T05:20:50Z, sourceCommitDate 2026-05-10, expirationDate null
curl -sL -o repo.zip https://anonymous.4open.science/api/repo/BayesJudge-0879/zip # 212,458 bytes
unzip -l repo.zip | tail -1 # 64 files: 22 .py (4,095 lines), 40 .pyc, README.md, requirements.txt
curl -s https://anonymous.4open.science/api/repo/BayesJudge-0879/file/bayesian_eval/experiments/__init__.py
# {"error":"file_not_found"} (imported by bayesian_eval/__init__.py line 9)
grep -c "def update_online\|def infer_with_fixed_params" bayesian_eval/core/evaluator.py # 0
# no LICENSE file; no dataset/summeval_pairwise_1000.json; no llm_judgments anywhereThe data check needs SummEval's annotation file (5.8 MB) and, for the repo's converter, the paired copy with article text. The aligned file has fields id, model_id, expert_annotations (3 dicts), turker_annotations (5 dicts). The README's Method 1, mteb/summeval on Hugging Face, holds 100 rows of averaged scores and cannot feed the converter.
curl -sLO https://storage.googleapis.com/sfr-summarization-repo-research/model_annotations.aligned.jsonl
python3 - <<'PY'
import json, random, itertools, collections
R=[json.loads(l) for l in open("model_annotations.aligned.jsonl")]
def lab(a,b):
x=sum(a.values())/4; y=sum(b.values())/4
return "T" if abs(x-y)<1e-6 else ("A" if x>y else "B")
P=[]
for g in range(0,1600,16): # file is already in blocks of 16 per article
G=R[g:g+16]
for i,j in itertools.combinations(range(16),2):
a,b=G[i],G[j]
P.append([lab(x,y) for x,y in zip(a["expert_annotations"],b["expert_annotations"])]+
[lab(x,y) for x,y in zip(a["turker_annotations"],b["turker_annotations"])])
S=random.Random(42).sample(P,1000) # the paper's 1,000 pairs
print(len(P), collections.Counter(l for p in S for l in p)) # 12000; A 3231, B 3594, T 1175
print(sum(p[:3].count("T") for p in S), sum(p[3:].count("T") for p in S)) # 478 697
same5=sum(all(t==r["turker_annotations"][0] for t in r["turker_annotations"]) for r in R)
print(same5) # 807
PYKappa is Cohen's on the three labels per slot pair; clustering is scipy.cluster.hierarchy.linkage(..., "average") on one minus kappa, cut at two clusters. The held-out baseline predicts each withheld slot on pairs 801 to 1,000 from the majority of its same-cluster slots. Dawid-Skene is plain EM, 200 iterations, initialised from vote shares, with a 0.01 pseudo-count. Entropy is natural log, matching the paper's 0 to 1.0986 range. Running summeval_exp.py on the rebuilt file fails first with ModuleNotFoundError: bayesian_eval.experiments, then, with that import commented out, with AttributeError: 'StrictBayesianEvaluator' object has no attribute 'update_online'.
What would prove this wrong
Our claim is that BayesJudge's SummEval rater typing reflects block structure and duplicated crowd rows that simple statistics already expose, and that no released artefact lets anyone check the full model's numbers. Three things would break it.
First, if the authors publish a complete evaluator and the GPT-4o-mini swap judgments, and BayesJudge run on the 1,000 seed-42 pairs with the 807 duplicated summaries' crowd rows removed still recovers distinct tie-heavy crowd matrices (tie diagonal above 0.40 for at least four of five slots), the tie-fallback reading is real and ours is wrong.
Second, a dated prediction: by 31 March 2027, any public rerun of BayesJudge on these pairs will report held-out expert accuracy no more than 2 points above our 68.5% kappa-cluster baseline. A rerun clearing 71% on experts with the LLM channel would show the swap channel adds information beyond the expert peers.
Third, if the anonymous repo is updated so that summeval_exp.py runs end to end on the public SummEval file, the reproducibility finding is superseded; we will note the date here.
Sources
- Zhang, J., Feng, P., Meng, C., He, H., Hao, T. BayesJudge: Uncertainty-Aware Bayesian Meta-Evaluation of Human and LLM Judgments. arXiv:2610.11116v1, 8 October 2026. Sections 5.1 to 5.4, Tables 2, 3, 4, 5, 13, Appendix B.2, I.2, I.3, NeurIPS checklist items 5 and 12.
- BayesJudge anonymous release, anonymous.4open.science/r/BayesJudge-0879, source commit dated 10 May 2026. Accessed 9 October 2026.
- Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., Radev, D. SummEval: Re-evaluating Summarization Evaluation. TACL, 2021. Data: Yale-LILY/SummEval and model_annotations.aligned.jsonl. Accessed 9 October 2026.
- SALT-NLP. autometrics, which hosts
model_annotations.aligned.paired.jsonl(1,600 rows with article text), used to run the repo's converter. Accessed 9 October 2026. - Dawid, A. P., Skene, A. M. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Journal of the Royal Statistical Society C, 1979.
- Hovy, D., Berg-Kirkpatrick, T., Vaswani, A., Hovy, E. Learning Whom to Trust with MACE. NAACL, 2013.
- BLOMEGA. Half the human panel in an LLM-judge hardness study is SummEval's crowd, 5 October 2026.
FAQ
Can BayesJudge's SummEval results be reproduced from its released code?
Not as served on 9 October 2026. The anonymous archive has 64 files (22 Python, 40 compiled bytecode, a README and requirements.txt) and no licence. bayesian_eval/__init__.py imports a bayesian_eval.experiments module that is not in the archive and returns 404 from the file API. After commenting that import out, summeval_exp.py stops at evaluator.update_online: the served evaluator.py has 206 lines and none of update_online, infer_with_fixed_params, get_sample_verdict, get_llm_posterior_stats or get_model_win_rate. Only a stale Python 3.11 bytecode file contains those names. The precomputed GPT-4o-mini, Claude-Haiku-4.5 and DeepSeek-V4-Flash judgments are not shipped either.
Do you need a Bayesian model to separate SummEval experts from crowdworkers without metadata?
No. On the paper's own 1,000 pairs, which BLOMEGA reproduced exactly with random.Random(42).sample, expert-expert Cohen kappa averages 0.402, crowd-crowd 0.408 and expert-crowd 0.007. Average-linkage clustering on one minus kappa returns the three experts and five crowd slots exactly, also from the 800 training pairs alone. Predicting a withheld rater from the majority of its own cluster scores 68.5% on experts and 68.7% on crowd for the 200 test pairs, against BayesJudge's 65.3% and 42.3% in Table 2.
Why do SummEval crowd raters look so uniform in BayesJudge?
Largely because of duplicated rows. In 807 of 1,600 SummEval summaries all five crowd score vectors are identical (46 of 1,600 for the three experts). On the 224 of 1,000 pairs where both summaries carry identical crowd rows, crowd-crowd kappa is 1.0; on the 229 pairs where neither does, it is 0.055. The crowd tie rate is 13.9% overall, below the experts' 15.9%, so a crowd tie fallback is not visible in the raw labels.