Treat the three annotators as a sample and eight LLMs tie: F(7,14) = 0.59 on 6,960 judgements

What changed, and when: a benchmark's authors retract their own leaderboard

Fardeen Sadab and Adib Sakhawat (Islamic University of Technology, Dhaka) posted Two Emojis of Difference as arXiv:2609.29445v1 on 24 September 2026. The arXiv comment line says it is accepted at the 6th Multilingual Representation Learning (MRL) workshop at EMNLP 2026 in Budapest. The benchmark is theirs: eight instruction-tuned LLMs (Claude-3-Haiku, DeepSeek-V3.2, Gemini-2.0-Flash, Gemma-3-27B, Mistral-Large-2512, GPT-4.1-nano, Qwen3-VL-235B, Grok-4-Fast) emit one to five emojis for each of 17,100 Bangla, English and Hindi sentences (Section 3).

The human study is the part that matters for anyone designing an evaluation. Three fluent Bangla speakers rated a stratified 5% Bangla sample, 290 sentences, for all eight systems on a 1 to 5 scale. Fully crossed: 290 x 8 x 3 = 6,960 judgements. That design is rare, and it is the only reason the system x annotator interaction can be separated from residual noise at all. Their earlier analysis of the same data had reported a winner (Gemma-3-27B), a "smaller model wins" effect and "low but positive agreement". This paper takes each claim apart.

No code or data repository is linked from the paper or the arXiv page as of 27 September 2026, and none of the first author's public GitHub repositories matches the benchmark. Every check below therefore runs on the printed tables.

How far apart are the two analyses of the same 6,960 ratings?

All values from arXiv:2609.29445v1. Brackets are 95% intervals.

The agreement numbers explain the rest. A Pearson of 0.17 was read as "low but positive agreement". Once the offset between an annotator who averages 2.86 and one who averages 4.35 is penalised, alpha falls to 0.014, and for one pair it is -0.109, slightly worse than chance. Per-annotator length slopes are 0.085, -0.024 and 0.329 MOS per emoji: one rater's system ranking follows output length at Spearman 0.95, another's at -0.26. The panel mean averages three different instruments.

Where do 14 degrees of freedom come from, and why do they decide the result?

The degrees of freedom check out, and they carry the whole argument. The system effect has 8 - 1 = 7. The system x rater interaction has 7 x (3 - 1) = 14. The residual (item x system x rater) has 289 x 7 x 2 = 4,046, which is exactly the denominator in the paper's F(7,4046). With three annotators the correct test for "system A beats system B for annotators in general" has 14 denominator degrees of freedom, not four thousand. Most of the 6,960 judgements tell you about items, not about whether a different set of raters would agree.

The interaction mean square (20.25) is larger than the system mean square (11.85). That is the plain-language result: the three annotators disagree about which systems are better by more than the systems differ. A paired t-test over 290 items on the panel average never sees that disagreement, because averaging first throws the system x rater term into the item means and then treats 290 items as the replication unit.

Our check 1: Table 1 reproduces Figure 1(a)

Table 1 prints four mean squares. We back out the rest. The residual mean square implied by F = 18.62 is 11.85 / 18.62 = 0.6364, and 20.25 / 0.6364 = 31.82 matches the printed system x rater F exactly. The rater F of 319.03 only works if the rater effect was tested against the item x rater interaction, whose mean square is then 1345.16 / 319.03 = 4.216 (not printed). The rater sum of squares, 2 x 1345.16 = 2,690.3, is stated to be 22.1% of the total, so the total is about 12,173; item at 18.8% then gives the item mean square, and the item x system term is the remainder. Plugging these into the standard expected-mean-square equations for a crossed random design gives the variance shares below. The three annotator-related terms (rater 29.1, item x rater 22.9, system x rater 3.5) add to 55.5% of rating variance. Our values match the paper's Figure 1(a) to within 0.1 point in every row, so the ANOVA and the variance decomposition come from the same data and the same model. The system sum of squares recomputed from the eight MOS values in Table 3 is 870 x (sum of squared deviations) = 83.0, against 7 x 11.85 = 82.95 in Table 1.

How many annotators would this evaluation need? More than the paper says

The paper's decision study reports G = 0.336 for the current design, 27 annotators to reach G of 0.8 and 70 to reach 0.9 at 290 items, and a ceiling below 0.8 at any number of items with three raters. We reproduce those numbers, and the reproduction shows where they come from.

Our check 2: the 27-annotator figure assumes a system variance the data do not support

G = 0.336 comes out exactly (0.3360) only if the system variance plugged in is the plain variance of the eight observed MOS means in Table 3, divided by 8: 0.01193. Using the n - 1 divisor gives 0.366, 23 annotators and 58. With the paper's choice we get 27 annotators for 0.8 and 69 for 0.9 (the paper says 70; a rounding difference). The ceiling with three annotators and unlimited items is 0.346.

The observed spread of system means is not the true system variance. It contains the noise the paper itself identifies. Using our reconstructed components, noise alone would give the eight means an expected variance of (0.0676/3 + 0.0871/290 + 0.6364/870) x 7/8 = 0.0206. The observed variance is 0.0119, 58% of that. That ratio is the F of 0.59 in another form. The unbiased ANOVA estimate of the system component, (MS_s - MS_ps - MS_sr + MS_res) / (290 x 3), is (11.85 - 0.90 - 20.25 + 0.64) / 870 = -0.010. Negative variance estimates are truncated to zero in practice, and at zero no panel size reaches any G above zero. Our judgement: "27 annotators" should be read as the minimum under the most generous assumption, and the paper's own F test says that assumption is not established.

If you run human preference evaluations, the design decision comes first. Run a small fully crossed pilot, estimate the system x rater variance, and compute the panel size from it before collecting 6,960 judgements. The G formula divides system x rater variance by the number of annotators only, so more items cannot fix a thin panel. A three-person panel on 290 items spent its budget on the axis that does not help.

Report agreement with a coefficient that sees offsets. A Pearson of 0.17 and an alpha of 0.014 describe the same ratings. Report alpha or ICC(2,1) with a bootstrap interval, plus per-annotator means, as the authors now recommend (Section 10).

Length-match before you believe a short-output leaderboard. The top-to-bottom MOS spread is 0.341 points, and at 0.159 points per emoji that is about 2.1 emojis of output length, which is the paper's title. Restrict to same-length pairs on the same item (2,599 of them here) and Gemini-2.0-Flash goes from sixth to first with a win rate of 0.603 [0.540, 0.664], Grok-4-Fast from last to fourth. Recomputing from Table 3 we get Spearman 0.905 (p = 0.002) for emoji count against MOS, Pearson R² 0.786, and Spearman 0.119 for the length-matched order, all matching the paper. Note that the 78.7% is a Pearson R² while the 0.90 is a Spearman, two statistics on the same eight points.

Split multi-view tables by source item. With 8 generator views per sentence, a row-wise 80/20 split puts 1 - 0.2^7 = 99.9987% of test rows next to a training copy of the same sentence; the paper's 99.998% checks out, as does 99.84% for 5 views. The measured optimism is 0.630 vs 0.598 macro-F1, and the top-ranked system changes from Mistral-Large to Gemma-3-27B.

Our check 3: two numbers that do not line up

Section 9 reports +3.1 points of inflation for the 80/20 split, then says the inflation "is stable at 1.7 to 2.3 points across split ratios from 0.5 to 0.9". An 80/20 split has ratio 0.8, inside that range, yet 3.1 is outside it. Either the sweep used different replicates or a different table, and the paper does not say which. Section 8 gives Grok-4-Fast 294.7 completion tokens per Bangla item and 2.60 emojis, "124 tokens per emoji"; 294.7 / 2.60 = 113. A mean of per-item ratios would explain the gap, but it is not stated. Neither changes a headline conclusion. Both are the kind of thing a released analysis script would settle in a minute.

This fits a pattern we have seen elsewhere in the annotation cluster: how many humans a judge panel is really worth depends on the same between-rater variance, and five machine labellers disagreeing on the same 100 scenes is the LLM version of three annotators averaging 2.86, 4.35 and 3.31.

Check it yourself

No ratings are released, so this runs on the printed numbers from Table 1, Table 3 and Section 4.3 of arXiv:2609.29445v1. It reproduces the degrees of freedom, Figure 1(a), G = 0.336, the 27-annotator figure and the negative system variance.

open https://arxiv.org/abs/2609.29445        # Table 1 (ANOVA), Table 3 (MOS, emojis), Section 4.3

python3 - <<'PY'
import numpy as np
np_, ns, nr = 290, 8, 3
print("df sys", ns-1, "df sys x rater", (ns-1)*(nr-1), "df resid", (np_-1)*(ns-1)*(nr-1))
MSs, MSr, MSsr = 11.85, 1345.16, 20.25
MSres = MSs/18.62                 # from F(7,4046) = 18.62
MSpr  = MSr/319.03                # rater tested against item x rater
T = 2*MSr/0.221                   # rater SS = 22.1% of total
MSp = 0.188*T/289                 # item SS = 18.8% of total
SSps = T - (7*MSs + 2*MSr + 14*MSsr + 4046*MSres + 578*MSpr + 289*MSp)
MSps = SSps/2023
v = dict(res=MSres, sr=(MSsr-MSres)/np_, pr=(MSpr-MSres)/ns,
         r=(MSr-MSpr-MSsr+MSres)/(np_*ns), p=(MSp-MSps-MSpr+MSres)/(ns*nr),
         ps=(MSps-MSres)/nr)
t = sum(v.values()); print({k: round(100*x/t, 1) for k, x in v.items()})
mos = np.array([3.667,3.638,3.579,3.516,3.454,3.443,3.421,3.325])
s2 = mos.var()                    # divide by 8
G = lambda n, r, s2=s2: s2/(s2 + v["sr"]/r + v["ps"]/n + v["res"]/(n*r))
print("G(290,3) =", round(G(290,3),3), " ceiling at 3 =", round(s2/(s2+v["sr"]/3),3))
print("annotators for 0.8:", next(r for r in range(1,500) if G(290,r) >= 0.8))
print("ANOVA sigma2_sys =", round((MSs-MSps-MSsr+MSres)/(np_*nr),4))
PY
# df sys 7 df sys x rater 14 df resid 4046
# {'res': 32.6, 'sr': 3.5, 'pr': 22.9, 'r': 29.2, 'p': 7.3, 'ps': 4.5}
# G(290,3) = 0.336  ceiling at 3 = 0.346
# annotators for 0.8: 27
# ANOVA sigma2_sys = -0.01

The item x rater mean square (4.216) and the item and item x system mean squares are our reconstruction, not printed values; the Figure 1(a) match is what validates them. The rater F of 319.03 uses the item x rater term as denominator, not a quasi-F, which is a simplification in the paper's table that does not affect the system test.

What would prove this wrong

The claim at stake is that this preference benchmark cannot rank the eight systems and that its ordering is a length effect. The paper's own limitation is the panel: three university-educated Bangla speakers in one age band, so the variance components are estimated from nr = 3 and could shrink with a broader panel.

A dated prediction: if anyone, the authors included, re-rates these 290 sentences x 8 systems with at least 27 annotators and publishes it by 30 September 2027, the system effect tested against system x rater will stay non-significant at p > 0.05, and emoji count will still correlate with MOS at Spearman 0.7 or above across the eight systems. We are wrong if a 27+ annotator panel yields a significant system effect under that test with a leader that is not among the three highest emoji emitters (Gemma-3-27B at 4.95, Claude-3-Haiku at 4.92, Mistral-Large at 4.87). That result would mean real quality differences were hiding under a thin panel, and our negative variance estimate was sampling noise from nr = 3.

Sources

1. Sadab, F., Sakhawat, A. Two Emojis of Difference: What Multilingual Affective Generation Benchmarks Actually Measure. arXiv:2609.29445v1, 24 September 2026. Accepted at MRL 2026 (EMNLP 2026). Tables 1 to 3, Sections 4, 5, 8, 9, Figure 1.
2. Brennan, R. L. Generalizability Theory. Springer, 2001. The decision-study formula used in Section 4.3.
3. Shrout, P. E., Fleiss, J. L. Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin 86(2), 1979.
4. Dubois, Y. et al. Length-controlled AlpacaEval. arXiv:2404.04475, April 2024. Prior evidence of length bias in preference judgements.
5. BLOMEGA. One 32-model judge panel, two answers: worth 4.24 humans, or worth 2.30. September 2026.
6. BLOMEGA. Five machine labellers marked 0, 1, 40, 72 and 78 of the same 100 scenes positive. September 2026.