Scoring against annotator disagreement instead of majority vote flips which model wins

What changed, and when

On 20 March 2026 Tomas Ruiz, Tanalp Agustoslu and Carsten Schwemmer posted Rethinking Ground Truth: A Case Study on Human Label Variation in MLLM Benchmarking (arXiv:2603.19744, v1), published at the 2025 IEEE International Conference on Big Data. Their affiliations are not stated on the arXiv page.

The underlying data is PoliTok-DE, a corpus of 195,373 TikTok posts tied to the 2024 Saxony state election, including 18,842 posts the platform removed, first released as its own dataset paper by Ruiz, Nanz, Schmid, Schwemmer, Theocharis and Rieger in arXiv:2509.15860 (September 2025). For this follow-up analysis, 300 of the removed posts were hand-labeled by multiple human annotators on five yes/no questions: is_political, is_saxony (whether the content concerns the Saxony election specifically), is_intolerant, is_hedonic_entertainment and is_eudaimonic_entertainment, producing 1,500 individual labels.

Instead of collapsing those labels to a single majority-vote ground truth and scoring every model against it, as most MLLM benchmarks do, the authors kept every individual annotator's vote and built two separate evaluation protocols: one for questions where annotators mostly agreed, and a different one for questions where they didn't.

The evidence table

Agreement is Krippendorff's alpha (Table III of arXiv:2603.19744v1), split at the standard 0.667 reliability threshold.

On the agreement subset, models are scored with ordinary F1 against the majority label (Table I). On the disagreement subset there is no single ground truth to score against, so each model's predicted probability of "yes" is compared to the full distribution of human votes using Brier score and Jensen-Shannon divergence, lower is better on both (Table II, shown here for is_intolerant, the most-disputed question with a usable per-model breakdown).

On is_political, scale helps and the two largest models tie for best (0.95). On is_intolerant, the pattern reverses inside both families: Gemma-3 12B's Brier score of 0.17 beats the larger 27B's 0.31, a model 2.25x smaller doing 1.82x better, and Qwen2.5-VL 7B's 0.12 beats the 72B model's 0.20, a model roughly ten times smaller doing 1.67x better. In both families, the middle-sized model wins the disagreement-subset question and the largest model wins (or ties for) the clearest agreement-subset question.

Why the biggest model loses on the question humans split on

The two scoring protocols reward different things. F1 against a majority label rewards confidently picking the single most common answer, something larger models with more training signal to draw on tend to do well when there is a real consensus answer to find. Brier score and Jensen-Shannon divergence against the full annotator-vote distribution reward something else: calibration against genuine human split. A model that answers "yes" with 90% confidence on a question where humans voted 55/45 is miscalibrated even if 55% of humans did say "yes," because it is more certain than the evidence warrants.

The paper's own framing is that "parameter count alone does not determine sensitivity to ambiguity and subjectivity." Our reading of why, which the paper does not spell out mechanistically: larger models in the same family tend to produce sharper, more confident output distributions, a property that helps when there is a consensus to be confident about and hurts when there is not. is_intolerant is exactly the question with the least consensus among the three measured here (α=0.48, against 0.81 for is_political), so it is the question where confident-by-default behavior is most likely to be punished by a scoring rule built to reward matching the spread of human opinion, not just its center.

What it means for anyone benchmarking a labeling model

Check your agreement level before trusting a majority-vote leaderboard. A benchmark question with α<0.667 does not have a reliable single ground truth. Scoring any model against a shaky majority vote on such a question measures how well the model matches whichever way the vote happened to tip, not how well it understands the content.

Don't default to the largest available model for subjective labeling tasks. On this study's most disputed question, the mid-size model in each family won, and the largest model was not even second-best in one family (Gemma-3 27B placed third of three on is_intolerant). If your annotation task includes genuinely contested categories, like intolerance, offensiveness, or borderline policy violations, a smaller model evaluated for calibration may outperform a larger one evaluated for confidence.

Split your own benchmark the same way before reporting one leaderboard. A single aggregate score across questions at every agreement level hides this reversal. The fix is cheap: compute Krippendorff's alpha per question against your own annotator pool, then score the low-alpha ones against the vote distribution rather than the majority label.

Check it yourself

The ratios in the evidence section are a one-line check:

python3 -c "print(round(0.31/0.17,2), round(0.20/0.12,2), round(27/12,2), round(72/7,1))"
# 1.82 1.67 2.25 10.3

PoliTok-DE's post identifiers and rehydration code are public on Hugging Face per the parent dataset paper, arXiv:2509.15860; the content of removed posts requires a research-access request, which we have not made. The alpha and per-model score tables are in the HTML version at arxiv.org/html/2603.19744, Tables I through III. No code repository for the benchmarking scripts in arXiv:2603.19744 itself was found at the time of writing.

What would prove this wrong

This is one case study on five questions from one corpus. If an independent replication on a different human-label-variation benchmark (for example a similarly annotated content-moderation dataset in another language) finds that the largest model in each tested family wins on the lowest-agreement question as often as the mid-size model does, by 31 December 2027, the scale-reversal pattern here would look specific to PoliTok-DE's is_intolerant question rather than a general property of disagreement-aware scoring.

Sources

1. Ruiz, T., Agustoslu, T., Schwemmer, C. Rethinking Ground Truth: A Case Study on Human Label Variation in MLLM Benchmarking. arXiv:2603.19744v1, 20 March 2026; IEEE International Conference on Big Data 2025; HTML version.
2. Ruiz, T., Nanz, A., Schmid, L., Schwemmer, C., Theocharis, Y., Rieger, D. PoliTok-DE dataset paper. arXiv:2509.15860, September 2025.
3. BLOMEGA. Culture-blind judges vs same-culture annotators and Data annotation research: the latest.