Majority Vote is underconfident, Dawid-Skene is overconfident, in opposite directions, across 39 crowdsourcing datasets

What changed, and when

On 12 October 2026, Gabriel Singer, Reda Arab, Maïssa Frikha, Samuel Gruffaz, Olivier Vo Van, Nicolas Vayatis and Argyris Kalogeratos (ENS Paris-Saclay, CentraleSupélec, Tampere University, and SNCF) posted On the Calibration of Crowdsourcing Aggregators (arXiv:2610.16815v1). Crowdsourcing aggregation methods, human and now LLM-based, are widely used to turn multiple noisy annotations per item into a single probabilistic prediction. Calibration, whether a predicted 80% confidence is actually correct 80% of the time, has been studied extensively for classifiers but, per the authors, never systematically for the aggregation step itself.

The paper combines a theoretical result with a large empirical sweep. Theoretically, it proves the Bayesian aggregator (the one that minimizes classification error using true per-annotator error rates) is perfectly calibrated by construction, for any number of annotators. It also derives the condition under which a weighted majority vote can be exactly calibrated in the limit, and shows that condition requires the crowd's average per-class accuracy to approach perfection, which real, noisy crowds essentially never do. Empirically, it measures Expected Calibration Error for Majority Vote, Dawid-Skene, GLAD and CrowdFM across 39 real-world human crowdsourcing datasets (image annotation, sentiment, toxicity, and more, drawn from LabelMe, Snow et al.'s NLP tasks, Jigsaw toxicity data, CIFAR-10H and others) plus an LLM-crowd version of the PASCAL RTE-1 entailment benchmark and an MMLU Clinical Knowledge set.

The evidence table

Expected Calibration Error (ECE, lower is better) for the three weighted-vote aggregators and three model-based aggregators, raw versus standard temperature scaling (TS) versus the paper's proposed fix (Soup+TS, averaging complementary aggregators' predictions before scaling), averaged across the 39 human datasets.

Reproduced from Table 1 of arXiv:2610.16815v1, Section 6.3, averaged over 39 human crowdsourcing datasets (N=39).

Plain temperature scaling nearly fixes the model-based aggregators (Dawid-Skene drops from 13.58% to 5.88%) but barely moves the weighted-vote ones (Majority Vote only drops from 11.98% to 10.64%), because temperature scaling cannot correct for a miscalibration direction baked into how the vote is computed. The paper's Soup+TS brings the weighted-vote family down further still (Majority Vote to 8.48%), at a small cost to the already-good model-based aggregators (Dawid-Skene rises slightly from 5.88% to 7.17% under Soup+TS versus plain TS, though both are far below the 13.58% raw figure).

The mechanism: why Majority Vote structurally resists calibration

The paper's theoretical contribution explains the direction, not just the magnitude, of the miscalibration. It proves that the Bayesian vote, which weights each annotator by their true, known error rates, is perfectly calibrated for any number of annotators. It then shows that for the broader class of weighted majority votes (which includes plain Majority Vote and Wawa), exact asymptotic calibration requires a specific condition: as the crowd grows, its average per-class accuracy must converge to a value such that the weighted sum of annotator accuracies, for any true class, approaches 1, meaning the crowd is already almost never wrong. Real crowds, human or LLM, essentially never meet this condition, which the authors offer as the structural reason Majority Vote is calibrated so inconsistently in practice.

What it means for anyone aggregating crowd or LLM labels

A confidence score from Majority Vote should be read as a floor, not an estimate. Because it is systematically underconfident, a pipeline that discards items below a Majority Vote confidence threshold is probably discarding more usable items than the threshold intends. Conversely, a Dawid-Skene or GLAD confidence score should be read as an upper bound: the model is telling you it is more certain than it has earned.

The miscalibration extends to LLM crowds, not just human ones. On the LLM-annotated RTE dataset, Majority Vote's ECE was 1.5% (close to calibrated) while Dawid-Skene's was 8.9%, a different magnitude than on the human RTE data but the same qualitative direction: the model-based aggregator overclaims confidence relative to the vote. Anyone treating an ensemble of LLM annotators as automatically well calibrated because "it's just an LLM crowd, not real crowdsourcing" is working from an untested assumption this paper directly tests and rejects.

Fixing calibration does not require discarding the aggregator you already use. Soup+TS is a post-hoc wrapper: it combines a target aggregator's output with a small set of complementary aggregators before applying standard temperature scaling, and the paper reports it generally preserves, and on human datasets often improves, the underlying accuracy (macro-F1) of the base aggregator while cutting calibration error roughly in half for the weighted-vote family.

Our judgement, specific to any pipeline that reports a confidence score alongside an aggregated label: if that confidence feeds a downstream decision (routing to review, setting a payout tier, flagging for audit), it needs its own calibration check against held-out ground truth before being trusted, exactly as a classifier's confidence would. An aggregation method's label accuracy and its probability calibration are two different properties, and this paper is the first to measure the second one directly.

Check it yourself

open https://arxiv.org/abs/2610.16815      # paper, Table 1, Figures 1-3, Sections 4-6

# what to check without rerunning anything:
# Figure 1 (Section 1): reliability diagrams for Majority Vote and Dawid-Skene
#   on the Adult dataset and averaged over 39 datasets
# Section 6.2: "On RTE dataset and the LLM crowd dataset, Dawid-Skene reaches
#   respectively ECE of 6.8% and 8.9%, while MV reaches respectively ECE of
#   16.9% and 1.5%"
# Table 1 (Section 6.3): ECE for MV/Wawa/ZBS and DS/GLAD/CrowdFM,
#   raw vs. TS vs. Soup+TS, on 39 human datasets (N=39) and 1 LLM crowd (N=1)
# Section 4: Proposition 1 (Bayesian vote perfectly calibrated),
#   Theorem 1 + Corollary 1 (weighted majority vote calibration condition)

python3 - <<'PY'
# Table 1: ECE improvement from raw to Soup+TS, human datasets (N=39)
rows = {
    "Majority Vote": (11.98, 8.48),
    "Wawa":          (10.67, 8.24),
    "ZBS":           (10.68, 8.20),
    "Dawid-Skene":   (13.58, 7.17),
    "GLAD":          (13.69, 6.27),
    "CrowdFM":       (12.50, 4.62),
}
for name, (raw, fixed) in rows.items():
    pct = (raw - fixed) / raw * 100
    print(f"{name:14s} raw {raw:5.2f}%  Soup+TS {fixed:5.2f}%  relative cut {pct:4.1f}%")
PY
# Majority Vote  raw 11.98%  Soup+TS  8.48%  relative cut 29.2%
# Wawa           raw 10.67%  Soup+TS  8.24%  relative cut 22.8%
# ZBS            raw 10.68%  Soup+TS  8.20%  relative cut 23.2%
# Dawid-Skene    raw 13.58%  Soup+TS  7.17%  relative cut 47.2%
# GLAD           raw 13.69%  Soup+TS  6.27%  relative cut 54.2%
# CrowdFM        raw 12.50%  Soup+TS  4.62%  relative cut 63.0%

What would prove this wrong

The empirical claim rests on 39 human datasets but only one LLM-crowd dataset (PASCAL RTE-1) plus one additional medical multiple-choice set reported in the appendix, which the authors flag as a real limitation: "they are based on two datasets in the case of LLM annotators, which limits the scope of our findings in that setting."

A dated prediction: by 31 December 2027, a published study measuring calibration of Majority Vote versus Dawid-Skene-family aggregators on five or more additional LLM-crowd datasets will still find the two in opposite miscalibration directions (Majority Vote underconfident, model-based aggregators overconfident) on a majority of them. If a broader LLM-crowd study finds the two aggregator families miscalibrated in the same direction, or finds Majority Vote overconfident on LLM crowds specifically, this prediction is wrong, and the direction of miscalibration would need to be treated as annotator-population-specific rather than a general property of the aggregation method.

Sources

1. Singer, G., Arab, R., Frikha, M., Gruffaz, S., Vo Van, O., Vayatis, N., Kalogeratos, A. (ENS Paris-Saclay, CentraleSupélec, Tampere University, SNCF). On the Calibration of Crowdsourcing Aggregators. arXiv:2610.16815v1, 12 October 2026. Table 1; Figures 1-3; Sections 4-7.
2. BLOMEGA. Goldsmith cuts gold loss 93% on its own 15 examples; held-out gain is 0.48 F1 points.
3. BLOMEGA. Data annotation research: the latest.