One dissenting vote in 51 makes a CIFAR-10H image "ambiguous": the 50.4% headline falls to at most 26.2%

What the paper claims, and when

On 28 September 2026 Manya Singh and Arjun Pakrashi (School of Computer Science, University College Dublin) posted Does Model Uncertainty Track Human Ambiguity? Evidence from Multi-Annotator Vision Benchmarks (arXiv:2609.34506v1). They fine-tuned eight timm models (ResNet-18/34/50, EfficientNet-B0/B1/B2, MobileNetV3 Small and Large) on CIFAR-10 and FER+ and compared their predictions with the multi-annotator test labels of CIFAR-10H (about 51 raters per image) and FER+ (10 raters per image).

Two results carry the paper. At the continuous level, single-model softmax confidence and predictive entropy correlate with normalised human vote entropy at Spearman |rho| of 0.24 to 0.35 on CIFAR-10H and 0.41 to 0.55 on FER+ (Table V). At the discrete level, each image is sorted into four profiles by whether humans and models are each unanimous. The human-only-disagreement cell, HoD, is 5,044 images (50.4%) on CIFAR-10H and 1,195 (33.5%) on FER+ (Tables IV and III). The abstract reads HoD as images that "receive multiple valid classifications from humans, while the models converge on only one."

The definition matters here. Section III-F sorts images by "whether the annotators and the models are each internally unanimous or divided". Humans agree only when every vote lands on one class. There is no minimum vote count or share for the second class, so one dissent among 51 is enough.

How much of the headline survives a stricter definition of "divided"?

We downloaded the public vote files and recomputed the human side under the paper's rule and under stricter ones. First, a replication. From cifar10h-raw.csv with attention-check trials removed and every annotator kept, we get 3,565 unanimous and 6,435 divided images, the paper's exact Table IV totals, and a mean normalised vote entropy of 0.075, the paper's figure. From FER+ fer2013new.csv (PrivateTest rows with an image and at least one emotion vote, 3,573 images against the paper's 3,572) we get 1,004 unanimous against the paper's 1,001 and a mean entropy of 0.247, again the paper's figure. The human side reproduces.

HoD ranges are our arithmetic, since the paper does not release per-image model predictions. Under a stricter rule the divided set shrinks, so HoD can be no larger than the new divided count and no larger than the paper's 5,044 (CIFAR-10H) or 1,195 (FER+), and no smaller than the new divided count minus the paper's mutual-disagreement cell, 1,391 and 1,376. On FER+ one vote is 10% of a 10-vote image, so the 10% rule equals the any-vote rule there.

The two datasets react very differently. CIFAR-10H falls from 64.4% to 8.9% divided at a 10% threshold, a factor of seven, and to 1.7% at 30%. FER+ holds at 71.9% through 10%, falls to 45.1% at 20% and 23.8% at 30%, and 9.0% of its images still have two emotions each above 40% of the votes. FER+ ambiguity is real at any sensible threshold. Most CIFAR-10H "ambiguity" is a small tail of stray votes on otherwise clear images, which the paper itself half-concedes in Section IV-B when it notes that most divided CIFAR-10H images sit well below the mean vote entropy of 0.075.

Why one dissenting vote carries so much weight on CIFAR-10H

With about 51 raters per image, the chance that at least one of them slips is high even on an unambiguous picture. Our recount finds that 2,688 of the 6,435 divided images (41.8%) have exactly one dissenting vote, and the median divided image has two. The median second-class share among divided images is 2% of the votes. A 50 to 1 split is a label that humans agree on, with a lapse attached.

Some of the lapses are not ambiguity at all. The CIFAR-10H raw file records 10 attention checks per annotator. The cifar10h-counts.npy file that the dataset authors publish, and describe as the labels used in their own work, keeps only the 2,555 of 2,571 annotators who passed at least 80% of those checks. We confirmed this by rebuilding the file: raw votes from annotators at or above 80% attention accuracy reproduce it cell for cell. The 16 dropped annotators cast 3,200 votes and were right on 37.1% of normal trials, against 95.2% for everyone else. On their own they turn 828 unanimous images into divided ones. The paper's 3,565 unanimous images match the unfiltered file exactly, so the 50.4% includes those 828. That is an inference from the exact match, not something the paper states.

The effect is lopsided because the rule is. Adding raters can only move an image from unanimous to divided, never back, so on a 51-rater dataset the one-vote rule measures rater count and rater carelessness as much as the image. FER+ has 10 raters, so one vote is already 10% of the distribution, and its headline is far less sensitive. Judgement: the one-vote rule makes the two datasets' HoD shares incomparable, which undercuts the paper's comparison of label provenance across them.

Per class, the one-vote rule puts every CIFAR-10H class between 51.3% (horse) and 77.6% (deer), with airplane second at 74.0%. At a 10% threshold the order becomes interpretable: cat 18.3%, deer 16.4% and dog 11.1% lead, airplane falls to 7.5% and ship to 2.4%. The paper's Fig. 6 also finds deer and cat on top using mean entropy, which agrees with the strict rule, not the loose one. On FER+ at the 30% multi-target rule, fear (38.3%, 94 images), sadness (35.9%), disgust (34.8%, 23 images) and contempt (31.0%, 29 images) lead and happiness trails at 8.6%.

What this means if you use human disagreement to audit model uncertainty

The paper's correlations stand; its discrete headline does not travel. The Spearman values of 0.24 to 0.55 (Table V) and the aggregate 0.363 and 0.572 for mean predictive entropy (Table VI) use continuous vote entropy, which we reproduce, and no threshold enters them. The claim that half of CIFAR-10H gets "multiple valid classifications" from humans depends on counting one vote in 51 as a valid class. Under the FER+ authors' own outlier rule, which zeroes any emotion with a single vote, CIFAR-10H HoD is between 12.3% and 26.2%.

Fix the threshold before you count ambiguity, and report the sweep. A single cut-off turns a continuous vote distribution into a headline, and on CIFAR-10H the headline moves from 64.4% to 1.7% across reasonable cut-offs. Reporting divided shares at any-vote, 2-vote and 30% levels costs three lines of code and shows a reader whether the ambiguity is in the images or in the raters.

Apply the dataset's own rater filter. CIFAR-10H ships a filtered counts file for a reason. Here 16 annotators out of 2,571, 0.6% of them, account for 828 of the divided images. If your ambiguity budget, active-learning pool or expert-review queue is sized from raw votes, part of it is paying for inattention. This is the same trap we found in HH-RLHF label noise, where the noise estimate swung with the instrument.

Rater count is a confound. More raters raise the any-vote divided share mechanically. When you compare ambiguity across datasets, as the paper does across FER+ and CIFAR-10H, compare entropy or a share-based threshold, not unanimity. Our related note on LLM judges and soft labels shows the same point from the other side: how many voters you have sets how much disagreement you can see.

Where the paper's warning still holds. On FER+, 23.8% of test faces have two emotions each above 30% of votes, and the paper finds models unanimous on 1,195 human-divided images. Even under the strictest rule we tested, FER+ keeps a large, genuinely contested set that model agreement cannot surface. For subjective labels, the paper's advice to validate uncertainty against multi-annotator data before deploying it stands.

Check it yourself

Both vote files are public. The script below reproduces the paper's human-side totals and our threshold sweep in under a minute (needs numpy and pandas).

curl -sLO https://github.com/jcpeterson/cifar-10h/raw/master/data/cifar10h-raw.zip && unzip -o cifar10h-raw.zip
curl -sLO https://github.com/jcpeterson/cifar-10h/raw/master/data/cifar10h-counts.npy
curl -sLO https://raw.githubusercontent.com/microsoft/FERPlus/master/fer2013new.csv

python3 - <<'PY'
import numpy as np, pandas as pd
r = pd.read_csv('cifar10h-raw.csv'); t = r[r.is_attn_check == 0]
raw = np.zeros((10000, 10), int)
np.add.at(raw, (t.cifar10_test_test_idx, t.chosen_label), 1)   # column name as in the file
filt = np.load('cifar10h-counts.npy')                           # annotators with >=80% attention checks
d = pd.read_csv('fer2013new.csv'); d = d[(d.Usage == 'PrivateTest') & d['Image name'].notna()]
E = ['neutral','happiness','surprise','sadness','anger','disgust','fear','contempt']
fer = d[E].values; fer = fer[fer.sum(1) > 0]
def div(m, share=0, votes=1):
    s = m / m.sum(1, keepdims=True)
    ok = (m >= votes) & ((s > 0) if share == 0 else (s >= share - 1e-12))
    return 100 * (ok.sum(1) >= 2).mean()
for name, m in [('cifar raw', raw), ('cifar filtered', filt), ('fer+', fer)]:
    print(name, len(m), 'any %.2f  2votes %.2f  10%% %.2f  30%% %.2f' %
          (div(m), div(m, votes=2), div(m, 0.10), div(m, 0.30)))
dis = raw.sum(1) - raw.max(1); print('cifar divided with exactly 1 dissent:', (dis == 1).sum())
PY
# cifar raw 10000 any 64.35  2votes 26.17  10% 8.86  30% 1.68
# cifar filtered 10000 any 56.07  2votes 25.14  10% 8.96  30% 1.72
# fer+ 3573 any 71.90  2votes 44.89  10% 71.90  30% 23.82
# cifar divided with exactly 1 dissent: 2688

The field to watch is is_attn_check: filter it out, then compare the per-annotator mean of correct_guess on attention-check rows against 0.8 to rebuild cifar10h-counts.npy exactly. The HoD ranges in the table then follow from the paper's Table IV (MD = 1,391) and Table III (MD = 1,376): 2,617 minus 1,391 is 1,226, or 12.3%. Everything in this section is our computation, not the paper's.

What would prove this wrong

Our HoD ranges are bounds, because the paper does not publish per-image model predictions. If the eight models happen to be unanimous on every CIFAR-10H image with a two-vote second class, HoD under that rule is 26.2%, still about half the headline. The one outcome that would rescue the 50.4% as a statement about ambiguity is evidence that single-vote dissents mark genuinely ambiguous images, for example if a fresh, attention-filtered panel of 51 raters splits those 2,688 images again at well above the rate it splits the 3,565 unanimous ones.

A dated prediction: if the authors or anyone else release per-image predictions for these eight models by 30 June 2027, CIFAR-10H HoD recomputed on the dataset's filtered cifar10h-counts.npy will be below 44%, and under the two-vote rule below 22%. Our estimate for the first figure is about 42.8%, assuming the 828 images that flip behave like the images humans agreed on, where the paper finds the models unanimous 93% of the time (3,310 of 3,565). A value above 46% would mean those 828 images are harder for models than ordinary unanimous ones, and that the low-attention votes carried some signal after all.

Sources

1. Singh, M., Pakrashi, A. Does Model Uncertainty Track Human Ambiguity? Evidence from Multi-Annotator Vision Benchmarks. arXiv:2609.34506v1, 28 September 2026. Section III-F, Section IV-B, Tables III to VI.
2. Peterson, J. C., Battleday, R. M., Griffiths, T. L., Russakovsky, O. CIFAR-10H repository (cifar10h-raw.zip, cifar10h-counts.npy), accompanying Human uncertainty and classification, ICCV 2019, arXiv:1908.07086. Files retrieved 29 September 2026.
3. Microsoft. FERPlus repository, fer2013new.csv and src/ferplus.py (single-vote outlier removal, 30% multi-target rule). Retrieved 29 September 2026.
4. Barsoum, E., Zhang, C., Canton Ferrer, C., Zhang, Z. Training deep networks for facial expression recognition with crowd-sourced label distribution. ICMI 2016, arXiv:1608.01041.
5. BLOMEGA. An LLM judge matches the majority label, but predicts human disagreement worse than 3 voters.
6. BLOMEGA. HH-RLHF's label noise is 39% or 1.8%, depending on the instrument.