Once individual accuracy drops below 50%, majority vote makes rare-event labels worse

What changed, and when

On 12 March 2026, Gunnar P. Epping, Andrew Caplin, Erik Duhaime, William R. Holmes, Daniel Martin and Jennifer S. Trueblood posted Managing Cognitive Bias in Human Labeling Operations for Rare-Event AI: Evidence from a Field Experiment (arXiv:2603.11511v1, cs.HC and econ.GN, CC BY 4.0). It is framed as an operations paper, not a machine learning paper, and it treats the gold-standard feedback stream as a policy lever rather than a fixed property of the task.

The prevalence effect itself is old. Wolfe and colleagues established that observers miss rare targets at elevated rates in visual search, and the effect survives training and incentives. What is new here is running it inside a working annotation pipeline, then following the resulting labels all the way into a trained model.

Study 1 is a lab replication that scales the effect to the crowd. Study 1a used 39 students (mean age 19.7, 77% female) across blocks at 75%, 50% and 25% target prevalence. Study 1b used 57 students (mean age 19.2, 58% female) split into a high group (90% and 50% blocks) and a low group (10% and 50% blocks). Study 2 is the field experiment: 290 participants recruited through DiagnosUs, the gamified labeling contest run by Centaur Labs, classifying white blood cell images as blast (cancerous) or non-blast.

The evidence table

Study 2 crossed two levers. The response interface was either binary choice (BC, "is this a blast cell?") or elicited beliefs (EB, "what is the likelihood that this is a blast cell?"). The gold-standard feedback stream was either matched to the unlabeled stream at 20% positives or balanced at 50%. Cell sizes: BC 20% n=75, BC 50% n=67, EB 20% n=75, EB 50% n=73. The unlabeled QA set was 750 images, 150 blast and 600 non-blast, the latter produced by rotating 150 distinct non-blast images by 90, 180 and 270 degrees.

Most Study 2 rates are reported in figures rather than in a table, and the text describes them with "around" and "roughly". We reproduce them the way the paper states them, and write "not reported" where no number appears.

Two rows carry most of the argument. The rEB individual row shows that recalibrating each worker on its own made the miss rate worse in the low-prevalence feedback condition, roughly 60% against 35% to 40%, while cutting false alarms to about 3%. The rEB-with-crowd-recalibration row shows that doing the same correction on the aggregate, after pooling nine judgements, takes the miss rate to about 9% while leaving false alarms near 3%. Same workers, same images, same nine votes. The difference is where in the pipeline the correction is applied.

Why the crossover sits at exactly p = 0.5

The paper's cleanest result is a piece of arithmetic it states in two lines, and it explains why redundancy is not a safety net for rare events. Suppose every annotator has the same probability p of being correct on an item, and errors are independent. A crowd of one is correct with probability p. A crowd of three, decided by majority, is correct with probability p³ + 3p²(1 − p). Those two curves cross at p = 0.5. Below it, every annotator you add makes the aggregate worse.

Prevalence pushes p down. That is the mechanism by which a rare-event stream converts a wisdom-of-crowds pipeline into a wisdom-destroying one: the prevalence effect does not just add noise, it biases every annotator in the same direction, which breaks the error-independence that aggregation depends on. Study 1 confirms both halves. Crowd miss rates fell and crowd false alarm rates rose as prevalence increased, exactly tracking the individual pattern, and in the 10% and 90% extremes the crowd was worse than a randomly chosen individual.

Study 2's fix is two interventions in the pipeline, neither of which changes what an annotator is paid or how many of them there are.

Why worker-level recalibration alone backfires is worth saying plainly. Fitting each worker's probabilities to gold-standard outcomes corrects overconfidence in both directions, and in a 20% feedback stream that mostly means pushing probabilities down. Individual false alarms fall to about 3%, which looks like a win, and individual misses climb to about 60%, which is the same bias made sharper. The crowd-level fit works because it is applied after the averaging, where the systematic underestimation shows up as a shift in the calibration curve rather than as per-worker noise. The paper's calibration curves make this concrete: in one bin, roughly 75% of images carrying crowd labels between 2/7 and 3/7 were in fact blast cells.

The bias reaches the model. Expected calibration error, computed over ten equal-width bins on 750 images, was best for the crowd-recalibrated variant in both feedback conditions and worst for the worker-recalibrated variant without it, and the ranking survived into the trained CNNs. One asymmetry is worth noting: the CNNs came out slightly less miss-prone than their training labels (0.24 against 0.28 for the binary-choice variant), so models do not fully inherit label bias on miss rate, but their expected calibration error was worse than their labels' across the board.

What it means if you run or buy rare-event annotation

Your gold-standard stream is a design parameter, not a sample. Most QA schemes interleave gold items drawn from the same distribution as the work, because that feels neutral. This experiment shows the composition of that stream sets the base rate annotators experience and therefore moves their decision criterion. Balancing it to 50% cost nothing per label and moved the error split from miss-heavy toward even at both individual and crowd level. If misses are the expensive error, this is the cheapest lever in the pipeline.

Stop buying redundancy blind. Nine votes per image is a real cost, and in a rare-event regime it can buy negative value. Before scaling redundancy, estimate the per-annotator accuracy on the target class. If it is under 0.5, majority vote is the wrong aggregator and more votes make the aggregate worse, which is a claim with a closed-form proof and a matching empirical block in this paper.

Elicit probabilities and score them properly. Asking "how likely is this a blast cell?" rather than "is this a blast cell?" gave a lower crowd miss rate in the 20% condition and is a prerequisite for recalibration. Our judgement: this is under-adopted because binary labels are what most annotation tools and most downstream training loops want. The paper's answer is that the probability is the thing to store and the binary is the thing to derive later, after correction.

Recalibrate after aggregation, not before it. This is the counterintuitive result and the most transferable one. Worker-level correction alone moved the individual miss rate the wrong way. The same functional form applied to the pooled label moved the crowd miss rate from about 55% to about 9%. If you already collect gold-standard items, you already have the fitting set.

Grade the labels, not just the model. Eight data variants were built and evaluated as labels before any network was trained, and the ranking of the labels predicted the ranking of the models. A pipeline that only measures downstream test accuracy cannot see that its miss-heavy labels are miss-heavy, because the test set is labelled the same way.

Check it yourself

The crossover result needs nothing but the binomial. The field-experiment rates are reported in figures, so they cannot be recomputed from the PDF; what follows is the arithmetic that makes the crowd result predictable, plus the checks we ran on the paper's availability.

# the paper (no arXiv HTML gaps; both formats resolve)
curl -s -o /dev/null -w "%{http_code}\n" https://arxiv.org/abs/2603.11511    # 200
curl -sL -o 2603.11511.pdf https://arxiv.org/pdf/2603.11511

# where majority vote starts to hurt: crowd of N vs a single annotator
python3 - <<'PY'
from math import comb
def crowd(p, n):                       # majority of n independent annotators
    return sum(comb(n, k) * p**k * (1-p)**(n-k) for k in range((n//2)+1, n+1))

p_10pct = 1 - 0.511                    # Study 1b, 10% prevalence block: miss 51.1%
for n in (1, 3, 5, 7, 9):
    print(f"n={n}: p=0.489 -> {crowd(p_10pct, n):.3f}   p=0.650 -> {crowd(0.65, n):.3f}")
PY
# n=1: p=0.489 -> 0.489   p=0.650 -> 0.650
# n=3: p=0.489 -> 0.484   p=0.650 -> 0.718
# n=5: p=0.489 -> 0.479   p=0.650 -> 0.765
# n=7: p=0.489 -> 0.476   p=0.650 -> 0.800
# n=9: p=0.489 -> 0.473   p=0.650 -> 0.828

# the linear-in-log-odds recalibration the paper fits at worker and crowd level
python3 - <<'PY'
import math
def llo(p, a, b):                      # Gonzalez-Wu form: logit-linear in log odds
    if p in (0.0, 1.0): return p
    z = a * math.log(p/(1-p)) + b
    return 1 / (1 + math.exp(-z))
# an underestimating crowd (a<1 compresses, b>0 shifts the whole curve up)
for p in (0.20, 0.30, 0.40, 0.50):
    print(f"raw {p:.2f} -> recalibrated {llo(p, 1.4, 0.9):.3f}")
PY
# raw 0.20 -> recalibrated 0.261
# raw 0.30 -> recalibrated 0.429
# raw 0.40 -> recalibrated 0.582
# raw 0.50 -> recalibrated 0.711

The LLO parameters above are ours, chosen to illustrate the shape; the paper fits them per worker and per crowd dataset from gold-standard items, and does not publish the fitted values. What the second snippet shows is the mechanism the paper describes in Section 3.2.2: a curve of this shape moves a band of crowd labels from below 0.5 to above it, which is precisely where the miss rate lives.

Two things we could not verify. The paper lists no public code or data repository, so the 750-image QA set, the 580-image gold-standard set and the per-worker judgements are not available for re-analysis. The Study 2 miss and false alarm rates in the table above are read off figures by the authors' own prose, so they carry the precision of "around" and should not be quoted to a decimal place.

What would prove this wrong

The transferable claim is that in a rare-event labeling stream, balancing the gold-standard feedback prevalence and applying a calibration fit to the aggregated label both reduce misses, and that the second matters more than adding annotators.

A dated prediction. By 31 March 2027, a replication on a different rare-event annotation task with true prevalence at or below 20%, at least 100 annotators, and the same two levers, will show crowd-level recalibration reducing the miss rate by at least 20 percentage points relative to an uncalibrated majority vote on the same judgements, at a false alarm rate no more than 10 percentage points higher. A replication that finds crowd recalibration neutral or harmful on that comparison, with the judgements published, falsifies the reading here. The weakest point is the population: DiagnosUs contestants self-select into contests and compete for prizes, which is a specific incentive structure and not every annotation workforce.

Sources

1. Epping, G. P., Caplin, A., Duhaime, E., Holmes, W. R., Martin, D., Trueblood, J. S. Managing Cognitive Bias in Human Labeling Operations for Rare-Event AI: Evidence from a Field Experiment. arXiv:2603.11511v1, 12 March 2026. Sections 2.1 to 2.4, 3.1 to 3.4 and 4.1; PDF.
2. Wolfe, J. M., Horowitz, T. S., Van Wert, M. J., et al. Low target prevalence is a stubborn source of errors in visual search tasks. Journal of Experimental Psychology: General, 2007. The prevalence effect the paper builds on.
3. Centaur Labs. Centaur Labs Medical Data Labeling Demo. YouTube, 2 December 2021. The DiagnosUs platform used in Study 2.
4. Guo, C., Pleiss, G., Sun, Y., Weinberger, K. Q. On Calibration of Modern Neural Networks. arXiv:1706.04599, 2017. Source of the expected calibration error definition used with ten bins.
5. BLOMEGA. Half of NVD's CWE labels match the vendor's own, on what an unaudited label layer costs downstream.
6. BLOMEGA. Data annotation research: the latest.