Majority vote throws away 63 to 83 percent of the positives your annotators found

What changed, and when

On 18 September 2026 Maciej Skorski posted Moral Entropy: Auditing Bias and Uncertainty in Moral Judgment (arXiv:2609.21992v1), accepted to the UncertaiNLP workshop at EMNLP 2026. The contribution that matters outside computational ethics is the audit, not the framework: a per-annotator confusion-matrix model gives a posterior over the true label, and every heuristic consensus rule can then be scored against it with cross-entropy, Brier score and expected calibration error.

Three corpora, fifteen discourse domains. MFTC contributes roughly 125,000 annotations from 23 trained annotators over 33,858 tweets across seven domains. MFRC contributes 17,886 Reddit posts across Everyday Morality, US politics and French politics. eMFD contributes 54,883 GDELT-indexed news paragraphs, grouped for this study into five theme categories. The model is fitted per foundation and per corpus.

The domain is moral-foundation labelling, which is unusually subjective. That makes the effect large, not unique. Any label where competent annotators genuinely disagree, and that covers toxicity, harm, intent, sentiment, relevance and most safety taxonomies, has the same structure.

The evidence table

Rates are means over the five moral foundations, measured against the calibrated posterior except where noted. The pooled row is the headline the abstract leads with; the foundation-level rows are the ones that survive disaggregation.

All values from arXiv:2609.21992v1. The two best and worst domain figures come from different domains: false-positive range is 12.8 percent on Election to 30.4 percent on MeToo, false-negative range is 20.2 percent on Sandy to 83.8 percent on Davidson. MeToo and Sandy are the only two of the seven MFTC domains where false positives still exceed false negatives.

Read the pooled 0.0 percent false-negative rate for the OR rule as an artefact and not a result, which is what the paper does. A negative output requires every annotator to vote negative, and with a diagonal-dominant confusion prior an all-negative pattern almost never produces a posterior above 0.5. The asymmetry is close to structurally guaranteed. The 30.6 percent false-positive figure is the genuine one: roughly a third of items the permissive rule flags are not supported by the calibrated consensus.

One vote vector, four verdicts

The entropy decomposition is the result that makes the rest actionable. Total posterior entropy splits into an aleatoric part, irreducible disagreement about the content, and an epistemic part, uncertainty coming from thin or noisy annotation. In every corpus and foundation cell the epistemic term is one to three orders of magnitude smaller: under 1 percent of total entropy on MFTC and MFRC, mean 0.3 percent each, and never above 6 percent on eMFD, the corpus with the fewest annotators per item.

That rules out the comfortable reading. The entropy is not there because the model is underfitted or the data is bad. It is real disagreement, and a rule that collapses it is discarding measured signal.

The item-level evidence points the same way. Among the 100 highest-entropy Care items in the BLM domain, disagreement concentrates on hostile political rhetoric rather than on compassion-laden content: harm-adjacent language with no concrete act described, which is precisely the boundary where the moral-psychology literature says foundation assignment is hard. Annotator disagreement rates against the BLM majority run from 7.6 percent for A04 to 31.3 percent for A02, and yet the most discordant annotator corpus-wide agrees with the majority on four of five representative high-entropy items and dissents alone on the least ambiguous one. No annotator holds a fixed permissive or conservative role, which is exactly what a per-annotator confusion matrix is built to capture and what a global majority rule cannot.

Which rule degenerates is set by how many annotators you bought

At 3.00 annotations per item on MFRC a majority is two votes, so a majority rule and an absolute two-vote threshold are the same rule. Thirteen of the fifteen MFRC false-negative cells coincide for exactly that reason. They separate where items carry more annotators: on BLM at 4.92 per item, the false-negative rates are 0.64 and 0.55.

eMFD is where the point becomes unavoidable. Its subcorpora carry 1.11 to 1.83 annotations per text. A two-vote threshold on a corpus where most items never receive two annotations is not conservative, it is degenerate, and the independent expert-gold check measures the consequence at 1.5 percent false positives and 96.6 percent false negatives. The same absolute threshold looked comparatively clean on MFTC.

The practical form of this: no fixed vote-count rule generalises across annotation designs, and a rule's apparent safety is a property of the dataset you last used it on. A rule inherited from a corpus with five annotators per item and applied to one with 1.2 will fail silently and in the direction that looks like good precision.

What this means for anyone commissioning labels

Ship the vote vector, not the verdict. Every number above is computable only because the per-annotator votes survived to analysis time. A delivery format that hands over one aggregated label per item forecloses the audit permanently, and the loss is not recoverable by relabelling because the annotators have moved on. This is the same collection-side argument we made about keeping soft labels, now with a cost attached.

Pick the rule against the consequence, and say which one you picked. The choice between OR and majority is a choice between a 30.6 percent false-positive rate and a 63 to 83 percent false-negative rate. For a safety filter where a miss is expensive, the permissive rule may be correct. For a training set where precision drives everything, the strict rule may be. What is not defensible is inheriting the rule from the previous project and not reporting it.

Soft targets are close to free and measurably better. Fine-tuning on the calibrated posterior instead of the collapsed label gave consistent 2 to 3 percent accuracy gains across all five foundations, with final ROC-AUC against the hard label of 0.955 on Care, 0.950 on Sanctity, 0.949 on Fairness, 0.944 on Authority and 0.934 on Loyalty, plateauing within the first epoch. The stated mechanism is that the posterior never forces false confidence onto an ambiguous item.

The Bayesian step is cheap enough to be default. Because the epistemic share is a fraction of a percent, a Laplace approximation around the MAP fit is sufficient and full MCMC buys little. That is what turns this from a paper method into something an annotation pipeline can run on every batch.

Annotation depth is a design parameter with a known failure mode. Before choosing annotators per item, decide which aggregation rule the downstream consumer will apply, then check the rule is even expressible at that depth. Two annotations per item and a two-vote threshold is a specification that discards almost everything, and the eMFD row says so at 96.6 percent.

Check it yourself

All three corpora are public and the audit is a comparison between two label columns you can compute from raw votes. The cheapest version needs no Bayesian fit at all: count how many positives your current rule drops relative to the permissive one.

open https://arxiv.org/abs/2609.21992     # Tables 2 to 4, Section 4.1, Figures 3 to 5

# the three corpora, their sizes and their annotation depth (Table 2)
#   MFTC  33,858 tweets, ~125,000 annotations, 23 trained annotators, 7 domains
#         annotations per text: ALM 3.03, BLM 4.92, Baltimore 3.32, Davidson 3.78,
#         Election 3.88, MeToo 3.65, Sandy 3.03
#   MFRC  17,886 Reddit posts, 3 subcorpora, 2.99 to 3.00 annotations per text
#   eMFD  54,883 GDELT news paragraphs, 1.11 to 1.83 annotations per text

# the minimum viable audit on your own data: no model fit required
python3 - <<'PY'
votes = [[1,0,0,0], [1,1,0,0], [0,0,0,0], [1,1,1,0], [1,0,0], [1,1,0]]
def any_rule(v):  return int(sum(v) >= 1)
def two_rule(v):  return int(sum(v) >= 2)
def maj_rule(v):  return int(sum(v) > len(v) / 2)
for name, rule in (("any-annotator", any_rule), ("two-vote", two_rule), ("majority", maj_rule)):
    pos = sum(rule(v) for v in votes)
    print("%-14s positives: %d of %d" % (name, pos, len(votes)))
base = sum(any_rule(v) for v in votes)
for name, rule in (("two-vote", two_rule), ("majority", maj_rule)):
    kept = sum(rule(v) for v in votes)
    print("%-14s drops %d of the %d items any annotator flagged (%.0f%%)"
          % (name, base - kept, base, 100 * (base - kept) / base))
PY
# any-annotator  positives: 5 of 6
# two-vote       positives: 3 of 6
# majority       positives: 2 of 6
# two-vote       drops 2 of the 5 items any annotator flagged (40%)
# majority       drops 3 of the 5 items any annotator flagged (60%)

That snippet measures disagreement between rules, which is a lower bound on the problem and not the paper's quantity. The 63 to 96.6 percent false-negative rates are measured against a calibrated posterior or, for eMFD, against independent expert gold. If your own drop rate between rules is already large, the calibrated version will be larger.

What would prove this wrong

The audit's reference is a model, and the paper is candid that its entropy decomposition rests on a Laplace approximation around the MAP fit rather than full posterior draws. A direct NUTS validation mixed badly, with effective sample size as low as 5 out of 300 draws, and matched the Laplace result only on the small number of items where it produced usable draws. If the epistemic share is being understated by that approximation, the claim that the entropy is real disagreement rather than model uncertainty weakens.

A dated prediction: by 31 December 2027, a reparameterised sampler or an ensemble of independently seeded MAP fits, applied to MFTC, will put the epistemic share of total entropy below 5 percent, consistent with the Laplace figure of 0.3 percent on that corpus. If it comes back above 20 percent, the decomposition is approximation-driven and the false-negative audit should be reread as partly a statement about model fit.

A second falsifier that needs no new statistics: the annotator pool. MFTC, MFRC and eMFD annotators were not sampled for cultural or ideological diversity, so a well-calibrated consensus over this pool can still encode a narrow view of which cues are morally salient. If the same audit run on a differently recruited pool produces materially different rule error rates on the same items, the numbers above describe these annotators rather than these rules.

Sources

1. Skorski, M. Moral Entropy: Auditing Bias and Uncertainty in Moral Judgment. arXiv:2609.21992v1, 18 September 2026. Accepted to UncertaiNLP at EMNLP 2026. Tables 2 to 5, Sections 4.1 to 4.3, Limitations. HTML version.
2. Hoover, J., et al. Moral Foundations Twitter Corpus (MFTC), 2020. 33,858 tweets, roughly 125,000 annotations from 23 trained annotators.
3. Trager, J., et al. Moral Foundations Reddit Corpus (MFRC), 2022. 17,886 posts across three subcorpora.
4. Hopp, F., et al. extended Moral Foundations Dictionary corpus (eMFD), 2021. The 54,883 GDELT-indexed news paragraphs used here, and the expert gold labels the two-vote rule is checked against.
5. BLOMEGA. LLM judges, soft labels and human disagreement.
6. BLOMEGA. Data annotation research: the latest.