Asking an LLM twenty yes/no questions predicts annotator disagreement 10.5 points better than asking it to guess

What changed, and when

On 13 October 2026, Irem Ecem Yelkanat, Faeze Ghorbanpour and Alexander Fraser at TU Munich posted Decomposing and Predicting Annotation Disagreement in Harmful Content Across Languages (arXiv:2610.18588v1). The paper treats annotator disagreement not as noise to discard but as a property worth predicting in advance, so contested items can be routed to extra human review before a dataset ships.

Their method asks an LLM twenty fixed binary questions per item, grouped into content questions (slurs, dehumanization, blaming), boundary questions (ambiguous target, borderline severity, missing context) and language and context questions (local slurs, coded insults, community-specific cues). A random forest trained on the yes/no answers, plus the model's token-probability confidence in each answer, predicts whether at least two annotators will assign different labels. The questions never reference a specific dataset's label scheme, so the same twenty-question feature vector applies across datasets and languages.

Three open-weight LLMs answer the questions: Gemma 3 27B, Mistral Small 3.2 24B and Qwen 3.6 27B. Four datasets provide the disagreement labels: MHS (Measuring Hate Speech, 29,453 English comments, 40.4% disagreement), HateXplain (19,192 English posts, 48.7%), the Turkish Hate Speech Dataset (60,308 tweets, 46.6%) and German DeTox (9,689 comments, 21.5%).

The evidence table

Direct prompting asks the same LLM, in one call, whether annotators would disagree on an item, scored by its probability of answering "disagree". The question-based features ask twenty separate yes/no questions, then predict disagreement from the pattern of answers and confidences. Averaged across the three LLMs, from the paper's own Table 2:

Computed by averaging Gemma 3 27B, Mistral Small 3.2 24B and Qwen 3.6 27B rows from Table 2 of arXiv:2610.18588v1 (direct prompting, and content+boundary+language questions plus confidence). Gains match the paper's own stated per-dataset averages in Section 5.1.

The mechanism: structured questions instead of a holistic judgment

The paper's explanation for the gap is not that the LLM "knows" more when asked twenty questions. It is that direct prompting asks the model to simulate the aggregate outcome of multiple human annotators in one step, a task LLMs are documented to do poorly (Ni et al., 2026; Lu et al., 2026, both cited in the paper). Targeted questions instead ask the model to judge properties it can assess more reliably, such as whether a slur is present or whether the severity looks borderline, and leave the harder job of combining those signals into a disagreement prediction to a separate, trained classifier.

Two questions carry the most weight across every dataset and every LLM tested, per TreeSHAP attribution: the derogation question ("does the text use a negative or derogatory description of a group?") ranks in the top five most important features in all twelve dataset-LLM combinations, and the context-or-intent question in ten of twelve. Which other questions matter is dataset-specific: on the Turkish dataset, the five boundary questions together carry more attribution than all ten content questions combined, consistent with disagreement there depending more on slang, sarcasm and borderline severity than on the presence of harmful content itself.

What it means for anyone building or buying annotation

A disagreement predictor is a routing tool, not a labeling shortcut. The paper is explicit that the score should send contested items to human review, not replace human adjudication or auto-resolve them. Treating a high disagreement score as a reason to discard an item would suppress exactly the perspectivist signal the method is designed to surface.

The eight-question compact set makes this cheap enough to run on every item. A beam-searched subset of eight questions reaches 71.2 to 71.9 ROC-AUC against the full twenty-question model's 72.5 to 73.4, a gap of at most 1.5 points, while cutting LLM calls per item by 60%. For a vendor scoring millions of items, that is the difference between a pre-filter that is affordable to run on everything and one reserved for a sample.

Cross-lingual transfer works without any language feature, but not for every metric. Trained on three datasets and evaluated on a fourth, unseen one, the predictor keeps 89.5 to 95.2% of in-domain ROC-AUC. PR-AUC and MCC degrade more on the datasets with rarer disagreement (MHS, DeTox), meaning a transferred model should be re-thresholded on a small target-language sample rather than reused with its source threshold.

Our judgement, specific to multilingual annotation programs: this is a usable pre-annotation triage step for any pipeline labeling subjective content across languages, provided the output is treated as a queue-prioritization signal and re-validated per language before being trusted for anything downstream of routing. It pairs with our separate finding that five LLM annotators can disagree with each other almost as much as they disagree with a human: predicting where that disagreement will land is a different, and more tractable, problem than eliminating it.

Check it yourself

open https://arxiv.org/abs/2610.18588     # paper, Tables 1-3, Figures 1-2, Section 5

# what to check without rerunning anything:
# Table 2 (Section 5.1): direct prompting ROC-AUC per LLM per dataset,
#   vs. the "C + B + L + conf." row (full question set)
# Section 5.1: "question features ... outperform direct prompting by 10.5
#   ROC-AUC points ... and the strongest supervised embedding baseline by 5.0 points"
# Section 5.1, compact set: 8-question ROC-AUC 71.2-71.9 vs full-set 72.5-73.4,
#   "reducing LLM calls by 60%"
# Table 3 (Section 5.3): leave-one-dataset-out transfer, Rel. column = 89.5-95.2%
# Figure 2: per-question TreeSHAP importance, by dataset and LLM

python3 - <<'PY'
# average direct-prompting and full-question-set ROC-AUC across the 3 LLMs,
# reproducing the Table 2->Section 5.1 per-dataset gains
rows = {
    "MHS":        {"direct": [63.8, 59.6, 59.9], "full": [74.7, 76.1, 76.3]},
    "HateXplain": {"direct": [58.1, 59.0, 59.1], "full": [64.6, 65.5, 66.0]},
    "Turkish":    {"direct": [68.0, 67.3, 67.1], "full": [73.4, 72.2, 73.2]},
    "DeTox":      {"direct": [64.3, 70.6, 67.5], "full": [81.6, 82.4, 84.1]},
}
for name, v in rows.items():
    d = sum(v["direct"]) / 3
    f = sum(v["full"]) / 3
    print(f"{name:12s} direct {d:5.1f}  full {f:5.1f}  gain {f-d:+.1f}")
PY
# MHS          direct  61.1  full  75.7  gain +14.6
# HateXplain   direct  58.7  full  65.4  gain  +6.6
# Turkish      direct  67.5  full  72.9  gain  +5.5
# DeTox        direct  67.5  full  82.7  gain +15.2

What would prove this wrong

This is a result on three specific open-weight LLM snapshots (Gemma 3 27B, Mistral Small 3.2 24B, Qwen 3.6 27B) and four specific datasets. The paper's own limitations note that extending it to more languages, tasks and model scales "is straightforward given the dataset-independent design of the questions", which makes the claim easy to test further.

A dated prediction: by 31 December 2027, a published replication using a frontier-tier LLM (GPT-5-class, Claude Opus-class, or Gemini-class) on a fifth harmful-content dataset in a language not tested here will still show the same qualitative pattern: question-based features beating direct prompting by at least 5 ROC-AUC points, with the derogation and context-or-intent questions among the top predictors. If a replication shows direct prompting matching or beating question-based features on a new dataset, or shows the derogation question dropping out of the top predictors entirely, this prediction is wrong.

Sources

1. Yelkanat, I. E., Ghorbanpour, F., Fraser, A. (TU Munich). Decomposing and Predicting Annotation Disagreement in Harmful Content Across Languages. arXiv:2610.18588v1, 13 October 2026. Tables 1-3; Figures 1-2; Sections 3, 5.
2. BLOMEGA. Five LLM annotators compared against one human on 100 scenes.
3. BLOMEGA. Data annotation research: the latest.