Across 14 Political-Science Studies, LLMs Match Expert Coder Agreement on Easy Tasks (0.89) and Beat 165 Crowd Workers by 0.086

What did the paper measure, and when?

On 24 August 2026, Kentaro Nakamura, Jing Ling Tan and George Yean, PhD students at Harvard's Kennedy School and Department of Government, posted Observational Equivalence of LLM and Human Annotation (arXiv:2609.22133v1, cs.CL), funded by Harvard's Institute for Quantitative Social Science. The acknowledgments thank participants at PolMeth 2026, the Society for Political Methodology's summer meeting, among a list of named readers.

The design replicates the coding step of 14 studies from general-interest political science journals, listed with the concept each one coded in the table below (Arias 2022 through Thrall 2025). For each study the authors sampled 100 texts and had three expert coders and ten LLMs, Mistral 7B Instruct v0.3, Llama 3.1-8B, Qwen2.5-7B, Qwen2.5-14B, Gemma 3-12B, Gemma 3-27B, GPT-OSS-20B, GPT-5.4 mini, GPT-5.4 and GPT-5.5, apply that study's own published codebook. A separate sample of 165 US-based crowd workers from CloudResearch Connect coded 20 texts per paper: the ten texts the LLMs agreed on most and the ten they agreed on least.

The central hypothesis, preregistered at osf.io/5wp2k, is that "recent LLMs achieve agreement with expert coders comparable to the agreement observed among experts themselves." The authors test it by comparing mean LLM-expert agreement to mean expert-expert agreement across the 14 studies, with between-study confidence intervals and Krippendorff's alpha as a robustness check, rather than a formal statistical-equivalence test with a prespecified margin.

How close is close? The study-by-study numbers

Table S8 of the paper gives mean pairwise expert-expert agreement and Krippendorff's alpha for all 14 studies. The authors classify a study as "easy" when that mean agreement is above 0.8 and "difficult" at or below 0.8, which puts nine studies in the easy group and five in the difficult one. We recomputed both group averages ourselves from the rows below and got 0.891 and 0.669, matching the paper's reported 0.89 and 0.67.

All rows from arXiv:2609.22133v1, Table S8, as posted 24 August 2026. "not reported" rows are omitted here because none occur in this table; Krippendorff's alpha runs far lower than pairwise agreement on several studies because it corrects for chance agreement on skewed label distributions.

Individual LLM scores track the group lines closely. On the nine easy studies, nine of the ten models score 0.89 to 0.91 against the 0.89 expert baseline; the tenth, Mistral 7B, scores 0.78. A majority vote across all ten models reaches 0.92. On the five difficult studies, against a 0.67 expert baseline, most models hold at 0.69 to 0.76; Mistral 7B falls to 0.57. Splitting each study's texts by whether a held-out expert called them clear or ambiguous moves every number down: on easy studies, LLM-expert agreement runs 0.91 on clear texts to 0.67 on ambiguous ones (majority vote 0.94 to 0.72); on difficult studies it runs 0.74 to 0.62 (majority vote 0.78 to 0.66).

The crowd comparison, Table S16, uses the 20-text-per-paper subsample the crowd workers also coded. Mean LLM-expert agreement on that subsample is 0.736; mean crowd-expert agreement is 0.650, a gap of 0.086. The LLM panel is closer to the experts in 12 of the 14 studies, ties in one (Blumenau and Lauderdale, 2018), and loses in one (Mattingly et al., 2025, where crowd workers beat the LLMs by 0.065). Running a two-sided sign test on the 13 non-tied studies ourselves gives p = 0.0034, so the direction of the gap is not an artifact of a handful of studies.

Why does the gap open on harder tasks, and can a codebook close it?

The paper's account is that disagreement, by experts, LLMs and crowd workers alike, is mostly a property of the text and the coding rule, not of who or what is reading it. A text where the three experts split evenly produces LLMs that are roughly three and a half times as split: cross-model Gini impurity runs 0.067 when the experts are unanimous versus 0.238 when they split, on the easy studies, and 0.141 versus 0.254 on the difficult ones. Crowd workers show the same pattern from the other direction: their own Gini impurity runs 10 to 14 points higher on the texts the LLMs disagreed on most than on the texts they agreed on most.

The clearest test of that account is the one place the authors manipulated the rule itself rather than just observing it. Of the 22 crowd workers assigned to the Schub (2022) task, 10 got the study's original codebook, a single yes/no question with no worked examples, and 12 got a revised version with explicit yes/no criteria, a "tricky or borderline" escape category, and three worked examples, all coding the same 20 texts.

Crowd Gini impurity on the same 20 texts fell from 0.418 to 0.288 (p=0.005) and coding entropy fell from 0.862 to 0.631 (p=0.011), both significant drops in how much workers disagreed with each other, and Krippendorff's alpha nearly tripled, from 0.047 to 0.355. But agreement with the experts rose only from 0.603 to 0.658, a 0.055-point gain that misses the 0.05 threshold (p=0.167). Clarifying the rule makes annotators converge on each other; it does not reliably make them converge on the expert's specific answer, because part of what the expert and the rule-writer settled on is itself a judgment call, not a fact the vaguer rule was hiding.

What this means for routing labeling work between LLMs, crowd workers and experts

Match the bar to your own schema, not to a round number. On the nine studies the authors call easy, expert-expert agreement is itself only 0.89, below what "trained human coder" implies to most people commissioning annotation. A ten-model majority vote, at 0.92, cleared that bar; a single weak model, Mistral 7B, did not (0.78). Measure the pairwise agreement your own expert reviewers reach on a schema before judging an LLM pipeline against it; judging against 1.0 guarantees disappointment regardless of annotator.

Do not treat crowd workers as a cheaper, interchangeable LLM. At a mean of 0.650 against the LLMs' 0.736 on the identical subsample, and behind in 12 of 14 studies, unscreened crowd workers trailed both experts and LLMs in this design. One caveat the authors built in matters here: crowd workers only saw the 10 most- and 10 least-LLM-agreed texts per paper, a harder-than-average mix skewed toward disagreement, so this is not a claim about crowd performance on an unselected sample of easy and hard texts together.

Use cross-model disagreement as a routing signal, not model self-reported confidence. Gini impurity across the ten LLMs was 0.067 when the three experts agreed and 0.238 when they split, on the easy studies alone. Running several models and flagging texts where they scatter costs nothing extra once you already have the ensemble for the majority vote, and it flags the same texts a held-out expert would call ambiguous.

Spend codebook-revision effort where experts themselves disagree, and expect it to buy clarity more reliably than agreement. The Schub revision cut crowd disagreement with itself by close to a third (Gini 0.418 to 0.288) without a significant move in crowd-expert agreement (0.603 to 0.658, p=0.167). That is a real result, not a null one: it tells you a clearer rule converges your annotators on a shared answer, but whether that shared answer matches the expert's is a second, separate question the rewrite does not automatically answer.

Two things from BLOMEGA's own count line up with this. Asking an LLM a battery of twenty yes/no questions about an item predicts human annotator disagreement 10.5 points better than asking it to guess a label outright, the same ambiguity-as-signal idea applied to one item instead of a whole model panel (our recount here). And the 0.086-point crowd gap here is a mild case: an earlier count of five machine labellers on the same 100 scenes found them marking 0, 1, 40, 72 and 78 positive against a human's 9, a reminder that equivalence this clean is a property of this task and these models, not a universal one (that count).

Limits. Three expert coders per study is a small panel. The crowd sample saw a selected 20-text subset per paper, not the full 100. Every number here measures agreement, not accuracy against an independent gold standard, since none of the 14 studies has ground truth beyond its own experts. All 14 are general-interest political science journals, which the authors say may not generalize to specialized subfields. No code or data repository is linked from the arXiv listing as of 24 October 2026; this note works from the arXiv HTML, PDF, and the OSF preregistration page.

Check it yourself

The paper does not (yet) link a public repository, so this reproduces two things anyone can check independently: the easy/difficult group averages from the paper's own Table S8, and the statistical significance of the 12-of-14 crowd-vs-LLM win count reported in Section 3.3 and Table S16.

curl -s https://arxiv.org/abs/2609.22133 -o abs.html
curl -sL https://arxiv.org/html/2609.22133v1 -o paper.html   # ~250KB; Table S8 is in Appendix S1

python3 - <<'PY'
# Table S8, mean pairwise expert-expert agreement, 14 studies (arXiv:2609.22133v1)
easy = {
 "Feltovich & Giovannoni 2024": 0.987, "Bush & Clayton 2023": 0.933,
 "Thrall 2025": 0.900, "Mattingly et al. 2025": 0.887, "Clark & Dolan 2021": 0.880,
 "Arias 2022": 0.873, "Blumenau & Lauderdale 2018": 0.860,
 "Parthasarathy et al. 2019": 0.853, "Jung 2020": 0.847,
}
hard = {
 "Blaydes et al. 2018": 0.753, "Lacombe 2019": 0.720, "Gilardi et al. 2021": 0.700,
 "Schub 2022": 0.640, "Osnabrugge et al. 2021": 0.533,
}
print(round(sum(easy.values()) / len(easy), 3), round(sum(hard.values()) / len(hard), 3))
# 0.891 0.669  -- matches the paper's reported 0.89 (easy) and 0.67 (difficult)

from scipy.stats import binomtest
# LLM-expert agreement beat crowd-expert agreement in 12 of 14 studies, tied in 1
# (Blumenau & Lauderdale 2018), lost in 1 (Mattingly et al. 2025, by 0.065 points).
# Drop the tie and run a two-sided sign test on the remaining 13.
r = binomtest(12, 13, p=0.5, alternative="two-sided")
print(13, 12, round(r.pvalue, 4))
# 13 12 0.0034
PY

Fields that matter if you read the full HTML yourself: Table S8 lists paper, mean_pairwise_agreement and krippendorff_alpha for each of the 14 studies; Table S16 lists the per-paper LLM-expert and crowd-expert means used for the win count above; Appendix S3.2.8 has the Schub codebook-revision numbers (Gini impurity, entropy, agreement, each with a one- or two-sided p-value as reported in the appendix).

What would prove this wrong

This is a single preprint, version 1, posted 24 August 2026 with no public code or data as of 24 October 2026. If a later version or a published replication, due on the paper's own revision timeline by 31 December 2027, releases per-study LLM-expert agreement broken out by individual model rather than by difficulty group, and more than two of the ten models fall outside the 0.69 to 0.92 range reported here on the same 14 studies, the headline equivalence claim would need revising downward for those models specifically.

The mechanism claim, that ambiguity in the text and the rule, not the reader, drives disagreement, predicts that a second controlled codebook-revision experiment on a different one of the 14 studies should show the same shape: a large, significant drop in impurity alongside a small, non-significant move in crowd-expert agreement. If a future revision experiment on another study in this set, or a close replication of the Schub revision itself, instead produces a significant rise in crowd-expert agreement of 0.1 or more, the result reported here would look like a single-study exception rather than the general mechanism the paper proposes.

Sources

1. Nakamura, K., Tan, J. L., Yean, G. Observational Equivalence of LLM and Human Annotation. arXiv:2609.22133v1, cs.CL, posted 24 August 2026. HTML version: arxiv.org/html/2609.22133v1. Tables S8, S16, S17; Appendix S3.2, S3.2.8, S4.2. Accessed 24 October 2026.
2. Nakamura, Tan, Yean. OSF preregistration 5wp2k, referenced in the paper as the source of hypotheses H1 through H3.
3. The 14 replicated studies (by coded concept, per the arXiv paper's bibliography and Table S8): Arias (2022); Blaydes, Chaney & Dichter (2018); Blumenau & Lauderdale (2018); Bush & Clayton (2023); Clark & Dolan (2021); Feltovich & Giovannoni (2024); Gilardi, Shipan & Wueest (2021); Jung (2020); Lacombe (2019); Mattingly et al. (2025); Osnabrügge, Hobolt & Rodon (2021); Parthasarathy, Rumble & Butler (2019); Schub (2022); Thrall (2025).
4. BLOMEGA. Asking an LLM twenty yes/no questions predicts annotator disagreement 10.5 points better than asking it to guess.
5. BLOMEGA. Five Machine Labellers Marked 0, 1, 40, 72 and 78 of the Same 100 Scenes Positive. The Human Marked 9.