Two experts, 5.6 hours of rationale labels: an LLM codebook hits 64.9 percent and beats the six-month expert version at 57.8

Six LLMs pointed two experts at 120 utterances, and that beat six months of codebook meetings

Zeyu He, Zhuqian Zhou, Kirk Vanacore, Rene F. Kizilcec and Ting-Hao 'Kenneth' Huang posted Experts Rise Where LLMs Disagree (arXiv:2609.26926v1, cs.CL, 45 pages) on 22 September 2026. The setting is a real project, not a benchmark: two education experts with doctorates in Learning Sciences built a 29-move taxonomy of tutoring behaviour (30 classes with None) for maths tutoring transcripts from Eedi, UPChieve and Third Space Learning.

The project kept two snapshots. The Initial codebook dates from January 2026, when the experts had finalised 825 utterances. The Final codebook dates from July 2026, after weekly Zoom sessions produced another 4,032 consensus labels from 110 sessions. Those 4,032 labels are the held-out test set (Section 4.1.3 and 4.2.1).

The authors took the January codebook, had six LLMs from six providers label 6,595 unseen utterances from 165 other sessions, and used where the models split to decide what the experts should look at. Three feedback formats were compared: editing LLM-proposed codebook text (Codebook Verifying, CV), answering LLM-generated questions (Question Answering, QA), and labeling contested utterances with a free-text rationale (Rationale Labeling, RL). GPT-5.6 Sol turned each kind of input into a revised codebook.

Which feedback format moved LLM accuracy, and by how much?

Each codebook was applied by the same six models at temperature 0, and the six votes were combined with eight Crowd-Kit aggregators (majority vote, Dawid-Skene, one-coin Dawid-Skene, GLAD, MMSR, Wawa, zero-based skill, MACE). Every number below is the mean over those eight.

All values from arXiv:2609.26926v1. Expert time is recorded interaction time for the revision task (Table 5: CV 25.6 and 80.5 minutes, QA 222.5 and 136.0, RL 146.9 and 191.1 for Experts A and B). The Final codebook figure is each expert's own retrospective estimate of about 300 hours on the whole project, which the authors flag as not directly comparable.

Two results are clean. Rationale Labeling lifted accuracy 11.5 points over the January codebook (53.4 to 64.9) and weighted F1 on the ten high-disagreement labels from 0.295 to 0.568. Codebook Verifying, the format most teams default to when an LLM drafts guidelines, landed at 52.0 or 53.7 percent depending on how unreviewed labels were filled, which is within about a point of doing nothing. The AI-only rewrite of all 30 labels scored 53.1.

The paper also reports, in Appendix A.2, that two automatic prompt optimisers run without gold labels (SPO at 48.7 percent, APO at 51.2) came in below the unrevised codebook (55.0) on a separate 586-utterance set. Without a ruling from someone who knows the answer, rewriting the instructions did not help.

How cross-model disagreement chooses what the experts see

Disagreement is normalised vote entropy over the six predictions, D(u), 0 when all six agree and 1 when all six differ. Each label then gets a vote-weighted mean of D(u) over utterances where at least one model chose it. Revoicing tops the ranking at 0.638 and Feedback_Correct sits at the bottom at 0.186 (Table 3). The authors split the ranking into tiers and gave CV 12 target labels and QA and RL a disjoint, adjacent 12, six per expert.

In RL the expert sees neither the target label nor any model prediction. A greedy picker chooses 10 utterances per target label that cover the most competing labels, ties broken by higher entropy, then shuffles the 120 across labels. The experts assigned 24 distinct labels to those 120 cases, so the revision touched 24 entries rather than 12 (Section 4.3.2).

Our reading of why RL wins: it is the only format where the expert rules on the exact ambiguity the models hit, in context, and writes down the reason. CV asks the expert to judge prose an LLM wrote without knowing which side of a boundary is correct. QA asks for rules in the abstract, without the transcripts; the authors' own survey has both experts rating QA 7 out of 7 for mental effort against 3.5 for RL (Table 6). That is a judgement, and the study's design cannot isolate it (see below).

Why the evaluation leans in RL's favour, by construction: the gold labels are the consensus of the same two people whose rationales RL fed to the rewriter. A codebook that encodes their judgement on hard cases should score well against their judgement. That is the goal of an annotation project, so it is fair, but it means the test is "does the LLM reproduce these two experts", not "is the label correct".

Can LLMs replace the annotators here? They replace the volume, not the expert rulings

Our recomputation of cost per point. Dividing total expert minutes (both experts) by accuracy points gained over the January codebook: RL spent 338 minutes (5.6 hours) for 11.5 points, 29.4 minutes per point. The best QA variant spent 358.4 minutes for 7.1 points, 50.5 per point. CV with LLM fill-in spent 106.2 minutes for 0.3 points, 354 per point, and CV with initial fill-in lost 1.4 points. The paper plots time against gain (Figure 6) but does not state these ratios.

The human ceiling is lower than it looks. Appendix A.1 says the experts initially disagreed on 336 of the 825 January utterances. That is 489 of 825 in agreement, 59.3 percent (our arithmetic). The best LLM codebook, at 64.9 percent against their negotiated consensus, sits above what one expert achieved against the other before discussion. The comparison is loose: different utterances, an earlier codebook, and a single rater against a pair's consensus. It still tells anyone asking "can LLMs replace annotators" that on a 30-class discourse scheme, a single trained human is not a 90 percent reference either. We made a related point about agreement statistics in LLM annotators with kappa near zero.

How good is 64.9 percent on 30 classes? The paper gives no majority-class baseline and does not publish the test-set label distribution. As a proxy, the most frequent label in the six models' 39,565 votes on the revision corpus is Prompting_Action with 5,839, 14.8 percent (Table 3, our arithmetic). The LLMs are far above a plurality guess and still wrong on roughly one utterance in three.

Decisions this supports, for a team building an LLM codebook:

• Spend expert hours on labeling contested items with a one-paragraph reason, not on reviewing LLM-drafted guideline text. In this study the second format cost a third of the time and bought nothing.
• Use models from different providers to find disagreement. Appendix A reports pairwise Cohen's kappa of 0.68 to 0.90 within a model across personas and 0.48 to 0.64 across models: provider diversity, not persona prompts, creates the signal.
• Let the rewriter touch every label an expert ruling mentions. The QA variant allowed to revise 25 labels scored 60.5 against 58.0 for the same input restricted to 12.
• Expect the gain where the models disagree most. On low-disagreement labels RL moved weighted F1 from 0.660 to 0.727; on high-disagreement labels from 0.295 to 0.568.

For the price side of this tradeoff, our note on decision models for text annotation covers cost against accuracy for cheaper labelers.

Check it yourself

There is no released code, codebook or data: the paper links no repository, the transcripts are provider-licensed, and a GitHub repository search on 27 September 2026 returned 0 hits. Everything below reproduces from the PDF tables.

curl -sL https://arxiv.org/pdf/2609.26926v1 -o p.pdf && pdftotext -layout p.pdf p.txt
grep -n "Heldout Evaluation Test Set\|averaged 53.1\|initially disagreed\|6.1% of utterances" p.txt
curl -s "https://api.github.com/search/repositories?q=codebook+disagreement+expert+llm" | grep total_count

python3 - <<'PY'
import math
acc = {"initial":.534,"cv_llm":.537,"qa_best":.605,"rl":.649,"final":.578}   # Table 1
mins = {"cv_llm":53.1,"qa_best":179.2,"rl":169.0}                               # Table 5, per expert
n = 4032
for k,m in mins.items():
    g = (acc[k]-acc["initial"])*100
    print(f"{k:8s} +{g:4.1f} pts  {2*m:6.1f} min  {2*m/g:6.1f} min/pt")
for k in ("qa_best","rl"):
    d = acc[k]-acc["final"]
    se = math.sqrt(acc[k]*(1-acc[k])/n + acc["final"]*(1-acc["final"])/n)
    print(f"{k} - final = {d*100:.1f} +/- {1.96*se*100:.2f} pts (unpaired, iid)")
print("expert-expert raw agreement:", (825-336)/825)                          # App. A.1
votes = [198,371,441,281,846,946,575,212,2672,1457,430,857,338,1350,977,2125,
         1489,627,1796,28,438,568,971,778,3706,577,5839,3075,2148,3449]      # Table 3
print("votes", sum(votes), "expected", 6*6595, "plurality share", max(votes)/sum(votes))
PY
# cv_llm   + 0.3 pts   106.2 min   354.0 min/pt
# qa_best  + 7.1 pts   358.4 min    50.5 min/pt
# rl       +11.5 pts   338.0 min    29.4 min/pt
# qa_best - final = 2.7 +/- 2.14 pts (unpaired, iid)
# rl - final = 7.1 +/- 2.12 pts (unpaired, iid)
# expert-expert raw agreement: 0.5927...
# votes 39565 expected 39570 plurality share 0.1475...

Three things the script shows that the paper does not say. First, the paper's significance test (Figure 5) is a paired t-test over the eight aggregators, which are "alternative summaries of the same model predictions", in the authors' own words (Section 4.2.4). It measures aggregator sensitivity, not whether a new sample of utterances would repeat the result. Our binomial interval is the missing sampling view: RL's 7.1 point lead clears it comfortably; the best QA lead of 2.7 points clears plus or minus 2.14 by little, and a session design effect above about 1.6 on 110 sessions would erase it (2.7 / 2.14 = 1.26, squared). Pairing on the same utterances would narrow it again; only the unpublished per-utterance predictions can settle which effect dominates.

Second, Table 3's vote counts sum to 39,565 against 6 x 6,595 = 39,570 expected, 5 votes short, most likely parse failures; it does not change any ranking. Third, Table 2 is internally consistent: weighting each group's F1 by its utterance count (1,141, 740, 2,151) reproduces the overall weighted F1 within 0.003 on all ten rows. For 6.1 percent of test utterances, about 246 (our arithmetic), the experts recorded more than one label and only the first was scored, so part of the remaining 35.1 point error is scoring convention.

What would prove this wrong

The biggest threat is order. Every expert did CV first, then QA, then RL, one to two weeks apart (Section 4.2). Expert B wrote that "the quality of my responses changed as I worked". RL always had the most practised experts. Target labels were swapped between QA and RL to limit carryover, but order itself was never counterbalanced, and with two experts it could not be.

A second threat is scope: one project, two experts, a label set that barely changed between January and July. The experts had already spent about 300 hours each before the study began, so the "hours instead of months" claim holds for experts who already know the data, which the authors concede in Section 6.3.

Dated prediction: by 30 June 2027, a replication of this workflow on a second annotation project with at least three experts and counterbalanced workflow order will still find Rationale Labeling above both Codebook Verifying and the expert-refined codebook, with a lead over the expert codebook of at least 3 accuracy points on a session-clustered bootstrap. If the lead falls under 3 points or reverses once order is counterbalanced, the RL advantage here was mostly learning from doing CV and QA first. If per-utterance predictions are released, a clustered bootstrap that puts the best QA gap's interval across zero would confirm our fragility reading of the 60.5 versus 57.8 comparison.

Sources

1. He, Z., Zhou, Z., Vanacore, K., Kizilcec, R. F., Huang, T.-H. K. Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision. arXiv:2609.26926v1, 22 September 2026. Tables 1 to 6, Sections 3 to 7, Appendices A to E.
2. Zhou, Z., Vanacore, K., Thompson, T., St John, J., Kizilcec, R. Tutor Move Taxonomy: A Theory-Aligned Framework for Analyzing Instructional Moves in Tutoring. arXiv:2603.05778, 2026. The 29-move label set.
3. Ustalov, D., Pavlichenko, N., Tseitlin, B. Crowd-Kit. Journal of Open Source Software 9(96), 2024. The eight aggregation methods.
4. He, Z., Naphade, S., Huang, T.-H. K. Prompting in the Dark. CHI 2025. Only 9 of 20 participants improved labeling accuracy by revising prompts without gold labels.
5. Chang, J. C., Amershi, S., Kamar, E. Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets. CHI 2017. Annotator disagreement as a signal about label definitions.
6. BLOMEGA. LLM annotators with kappa near zero.
7. BLOMEGA. Decision models for text annotation: cost against accuracy.