Two coding teams agree on 73% of ICD codes, and single-gold F1 bills the rest to the model

What changed, and when

On 21 September 2026 Han-Chin Shing and Joseph Paul Cohen (equal contribution) and seven Amazon colleagues posted Decomposing Error and Style in Automated Clinical Coding (arXiv:2609.24877v1, cs.CL). The claim is narrow and testable: part of what single-gold F1 counts as model error in ICD-10 coding is a systematic, coder- or site-level policy they call coding style, written ψ, and coding should be modelled as p(code | note, ψ).

The evidence rests on a design that is rare in annotation work. ACI-Bench (Yim et al., Scientific Data 2023) is a public set of doctor-patient conversations with visit notes. Two independent groups had already coded it: ACI-Y, double-annotated and adjudicated for Yuan et al. (EMNLP Industry 2025), and ACI-M, single-coder output from MedCodER (Baksi et al., NAACL Industry 2025). On the 110 notes both cover, the note is held fixed, so any difference in codes is the coder.

The authors add two things on top. An independent audit, in which two clinical coders marked every one of the 453 codes as defensible or not (Appendix A). And a 10-dimension rubric (6 ICD dimensions such as Problem Scope, Inferential Aggressiveness and Specificity Preference, 4 CPT dimensions such as E/M Level Philosophy) scored 1 to 5 by an LLM profiler, averaged per dataset and pasted into the coder's prompt as a <coding_style> block. No fine-tuning. Anthropic Opus 4.8 runs every prompt-based method.

What the paper measured, number by number

All values from arXiv:2609.24877v1 unless marked. F1 is sample-averaged: set F1 per encounter, then the unweighted mean. Test sets: CONV 273, EHR 300, ACI-Y 110, ACI-M 110, MIMIC-IV 300 (Appendix B). CONV and EHR are internal Amazon data and cannot be checked.

The controls do the persuading. If the style block were decoration, all-1s and all-5s would leave F1 flat. Instead all-5s costs 13.5 points on CONV, 14.4 on EHR, 20.2 on ACI-Y, 20.5 on ACI-M and 10.4 on MIMIC-IV (our subtraction from Table 4), and all-1s raises EHR by 22.3, because EHR's own profile is the most minimal of the five (S1 Problem Scope 1.4, S6 Comorbidity Inclusion 1.0, Table 3). The abstract's "lowers it by up to 21" is the all-5s figure. The largest mismatched drop in Table 4 is all-1s on MIMIC-IV, 38.1 points, which the abstract does not quote.

How much of the inter-team gap is error, and how much is style?

This is our recomputation, not a figure in the paper. Non-overlap before the audit is 1 − 0.73 = 27 points of Jaccard. After the audit it is 23. So removing every code the auditors rejected closes 4 of 27 points, about 15 percent of the disagreement. The other 85 percent lies between codes that a professional auditor judged supportable on the note. The paper calls that residual style. A more cautious reading, which is ours: it is disagreement that the audit protocol cannot call an error, which includes style and also anything the two auditors themselves would not agree on (their own agreement is not reported).

The second recomputation turns the 0.73 into an F1 ceiling. For two code sets A and B on one note, set F1 = 2|A∩B| / (|A| + |B|), and Jaccard J = |A∩B| / |A∪B|. Substituting gives F1 = 2J / (1 + J) exactly, per note. That function is concave, so by Jensen's inequality the mean per-note F1 is at most 2(0.73)/1.73 = 84.4. Graded against ACI-Y gold, ACI-M, a published coding system built for this job, would score no more than 84.4 F1. On the audited sets the bound is 2(0.77)/1.77 = 87.0.

Against that ceiling the model numbers read differently. The zero-shot coder sits 36.7 points below 84.4 on ACI-Y; the matched style block recovers 21.9 of them (47.7 to 69.6), which is at least 60 percent of the gap to human-level agreement, since 84.4 is an upper bound. On ACI-M the same arithmetic gives at least 52 percent (45.8 to 65.9 against a ceiling of 84.4). Our judgement: reporting 69.6 F1 as "30 points short" is the error the paper is about. The honest comparison is to the second team, and there the shortfall is under 15 points.

The mechanism behind the gains is visible in the paper's worked example D2N093 (Figure 4). With no style block the coder emits 6 codes, including symptom codes R06.00 (dyspnea) and R60.0 (edema) that both human teams omitted. With the ACI-Y profile it emits the same 4 codes ACI-Y did. With all-5s it emits 13. Outpatient gold averages 2.3 codes per encounter (Section 5.2), about 2.1 on the ACI sets by our count (225/110 and 228/110). Much of what the style block does is tell the model how many codes, and which kinds, a given shop writes down.

Does the choice of coding method still matter once style is given?

Much less. Without style the four methods span 10.5 points, from Two-Step at 39.0 to Optimized ZS at 49.5. With a matched profile they span 4.1, from 52.1 to 56.2 (Table 6). The minimal zero-shot prompt gains 14.4 and ties the prompt-engineered one at 56.2. Two published pipelines improve by 13.1 (Two-Step, Boyle et al. 2023) and 11.2 (Yuan-Verify, Yuan et al. 2025) on the pooled average. The Introduction quotes 18.1 and 16.0 for those two pipelines; see the next paragraph for where those come from.

The text quotes Zero-shot+matched at 60.0, Optimized ZS+matched at 59.8, prompt engineering alone at +11.8, and gains of 18.1 (Two-Step) and 16.0 (Yuan-Verify), and none of those appears in Table 6. We reproduced all five: they are encounter-weighted means over the four outpatient test sets only (n = 793), where the no-style range is 38.2 to 51.0 and the matched range 56.3 to 60.0. The numbers are right, the scope is unstated. Readers comparing the text to the table will otherwise think something is off.

What this changes for annotation teams and vendor evaluations

Write the style into the guideline, as numbers. The 10 rubric dimensions in Table 1 are the decisions a coding guideline usually leaves to habit: code only the chief complaint or every documented problem, add Z-codes or not, prefer unspecified or the most specific inferable code, include comorbidities or only focal conditions. ACI-Y and ACI-M in Figure 1 split exactly there (K59.09 vs K59.00, a Z28.310 on one side and G82.50 on the other), and all four codes passed the audit. A guideline that pins each dimension to a level makes that variation a spec rather than noise.

Report agreement next to the model score, as a ceiling. A Jaccard of 0.73 is a single number a buyer can turn into an F1 ceiling with 2J/(1+J). Any vendor quoting single-gold ICD F1 without the double-coded agreement on the same notes is quoting a number with no scale. Our judgement: an evaluation set without a second independent coding pass should not be used to rank vendors separated by less than roughly the gap between 100 and that ceiling.

Split disagreement before you pay to fix it. An audit that marks each code defensible or not separates error (fix with training and QA) from policy (fix with a guideline decision). Here the split was about 15 to 85. Spending on more coder training would have targeted the smaller part.

Evaluate vendors on your style, not theirs. Matched style lifted every method on outpatient data and flattened the ranking to 4.1 points. A procurement bake-off scored against one internal gold measures which vendor's defaults happen to resemble your shop. Give each vendor the same written style profile, then compare.

Two limits. The style effect did not transfer to inpatient MIMIC-IV (48.8 to 46.1 matched), where encounters carry 14.5 codes and the outpatient-derived rubric saturates (S1 at 4.5). And ACI-Y and ACI-M have nearly identical averaged profiles, differing mainly on S2 (1.76 vs 1.40): the rubric cannot tell apart the two teams whose disagreement motivates the paper, which the authors state openly as a coverage limit.

Check it yourself

The ACI-Bench notes are public. We pulled them on 23 September 2026 to test the paper's dataset sizes, and ran the arithmetic above from the paper's own numbers.

# 1. ACI-Bench encounter count (public repo, last push 28 Jan 2026)
for f in train valid clinicalnlp_taskB_test1 clinicalnlp_taskC_test2 clef_taskC_test3; do
  curl -sL -o $f.csv https://raw.githubusercontent.com/wyim/aci-bench/main/data/challenge_data/$f.csv
done
python3 - <<'PY'
import csv, glob
ids = {r["encounter_id"]: f for f in glob.glob("*.csv") for r in csv.DictReader(open(f))}
print(len(ids))                                  # 207  (train 67, valid 20, three test splits of 40)
for e in ["D2N093", "D2N128", "D2N182"]:         # the paper's three worked examples
    print(e, ids[e])                             # all three sit in the test splits
PY

# 2. The paper's arithmetic we added
python3 - <<'PY'
f1 = lambda J: 2*J/(1+J)
print(round(100*f1(0.73), 1), round(100*f1(0.77), 1))      # 84.4 87.0  F1 ceilings
print(round((0.77-0.73)/(1-0.73), 3))                       # 0.148 share removed by audit
print(round((69.6-47.7)/(100*f1(0.73)-47.7), 2))            # 0.6  gap closed on ACI-Y
n = [273, 300, 110, 110]                                    # outpatient test sets
zs_m  = [59.1, 55.2, 69.6, 65.9]; ozs_m = [59.8, 56.1, 65.7, 64.1]
zs_n  = [44.5, 28.8, 47.7, 45.8]; ozs_n = [57.0, 38.5, 60.9, 60.1]
w = lambda v: sum(a*b for a, b in zip(v, n))/sum(n)
print(round(w(zs_m), 1), round(w(ozs_m), 1), round(w(ozs_n)-w(zs_n), 1))   # 60.0 59.8 11.8
ts = ([45.1, 24.2, 49.5, 48.1], [55.2, 51.2, 65.9, 63.6])      # Two-Step no style, matched
yv = ([45.0, 30.8, 49.6, 48.1], [57.4, 49.9, 65.1, 65.5])      # Yuan-Verify no style, matched
print(round(w(ts[1])-w(ts[0]), 1), round(w(yv[1])-w(yv[0]), 1))  # 18.1 16.1 (paper rounds 16.05 to 16.0)
PY

Three things we could not reconcile, all ours. One: Table 2 lists ACI-Y and ACI-M at 193 encounters each, and Section 4.2 holds out 150 per dataset as a style train split. For CONV (423 − 150 = 273), EHR and MIMIC-IV (450 − 150 = 300) that matches the test sizes in Appendix B. For ACI it gives 43, not the 110 test encounters reported. Either the ACI profiles came from 83 encounters (193 − 110) or the style train split overlaps the test notes; the paper does not say which. Two: ACI-Bench has 207 encounters, not 193, so 14 were dropped by the source annotation sets for reasons not given. Three: Table 7 gives Dataset Avg Style at 67.5 against both ACI-Y and ACI-M gold and Oracle at 68.5 against ACI-M, while Table 4 gives 69.6, 65.9 and 70.9 for the same conditions. The zero-shot, all-1s and all-5s rows agree across the two tables, so the matched and oracle rows were probably computed differently in Table 7, but the text does not explain how.

What would prove this wrong

The strongest alternative is that the style block is mostly a code-count instruction. Outpatient gold averages 2.3 codes per encounter, the unstyled coder over-codes (6 against 4 on D2N093), and all-1s lifts EHR by 22.3 points. The paper runs no count-only control, such as "output about k codes" with k set from the train split. Our judgement: that baseline is the omission that matters most. If a count-only prompt recovers most of the +26.4 on EHR and the +21.9 on ACI-Y, the 10 dimensions add little beyond volume calibration, and "style" reduces to a length prior.

A dated prediction: by 30 June 2027, a count-matched baseline run on the same five test sets (by the authors or anyone with ACI-Bench plus the ACI-Y and ACI-M code sets) will recover less than half of the matched-style gain on ACI-Y, that is, under 11 of the 21.9 points. If it recovers more than half, the rubric is mainly a proxy for how many codes to emit and this note overstates it.

A second test anyone with a double-coded set can run: compute mean per-note F1 of team B against team A gold directly. It must come in at or below 2J/(1+J) of the mean Jaccard. If the ACI-Y and ACI-M code sets are released and that F1 exceeds 84.4, the reported 0.73 is not a per-note mean Jaccard as described.

Sources

1. Shing, H.-C., Moriarty, J., Ware, R., Marchbanks, A., Canvasser, C., Higgins, S., Gupta, H., Wang, F., Cohen, J. P. Decomposing Error and Style in Automated Clinical Coding. arXiv:2609.24877v1, 21 September 2026. Tables 1 to 8, Appendix A and B, Figures 1, 4, 5.
2. Yim, W., Fu, Y., Ben Abacha, A., Snider, N., Lin, T., Yetisgen, M. ACI-Bench, a dataset for benchmarking visit note generation. Scientific Data, 2023. Data: github.com/wyim/aci-bench, fetched 23 September 2026 (207 encounters).
3. Yuan, Z., Shing, H.-C., Strong, M., Shivade, C. Toward reliable clinical coding with language models: verification and lightweight adaptation. EMNLP 2025 Industry Track. doi:10.18653/v1/2025.emnlp-industry.12. Source of ACI-Y.
4. Baksi, K. D., et al. MedCodER: a generative AI assistant for medical coding. NAACL 2025 Industry Track. arXiv:2409.15368. Source of ACI-M.
5. Weber-Genzel, L., Peng, S., de Marneffe, M.-C., Plank, B. VariErr NLI: separating annotation error from human label variation. ACL 2024. aclanthology.org/2024.acl-long.123.
6. Plank, B. The "problem" of human label variation. EMNLP 2022. doi:10.18653/v1/2022.emnlp-main.731.
7. BLOMEGA. Five LLM runs agree at 0.995, and the measure correlates 0.029 with what it was built to measure. 22 September 2026.