An LLM-Assisted Audit Finds 11 Gold-Label Errors in SciFact's Dev Set

What changed, and when

SciFact is the standard benchmark for scientific claim verification: given a claim and evidence sentences pulled from PubMed abstracts, a system labels the claim SUPPORT, CONTRADICT, or NOT ENOUGH INFO (NEI). David Wadden and co-authors introduced it at EMNLP 2020 (arXiv:2004.14974), and it has since been folded into the BEIR retrieval suite and cited more than 645 times on Semantic Scholar as of April 2026. Its gold evidence labels were assigned by domain experts at a reported Cohen's kappa of 0.75, a respectable but imperfect agreement score that leaves room for disagreement by construction.

Nobody checked that room for six years. Julien Sylvestre, an independent researcher at BXCT Partners Ltd, ran the first systematic audit of SciFact's gold labels and published it at BioNLP 2026 (the 25th Workshop on Biomedical Language Processing, ACL SIGBIOMED, San Diego, 3-4 July 2026), pages 97 to 103, DOI 10.18653/v1/2026.bionlp-1.9. The audit covers the 209 development and 564 training claim-document pairs that carry gold evidence annotations (SciFact also has 300 hidden test claims that nobody outside the leaderboard organizers can check). Sylvestre's review of the dataset's GitHub repository (27 open issues, none about annotation quality), its Hugging Face page, and the citing literature found zero prior corrections or errata in that time.

The evidence table

Two numbers anchor the audit: an 11-error, 5.3% rate in the development set and a 13-error, 2.3% rate in training. Both carry 95% confidence intervals wide enough to reflect how few pairs were checked, and both sit inside the 3-to-7% range that comparable audits have reported for other benchmarks.

The errors are not evenly spread by type. Four of the eleven dev errors are outright label reversals, where the gold tag contradicts what the cited evidence sentence says (one claim about T-cell signaling is tagged SUPPORT, while the assigned rationale states the opposite effect was found). Three are outcome mismatches, where claim and evidence describe different clinical endpoints: one claim about "bleeding events" is graded against a rationale discussing cardiovascular mortality, MI, stroke, and congestive heart failure, which never mentions bleeding at all. Three are entity mismatches across species or organisms, including a claim about "human beta cells" graded SUPPORT against a mouse-only study. One is an overgeneralization: a claim that CHEK2 "is not associated with" breast cancer, graded SUPPORT against an abstract whose first sentence establishes the opposite association and reports an odds ratio of 2.26 for a specific mutation.

Fixing the 11 dev errors is not a bookkeeping exercise. Sylvestre reran zero-shot evaluation on all 209 dev claims before and after correction, using GPT-5.4-mini, GPT-5.4-nano, and (as an independent check outside the GPT family) Claude Haiku 4.5.

A $0.11 screen finds eight errors and misses three entirely

The audit ran in two stages. Stage one used GPT-5.4-mini at temperature 0.3 to screen all 209 dev pairs against a structured prompt (claim text, gold label, gold rationale sentences, and the full abstract), asking for an AGREE, DISAGREE, or DEBATABLE verdict plus a suggested correction. This stage flagged 57 pairs (43 DISAGREE, 14 DEBATABLE) and cost $0.11 in API fees, roughly 440,000 tokens at March 2026 cached-input pricing.

Stage two put a human in the loop. Sylvestre, working alone, adjudicated each of the 57 flagged pairs with GPT-5.4 (the frontier-size model, not mini) running in chat mode as a second opinion: the model surfaced its own reading of the claim and evidence, Sylvestre challenged it, and Sylvestre made the final call. Of the 57 flagged pairs, 8 were confirmed errors, 8 stayed debatable but defensible, and 41 were false alarms, mostly the screening model confusing claim direction with evidence direction (flagging a CONTRADICT label as wrong when the claim genuinely reversed the evidence). The remaining 152 unflagged pairs got an exhaustive manual pass too, without the chat-mode loop, since the screen had implicitly endorsed their gold labels. That pass caught 3 more confirmed errors the automated screen had rated AGREE and missed outright: one direct claim-evidence contradiction, one case requiring a check of a trial's reported significance level (p=0.07, not significant, against a claim of "reduced efficacy"), and one case requiring a check of which species a cleavage result had actually been measured in (mouse, not human, as the claim stated).

The screen's own hit rate is worth separating from the final count. Of the 57 pairs it flagged, only 8 turned out to be real errors: 14% precision. Of the 11 real errors in the full dev set, the screen caught 8: 73% recall. The three it missed needed a kind of check a prompt comparing claim text to rationale text will not reliably do on its own: reading a reported significance level, or tracking which species or molecule a finding was actually measured in.

One more pattern surfaces in the confirmed errors: 9 of 11 mislabel a claim as SUPPORT when the evidence does not support it. The asymmetry is statistically borderline on 11 data points (one-sided binomial p=0.033, two-sided p=0.065), and Sylvestre reports it as suggestive, not confirmed. A separate check argues the F1 gains are not just the models' own training data leaking back into the corrections: asked to classify the same 209 claims from the claim text alone, with no abstract, no rationale, and no title, Claude Haiku 4.5 reached 70.81% accuracy, only 4.78 points above the 66.03% majority-class baseline, with 18% of its guesses landing on NEI. A model that had memorized SciFact's gold labels would be expected to commit confidently to one side rather than hover just above a majority-class guess.

What it means for someone doing this work

For anyone building on SciFact, the audit changes three practical calculations. Training on the 809-claim training set means training against roughly 2.3% mislabeled targets among the 564 pairs with gold evidence annotations, a small but nonzero source of noise in any fine-tuning run. Comparing systems on the 300-claim dev set means reading a leaderboard where correcting 11 pairs moves binary macro-F1 by 1.7 to 3.8 points, the same order of magnitude as the 0.7-to-2.7-point gaps that separate the top models in prior dev-set comparisons, and close to the 1.25-point gap between the top two systems (MultiVerS at 72.5, ARSJoint at 71.2) on the hidden test leaderboard. Submitting to that hidden 300-claim test set means trusting labels nobody, including Sylvestre, has been able to audit.

The other practical lesson sits in the screen's own precision and recall numbers. A pipeline that runs an LLM screen over a benchmark and stops there, without a human (or a second, differently-scaled model) checking the unflagged majority, will under-count errors of exactly the kind this one missed: statistical-significance checks and entity or species mismatches. 73% recall on a $0.11 pass is a reasonable return for the price, but it is not a substitute for the exhaustive manual pass that actually found the last three errors.

Our judgement: the headline counts deserve real weight but not full confidence yet. Sylvestre is a single annotator, assisted by a frontier LLM in chat mode but not checked by a second independent human rater, which matters given that the original annotation itself only reached Cohen's kappa 0.75. The directional claim that errors skew toward mislabeling SUPPORT (9 of 11, two-sided p=0.065) is, on 11 data points, suggestive rather than established, and the paper says so explicitly. What is on firmer ground is the existence and rough size of the problem: the 95% confidence intervals (2.7-9.2% dev, 1.2-3.9% train) land inside the 3-to-7% range other benchmark audits have reported for ImageNet, MNIST, CoNLL-03, and factual-consistency benchmarks, and the F1 shift replicates across three models from two different labs (GPT-5.4-mini, GPT-5.4-nano, and Claude Haiku 4.5), which rules out the correction simply flattering one model family's own preferences.

Check it yourself

Sylvestre released the corrected labels, a blind re-annotation packet, and the scripts behind every number in the paper. None of this requires access to anything beyond a terminal and the public SciFact release.

git clone https://github.com/Kefez/scifact-audit-bionlp2026
cd scifact-audit-bionlp2026

# fetch the original SciFact release the audit checked (not vendored in the repo)
bash data/download_scifact.sh

# the 11 dev-set and 13 training-set corrections, as JSON
python3 -m json.tool < corrections/dev_corrections.json | head -40
python3 -m json.tool < corrections/train_corrections.json | head -40

# recompute the two headline error rates
python3 -c "print(round(11/209*100,1), round(13/564*100,1))"
# -> 5.3 2.3

# the Stage 1 LLM screen's own flags, for checking precision and recall
ls results/stage1_llm_screen/

# a blind packet (no gold label, no LLM signal shown) for an independent re-audit
head -3 annotator_packets/blind_annotator_packet_dev.csv

# the binomial tests and confidence intervals reported in the paper
python3 revision/stats_compute.py

The paper itself is pages 97 to 103 of the BioNLP 2026 proceedings (DOI 10.18653/v1/2026.bionlp-1.9). The per-claim justification for all 11 dev errors (claim IDs 593, 879, 1385, 808, 343, 1368, 770, 1216, 1274, 847, 208) and all 13 training errors is in the PDF linked from that page and in the repository's supplementary materials.

What would prove this wrong

Sylvestre built the blind re-annotation packet specifically so someone else could check the count without seeing his verdicts or the screening LLM's. That packet is the mechanism by which this finding gets tested. If an independent re-audit of the 209-pair blind packet, carried out by one or more raters who have not seen the original paper and reported before 3 July 2027 (one year after the BioNLP presentation), finds a confirmed dev-error count outside the paper's own 95% confidence interval, meaning fewer than 6 errors (under 2.9%) or more than 19 errors (over 9.1%), treat the 5.3% headline rate as an artifact of single-annotator judgment rather than a property of the dataset. If that same re-audit finds the errors split evenly, or skewed toward CONTRADICT or NEI instead of SUPPORT, the directional-asymmetry claim (9 of 11 toward SUPPORT) specifically falls, even if the overall error count holds up. Absent such a re-audit by that date, the finding stands as a stated, wide-interval single-annotator estimate, not as a settled fact about SciFact.

Sources

1. Julien Sylvestre. "Gold Label Errors in the SciFact Benchmark: An LLM-Assisted Annotation Audit." Proceedings of BioNLP 2026 (25th Workshop on Biomedical Language Processing), ACL Anthology, pp. 97-103, San Diego, 3-4 July 2026. DOI: 10.18653/v1/2026.bionlp-1.9. https://aclanthology.org/2026.bionlp-1.9/
2. Julien Sylvestre. Corrected annotations, blind annotator packet, and reproduction scripts. github.com/Kefez/scifact-audit-bionlp2026 (2026).
3. David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, Hannaneh Hajishirzi. "Fact or Fiction: Verifying Scientific Claims." EMNLP 2020. arXiv:2004.14974.
4. Curtis G. Northcutt, Anish Athalye, Jonas Mueller. "Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks." NeurIPS 2021. arXiv:2103.14749.
5. BLOMEGA. Lexicographers agree at kappa 0.537 on hard WordNet senses; relabeling SemCor lifts three WSD systems. Related internal coverage of relabeling a benchmark and measuring the downstream score shift.
6. BLOMEGA. NVD CWE label audit. Related internal coverage of a dataset label audit.
7. BLOMEGA. HH-RLHF preference label noise. Related internal coverage of label noise in a widely-cited benchmark.