Only 49.7% of NVD CWE labels match the code they describe

What changed, and when

On 22 August 2026 Yu Nong and Haipeng Cai (University at Buffalo), Yao Du (Macau University of Science and Technology) and Majid Behravan (Virginia Tech) posted How Reliable Are NVD CWE Labels? A Large-Scale Semantic Audit with Seclometry (arXiv:2608.21977, v1). It is the first measurement at this scale of a label set that vulnerability detectors, scanner evaluations and security benchmarks routinely treat as ground truth.

The instrument is an agent pipeline the authors call CweAgent. It builds an eight-field "seclometry" record for each CWE definition and each CVE (root cause, trigger, violated invariant, observable effect, code pattern, source and sink roles, non-examples), gathers evidence with web search, archived advisories, the fix commit and CodeQL, and matches the two. Phase 1 uses Claude 4.6 Opus, phase 2 GPT-5.4-mini, and a Claude Sonnet 4.6 arbitrator sorts each disagreement into an outcome class. On a manually curated benchmark of 100 CVEs, CweAgent reaches 85% top-1 exact-match accuracy and 92% ambiguity-aware accuracy; the arbitrator reaches 90% overall accuracy on a separate 100-CVE test set.

The context matters for anyone building on these labels. On 15 April 2026 NIST announced that CVE submissions had grown 263% between 2020 and 2025 and that it would prioritise enrichment for CVEs in CISA's Known Exploited Vulnerabilities catalog, CVEs in software used by the federal government, and CVEs in critical software as defined by Executive Order 14028. Everything else becomes "Lowest Priority - not scheduled for immediate enrichment", which means no NIST severity score, product mapping or weakness classification. Unenriched CVEs with an NVD publish date before 1 March 2026 moved to "Not Scheduled". NIST's announcement gives no count for that backlog.

So for most CVEs published after April, the only CWE a dataset builder will get is the one the CVE Numbering Authority (CNA) assigned. The audit measures exactly how much that source varies.

The evidence table

Counts and shares are transcribed from Tables IV, V, VII and X of arXiv:2608.21977v1. "Ambiguity-aware" is S1+S2+S3. "Classifier error" is the share where CweAgent was wrong and NVD was right, which is the instrument's own error rate on this corpus.

Two things in these rows are easy to miss. First, the instrument is wrong more often than NVD is inconsistent: 2,380 classifier errors against 564 S4 records, a ratio of 4.2 to 1. The headline 49.70% is a floor on NVD quality as seen through an 85%-accurate lens, and the authors separate the two cases in the table rather than hiding them. Second, the CNA spread among the 15 largest assigners runs from 39.20% exact match (redhat) to 83.54% (wordfence), a 44.3 point range. The weakness type moves the rate as much as the assigner: CWE-20, CWE-200 and CWE-284 score 0.00% exact match, which reads as the audit never accepting those broad categories as the most specific label (our reading of the table, not a sentence in the paper).

A heap over-read becomes an out-of-bounds write in three hops

The 434 confirmed mislabels fall into six patterns in the authors' open coding (Table XII): consequence named instead of root cause 23.1%, wrong sibling within a family 21.3%, wrong injection sink or interpreter 19.5%, authentication and authorisation conflated 15.3%, discouraged or wrong-branch label 10.3%, and several weaknesses collapsed into one CWE 7.8%, with a 3.9% residual on file and path handling.

The sibling pattern has a structural cause the paper names. NVD normalises labels to MITRE's CWE View-1003, which does not contain CWE-122, so a CNA's heap-overflow label gets mapped up to the parent CWE-787, out-of-bounds write. If the actual bug is a read, the normalisation step moves the label further from the truth. CVE-2022-1160 in vim is the paper's worked example, and the NVD API still shows both labels today.

What it means for anyone training or scoring on CWE labels

An exact-match score against NVD CWE has a ceiling well below 100% for a correct model. A classifier that is right 85% of the time on a curated benchmark agreed exactly with NVD on 49.70% of this corpus. If your vulnerability-type benchmark reports top-1 accuracy against NVD labels, part of the gap between models is label choice among defensible siblings, not model skill. The 25.01% overlap-ambiguity bucket is the size of that problem.

Score ambiguity-aware, and drop the umbrella categories as targets. CWE-20, CWE-200 and CWE-284 together account for 1,109 CVEs in Table VII with 0.00% exact match each. Training a model to emit them teaches it to be unspecific. Our judgement: map them to "unspecified" in training data and exclude them from exact-match evaluation.

Weight labels by who assigned them. After NIST's April change, the assigning CNA's label is the label for most new CVEs. A 44.3 point exact-match range across the 15 largest CNAs is a larger quality signal than the year. For memory-safety data specifically, re-derive the read or write distinction from the patch; CWE-122 at 11.94% inconsistency is the worst of the top 20.

An LLM auditor needs a human pass on the tail. The pipeline flagged 564 records; manual review kept 434 and set aside 130 (23%). That is the same human-in-the-loop shape we see in text annotation: the model narrows the search, a person decides. For procurement, the fields to ask a security-data vendor for are the assigning CNA, the CWE as assigned before normalisation, and whether a human checked the root cause against the fix. This is the provenance argument we make for chain of title, applied to labels.

Check it yourself

The worked example is live in NVD. The API returns both the CNA's label and NVD's.

curl -s "https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2022-1160" \
 | python3 -c 'import json,sys; c=json.load(sys.stdin)["vulnerabilities"][0]["cve"]
for w in c["weaknesses"]: print(w["source"], [d["value"] for d in w["description"]])'
# [email protected] ['CWE-122']
# [email protected] ['CWE-787']

The outcome table sums exactly, which is a quick transcription check:

python3 -c "print(7732+3890+990+564+2380, round(100*434/15556,2), round(2380/564,2))"
# 15556 2.79 4.22

The per-year contribution of 2024 to 2026 comes from Table V: 2,074 x 2.85% + 2,179 x 4.54% + 2,265 x 3.09% rounds to 228 CVEs. The authors' code, data and confirmed-mislabel set are at figshare.com/s/6ba6621a69a2dac5b516, which responded to a request on 15 September 2026; we have not re-run the pipeline. The CWE-122 and View-1003 relationship can be checked at cwe.mitre.org/data/definitions/122.html and cwe.mitre.org/data/definitions/1003.html.

What would prove this wrong

The central number depends on an 85%-accurate instrument, so the obvious attack is the instrument. If an independent team hand-labels a random sample of at least 300 CVEs from the same corpus and finds exact agreement with NVD above 60%, the 49.70% figure reflects CweAgent's taxonomy preferences more than NVD's errors.

A dated prediction: by 31 March 2027, fewer than half of the 434 confirmed mislabels in the figshare set will show a changed NVD CWE in the CVE API. The authors report NVD acknowledged the list; acknowledgment is not correction, and NIST's April policy puts most of these older records outside its enrichment priorities. If more than half have been corrected by that date, this prediction is wrong and NVD's correction loop is healthier than its April notice suggests.

Sources

1. Nong, Y., Du, Y., Behravan, M., Cai, H. How Reliable Are NVD CWE Labels? A Large-Scale Semantic Audit with Seclometry. arXiv:2608.21977v1, 22 August 2026. Tables IV, V, VII, X, XII; HTML version.
2. NIST. NIST Updates NVD Operations to Address Record CVE Growth. 15 April 2026.
3. NVD. CVE-2022-1160 detail and CVE API 2.0. Retrieved 15 September 2026.
4. MITRE. CWE View-1003: Weaknesses for Simplified Mapping of Published Vulnerabilities.
5. Authors' artifact: figshare.com/s/6ba6621a69a2dac5b516.
6. Gula Tech Adventures. NVD Gives Up: How NIST Made 29,000 CVEs Disappear Overnight. YouTube, 22 April 2026.
7. BLOMEGA. Data annotation research: the latest and Unpaid annotation tasks in ACL papers.