RSNA-ICA adjudicated 2,952 of 7,202 aneurysm series and reports no agreement statistic; the Kaggle labels do not match its own table

What RSNA published on 1 October 2026

Maria Correia de Verdier (Uppsala), Rachit Saluja (Cornell) and 38 further authors, with Evan Calabrese (Duke) and Jeffrey D. Rudie (UC San Diego, Scripps) as co-senior authors, posted The RSNA Intracranial Aneurysm (RSNA-ICA) Dataset as arXiv:2610.01135v1 (cs.CV, 48 pages with supplement). The work is by the Radiological Society of North America with ASNR, SNIS and ESNR. It documents the data behind the 2025 RSNA Intracranial Aneurysm Detection Challenge on Kaggle, which RSNA's press release says drew 1,147 teams and paid $50,000 to nine of them.

The full set is 7,202 series from 4,278 adult patients at 21 institutions: 2,566 CTA, 2,166 MRA and 2,470 routine MRI (707 post-contrast T1, 1,763 T2). The challenge used 6,509 series from 3,848 patients at 18 sites, split 4,348 / 689 / 1,472 into training, public test and private test. After the challenge, 695 series from 430 patients at three new sites (19 to 21) were added as an out-of-distribution cohort. MIRA (mira.rsna.org/dataset/7) serves 5,041 series: the training set minus two removed for artifacts (4,346) plus the 695. The 2,161 test series stay hidden.

Labels are 13 binary location flags per series plus an "Aneurysm Present" flag, a point localizer per aneurysm (an x, y position on one DICOM image), a size category (under 3 mm or 3 mm and over) in the MIRA release only, and 3D vessel segmentations for 178 series from 105 patients. The paper's main new information for annotation teams is in the Discussion, Appendix S1 and the attached annotation instructions dated 3 February 2025.

Does the paper's arithmetic hold, and do the released labels match it?

The paper is internally consistent. Every Table 2 site row sums to the Table 1 totals on all 24 columns (sex, four modalities, series, negatives, positives, aneurysms, size, 13 locations), sites 19 to 21 sum exactly to the post-challenge row, modality columns add to series totals, negatives plus positives add to series, and the 13 location columns add to the aneurysm count in every row. Sites 1 to 18 hold 3,848 sexed patients, matching the challenge figure; 4,263 sexed patients plus the 15 unsexed in sites 19 and 21 give 4,278.

Paper numbers from arXiv:2610.01135v1 (PDF SHA-256 4bc5559c...b24be1). BLOMEGA rows computed 4 October 2026 from train.csv (SHA-256 76560ca3...e083f0, 4,349 lines) and train_localizers.csv (SHA-256 968ed62a...5610da, 2,255 lines), copies of the Kaggle competition files found in two public GitHub repositories dated 4 October 2025 and 3 November 2025. The two files are mutually consistent: all 1,863 positive series have a localizer and no localizer points at a negative or unknown series.

The Kaggle versus paper gap is small and specific. Removing two series cannot raise the positive count. If both removed series were negative, the Kaggle file would give 1,863 positives and 2,483 negatives; Table 1 has 1,864 and 2,482. So at least one training series flipped from negative to positive between the Kaggle release and MIRA, or Table 1 is off by one. The modality columns locate the two removed series as one MRA and one T2.

Nine of 13 locations differ. The largest gap is the right middle cerebral artery, 307 in the paper against 311 Kaggle localizers. Opposite-sign gaps on matching left and right pairs (right supraclinoid ICA minus 1 and left supraclinoid plus 1; left PComm minus 1 and right PComm plus 1) look like side corrections. That reading is our judgement: the paper does not say whether labels were corrected between releases, and without MIRA access we cannot match series.

The Kaggle files also disagree with themselves in two series. In one CTA series (UID ending 717242994172) train.csv flags the left supraclinoid ICA while the only localizer says right supraclinoid ICA. In another CTA series (ending 020706657426) train.csv flags the basilar tip while the localizer says left posterior communicating artery. Four localizer rows are exact duplicates (same series, image and coordinates), which explains part of the 2,254 versus 2,252 gap. 146 of 2,254 localizers carry an extra f key, a frame index for multi-frame DICOM, which any parser that expects only x and y must handle.

How one RSNA-ICA label is made, and where "negative" gets wider

The design is the RSNA challenge pattern used for the abdominal trauma and lumbar spine datasets: the site's report label is the first opinion, one volunteer radiologist is the second, and an organizing-committee adjudicator settles conflicts with both visible. The Discussion's phrase "a third adjudication review" confirms the count: two opinions, then a third. The paper calls the pool crowdsourced, with a public scoreboard to drive participation. A standard deviation of 91.6 on a mean of 86.8 cases means the load was heavily skewed towards a few prolific readers, though the paper gives no distribution.

Adjudication was not only for disagreement. The instructions told readers to pick "possible small aneurysm (<3mm) versus infundibulum" when unsure, and those lesions went to neurointerventional reviewers unless the reader's call matched the site label. So 2,952 is an upper bound on disagreement, not a disagreement rate, and the paper does not split it. The adjudication rule was asymmetric: keep an aneurysm only when "sufficiently confident", otherwise call it negative. The authors state this may have produced false negatives for very small lesions.

Two further rules widen "negative". For routine MRI, the reader looked at the best CTA or MRA first, and the MRI got a positive label only if the aneurysm was clearly visible on it. The instructions had a fifth sublabel, "Aneurysm seen on other series but not this one", and the paper says those MRI series were "included among the negative cases". And the annotation tool offered 15 locations (left and right posterior cerebral artery separately), which the released schema merges into "other posterior circulation" to 13.

Why the test sets and the new sites are harder or easier than the training set

Table 1 gives aneurysm counts and the number at 3 mm or more per split, so the under-3 mm share is recoverable. The organizers state they oversampled small and multiple aneurysms, young and male patients and basilar tip lesions into the test sets. The size effect of that sampling is large.

The private test set that ranked the 1,147 teams has 41.4% small aneurysms against 27.8% in training. The out-of-distribution cohort has 10.4%: its three sites run 8.1%, 14.8% and 11.3%, while the 18 challenge sites range up to 44.8%. A model evaluated on the new sites therefore faces a site shift and an easier size mix at the same time, and a score on that cohort cannot be read as pure generalisation. Positives per split also move: 42.9% of training series, 52.5% of private test series.

Site enrichment was uneven. Sites were asked for about half positive cases; Table 2 shows positive series shares from 30.1% (site 20) to 95.5% (site 7, 276 of 289 series). The paper's "average of 1.8 aneurysms/patient" is 4,099 divided by 2,261 patients, but 4,099 counts aneurysm labels per series, so one aneurysm visible on CTA, MRA and T2 counts three times. Per positive series the figure is 1.26 (4,099 of 3,258), and the Kaggle localizers give 1.21. Our judgement: 1.8 per patient overstates multiplicity.

What to change if you train on RSNA-ICA or copy its protocol

Treat MRI negatives as "not visible on this series". In the Kaggle training file, 53.8% of CTA series (973 of 1,808) and 44.3% of MRA (555 of 1,252) are positive, against 26.2% of T2 (258 of 983) and 25.2% of post-contrast T1 (77 of 305). Part of that gap is by design: aneurysm patients whose lesion was not clearly visible on MRI sit among MRI negatives. Group by patient (MIRA's PatientGroupID) before using MRI negatives as clean controls. The Kaggle release blinds patient linkage at series level, so you cannot do this there.

Use the two releases as two label versions. One net positive flip, 14 units of location movement and two internally inconsistent series in 4,348 is a low error rate. It still matters if you report per-location sensitivity on the left and right ACA, where the training counts are 44 to 63. Fix a version, record the file hash, and say which.

Budget for a second read where it counts. A report label plus one blind read plus adjudication is cheap and catches gross errors. It cannot measure reader reliability, and the paper reports none. If you run a similar protocol, a double read on a random 5 to 10% gives a kappa for the cost of a few hundred extra reads. Record adjudication by trigger (disagreement versus "unsure" flag) so 2,952 can be split.

Check what drafted the site labels. Two of 21 sites used local LLMs on reports. UC Irvine's model was asked only for aneurysms of 3 mm or more, and a neuroradiologist verified model positives but the text does not say negatives were checked. The independent read is what catches the misses, which makes the one-reader design more load-bearing at those sites.

Read the terms before you mirror anything. RSNA's terms allow non-commercial research and education only and forbid sharing or redistributing "any portion" of the data. At least two public GitHub repositories hold copies of the Kaggle label CSVs. We used them for this count and do not link them; reproduce from Kaggle or MIRA under your own agreement. Commercial model training on RSNA-ICA is outside the licence.

Check it yourself

The table check needs only the PDF. The label check needs the two Kaggle CSVs (about 0.96 MB together, no images), downloaded after accepting the competition rules.

# 1. Paper tables (pdftotext missing here, pypdf layout mode works)
curl -sL https://arxiv.org/pdf/2610.01135 -o p.pdf      # sha256 4bc5559c41ae...0b9b0b24be1
python3 -c "import pypdf;r=pypdf.PdfReader('p.pdf');open('p.txt','w').write('\n'.join(p.extract_text(extraction_mode='layout') for p in r.pages))"
grep -n '^Site [0-9]' p.txt            # 21 rows of Table 2, 24 numeric columns after the age field
# sum each column over sites, compare with Table 1 'Total'; check CTA+MRA+T1post+T2 == Total series,
# Negative+Positive == Total, sum(locations 1..13) == Total aneurysms. All match.

# 2. Kaggle labels (kaggle competitions download -c rsna-intracranial-aneurysm-detection -f train.csv)
#    train.csv sha256 76560ca31dd3...ccd660e083f0 ; train_localizers.csv sha256 968ed62aa0dd...b6216858b55fe655610da
python3 - <<'PY'
import csv,collections
R=list(csv.DictReader(open('train.csv'))); loc=list(R[0])[4:17]
P=[r for r in R if r['Aneurysm Present']=='1']
print(len(R),len(P),collections.Counter(r['Modality'] for r in R),collections.Counter(r['Modality'] for r in P))
L=list(csv.DictReader(open('train_localizers.csv')))
print(len(L),collections.Counter(x['location'] for x in L))
by=collections.defaultdict(set)
for x in L: by[x['SeriesInstanceUID']].add(x['location'])
print([(r['SeriesInstanceUID'][-12:],l) for r in P for l in loc if (r[l]=='1')!=(l in by[r['SeriesInstanceUID']])])
PY
# 4348 1863 {CTA:1808, MRA:1252, MRI T2:983, MRI T1post:305} {CTA:973, MRA:555, MRI T2:258, MRI T1post:77}
# 2254 ... Right Middle Cerebral Artery 311 ... ; 4 mismatching (series, location) pairs in 2 series

# 3. Release metadata
curl -sL https://raw.githubusercontent.com/awslabs/open-data-registry/main/datasets/rsna-intracranial-aneurysm-detection-dataset.yaml
curl -s https://doi.org/api/handles/10.1148/dataset.ica.2025      # {"responseCode":100,...} = handle not found, 4 Oct 2026

Fields used: SeriesInstanceUID, Modality, the 13 location columns and Aneurysm Present in train.csv; SeriesInstanceUID, SOPInstanceUID, coordinates (a Python dict string with x, y and sometimes f) and location in train_localizers.csv. MIRA's dataset.csv and the 178 NIfTI segmentations need a MIRA account and were not checked. Kaggle files may have been revised after the mirror snapshots of 4 October 2025 and 3 November 2025; the hashes let you test that.

The release metadata outside the paper is less careful than the paper. The AWS Open Data registry entry (last edited 24 April 2026) describes "over 4,000 CT brain scans annotated by a cohort of over 40 volunteer radiologists" from 18 institutions, links its publication to DOI 10.1148/ryai.2021200254, which resolves to the RSNA Pulmonary Embolism CT Dataset (2021), and asks users to cite DOI 10.1148/dataset.ica.2025, which did not resolve on 4 October 2026. RSNA's challenge page says 60 radiologists. The paper says 65, and its annotator group lists 65 names in which "Anthony Kam" appears twice with different affiliations, and five sit in a company and a quantitative biomedicine department (plausibly the segmentation contributors, our inference). Appendix S1 lists 21 institutions in 13 jurisdictions; the paper's "12 countries" holds if Taiwan is not counted separately.

What would prove this wrong

Our claim is that the Kaggle and MIRA training labels are different label versions, not the same labels minus two series. Prediction: when anyone with MIRA access joins dataset.csv to Kaggle train.csv on SeriesInstanceUID, at least one shared series will have a different Aneurysm Present value, and at least five shared series will have different location flags. If by 31 March 2027 such a join shows identical labels on all 4,346 shared series, Table 1 is simply wrong by one and the per-location gaps are a table error, not relabelling.

Second prediction: if RSNA or the authors publish a double-read sample or split the 2,952 adjudications by trigger, disagreement-triggered adjudications will be under half of 2,952, because "unsure" flags on sub-3 mm lesions also routed series to adjudication and 30.7% of all aneurysms are under 3 mm.

Sources

1. Correia de Verdier, M., Saluja, R., et al., Calabrese, E., Rudie, J. D. The RSNA Intracranial Aneurysm (RSNA-ICA) Dataset. arXiv:2610.01135v1, 1 October 2026. Tables 1 and 2, Discussion, Appendix S1, "RSNA-ICA Annotation Instructions" (3 February 2025) and "Segmentation Instructions".
2. Kaggle. RSNA Intracranial Aneurysm Detection competition data, train.csv and train_localizers.csv. Checked 4 October 2026 via public copies (hashes above).
3. RSNA. MIRA dataset 7 (login required; page returned HTTP 200 on 4 October 2026).
4. AWS Open Data. RSNA Intracranial Aneurysm Detection Dataset (RSNA-ICA), registry YAML in awslabs/open-data-registry, created 14 November 2025, last edited 24 April 2026. Licence text and bucket intracranial-aneurysm-detection (us-west-2, controlled access; anonymous listing returns 403).
5. RSNA. RSNA Intracranial Aneurysm Detection AI Challenge (2025) and 2025 AI Challenge winners. Accessed 4 October 2026.
6. Colak, E., et al. The RSNA Pulmonary Embolism CT Dataset. Radiology: Artificial Intelligence, 2021 (the DOI the AWS entry links as the aneurysm publication).
7. BLOMEGA. VoiceNet-Emo's three experts say "present" 11.6% to 67.4% of the time on the same prompts.

FAQ

How were the RSNA-ICA aneurysm labels made?

Each contributing site supplied a label from the clinical radiology report (two sites, Duke and UC Irvine, used a local LLM to extract it). One radiologist from a pool of 65 then read the series independently in MD.ai without seeing the site label and placed a point localizer on each aneurysm. Disagreements and lesions marked possible small aneurysm versus infundibulum went to an adjudicator from the organizing committee; 2,952 series were adjudicated. Indeterminate lesions became negatives. The paper, arXiv:2610.01135 (1 October 2026), reports no inter-rater agreement statistic.

Do the RSNA-ICA Kaggle labels match the paper?

Almost, not exactly. The Kaggle train.csv has 4,348 series with 1,863 positive. The paper's Table 1 lists 4,346 training series (after removing two for artifacts) with 1,864 positive, so at least one series changed from negative to positive between the Kaggle and MIRA releases, or the table is off by one. The Kaggle localizer file has 2,254 aneurysm points against 2,252 in Table 1, and the per-location counts differ by 14 units in total. BLOMEGA computed this on 4 October 2026 from copies whose SHA-256 is given in the article.

What does a negative label mean in RSNA-ICA?

It depends on the series. On CTA and MRA a negative means no aneurysm was retained after adjudication. On routine MRI (T2 or post-contrast T1) a negative also covers patients whose aneurysm was seen on CTA or MRA but was not clearly visible on that MRI series; the annotation tool had a separate label for this case and it was folded into negative. Lesions that stayed indeterminate after adjudication are also negatives. In the Kaggle training file 26.2% of T2 series and 25.2% of post-contrast T1 series are positive, against 53.8% of CTA series.