Hugging Face serves zero RAI fields for 533 CroissantMiner datasets, and the scorer counts the string "null" as a hallucination

What the ETH Zurich team released on 5 October 2026

Berke Arda, Ahmetcan Yavuz and eight co-authors (ETH Zurich and the ETH AI Center, MIT CSAIL, Helmholtz Zentrum München, UMBC, Barcelona Supercomputing Center, Google and Duke Kunshan University) posted CroissantMiner: Automated Extraction and Validation of Croissant Metadata for ML Datasets as arXiv:2610.07132v1 (cs.CL). The repository describes it as a NeurIPS 2026 Evaluations and Datasets track paper. NeurIPS 2026 requires Croissant RAI files for dataset submissions, which is the practical pressure behind the work.

The benchmark pairs 602 dataset papers with the 30-field Croissant 1.1 schema: 10 core fields (sc:name, sc:license, cr:citeAs and so on) and 20 RAI fields such as rai:dataCollection, rai:annotatorDemographics and rai:dataAnnotationPlatform. 102 papers carry human-validated gold, 500 carry silver labels. Both started from one Claude Sonnet 4.5 single-pass extraction in mid-January 2026; 22 unpaid annotators (eight of them co-authors) then rated the gold pre-fills, at least three ratings per cell.

The paper evaluates 24 systems on an 88-paper test split. Single-pass Claude Sonnet 4.6 leads at 0.709 composite. All 13 agentic configurations (ReAct, Parallel Specialists, Triage + Critique, Locator-Extractor, on three backbones) score below the single-pass run on the same backbone. Long-form RAI fields are scored by GLM-5 as judge, chosen from six candidates at quadratic-weighted kappa 0.890 on 60 cells, a figure that drops to 0.708 with the final rubric and audited gold.

The release is real and complete. berkearda/croissantminer is MIT, created 28 September 2026, 173 commits by 7 October. The data sits at bearda/croissantminer (CC BY 4.0, v1.2 of 1 October 2026) with 13 configs, from gold and raw ratings to every system output and judge verdict. The NeurIPS review copy is frozen at croissantminer/croissantminer (v1.0, 7 May 2026). The demo Space reported stage RUNNING when we checked.

Do the released files match the paper's counts?

Almost all of them, and the scoring script reproduces Table 2 to the third decimal. The rows marked BLOMEGA are our own checks.

Paper numbers from arXiv:2610.07132v1. Release files from bearda/croissantminer v1.2 and github.com/berkearda/croissantminer at commit 12f7791 (7 October 2026). Our reading of the adjudication gap: 154 minus 132 equals 22 cells that the 191-cell audit relabelled; 3,060 minus 154 is 2,906, the paper's 95.0% majority share.

How a single pre-fill becomes both the gold and the silver

Annotators did not extract from scratch. They saw Sonnet 4.5's value for each cell and rated it Correct, Partially Correct or Not Correct, optionally typing a correction. We compared each final gold value with the pre-fill shown in ratings.extracted_value, after collapsing whitespace and case. 2,684 cells are identical. Of the 376 that differ, 89 are values annotators added where the pre-fill was empty, 37 are pre-fill values they deleted as unsupported, and 250 are rewrites. The paper's 376 rewritten cells (12.3%) match our count exactly.

Change is concentrated where the paper says agreement is low. sc:publisher changed in 37.3% of papers, rai:dataCollectionType in 36.3%, rai:annotationsPerItem in 28.4% and rai:dataPreprocessingProtocol in 24.5%. rai:annotatorDemographics changed in only 2.9%, and cr:isLiveDataset in none. Overall 90.0% of core cells and 86.6% of RAI cells kept the pre-fill.

Then scoring. For each test cell, is_null_gold() treats the gold as empty if it is missing, blank, or one of a dozen strings: null, none, n/a, unknown, not found, not specified, [NULL ...]. The system output gets a narrower test: cand_empty = cand is None or not str(cand).strip(). A model that answers "null" as a JSON string on a field the paper never documents is therefore scored 0 as a hallucination, while the identical token in the gold is read as absence.

Does Hugging Face already publish RAI metadata for these datasets? No, for 533 of 533

The paper states that platform-generated Croissant covers only structural fields. That is the case for the benchmark's own datasets. We requested https://huggingface.co/api/datasets/<id>/croissant for the 500 silver datasets and for the 102 gold datasets, mapping gold paper IDs such as AI4Math_MathVista to AI4Math/MathVista. 444 silver and 89 gold requests returned a file; 53 returned 401 (gated datasets such as ILSVRC/imagenet-1k) and 16 returned 400 (no viewer-supported format, such as allenai/objaverse).

All 533 files carry name, description, url, creator and keywords, and 433 (81.2%) carry a licence. 327 declare Croissant 1.1, 131 declare 1.0, 75 declare nothing. Not one contains a rai: key, and none carries citeAs or datePublished, which papers state in 99.0% and 100% of gold cases. The licence result inverts the paper picture: only 24.5% of gold papers state a licence, while Hugging Face has one for 81.2% of datasets, so for licence the hub is the better source.

The sharpest case is the benchmark's own dataset. The croissant.json the authors uploaded to bearda/croissantminer holds 20 rai: keys. The file the API serves for the same dataset holds none. An author who writes RAI metadata by hand and pushes it to the hub does not see it in the hub's machine-readable endpoint. The authors' own file also covers a different 20: it omits rai:dataCollectionMissingData, rai:dataImputationProtocol, rai:dataManipulationProtocol and rai:dataPreprocessingProtocol, and adds rai:hasSyntheticData, rai:isSyntheticData, rai:provenanceActivities and rai:sourceDatasets.

How much does the "null" string rule move Table 2?

We ran the authors' Table 2 code twice: as released, then with one line changed so that a system output equal to null or none counts as empty, the same rule the gold side already uses. Nothing else changed, including judge verdicts.

Mistral Small 4 writes the literal string in 252 cells across its extraction files, 181 of them in test RAI cells, and 140 of those sit on fields the gold marks as undocumented. Its RAI score rises from 0.482 to 0.536 and its composite from 0.527 to 0.566, which lifts it above Triage + Critique (GPT-5.4) at 0.557 and Parallel Specialists (Gemini 3.1 Pro) at 0.539. Llama 4 Scout gains 0.016, GPT-5.4 Mini 0.008, Gemini 2.5 Flash 0.006. Sonnet 4.6, Opus 4.7, GPT-5.4 and all 13 agentic rows emit no such string and do not move, so the headline order and the single-pass versus agentic finding survive intact.

Our judgement: writing "null" in place of JSON null is a format slip, and it is fair to report it. Scoring it as invented content mislabels it. The paper's own failure taxonomy separates "format error" from "hallucination", and the scorer should too.

Is a silver null evidence that the paper is silent?

Often not. Appendix F.2 reports that silver fill rates track gold (Pearson 0.839 across 30 fields) and explains the cr:isLiveDataset outlier (gold 100%, silver 7.8%) by annotators being told to decide for every paper. The ratings file says otherwise: the pre-fill shown to annotators already had a value (Yes or No) in all 306 isLiveDataset ratings, before anyone touched it. Across all fields, Sonnet 4.5's pre-fill on gold papers is non-empty in 71.0% of cells and the reviewed gold in 72.7%, so review added 1.7 points. The silver split sits at 62.3%.

The gap shows up on fields every paper has. Silver leaves cr:citeAs empty for 195 of 500 papers (39.0%) and sc:datePublished empty for 54 (10.8%), although every silver paper was selected through its arXiv identifier, which encodes the month and year. On gold papers the same model filled citeAs 95.1% of the time and the date 100%. Some of the difference may come from paper selection, since gold papers were curated and document more fields (21.7 against 18.7 on average). For an empty citation or date, selection cannot be the full story; our reading is that the silver run behaved differently from the gold pre-fill.

The re-judge in the repo's leaderboard is a second quiet shift. leaderboard.csv rescored all 24 paper systems on 1 October 2026 with GLM-5 served by Z.AI through OpenRouter, where the paper used GLM-5 on DeepInfra. Every composite fell, by 0.006 to 0.025 (mean 0.015); Sonnet 4.6 went from 0.709 to 0.692. Rank order held (Spearman 0.999, one adjacent swap, GLM-5.1 and Triage + Critique on Sonnet 4.6) and all 13 agentic rows stay below their single-pass match, but ReAct on Gemini 3.1 Pro now trails by 0.002, against 0.008 in the paper, inside overlapping intervals.

What this means if you document, buy or annotate datasets

Do not expect the hub to carry your RAI answers. If a buyer, a NeurIPS reviewer or an EU AI Act Article 11 file needs annotatorDemographics, dataAnnotationPlatform or consent details, ship them in the dataset card and in a Croissant file you host and link, and verify what /api/datasets/<id>/croissant returns. On our sample it returned none of the 20 fields.

Treat extracted RAI drafts as drafts. The best RAI score in Table 2 is 0.711 (Opus 4.7), and the overall leader manages 0.687. The paper's deployment advice (review the low-agreement fields) is right; we would add rai:dataCollectionType, which annotators changed in 36.3% of pre-fills, to the must-review list alongside sc:publisher and rai:dataAnnotationAnalysis.

Use silver labels for trends, not for absence. A silver empty cell says the extractor did not answer. It does not say the paper is silent. If you count "how many datasets disclose annotator demographics" from silver, you will count extractor abstention as missing disclosure.

If you build on the leaderboard, normalise null tokens on both sides and pin the judge provider as well as the judge model. A provider switch moved every score by about 0.015, the size of many gaps between adjacent systems.

For annotation programmes, the pre-fill paradigm is cheap but sticky. 87.7% of cells survived review unchanged. That is either a well-calibrated pre-fill or anchoring, and the paper's 10-paper re-seed (GPT-5.4 jumps from 0.673 to 0.930 against gold it seeded) shows the anchor is real. When the gold will rank models, seed a random slice with a second model or from scratch, and report agreement on that slice.

Check it yourself

Everything below runs in a fresh venv in under ten minutes; the Hugging Face crawl is the slow part (about 8 minutes at 4 requests a second). Total download is about 13 MB of parquet plus the 26 MB repo.

# 1. Table 2 exactly as printed (repo commit 12f7791, 7 Oct 2026)
git clone https://github.com/berkearda/croissantminer repo
python3 -m venv venv && venv/bin/pip install pandas pyarrow scipy
cd repo && ../venv/bin/python scripts/figures/print_table2.py   # Sonnet 4.6 0.752 0.687 0.709 [0.688, 0.729]

# 2. The one-line null-string change: in scripts/figures/build_test88_headline_table.py,
#    per_cell_scores_test88(), replace
#      cand_empty = cand is None or not str(cand).strip()
#    with
#      cand_empty = cand is None or not str(cand).strip() or str(cand).strip().lower() in ('null','none')
#    and rerun step 1.  Mistral Small 4: 0.616 0.482 0.527  ->  0.627 0.536 0.566

# 3. HF-served Croissant for one dataset vs the authors' uploaded file
curl -s https://huggingface.co/api/datasets/bearda/croissantminer/croissant | python3 -c \
 "import json,sys;print([k for k in json.load(sys.stdin) if k.startswith('rai:')])"          # []
curl -sL https://huggingface.co/datasets/bearda/croissantminer/resolve/main/croissant.json | python3 -c \
 "import json,sys;print(len([k for k in json.load(sys.stdin) if k.startswith('rai:')]))"     # 20

# 4. All benchmark datasets: loop over silver.parquet paper_id (500) and gold paper_id
#    (first "_" -> "/"), GET https://huggingface.co/api/datasets/<id>/croissant,
#    record status and keys starting with "rai:".  Ours: 533 x 200, 53 x 401, 16 x 400, rai keys 0.

# 5. Pre-fill vs gold (files from huggingface.co/datasets/bearda/croissantminer, v1.2)
#    gold.parquet: paper_id, field_id, gold_value, gold_method
#    ratings.parquet: paper_id, field_id, annotator, extracted_value (the pre-fill), rating, corrected_value
#    Take the modal extracted_value per cell, treat [NULL...]/null/none/blank as empty,
#    compare after lower-casing and collapsing whitespace: 2,684 of 3,060 identical.

For the leaderboard comparison, repo/leaderboard/leaderboard.csv has both composite (re-judged, column judged_on = 2026-10-01) and paper_composite for every row with source = paper. Our replication of the RAI column from the Hugging Face judge_verdicts and system_outputs configs alone lands within 0.003 of the paper on every system; the repo's local extraction JSONs give the exact values.

What would prove this wrong

The claim that the hub drops RAI fields is falsified by one fetch. If, on or before 30 June 2027, huggingface.co/api/datasets/bearda/croissantminer/croissant returns any rai: key, or a rerun of step 4 finds more than 5% of the 533 files carrying one, Hugging Face has started merging RAI metadata and the extraction bottleneck the paper targets is shrinking from the platform side. Our prediction is that neither happens by that date.

The scoring claim is falsified if the authors show that literal "null" strings were intended as hallucination penalties in the evaluation protocol, or if a future release changes the gold-side null list instead of the candidate side. The silver claim is falsified if a rerun of the released silver prompt on the 102 gold papers leaves cr:citeAs empty for more than 30% of them, which would mean the gap is paper selection after all.

Sources

1. Arda, B., Yavuz, A., et al. CroissantMiner: Automated Extraction and Validation of Croissant Metadata for ML Datasets. arXiv:2610.07132v1, 5 October 2026. Sections 3, 5, App. A.2, A.4, D.2, F.1, F.2, Tables 2, 3, 4, 7, 8.
2. berkearda. croissantminer (MIT), created 28 September 2026, commit 12f7791 of 7 October 2026. Files: scripts/figures/print_table2.py, scripts/figures/build_test88_headline_table.py, leaderboard/leaderboard.csv. Accessed 7 October 2026.
3. bearda. bearda/croissantminer dataset (CC BY 4.0), v1.2 of 1 October 2026; v1.0 at croissantminer/croissantminer. Accessed 7 October 2026.
4. bearda. CroissantMiner demo Space, runtime stage RUNNING on 7 October 2026.
5. Hugging Face. Croissant in the dataset viewer documentation, and the /api/datasets/<id>/croissant endpoint, queried 7 October 2026 for 602 dataset IDs.
6. MLCommons. Croissant 1.1 specification.
7. NeurIPS. NeurIPS 2026 Evaluations and Datasets FAQ, as cited in the paper.

FAQ

Does Hugging Face's Croissant endpoint include Responsible AI fields?

Not for the datasets we checked. On 7 October 2026 BLOMEGA requested huggingface.co/api/datasets/<id>/croissant for all 500 CroissantMiner silver datasets and 102 gold datasets. 533 returned a Croissant file; none contained any rai: key. Every file carried name, description, url and creator, and 433 carried a licence. The served file for bearda/croissantminer has no rai: key either, although the croissant.json the authors uploaded to that repository has 20.

Can the CroissantMiner Table 2 be reproduced?

Yes. Running python scripts/figures/print_table2.py from github.com/berkearda/croissantminer prints all 24 ranked rows and the Sonnet 4.5 reference to three decimals as in the paper, in about 5 seconds, with no API calls. One scoring rule is asymmetric: gold values such as null or not found count as empty, but a system output counts as empty only when missing or blank. Treating a literal null string as empty raises Mistral Small 4 from 0.527 to 0.566 and Llama 4 Scout from 0.391 to 0.407; the top three systems do not move.

How close are the CroissantMiner silver labels to human gold?

On the 102 gold papers, 2,684 of 3,060 gold cells (87.7%) are identical to the Claude Sonnet 4.5 pre-fill that also produced the silver labels: 90.0% of core cells and 86.6% of RAI cells. The paper's own metric scores Sonnet 4.5 at 0.861 composite on the test split. A silver null is weak evidence that a paper is silent: the silver split leaves cr:citeAs empty for 39.0% of papers and cr:isLiveDataset for 92.2%, while the same model filled both for at least 95% of gold papers.