PubChem's missing assay formats are mostly already in ChEMBL, and a field lookup scores 0.990 where seven LLMs score 0.951 to 0.979

What the AstraZeneca paper claimed on 1 October 2026

Laura van Weesep and Riccardo Tedoldi (equal contribution) and seven co-authors from AstraZeneca Molecular AI, Uppsala University, KTH, SciLifeLab and Chalmers posted Can LLMs Reliably Annotate Bioassay Metadata to Improve Data Readiness? as arXiv:2610.01616v1 (cs.CL) on 1 October 2026. The arXiv comment says it is accepted to the AI Data Readiness (AIDaR) workshop at NeurIPS 2026.

The paper makes two claims. First, from PubChem's bulk files (1,994,310 AIDs at their download), coverage is "critically sparse": 36% of assays lack an Assay Format, 89% a BioAssay Type, and over 99.9% any BAO assay format or detection technology term (Section 4, Tables 5 to 9). Second, seven LLMs (Claude Sonnet 4.6, Gemini 3.6 Flash, GPT-4o, GPT-OSS 20B, Gemma 3 27B, Gemma 4 31B, Llama 3.3 70B) label BAO depth-one assay format from assay text with 0.951 to 0.979 accuracy against ChEMBL 37 labels on 1,097 PubChem assays (Table 1), and detection technology with 0.883 to 0.905 on 308 assays (Table 4).

Prompts are printed in Appendix E; code and evaluation labels are promised "upon acceptance of the full paper" (footnote 1). On 5 October 2026 we found no matching repository in the AstraZeneca MolecularAI GitHub organisation, whose newest repository was created 5 August 2026, and GitHub repository search for the obvious terms returned zero hits. One small inconsistency: Section 3 lists Qwen 3.5 among the four open-weight models, but every results table reports Gemma 3 27B and no Qwen model.

How much of PubChem really lacks an assay format?

We downloaded bioassays.tsv.gz, Aid2Annotation.gz and Aid2CategorizedComment.gz from PubChem's Bioassay/Extras directory (server Last-Modified 29 September 2026) and counted every live AID. The bulk BAO counts match the paper exactly: 280 AIDs carry a BAO assay format comment and 348 a BAO detection technology comment. Our Assay Format gap is smaller than the paper's 36%, probably because PubChem added annotations between their download and ours; the paper does not give a download date.

Two facts the paper does not report change the reading. ChEMBL deposited 1,953,598 of the 1,982,401 live AIDs (98.5%), and 438,967 of the 644,193 AIDs without a format (68.1%) are ChEMBL assays whose PubChem Assay Type is Binding. We drew 500 random ChEMBL-sourced AIDs without a PubChem format (seed 20261004) and looked each one up in the ChEMBL web services. ChEMBL 37 gives 334 of them a specific BAO format (66.8%, Wilson 95% interval 62.6% to 70.8%), mostly single protein format.

Adding the 1,338,208 AIDs that have a PubChem format to the 66.8% of the ChEMBL gap that ChEMBL fills, about 88.2% of PubChem assays (86.9% to 89.5%) have a BAO depth-one or finer format in one of the two databases. The 0.018% figure is true of PubChem's categorized comments, which is where BARD's curation lives; it is not the share of PubChem assays for which a BAO format exists. We could not rebuild the paper's 236 BARD and ChEMBL overlap from the Extras files: only 42 BARD-annotated AIDs are among ChEMBL's PubChem-sourced assays, and 25 of those have a specific ChEMBL label.

Why a field lookup scores 0.990: PubChem's format is ChEMBL's label with the biochemical subtypes dropped

Aid2Annotation is PubChem's third-party annotation file, and its contents look like ChEMBL's: its Assay Type values (Binding, Functional, ADME, Toxicity, Physicochemical) are ChEMBL's assay types, present on 1,956,839 AIDs. In our random sample of 300 ChEMBL-sourced AIDs that do have a PubChem Assay Format, the value equals ChEMBL's BAO label in all 300 (Wilson lower bound 98.7%). In the 500-AID gap sample, not one ChEMBL label is cell-based or organism-based (upper bound 0.8%).

The Biochemical value is the telling case. All of PubChem holds 119 AIDs with Assay Format Biochemical, and ChEMBL's PubChem-sourced assays hold exactly 119 with the generic label "biochemical format"; the two sets coincide. The specific children (single protein format, protein complex format, nucleic acid format, protein format) are not mapped and arrive as an empty field. Our inference, which PubChem has not confirmed, is that the import keeps a ChEMBL label only when it matches one of PubChem's three words verbatim. That one vocabulary mapping produces most of the "missing" 32.5%.

To find the paper's test pool we pulled all 2,999 ChEMBL assays whose source is "PubChem BioAssays" (ChEMBL src_id 7); each carries the PubChem AID in src_assay_id. 1,706 (56.9%) have the unspecified label "assay format"; the paper discards those. The remaining 1,293 split 926 cell-based, 282 biochemical, 76 organism-based, 7 tissue-based and 2 cell-free, against Table 1's 784, 252, 53, 6 and 2. The paper says it also removed duplicate descriptions and AIDs mapping to several ChEMBL records; none of our 2,999 maps to more than one. 1,117 of the 1,293 (86.4%) already have a PubChem Assay Format and it equals the ChEMBL label in all 1,117. Predicting that field where present and "biochemical" otherwise gets 1,280 of 1,293 right (0.990, interval 0.983 to 0.994). The 13 misses are 7 tissue-based, 4 cell-based and 2 cell-free.

This baseline is label leakage, and that is the point: on this test set the reference labels already sit in PubChem, so the evaluation measures how well an LLM re-derives a label PubChem already shows, not how well it fills the gap. Rules that do not read the ChEMBL label give the fairer floor. A title keyword rule (cell-based, cell line or cells gives cell-based; else purified, enzyme, enzymatic, binding, recombinant or biochemical gives biochemical; else cell-based) scores 0.882 (0.863 to 0.898); the depositor's BioAssay Types field, read as Organism-based, Cell-based or Biochemical with cell-based as fallback, scores 0.889 (0.870 to 0.905). Measured against those floors, the LLM majority vote (0.961) removes about two thirds of the remaining error, with the caveat that our pool and the paper's 1,097 overlap heavily but are not identical.

The paper calls the BioAssay Type field unsuitable because it mixes format, method and endpoint. When the field names a format, it fires on 924 of the 1,293 pool assays and agrees with ChEMBL on 895 of them (96.9%, 95.5% to 97.8%), within the LLM range. Our judgement: the field is incomplete, not unreliable, and it is worth using as a free first pass.

How far above a majority-class guess are the LLMs on each of the paper's tables?

None of the paper's tables prints a majority-class row, but each prints class supports, so the trivial baseline can be read straight off them. The gap is wide on every task. Table 18 also lists parse failures, and Table 1's caption says invalid predictions were excluded from the metrics. Claude Sonnet 4.6 failed to return parseable output on 29 of 1,097 format items, 21 of 226 PubChem items and 34 of 308 detection items; every other model had at most 3 failures in any one experiment. Counting those as wrong moves Claude from third place to last on assay format and from first to last on detection.

The parse-failure counts sit in Table 18 in a cell that pdftotext merges with the token count ("292,302,201"); reading the PDF's text positions separates it into 29 failures and 2,302,201 input tokens, consistent with the other models' 2.0 to 2.2 million. The adjusted figures assume failures were dropped from the denominator, as Table 1's caption states; if the authors scored them some other way the adjustment changes.

Would the LLMs score 0.96 on the assays that are actually missing a format?

Probably not, and the reason is the class mix. The test set is 71.5% cell-based and 0.7% tissue-based or cell-free. The ChEMBL-sourced gap, after dropping unspecified and physicochemical labels as the paper does, is 80.3% biochemical, 7.1% tissue-based and 12.6% cell-free (mostly microsome and cell membrane formats from ADME assays), with no cell-based or organism-based assays at all. Table 1's recall on tissue-based (n=6) is 0.00 for six of seven models and on cell-free (n=2) is 0.00 for six of seven.

If Table 1's per-class recalls carried over unchanged, majority-vote accuracy on the gap mix would be about 0.779 and the best single model (Gemini 3.6 Flash, thanks to one correct cell-free item) about 0.834. Always answering "biochemical" scores 0.803 (0.755 to 0.844) on the same mix. This is a projection from recalls measured on 6 and 2 items, so treat it as a reason to test, not as a measurement; the paper's own expert review found several of those rare-class "errors" were silver-label errors.

What should an assay-data team do before paying an LLM to fill PubChem's format gap?

Join before you generate. About two thirds of the ChEMBL-sourced gap (an estimated 307,000 AIDs biochemical, 48,000 cell-free, 27,000 tissue-based, 30,000 physicochemical) can be filled from ChEMBL 37 by matching Source ID to assay_chembl_id. That costs nothing and gives the curated label. At the paper's Table 18 rates ($3.47 to $9.09 per 1,097 records), labelling all 616,509 ChEMBL gap AIDs would cost roughly $2,000 to $5,100 in API fees, cheap but pointless where the curated label is one join away.

Point the LLM at the true residue. The assays with no format anywhere are about 204,700 ChEMBL AIDs labelled "assay format" plus 27,684 non-ChEMBL AIDs without a PubChem format. These are exactly the records the paper's evaluation excludes, because there is no silver label to score against. Measuring LLM accuracy there needs a fresh, independently labelled sample, stratified so that tissue-based and cell-free get enough items for recall to mean something (judgement: 50 or more per class).

Report the floors and the failures. For any LLM annotation study, print the majority-class score, the best cheap rule using existing fields, and accuracy with unparsed outputs counted as wrong. On this paper those three numbers change the ranking of the seven models and shrink the apparent gain over existing metadata from about 0.25 to about 0.07.

Use the audit role, where the evidence is better. The strongest result in the paper is qualitative: in 17 of the 29 Table 2 disagreements that have a BARD label, the LLM majority sided with BARD against ChEMBL, and a senior curator agreed with the LLM majority on 12 of 17 reviewed assays, six of them only after reading the model's evidence. That supports using LLMs to flag conflicts between ChEMBL, BARD and PubChem for human review, which needs no claim about fill-in accuracy.

Check it yourself

About 87 MB of PubChem files and 19 ChEMBL API calls; under two minutes on a laptop. Fields used: Source Name (column 5), Source ID (6) and BioAssay Types (11) of bioassays.tsv; title Assay Format and Assay Type in Aid2Annotation; ChEMBL bao_label, src_assay_id and src_id.

B=https://ftp.ncbi.nlm.nih.gov/pubchem/Bioassay/Extras
for f in bioassays.tsv.gz Aid2Annotation.gz Aid2CategorizedComment.gz; do curl -sO $B/$f; done
python3 - <<'PY'
import gzip,csv,json,collections,urllib.request
csv.field_size_limit(10**9)
cur={}
for row in csv.reader(gzip.open('bioassays.tsv.gz','rt',newline=''),delimiter='\t',quoting=csv.QUOTE_NONE):
    if row[0].isdigit(): cur[int(row[0])]=(row[4],row[5],row[10])
fmt={}
for l in gzip.open('Aid2Annotation.gz','rt',errors='replace'):
    p=l.rstrip('\n').split('\t')
    if len(p)>2 and p[0].isdigit() and p[1]=='Assay Format': fmt.setdefault(int(p[0]),p[2])
N=len(cur); miss=[a for a in cur if a not in fmt]
print(N, len(miss), sum(cur[a][0]=='ChEMBL' for a in cur))   # 1982401 644193 1953598
# ChEMBL assays imported from PubChem (src_id 7): the paper's test pool
rows=[]
for off in range(0,3000,1000):
    u=f'https://www.ebi.ac.uk/chembl/api/data/assay.json?src_id=7&limit=1000&offset={off}&only=src_assay_id,bao_label'
    rows+=json.load(urllib.request.urlopen(u))['assays']
bio={'single protein format','protein complex format','nucleic acid format','protein format','biochemical format'}
d1=lambda l:'biochemical' if l in bio else l.replace(' format','')
E=[(int(r['src_assay_id']),d1(r['bao_label'])) for r in rows if r['bao_label']!='assay format']
print(len(rows), len(E), collections.Counter(y for _,y in E))   # 2999 1293 {cell-based 926, biochemical 282, ...}
look={'Cell-based':'cell-based','Organism-based':'organism-based','Biochemical':'biochemical'}
ok=sum(look.get(fmt.get(a),'biochemical')==y for a,y in E); print(ok, ok/len(E))   # 1280 0.9899
PY

The 500-AID gap sample uses random.Random(20261004).sample over the sorted ChEMBL-sourced AIDs without an Assay Format whose Source ID starts with CHEMBL, then /chembl/api/data/assay.json?assay_chembl_id__in=... in batches of 50. Cell-free means ChEMBL's cell-free, microsome, cell membrane and subcellular formats, following the paper's Appendix E prompt. ChEMBL's status endpoint reported ChEMBL_37 (release 1 May 2026) on 5 October 2026. Paper numbers come from arXiv:2610.01616v1: Tables 1, 3, 4, 5, 6, 9 and 18; Table 18's parse-failure column was read from PDF text positions with pypdf.

What would prove this wrong

Two predictions, to be checked against the code and labels the authors promise to release, or by 30 June 2027 at the latest. First, at least 95% of the 1,097 ChEMBL test AIDs will be ChEMBL src_id 7 assays, and at least 85% of them will already carry a PubChem Assay Format equal to their ChEMBL label (our pool: 86.4% and 100%). Second, on a random sample of 300 or more PubChem AIDs that have no PubChem Assay Format but a specific depth-one ChEMBL label, the paper's prompt and models will reach a majority-vote accuracy below 0.90 against ChEMBL, under 0.85 if parse failures count as errors. If the released AIDs are mostly not src_id 7, or the gap-sample accuracy lands at 0.95 or above, our reading that the evaluation measures re-derivation rather than gap filling is wrong.

Sources

1. van Weesep, L., Tedoldi, R., Sjölund, J., Azizpour, H., Winiwarter, S., Engkvist, O., Janet, J. P., Genheden, S., Viguera Diez, J. Can LLMs Reliably Annotate Bioassay Metadata to Improve Data Readiness? arXiv:2610.01616v1, 1 October 2026. AstraZeneca, Uppsala University, KTH, SciLifeLab, Chalmers. Sections 3 to 6, Tables 1 to 9 and 18, Appendix E.
2. NCBI PubChem. Bioassay/Extras: bioassays.tsv.gz, Aid2Annotation.gz, Aid2CategorizedComment.gz, Last-Modified 29 September 2026. Accessed 5 October 2026.
3. EMBL-EBI. ChEMBL web services, ChEMBL_37 (released 1 May 2026), assay and source endpoints. Accessed 5 October 2026.
4. BioAssay Ontology (BAO), v2.8.19 as cited by the paper.
5. Howe, E., et al. BioAssay Research Database (BARD): chemical biology and probe-development enabled by structured metadata and result types. Nucleic Acids Research 43(D1), 2015.
6. Smit, I., et al. Integrating artificial intelligence and manual curation to enhance bioassay annotations in ChEMBL. Journal of Cheminformatics 18, February 2026.
7. AstraZeneca. MolecularAI on GitHub, repository list checked 5 October 2026 (no repository for this paper).
8. BLOMEGA. VoiceNet-Emo's sampling column scores 0.716, an earlier trivial-baseline check in this series.

FAQ

How much of PubChem BioAssay really lacks an assay format?

In PubChem's files of 29 September 2026, 644,193 of 1,982,401 live AIDs (32.5%) have no Assay Format; arXiv:2610.01616 reports 36% on an earlier download. ChEMBL deposited 98.5% of PubChem's assays, and in a random sample of 500 ChEMBL-sourced AIDs with no PubChem format, ChEMBL 37 gives 334 (66.8%, 95% interval 62.6% to 70.8%) a specific BAO format, mostly single protein format. BLOMEGA estimates about 232,000 AIDs (11.7%) have no format in either database, and about 88% have a BAO depth-one or finer format in one of the two.

Do LLMs beat a trivial baseline at labelling bioassay format?

They beat the majority class by a wide margin: 0.951 to 0.979 accuracy against ChEMBL labels versus 0.715 for always answering cell-based (paper Table 1), and 0.883 to 0.905 versus about 0.46 to 0.52 on detection technology (Table 4). They beat cheap rules by less: a title keyword rule scores 0.882 and a BioAssay Types rule 0.889 on 1,293 ChEMBL assays imported from PubChem, computed by BLOMEGA. Reading PubChem's Assay Format field, which is copied from ChEMBL's label, scores 0.990. Counting its unparsed outputs as wrong, Claude Sonnet 4.6 falls from 0.963 to 0.938 on format and from 0.905 to 0.805 on detection.

Are the code and labels for arXiv:2610.01616 released?

Not as of 5 October 2026. Prompts are printed in Appendix E, and footnote 1 says code and data will be released upon acceptance of the full paper. BLOMEGA found no matching repository in the AstraZeneca MolecularAI GitHub organisation (newest repository created 5 August 2026) and zero GitHub search hits. The test pool can be approximated from ChEMBL's 2,999 assays with source PubChem BioAssays (src_id 7), whose 1,293 labelled assays have almost the same class mix as the paper's 1,097.