PaperlayerView exact revision on arXiv

Source-linked AI summary

Can LLMs Reliably Annotate Bioassay Metadata to Improve Data Readiness?

Laura van Weesep, Riccardo Tedoldi, Jens Sjölund, Hossein Azizpour, Susanne Winiwarter, Ola Engkvist, Jon Paul Janet, Samuel Genheden, Juan Viguera Diez

arXiv:2610.01616v1cs.CLcs.AIcs.DBq-bio.QM

TL;DR

Bioassay repositories have missing, inconsistent, or conflated metadata, limiting data readiness for machine-learning research. This paper tests seven LLMs on assay-text annotation and audits disagreements with existing labels. LLMs recover supported assay-format classes with recall above 0.96, while review reveals source inconsistencies and cases where curator labels changed after seeing model evidence.

  • Problem

    Public and industrial bioassay repositories contain missing, inconsistent, or conflated metadata, while LLM performance on multidimensional BAO labels remains underexplored.

    Source: §1 · p. 2

  • Method

    The authors evaluated seven open-source and proprietary LLMs on assay-text prediction of BAO labels and manually reviewed disagreements, including with a senior industrial curator.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    LLMs recovered supported assay-format classes with recall above 0.96, and many disagreements with silver labels reflected source inconsistencies or mislabeling rather than model error.

    Source: §6 · p. 9

  • Takeaways & Limitations

    LLMs may serve as annotators and auditors by surfacing evidence and routing ambiguous or conflicting cases for human review with per-class reliability estimates.

    Source: §6 · p. 9, §5.3 · p. 9

    Silver labels are only partially reliable, so recall measures agreement with existing curation rather than ground-truth accuracy; evaluation also stays at BAO depth one.

    Source: §6 · p. 9

Abstract

from arXiv · show

The emergence of foundation models for molecular property prediction requires a high degree of AI data readiness, including reliable metadata annotation. However, both public repositories and industrial screening databases suffer from missing, inconsistent, or conflated assay annotations. In this work, we quantify the extent of missing annotations in PubChem for the BioAssay Ontology (BAO) assay format and physical detection method fields and investigate whether open-source and proprietary large language models (LLMs) can reliably predict and audit metadata annotations directly from the assay text. In our assessment, we found that the annotation coverage across PubChem's $\sim$2 million bioassays is critically sparse, 36\% lacking an assay format, 89\% a BioAssay type, and >99.9\% any BAO-mapped assay format or detection technology term. This motivates the need for automated test-metadata curation. Using evaluation sets derived from PubChem and ChEMBL, we assess the agreement of seven open-source and proprietary LLMs with existing silver labels. Recall is at least 0.96 for biochemical and cell-based assay formats, with a similar pattern for detection technology, although disagreements increase on under-represented classes. Manual inspection shows that many of these disagreements trace back to inconsistencies between silver sources rather than to LLM error. Moreover, in a qualitative study with a senior industrial curator, LLM-generated evidence prompted the expert to revise some of their own labels, showing LLMs can flag potentially mislabeled assays. Across the study, performance differences between proprietary and open-source models were small. Together, these results suggest LLMs can support the large-scale annotation and auditing of assay metadata, though per-class reliability estimates and targeted human review remain necessary before such labels enter downstream ML pipelines.

1 Introduction

Bioassay repositories contain missing, inconsistent, and conflated metadata, limiting AI readiness for data used in drug-discovery machine learning. The paper evaluates LLMs for assigning BAO assay formats and detection methods and reviews disagreements with existing labels.

1 Introduction

  • Bioassay metadata are often missing or inconsistent in public repositories and industrial databases, while single labels can conflate distinct assay aspects.PubChem records may predate standards or use inconsistent terminology; industrial databases also contain missing historical data and composite labels.Source: §1 · p. 2
  • BAO separates assay format from physical detection method and other axes, addressing ambiguity in composite labels such as PubChem’s BioAssay Type.The framework maps unstructured descriptions to standardized entities, but manually disentangling millions of records does not scale.Source: §1 · p. 2
  • Whether LLMs can reliably assign labels across a multidimensional ontology such as BAO remains underexplored.Prior work cited in the paper reports instruction-tuned LLM performance on biomedical ontology alignment.Source: §1 · p. 2
  • The paper evaluates seven LLMs on BAO assay-format and detection-method assignment, and manually reviews disagreements with existing labels.The review included a senior industrial curator, who agreed with LLM evidence over original labels in some cases.Source: §1 · p. 2

2 Background and Related Work

BAO provides structured, orthogonal assay descriptions, but annotation coverage and consistency remain limited across public repositories. Prior manual and automated curation approaches leave a need to test whether LLMs can assign BAO labels from assay text.

2.1 Bioassay Ontology (BAO)

  • BAO organizes assays across orthogonal axes and has expanded over time, while earlier manual work mapped PubChem assay names to controlled terms.Vempati et al. mapped 299 names to 20 BAO terms; the ontology’s development has also addressed the breadth of assay types and interoperability.Source: §2.1 · p. 2

2.2 Annotated public data repositories

  • Public annotation sources differ in coverage and reliability: BARD expert-reviewed a small PubChem subset, while ChEMBL coverage is broader but incomplete.For records shared across all three sources, the paper treats BARD labels as most trustworthy, ChEMBL as secondary, and native PubChem labels as least reliable.Source: §2.2 · p. 3
  • Existing curation pipelines and benchmarks address data quality, but often depend on existing metadata or keyword searches rather than comprehensive structured annotations.CAPRICHO case studies report that grouping by assay type and cell type improved cross-assay agreement; the paper identifies LLM assignment of BAO labels from free text as an open question.Source: §2.2 · p. 3, §2.3 · p. 4

2.3 Bioassay curation with language models

  • Prior LLM-based biomedical extraction and ontology-grounding work motivates evaluating whether LLMs can consistently assign BAO format, detection, and target labels from free text.The paper notes that grounding outputs in formal ontology terms can address LLMs’ tendency to generate non-standard labels.Source: §2.3 · pp. 3–4

3 Methods

The evaluation combines PubChem assay text with linked database annotations to create labeled datasets for BAO format and detection-method prediction. Seven LLMs predict labels from assay text, with existing database annotations treated as silver rather than ground-truth labels.

3 Methods

  • The study assembled PubChem assay titles, descriptions, comments, and protocols from bulk data, supplementing missing fields through the PUG REST endpoint.Of 1,994,310 downloaded assays, 1,770,568 had a name, comment, and/or description.Source: §3 · p. 4
  • Linked ChEMBL and PubChem annotations were filtered, mapped to depth-one BAO categories, and used to form three evaluation sets.The sets comprised 1,097 assays with ChEMBL format labels, 226 PubChem-only format labels, and 308 assays with physical detection-method labels.Source: §3 · p. 4
  • Seven proprietary and open-weight LLMs predicted assay format or physical detection method from assay text, while reference database labels were treated as silver labels.Prompt ablations used a stratified subset of 500 ChEMBL format assays; silver labels may not reflect current BAO ground truth because source annotations can be inconsistent.Source: §3 · p. 4

4 Missingness and Inconsistency in PubChem Bioassay Metadata

PubChem’s assay metadata are sparse and inconsistent, and some fields conflate distinct assay properties. These issues limit the usefulness of native labels for BAO-aligned annotation.

4 Missingness and Inconsistency in PubChem Bioassay Metadata

  • The BioAssay Type vocabulary mixes format, methodology, and endpoint, making it unsuitable for assigning BAO assay format.Some entries combine multiple labels that are inconsistent with BAO.Source: §4 · p. 5
  • 36% of PubChem entries lacked an assay format, and fewer than 0.1% had BAO assay format and physical detection method annotations in comments.The PubChem Assay Format field has only three categories, whereas the BAO assay-format branch is richer.Source: §4 · p. 5
  • Among 236 assays with both BARD and ChEMBL format labels, 24 labels disagreed, and some PubChem records differed between bulk and web views.The paper reports AID 1885 as Cell-based in the web interface but Cell-based|Organism-based in the bulk download.Source: §4 · p. 5

5 Results: An LLM-based Auditing Workflow

The LLMs generally agreed with reference labels, while manual review found that some disagreements reflected inconsistent or ambiguous annotations rather than model error. Detection-method results and confidence-based triage suggest potential for auditing, but rare classes and methods that do not fit BAO categories remain difficult.

5.1 Reliability against silver sources

  • 96% of majority-vote predictions agreed with ChEMBL; every model achieved at least 96% recall on cell-based and biochemical formats, while rare classes performed poorly.Tissue-based and cell-free classes had very few examples, and manual inspection found BARD labels differed from ChEMBL labels for 7 of 8 such assays.Source: §5.1 · pp. 5–6, Table 1 · p. 6
  • Among 29 reviewed disagreements with ChEMBL that had BARD labels, 17 (59%) matched BARD instead, whose annotations include additional verification.The study also found that LLMs agreed with BARD in most cases where BARD and third-party PubChem annotations differed.Source: §5.1 · pp. 6–7
  • Experts agreed immediately with LLMs in 6 cases, changed their view after reviewing model evidence in 6 more, and found 4 further classifications plausible.The evidence did not persuade experts in every case: for AID 504631, the expert preferred ChEMBL because the models misread a living-organism assay as cell-line format.Source: §5.1 · pp. 6–7

5.2 Extending to a Second BAO Axis: Detection Technology

  • On the detection axis, LLMs had high recall for fluorescence, luminescence, and radiometry, but more moderate spectrophotometry results; rare classes had too few labels for firm conclusions.This complementary probe covered 12 depth-one detection classes, compared with 6 assay-format classes, and used fewer ChEMBL cross-referenced labels.Source: §5.2 · p. 7, Table 4 · p. 8
  • Of 24 reviewed detection-method disagreements with available BARD annotations, LLMs agreed with BARD in 15, suggesting many conflicts with silver labels may reflect label inaccuracies.BARD disagreed with the third-party annotation in all 24 cases with an available BARD annotation.Source: §5.2 · p. 8
  • In expert review of 11 detection records, 6 were judged undefinable and 3 had multiple acceptable methods; only 2 remained true single-label disagreements.For the undefinable assays, non-erroring models reported low confidence even while emitting a best-guess label.Source: §5.2 · p. 8

5.3 Calibrated confidence and Prompt Ablations

  • For high-confidence predictions, disagreement with silver labels reached at most 5% on the format task and 10% on detection, supporting confidence-based review triage.The authors still call for further verification because these disagreement rates are measured against silver labels.Source: §5.3 · pp. 8–9
  • Reversing label order changed predictions in at most 2.6% of cases, while removing BAO definitions changed them by 3.1%–10.8% across reported models.Gemini changed more often away from the silver label without definitions, suggesting greater reliance on the in-prompt definitions.Source: §5.3 · p. 9

6 Discussion and Conclusion

PubChem’s assay metadata is sparse and inconsistent, while LLMs agree strongly with existing assay-format labels and can help identify disputed annotations. The authors propose using them as annotators and auditors, with human review and per-class reliability estimates before downstream use.

6 Discussion and Conclusion

  • Roughly 36% of PubChem records lack an assay format, fewer than 0.1% have structured BAO annotations, and 10% of 236 jointly labeled assays disagree across BARD and ChEMBL.The authors attribute the gaps in part to optional deposition fields and PubChem’s narrower native vocabulary.Source: §6 · p. 9
  • LLMs achieved recall above 0.96 for well-supported assay-format classes, while over a third of majority-vote disagreements with ChEMBL matched PubChem/BARD labels instead.A senior annotator revised some labels after reviewing model evidence; proprietary and open-source model differences were small.Source: §6 · p. 9
  • The authors propose LLMs as annotators and auditors that flag inconsistencies, surface evidence, and route ambiguous records to human curators before downstream use.They call for per-class reliability estimates before labels enter ML pipelines.Source: §6 · p. 9
  • Recall against silver labels measures agreement with existing curation, not ground-truth accuracy, and the evaluation is limited to BAO depth one and one expert’s judgment.The live PUG JSON endpoint’s rate limits constrained the depth and breadth of PubChem data examined.Source: §6 · p. 9
  • For AID 504631, models cited correct evidence about a single-celled organism but still assigned the cell-based rather than organism-based BAO class.The authors recommend continued human spot-checking that cited evidence supports the assigned label in known tricky cases.Source: §6 · pp. 9–10
  • LLM auditing also exposed BAO gaps: “biosensor” lacks a definition, while “simple measurement” and “visual observation” definitions divided models and the curator.The authors suggest this auditing can inform ontology maintenance as well as data curation.Source: §6 · p. 10
  • The authors identify deeper BAO granularity, broader and industrial datasets, and LLM-mediated debate as next steps for resolving remaining silver-label conflicts.Source: §6 · p. 10

A Dataset Creation Details

The dataset was assembled from PubChem bulk resources, with direct PUG REST queries used to supplement missing assay descriptions.

A Dataset Creation Details

  • The authors downloaded PubChem JSON zip files, bioassays.tsv.gz, Aid2CategorizedComment.gz, and Aid2Annotation.gz for evaluation.Source: A Dataset Creation Details · p. 13
  • Seven JSON zip files containing categorized assay names, comments, and descriptions were empty at download time.Source: A Dataset Creation Details · p. 13
  • Of 1,994,310 downloaded BioAssay AIDs, 1,770,568 contained a name, comment, or description; PUG REST queries supplied descriptions for all analyzed assays.Source: A Dataset Creation Details · p. 13

B Dataset information from bulk PubChem

This appendix summarizes PubChem bulk-data statistics and reproducible LLM prompt configurations, illustrating conflicts among assay labels and reporting pairwise agreement across assay-format and detection-technology tasks.

B Dataset information from bulk PubChem

E Prompts used

E.1 Case studies of annotation inconsistencies in PubChem

  • For AID 1885, the PubChem interface labels the assay cell-based, while the bulk download also labels it organism-based, despite no intact organism being used.The same ambiguity may affect 42 other entries carrying both labels and additional label combinations.Source: §E.1 · p. 23
  • AID 588585 is labeled biochemical in a categorized-comment BAO field and subcellular in BARD-derived annotations, yet is absent from the BARD bulk download.Across the downloaded AID JSON files, only one third-party BAO label was found, and BARD and the third party agreed on it.Source: §E.1 · p. 23
  • 24 of 236 assays with BARD and ChEMBL assay-format labels disagreed (10.2%), including cell-based versus organism-based and biochemical versus cell-based labels.Other disagreements involved tissue-based versus cell-based labels and small-molecule physicochemical versus biochemical labels; the examples show that expert-driven curation does not guarantee consistency.Source: §E.1 · p. 23

F.1 Structured output

  • JSON output was enforced by API JSON mode for GPT-4o and Gemini, prompt instructions for Claude, and JSON-Schema-constrained decoding for open-weights models.For the open-weights models, the schema restricted generation to the valid BAO-label vocabulary.Source: §F.1 · p. 24

F.2 Proprietary models (API-served)

  • The three proprietary API-served models used temperature 0.0 for deterministic decoding, with maximum output length left at provider defaults.The compact JSON response contained three short fields, described as well below provider output caps.Source: Table 10 · p. 24

F.3 Open-weights models (Ollama v0.17.4)

  • Open-weights models ran locally through Ollama with fixed seed=42; GPU power draw was logged once per second for energy and CO2 accounting.Runs used JSON-Schema-constrained decoding to enforce the valid BAO-label enum.Source: §F.3 · p. 24, Table 11 · p. 24

G Inter-model agreement (Cohen’s κ)

  • Across seven models, ChEMBL format pairs all exceeded κ > 0.9 (n = 1097), PubChem pairs were mostly κ > 0.8 (n = 226), and detection-technology agreement was near-perfect.On the PubChem subset, disagreements concentrated on organism-based and cell-free classes; Cohen’s κ measures agreement beyond chance.Source: Figure 3 · p. 25, Figure 4 · p. 25, Figure 5 · p. 25

H Full manual-inspection table for ChEMBL assay format disagreements

This appendix provides row-level details for 43 assays where most evaluated LLMs disagreed with ChEMBL’s BAO assay-format labels, including silver labels, predictions, and model votes. Expert-review examples illustrate that disagreements may reflect incorrect labels or ambiguity, including cases where review favored the LLM majority and a failure to distinguish cell-based from organism-based assays.

H Full manual-inspection table for ChEMBL assay format disagreements

  • Table 12 covers 43 assays where most LLMs disagreed with ChEMBL’s BAO assay-format label, listing silver labels, top predictions, and model vote counts.It also indicates whether a BARD annotation was available and which rows received senior industrial curator review.Source: Table 12 · p. 26, H Full manual-inspection table for… · p. 26
  • In reviewed examples, the expert agreed with the LLM majority and judged the ChEMBL/BARD label incorrect.Examples include assays with cell-based top predictions and ChEMBL labels marked incorrect.Source: H Full manual-inspection table for… · pp. 29, 31
  • For a protease assay, the expert initially leaned toward ChEMBL but revised the assessment after LLMs highlighted membrane preparations in the assay description.The curator noted that the text described characterization of yeast and human CaaX proteases.Source: H Full manual-inspection table for… · p. 29
  • A cell-lysate assay exposed ambiguity between cell-free and biochemical formats, depending on whether the proteome was considered sufficiently purified.The expert reasoned that a lysate-derived mixture could be cell-free but questioned whether it qualified as biochemical.Source: H Full manual-inspection table for… · p. 30
  • For another assay, the expert agreed with the LLM majority that the ChEMBL/BARD label was incorrect, while noting that the extract-based assay might be biochemical.The table identifies the assay as using yeast nuclear extracts.Source: H Full manual-inspection table for… · p. 30
  • In a separate reviewed case, the expert agreed with the LLM majority that the ChEMBL/BARD label was incorrect, citing protocol language that cells were lysed.The short title and protocol indicated a cell-based format, though the assay involved yeast.Source: H Full manual-inspection table for… · p. 31
  • For a C. albicans growth assay, the expert agreed with the LLM majority and judged the ChEMBL label incorrect.The table entry identifies the assay as fluorescence cell-based retest of C. albicans growth.Source: H Full manual-inspection table for… · p. 31
  • In another case, the expert agreed with the LLM majority that the ChEMBL/BARD label was incorrect after discussing the definition.The expert initially leaned toward the ChEMBL label before adapting their judgment.Source: H Full manual-inspection table for… · p. 31
  • A further entry records expert agreement with the LLM majority and a ChEMBL/BARD label judged incorrect after discussion of the definition.The expert initially leaned toward ChEMBL; the assay title indicated cell-based format.Source: H Full manual-inspection table for… · p. 31
  • For AID 504631, the expert agreed with ChEMBL while the LLM majority was incorrect for the single-celled Plasmodium falciparum assay.This counterexample illustrates that model majorities can follow the cell-based title rather than the BAO organism-based classification.Source: H Full manual-inspection table for… · p. 31, §H.1 · p. 32

H.1 Failure case: AID 504631

  • BAO assigns single-celled organisms, including eukaryotic parasites, to organism-based rather than cell-based format.Cell-based format is restricted to eukaryotic cells derived from multicellular organisms.Source: §H.1 · p. 32
  • For AID 504631, four of seven models predicted cell-based instead of organism-based format, while three correctly predicted organism-based format.The incorrect predictions persisted despite evidence mentioning a malarial parasite or microorganism system, suggesting surface-cue pattern matching rather than application of the BAO distinction; the assay title may reinforce this bias.Source: §H.1 · p. 32, Table 14 · p. 32

I Detection Disagreement Details

The appendix details 34 detection-method cases where LLM majorities disagreed with PubChem labels, including selected expert reviews and per-model evidence. Examples show both sparse assay descriptions and differences in how assay readouts are categorized.

I Detection Disagreement Details

  • The appendix reports all 34 cases where model majorities disagreed with PubChem labels and provides expert-reviewed cases with model evidence spans.Tables 15–17 cover the disagreement records, expert annotations, and selected model predictions.Source: I Detection Disagreement Details · p. 33, Table 15 · p. 33, Table 16 · p. 34
  • In several probe-development records, the expert marked the detection method “cannot be defined,” citing pre-work rather than an assay or missing information.Examples include efforts to identify probes for MsrA and muscarinic receptors.Source: I Detection Disagreement Details · p. 35
  • Plaque counts were determined by visual counting, although the expert was reluctant to classify that readout as “visual observation.”The expert reserved that category for larger-organism behavioral readouts.Source: I Detection Disagreement Details · pp. 35, 38
  • For MsrA probe-development records, models offered low-confidence methods based on common practice despite descriptions lacking explicit detection evidence.Suggestions included fluorescence, spectrophotometry, simple measurement, and visual observation.Source: I Detection Disagreement Details · p. 36
  • For the RecBCD assay, models split between simple measurement and visual observation, citing differential plating, colony counts, or recombinant frequency.The assay title describes identifying inhibitors of RecBCD recombination-promoting activity in E. coli.Source: I Detection Disagreement Details · pp. 35–36

J Confidence calibration on both tasks

Figure 6 shows error rates by predicted-confidence category for the ChEMBL baseline and detection task.

J Confidence calibration on both tasks

  • Figure 6 groups incorrect predictions by confidence category, with sample counts in parentheses, for the ChEMBL baseline and detection task.The top panel is the ChEMBL baseline; the bottom panel is the detection task.Source: Figure 6 · p. 40

K Prompt ablation details

For the ChEMBL assay-format subset, removing BAO definitions affected proprietary-LLM predictions more than reversing class order.

K Prompt ablation details

  • 10.8% of Gemini 3.6 Flash predictions changed when BAO definitions were removed, and this perturbation changed far more predictions than reversing class order.Figure 7 segments changed predictions by whether they moved to, from, or between labels different from the silver label.Source: Figure 7 · p. 41

L Resource Usage: Cost and Energy

The appendix reports resource use across the LLM auditing workflow, with API costs for proprietary models and energy use for self-hosted open-weight models.

L Resource Usage: Cost and Energy

  • The appendix presents compute cost, wall-clock throughput, and proxy energy consumption as resource measures for the auditing workflow.Input and completion token counts are also included in Table 18.Source: L Resource Usage: Cost and Energy · pp. 42–43, Table 18 · p. 42
  • Open-weight models ran locally through Ollama on a single cluster node, and their energy-per-record value is a proxy based on GPU-hours × PGPU.The proxy excludes CPU and cooling overhead, and the reported wall-clock time could be improved for open-source models.Source: L Resource Usage: Cost and Energy · p. 42
  • Proprietary models are reported by USD cost per completion, while open-weight models are reported by energy consumption per completion.Table 18 covers all five experimental conditions; proprietary-model energy figures are unavailable because providers do not expose token-level energy accounting.Source: L Resource Usage: Cost and Energy · p. 42, Table 18 · p. 42
Loading 2610.01616v1…