PaperlayerView exact revision on arXiv

Source-linked AI summary

PlantMarkerBench: A Multi-Species Benchmark for Evidence-Grounded Plant Marker Reasoning

Sajib Acharjee Dip, Song Li, Liqing Zhang

arXiv:2605.10032v2cs.CL

TL;DR

PlantMarkerBench addresses the lack of explicit, literature-grounded evidence interpretation in plant marker resources. It constructs and evaluates a multi-species benchmark from full-text papers, finding that models handle direct expression evidence better than functional, indirect, and weak-support evidence. The benchmark provides a challenging framework for evidence attribution and trustworthy scientific information extraction.

  • Problem

    Existing plant marker resources and evaluations do not adequately test whether literature supports a gene–cell-type association or what evidence type it represents.

  • Method

    The authors construct PlantMarkerBench with a modular pipeline for retrieval, species-aware grounding, evidence grading, aggregation, and human review across four plant species.

  • Results

    Frontier models perform relatively well on direct expression evidence but substantially worse on functional, indirect, and weak-support evidence, with evidence-type confusion dominant.

  • Takeaways & Limitations

    PlantMarkerBench offers a reproducible evaluation framework for fine-grained, literature-grounded plant marker evidence attribution.

  • Takeaways & Limitations

    The dataset may contain residual labeling noise or biologically ambiguous cases and currently focuses mainly on root and developmental cell types from four species.

Abstract

from arXiv · show

Cell-type-specific marker genes are fundamental to plant biology, yet existing resources primarily rely on curated databases or high-throughput studies without explicitly modeling the supporting evidence found in scientific literature. We introduce PlantMarkerBench, a multi-species benchmark for evaluating literature-grounded plant marker evidence interpretation from full-text biological papers. PlantMarkerBench is constructed using a modular curation pipeline integrating large-scale literature retrieval, hybrid search, species-aware biological grounding, structured evidence extraction, and targeted human review. The benchmark spans four plant species -- Arabidopsis, maize, rice, and tomato -- and contains 5,550 sentence-level evidence instances annotated for marker-evidence validity, evidence type, and support strength. We define two benchmark tasks: determining whether a candidate sentence provides valid marker evidence for a gene-cell-type pair, and classifying the evidence into expression, localization, function, indirect, or negative categories. We benchmark diverse open-weight and closed-source language models across species and prompting strategies. Although frontier models achieve relatively strong performance on direct expression evidence, performance drops substantially on functional, indirect, and weak-support evidence, with evidence-type confusion emerging as a dominant failure mode. Open-weight models additionally exhibit elevated false-positive rates under ambiguous biological contexts. PlantMarkerBench provides a challenging and reproducible evaluation framework for literature-grounded biological evidence attribution and supports future research on trustworthy scientific information extraction and AI-assisted plant biology.

1 Introduction

PlantMarkerBench addresses the difficulty of interpreting heterogeneous, context-dependent plant marker evidence from literature. It introduces a multi-species benchmark and evaluates models on validity and evidence-type reasoning.

  • Motivation: Reliable literature-based marker identification is difficult because evidence is heterogeneous and gene–cell-type co-occurrence can be misleading.Interpretation may require distinguishing direct from indirect evidence, resolving species and gene-alias ambiguity, and rejecting unsupported statements.
  • Motivation: Existing plant-biology evaluations largely emphasize entity extraction, marker lookup, or expression-based annotation rather than evidence attribution.They do not directly test whether models validate gene–cell-type associations, classify evidence types, and reject misleading biological context.
  • Benchmark: PlantMarkerBench contains 5,550 sentence-level instances across Arabidopsis, maize, rice, and tomato, annotated for validity, evidence type, and support strength.The benchmark covers 1,036 unique genes and 127 observed cell types, with categories including expression, localization, functional, indirect, and negative evidence.
  • Benchmark: The benchmark is built with a reproducible modular pipeline integrating full-text retrieval, biological grounding, hybrid retrieval, evidence grading, aggregation, and human review.Its core evaluations are marker-evidence validity prediction and evidence-type classification.
  • Results: Frontier models perform relatively well on direct expression evidence but drop substantially on functional, indirect, and weak-support evidence.Evidence-type confusion is a dominant failure mode, alongside hard negatives involving aliases, wrong genes, and cell-type ambiguity.

2 Dataset Overview

PlantMarkerBench is a sentence-level, multi-species dataset designed to test marker-evidence reasoning rather than simple marker lookup. It combines diverse evidence regimes, hard negatives, and auditable curation outputs.

  • Benchmark design: PlantMarkerBench requires models to determine whether an evidence window supports a gene as a valid marker and to classify its evidence type.The benchmark targets literature-grounded reasoning over gene–cell-type associations.
  • Dataset scale: The release contains 5,550 sentence-level instances across four plant species, covering 1,036 genes and 127 observed cell types.For controlled evaluation, the authors construct balanced pilot subsets with 2,400 manually reviewed instances.
  • Evidence composition: Unlike positive-association resources, the dataset includes literature noise, weak grounding, indirect associations, and hard negatives.Roughly two-thirds of instances are invalid, weak, indirect, or ambiguous, and the benchmark spans expression, localization, functional, indirect, and negative evidence.
  • Curation pipeline: The curation pipeline uses specialized retrieval, grounding, evidence-grading, and aggregation components whose structured outputs support reproducibility and targeted review.The pipeline proceeds through literature filtering, species assignment, biological grounding, hybrid retrieval, candidate generation, and evidence aggregation.
  • Dataset overview: The benchmark overview organizes dataset scale across four species, evidence-type composition, and long-tail support-strength distributions.The support-strength distribution indicates that most literature evidence is weakly supported, reflecting scientific ambiguity.

3 PlantMarkerBench Construction and Task Formulation

PlantMarkerBench is constructed through species-aware literature curation, retrieval, grounding, and structured evidence review. It supports fine-grained evaluation of marker validity and evidence type across multispecies plant literature.

  • Literature collection: Candidate papers are collected from PubMed and PMC, then filtered to retain evidence-rich sections and exclude insufficient full-text records.The cleaned corpus retains abstracts, introductions, results, discussions, and conclusions while excluding methods, references, acknowledgments, and supplements.
  • Species and gene grounding: Articles receive a primary species assignment from title, abstract, and early-text signals, while unreliable species assignments are excluded to reduce contamination.Species-specific gene matchers map identifiers, symbols, and literature aliases to canonical genes for candidate grounding.
  • Dataset composition: PlantMarkerBench spans four plant species and contains 5,550 sentence-level instances covering 1,036 genes and 127 observed cell types.Its five evidence regimes are expression, localization, functional, indirect, and negative/noise, with support-strength annotations.
  • Cell-type grounding: Species-specific controlled vocabularies provide terminology for retrieving and grounding plant cell types across tissues and developmental contexts.The vocabularies include root, vascular, leaf, meristematic, reproductive, and species-specific tissue cell types.
  • Candidate generation: Retrieval combines keyword matching, BM25, dense embeddings, and hybrid fusion over sentence-centered evidence windows.Grounding identifies genes and cell types, deduplicates candidate pairs, and preserves retrieval provenance such as aliases, sections, scores, and local context.
  • Evidence labeling: An LLM-based grader records evidence validity, type, support strength, confidence, and rationale in structured outputs.The labeling scheme is conservative: co-occurrence, homology-only statements, and nonspecific developmental evidence are not direct marker evidence.
  • Aggregation and review: Judged evidence is aggregated into gene–cell-type evidence graphs, producing strict markers, expanded candidates, functional regulators, and indirect associations with provenance and confidence statistics.Two reviewers inspect difficult cases and remove malformed or unsupported instances before release.
  • Tasks and evaluation splits: The benchmark evaluates marker-evidence validity prediction and five-way evidence-type classification, while the released pipeline also supports downstream curation tasks.A balanced Arabidopsis pilot split contains 600 examples with equal valid and invalid instances for stable precision, recall, and F1 comparison.

4 Benchmarking Results

Benchmark results show that models handle direct expression evidence better than fine-grained evidence attribution. Performance varies by species and prompting strategy, with evidence-type confusion, localization, and indirect evidence remaining difficult.

  • Overall performance: Models achieve moderate binary validity F1 across species but often fail to identify the underlying biological evidence type.This gap indicates that recognizing relevant context does not ensure accurate gene–cell-type grounding or mechanistic classification.
  • Model trends: Qwen2.5-32B-Instruct achieves the strongest open-model validity F1 on both evaluated species, yet evidence-type macro-F1 remains substantially lower.Smaller models often over-predict positive evidence or collapse on localization and indirect categories.
  • Evidence-type difficulty: Expression evidence is consistently easier than localization or indirect evidence, with localization and indirect evidence frequently below 0.4 macro-F1.Functional and weak-support evidence also produce substantial performance drops.
  • Cross-species evaluation: Cross-species performance varies substantially: rice is strongest overall, while maize and tomato are more challenging, especially for localization and indirect evidence.Species-specific nomenclature, synonym ambiguity, and heterogeneous literature conventions limit transfer across biological domains.
  • Prompting effects: Few-shot prompting improves binary validity F1 but does not consistently improve fine-grained evidence attribution.Direct prompting gives GPT-5.4 the strongest average evidence macro-F1, while localization and indirect evidence remain difficult across prompt settings.
  • Error analysis: Evidence-type mismatch is the dominant error mode, and open-weight models show substantially higher false-positive rates.The error taxonomy distinguishes these failures from correct predictions and other error categories.
  • Biological failure modes: Hard examples require distinguishing expression, localization, and functional support while resolving aliases, gene-family attribution, indirect associations, and cell-type granularity.These cases test literature-grounded attribution rather than simple entity extraction or keyword matching.

5 Conclusion

PlantMarkerBench provides a multispecies framework for evaluating literature-grounded plant marker evidence attribution. Results show that current LLMs handle direct expression evidence better than functional, indirect, and weak-support evidence, with evidence-type confusion remaining dominant.

  • Conclusion: PlantMarkerBench evaluates diverse gene–cell-type evidence regimes across four plant species using full-text biological literature.The benchmark is paired with a reproducible curation pipeline integrating retrieval, grounding, grading, aggregation, and human review.
  • Conclusion: Current LLMs still struggle with fine-grained evidence attribution despite reasonable performance on direct expression evidence.Performance drops substantially on functional, indirect, and weak-support evidence, where evidence-type confusion is dominant.

A Extended Dataset Construction Details

PlantMarkerBench combines species-aware literature retrieval, biological grounding, structured evidence annotation, aggregation, and review into an auditable multi-species benchmark. Its sentence-level records preserve evidence context, labels, provenance, and realistic biological diversity.

  • Corpus construction: The pipeline retrieves full-text plant biology papers with species-aware queries, retains relevant sections, and filters insufficient or ambiguous content.Species assignment uses weighted title, abstract, and body mentions, while mixed-species papers are conservatively filtered.
  • Biological grounding: Species-specific gene normalization and curated cell-type vocabularies reduce spurious matches across Arabidopsis, rice, maize, and tomato.The resources include identifiers, aliases, synonyms, canonical cell-type names, abbreviations, and related variants.
  • Candidate retrieval: Candidate evidence windows combine BM25, dense embeddings, keyword matching, and hybrid fusion around co-occurring gene and cell-type mentions.Each candidate stores retrieval provenance, evidence sentences, local context, normalization outputs, and cell-type metadata.
  • Evidence annotation: Candidates receive validity, evidence-type, support-strength, and rationale annotations through structured grading followed by targeted human review.The five categories are expression, localization, function, indirect, and noise; review targets biological edge cases such as species mismatch and gene-family ambiguity.
  • Marker aggregation: Strict markers retain direct, expression, or localization evidence above threshold, whereas expanded candidates also include functional and high-confidence indirect evidence.Gene–cell-type aggregation incorporates evidence-type weights, confidence, section reliability, supporting-paper counts, and retrieval consensus.
  • Release and coverage: The released JSONL benchmark preserves grounded pairs, evidence context, structured labels, retrieval provenance, and audit artifacts for reproducibility and extension.It includes full corpora, balanced pilot splits, species statistics, prediction files, prompt templates, evaluation scripts, and error-analysis utilities.
  • Release and coverage: The benchmark spans diverse plant cell types and full-text evidence regimes, with common tissues frequent and specialized types forming a pronounced long tail.This distribution reflects uneven biological literature coverage and tests reasoning over sparse, heterogeneous evidence.

F Prompt Templates and Evaluation Protocols

The evaluation protocol standardizes structured model outputs across prompt modes, evidence labels, metrics, and reproducibility settings. It compares direct classification, reasoning, conservative filtering, and few-shot prompting under fixed evaluation procedures.

  • F.1 Evaluation Prompt Modes: Four prompt modes—direct, structured, conservative, and few-shot—use identical biological input fields and require JSON predictions for validity, type, strength, and rationale.Direct marker evidence is normalized to expression during evaluation.
  • F.2.2 Structured Reasoning Prompt: The structured prompt checks gene grounding, cell-type grounding, relation type, and the final valid-or-invalid marker decision.It instructs models to remain conservative without rejecting valid evidence merely because “marker” is absent.
  • F.2.3 Conservative Prompt: The conservative prompt rejects wrong-gene, wrong-species, homology-only, pathway, citation-noise, and organ-level evidence lacking cell-type grounding.Its evidence labels remain expression, localization, function, indirect, and noise.
  • F.2.4 Few-Shot Prompt: Few-shot prompting supplies examples linking expression, localization, function, indirect evidence, and noise to corresponding validity and type labels.Examples include root-hair expression, endodermal reporter localization, guard-cell developmental effects, and pathway- or homology-mediated associations.
  • F.3 Output Schema and Normalization: Prediction normalization maps evidence types to five labels, direct marker predictions to expression, support strength to four levels, and failed parses conservatively.The expected output is a JSON object containing validity, evidence type, support strength, and a short rationale.
  • F.4 Evaluation Metrics: Validity classification evaluates supported-versus-unsupported evidence, while evidence-type classification evaluates five biological categories using accuracy and macro-F1.Macro-F1 is emphasized because category frequencies are imbalanced and some evidence types are sparse.
  • F.4 Evaluation Metrics: Pilot evaluations use fixed 600-example species-level splits and deterministic normalized labels.The protocol also reports bootstrap confidence intervals for representative validity-F1 evaluations.
  • F.8 Reproducibility Settings: Runs preserve configurations, raw outputs, normalized predictions, and metrics, while comparable prompt families support cross-model evaluation and separate ablations.The main leaderboard uses default stable settings; additional closed-source prompt ablations are reported separately when smaller models show instability.

G.1 Hard-Subset and Support-Strength Analysis

Hard-subset analyses show that PlantMarkerBench distinguishes explicit expression reasoning from more difficult indirect and weak-support interpretation. These patterns persist across models, species, and prompt settings.

  • Hard subsets: Curated hard subsets isolate expression, localization, functional, indirect, negative, and strong-, medium-, and weak-support evidence cases.The subsets are designed to expose reasoning failures beyond aggregate benchmark scores.
  • Evidence-type performance: Expression evidence is substantially easier than indirect evidence across nearly all models, indicating reliance on explicit lexical grounding cues.Weak-support examples remain difficult even for stronger models.
  • Cross-species evaluation: OpenAI results across all four species and prompt modes likewise show consistent advantages on expression evidence, while indirect and weak-support cases remain difficult.The comparison is reported as evidence-type classification accuracy for each corresponding subset.
  • Human review: Independent review by two computational-biology reviewers focused on ambiguous grounding, indirect evidence, gene-family ambiguity, and species mismatch, resolving disagreements through adjudication.This review targeted difficult benchmark instances rather than replacing the benchmark labels wholesale.

H.1 Per-Model Error Breakdown

Error analyses reveal distinct model and species failure patterns, while prompting changes the balance between false positives, false negatives, and evidence-type mismatches. Dataset composition also constrains interpretation of some categories.

  • Per-model error breakdown: Closed models generally achieve higher correct-prediction rates and lower false-positive rates, whereas several open-weight models show evidence-type confusion or over-prediction.The error taxonomy separates behaviors that aggregate F1 can obscure.
  • Species-specific trends: Maize has the largest false-negative rates, Arabidopsis errors more often reflect evidence-type confusion, and rice shows the most stable behavior across settings.These dominant trends vary substantially across species.
  • Prompting effects: Conservative prompting reduces false positives but increases false negatives, while few-shot prompting improves grounding accuracy and reduces evidence-type mismatch.The prompting comparison concerns average error distributions across the four species.
  • Label design: The negative/noise label aggregates multiple failure modes because they commonly co-occur during realistic literature retrieval and test grounding robustness under ambiguity.The retained subtypes are documented for evidence adjudication.
  • Low-resource evidence categories: Localization-specific metrics require caution because localization evidence is underrepresented, particularly for maize and tomato.The category remains separate because localization is biologically distinct for marker interpretation.
  • Borderline evidence cases: Developmental perturbation studies can imply cell-type specificity without directly demonstrating marker enrichment, leaving boundaries among indirect, functional, and expression evidence ambiguous.The benchmark uses conservative adjudication and rationale metadata to support future refinement.
  • Retrieval analysis: Retrieval comparisons include BM25, embedding-based, keyword, and hybrid methods, with candidate quality analyzed as retrieval depth changes.These analyses complement the model error breakdown by examining candidate generation.

I.3 Gene Matcher Quality Analysis

Species-aware gene normalization substantially improves candidate quality, with locus-aware matching producing far more candidates and valid evidence instances in preliminary maize experiments.

  • Species-aware gene normalization substantially improves candidate quality.
  • 1027 candidates and 341 valid evidence instances resulted from locus-aware matching, compared with 14 candidates and 4 valid evidence instances using a weak matcher.
  • The maize comparison demonstrates the importance of species-specific biological grounding for candidate generation.

J Reproducibility and Resource Release

The release provides benchmark data, evaluation resources, and intermediate artifacts for reproducible, auditable multi-species evidence reasoning experiments.

  • The complete PlantMarkerBench benchmark, dataset artifacts, and evaluation code are released through an anonymous Zenodo archive.
  • Released resources include full sentence-level datasets, balanced pilot subsets, benchmark statistics, retrieval code, evaluation scripts, and analysis utilities.
  • Each benchmark instance records species and paper identifiers, gene and cell-type information, evidence context, structured labels, support strength, and reasoning traces.
  • Intermediate retrieval outputs, candidate windows, judged evidence files, and species-specific statistics enable reconstruction and auditing of the curation pipeline.
  • The release contains approximately 24 MB of data and code artifacts spanning four species and more than 5,500 evidence instances.
  • Pilot splits contain 600 manually reviewed instances per species and are intended for controlled benchmarking rather than distribution-matched sampling of the full literature corpus.

K Limitations

The benchmark is constrained by labeling ambiguity, uneven evidence coverage, limited biological and document scope, possible pretraining overlap, and incomplete baseline coverage.

  • LLM-assisted evidence grading may leave residual labeling noise or biologically ambiguous cases despite human review and multi-stage filtering.
  • The release emphasizes root and developmental cell types from four species and does not cover the full diversity of plant tissues, conditions, stages, or modalities.
  • Localization evidence is comparatively sparse in some species, while weakly supported and indirect evidence is abundant and increases task difficulty.
  • Evaluation primarily targets sentence-level reasoning rather than full document understanding, leaving figure interpretation, supplementary materials, and multi-hop reasoning for future extensions.
  • Some benchmark papers or marker associations may overlap with language-model pretraining corpora because the benchmark uses publicly available literature.
  • The current release focuses on language-model evaluation and lacks dedicated supervised encoder and retrieval-only baselines.
Loading 2605.10032v2…