Source-linked AI summary
Quantitative Evidence Mining for Plausibility-Aware Biomedical AI
Negin Sadat Babaiha, Stefan Geissler, Marie-Christine Simon, Martin Hofmann-Apitius, Marc Jacobs
TL;DR
Biomedical AI increasingly reuses extracted claims, but relation-centric outputs often omit the quantitative details, context, uncertainty, and provenance needed to treat them as evidence. The paper proposes quantitative evidence mining and a plausibility-aware framework for representing claims as auditable evidence objects, illustrated with structured vaccine-efficacy information and validation checks. It concludes that this framework is a research agenda requiring prospective evaluation and expert review before high-stakes use.
Problem
Relation-centric extraction can omit doses, effect sizes, populations, comparators, conditions, uncertainty, and provenance needed to verify and reuse biomedical claims.
Method
The paper proposes quantitative evidence mining that structures values, units, measured entities, context, uncertainty, provenance, and plausibility information into evidence-aware knowledge graphs.
Results
The framework represents a vaccine-efficacy measurement with a 95.0% value and a 95% credible interval of 90.3%–97.6%, while checking grounding, scale consistency, and contextual completeness.
Takeaways & Limitations
Extracted biomedical claims should remain source-grounded, contextualized, uncertainty-preserving, and open to plausibility checks rather than being treated as final answers.
Takeaways & Limitations
The framework is a research agenda rather than an already validated system, and extracted evidence requires expert inspection and correction before influencing decisions.
Abstract
from arXiv · showhide
Biomedical artificial intelligence (AI) systems increasingly extract, organize, and reuse scientific claims from literature, clinical trials, and regulatory documents. But automatic extraction alone does not make a claim reliable evidence: a claim becomes useful only when it can be traced to its source, linked to the quantitative details that support it, and read within its biomedical context and uncertainty. This matters as large language models (LLMs) and increasingly autonomous systems drive evidence synthesis, knowledge graph (KG) construction, and decision support. Many text-mining and LLM pipelines remain relation-centric: they capture entities and relations such as Drug--TREATS--Disease, but drop the dose, effect size, population, comparator, uncertainty, and conditions under which a claim holds. Such relations can look actionable yet remain hard to verify, compare, or reuse. In this perspective, we argue for a shift toward quantitative evidence mining---extracting values, units, measured entities and properties, context, uncertainty, provenance, and plausibility as structured evidence units that populate evidence-aware KGs and can be checked for source grounding, unit consistency, completeness, and biological plausibility. We outline a framework for plausibility-aware AI that treats extracted claims not as final answers but as auditable evidence objects, making clear what was measured, how much it changed, in which setting, with what uncertainty, and from which source. The central risk is not only incorrect extraction, but claims that look like evidence while lacking the structure needed to trust them.
I. INTRODUCTION
Biomedical AI is shifting from simple relation extraction toward quantitative evidence mining because downstream reuse requires claims to retain measurements, context, uncertainty, and provenance. The paper frames this as a way to make extracted biomedical evidence more verifiable and reusable.
- I. INTRODUCTION: Downstream systems need claims that specify supporting evidence, conditions, and uncertainty rather than only detected relations.This is especially relevant for knowledge graphs, evidence synthesis, decision support, regulatory assessment, and model parameterization.
- I. INTRODUCTION: Quantitative evidence mining preserves values, units, measured entities, properties, conditions, comparators, uncertainty, provenance, and plausibility information.Its aim is to represent what was measured, how much it changed, in which setting, and from which source.
- I. INTRODUCTION: Relation-extraction systems provide an entity-and-relation backbone but do not fully capture how much was measured, under which conditions, or whether a value is plausible.This limitation motivates extending semantic relations with structured quantitative evidence.
- I. INTRODUCTION: Evidence-sentence extraction locates textual support, but useful vaccine claims also require dose schedule, population, endpoint, effect size, comparator, uncertainty, and time point.The BNT162b2 example links two doses given 21 days apart with efficacy against confirmed symptomatic COVID-19 in participants aged 16 years and older.
- I. INTRODUCTION: Numbers require contextual linking because identical-looking values can represent different effects, biological measurements, durations, or subgroup proportions.The paper illustrates this with 35%, 5.4 pg/mL, and 12 weeks, whose meanings depend on their measured entities and settings.
- I. INTRODUCTION: Intervention claims remain incomplete without quantitative and contextual conditions such as strain, formulation, dose, duration, population, and outcome definition.The paper contrasts generic treatment claims with evidence statements that preserve magnitude, time point, comparator, and additional study details.
B. RULE-BASED, HYBRID, AND MACHINE-LEARNING SYSTEMS FOR QUANTITY EXTRACTION
Quantity-extraction systems progressed from rule-based pattern matching to hybrid, machine-learning, and LLM-assisted approaches that link values with biomedical context. The section emphasizes that reusable quantitative evidence requires structured statements, validation, provenance, and plausibility checks.
- Rule-based systems: Rule-based systems detect visible quantity patterns such as values with units, percentages, ranges, p-values, and statistical expressions.They can extract examples including melting points and body-mass-index thresholds with high precision.
- Rule-based systems: Scientific language and document structure can separate a value from its unit, measured entity, role, or associated outcome.A dose may appear in Methods while its effect is discussed separately in Results, making sentence-level matching insufficient.
- Hybrid and machine-learning systems: Hybrid and machine-learning systems combine parsing, dictionaries, normalization, sequence labeling, and relation extraction to identify quantities and their properties.MeasEval systems used transformer models such as SciBERT with CRF layers to identify quantities, units, entities, modifiers, qualifiers, and relations.
- LLM-assisted systems: LLMs can connect numbers with flexible context and distinguish roles such as dose, time point, sample size, effect size, and uncertainty.They may extract treatment arm, dose, population, endpoint, effect size, comparator, confidence interval, and time point in one structured output.
- LLM-assisted systems: Because fluent LLM outputs may contain unsupported or misassigned values, extraction should propose evidence units that are checked against source passages, tables, or figures.Checks include value presence, unit correctness, entity linkage, comparator and time-point attachment, and uncertainty preservation.
- Evidence-aware workflows: Quinex and related workflows represent quantities as structured statements and validate numerical claims against contextual or external evidence.Quinex normalizes values and units, then extracts measured properties, entities, and scope qualifiers; related work combines extraction with retrieval-augmented validation.
IV. BIOMEDICAL QUANTITATIVE EVIDENCE UNITS
The biomedical quantitative evidence unit is proposed as a structured container linking a scientific claim to its quantitative support, biomedical context, uncertainty, provenance, and plausibility assessment. An illustrative vaccine-efficacy unit shows how these fields preserve both the reported result and the conditions needed to interpret it.
- Core representation: A biomedical quantitative evidence unit links a claim to its value, unit or scale, measured entity and property, context, uncertainty, provenance, and source span.Extraction confidence, multidimensional validation results, and expert-review status remain separate fields rather than being collapsed into one plausibility label.
- Illustrative evidence unit: The illustrative claim reports 95.0% vaccine efficacy for BNT162b2 relative to placebo.The underlying observations were 8 cases in the BNT162b2 group and 162 cases in the placebo group.
- Illustrative evidence unit: The example specifies two 30-μg intramuscular doses 21 days apart, participants aged ≥16 years without previous SARS-CoV-2 infection, and confirmed cases at least seven days after dose 2.These fields connect the efficacy estimate to its intervention, population, and temporal context.
- Uncertainty and assessment: The efficacy estimate is accompanied by a 95% credible interval of 90.3%–97.6% and provenance to an FDA briefing document, Table 6, and its accompanying efficacy paragraph.The example also records source grounding, unit/scale consistency, contextual completeness, statistical coherence, and biological feasibility as assessment dimensions.
- Completeness and governance: Missing comparators, confidence intervals, or provenance links should be represented explicitly because incompleteness lowers downstream confidence without automatically invalidating the evidence unit.The framework calls for flagging missing fields rather than silently filling gaps or discarding the claim.
- Plausibility assessment: Plausibility-aware AI interrogates whether entities, values, populations, time windows, trial arms, intervals, and source links are correctly connected and biologically consistent.The paper positions this process as a shift from extracting statements alone toward quantitative, context-preserving, plausibility-checked evidence units.
B. NON-EXCLUSIVE FACETS OF BIOMEDICAL QUANTITATIVE EVIDENCE
Biomedical quantitative evidence is organized along complementary, non-exclusive axes that capture evidence role, biomedical domain, and quantitative form, together with the context needed to interpret each measurement. Missing fields should be recorded explicitly rather than silently omitted or inferred.
- Three complementary axes classify quantities by evidence role, biomedical domain, and quantitative form rather than assigning one exclusive type.
- Study-design quantities require the study design, sample size, follow-up or study duration, trial arm, analysis population, and denominator.
- Population quantities require the exact cohort or subgroup, denominator, inclusion criteria, demographic definition, and disease stage.
- Intervention quantities require intervention identity, dose unit, route, frequency, duration, exposure window, treatment arm, and target population or species.
- Quantitative evidence also includes outcomes, molecular and clinical measurements, effect estimates, uncertainty, and thresholds with their relevant contextual requirements.Examples include cytokine concentration, blood pressure, allele frequency, odds ratio, confidence interval, p-value, and decision boundary.
- Absent contextual fields should be recorded as “not reported in the source” rather than silently omitted or inferred.
V. PLAUSIBILITY-AWARE FRAMEWORK
The proposed plausibility-aware framework is a flexible reference architecture for transforming biomedical sources into structured, traceable, and plausibility-checked evidence units. Its workflow preserves end-to-end provenance and flags questionable evidence for review rather than automatically discarding it.
- The framework transforms raw biomedical sources into structured, traceable, and plausibility-checked evidence units.
- Eight stages span corpus selection, entity recognition, quantity extraction, context linking, normalization, assembly, plausibility assessment, and export and governance.JSON is one structured export format; traceability preserves source location, evidence span, document and software versions, preprocessing, and the audit trail.
A. CORPUS GENERATION AND SOURCE SELECTION
Corpus selection defines which biomedical sources are mined and should be guided by the target evidence question. The proposed source scope spans literature, trials, reviews, guidelines, regulatory documents, and supplementary files.
- Corpus selection may include PubMed abstracts, PMC full texts, clinical trial records, systematic reviews, preprints, guidelines, regulatory documents, or supplementary files.
- The target biomedical question should guide source selection.Examples include vaccine response, ageing biomarkers, immune-mediated disease, post-viral syndromes, drug efficacy, and neurodegenerative comorbidity.
- Different evidence questions may therefore require different combinations of biomedical source types.
B. BIOMEDICAL ENTITY AND CONCEPT RECOGNITION
Biomedical entity and concept recognition identifies relevant biomedical entities and maps them to controlled vocabularies. The recognized concepts support structured evidence mining across molecular, clinical, and experimental domains.
- The entity-recognition layer identifies genes, proteins, diseases, chemicals, drugs, phenotypes, biomarkers, pathways, variants, cell types, tissues, organisms, interventions, and clinical outcomes.
- Recognized entities are mapped to controlled vocabularies.Listed resources include PubTator, UMLS, MeSH, MONDO, HPO, ChEBI, DrugBank, and Gene Ontology.
- Entity normalization provides the concept layer needed for downstream quantitative evidence extraction.
C. QUANTITY AND UNIT EXTRACTION
The framework extracts numerical values and units, links them to biomedical context, normalizes representations, and assembles structured evidence units for plausibility-aware downstream use.
- The quantity layer extracts values, ranges, percentages, statistical measures, sample sizes, durations, concentrations, doses, and thresholds with their units.
- Context links each quantity to the measured entity and property, population, biological material, assay, disease stage, comparator, time point, endpoint, and condition.A numerical value without context can be misleading or unusable.
- Normalization harmonizes entity names, units, quantity types, statistical measures, variant identifiers, genome builds, and effect-allele orientation.
- Structured evidence units assemble extracted components for inspection, comparison, and checking in dashboards, reviews, evidence tables, RAG systems, and decision-support tools.They should not yet be treated as trustworthy evidence until plausibility and provenance checks are applied.
- Plausibility assessment reports dimension-specific results and review flags for grounding, confidence, consistency, completeness, assay compatibility, statistical coherence, and biological feasibility.Illustrative checks verify source support, compare compatible contexts, and flag missing information without automatically discarding the evidence unit.
- Plausibility supports screening and prioritization rather than establishing scientific truth, because extracted values may come from biased or underpowered studies.Conversely, an apparent outlier may be a genuine novel finding, so uncertainty and expert judgment must be preserved.
H. PROVENANCE, MODALITY, AND PROCESS
The framework preserves how quantitative evidence is produced and where it appears, enabling traceable comparison and reuse across clinical, biomarker, comorbidity, KG, pharmacovigilance, and regulatory applications.
- Provenance records the source document, location, extraction model, prompt, model version, execution date, segmentation strategy, and post-processing steps.LLM outputs can vary with model version, prompt design, chunking, and decoding parameters.
- Modality records whether a value came from narrative text, a table cell, figure panel, axis label, caption, graphical abstract, or supplementary material.
- Quantitative evidence mining supports comparison, validation, and reuse across heterogeneous documents, populations, measurement platforms, and biological scales.
- Clinical-trial evidence units preserve eligibility criteria, dose and schedule, duration, comparator, endpoints, effect estimates, confidence intervals, adverse events, and follow-up periods.
- Biomarker evidence requires concentrations, units, sample type, patient group, comparator, endpoint, and confidence interval to interpret associations and detect mismatches or missing information.
- Comorbidity evidence preserves cohort size, severity, follow-up, age, vaccination status, outcome definition, comparator, and effect size for cross-study comparison.This avoids reducing heterogeneous findings to a generic disease–disease link.
- Evidence-aware KGs connect claims to values, units, context, uncertainty, provenance, validation dimensions, and expert-review status rather than only entity relations.Quantitative units complement source-level evidence weighting by structuring and checking numerical evidence attached to each relation.
- Pharmacovigilance and regulatory assessment require quantitative details including dose, exposure, adverse-event frequency, risk estimates, uncertainty, population, and reporting context.
E. FROM QUANTITATIVE EVIDENCE UNITS TO SIMULATION-READY PARAMETERS
Quantitative evidence units can make simulation parameters traceable and reproducible while supporting domain-specific plausibility checks and multi-level evaluation of complete evidence units.
- Simulation-ready parameters: Agent-based and immune-system models require mechanistic structure together with parameter values, ranges, assumptions, and calibration targets.Examples include viral load, infection rate, cytokine concentration, antibody titre, immune-cell count, dose, and time-dependent changes.
- Simulation-ready parameters: Evidence units preserve what was measured, its unit and biological material, time point, population or experimental system, and uncertainty for candidate model parameters.
- Simulation-ready parameters: Mapping values to model variables, harmonizing units, representing ranges, and labeling calibration or validation roles makes source-to-model assumptions more transparent and reproducible.
- Cross-cutting population context: Population composition, dose, comparator, outcome, and uncertainty can expose when evidence is under-validated for another population rather than silently generalized.
- Domain-specific evidence: Quantitative genetics requires variant, genome-build, allele, phenotype, effect, uncertainty, sample, ancestry, sex, model, covariate, and replication context.
- Domain-specific evidence: Genetic evidence must preserve tissue, population, ancestry, allele, effect-size, and uncertainty information because predictive performance and clinical transferability can differ across populations.
- Domain-specific evidence: Genetic studies require scalable handling of supplementary tables whose headers, allele conventions, effect scales, units, and significance thresholds vary across studies.
- Domain-specific evidence: Plausibility checks can test allele-frequency bounds, beta–standard-error–p-value consistency, odds-ratio positivity, and effect-direction harmonization.
VIII. DISCUSSION AND OUTLOOK: BEYOND BIOMEDICAL EVIDENCE
The paper extends quantitative evidence mining beyond biomedicine to public health, technical assessment, and security, while emphasizing traceability, uncertainty, and human oversight.
- Across domains, extracted claims should remain measurable, traceable, contextualized, and open to plausibility checks before reuse.
- Public-health claims require exposure level, threshold, location, population, time period, endpoint, effect estimate, uncertainty, and provenance.
- Technical assessment cannot meaningfully interpret a reported accuracy of 95% without the dataset, target population, operating conditions, baseline, uncertainty, and testing protocol.
- Security and infrastructure analysis can structure severity scores, affected versions, incident frequency, time windows, confidence, geographic scope, and mitigation effectiveness.
- In sensitive domains, quantitative evidence mining is framed for lawful analysis, documentation, and risk assessment rather than autonomous decision-making.Its value is making numerical sources, measurements, reliability, and missing context transparent.
C. LIMITATIONS AND RESPONSIBLE USE
The framework positions quantitative evidence mining as an auditable aid to evidence organization and decision support, while retaining human judgment and acknowledging that it remains conceptual and requires prospective validation.
- Responsible use: Quantitative evidence mining should organize evidence and support decisions, not autonomously judge scientific truth.Extraction, unit conversion, context attachment, source linking, and statistical interpretation can alter a claim’s meaning, especially in safety-critical settings.
- Responsible use: Genetic associations must not be treated as patient-specific risk estimates or clinical genetic advice without governed individual-level data, relevant-population validation, specialist interpretation, and safeguards.
- Responsible use: Extracted evidence should retain source links, context, provenance, uncertainty, validation results, and review status before downstream AI reuse.Without these elements, errors can pass unnoticed into later analyses or decisions.
- Responsible use: The framework makes claims easier to measure, compare, trace, and audit while keeping human judgment in the workflow; it is not a complete solution to evidence assessment.In biomedicine, this boundary is connected to scientific reliability and patient safety.
- Scope and validation: The framework remains conceptual and unimplemented as a complete end-to-end system, requiring benchmarks that preserve quantitative evidence across text, tables, figures, and supplementary materials.Plausibility rules may not cover every domain, and evidence from different assays, populations, endpoints, or study designs may not be directly comparable.
- Scope and validation: Future work should develop representative annotations and benchmarks, evaluate extraction components, and assess multidimensional validation for calibration, genuine-error sensitivity, and rejection of valid outputs.