PaperlayerView exact revision on arXiv

Source-linked AI summary

From archive to resource: bringing legacy data back to life at scale with large language models

C. H. Vieira-Vieira, I. Schneider Chernov, A. Serra, M. Correa Marrero, P. L. Bhavani Mohan, G. Duchateau-Nguyen, T. Y. Doktorova, A. Kreuchwig

arXiv:2610.32616v1cs.IR

TL;DR

Legacy preclinical reports contain contextual descriptors that are often missing from structured standards, complicating systematic reuse of historical data. Two independent LLM pipelines recovered metadata with high agreement and traceable evidence; fasting status was associated with clinical-chemistry differences.

  • Problem

    Contextual factors that may affect clinical-pathology variability are often absent from structured preclinical standards and are difficult to identify across heterogeneous reports.

    Source: Introduction · p. 2

  • Method

    Two independent LLM-based pipelines extracted contextual metadata from legacy study reports and linked extracted values to their verbatim source text, page, and report section.

    Source: Introduction · p. 2

  • Results

    Both pipelines achieved high agreement with expert curation; extraction took 1.6 minutes per study, and fasting status was associated with clinical-chemistry differences.

    Source: Bayer results · p. 7, Discussion · p. 9

  • Takeaways & Limitations

    Recovered metadata strengthens the evidence base for translational and regulatory decisions and supports efforts to reduce animal use.

    Source: Methodological lessons, limitations,… · p. 10

    Defining metadata schemas and extraction instructions for new domains still requires manual expert input.

    Source: Methodological lessons, limitations,… · p. 10

Abstract

from arXiv · show

Pharmaceutical companies maintain vast archives of legacy preclinical study reports that contain study-context details, such as fasting, anesthesia, and laboratory instrumentation, that are rarely captured in structured data formats, yet they can correlate with clinical pathology parameters values and limit the reuse of historical data. This work demonstrates practical, scalable solutions to recover such descriptors from heterogeneous report texts using large language model (LLM)-based extraction pipelines developed independently by two industry teams. The pipelines identify and normalize metadata fields and return verbatim evidence with page and section reference, hasting human verification and iterative refinement. Evaluation of over 400 documents against expert-curated reference values showed high agreement across a broad set of metadata fields. Performance was strongest for consistently reported items and lower for descriptors subject to linguistic or report structure variability. Extraction time has shrunk from 25 minutes in the manual benchmark to 1.6 minutes. As a concrete biological demonstration, we show that fasting status extracted from report text was associated with species- and analyte-specific shifts in clinical chemistry parameters, illustrating how recovered metadata can improve selection of historical controls. Key strengths of the approach are scalability, traceability, and the ability to recover information missing from current industry standards. Main limitations are dependence on source-report quality and initial expert effort to define and refine schemas. These results establish how LLMs can convert heterogeneous, unstructured legacy reports into evidence-grounded, reusable and structured metadata, leading to better experimental comparability, and thus enabling reliable data pooling or informed virtual control selection.

Introduction

Legacy preclinical reports contain contextual metadata that is often absent from structured standards, making systematic reuse of historical data challenging. The introduction presents two independent LLM pipelines to recover these descriptors and reports high agreement with expert curation across both teams’ reports.

Introduction

  • Legacy reports contain factors that may confound cross-study comparisons, but many are not consistently represented in structured standards such as SEND.The authors frame this unstructured information as difficult to identify systematically and note that manual recovery does not scale to thousands of reports.Source: Introduction · p. 2
  • Both pipelines matched expert curation for most study metadata across 244 Bayer and 202 Roche reports; extracted fasting status was associated with clinical-chemistry changes.The authors identify fasting status as a candidate matching criterion for virtual control groups.Source: Introduction · p. 3

System Architecture – PRINCE (Bayer)

  • Bayer’s PRINCE pipeline uses target-study-filtered retrieval and schema-constrained LLM extraction, then reconciles extracted values with existing database entries using field-dependent rules.It accepts some absent fields, uses existing curated values for some fields, and prefers extraction for others.Source: System Architecture – PRINCE (Bayer) · pp. 3–4

System architecture - Controlled Generation (Roche)

  • Roche’s workflow identifies report cores, parses reports, and applies GPT-4.1 with field-specific schema instructions; optional section selection was not used in this study.The fields were selected based on SEND, preclinical-toxicity relevance, and VICT3R requirements, with extensions for cross-study comparison and VCG development.Source: System architecture - Controlled… · p. 4

Iterative human curation of reference values with LLM assistance

  • Both workflows return extracted values with verbatim source text, page, and report-section references, supporting human review and error diagnosis.Curators distinguished incorrect source selection from incorrect interpretation of a correct source, refined instructions with domain experts, and repeated curation until test-set entities were validated.Source: Iterative human curation of reference… · p. 4

Automatic evaluation with LLM-as-a-Judge

The paper uses an LLM judge to compare extracted values with curated references and fits species-specific mixed-effects models to examine fasting-associated analyte shifts. The models adjust for sex, specimen type, and study-level variation, while facility-related confounding limits interpretation.

Automatic evaluation with LLM-as-a-Judge

  • An LLM judge classified extracted values as correct, partially correct, incorrect, or not extracted, using curated values and field descriptions as reference.The resulting categories were mapped to a common confusion matrix for precision, recall, and F1 calculations.Source: Automatic evaluation with LLM-as-a-Judge · p. 5
  • A curated value of “Not Available” meant the report lacked the entity, so any extraction for that field counted as a false positive.The evaluation also distinguished cases with no curation and no extraction from cases with no curation but an extracted value.Source: Automatic evaluation with LLM-as-a-Judge · p. 5

Linear mixed-effects model

  • For each of 16 species–analyte combinations, separate log-scale models estimated fasting-associated shifts while adjusting for sex, specimen type, and study-level variation.The models used main effects without interactions; fasting p-values were corrected across 16 tests with the Benjamini-Hochberg procedure.Source: Linear mixed-effects model · pp. 5–6
  • A pooled fasting-by-species interaction served only as a qualitative indicator because unfasted studies were limited; facility was nearly collinear with fasting and was absorbed by the study random intercept.The authors caution that a non-significant interaction does not establish equivalence between species.Source: Linear mixed-effects model · p. 6

Results

The Bayer pipeline extracted metadata rapidly and with high agreement, including fields missing from PRINCE; the Roche pipeline was strongest on consistently reported fields and weaker where language or domain meaning varied.

Bayer results

  • 1.6 minutes per study (5.5 hours for 44 fields) versus 25 minutes manually; across 44 fields, mean recall, precision, and F1 were 95.8%, 96.2%, and 95.6%.Manual curation of 44 fields across 205 studies took approximately 86 hours.Source: Bayer results · pp. 6–7
  • Mean F1 was 98.8 ± 1.3% across 28 PRINCE-comparable fields, versus PRINCE mean F1 of 82.5 ± 24.7%.Fields absent from or poorly annotated in PRINCE, including housing, diet, dosing frequency, and exposure start date, were extracted from reports.Source: Bayer results · pp. 6–7
  • Anesthesia method had 55.7% recall and 32.5% precision, while animal supplier country had 35.5% recall and 93.4% precision.Age and age units also performed less well, consistent with their frequent absence from reports.Source: Bayer results · p. 7
  • On 28 PRINCE-comparable fields, validation-set recall was 99.1 ± 2.2%, precision 95.1 ± 7.1%, and F1 97.0 ± 4.6%.Performance remained comparable to training; the authors attributed a modest precision reduction likely to unextractable report values or missing reference values, alongside more non-GLP studies.Source: Bayer results · p. 7

Roche results

  • Test Facility Name and Location each reached 100% accuracy, recall, precision, and F1; consistently reported metadata fields were strongest.These fields were typically on the cover page or in a dedicated test-site section.Source: Roche results · p. 8
  • Anesthetic Prior Blood Collection had 100% recall and 73% precision, while Anesthetic Prior Death had 99% recall and 72% precision.A recurrent false-positive pattern was confusing anesthetic agents with euthanasia agents when reports did not distinguish their purpose.Source: Roche results · p. 8
  • Animal Supplier Name had an F1 score of 79%, and Hematology Instrument had an F1 score of 85%.Errors included interpreting animal geographic origin as supplier identity and ambiguity between instruments, associated software, or models not used in the study.Source: Roche results · p. 8

Fasting status as a relevant selection criterion for virtual control groups

Fasting status was associated with shifts in some clinical chemistry analytes, and the effects for triglycerides and phosphate differed by species. Facility-level confounding limits interpretation of these associations.

Fasting status as a relevant selection criterion for virtual control groups

  • Analyses included 44 fasted and 12 unfasted monkey studies and 31 fasted and 9 unfasted rat studies; other species or strains lacked sufficient studies in one or both groups.Source: Fasting status as a relevant selection… · p. 8
  • Fasting was associated with lower glucose in cynomolgus monkeys, lower triglycerides in Wistar Han rats, and higher phosphate in cynomolgus monkeys after FDR correction.The reported fold changes were 0.87 for monkey glucose, 0.59 for rat triglycerides, and 1.12 for monkey phosphate.Source: Fasting status as a relevant selection… · p. 9
  • Fasting effects differed by species for triglycerides (interaction p = 0.009) and phosphate (interaction p = 0.037), supporting species-specific models.Source: Fasting status as a relevant selection… · p. 9
  • Fasting shifted three of eight analytes; ALT, AST, calcium, and cholesterol showed no significant shifts in either species.Rat ALP decreased by 25% but did not remain significant after FDR correction; the small number of unfasted studies does not exclude smaller effects for other analytes.Source: Fasting status as a relevant selection… · p. 9
  • Fasting was nearly collinear with test facility, so facility-level differences such as analytical platform cannot be separated from fasting effects; results should be interpreted as associations.Source: Fasting status as a relevant selection… · p. 9

Discussion

Independent Bayer and Roche pipelines recovered contextual metadata from heterogeneous legacy reports with high agreement to expert curation. The findings support capturing fasting status in structured standards and highlight metadata's relevance to historical-control selection.

Discussion

  • Independent Bayer and Roche LLM pipelines achieved high agreement with expert curation, extracting metadata previously accessible only through costly, time-consuming manual review.Source: Discussion · p. 9
  • Missing study descriptors are relevant to VCG construction because they inform historical-control selection (Gurjanov et al., 2023, Palazzi et al., 2024; Steger-Hartmann et al., 2025).Source: Discussion · p. 9
  • The findings support capturing fasting status in structured form in future versions of preclinical data standards.Source: Discussion · p. 9

Methodological lessons, limitations, and outlook

Both pipelines performed similarly well, while reporting heterogeneity explained many extraction errors and the authors emphasize validation and traceable evidence. Adapting the approach to new domains still requires expert input.

Methodological lessons

  • Both Bayer and Roche pipelines achieved similarly strong performance, while false positives and missed values often reflected reporting heterogeneity.The cited discussion identifies linguistic variability, domain-specific ambiguity, and differences in facility and time-period reporting.Source: Methodological lessons, limitations,… · p. 10
  • Verbatim evidence and report locations support efficient human review and provide a basis for automated verification and retry-based correction.The authors also describe iterative schema refinement, clearer instructions, and targeted prompt updates as ways to mitigate extraction issues.Source: Methodological lessons, limitations,… · p. 10

Limitations and outlook

  • Adapting the approach to new domains requires manual expert input to define metadata schemas and extraction instructions.The authors suggest automated assessment, prompt optimization, and agent-based support as potential ways to reduce this burden.Source: Methodological lessons, limitations,… · p. 10

Outlook

  • The authors conclude that LLM extraction can recover information from legacy preclinical reports accurately, at scale, and with traceable evidence.They state that the metadata strengthens the evidence base for translational and regulatory decisions and supports efforts to reduce animal use.Source: Methodological lessons, limitations,… · p. 10

Funding

The project acknowledges support through an IHI Joint Undertaking grant and lists the participating public and private partners; the authors report no further financial support.

Funding

  • VICT3R received support from the Innovative Health Initiative Joint Undertaking under grant agreement No 101172693.The JU receives support from the EU Horizon Europe programme and named partner organizations.Source: Funding · p. 11
  • The JU funding partners named include COCIR, EFPIA, Europa Bío, MedTech Europe, Vaccines Europe, and Instem Scientific Limited.The agreement also receives support from the European Union’s Horizon Europe research and innovation programme.Source: Funding · p. 11
  • The authors state that the expressed views do not necessarily reflect those of the funding parties and that no further financial support was received.The funding disclaimer says the listed parties cannot be held responsible for the views expressed.Source: Funding · p. 11

Figure Legends

The figure legends describe a PDF-to-metadata workflow, benchmark visualizations, and a fasting-status analysis; Table 1 illustrates confusion-matrix categories.

Figure 1

  • Figure 1 depicts PDF report preprocessing and parsing, optional section selection, schema-based LLM extraction, and expert curation and review.Expert feedback was used to refine preprocessing, section extraction, and schema definitions.Source: Figure 1 · p. 16

Figure 2

  • Figure 2 compares PRINCE baseline and merged extraction outputs against manually curated fields in training and validation sets.Panels A and B show training data (n = 205); panels C and D show validation data (n = 39), with bold labels marking fields absent from PRINCE.Source: Figure 2 · p. 15

Figure 3

  • Figure 3 shows Roche extraction counts by metadata field and grouped accuracy, recall, precision, and F1 scores for 202 reports.Facility name and location each include 220 evaluated instances because some studies involved multiple facilities.Source: Figure 3

Figure 4

  • Figure 4 presents control-animal clinical chemistry values for fasted and unfasted cynomolgus monkeys and Wistar Han rats across eight analytes.Per-animal values appear as boxes on a log scale and per-study means as black dots; analyte coverage counts are shown in panel A.Source: Figure 4 · p. 17
  • The fasting-effect models use study-level mixed models adjusted for sex and specimen matrix, with false-discovery-rate significance marked in red.Fold changes are fasted/unfasted estimates with 95% confidence intervals from log-value models with a study random intercept.Source: Figure Legends · p. 18

Table 1

  • Table 1 gives examples of confusion-matrix categories, with “No clear answer” denoting a standardized response when no value was extracted.Source: Table 1 · p. 18

Supplementary Material 1

The supplementary material specifies how extracted answers are judged against curated references, including rules for exact matches, partial matches, and domain-specific equivalences.

Correctness

  • An answer is correct when it matches the reference in meaning; wording, order, minor typos, and integer-versus-float notation may vary.Empty answers, “Not Found,” “Not Reported,” and “Not Available” are equivalent, but “Not applicable” is not.Source: Supplementary Material 1 · p. 19

Partial correctness

  • Partial correctness applies when an answer captures the reference partly, including an address with the correct country but no city.Overlapping number ranges may count as partial matches; for example, 19–24 overlaps 22–25 but not 25–27.Source: Supplementary Material 1 · pp. 19–20
  • Dates require the correct year, month, and day for a correct judgment; a missing or incorrect day is not partial correctness.Approximate durations expressed in days, weeks, months, or years may count as partial matches.Source: Supplementary Material 1 · p. 20
  • Administration-route judgments distinguish broad terms from specific routes: “ORAL” may mean dietary administration, while “intravenous” alone is partial for bolus or infusion.Intragastric is equivalent to oral gavage, and “one time” or “single dose” may match “ONCE.”Source: Supplementary Material 1 · p. 20

Additional considerations

  • Accepted equivalences include “Kolliphor” and “Solutol” as vehicle components, “single housed” and “singly housed,” and filtered and tap water.Dashes in codes and answer order do not change the evaluated meaning; overlapping ranges are considered partially correct.Source: Supplementary Material 1 · pp. 20–21
  • Answers that misunderstand the reference or omit important aspects are incorrect, and evaluation should account for relevant toxicology and biology terminology.Names may be written in full or abbreviated when they identify the same person; integers and floats can represent the same number.Source: Supplementary Material 1 · pp. 20–21

Supplementary Material 2

The strain-extraction instructions prioritize report text and permit species-based assumptions only when the strain is absent. Extracted strain values are then harmonized using supplied mappings.

Supplementary Material 2

  • Check the study description for the animal strain first; only when it is absent, use the specified species-based default.The instructions give defaults including WISTAR for rats, BEAGLE for dogs, and CYNOMOLGUS for non-human primates.Source: Supplementary Material 2 · p. 21
  • Map extracted strain variants to harmonized answers using the supplied examples, such as “Wistar” and “WISTAR” to “WISTAR.”The mappings also include multiple spelling and formatting variants for WISTAR HAN and CD1(ICR).Source: Supplementary Material 2 · p. 22

Supplementary Material 3

The supplementary material describes report preprocessing, a class-based extraction schema, and conventions for missing values. It also identifies the training- and validation-set panels used to display per-field extraction outcomes.

Supplementary Material 3

  • Report-core detection uses the table of contents when available, otherwise parsing initial pages and trigger terms to set an endpoint; if that fails, the first 100 pages are retained.The fallback can include appendix material, which was preferred to potentially excluding information needed for extraction.Source: Supplementary Material 3 · p. 22
  • Related metadata fields are grouped by topic in a class-based Python schema, with field-specific instructions and output formats.Study-specific extensions include anesthesia before blood collection because protocols can affect clinical-pathology measurements and historical-control comparability.Source: Supplementary Material 3 · p. 23
  • When extraction fails, fields generally receive a standardized missing-value response; GLP status, recovery period, and measurement-domain availability instead encode absence as negative.Source: Supplementary Material 3 · p. 23
  • The selected metadata span study conditions, suppliers, facilities, treatment, study design, and measurement-domain availability, alongside study-specific anesthesia fields.Examples include housing temperature and humidity, test-facility and supplier details, dose information, and whether laboratory or pathology results were reported.Source: Supplementary Material 3 · p. 24
  • The per-field outcome plots cover training and validation sets, contrasting PRINCE baseline panels with panels combining PRINCE and pipeline output after rule-based merging.The caption identifies training panels A–B (n = 205) and validation panels C–D (n = 39); bold labels mark fields absent from PRINCE in combined panels.Source: Supplementary Material 3 · p. 29
Loading 2610.32616v1…