Source-linked AI summary
HERMES: a multi-agent framework for structured knowledge extraction from ultra-long documents in geoscience
Ziqi Song, Zongyuan Xiang, James G. Ogg, Bruce S. Lieberman, Gabi Ogg, Natalia López Carranza, Wen Du, Yufei Ye, Shuan Li, Zhong Peng, Shaoqi Yu, Juye Wei, Ying Zhou, Jieping Ye, Jiang Yang
TL;DR
Scientific knowledge in geoscience remains difficult to access and computationally reuse from long, multimodal literature. HERMES addresses this gap with multi-agent extraction, achieving robust cross-domain performance and reducing per-volume turnaround to 7 calendar days from 45 manually.
Problem
Geoscience literature contains valuable knowledge that remains difficult to access, structure, and computationally reuse, especially in large historical monographs.
Method
HERMES coordinates specialized agents through shared document context, domain validation, multimodal extraction, evidence tracing, and expert verification.
Results
HERMES maintained robust performance across palaeontology, palaeomagnetism, and geochemistry, reducing per-volume turnaround from 45 to 7 calendar days.
Takeaways & Limitations
HERMES offers a scalable process for converting dispersed geoscience evidence into traceable structured data across distinct document structures and data types.
Takeaways & Limitations
The two-stage extraction strategy can propagate missed, misbounded, or improperly merged entities into cascading attribute errors.
Abstract
from arXiv · showhide
Authoritative scientific knowledge in geoscience remains largely trapped in legacy monographs and historical literature, where unstructured text and complex layouts hinder computational access. We introduce HERMES, a scalable multi-agent framework that extracts structured data from ultra-long scientific documents. Using a coordinating large language model, HERMES integrates domain constraints, validation rules and evidence tracing within a unified document-level extraction process that incorporates parsed text, tables, figures and captions. Applied to the 55-volume Treatise on Invertebrate Paleontology, the system produced a structured database of 32,277 fossil taxonomic entities and 451,878 attributes, released online at https://treatise.geolex.org. Extraction performance remained stable across fossil groups (average F1 scores of approximately 0.90 for entities and 0.91 for attributes), improving per-volume efficiency approximately sixfold relative to the tested fully manual baseline. Evaluation in palaeomagnetism and geochemistry, conducted without additional model training, demonstrated transfer across distinct geoscience domains. This work provides a practical pathway to transform historical scientific literature into FAIR-oriented structured data, offering a sustainable infrastructure for data-intensive disciplines and large-scale knowledge integration.
Introduction
The rapid growth of scientific literature makes published knowledge increasingly difficult to access, structure, and computationally reuse, motivating evidence-aware extraction with provenance and validation. HERMES addresses this need by integrating multimodal extraction with human-in-the-loop validation for geoscience literature.
- Motivation: Rapid literature growth increasingly hinders access to, structuring of, and computational reuse of published scientific knowledge.The passage links this challenge to data-intensive scientific discovery and the need to organize, integrate, and analyze existing data at scale.
- Related work: Scientific information extraction has progressed from task-specific entity and relation extraction toward document-level and LLM-based structured extraction.SciERC, SciREX, and S2ORC established benchmarks and corpora supporting machine-readable scholarly text research.
- Contribution: HERMES integrates natural language processing, computer vision, entity recognition, and human-in-the-loop validation to extract and structure multimodal geoscience literature data.The system is presented as an evidence-aware process combining structured outputs with fine-grained provenance and human validation.
Results
HERMES transferred across palaeontology, palaeomagnetism and geochemistry without changing model weights, while Treatise processing reduced turnaround relative to fully manual extraction. Geochemical evaluation achieved strong entity precision but lower attribute-extraction performance.
- Cross-domain transfer: Model weights remained unchanged across palaeontology, palaeomagnetism and geochemistry, with only extraction schemas and Validator rules adapted.The ReAct-based architecture and dynamic shared context supported application across these disciplines.
- Palaeontology: 55 Treatise volumes, averaging 160,437 words each, were processed in the evaluation.The Treatise on Invertebrate Paleontology has been published and updated since 1953 and synthesizes work across multiple geoscience subfields.
- Palaeontology: 45 calendar days were required for fully manual extraction of one Treatise volume, compared with 7 calendar days for HERMES plus manual review.The HERMES workflow combined automated extraction with manual review and revision.
- Geochemistry: 0.81 was the overall F1 score for geochemical entity extraction, versus 0.51 for attribute extraction.Entity extraction had recall 0.89 and precision 0.74; attribute extraction had recall 0.35 and precision 0.93.
- Geochemistry: 10 target attributes were selected from geochemical source tables containing at least 67 variables.HERMES was evaluated on selected attributes rather than every variable in the source dataset.
Discussion
HERMES addresses the conversion of dispersed scientific evidence into traceable FAIR data through coordinated document parsing and extraction. Its design improves controllability and scalability but remains vulnerable to cascading entity errors, efficiency–recall trade-offs, and complex table semantics.
- Contribution: HERMES organizes document parsing and extraction to convert evidence dispersed across narrative text, tables, figures, and captions into traceable FAIR data.The framework targets scientific information that is difficult to convert into FAIR data when evidence spans multiple document formats.
- Limitations: The two-stage entity–attribute strategy improves controllability and auditability but can propagate missed, misbounded, or improperly merged entities into incorrect attribute associations.The entity index created during recognition underpins attribute extraction, so recognition errors can cause omissions or misassignments downstream.
- Limitations: Rule-based filtering, text chunking, and RAG reduce computational complexity for extremely long documents but trade extraction efficiency against recall.Dimensionality reduction can impair reliable use of evidence when relevant information is sparsely distributed or located in less salient text.
- Limitations: Complex table structures remain difficult to interpret, particularly across pages, multiple related tables, and cases involving complex headers or merged cells.HERMES recognizes and parses most tabular data, but fully understanding table semantics remains challenging.
Methods · Orchestrator
HERMES used GeoGPT-R1-Preview as its Orchestrator, which interpreted queries, managed task state, decomposed extraction into executable steps, and coordinated the process. The framework also supported task-specific LLM backends, with DeepSeek-V3 used for entity extraction and GeoGPT-R1-Preview for attribute extraction in the reported experiments.
- Orchestrator: GeoGPT-R1-Preview served as the Orchestrator agent in the reported experiments.It is based on the Qwen2.5-72B foundation model.
- Orchestrator: The Orchestrator provided semantic understanding, reasoning, and task orchestration capabilities.The passage characterizes GeoGPT-R1-Preview as having strong reasoning capabilities for answering geoscience questions.
- Orchestrator: The Orchestrator interpreted user queries and determined the extraction task state.
- Orchestrator: The Orchestrator decomposed the extraction process into executable steps.
- Orchestrator: HERMES supported task-specific LLM backends for individual sub-agents.The implementation's controller choice was not presented as the only possible backend assignment.
- Orchestrator: DeepSeek-V3 was used by the Entity Recognizer for entity extraction in the reported experiments.
- Orchestrator: GeoGPT-R1-Preview was used by the Annotator for attribute extraction in the reported experiments.
- Orchestrator: All LLM inference settings used in HERMES were summarized in the Supplementary Information unless otherwise stated.
The Shared Context · Parser
HERMES uses a document-level shared context to coordinate multimodal extraction, preserve semantic consistency, and support auditing. Its Parser digitizes PDF content while retaining layout information, page locations, paragraph boundaries, and reading order for downstream processing.
- The Shared Context: The shared context consolidates intermediate results from sub-agents at the document level, with all agents exchanging data exclusively through this centralized structure.This architecture preserves semantic consistency across processing stages and supports result auditing.
- The Shared Context: The shared context links multimodal parsed content with structured entities, attributes, provenance, and validation information for document-level extraction.This summarizes the coordinated information architecture described for the shared context and Parser outputs.
- The Shared Context: It integrates PDF text, figures, captions, tabular data, layout information, paragraph and chunk structures, entity indices, attribute records, and field-level evidence provenance with validation flags.The same structure was reused for palaeomagnetism and geochemistry extraction and can be adapted to other disciplines with minimal modification.
- Parser: The Parser digitizes raw PDF content, including text, figures, and tables, using Mathpix OCR and an in-house layout analysis module.The layout analysis module is informed by document layout analysis methods.
- Parser: Parser outputs preserve page numbers and spatial coordinates for recognized document elements.These positional details support subsequent organization of extracted content.
- Parser: The Parser sorts recognized elements to restore paragraph boundaries and reading order after digitization.This processing specifically addresses multi-column layouts and mixed text structures.
Entity Recognizer
The Entity Recognizer extracts user-specified target entities from source documents and builds an entity index using query-provided descriptions and retrieval over encoded text chunks.
- Entity Recognizer: The Entity Recognizer defines each target entity by a name, definition, and example, then segments shared-context text into chunks and encodes them for retrieval indexing.It uses the Llama-Index framework and bge-base-en-v1.552 embeddings to construct a full-text retrieval space.
Annotator
The Annotator extracts user-specified attribute values for each entity. It uses attribute descriptions and entity information to retrieve relevant text and guide explicit-value extraction.
- The Annotator extracts values of user-specified attributes for each entity.
- Each attribute query combines its name, definition, example, and the target entity’s name and description.
- The system retrieves the top-10 similar full-text chunks and instructs GeoGPT-R1-Preview36 to extract values explicitly appearing in them.
Validator
The Validator standardizes extracted entities and attributes, checks scientific and logical consistency, and routes failed fields for tracing and expert verification. It also extracts citation metadata when captions lack repository code.
- The Validator applies rule-based constraints to normalize extracted entities and attributes according to domain-specific scientific expression standards.
- It checks temporal-range ordering and consistency between attribute values and entity types, flagging failures for the Tracer agent and expert verification.
- When captions contain no repository code, the Validator extracts citation information as the author and four-digit year.The passage gives “Smith, 1987” as an example.
Tracer
The Tracer agent links each extracted entity name or attribute value to source evidence using a coarse-to-fine retrieval and reranking process. It records the top-ranked evidence text and coordinates in shared context.
- Evidence matching: The Tracer matches each extracted entity name or attribute value to corresponding source evidence stored as sentence or table chunks.It first identifies chunks that may contain the target item during a coarse retrieval stage.
- Fine reranking: The BGE-reranker-large57 ranks candidate chunks, and the Tracer returns the top-1 result with its text and coordinates written to shared context.This second phase refines the coarse candidate set to the most relevant source evidence.
Human-in-the-loop Validation
HERMES supports human-in-the-loop validation through an interactive workbench that links extracted results to their source evidence in the original PDF. Users can inspect extraction outputs in a structured format alongside the document.
- Human-in-the-loop Validation: HERMES provides native support for human-in-the-loop validation through a dedicated interactive workbench.The workbench enables users to review extraction results within the system.
- Human-in-the-loop Validation: The workbench displays the original PDF alongside structured entity-attribute extraction results.This side-by-side layout supports direct comparison between the source document and extracted data.
- Human-in-the-loop Validation: Selecting an entity or attribute automatically highlights its source evidence in the original document.This evidence-tracing interaction connects extracted information to its location in the source PDF.
Generative AI assistance
ChatGPT was used only for language editing during manuscript preparation. The authors developed, verified, and approved all scientific content and take full responsibility for the work.
- ChatGPT was used solely to improve language clarity, grammar, and readability.
- The authors developed, verified, and approved the manuscript’s scientific content, methodological design, analyses, interpretations, and conclusions.
- The authors take full responsibility for the manuscript.
Competing Interests
The authors declare no competing interests.
- The authors declare no competing interests.