Source-linked AI summary
Automated Analysis Framework for Multilingual Climate-Health Literature Based on Multi-Agent Large Language Model
Yuze Sun, Shihui Zhang, Jiancheng Pan, Yunjia Ye, Wentao Luo, Jiahao Li, Quan Zhang, Wenjia Cai, Xiaomeng Huang
TL;DR
The paper addresses the difficulty of efficiently and reliably analyzing rapidly expanding multilingual, interdisciplinary climate-health literature. It presents a centrally coordinated multi-agent LLM pipeline for screening, structured extraction, standardization, and analysis, with hallucination controls and expert verification. The framework achieves an overall extraction F1-score of 0.92 and supports large-scale climate-health evidence mining.
Problem
Expanding multilingual and interdisciplinary literature makes manual and single-algorithm analysis insufficiently efficient, scalable, and domain-adaptable.
Method
A centralized pipeline assigns document appraisal, information extraction, and analytical review to three specialized agents, supported by hallucination controls and expert validation.
Results
0.92 overall F1-score was achieved for the three core information-extraction tasks.
Takeaways & Limitations
The framework provides an efficient evidence-synthesis tool and reusable multi-agent paradigm for climate-health literature-oriented data mining.
Takeaways & Limitations
Large-batch latency, LLM costs, prompt and knowledge-base version management, terminology maintenance, and continued expert supervision remain deployment challenges.
Abstract
from arXiv · showhide
The rapid proliferation of interdisciplinary and multilingual scientific literature has left traditional manual analysis and single-algorithm methods plagued by low efficiency, poor scalability, and insufficient domain adaptability. Targeting the literature analysis needs of the typical interdisciplinary climate-health field, this study proposes a multi-agent large language model automated analysis framework for multilingual scientific literature, which realizes full-process automation covering literature screening, structured information extraction, and standardized integration. With a central coordination module as the core, the framework deploys three dedicated agents for document evaluation, information extraction, and analytical review to mimic the literature analysis thinking of domain experts, and adopts a four-layer hallucination control strategy together with a manual verification procedure to ensure the accuracy and reliability of analytical outcomes. Validated on a bilingual Chinese-English corpus of 32,642 climate-health papers covering China from 1993 to 2023, the framework achieves an F1 score of 0.92 in core information extraction, and completes the extraction and standardization of 2,012 city-literature association pairs, offering effective technical support for large-scale evidence mining in the climate-health research domain.
Introduction
Climate-health literature is rapidly expanding across languages and disciplines, exposing the limits of manual review and single-LLM analysis. The paper addresses this gap with a modular multi-agent framework incorporating standardized extraction and hallucination control.
- Research gap: Traditional manual reading, screening, coding, and induction cannot meet the literature's requirements for timeliness, scalability, and standardization.
- Research gap: Single-LLM analysis struggles with specialized multi-task workflows and is prone to task confusion.
- Research context: More than 30,000 China-related climate-health papers create a large, multilingual, interdisciplinary corpus poorly suited to manual curation and classical topic models.The literature spans environmental science, public health, epidemiology, and atmospheric science, with close policy relevance.
- Proposed framework: The framework decomposes bilingual literature analysis into appraiser, extractor, and analyst agents managed by a central scheduler.
- Reliability design: A four-layer hallucination-control and consistency-verification pipeline combines prompt constraints, cross-agent checking, confidence scoring, external filtering, and expert validation.
- Validation scope: The system evaluates a bilingual corpus exceeding 30,000 papers, produces structured datasets, and identifies domain research gaps.
Method
The method is a centrally scheduled pipeline that screens multilingual literature, extracts standardized climate-health information, validates it against domain resources, and generates analytical reports. Three specialized agents divide these functions across document appraisal, information extraction, and senior analysis.
- Overall architecture: The centralized architecture manages data flow and execution order across document appraiser, information extractor, and senior analyst agents.
- Literature inputs: The pipeline compiles English literature from OpenAlex and Chinese literature from CNKI and Baidu Scholar into standardized title, abstract, and keyword inputs.
- Document appraisal: The document appraiser screens papers using climate exposure, health outcome, and Chinese city-location criteria, producing relevant/irrelevant classifications.
- Document appraisal: Screening uses only explicit title, abstract, and keyword information, with domain specifications and annotated examples supporting cross-language consistency.
- Information extraction: The information extractor produces standardized fields including cities, climate exposures, health outcomes, vulnerable populations, and socioeconomic factors.
- Information extraction: Extraction is restricted to explicitly stated content, pairs co-occurring entities within the same sentence fragment, and assigns confidence scores for verification.
- Senior analysis: The senior analyst checks completeness, ontology alignment, terminology, and logical consistency before generating regional, thematic, temporal, and outlier analyses.
Experiments
The experiments evaluate extraction accuracy, efficiency, multilingual capability, and practical application on a 32,642-document bilingual corpus. The framework achieves strong extraction performance, supports cross-language processing, and produces standardized city-literature evidence for identifying research patterns and gaps.
- Experimental Setup: 32,642 bilingual Chinese–English climate-health papers covering mainland China from 1993–2023 form the experimental corpus.The corpus was retrieved from OpenAlex, CNKI, and Baidu Scholar, then deduplicated, normalized, and structured.
- Core Information Extraction Accuracy: 0.92 overall F1-score is achieved across geographic, climate-exposure, and health-outcome extraction tasks.Geographic extraction reaches F1 = 0.94, climate-exposure extraction F1 = 0.91, and health-outcome extraction F1 = 0.92.
- Core Information Extraction Accuracy: 0.93 precision and 0.92 recall yield a 7% false-positive rate for the extraction system.The reported precision–recall profile is associated with the framework’s four-layer hallucination-control strategy.
- Baseline Comparison: The proposed pipeline achieves the best overall precision, recall, and F1-score against single-LLM, BERT-NER, AutoGen, and existing literature-analysis baselines.A straightforward single-prompt LLM obtains an F1-score around 0.82, while the proposed system performs best overall.
- Ablation and Confidence Analysis: Removing cross-agent verification, external knowledge filtering, confidence filtering, or coordination reduces performance, while the three-agent architecture achieves the highest F1 in agent-number ablation.High-confidence samples score F1 = 0.94 versus F1 = 0.66 for low-confidence samples.
- Multilingual Processing and Domain Application: 0.93 English-literature F1 and 0.91 Chinese-literature F1 indicate a small cross-language performance gap, while 2,012 city-literature association pairs are extracted and standardized.Among 1,151 relevant papers, 1,128 are successfully extracted, and the reported standardization rate is 99%; the dataset identifies spatial and thematic gaps, including a spatial Gini coefficient of 0.761.
Conclusion and Discussion
The framework demonstrates that a three-stage multi-agent architecture can support efficient, standardized multilingual literature mining while reducing hallucinations. Its current scope remains limited by metadata-only analysis, residual extraction errors, external-interface constraints, and deployment requirements for optimization and expert oversight.
- Main findings: The three-stage agent architecture improves extraction accuracy, mitigates hallucinations, and enhances stability over single-LLM baselines.The framework is positioned as an efficient evidence-synthesis tool and reusable paradigm for interdisciplinary literature-oriented data mining.
- Main findings: Table 3 examines agent architectures with different task-decomposition granularities.The comparison concerns how screening, extraction, and verification work are distributed across agents.
- Limitations: The system currently analyzes titles, abstracts, and keywords rather than full text, potentially missing research design, sample size, and statistical-model information.Future expansion includes PDF parsing, chapter segmentation, and table understanding for fuller extraction.
- Limitations: A small number of extraction errors may remain for vague, incomplete, or non-standard literature, so manual spot checks remain the final guarantee.The limitation follows the framework’s stated reliance on expert verification for reliability.
- Deployment considerations: External LLM interfaces may impose call-rate limits in extremely large-scale scenarios, while deployment also requires optimization of latency, costs, prompt versions, and knowledge bases.Public-health policy applications additionally require traceability, reliability, ontology maintenance, and continued human expert supervision.
Table
Table S1 reports ablations of hallucination-control components and the central coordination module, using mean ± std over three runs.
- Table S1: Table S1 ablates hallucination-control components and the central coordination module using mean ± std over three runs.The table is designed to quantify component contributions under repeated evaluation.
Evaluation Metrics
The framework is evaluated across extraction accuracy, operational efficiency, multilingual capability, and practical domain value.
- Evaluation dimensions: Evaluation covers extraction accuracy, operational efficiency, multilingual capability, and practical domain value.These four dimensions are intended to provide comprehensive and reliable assessment.
Information Extraction Accuracy Metrics
Core extraction accuracy is assessed against expert gold labels using precision, recall, and F1-score for geographic, climate-exposure, and health-outcome information.
- Metrics: Precision, recall, and F1-score are calculated against manual annotations from two climate-health experts.F1-score is the core metric because it balances precision and recall.
- Tasks: The three core extraction tasks are geographic information, climate exposure factor, and health outcome, with an overall average across tasks.Metrics are calculated separately for each task before averaging.
Experimental Data and Corpus Construction
The study constructs a Chinese-English bilingual climate-health literature corpus covering mainland China from 1993 to 2023, using authoritative academic databases to support representativeness and academic quality.
- The corpus is Chinese-English bilingual and covers mainland China from 1993 to 2023.It was constructed from authoritative academic databases.
Data Sources and Retrieval Strategy
The framework combines multi-database corpus preprocessing, centralized multi-agent scheduling, external domain databases, expert-annotated evaluation, and measures of accuracy, efficiency, and output usability. Its agent prompts constrain screening and extraction to explicit text while supporting verification and normalization of structured records.
- Data Sources: The corpus draws on OpenAlex and CNKI to collect English international and Chinese core-journal climate-health literature.OpenAlex retrieval covers 1993–2023, while CNKI retrieval covers 2009–2023.
- Corpus Preprocessing: Standardized preprocessing removes duplicates, unifies metadata, supplements missing core fields, and structures paper text.Deduplication uses fuzzy matching of title, author, and publication year with a 0.95 similarity threshold.
- Framework Architecture: A centralized Prefect-based coordinator schedules three agents connected to Deepseek, ChatGPT, and Ernie Bot models.The document appraiser, information extractor, and senior analyst operate with temperature set to 0.1.
- Domain Resources: External databases standardize climate variables, Chinese city names and administrative divisions, and health outcome terminology.The sources include WMO, a Chinese city place-name database, and WHO-ICD resources.
- Evaluation Design: Evaluation uses an 800-paper bilingual gold corpus annotated independently by two climate-health experts, whose macro-F1 agreement reaches 0.87.Disagreements are resolved through joint discussion, and experts do not annotate the full corpus.
- Evaluation Metrics: The evaluation framework measures cross-language extraction accuracy, processing efficiency, and practical output usability.Metrics cover F1-score, processing time, throughput, efficiency improvement, effective extraction, standardization, and research-gap identification.
- Agent Procedures: Agent prompts enforce explicit-text screening, atomic structured extraction with confidence scores, and verification through normalization and external lookup.Records below confidence 3 are filtered and logical consistency between climate exposure and health outcome is checked.