Automated Analysis Framework for Multilingual Climate-Health Literature Based on Multi-Agent Large Language Model

Research gap

• Traditional manual reading, screening, coding, and induction cannot meet the literature's requirements for timeliness, scalability, and standardization.Source: Introduction · p. 1
• Single-LLM analysis struggles with specialized multi-task workflows and is prone to task confusion.Source: Introduction · p. 1

Research context

• More than 30,000 China-related climate-health papers create a large, multilingual, interdisciplinary corpus poorly suited to manual curation and classical topic models.The literature spans environmental science, public health, epidemiology, and atmospheric science, with close policy relevance.Source: Introduction · p. 2

Proposed framework

• The framework decomposes bilingual literature analysis into appraiser, extractor, and analyst agents managed by a central scheduler.Source: Introduction · p. 2

Reliability design

• A four-layer hallucination-control and consistency-verification pipeline combines prompt constraints, cross-agent checking, confidence scoring, external filtering, and expert validation.Source: Introduction · p. 2

Validation scope

• The system evaluates a bilingual corpus exceeding 30,000 papers, produces structured datasets, and identifies domain research gaps.Source: Introduction · p. 2

Overall architecture

• The centralized architecture manages data flow and execution order across document appraiser, information extractor, and senior analyst agents.Source: Overall Design of the Multi-Agent… · p. 2, Figure 1 · p. 3

Literature inputs

• The pipeline compiles English literature from OpenAlex and Chinese literature from CNKI and Baidu Scholar into standardized title, abstract, and keyword inputs.Source: Overall Design of the Multi-Agent… · p. 2

Document appraisal

• The document appraiser screens papers using climate exposure, health outcome, and Chinese city-location criteria, producing relevant/irrelevant classifications.Source: Evaluation: Document Appraiser Agent · p. 2
• Screening uses only explicit title, abstract, and keyword information, with domain specifications and annotated examples supporting cross-language consistency.Source: Evaluation: Document Appraiser Agent · p. 2

Information extraction

• The information extractor produces standardized fields including cities, climate exposures, health outcomes, vulnerable populations, and socioeconomic factors.Source: Extraction: Information Extractor Agent · p. 3
• Extraction is restricted to explicitly stated content, pairs co-occurring entities within the same sentence fragment, and assigns confidence scores for verification.Source: Extraction: Information Extractor Agent · p. 3

Senior analysis

• The senior analyst checks completeness, ontology alignment, terminology, and logical consistency before generating regional, thematic, temporal, and outlier analyses.Source: Analysis: Senior Analyst Agent · pp. 3–4

Experimental Setup

• 32,642 bilingual Chinese–English climate-health papers covering mainland China from 1993–2023 form the experimental corpus.The corpus was retrieved from OpenAlex, CNKI, and Baidu Scholar, then deduplicated, normalized, and structured.Source: Experimental Settings · p. 4

Core Information Extraction Accuracy

• 0.92 overall F1-score is achieved across geographic, climate-exposure, and health-outcome extraction tasks.Geographic extraction reaches F1 = 0.94, climate-exposure extraction F1 = 0.91, and health-outcome extraction F1 = 0.92.Source: Core Information Extraction Accuracy · p. 4
• 0.93 precision and 0.92 recall yield a 7% false-positive rate for the extraction system.The reported precision–recall profile is associated with the framework’s four-layer hallucination-control strategy.Source: Core Information Extraction Accuracy · p. 4

Baseline Comparison

• The proposed pipeline achieves the best overall precision, recall, and F1-score against single-LLM, BERT-NER, AutoGen, and existing literature-analysis baselines.A straightforward single-prompt LLM obtains an F1-score around 0.82, while the proposed system performs best overall.Source: Core Information Extraction Accuracy · p. 4

Ablation and Confidence Analysis

• Removing cross-agent verification, external knowledge filtering, confidence filtering, or coordination reduces performance, while the three-agent architecture achieves the highest F1 in agent-number ablation.High-confidence samples score F1 = 0.94 versus F1 = 0.66 for low-confidence samples.Source: Core Information Extraction Accuracy · p. 4

Multilingual Processing and Domain Application

• 0.93 English-literature F1 and 0.91 Chinese-literature F1 indicate a small cross-language performance gap, while 2,012 city-literature association pairs are extracted and standardized.Among 1,151 relevant papers, 1,128 are successfully extracted, and the reported standardization rate is 99%; the dataset identifies spatial and thematic gaps, including a spatial Gini coefficient of 0.761.Source: Multilingual Processing Performance · pp. 4–5, Practical Domain Application Effects · p. 5

Main findings

• The three-stage agent architecture improves extraction accuracy, mitigates hallucinations, and enhances stability over single-LLM baselines.The framework is positioned as an efficient evidence-synthesis tool and reusable paradigm for interdisciplinary literature-oriented data mining.Source: Conclusion and Discussion · p. 5
• Table 3 examines agent architectures with different task-decomposition granularities.The comparison concerns how screening, extraction, and verification work are distributed across agents.Source: Table 3 · p. 6

Limitations

• The system currently analyzes titles, abstracts, and keywords rather than full text, potentially missing research design, sample size, and statistical-model information.Future expansion includes PDF parsing, chapter segmentation, and table understanding for fuller extraction.Source: Conclusion and Discussion · p. 6
• A small number of extraction errors may remain for vague, incomplete, or non-standard literature, so manual spot checks remain the final guarantee.The limitation follows the framework’s stated reliance on expert verification for reliability.Source: Conclusion and Discussion · p. 6

Deployment considerations

• External LLM interfaces may impose call-rate limits in extremely large-scale scenarios, while deployment also requires optimization of latency, costs, prompt versions, and knowledge bases.Public-health policy applications additionally require traceability, reliability, ontology maintenance, and continued human expert supervision.Source: Conclusion and Discussion · p. 6, Practical Deployment Roadmap · p. 6

Table S1

• Table S1 ablates hallucination-control components and the central coordination module using mean ± std over three runs.The table is designed to quantify component contributions under repeated evaluation.Source: Table S1 · p. 9

Evaluation dimensions

• Evaluation covers extraction accuracy, operational efficiency, multilingual capability, and practical domain value.These four dimensions are intended to provide comprehensive and reliable assessment.Source: Evaluation Metrics · p. 9

Metrics

• Precision, recall, and F1-score are calculated against manual annotations from two climate-health experts.F1-score is the core metric because it balances precision and recall.Source: Information Extraction Accuracy Metrics · p. 9

Tasks

• The three core extraction tasks are geographic information, climate exposure factor, and health outcome, with an overall average across tasks.Metrics are calculated separately for each task before averaging.Source: Information Extraction Accuracy Metrics · p. 9

• The corpus is Chinese-English bilingual and covers mainland China from 1993 to 2023.It was constructed from authoritative academic databases.Source: Experimental Data and Corpus… · p. 9

Data Sources

• The corpus draws on OpenAlex and CNKI to collect English international and Chinese core-journal climate-health literature.OpenAlex retrieval covers 1993–2023, while CNKI retrieval covers 2009–2023.Source: Data Sources and Retrieval Strategy · p. 9

Corpus Preprocessing

• Standardized preprocessing removes duplicates, unifies metadata, supplements missing core fields, and structures paper text.Deduplication uses fuzzy matching of title, author, and publication year with a 0.95 similarity threshold.Source: Data Preprocessing and Corpus Scale · p. 9

Framework Architecture

• A centralized Prefect-based coordinator schedules three agents connected to Deepseek, ChatGPT, and Ernie Bot models.The document appraiser, information extractor, and senior analyst operate with temperature set to 0.1.Source: Multi-Agent Framework Configuration · p. 10

Domain Resources

• External databases standardize climate variables, Chinese city names and administrative divisions, and health outcome terminology.The sources include WMO, a Chinese city place-name database, and WHO-ICD resources.Source: External Support Resource Configuration · p. 10

Evaluation Design

• Evaluation uses an 800-paper bilingual gold corpus annotated independently by two climate-health experts, whose macro-F1 agreement reaches 0.87.Disagreements are resolved through joint discussion, and experts do not annotate the full corpus.Source: Gold-standard Annotation Setup · p. 10

Evaluation Metrics

• The evaluation framework measures cross-language extraction accuracy, processing efficiency, and practical output usability.Metrics cover F1-score, processing time, throughput, efficiency improvement, effective extraction, standardization, and research-gap identification.Source: System Efficiency Metrics · p. 10, Multilingual Processing Performance… · p. 10, Domain Application Value Metrics · p. 10

Agent Procedures

• Agent prompts enforce explicit-text screening, atomic structured extraction with confidence scores, and verification through normalization and external lookup.Records below confidence 3 are filtered and logical consistency between climate exposure and health outcome is checked.Source: Agent System Prompts · p. 10, Document Appraiser Agent Prompt · p. 10, Information Extractor Agent Prompt · p. 10