PaperlayerView exact revision on arXiv

Source-linked AI summary

EpiQAL: Benchmarking Large Language Models in Epidemiological Question Answering and Reasoning

Mingyang Wei, Dehai Min, Zewen Liu, Yuzhang Xie, Guanchen Wu, Ziyang Zhang, Carl Yang, Max S. Y. Lau, Qi He, Lu Cheng, Wei Jin

arXiv:2601.03471v4cs.CLcs.AI

TL;DR

Epidemiological QA requires population-level synthesis of study evidence, yet existing medical QA benchmarks primarily target clinical knowledge and patient-level reasoning. EpiQAL benchmarks epidemiological question answering over research articles through three subsets and a quality-controlled construction pipeline. Across fifteen models, multi-step inference is the greatest challenge, rankings shift across subsets, and scale alone does not predict success.

  • Problem

    Existing medical QA benchmarks primarily emphasize clinical knowledge or patient-level reasoning, while epidemiological inference requires population-level synthesis of study evidence.

    Source: §1 · p. 1

  • Method

    EpiQAL evaluates research-literature QA with subsets for factual recall, multi-step inference, and conclusion reconstruction, built using taxonomy guidance, multi-model verification, and difficulty screening.

    Source: §1 · p. 2, §3.1 · pp. 2–3, §5 · p. 9

  • Results

    Across fifteen models, multi-step inference is the greatest challenge; rankings shift across subsets, and scale alone does not predict success.

    Source: §4.5 · pp. 7–8, §5 · p. 9

  • Takeaways & Limitations

    EpiQAL provides fine-grained diagnostic signals for evidence-grounding, inferential reasoning, and conclusion reconstruction.

    Source: §5 · p. 9

    The controls do not separate epidemiological reasoning from general scientific reading, and human evaluation covers 120 questions (8.4% of the benchmark).

    Source: Limitations · pp. 9–10

Abstract

from arXiv · show

Reliable epidemiological reasoning requires synthesizing study evidence to infer disease burden, transmission dynamics, and intervention effects at the population level. Existing medical question answering benchmarks primarily emphasize clinical knowledge or patient-level reasoning, yet few systematically evaluate evidence-grounded epidemiological inference. We present EpiQAL, to our knowledge the first diagnostic benchmark for epidemiological question answering over research literature, comprising three subsets built from open-access articles across diverse diseases. The three subsets progressively test factual recall, multi-step inference, and conclusion reconstruction under incomplete information, and are constructed through a quality-controlled pipeline combining taxonomy guidance, multi-model verification, and difficulty screening. Experiments on fifteen models spanning open-source and proprietary systems reveal that current LLMs show limited performance on epidemiological reasoning, with multi-step inference posing the greatest challenge. Model rankings shift across subsets, and scale alone does not predict success. Chain-of-Thought prompting benefits multi-step inference but yields mixed results elsewhere. EpiQAL provides fine-grained diagnostic signals for evidence-grounding, inferential reasoning, and conclusion reconstruction.

1 Introduction

Epidemiological public health practice requires population-level synthesis of heterogeneous study evidence, but existing QA benchmarks mainly address clinical knowledge or patient-level decisions. EpiQAL targets this gap with a broad, evidence-grounded benchmark and a quality-controlled pipeline for testing three forms of epidemiological reasoning.

Motivation

  • Epidemiological reasoning requires population-level inference about disease burden, transmission, and interventions, unlike biomedical QA focused on clinical knowledge and patient-level decisions.It also requires synthesizing heterogeneous, context-dependent study findings.Source: §1 · p. 1
  • Existing resources only partly provide controlled, trustworthy evaluation: they emphasize clinical exams, abstracts with constrained labels, or disease-specific extractive formats, while expert annotation limits scale and coverage.A suitable benchmark should reduce shortcut cues and ground answers in verifiable study evidence.Source: §1 · p. 1

Contributions

  • EpiQAL is presented as the first systematic epidemiological QA benchmark combining broad topic coverage, set-based evaluation, and document-grounded answer derivation.Its three subsets assess text-grounded factual recall, multi-step inference, and conclusion reconstruction with the Discussion withheld.Source: §1 · pp. 1–2
  • The construction framework uses an expert-curated taxonomy, evidence-grounded subset strategies, multi-LLM verification with targeted human review, and difficulty screening with stem refinement.These choices address broad scope, traceable answers, multi-answer validation, and superficial-cue exploitation.Source: §1 · p. 2

2 Related Work

Prior biomedical QA resources differ in evidence source, format, and domain scope; EpiQAL combines diverse epidemiological topics, set-based evaluation, and masked-input conclusion reconstruction.

Biomedical QA benchmarks

  • Exam-style MedQA and MedMCQA test broad medical knowledge, while BioASQ and PubMedQA ground questions in biomedical literature but use summaries or a constrained yes/no/maybe space (Jin et al., 2021; Pal et al., 2022; Krithara et al., 2023; Jin et al., 2019).COVID-QA, CoQUAD, and EPIC-QA typically focus on pandemic evidence and extractive formats.Source: §2 · p. 2

Benchmark distinctions

  • EpiQAL differs by covering diverse epidemiological topics, supporting single- and multi-answer set evaluation, and masking input for conclusion reconstruction.Prior construction work includes model-in-the-loop filtering, multi-judge verification, set-based metrics, and difficulty screening (Bartolo et al., 2020; Kiela et al., 2021; Bras et al., 2020; Lee et al., 2025; Liu et al., 2023; Ma et al., 2025; Yang et al., 2018; Lee et al., 2023; Bai et al., 2024).Source: §2 · p. 2

3 Method

EpiQAL formulates literature-based question generation and set-based evaluation, then builds three complementary subsets using taxonomy and evidence constraints. Verification and difficulty control are applied to improve instance quality, with difficulty screening limited to the inference and masked-input subsets.

3.1 Task Formulation

  • The task generates a question, correct-option set, and distractor set from a source document under a schema specifying topic scope, reasoning requirements, and option rules.Optional external knowledge is used during construction for EpiQAL-B only; EpiQAL-A and EpiQAL-C use none.Source: §3.1 · p. 2

3.2 Framework and Subset Design

  • The three subsets test text-grounded recall, multi-step inference, and reconstruction of author-stated conclusions when the Discussion is masked.A questions are explicitly supported by document spans; B derives implications by linking evidence and epidemiological knowledge; C conclusions must remain supportable by the unmasked input.Source: §3.2 · pp. 3–4

3.3 Input Constraints

  • A six-class, 25-topic taxonomy refined with an epidemiology expert steers evidence selection, question phrasing, and option design for EpiQAL-A and B.EpiQAL-B may use knowledge-graph relations during construction, but that knowledge is withheld at evaluation; EpiQAL-C derives supervision from paper structure.Source: §3.3 · p. 4

3.4 Multi-model Verification

  • Multiple LLM checkers assess option-label consistency against cited evidence and reasoning coherence, with stochastic votes routing uncertain cases to human review.For EpiQAL-C, checkers confirm conclusion traceability to the Discussion but judge derivability from the masked test-time input; reviewers decide flagged cases.Source: §3.4 · pp. 4–5

3.5 Difficulty Control

  • Difficulty control applies only to EpiQAL-B and C: model-pool F1 and Exact Match scores identify easy items, and stem refinement reduces lexical matching cues.Refinement replaces salient entities with descriptive phrases, iterating until the difficulty threshold or a three-iteration cap; later iterations can over-expand descriptions.Source: §3.5 · p. 5

4 Experiment

The experiments assess dataset construction, human-rated quality, and fifteen models across EpiQAL’s three subsets. Performance is uneven across tasks, with multi-step inference especially difficult and errors varying by distractor type.

4.1 Generation Settings

  • The benchmark draws from about 10,600 PLOS Neglected Tropical Diseases articles covering over 20 diseases, with 500 sampled for the main experiments.Generation uses Qwen3-30B-A3B-Instruct-2507; three-model voting verifies options, and difficulty screening uses a four-model pool for EpiQAL-B and C.Source: §4.1 · pp. 5–6

4.2.1 Verification Pipeline Statistics

  • The pipeline automatically resolves over 96% of options without human intervention, while 2.2–3.6% of instances are fully discarded.After verification and human review of ambiguous options, the retained sets contain 475, 478, and 479 instances for EpiQAL-A, B, and C.Source: §4.2.1 · p. 6, §4.1 · p. 6

4.2 Benchmark Quality Evaluation

  • Human evaluation scores a stratified sample of 120 questions across subset-specific dimensions, with three biomedical-trained computer science PhD students rating each item.A biostatistics and epidemiology faculty member independently scores 30 questions while blind to annotator scores.Source: §4.2.2 · p. 6

4.2.2 Human Evaluation

  • Human ratings exceed 2.0 on every dimension; binary agreement exceeds 91%, and expert agreement with annotator majorities is 96.0–98.3% across subsets.The expert scored 30 questions, and the authors caution that this small sample does not establish expert validation of every automatically verified label.Source: §4.2.2 · p. 6, Table 3 · p. 6

4.3.1 Setup

  • Fifteen models from seven families, ranging from 3B to over 100B parameters, are evaluated with set-based F1 and Exact Match under zero-shot and one-shot prompts.The zero-shot settings include both no-CoT and CoT; the temperature ablation changes mean per-model EM by less than 1%.Source: §4.3.1 · pp. 6–7

4.4 Distractor Error Analysis

  • Semantic near-miss distractors are roughly twice as deceptive as other categories on A; B categories range from 15.9–19.2%, while speculation/limitation leads C at 21.0%.The higher 39.3% background-only rate on C is based on only eight distractors and is excluded from comparison.Source: §4.4 · p. 7, Table 5 · p. 8

4.3.2 Results

  • EpiQAL-B is hardest: only four of fifteen models exceed 0.70 EM without CoT, while leaders differ by subset—DeepSeek-V3.2-Thinking on A/B and Mistral-7B on C.Table 4 reports Exact Match and F1 across subsets, with and without Chain-of-Thought prompting.Source: §4.5 · pp. 7–8, Table 4 · p. 8

4.5 Discussion

  • Model rankings are subset-dependent and scale alone does not predict success; CoT yields its largest gains on B but has mixed effects elsewhere.Mistral-7B outperforms Mistral-Large on B and C despite having a fraction of the parameters; on C, GPT-5-mini’s EM decreases from 0.599 to 0.555 with CoT.Source: §4.5 · pp. 8–9

5 Conclusion

EpiQAL evaluates evidence-grounded epidemiological reasoning through three complementary subsets and a quality-controlled construction framework. Across fifteen models, multi-step inference is the greatest challenge, and model rankings vary across subsets.

5 Conclusion

  • Current LLMs struggle with epidemiological reasoning; multi-step inference is hardest, and rankings shift across subsets while scale alone does not predict success.Source: §5 · p. 9
  • EpiQAL combines an expert-curated taxonomy, subset-specific grounding, multi-model verification, and difficulty screening across recall, inference, and conclusion-reconstruction subsets.Source: §5 · p. 9
  • Chain-of-Thought prompting benefits multi-step inference but has mixed results elsewhere; reasoning-level distractors challenge models on inference, while semantic near-misses dominate retrieval errors.Source: §5 · p. 9
  • An International Journal of Epidemiology evaluation indicates the pipeline generalizes beyond neglected tropical diseases, and the authors release EpiQAL for future research.Source: §5 · p. 9

Limitations

The authors identify limits in what EpiQAL measures, the breadth and construction of its source data, and its use as a proxy for real-world public health analysis.

Limitations

  • The controls do not isolate epidemiological reasoning from general scientific reading, and format differences confound cross-subset comparisons of Exact Match.EpiQAL-B grounding evidence is partial; B and C have more options and single answers, unlike A.Source: Limitations · p. 9
  • The main corpus is English-language research from PLOS Neglected Tropical Diseases, while the International Journal of Epidemiology check tests pipeline portability rather than epidemiology-wide coverage.Source: Limitations · p. 9
  • Each subset has approximately 500 instances generated by one 30B-parameter model; a preliminary alternative-generator check covers only 50 articles per subset and lacks matched distractor-deception rates.Source: Limitations · p. 9
  • Residual errors may remain despite verification; human review covers 120 questions, and LLM-based generation and evaluation leave item style and difficulty partly model-relative.Some evaluated models also serve as checkers or difficulty judges.Source: Limitations · p. 10
  • Post-collection training may indirectly expose models to source content, and single-article reasoning omits the multiple documents and temporal or geographic context of real public health analysis.Source: Limitations · p. 10

Ethical Considerations

These passages describe EpiQAL’s ethical scope and supporting appendix material on related work, evaluation, taxonomy, knowledge construction, distractors, and stem refinement.

Ethical Considerations

  • The benchmark uses published research articles without personally identifying information, and human annotation involved project researchers and a faculty collaborator rather than crowdworkers.The main benchmark is released under terms consistent with the source corpus’s CC BY 4.0 license.Source: Ethical Considerations · p. 10

A Additional Related Work

  • Prior resources include extractive reading comprehension, clinical QA, retrieval augmentation, and biomedical knowledge graphs; EpiQAL-B uses graph triples as generation signals withheld at evaluation.Source: A Additional Related Work · p. 14

B.1 Evaluation Metrics

  • The evaluation calculation uses the model-predicted and reference option sets, including their respective cardinalities.Source: §B.1 · p. 14

B.2 Epidemiology Taxonomy

  • Six taxonomy classes cover complementary stages of epidemiological investigation, with topics and descriptions steering question generation for EpiQAL-A and EpiQAL-B.EpiQAL-C derives supervision from paper structure instead of taxonomy guidance.Source: §B.2 · p. 15

B.3 External Knowledge Construction

  • EpiQAL-B’s external knowledge pipeline extracts and links disease entities, retrieves related knowledge-graph triples, and summarizes them into generation signals not given at evaluation.Source: §B.3 · p. 15

B.4 Distractor Design

  • Distractors are passage-grounded role or context confounders in A, reasoning-flaw adversaries in B, and plausible traps under masked input in C.Source: §B.4 · p. 15

B.5 Stem Refinement Procedure & Example

  • Retrieved descriptions iteratively replace salient entities until a difficulty threshold or iteration cap, reducing lexical anchors while only rewritten stems reach evaluation.Source: §B.5 · p. 16

B.6 Difficulty Control Analysis

Difficulty screening selects how much to refine each stem, using a threshold intended to avoid unnecessary rewriting. Uniform refinement modestly shifts difficulty upward, and most instances already meet the threshold before rewriting.

B.6 Difficulty Control Analysis

  • 33.7% to 61.9% of EpiQAL-B instances and 30.9% to 47.2% of EpiQAL-C instances are classified as easy across the θd = 0.20-to-0.25 threshold change, motivating θd = 0.2.The pipeline selects the first iteration meeting the threshold, using Iter 3 as fallback.Source: §B.6 · pp. 16–17
  • EpiQAL-B mean DiffScore rises from 0.275 to 0.298 and EpiQAL-C from 0.336 to 0.354 after three uniform refinement iterations.Easy-instance shares fall from 33.7% to 28.9% for B and from 30.9% to 29.4% for C.Source: §B.6 · p. 17
  • 317 B instances and 331 C instances retain original stems; most instances exceed the difficulty threshold without refinement.For B, 28, 12, and 121 use iterations 1, 2, and 3; for C, the counts are 31, 12, and 105.Source: §B.6 · p. 17

B.7 Construction Pipeline Pseudocode

  • Algorithm 1 summarizes the construction pipeline and marks which steps apply to each subset.Source: §B.7 · p. 17

C.1 Compute and Inference Settings

The appendix reports compute and preprocessing choices, model configurations for benchmark construction, and the scope of the taxonomy-based dataset analysis.

C.1 Compute and Inference Settings

  • Experiments used NVIDIA H100 and H200 GPUs; Llama-3.3-70B-Instruct and GLM-4.5-Air used four-bit inference, while other models used default precision.Source: §C.1 · p. 17

C.2 Preprocessing

  • Preprocessing removes references and non-content artifacts, assembles documents in a fixed section order, and excludes papers whose text or formatting prevents reliable parsing.Source: §C.2 · p. 17

C.3 Model Configuration

  • Construction used Qwen3-30B-A3B-Instruct-2507 for generation and knowledge-graph summaries, GLiNER for disease extraction, and SapBERT for entity linking in EpiQAL-B.Generation temperature was set to 0; checker and difficulty-pool models and thresholds are also specified in the configuration.Source: §C.3 · p. 18

D Dataset Analysis

  • The dataset-composition analysis covers taxonomy-guided EpiQAL-A and EpiQAL-B, but excludes EpiQAL-C, whose supervision derives from paper structure.Source: D Dataset Analysis · p. 18

D.1 Class and Topic Distribution

Taxonomy-based subsets have uneven class and topic representation, while a verification example shows the checking group rejecting a reordered correct answer as an invalid distractor.

D.1 Class and Topic Distribution

  • Modeling, Methods and Evaluation is EpiQAL-A’s largest class at 213 instances, while Determinants and Exposures leads EpiQAL-B at 222; two classes together have fewer than 10 instances in each subset.The distribution reflects the prevalence of topics in the source corpus of neglected tropical disease research.Source: §D.1 · pp. 18–19
  • Zoonotic and One Health interfaces is the most frequent topic in both subsets, at 17.7% in EpiQAL-A and 22.8% in EpiQAL-B.Measurement, modeling, and severity-related topics follow; the remaining topics are grouped as Other.Source: §D.1 · p. 19

D.2 Verification Example

  • All nine checker votes rejected a reordered version of the correct species answer as an invalid distractor, recognizing that reordering did not make it substantively wrong.Source: §D.2 · p. 19

E Human Evaluation Details

Human ratings assess benchmark quality and agreement, while the appendix examines refinement, distractor design, model vulnerabilities, and one-shot prompting across the subsets.

E Human Evaluation Details

  • Answer Correctness scored at least 2.81 across subsets, while Distractor Quality scored 2.52–2.56; Reasoning Depth was 2.45 on both B and C.EpiQAL-C’s Answerability score was 2.78, supporting that its conclusion-reconstruction questions are solvable from the provided evidence.Source: E Human Evaluation Details · p. 20
  • Binary Agreement with annotators’ majority vote was 97.5% on A, 96.0% on B, and 98.3% on C; exact agreement was lower, especially on B, at 47.5%.Severe Disagreement was 2.5%, 2.0%, and 0.0% across A, B, and C, respectively.Source: E Human Evaluation Details · pp. 20–21

F.1 Stem Refinement Effect on Model Performance

  • Average Exact Match fell from 0.613 on original EpiQAL-C stems to 0.586 after two refinement iterations, though Iter 3 partly reversed the decline for three of four models.The authors attribute the recovery to over-expansion that can add domain knowledge to the rewritten stems, and set the maximum at three iterations.Source: §F.1 · p. 21

F.2 Distractor Analysis Details

  • Distractors target wrong entity or context, metric, and semantic near-miss errors in A; methodological, causal/logical, variable, and assumption errors in B; and evidence-dependency or interpretation errors in C.Source: §F.2 · p. 21

F.3 Per-Model Deception Rate

  • Semantic near-miss was the top vulnerability for 13 of 14 models on A; Variable confusion led for 8 of 14 on B, while Speculation/limitation led for nearly all models on C.Mistral-7B’s top category on C was External dependency.Source: §F.3 · p. 22

F.4 One-Shot Results

  • All 14 models improved EpiQAL-B Exact Match with one-shot prompting without CoT; Llama-3.2-3B gained 0.394, while effects on A and C were smaller and less uniform.One-shot plus CoT reduced Llama-3.3-70B’s EpiQAL-A score from 0.789 to 0.679.Source: §F.4 · p. 22

F.5 Role and Limitations of Retrieved Knowledge in EpiQAL-B

Retrieved knowledge rarely contributed to EpiQAL-B reasoning, and question-only performance varied by subset, underscoring the relevance of document evidence in these tasks.

F.5 Role and Limitations of Retrieved Knowledge in EpiQAL-B

  • Only 27 of 478 EpiQAL-B instances (5.6%) incorporated retrieved knowledge, and in all 27 cases it added no inferential link beyond the source passage.The retrieved information restated passage facts or supplied generic epidemiological context.Source: §F.5 · p. 25
  • The authors attribute retrieved knowledge’s limited contribution to dense study-level evidence in full-text papers and knowledge graphs’ coarser, entity-level facts.They suggest domain-specialized resources with study-level granularity or sparser source documents as directions for future work.Source: §F.5 · p. 25

F.6 Question-Only Baseline

  • Removing document input dropped average EpiQAL-A Exact Match from 0.78 to 0.12, while question-only performance remained higher on B and C, where options carry partial information.For two C models, full-document Exact Match was no higher than question-only performance; the authors note that Exact Match can understate passage contributions in these cases.Source: §F.6 · pp. 25–26

F.7 Input Ablation Controls

Supplementary controls show that performance often depends on the correct article, while robustness checks find stable rankings across several evaluation conditions. Results remain subset- and model-dependent, and some controls leave explanations unresolved.

F.7 Input Ablation Controls

  • On EpiQAL-A and EpiQAL-C, wrong articles drive accuracy near zero, with 82–100% of answers empty; on EpiQAL-B, correct passages add 0.196 EM for DeepSeek-V4-Flash-Thinking and 0.240 for GPT-4.1-nano versus Q-only.Shuffled inputs score below Q-only, while option-only does not exceed the random single-option reference; question-conditioned option elimination remains unisolated.Source: §F.7 · p. 26, Table 18 · p. 26, Table 19 · p. 27

F.8 Temperature Sensitivity; F.9 Multi-Run Stability

  • Rankings remain highly stable: temperature rank correlations exceed 0.99 on every subset, while no-CoT run standard deviations stay below 1% on each subset.CoT varies slightly more across runs, but model rankings remain unchanged.Source: §F.8 · p. 27, Table 20 · p. 27, §F.9 · p. 27

F.10 Reasoning Budget Sweep; F.11 Token Usage

  • Reasoning-budget settings have small, model-dependent accuracy effects; DeepSeek-V4-Flash-Thinking averages 0.769 EM versus GPT-5-mini’s 0.667, using roughly 1.4 times as many reasoning tokens.Provider controls are not directly comparable, and the token figures are descriptive rather than a cost-efficiency comparison.Source: §F.10 · p. 27, §F.11 · pp. 27–28, Table 22 · p. 28

F.12 Out-of-Distribution Evaluation; F.13 Cross-Source Generalization

  • Original and post-release rankings correlate at 0.94, 0.95, and 0.96 on EpiQAL-A, B, and C; IJE results also reproduce key subset and model patterns.The IJE check uses the same generator and checkers without a separate expert pass, so it does not establish taxonomy-wide coverage.Source: §F.12 · p. 28, §F.13 · pp. 28, 30, Table 24 · p. 29

F.14 Cross-Generator Robustness

  • Across generators, per-model ordering is largely preserved, EpiQAL-A remains near saturation, and EpiQAL-B and C remain harder under the alternative generator.This is a preliminary check using one alternative generator and a 50-article sample per subset, not a controlled generator ablation.Source: §F.14 · p. 30, Table 26 · p. 30

F.15 Construction and Evaluation Roles; F.16 Model Version Comparison

  • Construction-model family overlap may leave checker and difficulty-judge blind spots; meanwhile, V4-Flash-Thinking matches V3.2-Thinking on EpiQAL-A but trails on B and C.No model generates items later evaluated on them beyond the generator itself, which ranks ninth on EpiQAL-B.Source: §F.15 · p. 30, §F.16 · p. 30, Table 27 · p. 30

F.17 Detailed Discussion

  • Distractor vulnerability varies by subset: Semantic near-misses deceive models on A, reasoning-level distractors expose weaknesses on B, and Mistral-7B has the lowest rate on C.Top models retain category-specific weaknesses: DeepSeek-V3.2-Thinking’s Semantic near-miss deception is 13.6% on A, while its Causal reversal deception is 0.7% on C.Source: §F.17 · pp. 30–31

F.17 Detailed Discussion; G Prompts

  • GPT-5-mini’s ranking falls from third on A to mid-tier on B and C; CoT gains on B but has mixed effects on C, and abbreviated prompts appear in Tables 28–31.The authors describe CoT-related error amplification under partial evidence as affecting multiple models, not just one.Source: §F.17 · p. 31, G Prompts · p. 31

Related papers

Loading 2601.03471v4…