PaperlayerView exact revision on arXiv

Source-linked AI summary

Generating Biomedical Fact-Checking Reports with RL-Enhanced Agentic Search

Jiongxiao Wang, Dingli Ma, Chaoqun Ni

arXiv:2608.23811v1cs.AI

TL;DR

Biomedical fact-checking needs rigorous evidence interpretation and explanations beyond isolated labels. The paper introduces BioCheck Agent with PubMed agentic search and EG-GRPO, which improves label prediction and report evidence quality while reducing hallucinations. Its current scope and training data limit coverage of complex real-world, multimodal misinformation.

  • Problem

    Biomedical fact-checking requires rigorous scientific-evidence interpretation and comprehensive justification, while existing retrieve-then-verify systems often produce isolated labels with limited explanatory depth.

  • Method

    BioCheck Agent generates structured reports from PubMed agentic searches, and EG-GRPO trains it with rewards for label alignment, advanced search, evidence quality, groundedness, and hallucination reduction.

  • Results

    Across SciFact and HealthFC, EG-GRPO outperformed baselines in label prediction; versus Qwen3.5-4B, it improved SciFact accuracy by 9.95%, increased EQS by 3.7%, and reduced EHR by 19.63%.

  • Takeaways & Limitations

    Structured biomedical fact-checking reports provide contextualized evidence and conclusions intended to better support human decision-makers than isolated prediction labels.

  • Takeaways & Limitations

    The agent primarily handles isolated atomic claims, whereas real-world misinformation often appears in complex narratives and multimodal contexts.

Abstract

from arXiv · show

Automated fact-checking is essential for ensuring the reliability of public health information, yet the biomedical domain poses unique challenges. Validating biomedical claims requires rigorous interpretation of scientific literature, assessment of retrieved evidence, and comprehensive justification toward the conclusion. Although Large Language Models (LLMs) enhanced by Retrieval-Augmented Generation (RAG) and agentic search perform automated fact-checking in a retrieve-then-verify paradigm, current methods still output isolated prediction labels, lacking explanatory depth and offers limited utility for human understanding. To bridge this gap, we introduce an LLM-based agent named BioCheck Agent that generates structured biomedical fact-checking reports with agentic search. Rather than merely outputting supported or refuted labels, our agent synthesizes final conclusions with retrieved evidence and rigorous analysis. To ensure domain-specific accuracy, BioCheck Agent exclusively searches high-quality scientific literature in PubMed, utilizing advanced Boolean search operators. Recognizing that direct prompting often results in hallucinations and low-quality reports, especially for lightweight open-source models, we further propose the Evidence-Grounded Group Relative Policy Optimization (EG-GRPO) to perform reinforcement learning on BioCheck Agent with a task-specific reward that incentivizes advanced search behavior and high-quality evidence retrieval while penalizing hallucinations. Our experimental results show that compared to the base model Qwen3.5-4B, BioCheck Agent with EG-GRPO improves label prediction accuracy on SciFact by 9.95%. Furthermore, it achieves a 3.7% higher evidence quality score and a 19.63% lower evidence hallucination rate, demonstrating its ability to generate biomedical fact-checking reports with improved accuracy and quality.

1 Introduction

Biomedical fact-checking must assess scientific evidence and explain conclusions, not merely predict labels. BioCheck Agent addresses this gap with PubMed agentic search and EG-GRPO, which improves prediction and report quality.

  • Biomedical misinformation can affect health behavior, public trust, and evidence-based public health interventions.
  • Biomedical fact-checking requires domain expertise, credible-evidence assessment, explanation generation, and attention to evidence strength and certainty.
  • BioCheck Agent generates structured reports by retrieving PubMed literature, analyzing supporting and refuting evidence, and producing a justified conclusion.It decomposes complex claims and links core entities with Boolean operators to construct search queries.
  • EG-GRPO uses task-specific rewards for label alignment, Boolean search, evidence quality and groundedness, while penalizing hallucinated evidence.
  • 9.95% higher prediction accuracy on SciFact and 7.03% higher accuracy on HealthFC were achieved versus the pre-RL base model.EG-GRPO also produced a 3.7% higher Evidence Quality Score and a 19.63% lower Evidence Hallucination Rate on SciFact than Qwen3.5-4B.

2 Related Work

Automated fact-checking has evolved from supervised classification toward open-world retrieval and agentic evidence gathering. Reinforcement learning with verifiable rewards, including GRPO, is increasingly used for agent tasks with explicit answers.

  • Traditional automated fact-checking trained supervised classifiers and often relied on pre-annotated dataset evidence.
  • LLM and RAG methods shifted fact-checking toward open-world retrieve-and-verify workflows using external databases and knowledge graphs.
  • LLM-based agents use external tools such as search engines for more comprehensive and real-time evidence retrieval.
  • Reinforcement learning with verifiable rewards is well suited to agent tasks with explicit answers, and GRPO has improved performance across varied tasks.

3 Method

BioCheck Agent uses iterative PubMed search to produce structured fact-checking reports, while EG-GRPO trains it with rewards for outcomes, search quality, evidence quality, and hallucination avoidance.

  • 3.1 BioCheck Agent: BioCheck Agent iteratively reasons, invokes a predefined search tool, and decides whether to continue searching or generate a report from collected evidence.LangGraph orchestrates agent nodes, while LangChain invokes the language model with customized tools.
  • 3.1 BioCheck Agent: PubMed search retrieves relevant PMIDs and returns concatenated titles and abstracts, with at most 5 papers per search and 5 iterative searches.The tool uses the PubMed API and limits responses because of context length and inconsistent full-text availability.
  • 3.1 BioCheck Agent: The structured report contains supporting evidence, refuting evidence, a summary with consensus checking and justification, and a conclusion labeled SUPPORTED or REFUTED.Evidence entries include PMIDs, verbatim top-sentence quotes from abstracts, and relevant context.
  • 3.2 Evidence-Grounded GRPO: Composite Evidence Reward scores retrieved claim-evidence pairs while retaining both supporting and contradicting evidence for critical retrieval from both sides.Evidence contributes only when its abstract-level probability score exceeds 0.5; otherwise its score is set to 0.0.
  • 3.2 Evidence-Grounded GRPO: EG-GRPO optimizes the policy using groups of sampled trajectories and advantages computed from reports’ relative rewards within each group.The objective uses importance sampling, clipping, and a KL penalty between the policy and reference models.

4 Experiment

The experiments evaluate BioCheck Agent and EG-GRPO on SciFact and HealthFC for label prediction and report generation. EG-GRPO improves label prediction and report quality relative to baseline approaches, although report-generation benchmarks remain limited.

  • Experimental Settings: Experiments use SciFact for training and evaluation, while HealthFC is used only for evaluation because it lacks explicit training, validation, and test splits.
  • Experimental Settings: Evaluation covers label accuracy, macro precision, macro recall, macro F1, Evidence Quality Score, and Evidence Hallucination Rate.Higher EQS and lower EHR indicate greater report quality.
  • Experimental Settings: Existing literature lacks established benchmarks for evaluating fact-checking report generation, so report-generation comparisons focus mainly on the base model, GRPO, and EG-GRPO.
  • Main Results: EG-GRPO enables Qwen3.5-4B to surpass GPT-5.2 on SciFact, with an 8.9% improvement in prediction accuracy.Baseline GRPO provides limited gains and a worse F1 score on HealthFC.
  • Main Results: EG-GRPO produces higher EQS and significantly reduced EHR than the base model and baseline GRPO on SciFact and HealthFC.Baseline GRPO marginally reduces EHR but yields lower EQS.
  • Ablation Study: Using GPT-4o as the judge changes EQS values but preserves the trend that EG-GRPO outperforms the base model and baseline GRPO.

5 Conclusion

The paper advances biomedical fact-checking from isolated label prediction to structured reports grounded in PubMed evidence. BioCheck Agent with EG-GRPO achieves accurate label prediction and higher-quality report generation to assist human assessment of health misinformation.

  • BioCheck Agent generates structured biomedical fact-checking reports through PubMed agentic search instead of producing isolated prediction labels.
  • EG-GRPO uses task-specific rewards to optimize the groundedness and quality of evidence in generated reports.
  • Experiments demonstrate accurate label prediction and high-quality report generation for biomedical fact-checking.

6 Limitation

BioCheck Agent is currently scoped to isolated atomic claims, while real-world misinformation often involves complex narratives and multimodal contexts. Training data are also limited and differ from general public-health distributions.

  • BioCheck Agent primarily handles isolated atomic claims rather than complex narratives across platforms or multimodal contexts.Extending verification to such environments remains a future challenge.
  • The SciFact training set contains around 500 labeled examples, lacks frequent real-world NEI cases, and differs from general public-health data distributions.This distribution shift is associated with more modest improvement on HealthFC than on SciFact.

A Implementation Details of BioCheck Agent

BioCheck Agent uses a LangGraph workflow to search PubMed, iteratively refine evidence retrieval, and produce a structured fact-checking report with a final label.

  • Agent architecture: BioCheck Agent is implemented as a LangGraph workflow that connects LLM, tool, and intervention nodes.Conditional edges route outputs with tool calls to search execution, while intervention ends searching after the maximum limit.
  • Search protocol: The agent analyzes claims into entities, relations, and context before constructing Boolean PubMed queries with expanded aliases and MeSH terms.It can relax constraints, incorporate newly discovered terminology, and search actively for contradictory evidence.
  • Report format: The report records supporting and refuting articles using PMIDs, abstract quotations, and contextual conditions before summarizing the evidence.Its summary includes support and refutation reasons, a consensus and context check, and a final justification.
  • Decision output: The conclusion outputs SUPPORTED or REFUTED, while insufficient evidence can trigger a NOT ENOUGH INFORMATION result.The intervention prompt forces final report generation when the search limit is reached.
  • Evidence scoring: Evidence confidence is scored from the LLM’s next-token probabilities for supported, refuted, and insufficient-information choices.The corresponding probabilities are computed as s = exp(z) from token logprobs.

C Training Details

EG-GRPO training uses task-specific reward components and selects the validation-best checkpoint from a 70-step training process.

  • Reward design: EG-GRPO combines outcome, search, evidence, and hallucination terms in a task-specific reward.The Biocheck Reward is formulated as Routcome + β1Rsearch + β2Revidence − β3Rhallucination.
  • Reward design: The reward weights are β1 = 0.5 and β2 = β3 = 1.0, with the RCER saturation factor set to α = 1.0.These settings control the relative contributions of search, evidence, and hallucination terms.
  • Training schedule: Table 3 reports the key hyperparameters used for EG-GRPO training.The section identifies the table as the source of the configured reinforcement-learning hyperparameters.
  • Training schedule: 70 training steps are run, and the step-60 checkpoint is selected because it achieves the highest average validation reward.The training curve is shown in Figure 4.

D Baseline Methods

The evaluation compares BioCheck Agent with retrieval, web-search, and outcome-only baselines spanning no retrieval, corpus retrieval, iterative search, and PubMed agent workflows.

  • No Retrieval: The No Retrieval baseline classifies claims using only the LLM’s internal biomedical knowledge.It directly prompts the model to produce one of three labels.
  • CER: CER retrieves 20 PubMed abstracts with BM25, reranks their sentences using a biomedical encoder, and retains up to 60 evidence sentences per claim.The implementation then uses an LLM for evidence-grounded reasoning and zero-shot decision-making.
  • FIRE: FIRE iteratively verifies claims with live Google Search evidence and can terminate early when queries or retrieved evidence become highly similar.Its loop runs for at most five steps, with up to ten retries for malformed outputs and a cosine-similarity threshold of 0.9.
  • PMSearch Agent: PMSearch Agent shares BioCheck Agent’s LangGraph structure but outputs only a predicted label after PubMed searches.It retrieves five papers per search and permits at most five search iterations.
  • GRPO comparison: Standard GRPO provides an outcome-only comparison by rewarding whether the generated report’s final conclusion matches the ground-truth label.Its training parameters are kept identical to EG-GRPO for comparison.

E Examples of Generated Fact-Checking Report

The examples compare reports generated by the Qwen3.5-4B base model and its EG-GRPO-optimized version on a supported biomedical claim.

  • Generated reports: The examples present reports from BioCheck Agent using Qwen3.5-4B before and after EG-GRPO optimization.They are intended to illustrate differences in generated biomedical fact-checking reports.
  • Example claim: The example claim concerns whether azo food dyes cause attention deficit disorder and hyperactivity in children, with a ground-truth label of SUPPORTED.Warnings identify mismatched PMIDs and low-quality evidence within the examples.

Fact-Checking Report Generated by BioCheck Agent with the Base Model

The base-model report presents conflicting evidence: several controlled studies associate azo dyes with hyperactivity, while other trials find no significant effects or effects limited to sensitive children. It therefore concludes that a universal causal claim cannot be definitively supported.

  • Supporting Evidence: Multiple controlled studies associate artificial food colorings with increased hyperactivity or ADHD symptoms in children.The report cites meta-analytic, randomized, and physiological evidence supporting an association.
  • Refuting Evidence: Several double-blind placebo-controlled studies found no significant association between food colorings and behavior in general-population children.The report highlights the Hong Kong trial and Feingold diet review as contradictory evidence.
  • Consensus & Context Check: Some reported effects appear limited to subgroups of sensitive or hyperactive children rather than children universally.The report describes variability by susceptibility and population.
  • Consensus & Context Check: The evidence is conflicting and inconsistent across studies.The report’s consensus check cautions against interpreting supporting findings as a blanket effect.
  • Conclusion: The report concludes that azo food dyes universally causing ADHD or hyperactivity in children cannot be definitively supported.Its final conclusion is REFUTED, while acknowledging that some studies show associations.

Fact-Checking Report Generated by BioCheck Agent with EG-GRPO

The EG-GRPO report finds evidence supporting behavioral effects of azo dyes in susceptible children, especially at higher doses, while rejecting a universal effect across all children. It concludes that the claim is supported when understood as a conditional risk affecting a subset of children.

  • Supporting Evidence: A systematic review reports that artificial azo dye food colourants cause disturbed behaviour in children.The cited review draws on observational and animal-model studies.
  • Supporting Evidence: Meta-analytic modeling estimated an overall effect size of 0.283 (95% CI, 0.079 to 0.488).After excluding the smallest and lowest-quality trials, the estimate fell to 0.210 (95% CI, 0.007 to 0.414).
  • Supporting Evidence: A double-blind placebo-controlled study reported dose-response behavioral changes associated with tartrazine in some children.Significant reactions were observed across six tested dose levels from 1-50mg.
  • Refuting Evidence: Typical population dietary intakes fall below the challenging doses associated with detectable effects.The report states that children and teenagers did not reach doses used in the relevant challenge study.
  • Supporting Evidence: Multiple controlled studies and a meta-analysis support behavioral changes in a subgroup of children exposed to specific azo dyes.The report describes irritability, restlessness, and sleep disturbance, particularly at doses of 50mg or more.
  • Refuting Evidence: Behavioral effects are not universal, because responses appear concentrated among sensitive or reactive children.The report identifies individual susceptibility, genetic factors, and co-exposure as possible moderators.
  • Consensus & Context Check: The evidence is best characterized as a conditional association and risk factor for a subset of children, not a universal cause of ADHD or hyperactivity.The final justification preserves support for effects at higher doses while rejecting universal causation.
  • Conclusion: The report concludes SUPPORTED for the claim that azo food dyes can cause behavioral changes in susceptible children.The conclusion qualifies the support by excluding a universal effect applicable to all children.
Loading 2608.23811v1…