PaperlayerView exact revision on arXiv

Source-linked AI summary

SSE-Bio: A Structured Self-Evolving Agent with Agentic Retrieval Policy for Multi-Hop Biomedical Reasoning

Zhaohan Meng, Zaiqiao Meng, Siwei Liu, Hao Xu, Ke Yuan, Iadh Ounis

arXiv:2608.22132v1cs.CLcs.AIcs.CE

TL;DR

Biomedical multi-hop QA requires models to connect evidence across specialized intermediate entities, while existing systems offer limited control over retrieval and memory evolution. SSE-Bio combines structured short- and long-term memory with a trainable Proxy for selective retrieval and fine-grained template editing, optimized using decision-contrastive GRPO. Across three biomedical benchmarks, it consistently outperforms strong baselines, including a 6.56-point improvement over STELLA on BioHopR single-answer Bothcor.

  • Problem

    Biomedical multi-hop QA requires resolving evidence chains across diseases, drugs, proteins, and other entities, while existing agents have limited retrieval control and can suffer reasoning drift or incomplete answers.

  • Method

    SSE-Bio combines structured short-term state, long-term template memory, explicit Proxy-controlled retrieval, fine-grained template editing, and GRPO over alternative retrieval branches.

  • Results

    SSE-Bio consistently outperforms strong baselines on BioHopR, MedHop, and Humanity’s Last Exam: Biomedicine, improving over STELLA by 6.56 points on BioHopR single-answer Bothcor.

  • Takeaways & Limitations

    The results support structured retrieval control and local memory evolution as effective components for biomedical multi-hop reasoning.

  • Takeaways & Limitations

    SSE-Bio remains dependent on retrieval quality and long-term template memory, which may yield mismatched evidence or redundant and outdated templates over time.

Abstract

from arXiv · show

Biomedical multi-hop question answering (QA) requires models to connect evidence across intermediate entities such as diseases, drugs, proteins, and phenotypes. Existing agents typically rely on static retrieval workflows or coarse-grained prompt rewriting, which can lead to instruction drift when reasoning procedures need to be updated. We propose SSE-Bio, a structured self-evolving agent with an agentic retrieval policy for multi-hop biomedical reasoning. Instead of globally rewriting agent instructions, SSE-Bio maintains a structured state, selectively retrieves knowledge triplets and prior templates through a trainable proxy policy, and improves its reasoning memory through fine-grained template editing. To optimise retrieval decisions, we introduce a proxy-training strategy based on group relative policy optimization, where the proxy is improved through decision-contrastive groups over alternative retrieval choices. Experiments on three biomedical multi-hop QA benchmarks show that SSE-Bio consistently outperforms existing baselines, achieving an improvement of 6.56 absolute points over the strongest self-evolving baseline on BioHopR.

1 Introduction

Biomedical multi-hop QA requires resolving chains across specialized biomedical entities, while existing retrieval and self-evolving agents provide limited control over evidence selection and memory updates. SSE-Bio addresses these challenges with structured memory, trainable retrieval control, fine-grained template editing, and GRPO-based proxy optimization.

  • Motivation: Biomedical multi-hop QA connects diseases, drugs, proteins, and other entities through complex relations, making unresolved bridge entities a major reasoning challenge.Multi-answer questions also require avoiding premature stopping after finding only a subset of valid answers.
  • Limitations of Existing Systems: Existing retrieval-augmented and agentic systems can leave intermediate entities unresolved and provide limited control over when and what evidence is retrieved.These limitations can contribute to reasoning drift and incomplete multi-answer responses.
  • Results: SSE-Bio consistently outperforms strong baselines on BioHopR, MedHop, and Humanity’s Last Exam: Biomedicine.On BioHopR, it improves over STELLA by 6.56 points on single-answer Bothcor and achieves the best multi-answer performance.
  • SSE-Bio: SSE-Bio unifies short-term structured state tracking, long-term template memory, and explicit retrieval management in one agentic framework.The Manager plans from the current state, the Dev agent executes with retrieved triplets, and the Critic supplies structured feedback.
  • SSE-Bio: Fine-grained template editing updates reusable memory locally from successful trajectories instead of rewriting whole templates.The design is intended to make memory evolution more controllable and auditable while reducing unnecessary changes.
  • Proxy Optimization: SSE-Bio trains a Proxy with SFT and GRPO to compare alternative retrieval branches and select decisions using answer-grounded supervision.The optimization couples decision-contrastive trajectory generation with rewards reflecting correctness and evidence support.

2 Related Work

Related work spans biomedical retrieval-augmented generation, multi-agent reasoning, and self-evolving systems. SSE-Bio differs by combining state-aware retrieval control with structured long-term memory and local template evolution.

  • Biomedical RAG Models: Biomedical RAG models incorporate external knowledge, with approaches such as MedGraphRAG organizing medical knowledge into graphs for relation-aware retrieval.The related-work discussion also identifies systems that retrieve medical evidence to support answer generation.
  • Biomedical Multi-agent Systems: Biomedical multi-agent systems support complex reasoning through specialized roles for step-wise planning, execution, feedback, and broader tool use.Examples include collaborative specialist agents, knowledge-graph agents, and systems with broader biomedical tools.
  • Self-evolving Agents: Self-evolving systems accumulate experience across questions, but STELLA offers limited control over retrieval timing and content while rewriting templates from successful cases.This positions retrieval policy and memory-update granularity as distinctions between prior self-evolving agents and SSE-Bio.
  • SSE-Bio’s Distinction: SSE-Bio couples state-aware retrieval control with structured long-term memory that evolves through fine-grained local editing.Its framework is presented as an alternative to fixed retrieval pipelines and unconstrained template rewriting.

3 Methodology

SSE-Bio formulates biomedical multi-hop reasoning as an iterative process that combines structured state tracking, retrieval control, agentic feedback, and evolving long-term memory. Its Proxy learns retrieval decisions through supervised initialization and action-level GRPO using answer and evidence-grounding rewards.

  • Iterative Agentic Process: SSE-Bio uses a Manager, Proxy, Dev agent, and Critic to iteratively plan, retrieve, execute, assess, and refine biomedical reasoning.The Manager updates structured state after failed reasoning and evolves long-term memory after successful reasoning.
  • Structured Memory: The structured state records task type, information gaps, critic feedback, and retrieval state to ground planning and retrieval in explicit reasoning progress.This short-term memory is updated across reasoning rounds.
  • Structured Memory: Long-term memory stores reusable templates containing task patterns, reasoning flows, verification criteria, and tool-use policies from successful cases.Templates provide reusable reasoning experience across questions.
  • Agentic Retrieval Policy: The Proxy selects among four retrieval actions: neither source, knowledge triplets only, templates only, or both.Knowledge-triplet retrieval uses the information gap, whereas template retrieval uses the task type; both use BiCA and return filtered top-K results.
  • Proxy Policy Training: The Proxy is the only trainable component, learning a policy that selects retrieval actions conditioned on the current structured reasoning state.The Manager, Dev, Critic, retrievers, and reasoning environment remain fixed.
  • Proxy Policy Training: SFT initializes retrieval pseudo-labels from the highest-reward alternative branch, after which GRPO refines the Proxy using decision-contrastive trajectory groups.Branches are expanded from the same reasoning state and pruned when they violate schemas, leave information gaps unresolved, or produce unsupported bridge entities.
  • Proxy Policy Training: The composite reward combines final-answer correctness with behavioral evidence support for key reasoning steps.Behavioral support requires retrieved or template-supported evidence for bridge resolution and answer verification.
  • Proxy Policy Training: GRPO compares discrete retrieval actions across trajectory groups using group-relative advantages, probability ratios, and regularization against a frozen SFT reference policy.The optimization operates at the action level because the Proxy outputs retrieval actions rather than tokens.

4 Experimental Setup

SSE-Bio is evaluated on BioHopR with paired single-answer and multi-answer metrics, then tested zero-shot for transfer on MedHop and HLE. The setup compares it with broad non-agentic, retrieval-augmented, agent-based, and self-evolving baselines while fixing the reasoning backbones.

  • Datasets: BioHopR contains 7.63K paired instances, uses a 7:3 train/test split, and supports both single-answer and multi-answer evaluation.The split yields approximately 5.34K training pairs and 2.29K test pairs.
  • Datasets: MedHop and Humanity’s Last Exam: Biomedicine provide zero-shot cross-benchmark generalization tests.SSE-Bio is trained on BioHopR and transferred directly to both datasets.
  • Baselines: The baseline suite spans general-purpose and medical LLMs, retrieval-augmented generation, agent-based systems, and self-evolving agents.These categories cover non-agentic, retrieval-augmented, and self-evolving reasoning settings.
  • Evaluation: BioHopR evaluation reports PrecH1, PrecH2, Bothcor, and Bothwr for linked-hop precision and paired joint correctness or incorrectness.Higher values are better for PrecH1, PrecH2, and Bothcor, while lower values are better for Bothwr.
  • Cross-Benchmark Evaluation: Figure 2 compares zero-shot accuracy with agentic baselines on HLE and MedHop and compares proxy-backbone performance on BioHopR.The figure reports percentage-based comparisons across its three panels.
  • Implementation: Evaluation fixes Gemini-2.5-Pro as Manager and Critic and Claude-4.5-Sonnet as Dev, isolating retrieval-policy learning from reasoning-backbone changes.Template memory is frozen during evaluation, with no held-out test instances used to create or edit templates.

5 Results and Analysis

SSE-Bio consistently outperforms strong baselines across biomedical reasoning evaluations, with gains supported by trainable retrieval control and structured self-evolution. Ablations and relation-type analyses indicate that its components contribute beyond fixed retrieval schedules and coarse rewriting.

  • Overall performance: SSE-Bio consistently outperforms strong baselines across BioHopR, MedHop, and HLE evaluations.It achieves the best performance across BioHopR metrics and reaches 28.4% accuracy on HLE.
  • Effect of the Proxy: Removing the Proxy lowers single-answer Bothcor from 16.52 to 10.62 and raises Bothwr from 47.93 to 55.00, indicating retrieval control is important.The learned Proxy also outperforms fixed retrieval schedules, and GRPO improves Bothcor from 15.08 to 16.52 with the Qwen-72B Proxy.
  • Structured self-evolution: Replacing structured state and fine-grained editing with free-form state and coarse rewriting lowers single-answer Bothcor from 16.52 to 11.72.Using only structured state or only fine-grained editing recovers part of the gap, reaching 13.86 and 14.52, respectively.
  • Retrieval headroom: Removing triplets or prior templates lowers single-answer Bothcor to 13.64 and 14.26, showing that factual evidence and reusable reasoning priors are complementary.Boundary analysis further evaluates improvement available from stronger evidence and template selection.
  • Relation-type robustness: SSE-Bio improves over STELLA across all listed BioHopR relation types, including Disease:Drug:Phenotype PrecH2 gains from 22.83 to 34.26 in single-answer evaluation.The same relation improves from 10.58 to 16.82 on multi-answer PrecH2.

6 Conclusions

SSE-Bio combines state-aware retrieval control through a trainable Proxy with structured templates and fine-grained memory editing. Across three benchmarks, it consistently outperforms strong baselines, including a 6.56-point BioHopR gain over STELLA on single-answer Bothcor.

  • SSE-Bio combines trainable state-aware retrieval control with structured templates and fine-grained editing for controllable biomedical multi-hop reasoning.The design avoids fixed retrieval pipelines and unconstrained self-rewriting.
  • SSE-Bio consistently outperforms strong baselines across three benchmarks and improves single-answer BioHopR Bothcor over STELLA by 6.56 points.It also achieves the best performance under multi-answer settings.

Limitations

The paper identifies limitations involving retrieval quality, long-term memory growth, Proxy scaling, fixed agent components, and transfer across agent backbones.

  • SSE-Bio remains dependent on retrieved triplets and prior templates, while oracle analysis indicates stronger evidence and template matching could improve performance.The paper also notes that memory may accumulate redundant or outdated templates, and larger Proxy backbones improve retrieval control.
  • SFT labels and GRPO rewards are induced within one fixed agent environment, so transfer of the learned retrieval policy across different agent backbones remains untested.

A.1 Computing Hardware

All experiments used a compute node equipped with six NVIDIA GH200 Grace Hopper superchips, providing sufficient memory and computational capacity for the reported training and inference settings.

  • All experiments were conducted on a node with six NVIDIA GH200 Grace Hopper superchips.The hardware supported the training and inference settings used in the work.

A.2 Examples of Structured Memory

SSE-Bio uses structured short-term state and long-term template memory to support retrieval control, planning, and reusable reasoning. The examples show compact state fields and localized template updates that preserve unaffected fields.

  • Memory design: The short-term state captures current reasoning status for retrieval control and replanning, while long-term templates store reusable reasoning priors for future questions.Table 4 defines the fields used by both memory types.
  • Short-term structured state: Figure 3 organizes task type, information gap, Critic feedback, and retrieval state into a compact representation for next-step planning.These fields support retrieval control during iterative reasoning.
  • Long-term template update: Figure 4 illustrates fine-grained template editing after a successful trajectory while keeping the task type fixed.The update changes selected fields rather than rewriting the entire template.
  • Template example: The template example includes a Disease–Protein–Drug multi-answer task type and a verification criterion requiring each predicted drug to link to at least one protein.The criterion makes the required evidence connection explicit.

A.3 Prompt and API Sensitivity

Sensitivity analyses on BioHopR test whether SSE-Bio depends on specific system prompts or commercial API backbones. The reported results indicate small prompt-related drops and moderate but non-catastrophic API degradation, supporting stability under both variations.

  • Prompt sensitivity: SSE-Bio remains reasonably stable when Manager, Dev, and Critic system prompts vary across Empty, Basic, and role-specific settings.All other components, including schemas, retrieval actions, template format, model APIs, and the trained Proxy, remain fixed.
  • Prompt sensitivity: Replacing role-specific prompts with Empty or Basic prompts causes only small drops, suggesting performance is not driven by brittle system-prompt wording.This analysis uses the BioHopR test split.
  • API sensitivity: Alternative Manager and Critic APIs produce moderate but not catastrophic degradation while the Dev agent, Proxy, retriever, memory, prompts, and protocol remain fixed.The comparison isolates the effect of changing the Manager and Critic backbones.

A.4 Efficiency Details

SSE-Bio improves paired correctness over agentic baselines, but uses more tokens and adds moderate per-example inference overhead alongside a one-time Proxy training cost.

  • Efficiency comparison: 16.52 Bothcor exceeds Biomni’s 10.45 and STELLA’s 9.96 on the BioHopR test split.SSE-Bio uses 28.3M tokens, compared with 23.6M for Biomni and 21.2M for STELLA.
  • Architecture: SSE-Bio uses four agents: a Manager, Proxy, Dev agent, and Critic, with only the Proxy trainable.The Manager and Critic use Gemini-2.5-Pro, the Dev agent uses Claude-4.5-Sonnet, and the Proxy uses open-weight backbones.
  • Inference cost: SSE-Bio adds 3.1K tokens, 1.8 calls, 5.4 seconds, and approximately $0.012 per BioHopR example relative to STELLA.The local Proxy adds no commercial API calls, and its latency is included in the end-to-end measurement.
  • Training cost: The one-time Proxy training cost is 138 GPU-hours and is not incurred during test-time inference.
  • Memory evolution: Fine-grained editing reduces average changed fields from 3.28 to 1.46 and improves unchanged-field preservation from 64.0% to 88.0%.Unsupported new instructions decrease from 12.0% to 4.0%, while contradictory instructions decrease from 7.0% to 2.0%.
Loading 2608.22132v1…