PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentEHR: Advancing Autonomous Clinical Decision-Making via Retrospective Summarization

Yusheng Liao, Chuan Xuan, Yutong Cai, Lina Yang, Zhe Chen, Yanfeng Wang, Yu Wang

arXiv:2601.13918v1cs.CL

TL;DR

Autonomous EHR navigation remains limited by curated inputs, simplified retrieval, and information loss in long-context clinical reasoning. The paper introduces AGENTEHR and RETROSUM, combining a realistic benchmark with retrospective summarization and evolving experience. RETROSUM reports performance gains of up to 29.16% over existing baselines, while the authors identify dataset and modality boundaries.

  • Problem

    Existing EHR applications often depend on manual curation, while unidirectional summarization can lose cross-temporal clinical correlations needed for long-context reasoning.

  • Method

    The paper introduces AGENTEHR for multi-step reasoning over raw EHRs and RETROSUM, which retrospectively re-evaluates interaction history while using accumulated experience.

  • Results

    29.16% performance gains over existing baselines are reported for RETROSUM in extensive experiments.

  • Takeaways & Limitations

    The results support retrospective mechanisms as important for clinical-agent reasoning and decision-making tasks in EHRs.

  • Takeaways & Limitations

    Evaluation relies primarily on MIMIC-IV and MIMIC-III from one medical center, and the framework does not directly analyze raw medical imaging or high-frequency waveforms.

Abstract

from arXiv · show

Large Language Models have demonstrated profound utility in the medical domain. However, their application to autonomous Electronic Health Records~(EHRs) navigation remains constrained by a reliance on curated inputs and simplified retrieval tasks. To bridge the gap between idealized experimental settings and realistic clinical environments, we present AgentEHR. This benchmark challenges agents to execute complex decision-making tasks, such as diagnosis and treatment planning, requiring long-range interactive reasoning directly within raw and high-noise databases. In tackling these tasks, we identify that existing summarization methods inevitably suffer from critical information loss and fractured reasoning continuity. To address this, we propose RetroSum, a novel framework that unifies a retrospective summarization mechanism with an evolving experience strategy. By dynamically re-evaluating interaction history, the retrospective mechanism prevents long-context information loss and ensures unbroken logical coherence. Additionally, the evolving strategy bridges the domain gap by retrieving accumulated experience from a memory bank. Extensive empirical evaluations demonstrate that RetroSum achieves performance gains of up to 29.16% over competitive baselines, while significantly decreasing total interaction errors by up to 92.3%.

1 Introduction

AGENTEHR addresses the gap between curated EHR experiments and realistic clinical decision-making by requiring multi-step reasoning over raw databases. RETROSUM uses retrospective summarization and evolving experience to preserve clinical reasoning continuity.

  • Existing EHR methods often rely on manual curation, sidestepping data noise and long-context processing in realistic clinical scenarios.
  • Prior agent-based EHR systems largely perform query rewriting or factual retrieval, limiting their use of advanced reasoning for complex clinical analysis.
  • AGENTEHR requires agents to seek information across multiple steps, synthesize it through clinical reasoning, and predict diagnoses or treatment plans.
  • Unidirectional summarization can sever cross-temporal EHR connections, fracturing the reasoning continuity needed for precise diagnosis.
  • RETROSUM combines retrospective summarization, which captures latent correlations, with an evolving strategy that accumulates experience to bridge the domain gap.
  • 29.16% performance gains over existing baselines are reported for RETROSUM in extensive experiments.

2 AGENTEHR Benchmark

AGENTEHR is an evaluation framework for autonomous agents performing interactive clinical reasoning and decision-making in real-world EHR data. It spans multiple datasets, distribution settings, clinical tasks, and database-access tools.

  • AGENTEHR evaluates EHR-based interactive reasoning and clinical decision-making in raw, high-noise databases.
  • AGENTEHR is designed to bridge idealized experimental settings and authentic medical scenarios through multi-step reasoning in raw EHR databases.
  • The benchmark uses MIMIC-IV and MIMIC-III, organized into Common, Rare, and systemic-shift subsets for robustness and generalization evaluation.
  • The six clinical tasks are Diagnoses, Labevents, Microbiology, Prescriptions, Procedures, and Transfers.
  • A Model Context Protocol toolbox provides over 19 specialized tools, including temporal filtering, keyword search, fuzzy matching, and direct SQL execution.

3 Methods

RETROSUM supports long-horizon clinical reasoning by retrospectively re-evaluating interaction history while preserving full context, then using accumulated experience to guide summarization and action selection.

  • Task Formulation: RETROSUM frames clinical agents as systems that interact with EHR databases through iterative actions and observations until producing a clinically instructed answer set.Each query includes a patient, timestamp, and clinical instruction; the interaction history accumulates action–observation pairs.
  • RETROSUM: RETROSUM addresses unidirectional summarization’s loss of latent EHR correlations and resulting fragmentation of multi-turn reasoning.The framework is motivated by heterogeneous tables and cross-temporal dependencies that can be severed by incremental compression.
  • Retrospective Summarization: Retrospective summarization periodically partitions distant and recent history, then re-evaluates both to update the summary in light of new findings.The summarizer is triggered every w steps and conditions on Hdist and Hrec, enabling previously ambiguous correlations to be reconsidered.
  • Retrospective Inference: Unlike summary-replacement methods, RETROSUM augments the complete interaction history with the latest retrospective summary so the actor retains the full reasoning chain.The summary acts as a high-level cognitive map while the actor continues accessing raw history.
  • Evolving Strategy: The evolving strategy extracts procedural and salience heuristics into an experience memory bank, retrieves similar experiences for new patients, and guides both framework modules.Retrieved experience helps the summarizer filter noise and the actor adopt learned reasoning strategies without updating model parameters.
  • Evolving Strategy: Conditioning on retrieved experience improves noise filtering in the summarizer and supports the actor’s use of established reasoning strategies.The paper describes this joint guidance as enhancing overall robustness.

4 Experiments

Experiments evaluate RETROSUM across benchmark settings, backbones, subsets, and ablations. The results consistently favor RETROSUM, including under distribution shifts and when combining retrospective and evolving components.

  • Main Results: RETROSUM demonstrates consistent superiority across the reported experimental settings, including configurations without the evolving mechanism.The comparison also identifies instability in the ReSum baseline across backbone capabilities.
  • Main Results: On Grok-4.1-fast, ReSum scores 0.2237 versus 0.2501 for standard ReAct, while RETROSUM retains robust improvements across backbone capabilities.The authors attribute ReSum’s weakness on stronger models to information loss from unidirectional compression.
  • Main Results: The evolving RETROSUM variant achieves the highest average score of 0.2880, whereas Reasoning Bank and ReflectTool show limited or unstable benefits.The paper suggests retrospective context compression makes retrieved experiences more useful for complex context synthesis.
  • Cross-subset Validation: RETROSUM remains robust across MIMIC-IV-Rare and MIMIC-III distribution shifts, maintaining state-of-the-art performance where ReSum degrades under systemic format changes.The cross-subset results include low-prevalence diagnoses in MIMIC-IV-Rare and denser, noisier records in MIMIC-III.
  • Ablation Studies: In ablations on Qwen3-30B-A3B, Actor-only and Summarizer-only retrospective variants score 0.1876 and 0.1798, the combined base variant scores 0.1957, and evolving optimization peaks at 0.2117.These results quantify contributions from the two retrospective applications and the subsequent evolving component.

5 Analysis

The analysis shows that RETROSUM’s retrospective mechanism supports performance, error reduction, and reasoning efficiency across summarization settings, context limits, and test-time scaling.

  • Retrospective mechanism: At small intervals (w ≤5), Act-Only maintains immediate reasoning coherence, whereas at large intervals (w ≥15), Sum-Only retains distant critical information.Combining both complementary roles enables RETROSUM to outperform ReSum across summarization frequencies.
  • Error analysis: 92.3% reduction in total errors is achieved by Evolved RETROSUM compared to ReasoningBank.Failed trajectories are dominated by No Candidate Tool and repetitive-behavior errors; RETROSUM reduces total errors across categories.
  • Efficiency analysis: RETROSUM shifts the interaction-turn distribution toward 20-40 turns, while ReSum frequently reaches the 100-turn limit.The analysis attributes ReSum’s longer trajectories to redundant information-seeking loops caused by lost context.
  • Test-time scaling: RETROSUM consistently outperforms ReAct and ReSum across the full range of Best@K F1 Score test-time scaling settings.Best@1 corresponds to the average F1 score across 256 samples.
  • Benchmark analysis: The benchmark is described as fundamentally solvable and reasonably designed despite its challenging EHR decision-making tasks.The experiments include diagnoses-task analyses and comparisons across agent methods.
  • Memory constraints: RETROSUM remains stable as the maximum context length decreases from 64k to 8k tokens, unlike ReAct and ReSum.The baselines show significant degradation under tighter context windows.

6 Conclusions

The paper introduces AgentEHR for complex decision-making in raw, high-noise EHR databases and proposes RETROSUM with retrospective reasoning and evolving experience. Its validation identifies retrospective mechanisms as essential for clinical-agent reasoning tasks.

  • Conclusion: AgentEHR benchmarks complex decision-making within raw, high-noise clinical databases.RETROSUM combines retrospective reasoning with an evolving experience strategy to capture latent cross-temporal correlations.

Limitations

The study’s limitations concern dataset scope and modality coverage: evaluation centers on single-center MIMIC data, while the framework currently handles text and structured tables rather than raw imaging or waveforms.

  • Dataset scope: Evaluation relies primarily on MIMIC-IV and MIMIC-III, both sourced from a single medical center.This may not capture broader administrative protocols or demographic variation across global healthcare systems.
  • Modality coverage: RETROSUM currently supports textual clinical notes and structured tabular data, not pixel-level medical imaging or high-frequency physiological waveforms.The framework relies on textual reports for those modalities; future work targets multimodal synthesis and multi-center generalization.
  • Data preparation: The benchmark construction uses MIMIC-IV as the primary corpus and MIMIC-III for cross-subset validation.The processing pipeline builds patient-level time series, enriches medical codes, standardizes storage, and provides a unified toolbox.
  • Leakage mitigation: Future information is restricted through temporal censoring, removal of outcome-related columns, and masking of pharmacy artifacts.These controls aim to prevent leakage from administrative or outcome information.
  • Benchmark sampling: Label-wise weighted sampling separates Common and Rare pools using inverse label frequency and produces 600 samples for each task type.The resulting benchmarks include MIMIC-IV-Common and MIMIC-IV-Rare after filtering samples without admission records.

B.4 Dataset Statistics

The benchmark spans Common, Rare, and MIMIC-III cohorts with differing prevalence distributions and input complexity, and agents interact through modular tools evaluated across multiple backbones and baselines.

  • Dataset complexity: MIMIC-IV subsets contain longitudinal histories spanning over a year with thousands of temporal records.This creates substantial long-term dependency modeling requirements across heterogeneous EHR tables.
  • Dataset complexity: MIMIC-III spans approximately 1/20 the time of MIMIC-IV but retains roughly 1/2 as many records, creating greater recording density, redundancy, and noise.The paper characterizes MIMIC-III as more challenging because of this heightened complexity.
  • Dataset statistics: The dataset statistics table reports candidate counts, average label-set size, source-table counts, record volume, and longitudinal span.These measures characterize both output-space size and input complexity.
  • Agent interface: The agent toolbox provides Record, Candidate, Table, Inner, and Retrieval modules for EHR access, candidate alignment, schema inspection, internal reasoning, and external knowledge retrieval.Record tools support temporal, textual, and numerical filtering, while Candidate tools combine keyword, fuzzy, and semantic matching.
  • Experimental setup: Evaluations compare six agent methods across five LLM backbones, including Qwen3, GPT-5-min, and Grok-4.1-fast.Baselines include ReAct, Reflexion, ReSum, Reasoning Bank, and ReflecTool variants.
  • Experimental setup: All agents use a 100-turn interaction limit and a 64,000-token context cap, with evolving methods receiving 100 common-set training examples per task.The same accumulated experiences are used for evaluation across methods.

D.3 Metrics

The section defines F1-based evaluation for predicted clinical label sets and introduces Best@K F1 to measure expected peak performance under a trajectory budget.

  • F1 Score: F1 evaluates predicted elements against ground-truth elements using Precision and Recall, combining them as their harmonic mean.This balances missed relevant items and incorrect predictions.
  • Best@K F1: Best@K F1 measures expected maximum F1 when K trajectories are sampled without replacement from N generated trajectories.For each instance, it averages the best trajectory score across all size-K subsets.
  • Best@K F1: At K = 1, Best@K F1 becomes the arithmetic mean of all generated trajectories, representing average performance without selection.This is the single-attempt boundary condition.
  • Best@K F1: At K = N, Best@K F1 reaches the model’s absolute capability upper bound, corresponding to Pass@N.The full generated pool can be evaluated when the budget equals N.
  • Best@K F1: The probabilistic definition removes random-sampling variance and deterministically evaluates the trade-off between computational budget and performance.This makes budget-performance comparisons reproducible.

E Error Definition

The benchmark categorizes agent failures from tool and format violations to repetitive retrieval and unresolved termination, providing distinct labels for pathological behaviors.

  • Overview: Failure analysis divides agent errors into six categories spanning low-level format violations and high-level reasoning stagnation.The categories are used to systematically diagnose pathological trajectories.
  • No Prediction: No Prediction covers episodes ending without a parsable or valid prediction, including incomplete termination after the maximum turn limit.It also includes tool parsing and empty-answer failures.
  • Tool Repeat: Tool Repeat denotes identical tool calls with identical parameters for five consecutive turns, indicating a rigid loop and severe information redundancy.The defining behavior is exact cognitive stagnation.
  • Looping Errors: Single-Tool Loop requires ten consecutive uses of one tool with parameter similarity Sro > 0.95, while Multi-Tool Cyclic Loop flags similar calls more than 15 times overall.These criteria capture sequential scanning and pervasive revisiting of prior states.
  • Tool Usage Error: Tool Usage Error occurs when an agent invokes an unavailable tool or supplies arguments that violate the tool schema.These failures reflect a disconnect between the reasoning core and environmental constraints.
  • No Candidate Tool: No Candidate Tool is assigned when an agent never invokes candidate-retrieval tools, preventing its prediction from mapping to a valid candidate-table entry.The benchmark therefore treats the trajectory as a task failure.

F.1 Turn Analysis

Turn-distribution analysis shows that baseline methods are highly sensitive to backbone capability, while alternative self-evolving methods can terminate prematurely on complex diagnostic tasks.

  • Backbone Sensitivity: Baseline methods, especially ReAct and ReasoningBank, show substantial interaction-turn volatility across model backbones.The analysis examines how model capability and agent method jointly affect turn distributions.
  • Backbone Sensitivity: Nearly 500 cases reach the maximum turn limit for ReasoningBank on weaker models.The passage attributes this stagnation to critical-detail omission from lossy context compression without effective memory.
  • Premature Termination: 316 ReflecTool-CS cases resolve within 10 to 20 rounds on Qwen3-30B-A3B.The passage associates this rapid convergence with superficial reasoning or insufficient differential-diagnosis verification.

F.2 Tool Analysis

Tool-use distributions reveal distinct navigation strategies: baselines often over-query raw records, whereas RETROSUM shifts activity toward candidate grounding and maintains schema awareness.

  • Behavior with Different Methods: On weaker Qwen3-30B-A3B backbones, ReAct and Reflexion devote over 60% of actions to the Records tool.This pattern indicates repeated database querying and inefficient evidence acquisition.
  • Behavior with Different Methods: RETROSUM consistently uses the Candidate tool more than baselines, indicating greater emphasis on aligning clinical findings with the candidate space.The passage connects this shift to retrospective context compression.
  • Behavioral Alignment with Stronger Models: RETROSUM enables smaller models to approximate the Candidate and Inner tool-use profile of GPT-5-mini baselines.This profile is presented as evidence of more purposeful reasoning behavior across model capabilities.
  • Schema Awareness: RETROSUM maintains steady, moderate Table-tool usage, suggesting retained database-schema awareness without redundant schema queries.The passage also links this behavior to precise SQL generation for record retrieval.

F.3 Computational Consumption Analysis

RETROSUM reduces computational resource use while preserving task-relevant retrieval behavior across EHR navigation tasks. Its retrospective summaries lower token consumption and execution latency, while retrieval patterns show both broad context gathering and task-focused data access.

  • Token Economy: 0.42M average input tokens per sample represents a 4.9× reduction versus ReAct and a 1.7× reduction versus ReSum.RETROSUM dynamically compresses interaction history into concise retrospective summaries.
  • Execution Latency: 133.08s average execution time is lower than ReSum at 142.11s and ReAct at 158.66s.The method reduces interaction turns despite summary-generation and evolving-experience overhead.
  • Computational Cost: RETROSUM combines lower input consumption with reduced execution latency compared with standard baselines.Figure 10 compares execution time, input tokens, and output tokens across agent frameworks.
  • Task-Specific Schema Alignment: Agents predominantly retrieve tables semantically aligned with the target task, such as labevents for the labevents task.The dominant diagonal retrieval pattern indicates awareness of task-relevant database structure.
  • Impact of Model Capabilities: Smaller models repeatedly concentrate on limited table sets, whereas stronger models distribute attention across a wider range of sources.The comparison includes Qwen3-30B-A3B, Qwen3-80B-A3B, GPT-5-mini, and Grok-4.1.
  • Strategies of Agent Methods: Summarization methods broaden table-query distributions, while self-evolution methods focus more sharply on task-relevant tables.RETROSUM integrates active exploration for broad context with task-oriented focus on critical data.

G AI Assistance Statement

The paper reports AI-assisted drafting and presents prompts, examples, benchmark outputs, and error analyses related to clinical EHR decision-making tasks. These materials include diagnosis, laboratory, microbiology, prescription, procedure, and transfer tasks, along with documented agent failures.

  • AI Assistance: The paper acknowledges language editing and stylistic refinement with assistance from large language models.The statement mentions ChatGPT 4 and Google Gemini 5.
  • Task Prompts: The benchmark prompts assign agents roles including diagnostician, laboratory medicine specialist, clinical microbiologist, pharmacist, surgical planner, and care coordinator.Tasks require producing standardized diagnosis, test, prescription, procedure, or transfer candidates.
  • Summarization Prompt: The summarization prompt instructs agents to extract only explicit, certain, and task-relevant information without assumptions or unsupported inferences.It specifies a structured summary format for essential information from conversation history.
  • Error Analysis: Error analyses document sequential scanning, schema violations, invalid free-text outputs, and moderate-similarity candidate mappings that can produce incorrect standardized predictions.Examples include excessive raw-data retrieval, nonexistent columns, missing candidate-tool calls, and clinically inaccurate semantic matches.
  • Example Results: One diagnosis example reports F1 Score 0.608695652173913, Precision 0.7, and Recall 0.5384615384615384.The reported scores accompany a completed diagnosis-task example.
Loading 2601.13918v1…