HealthFlow: A Self-Evolving AI Agent with Meta Planning for Autonomous Healthcare Research

• Current agents commonly use static, predefined high-level strategies, limiting adaptation in open-ended, noisy healthcare research problems.Existing systems may refine tools or subproblem reasoning, but problem decomposition, workflow orchestration, and plan revision remain hard-coded.Source: §1 · p. 1
• HealthFlow refines high-level orchestration policies by analyzing task successes, failures, and corrections into structured procedural knowledge.The resulting heuristics, workflow patterns, and data-handling warnings influence future task triage and analytical planning.Source: §1 · pp. 1–2
• The paper frames HealthFlow’s contribution as meta-level strategic learning rather than the operational-level optimization emphasized by prior systems.Source: §1 · p. 2
• EHRFlowBench provides a public benchmark of complex clinical research tasks derived from peer-reviewed literature for evaluating AI agent-driven EHR analysis.The benchmark is intended to address the lack of domain-specific resources beyond general agent benchmarks and closed-ended medical question answering.Source: §1 · p. 2
• HealthFlow’s self-evolving strategic approach significantly improves task success, robustness, and efficiency compared with state-of-the-art agent frameworks.Source: §1 · p. 2

• Existing agent systems generally retain hard-coded strategies for task decomposition, tool orchestration, and execution, including in scientific and biomedical applications.Examples include AutoGPT, GPT4Tools, HuggingGPT, MetaGPT, SciToolAgent, Robin, and Biomni.Source: §2 · p. 2
• Self-improvement methods such as STELLA, OriGene, self-reflection, Self-Refine, AlphaEvolve, and TextGrad improve tools, templates, feedback, or artifacts without learning a new meta-strategy.Source: §2 · p. 3
• HealthFlow’s architecture uses a meta agent, executor, evaluator, and reflector in a continuous loop that stores synthesized experiences for future strategic planning.Evaluation provides short-term correction, while reflection converts successful execution histories into persistent strategic knowledge.Source: Figure 1 · p. 3

3.1 Formalism and System Overview

• HealthFlow models research as iterative planning, execution, evaluation, and reflection, storing successful-task experiences in a growing memory to improve future plans.Each iteration produces a plan, execution trace, score, and feedback; successful traces contribute new experiences to memory.Source: §3.1 · p. 3, §3.2 · pp. 3–4, §3.3 · p. 4

3.2 A Collaborative Multi-Agent Architecture

• The meta agent retrieves relevant experiences and uses them to generate higher-level plans, while the executor converts those plans into auditable tool calls and code executions.The executor operates in a sandboxed environment and produces a detailed trace for evaluation.Source: §3.2 · p. 4
• The evaluator supplies quantitative scores and actionable feedback for revised plans, while the reflector distills successful traces into heuristics, workflow patterns, code snippets, and data warnings.Reflection is activated after successful completion and synthesizes procedural history into structured knowledge.Source: §3.2 · p. 4, §3.3 · p. 4

3.3 Meta-Level Evolution through Experience

• HealthFlow handles conflicting experiences through task-specific contextual synthesis and bootstraps its memory from curated tasks validated against reference solutions.The training mode restricts reflection to successfully validated tasks, producing verified initial knowledge.Source: §3.3 · p. 5

3.4 EHRFlowBench: A Benchmark for Agentic Healthcare Research

• EHRFlowBench is created by screening 51,280 papers, manually reviewing 118 relevant papers, and extracting an initial pool of 585 evidence-grounded tasks.The benchmark construction uses an LLM-assisted, human-verified process.Source: §3.4 · p. 5
• The benchmark is manually consolidated into 10 research categories and stratified to produce 110 tasks, including 100 evaluation tasks and 10 training tasks.The task-category distribution is refined from 585 extracted tasks into the final 110-task benchmark.Source: §3.4 · p. 5, Figure 2 · p. 5

4.1 Datasets, Tasks, and Evaluation

• Evaluation spans EHRFlowBench, MedAgentBoard, MedAgentsBench, HLE, and CureBench, covering open-ended research, EHR pipelines, medical knowledge, expert reasoning, and tool-augmented clinical decisions.EHRFlowBench uses weighted LLM-judge scores, while MedAgentBoard emphasizes manually evaluated task success and analytic dimensions.Source: §4.1 · p. 6

4.2 Baseline Methods

• HealthFlow is compared with general and medical LLMs, multi-agent systems, general agent frameworks, and biomedical agent frameworks.The listed baselines include DeepSeek-V3, DeepSeek-R1, HuatuoGPT-o1, MedGemma, MedAgents, MDAgents, ColaCare, AFlow, Alita, Biomni, and STELLA.Source: §4.2 · p. 6

4.3 Implementation Details

• Experiments use LLM ensembles for task curation and judging, bootstrap all test samples 100 times, and report means and standard deviations.Success-rate and accuracy metrics are multiplied by 100 for readability.Source: §4.3 · p. 6
• HealthFlow uses DeepSeek-V3 for its meta, evaluator, reflector, and main executor agents, with ToolUniverse integration and a self-correction loop capped at three attempts.An evaluator score below 6.0 triggers a retry.Source: §4.3 · pp. 6–7
• Table 2 compares HealthFlow, HealthFlow + ToolUniverse, and ablations that remove feedback, experience memory, or training bootstrapping across five medical benchmarks.The ablations isolate contributions from reflection or evaluation, experience memory, and initial training knowledge.Source: Table 2 · p. 7

5.1 Main Results and Ablation Study

• HealthFlow substantially outperforms all baselines across tasks, especially on EHRFlowBench and MedAgentBoard, while remaining competitive on knowledge-intensive, tool-light benchmarks.Non-agentic LLMs and frameworks without code execution struggle with multi-step analytical pipelines; specialized biomedical agents outperform general agents on MedAgentBoard.Source: §5.1 · p. 7
• HealthFlow achieves a dominant head-to-head win rate, maintaining a significant lead on both open-ended EHRFlowBench and structured MedAgentBoard tasks.Figure 3 categorizes each direct comparison as a win, loss, or tie when the score difference is ≤0.25.Source: §5.1 · p. 7, Figure 3 · p. 8
• Removing feedback reduces performance from 3.82 to 2.78 on EHRFlowBench and from 66.09% to 42.63% on MedAgentBoard, showing the importance of iterative critique and refinement.The feedback loop includes evaluator and reflector agents.Source: §5.1 · p. 7
• Disabling long-term experience memory lowers EHRFlowBench performance from 3.82 to 3.63, whereas omitting initial bootstrapping lowers it only from 3.82 to 3.80.The results distinguish the durable benefit of accumulated strategic knowledge from the smaller benefit of pre-populated memory.Source: §5.1 · p. 8

5.2 Further Analysis

• A capable executor is essential: replacing it with Qwen3-Coder causes MedAgentBoard performance to collapse from 66.09% to 6.04%, while the strongest configuration pairs DeepSeek-V3 with ToolUniverse.The failure analysis attributes the collapse to misinterpreted file-system instructions and premature task termination.Source: §5.2 · p. 8, Table 3 · p. 9
• HealthFlow excels at method soundness and artifact generation on EHRFlowBench, and at predictive modeling and report generation on MedAgentBoard.Reported scores are 3.72 and 3.96 for EHRFlowBench method soundness and artifact generation, and 4.21 and 4.10 for MedAgentBoard predictive modeling and report generation.Source: §5.2 · p. 8, Table 4 · p. 9
• Experience use adapts to task structure: EHRFlowBench emphasizes abstract heuristics and concrete code retrieval, whereas MedAgentBoard most often retrieves heuristics.Abstract heuristics comprise 31.1% of synthesized EHRFlowBench experiences, and code snippets are retrieved 1.91 times per task; MedAgentBoard also reports 1.91 heuristic retrievals per task.Source: §5.2 · p. 9, Figure 4 · p. 9
• Case studies show that retrieved outlier-handling knowledge produces interpretable visualizations and that HealthFlow can complete complex workflows with simulated data, CNN-LSTM training, evaluation, and required artifacts.Biomni and STELLA produce uninterpretable plots because of outliers, while HealthFlow generates a comparable visualization; a second case produces plots, accuracy metrics, and checkpoints.Source: §5.2 · p. 10, Figure 7 · p. 11

6 Discussion

• HealthFlow’s underlying LLM capabilities constrain performance because model biases and knowledge gaps can propagate into strategic plans and synthesized experiences.This limitation directly bounds the reliability of the agent’s learned strategic knowledge.Source: §6 · pp. 11–12
• Experience synthesis may distill flawed or overly specific heuristics from idiosyncratic successes, potentially degrading future performance if they are not properly generalized.The paper also identifies risks of flawed scientific conclusions and data privacy concerns, supporting human-in-the-loop deployment.Source: §6 · p. 12

7 Conclusion

• HealthFlow learns high-level research-management policies from experience, addressing the static-strategy bottleneck and achieving superior performance against state-of-the-art frameworks.The paper positions this as a step toward operationalizing procedural knowledge from scientific literature for autonomous healthcare research.Source: §7 · p. 12

A Ethical Statement

• HealthFlow and EHRFlowBench follow healthcare-AI principles, use de-identified datasets, and are intended to assist professionals rather than replace clinical judgment or provide medical advice.The authors also commit to publicly releasing source code and the benchmark for independent verification.Source: A Ethical Statement · pp. 15–16

B HealthFlow Algorithm

• The workflow retrieves relevant experiences, generates a meta-plan, executes it, evaluates the outcome, retries unsuccessful attempts, and stores synthesized experience after success.Algorithm 1 separates short-term evaluator-driven correction from long-term evolution through reflection and experience memory.Source: B HealthFlow Algorithm · p. 16

Assumptions

• The proof assumes that reflection produces meaningful, generalizable experiences and that the meta-agent reliably retrieves relevant experiences without harm from irrelevant context.The assumptions are explicitly presented as idealized simplifications of the learning process.Source: C Formal Proof of Evolving Efficacy · pp. 16–17

Theorem

• Under Effective Reflection and Effective Retrieval, the quality of HealthFlow’s planning policy is monotonically non-decreasing after successful task completions.The theorem defines policy quality as expected task success over a stationary task distribution.Source: C Formal Proof of Evolving Efficacy · pp. 16–17

Proof

• For relevant future tasks, the new experience cannot reduce success probability and may increase it, while irrelevant tasks retain their prior success probability.These two cases yield the overall inequality Q(π(Mt+1)) ≥ Q(π(Mt)).Source: C Formal Proof of Evolving Efficacy · p. 17

Limitations

• The authors acknowledge that poorly generalized experiences could degrade performance and that irrelevant context may not be perfectly ignored in practice.Thus, the monotonic-improvement guarantee is conditional rather than an unconditional empirical claim.Source: C Formal Proof of Evolving Efficacy · p. 17

D.1 EHRFlowBench Construction Details

• EHRFlowBench provides a realistic testbed for complex healthcare research tasks, with construction designed for relevance, diversity, and grounding in scientific literature.The benchmark is built through a systematic two-stage procedure.Source: §D.1 · p. 17

Stage 1: Candidate paper screening

• From 51,280 papers published at selected AI and data-mining conferences between 2020 and 2025, majority voting by three LLMs produced 162 EHR-relevant candidate papers.The screening classifies titles for explicit AI use on EHR data, including clinical notes, structured records, and EHR time series.Source: §D.1 · p. 18
• Manual expert review reduced the candidates to 118 seed papers for task extraction.The selected papers span conferences including AAAI, ICLR, ICML, IJCAI, KDD, NeurIPS, and WWW.Source: §D.1 · pp. 18–20

Stage 2: Task extraction and curation

• LLM extraction generated 585 initial tasks, which researchers consolidated, filtered, clarified, and stratified into 110 final tasks: 100 evaluation tasks and 10 training tasks.The training tasks bootstrap HealthFlow’s experience memory, with one task from each of the 10 final categories.Source: §D.1 · p. 21

Task extraction and curation

• Tasks must be self-contained and paper-independent, cover substantial research stages, and include verifiable answers tied to the source findings.The generation specification requires explicit formulas, parameters, constants, and brief interpretations where applicable.Source: §D.1 · pp. 21–22

Example task

• The example Heart Failure task contains 12,320 patients, a 3.0 control-to-case ratio, and 38.74 average visits per patient.Its interpretation identifies the cohort’s 3:1 imbalance as a consideration for model training and evaluation.Source: §D.1 · p. 22

Motivation for dataset selection

• Together, the selected datasets assess knowledge, practical EHR modeling, complex research execution, and clinical therapeutic reasoning.All datasets are publicly available or accessible upon request under their respective data-use agreements.Source: §D.2 · pp. 22–23

Benchmark selection

• EHRFlowBench evaluates end-to-end complex research workflows, while MedAgentBoard tests multi-step EHR data processing and modeling from extraction through report generation.MedAgentBoard uses real-world EHR data from MIMIC-IV and TJH.Source: §D.2 · p. 23
• MedAgentsBench and the medical subset of Humanity’s Last Exam assess foundational medical knowledge and expert-level reasoning required for high-quality analysis.The evaluation uses 100 randomly selected hard-set MedAgentsBench questions and 45 text-only HLE medicine and health-science questions.Source: §D.2 · p. 23
• CureBench evaluates tool-augmented therapeutic decision-making involving reasoning over patients, diseases, and drugs with external biomedical tools.The associated tools include FDA databases and PubMed.Source: §D.2 · p. 23

E.1 Computing Infrastructure

• All experiments run on an Apple Mac Studio with an M3 Ultra chip and 512 GB unified memory, using Python 3.12 and reproducible dependency management.Local HuatuoGPT-o1 and MedGemma models run through LMStudio, while online models use official platforms and, when supported, OpenAI-compatible APIs.Source: §E.1 · p. 23

E.2 Hyperparameter Details

• HealthFlow uses a maximum of 3 retries and a success threshold of 6.0 to balance correction opportunities, computational cost, and efficiency.The threshold ends retries once the evaluator assigns at least 6.0 on a 1.0–10.0 scale; a threshold of 8.0 caused unnecessary additional retries.Source: §E.2 · pp. 23–24

E.3 Implementation of Baseline Methods

• Baselines are reproduced from original papers or public implementations and generally use DeepSeek-V3 to normalize the backbone model and isolate agentic-architecture performance.Models without code execution are evaluated on generated code and answers for open-ended benchmark tasks, while multi-agent frameworks use official implementations or an identified community version.Source: §E.3 · p. 24

Standard metrics

• Ground-truth benchmarks use accuracy for MedAgentsBench and HLE, with LLM-based answer extraction followed by the official correctness evaluation prompt.Accuracy is the percentage of correctly answered questions.Source: §E.4 · p. 24

LLM-as-a-judge evaluation

• EHRFlowBench uses five LLM judges scoring methodology soundness, presentation quality, and artifact generation with weights of 70%, 20%, and 10%.The final score is the weighted average of the judges’ mean scores.Source: §E.4 · p. 24

Artifact evaluation

• Artifact-generation rubrics assess correctness, functionality, organization, and completeness, with scores from 1 to 5 ranging from failed or poor outputs to exemplary software-engineered artifacts.A holistic evaluator treats critical failures as heavily penalizing and exceptional performance in key areas as elevating the overall score.Source: §E.4 · p. 25

MedAgentBoard evaluation

• MedAgentBoard uses a single LLM, DeepSeek-V3, to provide a 1–5 score across four task-specific dimensions, with detailed rubrics customized for each task category.For data extraction and statistical analysis, rubrics assess data selection, transformation logic, missing values, and statistical-method appropriateness; for predictive modeling, they assess model choice, training, metrics, and validation.Source: §E.4 · p. 26

Data extraction and statistical analysis

• Data-analysis scoring separately evaluates data selection, transformation logic, missing-value handling, and statistical-method appropriateness against task requirements and standard answers.Each dimension uses a 1–5 rubric distinguishing exact or appropriate solutions from critically flawed or irrelevant processing.Source: §E.4 · p. 26

Predictive modeling

• Predictive-modeling rubrics assess model choice, training procedures, necessary evaluation metrics, and validation practices, including avoidance of data leakage.The criteria distinguish appropriate models and sound validation from incorrect, nonfunctional, or absent procedures.Source: §E.4 · pp. 26–27

Data visualization

• Visualization scoring covers plot correctness, analytical alignment, readability, and required file generation, including whether the correct image file is produced.The file-generation rubric assigns the lowest score when the required image file is not generated.Source: §E.4 · pp. 27–28

Report generation

• Report evaluation assesses completeness, factual accuracy, coherence, and clinical relevance, while solution quality heavily weights a well-reasoned, comprehensive justification of the chosen approach covering problem framing, results, and limitations.Source: §E.4 · pp. 25, 28

Experience-driven planning

• Retrieved experience memory guided early healthcare-data validation, including column-presence, numeric-type, and missing-value checks to prevent failures or distorted visualizations.These principles were synthesized into an explicit strategic workflow rather than a simple command sequence.Source: §F.1 · p. 29

Methodology & Implementation

• HealthFlow’s workflow validated required columns, removed missing target values, generated the labeled visualization, and completed successfully on the first attempt.The evaluator rated the result 9.0, while recommending removal of unrequested correlation annotation and redundant file-verification steps.Source: §F.1 · pp. 29–31

Results & Analysis

• 216,291 valid records yielded a systolic–diastolic blood-pressure scatter plot with correlation coefficient 0.008, indicating minimal linear dependence.The visualization was successfully created and saved as `bp_correlation_plot.png`.Source: §F.1 · pp. 30, 31, 33

Meta-level evolution

• The reflector converted the successful trace into reusable workflow and heuristic knowledge for validated clinical scatter plots and future healthcare analyses.The synthesized pattern specifies loading data, validating columns and types, removing missing target values, and saving clearly labeled visualizations.Source: §F.1 · pp. 30–31

Baseline comparison

• HealthFlow’s output included essential validation that Biomni and STELLA omitted, while Alita failed to generate the required image.Figure 9 compares the reference, HealthFlow, Biomni, STELLA, and Alita outputs for the same MedAgentBoard task.Source: §F.1 · p. 33, Figure 9 · p. 34

F.1 Case 1: Medically Informed Data Validation

• The execution log stated that a scatter plot was saved as 'blood_pressure_scatter.png' in the current directory, and the final answer repeated that claim.These outputs illustrate the mismatch between the logged failures and the agent’s asserted success.Source: §F.1 · p. 34
• Multiple tool calls failed, yet the synthesizer reported that the Systolic-versus-Diastolic blood-pressure scatter plot had been successfully created and saved.The failures included incorrect positional-argument counts for read_parquet_file, extract_blood_pressure_data, and save_plot_to_directory, although create_scatter_plot was logged as successful.Source: §F.1 · pp. 34–35
• The final response confidently provided an output path for the supposedly generated scatter plot despite the execution failures.The reported output location was '/PATH/TO/systolic_vs_diastolic_scatter.png'.Source: §F.1 · pp. 34–35

F.2 Case 2: Autonomous Research Simulation

• After initially generating datasets and altering the epoch count, HealthFlow’s evaluator triggered self-correction, leading the second attempt to implement the required architecture and reporting constraints.The first attempt scored 5.0; the second scored 7.1 after feedback addressing dataset use, 1000 epochs, softmax, class labels, error handling, and reporting scope.Source: §F.2 · p. 36
• HealthFlow evaluated four-class DBP classification using identical CNN-LSTM models, matched training conditions, and 1000 epochs on real and synthetic EHR datasets.The architecture used three 1D-CNN layers, a bidirectional LSTM with hidden size 128, and softmax outputs; training used identical hyperparameters and dataset sizes.Source: §F.2 · pp. 35, 37
• The study’s limitations include potentially incomplete synthetic clinical correlations, an arbitrarily chosen sequence length of 10, and evaluation on a single synthetic dataset.The authors recommend broader correlation analysis, architecture optimization, multicenter validation, and privacy-preserving metrics.Source: §F.2 · pp. 38–39
• Synthetic-data training achieved 0.5912 test accuracy versus 0.5875 for real data, with training losses of 0.5218 and 0.4852, respectively.The reported absolute accuracy difference was 0.37%, while the training-loss difference was 0.0366.Source: §F.2 · pp. 37–38

G Protocol for Human Evaluation and Rater Instructions

• Human evaluators review a medical task and reference answer, compare anonymized agent responses side by side, select the superior response or none, and export the results.The evaluation is conducted through an online platform with a save-and-proceed workflow and JSON export.Source: §G.1 · p. 39, §G.2 · p. 40

G.3 Evaluation Criteria

• The evaluation criteria emphasize factual medical accuracy, consistency with clinical guidance, avoidance of misleading or risky information, completeness, context, and critical omissions.Experts are also directed to assess readability and logic.Source: §G.3 · p. 40
• G.3 Evaluation Criteria assesses responses for clarity, structure, formatting, clinical grounding, and practical value for clinical practice.Responses should avoid overly theoretical or impractical statements.Source: §G.3 · pp. 40–41

H Prompt Design in HealthFlow

• HealthFlow separates static system prompts from dynamic task-specific user templates to maintain consistent agent behavior while adapting to different tasks.The section presents prompt templates for the meta, evaluator, and reflector agents.Source: H Prompt Design in HealthFlow · p. 41
• The meta agent converts every request into a single JSON-wrapped markdown plan, synthesizing retrieved experiences and requiring precise, privacy-conscious, verifiable steps.Complex tasks receive detailed plans, while simple questions receive a single command; feedback from prior attempts must be addressed explicitly.Source: H Prompt Design in HealthFlow · p. 42
• The evaluator agent returns a JSON score from 1.0 to 10.0 with actionable feedback and reasoning based on correctness, efficiency, safety, robustness, and data privacy.A training-mode variant adds a ground-truth reference answer and places greater weight on objective correctness for system evolution.Source: H Prompt Design in HealthFlow · p. 43
• The reflector agent analyzes successful task histories and outputs 1–3 generalized experiences, such as heuristics, code snippets, workflow patterns, or warnings.Experiences must be abstract, success-oriented, specific, immediately useful, and assigned contextually appropriate categories.Source: H Prompt Design in HealthFlow · pp. 43–44