Source-linked AI summary
CarePilot: A Multi-Agent Framework for Long-Horizon Computer Task Automation in Healthcare
Akash Ghosh, Tajamul Ashraf, Rishu Kumar Singh, Numan Saeed, Sriparna Saha, Xiuying Chen, Salman Khan
TL;DR
Long-horizon automation in healthcare lacks realistic, domain-specific evaluation across heterogeneous clinical software. The paper introduces CareFlow and CarePilot, an actor–critic framework with tool grounding and dual memory, and reports state-of-the-art performance against multimodal baselines.
Problem
Long-horizon multimodal agents struggle with interdependent healthcare workflows, while realistic benchmarks for heterogeneous clinical software are lacking.
Method
CarePilot uses an actor–critic architecture in which tool-grounded actions are conditioned on short- and long-term memories and refined through critic feedback.
Results
CarePilot achieves state-of-the-art results, outperforming strong open- and closed-source multimodal agents across healthcare software settings.
Takeaways & Limitations
CareFlow enables domain-grounded evaluation of long-horizon healthcare automation, while CarePilot supports robust semantic action prediction across clinical workflows.
Takeaways & Limitations
CareFlow covers only five healthcare platforms, and CarePilot predicts high-level semantic actions rather than exact GUI coordinates.
Abstract
from arXiv · showhide
Multimodal agentic pipelines are transforming human-computer interaction by enabling efficient and accessible automation of complex, real-world tasks. However, recent efforts have focused on short-horizon or general-purpose applications (e.g., mobile or desktop interfaces), leaving long-horizon automation for domain-specific systems, particularly in healthcare, largely unexplored. To address this, we introduce CareFlow, a high-quality human-annotated benchmark comprising complex, long-horizon software workflows across medical annotation tools, DICOM viewers, EHR systems, and laboratory information systems. On this benchmark, existing vision-language models (VLMs) perform poorly, struggling with long-horizon reasoning and multi-step interactions in medical contexts. To overcome this, we propose CarePilot, a multi-agent framework based on the actor-critic paradigm. The Actor integrates tool grounding with dual-memory mechanisms (long-term and short-term experience) to predict the next semantic action from the visual interface and system state. The Critic evaluates each action, updates memory based on observed effects, and either executes or provides corrective feedback to refine the workflow. Through iterative agentic simulation, the Actor learns to perform more robust and reasoning-aware predictions during inference. Our experiments show that CarePilot achieves state-of-the-art performance, outperforming strong closed-source and open-source multimodal baselines by approximately 15.26% and 3.38%, respectively, on our benchmark and out-of-distribution dataset.
1. Introduction
Long-horizon automation in heterogeneous, policy-constrained healthcare software lacks realistic domain-grounded evaluation. CareFlow and CarePilot address this gap with expert-annotated workflows and an actor–critic framework combining tool grounding with dual memory.
- Long-horizon agents must execute dozens of interdependent steps under partial observability, but realistic interactive benchmarks for domain-specific software remain scarce.
- Healthcare workflows span heterogeneous DICOM, annotation, EMR/EHR, and LIS systems, often requiring 10–15 dependent actions under privacy and audit constraints.
- CareFlow defines healthcare automation as predicting step-by-step semantic actions from natural-language goals and screenshot sequences across 8–24 decisions.
- CareFlow uses expert-designed and refined trajectories to retain high-frequency, high-value procedures representative of everyday clinical practice.
- CarePilot combines tool grounding, short-term memory, and long-term memory within an actor–critic loop for next-action prediction and corrective feedback.
- 15.26% improvement in task accuracy is reported over strong open- and closed-source baselines across CareFlow domains.
2. Related Work
Prior multimodal agents automate general digital environments, while healthcare automation remains limited by heterogeneous clinical interfaces and multi-stage workflow requirements. CareFlow and CarePilot extend evaluation and agentic control to these settings.
- Autonomous Multimodal Agents: Screenshot-based multimodal agents such as Mind2Web, SeeAct, and UI-TARS automate interactions across web and desktop applications.
- Autonomous Multimodal Agents: Voyager, Reflexion, and Jarvis 1 show the relevance of episodic memory, self-reflection, and long-term credit assignment for persistent task execution.
- Healthcare Software Automation: Rule-based healthcare automation improves efficiency but lacks generalization across heterogeneous clinical interfaces and cannot reason over multi-stage tasks.
- Healthcare Software Automation: CareFlow provides a human-annotated benchmark for long-horizon interactions across EMR systems, annotation tools, and hospital management applications.
- Healthcare Software Automation: CarePilot supplies an end-to-end multimodal framework that perceives, reasons, and acts within complex healthcare software ecosystems.
3. CareFlow
CareFlow is a human-annotated benchmark of long-horizon healthcare software workflows constructed through expert-informed task generation, screenshot–action annotation, and strict quality filtering. It spans multiple clinical software domains and provides validated trajectories for multimodal-agent evaluation.
- Dataset Pipeline: CareFlow is built through four stages: seed-task design, diversity expansion, stepwise GUI-state annotation, and quality assurance.
- Dataset Pipeline: Domain experts mapped software usage patterns and operational constraints to create executable end-to-end tasks representative of authentic clinical workflows.
- Dataset Pipeline: Controlled substitutions, parameter changes, and procedural edits broaden task diversity while preserving intent and executability.
- Dataset Pipeline: Annotators decomposed tasks into atomic steps, pairing each GUI screenshot with the precise next semantic action.
- Dataset Pipeline: Only trajectories with chronological screenshots, complete near-optimal sequences, and unambiguous instructions were retained.
- Dataset Validation: κ = 0.78 inter-annotator agreement was obtained, with the test set independently validated by domain experts.
- Dataset Composition: 1,100 tasks span four healthcare software categories, with each trajectory containing 8–24 consecutive GUI screenshots.
4. CarePilot
CarePilot combines multimodal tool grounding, dual memory, and an actor–critic loop to predict and correct semantic actions in long-horizon healthcare workflows. Critic feedback is distilled into an Actor that operates alone at inference.
- CarePilot targets long-horizon healthcare interfaces where VLMs struggle with low task completion despite moderate step-wise accuracy.
- Task Definition: The agent formalizes workflow completion as selecting semantic actions from screenshots, instructions, and history until a verifier confirms all required states and artifacts.
- Perceptual Grounding: Four lightweight perception tools—UI detection, zoom/crop, OCR, and template/icon matching—produce grounding signals for next-action prediction and memory updates.The outputs are aggregated into a unified representation ϕt.
- Dual Memory: CarePilot maintains short-term context and a long-term trajectory embedding, conditioning the next action on both memories.The long-term memory is updated using tool-grounding features ϕt.
- Actor–Critic Framework: The Critic evaluates each Actor proposal, approves sufficiently correct actions, or issues structured hierarchical feedback for localized and trajectory-level correction.Reflection examines consecutive states, short action windows, and the full trajectory, updating corresponding memories.
- Training and Inference: After simulation, Critic-verified successful trajectories train the Actor; inference retains only the Actor, which predicts actions without runtime Critic involvement.The design transfers Critic reasoning and memory use into the Actor while removing explicit evaluation overhead.
5. Experimental Setup
The experiments define implementation, evaluation, and baseline comparisons for CarePilot across healthcare software. They measure both exact next-action reliability and exact end-to-end workflow success.
- Implementation Details: Experiments use NVIDIA A100 GPUs and Google Colab Pro+ environments, with models trained for roughly 5–6 hours.
- Implementation Details: The framework uses PyTorch, Hugging Face Transformers, Unsloth, cosine learning-rate scheduling, LoRA adapters, and 4-bit base weights.Training uses a learning rate of 2×10−4, two epochs, and lightweight LoRA adapters.
- Evaluation Metrics: Step-Wise Accuracy measures exact next-action matches, while Task Accuracy requires every action in a workflow to be correct and ordered.SWA reflects fine-grained step reliability; TA captures end-to-end workflow success.
- Baselines: Baselines include open-source Qwen, Llama, Mistral, and Nemotron models plus closed-source GPT-4o, GPT-5, and Gemini 2.5 Pro systems.Non-GPT baselines are accessed through DeepInfra and evaluated zero shot.
6. Results and Findings
CarePilot consistently outperforms multimodal baselines across healthcare domains and shows robustness on out-of-distribution workflows. Ablations and task-length analyses indicate that tool grounding, memory, and critic feedback are important, while performance declines on longer tasks.
- R1) Baseline Comparison: CarePilot consistently outperforms open- and closed-source baselines across every healthcare domain.The Qwen 3 VL variant reaches 48.76% task accuracy and 92.50% step-wise accuracy, exceeding GPT-5 and Qwen 2.5 VL on both metrics.
- R2) Out-of-Distribution Performance: CarePilot with Qwen 2.5 VL achieves 77.93 SWA and 36.40 task accuracy on the Out of Distribution Open Hospital benchmark.These results are reported as superior in robustness relative to strong open-source and closed-source models.
- R3) Open- versus Closed-Source Models: GPT models lead proprietary systems at 36.19% average task accuracy, while Llama variants are strongest among open-source models.The comparison indicates a narrowing gap between closed- and open-source model classes on CareFlow.
- R4) Critic Ablation: Removing the Critic reduces task accuracy to 3.75% without tools and 12.5% with tool grounding.Both configurations perform substantially below the full CarePilot framework using Qwen-2.5 VL 7B.
- Component Ablations: Tool grounding is the most critical contextual component, with its removal reducing task accuracy to 9.37; long-term memory matters more than short-term memory.The ablation reports a larger performance decline when LTM is removed than when STM is removed.
- Task-Length Effects: Accuracy declines as workflows lengthen: it exceeds 64% below 10 steps, falls below 35% beyond 15 steps, and converges near 27% above 20 steps.The decline is steeper for the 7B variant between 10 and 15 steps, while both CarePilot variants converge for tasks exceeding 20 steps.
7. Conclusion
CarePilot combines actor-critic agents, tool grounding, and complementary short- and long-term memory for long-horizon healthcare software automation. CareFlow provides a multi-platform benchmark, while the reported experiments show state-of-the-art performance but leave platform coverage and pixel-level grounding for future work.
- Framework: CarePilot combines an action agent and critic agent with tool grounding, short-term memory, and long-term memory.STM stores recent step, outcome, and rationale information, while LTM maintains trajectory-level context.
- Benchmark: CareFlow is a benchmark for long-horizon healthcare software tasks spanning multiple platforms and diverse clinical subdomains.The benchmark is described as the first dedicated resource for this task setting.
- Findings: CarePilot achieves state-of-the-art results against strong open- and closed-source multimodal agents when equipped with contextual grounding and memory-based reasoning.The conclusion attributes the reported performance setting to accurate contextual grounding and memory-based reasoning.
- Limitations: CareFlow covers only five healthcare platforms, and CarePilot predicts semantic actions rather than exact GUI coordinates.The authors identify broader platform coverage, pixel-level grounding, and longer multilingual workflows as future directions.
Supplementary Material
The supplementary material reports ablations, qualitative analyses, ethical considerations, data-source descriptions, and inference-speed evaluation for CarePilot. Results emphasize tool grounding, memory, state verification, and affordance-aware planning as central to reliable healthcare workflow automation.
- Additional Experiments: Tool grounding contributes most to CarePilot performance, followed by long-term memory and short-term memory across additional experiments.The same component ordering is observed with the Qwen 3 VL 8B model.
- Additional Experiments: Removing Tool Memory causes the largest performance drop, while removing Zoom has the smallest impact among evaluated tools.The ablation also evaluates Object Detection and Optical Character Recognition.
- Qualitative and Error Analysis: Baseline agents repeatedly confuse tool selection with execution, producing missed zoom gestures, incomplete polygons, and uninvoked statistics panels.The errors affect both Llama-4 Maverick-17B and GPT-5 across CT abdomen and CT chest workflows.
- Qualitative and Error Analysis: Poor UI state tracking leads baselines to issue navigation actions when targeted annotation or statistics-tool clicks are required.These failures suggest policies do not verify whether the interface is in navigation or annotation mode.
- Qualitative and Error Analysis: GPT-5 emits free text before the text tool is armed and repeats it after activation, indicating weak coupling between language generation and GUI affordances.CarePilot addresses this failure through grounded annotation gated on the text-tool cursor state.
- Qualitative and Error Analysis: CarePilot completes all steps in the showcased CT abdomen and CT chest cases, producing correct ROIs and measurements on the first attempt.Action–mode verification, UI-aware planning, and grounded annotation support consistent end-to-end execution and reduce potential measurement variance.