PaperlayerView exact revision on arXiv

Source-linked AI summary

HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents

Qianchu Liu, Sheng Zhang, Guanghui Qin, Jeya Maria Jose Valanarasu, Maximilian Rokuss, Mingyu Lu, Timothy Ossowski, Juan Manuel Zambrano Chaves, Cliff Wong, Peniel Argaw, Yashna Hasija, Mu Wei, Wen-wai Yim, Qin Liu, Zilin Jing, Jason Entenmann, Naoto Usuyama, Tristan Naumann, Hoifung Poon

arXiv:2606.31179v1cs.AIcs.CLcs.CV

TL;DR

Healthcare lacks broad, agent-native benchmarks for evaluating frontier systems on realistic, end-to-end clinical workflows spanning multiple tasks, modalities, and data sources. HealthAgentBench addresses this gap with 54 tasks across 7 categories and finds that current agents remain challenged, with Codex GPT-5.5 achieving only approximately 42% task success.

  • Problem

    Existing healthcare benchmarks are fragmented, often evaluating isolated reasoning or narrow tasks, modalities, and workflows rather than agents’ full end-to-end capabilities.

  • Method

    HealthAgentBench evaluates agents across 54 patient-grounded tasks in 7 environments spanning diverse clinical workflows and modalities, with unified success criteria.

  • Results

    42% task success rate is achieved by Codex GPT-5.5, while the benchmark reveals strengths and limitations across task categories, workflows, and modalities.

  • Takeaways & Limitations

    HealthAgentBench is a challenging testbed showing substantial room for progress in agentic healthcare, especially for medical imaging, large search spaces, and compositional reasoning.

  • Takeaways & Limitations

    Medical imaging remains a persistent challenge, with Codex agents averaging 22% success versus 12% for Claude Code agents.

Abstract

from arXiv · show

As AI agents become increasingly capable of complex, long-horizon reasoning, rigorous and holistic evaluation is essential for measuring progress toward real-world healthcare applications. We introduce HealthAgentBench, a suite of 54 agentic healthcare tasks across 7 categories each with its unique environment. The benchmark suite spans diverse workflows throughout the patient journey and a broad range of modalities. Each task is designed to replicate an end-to-end clinical workflow: given minimal instructions, an agent must explore raw healthcare data, operate within a complex environment, and execute multi-step solutions that go beyond naive prompting. A final task success rate is reported to provide a single, interpretable metric for HealthAgentBench overall performance for each agent. Evaluating frontier agents on HealthAgentBench, we find that overall task success rate remains low, underscoring the difficulty of the suite. The strongest and the most cost effective agent, Codex GPT-5.5, achieves only approximately 42% success rate. Beyond aggregate performance, HealthAgentBench reveals nuanced strengths and weaknesses across task categories. Frontier agents show promise in automatically developing research modeling pipelines over EHR data, but medical imaging remains especially challenging, particularly for Claude Code models, while Codex GPT-5.5 shows emerging capability. Tasks that combine large search spaces with compositional reasoning requirements remain difficult for all current agents. Together, these results suggest that HealthAgentBench provides a challenging and realistic benchmark with substantial room for future progress. We release our benchmark at https://github.com/microsoft/HealthAgentBench.

1 Introduction

HealthAgentBench is a unified suite of 54 realistic agentic healthcare tasks across 7 categories and diverse clinical workflows and modalities. Frontier agents achieve low overall performance, with Codex GPT-5.5 reaching only around 42% success, while medical imaging and large-search-space compositional tasks remain major bottlenecks.

  • Motivation: Healthcare workflows require reasoning over data that cannot fit in short prompts, including 3D CT volumes, gigapixel pathology slides, longitudinal EHRs, trial protocols, and heterogeneous clinical documents.Traditional static benchmarks and recent agent-oriented efforts [7] [5] remain limited in scope for evaluating these capabilities.
  • Benchmark contribution: HealthAgentBench introduces 54 patient-grounded tasks across 7 categories, spanning medical imaging, free text, structured EHR data, and diverse clinical workflows.The suite includes unique environments and a principled workflow for sourcing and selecting tasks that can be extended in future editions.
  • Evaluation design: The benchmark evaluates end-to-end agent capabilities by requiring planning, tool use, environment exploration, and multi-step task execution over complex healthcare data.Examples include inspecting longitudinal chest radiographs, zooming and cropping images, cross-referencing prior studies, and writing a corrected report.
  • Empirical findings: Around 42% overall task success is achieved by Codex GPT-5.5, the strongest evaluated agent, showing that HealthAgentBench remains far from saturation.The benchmark supports nuanced analysis of agent strengths and weaknesses across task categories, clinical workflows, and data modalities.
  • Empirical findings: Medical imaging and tasks requiring large search spaces with complex compositional reasoning are major bottlenecks, while Codex GPT models generally outperform Claude Code models on imaging tasks.The analysis identifies CT, X-ray, and pathology-slide tasks as especially challenging and finds Codex GPT models, particularly GPT-5.5, generally more cost-effective.

2 Related Work

Related healthcare-agent benchmarks span medical question answering, interactive clinical dialogue, and executable environments over real data, but none provides a unified, executable, multimodal evaluation across the full patient journey.

  • Positioning: HealthAgentBench is positioned as filling the gap with a unified, executable, multimodal evaluation spanning the full patient journey.The paper groups prior work into three complementary lines and relates HealthAgentBench to each.
  • Medical question answering: Existing benchmarks primarily evaluate agentic strategies for medical question answering, including multi-agent collaboration, single-LLM prompting, conventional methods, and broad clinical capability suites [20].MedAgentBoard compares these approaches across medical task families, while ClinicalAgent Bench combines five clinical capability dimensions with a rich clinical toolbox.
  • Interactive clinical dialogue: Interactive evaluations instead center on clinical dialogue, spanning physician-authored multi-turn conversations, cost-aware sequential diagnosis, and simulated clinics with interacting medical agents [13] [26] [31].HealthBench and HealthBench Professional provide patient and clinician conversation rubrics; MAI-DxO uses challenging NEJM cases; AgentClinic simulates doctor, patient, and measurement agents.
  • Executable environments: The closest prior work places agents in executable environments over real clinical data, but benchmarks generally remain focused on particular settings such as structured EHR workflows [22] [35].MedAgentBench and PhysicianBench use FHIR-based EHRs for record retrieval and order placement, with PhysicianBench adding long-horizon, execution-verified workflows; MedAgentGym provides a scalable code-centric environment.

3 Task Creation

HealthAgentBench uses a four-stage workflow to create 54 tasks across 7 healthcare categories, emphasizing realistic agentic workflows, broad modality coverage, verifiable outcomes, and anti-cheating safeguards. Tasks are packaged as minimally instructed terminal environments, sampled representatively, and validated through baseline sweeps and trajectory review.

  • Task scope: The benchmark contains 54 tasks across 7 categories spanning data management, diagnostics, event modelling, and treatment planning with real clinical data.Categories include X-ray Report Correction, Pathology Tumor Area Selection, CT Abnormality Classification, Clinical Trial Matching, EHR Data Quality Auditing, EHR Event Modelling, and EHR Format Conversion.
  • Design requirements: Tasks require planning, tool use, multi-step reasoning, or environment interaction beyond naive prompting and span five input modalities across diverse clinical settings.The modalities are 2-D radiographs, 3-D chest CT, pathology whole-slide images, free-text clinical documents, and structured EHR data.
  • Task creation workflow: Each candidate passes four stages: selection criteria, construction from benchmarks or curated patient data, standardized design principles, and final checks against cheating, ambiguity, and triviality.Existing benchmarks are converted into terminal-based environments requiring autonomous end-to-end exploration without the human scaffolding used in original evaluations.
  • Design principles: The suite uses versatile terminal environments, representative 5–15-sample category sampling, runtime data downloads, opaque identifiers, disabled browsing, and minimalist instructions.These choices support heterogeneous data access while testing agents’ planning and strategy formulation rather than predefined workflows.
  • Evaluation and validation: Task success is evaluated with binary task-specific criteria, while metrics such as F1 and recall quantify partial progress; baseline sweeps and transcript reviews remove shortcuts and genuine ambiguity.Gold labels and verifier code remain outside agent containers, and tasks are redesigned when unintended shortcuts or multiple reasonable solutions are detected.

4 Baselines and Results

HealthAgentBench remains difficult for frontier agents: Codex GPT-5.5 achieves only 42% overall task success, while performance varies substantially by harness, cost, modality, and task category. Agents show promise on EHR machine-learning workflows but struggle with complex retrieval and medical imaging.

  • Harness effects: 42% to 35%: GPT-5.5’s success rate falls from Codex to Copilot CLI, whereas Opus-4.8 rises from 32% under Claude Code to 36% under Copilot CLI.These differences indicate that the harness can significantly influence agent performance; Copilot CLI uses a multi-agent architecture.
  • Overall results: 42%: Codex GPT-5.5 achieves the highest task success rate across 162 trials, leaving substantial room for improvement on HealthAgentBench.The evaluation covers 3 attempts on each of 54 tasks.
  • The Win: Machine Learning AutoResearch for EHR modelling: Nine of 10 agents reach 100% task success on EHR Format Conversion, showing strong capability in constructing clinical-data ETL pipelines.EHR Event Modelling and EHR Format Conversion replicate machine-learning research workflows over tabular clinical data, and agents can competitively match human performance in this setting.
  • The Challenge: Needle in a Haystack - Full Retrieval from Complex Search Space: 42%: Claude Code Opus-4.6 is the strongest agent on EHR Data Quality Auditing, yet no agent exceeds 50%, while Clinical Trial Matching requires searching roughly 400 free-text protocols.These tasks test retrieval across large or unstructured search spaces together with compositional reasoning.
  • The Challenge: Medical Imaging and the emerging capabilities from Codex GPT-5.5: 17% versus 49%: mean success is much lower on imaging than text tasks, with Codex GPT-5.5 averaging only approximately 35% across imaging tasks.Codex agents average 22% on imaging versus 12% for Claude Code, while the families are comparable on text tasks at 50% versus 48%.

5 Conclusion

HealthAgentBench is a unified, agent-native benchmark suite for healthcare AI agents spanning diverse data modalities and clinical workflows across the patient journey. It remains highly challenging for current frontier agents, with Codex GPT-5.5 achieving only a 42% mean success rate and substantial room for progress.

  • 5 Conclusion: 42% mean success rate is achieved by the strongest agent, Codex GPT-5.5, across the suite.This result underscores the benchmark’s difficulty and leaves substantial room for future progress.
  • 5 Conclusion: HealthAgentBench provides a unified, agent-native benchmark for evaluating healthcare AI agents across diverse data modalities and clinical workflows spanning the patient journey.

A Benchmark Comparison

Table 1 positions HealthAgentBench within healthcare-agent benchmarks by comparing modalities and interaction workflows, with the benchmark spanning five modalities and four workflows.

  • Benchmark scope: HealthAgentBench spans five input modalities and four workflows in the benchmark comparison.Modalities include text, structured EHR data, 2D and 3D medical images, whole-slide pathology images, and GUI interaction.
  • Modalities: The comparison defines modality by the raw data agents must process, including text, EHR, 2D, 3D, WSI, and GUI inputs.EHR inputs comprise structured or tabular records accessed through FHIR or SQL; imaging modalities cover radiographs, dermatology photographs, ultrasound, fundus images, blood smears, CT/MRI volumes, and whole-slide pathology.
  • Workflows: The workflow taxonomy covers treatment, communication, health-data operations, research, and administration.These categories include care planning and prescribing, patient or clinician communication, EHR and data-quality operations, analytic pipeline building and evidence synthesis, and hospital operations.

B The HealthAgentBench Benchmark Suite

HealthAgentBench packages seven healthcare benchmark categories as standardized Harbor tasks with containerized execution, hidden labels, verification, and diagnostic rewards. Its binary success framework and multidimensional coverage span varied clinical modalities, workflows, outputs, patient scopes, and data-access regimes.

  • Task Infrastructure: Seven benchmark categories are implemented as Harbor tasks with standardized containers, internet access, agent-visible instructions, hidden test labels, and verifiers producing binary rewards plus diagnostic metrics.The first release organizes the suite around the Harbor execution substrate.
  • Success Criteria: Each trial receives reward 1 only when it satisfies its category-specific success criterion, otherwise receiving 0.Criteria may be based on verifier checks, error counts, accuracy, F1, recall, or AUROC, and aggregate task success rates are reported across these binary outcomes.
  • Evaluation Coverage: The suite covers five evaluation axes: modality, clinical workflow stage, output shape, patient and temporal scope, and data-access regime.Figure 7 maps seven task categories across these dimensions, including radiology, CT, pathology, free text, and structured EHR modalities.

C Cost vs. performance trade-off · D Per-Category Performance Breakdowns

Cost-performance analysis shows that GPT-5 agents define the HealthAgentBench Pareto frontier, while Claude Code agents are more expensive for equal-or-lower success rates. The appendix complements these aggregate results with finer-grained per-category breakdowns.

  • C Cost vs. performance trade-off: The plotted performance metric is the pooled, trial-weighted task success rate, identical to the metric used in Figure 4.
  • C Cost vs. performance trade-off: Section 4 observes that the best-performing agents are not necessarily the most expensive.
  • C Cost vs. performance trade-off: The Pareto frontier is traced entirely by GPT-5 agents, including Codex and Copilot CLI.Figure 8 plots total full-suite cost against pooled task success rate.
  • C Cost vs. performance trade-off: Every Claude Code agent lies below-and-right of the Pareto frontier, indicating greater cost for equal-or-lower success rate.
  • C Cost vs. performance trade-off: Figure 8 measures cost as the total USD required for one full sweep over all 54 HealthAgentBench tasks.The x-axis uses a log scale.
  • D Per-Category Performance Breakdowns: The appendix provides finer-grained, per-category performance breakdowns that complement the aggregate results in Section 4.

D.1 Per-category results: success rate, time, and cost · D.2 Task-specific scores

Per-category evaluation reports success, wall-clock time, and cost across ten agents and seven task categories, while task-specific metrics expose performance beyond binary success. Results aggregate three attempts per instance at xhigh reasoning effort across Codex, Claude Code, and Copilot CLI harnesses.

  • D.1 Per-category results: success rate, time, and cost: Per-category results jointly report binary success rate, mean trial time, and mean trial cost for every agent–category cell.Figure 5 visualizes these quantities, while Tables 4–6 provide their exact values.
  • D.1 Per-category results: success rate, time, and cost: Each category cell pools three attempts per instance at xhigh reasoning effort across ten agents from the Codex, Claude Code, and Copilot CLI harnesses.Wall-clock time is measured in minutes and cost in USD per trial.
  • D.1 Per-category results: success rate, time, and cost: The per-category evaluation covers seven tasks spanning imaging, pathology, clinical-trial matching, EHR auditing, event modelling, and format conversion.The success-rate table lists X-ray Report Correction, CT Abnormality Classification, Pathology Tumor Area Selection, Clinical Trial Matching, EHR Data Quality Auditing, EHR Event Modelling, and EHR Format Conversion.
  • D.2 Task-specific scores: Figure 9 reports category-specific task metrics before reducing outcomes to binary success, except for EHR Format Conversion, whose score is already binary.Each panel uses its own y-axis because the metrics have different scales.
  • D.2 Task-specific scores: Task-specific scores are mean accuracy for CT classification, tumor-tile F1 for pathology selection, mean recall for EHR auditing, and mean recall@top-50 for trial matching.These metrics provide task-level performance measures before binary scoring.
  • D.2 Task-specific scores: EHR Event Modelling uses mean AUROC, while X-ray Report Correction uses the mean count of significant errors, for which lower values are better.All other Figure 9 panels are higher-is-better, and each panel identifies the task’s success criterion.

D.3 EHR Data Quality Auditing: the cost of a large search space … E.5 Pathology Tumor Area Selection: pathology whole-slide reasoning

Across HealthAgentBench’s EHR, trial-matching, imaging, pathology, and data-management tasks, agents struggle especially when success requires searching large spaces, interpreting complex modalities, or satisfying exact structured-output criteria. Performance varies substantially by workflow: narrowing search improves EHR auditing, while Codex GPT-5.5 leads the reported CT and pathology results and Copilot CLI’s opus-4.8 leads clinical-trial matching.

  • D.3 EHR Data Quality Auditing: the cost of a large search space: EHR Data Quality Auditing contains eight tasks spanning impossible values, demographic conflicts, cross-table conflicts or duplicates, and a combined scenario, each with base and clue variants.The clue variant additionally identifies injected sub-types and the table or tables to inspect, while data, labels, and scoring remain unchanged.
  • D.3 EHR Data Quality Auditing: the cost of a large search space: Narrowing the search to disclosed tables markedly improves EHR auditing recall, with Claude Code Opus-4.8 reaching 0.97 on clue variants.The benchmark uses eight tables and more than 800k rows, and the same data becomes more solvable when the search is narrowed, identifying search-space size as the dominant bottleneck.
  • D.4 EHR Event Modelling: AUROC vs. human engineered baseline: EHR Event Modelling compares four strongest agents with CLMBR and count+GBM on six new-onset diagnosis targets using the same first-prediction-time test cohort.The count+GBM model is a gradient-boosted classifier over hand-engineered count features, while CLMBR uses frozen clinical-language-model representations with logistic regression.
  • E.1 EHR Format Conversion: EHR ETL pipeline customization: Nine of ten frontier models completed the EHR format-conversion task perfectly at 3/3, while claude-opus-4-6 failed all three attempts at $0.88.Among passers, cost ranged from $0.29 for gpt-5.4-mini to $2.34 for claude-opus-4-7, an approximately 8× span at identical reliability.
  • E.3 Clinical Trial Matching: patient–trial eligibility matching: 0.667 mean success rate made Copilot CLI’s opus-4.8 the clinical-trial-matching leader, ahead of gpt-5.5 at 0.519 and claude-opus-4-8 at 0.481.The same four leading agents achieved soft recall_top_50 values between 0.82 and 0.93, while every other agent scored at or below 0.259 success.
  • E.4 CT Abnormality Classification: chest CT interpretation: 10/30 trials passed by Codex gpt-5.5 on CT abnormality classification, the best reported success rate, ahead of gpt-5.4 at 9/30 (0.300).Claude-family agents achieved 5/30, 5/30, and 4/30 across the listed models.
  • E.5 Pathology Tumor Area Selection: pathology whole-slide reasoning: Pathology tumor selection requires identifying all grid tiles with at least 20% tumor area in gigapixel whole-slide images averaging around 100,000 × 100,000 pixels.The task uses a fixed 256 × 256 grid at downsample 16 and rewards tile-F1 ≥0.90 against hidden masks.
  • E.5 Pathology Tumor Area Selection: pathology whole-slide reasoning: 12/30 trials passed by gpt-5.5 in pathology tumor-area selection, yielding a 0.400 mean success rate and the highest tile-F1 of 0.845.It was the only agent above 0.30; claude-opus-4-8 followed at 0.200 (6/30).

E.6 ehrshot: longitudinal clinical event prediction

EHRSHOT tasks evaluate whether agents can learn longitudinal clinical event predictors from labelled timelines and apply them to leak-proof, partially observed test timelines. Across six new-onset diagnosis tasks, performance varies substantially, with the strongest agent achieving a 0.778 mean pass rate and failures arising from overfitting, leakage, and resource exhaustion.

  • E.6 ehrshot: longitudinal clinical event prediction: The benchmark packages EHRSHOT as a reproducible Harbor-first workflow with hidden test labels, one-click data download, and mechanical leak-proofing.Test patients retain only events before their first prediction time, future end values are blanked, and labels remain on an isolated /tests/ mount unavailable to the main agent.
  • E.6 ehrshot: longitudinal clinical event prediction: Agents predict six new-onset diagnoses from train and validation timelines, then submit continuous per-patient probabilities for an unlabeled test set scored by AUROC.The selected targets are hyperlipidemia, celiac disease, acute myocardial infarction, pancreatic cancer, systemic lupus, and essential hypertension; success requires meeting each task’s count+LightGBM baseline.
  • E.6 ehrshot: longitudinal clinical event prediction: 0.778 mean pass rate was highest for claude-opus-4-7 (14/18), followed by Codex gpt-5.5 and Copilot’s opus-4.8 at 0.722 (13/18).claude-opus-4-8 and Copilot’s gpt-5.5 reached 0.667 (12/18), while gpt-5.4-mini and claude-opus-4-6 achieved 0.389 (7/18) and 0.333 (6/18), respectively.
  • E.6 ehrshot: longitudinal clinical event prediction: Rare outcomes can cause severe overfitting: GPT-5.5’s best celiac model fell from ∼0.74 validation AUROC to 0.30 on test.Self-inflicted validation leakage, including ensemble scores bleeding across folds, can mask the held-out generalization gap.
  • E.6 ehrshot: longitudinal clinical event prediction: Resource exhaustion is another failure mode, as weaker agents time out while streaming the ∼2.1 GB events.csv file and produce no submission.The task uses 16 CPUs, 64 GB RAM, and 65 GB storage and is described as the suite’s most expensive task.

E.7 EHR Data Quality Auditing: EHR data-quality auditing

This category benchmarks EHR data-quality auditing with deterministic hidden synthetic errors, requiring agents to flag clinically implausible rows under realistic disclosure and anti-cheating constraints. Performance is poor: the best agents achieve only 0.417 mean success, and incomplete high-recall runs fail the strict pass gate.

  • Benchmark design: The suite uses deterministic seeded synthetic errors with hidden labels across eight MIMIC-IV demo tables [16], covering impossible values, inconsistencies, and demographic contradictions.Gold labels are generated alongside the corruption and manually reviewed for genuine errors.
  • Tasks and outputs: Agents submit table and _row_id pairs in a flagged_rows.csv file, with base tasks paired with variants that disclose injected sub-types and tables to inspect.The benchmark includes impossible-value, inconsistency, demographic-conflict, and combined tasks, each with an additional _clues variant.
  • Reward and evaluation: The benchmark passes only when cluster-level recall is 1.0 and precision is at least 0.01, so incomplete high-recall runs score zero.Cluster-level recall measures whether every injected error cluster has at least one flagged member.
  • Anti-cheat and leak-proofing: Leak prevention relies on obfuscated source names and an instruction not to fetch the upstream dataset, while the planned /etc/hosts egress block is inactive.The instruction-level prohibition and source-name obfuscation are the current live defenses against retrieving a clean copy.
  • Baseline observations: 0.417 mean success rate (10/24) is the best result, jointly achieved by claude-opus-4-6 and Copilot’s gpt-5.5; no agent exceeds 0.42.claude-opus-4-7, claude-opus-4-8, and Copilot’s opus-4.8 reach 0.333 (8/24), while Codex gpt-5.5 reaches 0.250 (6/24).
  • Baseline observations: Agents generally under-flag subtle conflicting or duplicate records, while no agent over-flags enough to approach the 0.01 precision floor.This pattern indicates that failures primarily reflect missed injected-error clusters rather than excessive false positives.

F Trajectory Analysis for Successful Trials … F.6 X-ray Report Correction: read priors, let the image decide

Successful HealthAgentBench trajectories consistently prioritize orientation, verification, and disciplined interaction with raw data and environments over blind code or answer generation. Across task categories, agents succeed by scoping changes, triaging and rechecking evidence, constructing appropriate views or features, and grounding final outputs in validated observations.

  • F Trajectory Analysis for Successful Trials: A recurring signature across all seven categories is that strongest agents spend most steps orienting, verifying, enumerating workspaces, reading data, and checking intermediate outputs.The trajectories emphasize an orient → scope → execute → verify → clean workflow rather than blind generation.
  • F.1 EHR Format Conversion: minimal scoped config override: Codex GPT-5.5 achieved 100% success in the sampled EHR conversion trial by making a minimal copied-config override, validating parquet contents, and leaving the default YAML untouched.The agent first inspected repository wiring, configuration, and staged input before editing, then performed a final git-status sweep.
  • F.2 Clinical Trial Matching: triage, then parallel eligibility adjudication: Opus-4.8 recovered all four gold-eligible trials with recall 1.0 by triaging 301 XMLs in code, parallelizing adjudication across five subagents, and centrally rechecking borderline cases.The structured patient profile explicitly included absent conditions and devices so exclusion criteria were applied consistently; the next-best Codex agent had 52% task success.
  • F.3 EHR Event Modelling: leakage-safe ML pipeline: Claude Code Opus-4.7 achieved 78% success by building a leakage-safe pipeline over 29M events, using only pre-cutoff data to construct patient-level prediction features.Codex GPT-5.5 used comparable engineering, including Polars lazy joins over 29.3M rows and an anti-leakage cutoff enforced inside the feature table.
  • F.4 CT Abnormality Classification: multi-window rendering and noise control: Codex GPT-5.5 achieved 33% task success and passed the sampled CT trial with 1.0 accuracy by rendering multi-window, multiplanar, and z-slab views before making conservative label decisions.It supplemented rendered views with voxel-spacing and HU measurements, then checked that every requested label appeared exactly once with a yes/no value.
  • F.5 Pathology Tumor Area Selection: multi-resolution tiling and morphologydriven calls: Codex GPT-5.5 achieved tile-F1 0.967 with 163 true-positive tiles and no misses by aligning multi-resolution pathology views, reasoning morphologically, refining ambiguous boundaries, and running an outside pass.The trajectory used tissue and stain fractions only to avoid low-tissue overcalls; no learned classifier was trained, and Figure 13 shows the result with two spurious tiles.
  • F.6 X-ray Report Correction: read priors, let the image decide: Codex GPT-5.4 achieved 40% task success in the X-ray correction task through a 24-step, $0.314 trajectory that read prior reports, used the image to resolve contradictions, and edited only affected sentences.A JSON read-back completed the workflow, matching the edit-only scoring requirement to ground each correction in the image while preserving other text.

F.7 EHR Data Quality Auditing: scripted detection with iterative recall recovery · F.8 Common themes

EHR auditing succeeds through scripted inspection followed by iterative recall recovery, while across tasks agents prioritize verification and adapt their strategies to each workflow’s bottleneck. A pathology example illustrates this pattern with complete tumor-region recall despite two spurious tiles.

  • F.7 EHR Data Quality Auditing: scripted detection with iterative recall recovery: 42% success rate was achieved by Claude Code Opus-4.6 on the sampled demographic-conflict trial, with full recall of errors after inspecting eight gzipped tables.The agent wrote and iteratively refined a Python detector for demographic contradictions.
  • F.7 EHR Data Quality Auditing: scripted detection with iterative recall recovery: Iterative recall recovery caught missed Hemoglobin contradictions, found another gender-mismatched patient, and verified every flagged row ID against the source before resubmission.The detector progressed from an initial pass to deeper checks of gender-specific labs, reference ranges, and heights.
  • F.7 EHR Data Quality Auditing: scripted detection with iterative recall recovery: The EHR auditing workflow demonstrates that repeated scripted sweeps can recover errors missed initially, but the supplied passage cautions that this pattern is not sustained across the broader benchmark.The passage explicitly qualifies the result as applying to a single error family.
  • F.8 Common themes: Across all seven tasks, successful agents treated verification as a first-class activity by rereading artifacts, rechecking borderline decisions, and sweeping diffs.They spent substantially more steps checking than writing.
  • F.8 Common themes: Agents shaped their methods to each task’s bottleneck, including scripted triage with criterion-level reasoning, leakage-safe joins, multi-resolution imaging views, image-grounded edits, and iterative auditing.The examples span ETL, trial retrieval, prediction, medical imaging, report correction, and data-quality auditing.
  • F.8 Common themes: Recall 1.00 was achieved in a successful Codex GPT-5.5 pathology trial, with 29 true-positive tiles, 2 false positives, and 0 false negatives.The predicted tiles localized metastasis to the lower lymph-node fragments and recovered the entire tumor region.
Loading 2606.31179v1…