PaperlayerView exact revision on arXiv

Source-linked AI summary

BaT: Towards Self-Evolving Medical Research Agent with Stage Rubrics

Junqi Liu, Yufan He, Yexiao He, Pengfei Guo, Dong Yang, Andriy Myronenko, Can Zhao, Hanrong Ye, Tianhao Qi, Yuyin Zhou, Daguang Xu, Yucheng Tang

arXiv:2608.16211v1cs.AI

TL;DR

Long-horizon medical-agent training lacks a way to use stage-level benchmark diagnostics while keeping task content outside training. BaT creates a recursive benchmark-teaching loop that more than doubles Qwen Instruct Overall scores on AutoMedBench-Lite, with BaT-9B Agent reaching 79.6 Overall.

  • Problem

    Standard post-training reduces multi-stage trajectories to one outcome reward, discarding held-out stage diagnostics needed to identify which stage requires practice.

  • Method

    BaT combines asynchronous, content-isolated Stage Bank practice states with BiCuRL, which selects curricula from benchmark diagnostics, updates policies, and re-evaluates checkpoints.

  • Results

    79.6 Overall: BaT-9B Agent exceeds Claude Opus 4.6 with Claude Code by 2.1 points, while BaT-4B and BaT-9B more than double their Qwen Instruct Overall scores.

  • Takeaways & Limitations

    BaT shows that a structured benchmark can serve as both evaluator and post-training teacher for medical agents.

  • Takeaways & Limitations

    The evidence lacks a separate untouched final test and a measured semantic leakage audit, despite keeping task content outside training rows.

Abstract

from arXiv · show

Long-horizon agents are beginning to automate complete workflows that produce code, reports, and research artifacts. Medical imaging workflows are multi-stage and data-sensitive, while expert trajectories remain scarce and difficult to share. Structured benchmarks can localize failures through stage-level rubrics, but standard post-training discards these diagnostics before the next training round. We present Benchmark-as-Teacher (BaT), a recursive self-improvement system for agent post-training. BaT contains two linked components: the asynchronous Stage Bank data pipeline and BiCuRL (Bilevel Curriculum Reinforcement Learning), its self-improving post-training method. Stage Bank synthesizes content-isolated training states outside the policy-update loop. BiCuRL uses a fixed held-out evaluation to select the next stage curriculum, verifies rollouts with task rubrics, updates the policy with GRPO, and returns the candidate checkpoint to evaluation. On AutoMedBench-Lite, BaT-4B and BaT-9B more than double the Overall scores of their Qwen Instruct baselines. BaT-9B Agent reaches 79.6 Overall, exceeding Claude Opus 4.6 with Claude Code at 77.5.

1. Introduction

BaT turns structured benchmark diagnostics into a recursive, content-isolated post-training cycle for long-horizon medical agents. Its Stage Bank and BiCuRL components target weak stages, verify rollouts, update policies, and re-evaluate candidates while preserving held-out task boundaries.

  • Motivation: Long-horizon medical workflows are vulnerable to cascading stage errors, while expert trajectories remain scarce and difficult to share.Agents must plan, configure tools, validate data, run inference, and submit checked artifacts.
  • Problem: A single outcome reward cannot identify which stage needs practice in AutoMedBench-Lite runs averaging 33 interaction turns.Standard GRPO post-training discards stage-level diagnostics, and iterative schedules omit held-out stage diagnostics.
  • Method: BaT combines the asynchronous Stage Bank pipeline with BiCuRL to turn benchmark signals into recursive, stage-targeted post-training.Stage Bank synthesizes fictional tasks, reconstructs executable stage states, applies leakage checks, and exposes content-isolated training pools.
  • Method: BiCuRL selects curricula from fixed held-out stage scores, verifies rollouts with rubric items and artifact evidence, updates policies with GRPO, and retains or rejects checkpoints.Only aggregate scores cross the evaluation boundary; task IDs, answers, paths, reports, and traces remain held out.
  • Results: 79.6 Overall is achieved by BaT-9B Agent, exceeding Claude Opus 4.6 with Claude Code at 77.5, while BaT-4B and BaT-9B more than double their Qwen Instruct Overall scores.These results are reported on AutoMedBench-Lite; the full Stage Bank mixture leads every evaluated partial mixture.

2. Benchmark-as-Teacher

Benchmark-as-Teacher is a recursive self-improvement system coupling an asynchronous Stage Bank data pipeline with BiCuRL post-training. Stage Bank supplies isolated practice states, while BiCuRL selects curricula from held-out stage diagnostics, trains with rubric-verified GRPO, and returns retained checkpoints to evaluation.

  • Data Factory: Stage Bank prepares content-isolated practice states from synthetic, leakage-checked medical-imaging workflows outside the policy-update loop.It creates SFT slices for cold start and executable RL sandboxes for stage-level and end-to-end rollouts.
  • Results: 79.6 Overall is achieved by BaT-9B Agent on AutoMedBench-Lite, versus 77.5 for the strongest non-BaT systems.The agent combines the BiCuRL-trained policy with a fixed OpenHands execution environment across BaT Agent comparisons.
  • BiCuRL: The curriculum combines S-target for the selected weak stage, S-mix for remaining stages, and E2E for complete surrogate workflows.S-mix limits forgetting, while E2E preserves cross-stage coordination; the ablation changes only which pools enter training.
  • BiCuRL: BiCuRL uses fixed held-out Overall and stage scores to choose the next curriculum, update the policy with GRPO, and evaluate the candidate checkpoint.Its alternating loop runs a finite GRPO block, evaluates the candidate, updates the curriculum, and starts the next block from the retained checkpoint.
  • BiCuRL: Rubric verification scores rollouts against execution records and artifact evidence, with evidence completeness reducing rewards for unsupported success claims.Binary rubric decisions avoid free-form score calibration, and group normalization compares rollouts sharing one state and reward contract.

3. Experimental Setting

The experiments evaluate long-horizon medical research agents across complementary benchmarks, using process, outcome, and overall metrics. They compare Qwen3.5 post-training methods and external systems under repeated diagnostic runs.

  • Benchmarks: AutoMedBench-Lite evaluates five workflow stages across two difficulty tiers; the diagnostic suite contains 7 task tracks with 70 runs per system.Each track is repeated 10 times, and tested agents average 33 interaction turns.
  • Benchmarks: ABRA evaluates radiology agents across 655 tasks, while MedXpertQA-Text evaluates text-only medical questions spanning 17 specialties and 11 body systems.ABRA uses an OHIF viewer, an Orthanc DICOM server, and 21 tools across eight task types and three difficulty tiers.
  • Metrics: AutoMedBench-Lite reports Task, Agentic, and Overall scores, with Overall equally weighting Task and Agentic using unrounded values.Task measures the submitted result, while Agentic measures completion of Plan, Setup, Validate, Inference, and Submit.
  • Compared systems: Experiments train Qwen3.5-4B and Qwen3.5-9B Instruct models and compare their Baseline, SFT, GRPO, and BiCuRL variants.Baseline is the pre-post-training Qwen Instruct checkpoint; GRPO uses a single final task reward on E2E data, while BiCuRL runs inside the full BaT loop.
  • Compared systems: Figure 2 compares Qwen3.5, Gemini 3.1 Pro, GLM-5, GPT-5.5, and Claude Opus 4.6 systems using Claude Code or Codex across the three evaluation suites.Each system retains the execution setting named in the figure.
  • Data preparation: Stage Bank contains 20,299 candidate prompt states and projects a separate 1,008-row synthetic E2E source pool into E2E, S-target, and S-mix sandboxes.The 4B and 9B SFT starts each use 4,608 rows, and each GRPO group samples 4 continuations from one state.

4. Results

BaT-4B and BaT-9B more than double their Qwen Instruct AutoMedBench-Lite Overall baselines, while checkpoint retention and the full three-pool mix preserve or improve results. BaT-9B reaches 79.6 Overall on AutoMedBench-Lite, and its broader reasoning performance remains close to baseline while improving three long-horizon scores.

  • Overall performance: 22.9 Overall for BaT-4B versus 6.1, and 53.4 Overall for BaT-9B versus 19.9, more than double their corresponding Instruct baselines.These are the completed BiCuRL policy runs and their Qwen Instruct comparisons.
  • Iterative improvement: Through round ten, best-so-far checkpoint retention preserves accepted Overall gains even when later candidates score lower.Raw round curves fluctuate, motivating retention and fallback in BiCuRL’s outer loop.
  • Pool ablations: 53.4 Overall for the full three-pool mix exceeds E2E alone at 31.9 by 21.5 points, while every drop-one-pool run trails by at least 26 points.The compared pools are S-target, S-mix, and End-to-End.
  • Medical benchmark comparison: 79.6 on AutoMedBench-Lite for BaT-9B Agent is 2.1 points above Claude Opus 4.6 with Claude Code at 77.5.Each BaT Agent pairs its BiCuRL-trained policy with the fixed OpenHands environment.
  • Local deployment: BaT-9B ranks first and BaT-4B third among 10 tiny local LLMs, while BaT-4B rises from 6.1 to 22.9 Overall.Tiny local LLMs are defined as models with parameter size no more than 12B, evaluated with the same runner and protocol.
  • Generalization: BaT-9B remains within 3.4–5.8 points of its short-turn reasoning baseline, improves τ2-Bench by 5.4 points, and improves three long-horizon scores.BaT-4B scores lower on all eight evaluated benchmarks.

5. Discussion

BaT uses AutoMedBench-Lite both to evaluate the current agent and to control subsequent practice, while keeping task content outside training rows. The discussion also identifies unresolved evaluation and reporting limitations, including adaptation-aware diagnostics, incomplete leakage auditing, and incomplete local-run manifests.

  • Benchmark roles: AutoMedBench-Lite measures the current agent, and its aggregate diagnostics determine what the model practices next while task content remains outside training rows.This makes the benchmark both a diagnostic evaluation and a controller of later training mixtures, rather than an untouched final test.
  • Benchmark requirements: BaT requires benchmarks with stable stages, stage-level scores, and rubrics or checks capable of scoring a continuation.External benchmarks provide a separate transfer check, although their protocols differ.
  • System operation: Stage Bank synthesis and BiCuRL validation can run asynchronously, while policy updates read only versioned, executable states.The separation between these components matters operationally for the training pipeline.
  • Limitations: The discussion lists six limits, including adaptation-aware diagnostics, future content-isolation validation through an untouched final test and leakage audit, and incomplete local-run manifests.The external results table combines retained local runs with published baselines, while local aggregates lack complete per-run manifests.

6. Related Work

Prior work spans long-horizon agent benchmarks, healthcare-specific evaluation, reinforcement-learning and reasoning methods, and executable agent environments. BaT builds on these directions by using AutoMedBench’s stage structure to create content-isolated training sandboxes and select the next training focus.

  • Agent Benchmarks: Long-horizon benchmarks evaluate agents on tools, files, command-line tasks, function calling, or unseen instruction rules.Examples include Terminal Bench 2.0, BFCL, and IFBench.
  • Healthcare Benchmarks: Healthcare benchmarks cover physician-authored EHR tasks, realistic agentic healthcare settings, and staged medical-AI development.MedAgentBench, HealthAgentBench, and AutoMedBench measure agent performance, while AutoMedBench’s stage structure also supports diagnosis.
  • BaT’s Positioning: BaT uses public stage structure to build content-isolated training sandboxes and choose the focus of the next training round.This positions BaT’s training pipeline as an extension of stage-aware healthcare benchmarking.
  • Training Methods: Related methods include GRPO, ReAct, process supervision, curriculum learning, Self-Refine, and Agent Lightning for reasoning, tool use, feedback, and agent training.GRPO compares samples within groups without a learned value model; process supervision provides intermediate-step feedback, and curriculum learning schedules examples by difficulty or structure.
  • Agent Environments: Agent-training infrastructure includes workspaces, non-Docker software environments, executable engineering environments, and modular asynchronous tool-use reinforcement learning.Examples include OpenHands, SWE-World, SWE-Gym, and VerlTool.

7. Conclusion

BaT turns structured benchmark stages and checks into a recursive self-improvement system, combining Stage Bank data isolation with BiCuRL curriculum selection and policy updating. Its policies substantially improve AutoMedBench-Lite performance and show mixed transfer across external benchmarks.

  • System contributions: BaT converts public benchmark stages and checks into an RSI system through the asynchronous Stage Bank pipeline and BiCuRL.Stage Bank builds content-isolated E2E, S-target, and S-mix states, while BiCuRL selects a weak stage using aggregate diagnostics and updates the policy.
  • Benchmark results: More than double: BaT-4B and BaT-9B exceed their corresponding Qwen Instruct Overall scores on AutoMedBench-Lite.The data boundary keeps task-specific evaluation content outside training while aggregate scores guide routing.
  • System contributions: Stage Bank, BiCuRL, and the fixed BaT Agent execution layer demonstrate how a structured benchmark can serve as both evaluator and post-training teacher.This conclusion links evaluation, curriculum routing, and post-training within one recursive system.
  • Benchmark results: 2.1 points: BaT-9B Agent exceeds Claude Opus 4.6 with Claude Code.The passage reports the comparison as a 2.1-point margin.
  • External transfer: Within 3.4–5.8 points: the 9B policy remains near AIME and GPQA-Diamond performance on external benchmarks and gains on three of five long-horizon tasks.The 4B policy scores lower on all eight external tests.

A. Technical Details · A.1. Stage Bank Construction

Stage Bank is a versioned collection of stage- and end-to-end training rows built from checked teacher trajectories and executable sandbox states. It preserves upstream workflow context while isolating stage-scoped supervision and validating reduced end-to-end surrogates for repeated training.

  • A.1. Stage Bank Construction: Each versioned row keys to a workflow stage or E2E pool, stores executable sandbox state, attaches rubric evidence requirements and stage skills, and records provenance.The public benchmark rubric supplies item sets and evidence requirements used to compute ηᵢ,ₖ.
  • A.1. Stage Bank Construction: For each fictional task, a teacher model produces a checked multi-turn trajectory that Stage Bank transforms into single-response training rows.Rows retain the task, stage skill, earlier responses, and observations as history.
  • A.1. Stage Bank Construction: Each row keeps multi-turn history but applies loss only to the selected teacher response, excluding tool observations and other environment text.This isolates supervised learning to the teacher response rather than environment-generated content.
  • A.1. Stage Bank Construction: Stage sandboxes replay each trajectory prefix in the execution environment to materialize files, environment state, and intermediate artifacts at the stage boundary.Training starts with upstream context, while the row rubric scores only the selected stage’s work.
  • A.1. Stage Bank Construction: E2E surrogates reduce fictional tasks by selecting cases, capping input size, and limiting validation passes while preserving five-stage order, semantics, and output schema.The reduction parameters are chosen from a versioned task-specific grid and recorded in provenance.
  • A.1. Stage Bank Construction: The reduced E2E trajectory starts from an empty sandbox and attaches ordered stage skills with combined rubric and evidence contracts.The E2E construction therefore retains the complete ordered workflow in a reduced execution setting.
  • A.1. Stage Bank Construction: E2E surrogates are accepted only when sandbox initialization, rollout-limit compliance, every stage contract, and zero leakage all pass.Accepted rows preserve the full workflow while keeping each rollout small enough for repeated training.

A.2. Multi-Turn Supervision in BiCuRL

BiCuRL supervises multi-turn rollouts by propagating a single rubric-derived reward to every generated token while excluding sandbox observations from the policy loss. The curriculum selects which stage the supervised trajectories practice.

  • Multi-turn rollout structure: Each rollout is a multi-turn interaction in which the policy conditions on sandbox state, stage skill, and interaction history before receiving environment observations.The rubric verifier scores the completed rollout, requiring rollout-level supervision across all interaction turns.
  • Token-level supervision: After tokenization, BiCuRL joins policy responses into generated-token sequences and excludes observation tokens from the loss.The generated sequence is represented as y_i,k = (a_i,k,1, ..., a_i,k,T_i,k).
  • Token-level supervision: Each generated token is conditioned on the prompt, stage information, prior generated tokens, and every preceding sandbox observation.The full token context before each generated token includes p_i, κ_i, interaction history, and returned observations.
  • Token-level optimization: Every generated token receives the rollout advantage, with the policy update using clipped importance ratios and a KL penalty toward the current policy.The objective averages token terms over rollouts and groups.
  • Curriculum supervision: One group-normalized, evidence-discounted rubric decision supervises every turn, while curriculum q_r determines which stage those trajectories practice.This connects rollout-level rubric verification to stage-specific curriculum selection.

A.3. Migration to Other Staged Benchmarks … A.8. External Benchmark Protocol

The appendix establishes BiCuRL’s portability requirements, implementation and evaluation protocols, and comparative results across local models, AutoMedBench-Lite, ablations, and external benchmarks. It reports that BaT-9B scores 53.4 Overall among local models and uses track-level aggregation for AutoMedBench-Lite evaluation.

  • A.3. Migration to Other Staged Benchmarks: BiCuRL transfers to a staged benchmark when stages, rubric and evidence contracts, fixed evaluation, executable Stage Bank states, and policy likelihoods are available.The benchmark is represented as ℬ=(𝒮,𝒞,𝒱), and these assumptions define every term required by the BiCuRL update.
  • A.4. Post-Training and Runtime Details: The Qwen3.5-9B cold-start policy uses full-parameter SFT on 4,608 rows across eight GPUs, and BiCuRL initializes from the final archived checkpoint.Training uses PyTorch FSDP full_shard and fused AdamW.
  • A.4. Post-Training and Runtime Details: Each training round samples Stage Bank pools in proportions (ρtarget, ρmix, ρE2E)=(1/2,1/4,1/4), without replacement within the round and with resets between rounds.Within E2E, surrogate rows are sampled uniformly; matched ablations use a 275-row pool and renormalize after omitting a pool.
  • A.5. Local LLM Comparison: 53.4 Overall places BaT-9B third among 19 representative open-weight local models evaluated with the default runner.The two higher-ranked models have 27B and 35B total parameters.
  • A.6. AutoMedBench-Lite Protocol: AutoMedBench-Lite evaluates seven held-out medical AI tracks using ten separate agent executions per system on each track.The five-stage workflow covers Plan, Setup, Validate, Inference, and Submit.
  • A.6. AutoMedBench-Lite Protocol: Overall averages Task and Agentic scores, while Agentic weights Plan, Setup, Validate, Inference, and Submit by 0.25/0.15/0.35/0.15/0.10.For each system, metrics are averaged over ten repeats within each track, then across seven track means treated as independent units.
  • A.7. Matched Ablation Contract: The matched pool-ablation contract changes only which of S-target, S-mix, and E2E enters GRPO, using a 275-row matched pool.Table 8 records the associated controls for the Qwen3.5-9B ablation contract.
  • A.8. External Benchmark Protocol: The external protocol evaluates Instruct baselines and BiCuRL policies on eight benchmarks spanning competition mathematics, graduate-level science, multi-turn agents, function calling, and general assistants.Named benchmarks include AIME 2025, AIME 2026, GPQA-Diamond, τ2-Bench, BFCL-Parity, and GAIA.
Loading 2608.16211v1…