PaperlayerView exact revision on arXiv

Source-linked AI summary

MediSkill-Evo: Process-Constrained Self-Evolution for Evidence-Grounded Clinical Interaction

Ruoyu Wu, Shenfu Xie, Yinqian Sun, Haibo Tong, Feifei Zhao

arXiv:2608.23397v1cs.AI

TL;DR

Interactive clinical agents need to gather decisive evidence and convert it into safe, grounded actions under partial observability. MediSkill-Evo addresses this by evolving governed process knowledge in four typed banks and using a process-constrained harness to select actions. On fixed suites, complete-system evaluations report improved diagnosis, treatment-intent, and evidence coverage alongside fewer automatically scored critical failures, while remaining descriptive rather than causal or clinically validating.

  • Problem

    Under partial observability, a correct diagnosis alone does not establish that an agent gathered valid evidence or respected care-process constraints.

    Source: §1 · p. 1

  • Method

    MediSkill-Evo evolves four typed banks under provenance, support, replay, and safety checks, then ranks actions with a process-constrained harness and Clinical Process Critic.

    Source: §1 · p. 2, §2.3 · p. 4

  • Results

    Complete-system comparisons report higher treatment-intent and evidence coverage and fewer automatically scored safety-related failures across fixed suites and backbone endpoints.

    Source: §1 · p. 2, §3.4 · p. 6, §4 · p. 9

  • Takeaways & Limitations

    The results provide descriptive end-to-end evidence that governed process knowledge can support evidence-grounded clinical interaction within the evaluated benchmark settings.

    Source: §1 · p. 3, §4 · p. 9

    A failure-boundary case shows that governance checks cannot determine whether an image interpretation is clinically true, motivating independent image adjudication.

    Source: T15 MEASUREMENT · p. 31

Abstract

from arXiv · show

Interactive clinical agents must gather decisive evidence and convert it into grounded actions under partial observability. A correct final diagnosis alone does not show that an agent respected evidence and care-process constraints. We introduce MediSkill-Evo, a clinical agent that evolves governed process knowledge without backbone fine-tuning. It separates experience into four typed banks for clinical skills, process rules, symbolic schemas, and measurement procedures. Provenance, support, replay, and controller-defined safety checks govern publication to a frozen test-time snapshot. A Process-Constrained Preference Harness binds evidence to its source, rejects controller-invalid candidates, and ranks actions with a safety-prioritized Clinical Process Critic. We evaluate complete agent systems across two backbone endpoints and six controlled stress dimensions under the same Doctor-turn limit. On 300 held-out Qwen encounters, MediSkill-Evo improves diagnosis accuracy from 61.33 percent to 69.00 percent and treatment-intent coverage from 33.62 percent to 66.44 percent, while reducing automatically scored critical failures from 31.00 percent to 16.33 percent relative to AgentClinic. On 180 hard-isolation conditions derived from 30 cases, target recovery reaches 93.61 percent under patient-behavior pressure, 100.00 percent for temporal evidence, and 92.22 percent for triage red flags. An exploratory 100-case MedSAM comparison evaluates request-gated tool-interface feasibility. These results provide descriptive end-to-end evidence for the complete system on fixed evaluation suites, not causal evidence for an individual bank or clinical validation of the automatic judge.

1 Introduction

MediSkill-Evo addresses the challenge of improving clinical agents under partial observability while preserving evidence boundaries and care-process obligations. It combines governed four-bank self-evolution with process-constrained action selection and reports complete-system gains on fixed clinical interaction benchmarks.

Motivation

  • Clinical agents must gather decisive history and examinations, update assessments, and recommend treatment without treating unavailable tests as negative findings.A correct diagnosis can still conceal an unsupported or unsafe interaction trajectory.Source: §1 · p. 1

Scope

  • The system turns evolving external knowledge into controller-valid, evidence-grounded actions without claiming independent clinical legality or safety.Its harness binds results to valid requests, preserves unavailable evidence as unknown, and applies benchmark-defined safety checks without hidden-diagnosis access.Source: §1 · p. 2

Approach

  • MediSkill-Evo evolves clinical experience through four typed banks for strategies, workflow rules, evidence semantics, and visual measurement procedures.Provenance, replay, support, and safety checks govern publication into the next frozen snapshot.Source: §1 · p. 2, §2.1 · p. 3, §2.2 · p. 3
  • The Process-Constrained Preference Harness retrieves state-relevant knowledge, rejects controller-invalid actions, and selects candidates with a safety-prioritized Clinical Process Critic.Hard-invalid candidates cannot return through finite penalties; persistent safety-threshold failure leads to safe termination and human escalation.Source: §1 · p. 2, §2.3 · p. 4

3 Experiments

The experiments evaluate MediSkill-Evo as a complete system across fixed end-to-end, stress, multimodal, and component-comparison suites under controlled interaction settings. Results show stronger process completion on several dimensions, but also reveal bounded performance and prevent single-component causal conclusions.

Experimental setup

  • The evaluation uses two hosted backbone endpoints with fixed cases, environments, evaluation code, and Doctor-turn ceilings; multimodal conditions additionally share cases and interaction settings but not internal compute.The study reports fixed-suite point estimates and explicitly makes no computational-efficiency claim.Source: §3.1 · pp. 4–5

Cross-backbone FullChain results

  • 69.00% diagnosis accuracy and 66.44% treatment-intent coverage are achieved by MediSkill-Evo on 300 held-out Qwen3.6-Flash encounters, versus 61.33% and 33.62% for AgentClinic.Evidence recall rises by 36.12 points, required-history recall by 86.74 points, and critical failures fall from 31.00% to 16.33%.Source: §3.4 · p. 6
  • 47.30% treatment-intent coverage, 40.11% evidence recall, and 73.65% history recall are reported on DeepSeek-V4-Flash, with lower safety violations, critical failures, and unnecessary examinations than AgentClinic.The authors characterize this as consistency within the evaluated endpoint families, not model-family-wide generalization.Source: §3.4 · p. 7

Process Correctness and Safety under Controlled Stress

  • 83.79% stress-process score, 88.30% required-action recall, 82.40% treatment-intent coverage, and 80.03% core score are reported in auxiliary stress results, while diagnosis accuracy is 93.89%.These results support stronger process completion in selected dimensions rather than uniform dominance in diagnosis or safety.Source: §3.5 · p. 7

Modular Visual Measurement on NEJM Cases

  • The exploratory MedSAM-enabled condition is 3.24 points higher in core-case score than the original-image condition on 100 NEJM cases.The authors describe this as tool-interface feasibility evidence, not a MedSAM improvement or segmentation-only causal estimate.Source: §3.6 · p. 8

Component Analysis

  • Component diagnostics show no profile dominates every endpoint, and one frozen run per profile does not establish that any bank is necessary or causally beneficial.Observed shifts support a tradeoff interpretation among banks rather than a single-component explanation of the end-to-end result.Source: §3.7 · pp. 8–9

4 Conclusion

MediSkill-Evo frames clinical-agent evolution as governed process knowledge, combining typed update paths with stress testing of evidence-grounded interaction. The reported results remain bounded system-level evidence rather than clinical validation or causal attribution to individual components.

4 Conclusion

  • Four typed banks provide distinct update and validation paths for reusable strategies, workflow rules, evidence semantics, and visual procedures.The Process-Constrained Preference Harness evaluates these elements jointly during interaction.Source: §4 · pp. 9–10
  • Stress testing shows stronger target recovery under patient-behavior, temporal, and triage pressure, alongside weaker diagnostic-discriminator and unavailable-test acquisition.Source: §4 · p. 9
  • The study does not establish clinical safety, population-level generalization, judge construct validity, or causal credit for individual components.Source: §4 · p. 10
  • The evaluation uses deidentified MIMIC-IV-derived records and published NEJM image cases under their original access, licensing, and redistribution terms.Restricted records and images are not released through the benchmark artifact.Source: §5 · p. 10, Supplementary Materials · p. 12

S1.1 Clinical Skill Bank

The supplementary design separates clinical skills, workflow constraints, evidence semantics, and visual procedures into typed banks with explicit provenance, safety, and lifecycle controls. Post-episode proposals are restricted to reusable, controller-valid updates rather than case answers.

S1.1 Clinical Skill Bank

  • The Clinical Skill Bank stores reusable case strategies with applicability conditions, evidence-acquisition or management sequences, and misuse warnings.Symbolic preconditions and semantic gating remove skills that conflict with visible evidence or diagnostic boundaries.Source: S1.1 Clinical Skill Bank · p. 12
  • Process Rules encode cross-disease workflow constraints, including triggers, required or prohibited actions, release conditions, and priorities.Rules constrain registered state without creating clinical facts or overriding evidence semantics.Source: S1.2 Process Rule Bank · p. 12
  • Symbolic Schemas define legitimate observation sources, permitted state transitions, request–result relations, and provenance-bearing fact consumers.Missing, pending, and unavailable states remain distinct and cannot default to normal or negative.Source: S1.3 Symbolic Schema Bank · p. 12
  • Measurement procedures specify modality, targets, prerequisites, steps, report fields, quality checks, and failure modes while separating procedures from case findings.Source: S1.4 Measurement Bank · p. 12
  • Reflection runs only after training encounters and may propose at most one Process Rule and one Symbolic Schema mutation using existing active identifiers.Proposals must be reusable, avoid hidden targets, and pass anti-leakage and runtime-judgability checks.Source: S2.1 Post-episode trajectory reflection · p. 12, S2.2 Typed Doctor-bank mutation… · pp. 13–14
  • Clinical Skill mutations may add, merge, patch, deprecate, or discard artifacts, with failed cases producing correction strategies rather than memorized answers.Medication, procedure, escalation, and monitoring policies include relevant safety checks.Source: S2.2 Typed Doctor-bank mutation… · p. 13

S2.3 Measurement Agent prompts

The Measurement Agent uses staged visual localization and review to produce evidence reports for the Doctor without issuing final diagnoses. Its prompts separate image observations from non-image evidence and govern reusable measurement-bank updates after completed cases.

S2.3 Measurement Agent prompts

  • A two-stage image prompt first identifies modality and defensible regions, then combines original pixels, optional MedSAM outputs, context, and retrieved measurement skills.Source: S2.3 Measurement Agent prompts · p. 14
  • The visual locator returns one structured entry per panel with modality, segmentation applicability, visible findings, confidence, and at most two ROI boxes.ROI coordinates use a 0..1000 panel frame and are limited to defensible abnormal or decision-salient regions.Source: S2.3 Measurement Agent prompts · p. 14
  • The reviewer independently verifies localization against original pixels, treats MedSAM as a localization aid, and separates non-image evidence from image observations.Unreliable masks are reported as unavailable rather than implying segmentation findings.Source: S2.3 Measurement Agent prompts · p. 15
  • The visual report contains panel findings, mask-derived observations, cross-panel synthesis, limitations, and a segmentation assessment for the Doctor.Source: S2.3 Measurement Agent prompts · pp. 14–15
  • Post-training reflection evaluates whether visual measurement helped, what evidence was missed or overstated, and whether a reusable modality–task checklist should be maintained.Measurement-bank proposals prefer discard when no generalizable visual lesson exists and require verification of mask-derived measurements against original pixels.Source: S2.3 Measurement Agent prompts · p. 15

S2.4 Online candidate generation and preference criticism

Online candidate generation produces distinct next actions from visible state, while a Clinical Process Critic ranks them using evidence alignment, safety, triage, and process constraints. Final-turn candidates must be complete diagnosis-ready plans.

S2.4 Online candidate generation and preference criticism

S2.5 Final risk audit, rewrite, and certification

The evaluation pipeline independently audits clinical risk, rewrites the final response, and certifies release using visible evidence, process obligations, and strict output checks.

Final risk audit, rewrite, and certification

  • The Final Rewriter produces one complete JSON response containing diagnosis, differential diagnoses, evidence, tests, treatment, safety checks, and follow-up or escalation.It cannot request more evidence and must incorporate every evidence-supported required correction.Source: S2.5 Final risk audit, rewrite, and… · pp. 16–17
  • The Release Certifier independently reconstructs the highest-risk problem and releases the answer only when no material evidence-integrity, treatment, medication, or disposition defect remains.The certifier assumes the rewritten diagnosis may be wrong.Source: S2.5 Final risk audit, rewrite, and… · pp. 16–17
  • The diagnosis-blind safety frame identifies visible red flags, high-harm pathways, missing prerequisites, and required monitoring before seeing a proposed diagnosis or treatment.The frame forbids guessing hidden labels, inventing findings, or treating missing evidence as negative.Source: S2.5 Final risk audit, rewrite, and… · p. 16
  • The final risk auditor reconciles medications and procedures with allergies, contraindications, physiology, interactions, prerequisites, red flags, escalation needs, and time-critical care.Unknown prerequisites require active acquisition, a safe alternative, or deferral.Source: S2.5 Final risk audit, rewrite, and… · p. 16
  • Deterministic validators reject unavailable-result citations, controller-invalid sources, missing final fields, invalid action contracts, schema errors, and unsafe memory mutations.The evaluator combines deterministic rules with a semantic judge only where exact matching cannot represent clinical equivalence or observable process quality.Source: S2.6 Prompt-free guards and… · p. 17, S3 Evaluator Prompts and Metric… · p. 17, S3.1 Standard and Multimodal Evaluator… · p. 17
  • The evaluator scores treatment-intent coverage rather than raw drug-string equality and penalizes missing critical intents most strongly.It also caps treatment accuracy at 0.4 for contraindicated or materially unsafe treatment.Source: S3.1 Standard and Multimodal Evaluator… · pp. 17–18

S3.2 Controlled Clinical Stress Evaluator Prompt

Stress V2 evaluates whether agents recover and use controller-released evidence safely across diagnosis, evidence, behavior, treatment, temporal, and triage dimensions.

S3.2 Controlled Clinical Stress Evaluator Prompt

  • Stress V2 separates deterministic target release and unavailability from semantic assessment of visible clinical use and safety.Recovery credit cannot be created by the semantic judge, and hidden source values do not enter runtime prompts merely because they are later evaluated.Source: S3.2 Controlled Clinical Stress… · p. 18
  • The evaluator returns used target IDs, integrated timeline IDs, Boolean process and safety fields, transcript evidence indexes, and a rationale constrained to released targets and existing turns.The controller log is authoritative about release and unavailability.Source: S3.2 Controlled Clinical Stress… · p. 18
  • Diagnosis difficulty measures delayed-discriminator release and use, while evidence completeness measures unavailable-history or unavailable-test requests and safe handling of unavailable responses.Hallucinating an unavailable value is penalized.Source: S3.2 Controlled Clinical Stress… · p. 18
  • Patient-behavior and treatment dimensions measure focused-question recovery and delayed medication, allergy, pregnancy, or renal prerequisite recovery, respectively.The patient-behavior dimension also records unrecovered-target failure, while treatment evaluates prerequisite-sensitive prescription handling.Source: S3.2 Controlled Clinical Stress… · p. 18
  • Temporal and triage dimensions measure delayed timeline-fact recovery and integration, delayed-red-flag recovery, escalation adequacy, and unsafe reassurance.The stress evaluator distinguishes safe conditional treatment, alternatives, or deferral from unsafe action.Source: S3.2 Controlled Clinical Stress… · p. 18, S3.3 Case-Level and Aggregate Formulas · p. 19

S3.3 Case-Level and Aggregate Formulas

  • Case-level formulas define diagnosis, treatment, history, test, safety, critical-failure, efficiency, and stress-process quantities from eligible sets and validated outputs.The auxiliary required-action value combines dimension-specific recovery and use obligations.Source: S3.3 Case-Level and Aggregate Formulas · pp. 18–19
  • Unavailable components are removed and remaining weights renormalized; diagnosis and treatment receive half the auxiliary stress composite’s nominal weight, so it is reported only as an auxiliary summary.Empty required sets receive recall one, empty request sets unnecessary-test rate zero, and missing required treatment receives zero.Source: S3.3 Case-Level and Aggregate Formulas · pp. 18–19
  • Aggregate percentages are macro-averages over declared eligible sets, with fixed-denominator recovery and adverse-event metrics using 30 cases per dimension.HqR and KTR use 10 construction-assigned cases, while conditional metrics follow their eligibility rule.Source: S3.3 Case-Level and Aggregate Formulas · p. 19

S3.4 Registered Comparator and Artifact Ledger

The comparator ledger standardizes training, testing, endpoints, and interaction budgets while documenting limits on reproducibility and clinical interpretation.

S3.4 Registered Comparator and Artifact Ledger

  • All learned Qwen comparators use the same 700 training encounters, publish method-native memory before testing, and remain frozen for the same 300-case test.Only the Doctor varies; Patient, Measurement, moderator/evaluator, case order, and the six-turn ceiling are shared.Source: S3.4 Registered Comparator and… · p. 19
  • Agent-KB, ExPeL, MemP, Reflexion, and SkillWeaver use adapted native memory prompts without the MediSkill-Evo harness, audit, rewrite, or certification path.The ledger distinguishes these comparator adaptations from the proposed system.Source: S3.4 Registered Comparator and… · p. 19, Table S2 · p. 20
  • The registered calls use Qwen3.6-Flash and DeepSeek-V4-Flash aliases through specified endpoints at temperature zero, without portable seeds or immutable provider revisions.The shared budget is a Doctor-turn ceiling rather than a match on semantic items, words, patient burden, internal calls, tokens, latency, or cost.Source: S3.4 Registered Comparator and… · p. 19
  • Method blindness and evidence indexing improve internal comparability but cannot eliminate controller–evaluator rubric alignment, automatically generated target validation, or same-backbone moderator calibration to independent clinical judgment.Absolute clinical interpretation remains unsupported.Source: S3.4 Registered Comparator and… · p. 20

S3.5 Existing-Trace Paired Transitions

The paired-trace section documents frozen FullChain case transitions and one observable clinical encounter, including request handling, evidence use, treatment planning, and scoring.

S3.5 Existing-Trace Paired Transitions

  • Table S3 compares already frozen AgentClinic and MediSkill-Evo outputs across 300 shared FullChain case indices without rollout or re-judging.Metric direction determines which transition is labeled better, and released artifacts include hashes and per-case rows.Source: S3.5 Existing-Trace Paired Transitions · p. 20
  • The recorded case concerns a 24-year-old woman with acute right upper-quadrant pain, with focused history, examination, laboratory, diagnostic, and treatment objectives.The protocol includes registered requestable tests.Source: S4 Text-only FullChain Test Example:… · p. 20, S4.1 Original frozen FullChain test… · p. 20
  • The retrieved memory context supplies biliary triage and acute-abdomen baseline protocols for the encounter.These memories are shown as part of the observable interaction context.Source: S4.2 Complete MediSkill-Evo FullChain… · pp. 20–21
  • The interaction elicits fever, chills, nausea, vomiting, jaundice, medication use, allergies, substance use, exposures, family history, and pregnancy status; one requested test is unavailable.The transcript records the unavailable result explicitly rather than treating it as negative.Source: S4.2 Complete MediSkill-Evo FullChain… · p. 21
  • The scored prediction identifies acute cholecystitis and cites persistent fatty-meal-triggered pain, nausea, vomiting, leukocytosis with neutrophil predominance, and absent fever, chills, or jaundice.The recorded outcome marks the diagnosis-ready prediction correct against the gold diagnosis.Source: T11 DOCTOR · pp. 21–22, S4.2 Complete MediSkill-Evo FullChain… · p. 21
  • The treatment plan requests baseline laboratory testing, withholds medications pending renal, hepatic, bleeding, allergy, and pregnancy verification, and seeks alternative imaging because ultrasound is unavailable.It also specifies urgent admission or emergency transfer, monitoring, serial examinations, and escalation triggers.Source: T11 DOCTOR · pp. 21–22

S5 NEJM Boundary Case: Original Record and Paired Interaction

The NEJM boundary case uses a frozen interactive record in which the agent must gather focused evidence and request available tests before diagnosing. Evaluation reserves reference information and unrevealed results from the interaction.

S5.2 Paired test protocol and outcomes

The paired protocol compares identical interactive records under local no-MedSAM and remote with-MedSAM conditions. It controls the model, banks, evaluator, request gate, and inference budget, while acknowledging that the comparison is not tool-only.

S5.3 Local condition: Measurement learning without MedSAM

The local condition develops Measurement behavior without MedSAM and conducts a request-gated multimodal encounter using qualitative morphology rather than automated segmentation metrics. Its final diagnosis is scored incorrect against the recorded gold diagnosis.

S5.4 Remote condition: Measurement learning with MedSAM

The remote condition enables MedSAM and produces segmented multimodal observations during the same style of request-gated encounter. The system reaches the recorded diagnosis, but the trace remains a qualitative failure-boundary illustration rather than clinical or causal evidence.

Related papers

Loading 2608.23397v1…