PaperlayerView exact revision on arXiv

Source-linked AI summary

MedVLA: A Hierarchical Vision-Language-Action Framework for Closed-Loop Precision Medical Robot Manipulation

Junjie Xie, Chuxuan He, Angen Ye, Yujia Song, Dapeng Zhang

arXiv:2609.25756v1cs.RO

TL;DR

Continuous-action VLA paradigms are not well suited to precision medical tasks that require safety-conscious, interpretable decision-making and reliable closed-loop operation. MedVLA decouples high-level judgment from low-level execution, gates structured function calls against state-based preconditions, and uses a multi-agent pipeline to generate skill-oriented training data. In system-level validation on a real platform, MedVLA achieved a 95.0% success rate in complete closed-loop tasks, outperforming OpenVLA and π0 by 87.0 and 80.0 percentage points, respectively.

  • Problem

    Continuous-action VLA paradigms are not well suited to precision medical tasks that require safety-conscious, interpretable decision-making and reliable closed-loop operation.

    Source: §I · p. 1

  • Method

    MedVLA decouples high-level judgment from low-level execution, gates structured function calls against state-based preconditions, and uses a multi-agent pipeline to generate skill-oriented training data.

    Source: §I · pp. 2–3, C. Structured Data Generation via… · pp. 3–4

  • Results

    In system-level validation on a real platform, MedVLA achieved a 95.0% success rate in complete closed-loop tasks, outperforming OpenVLA and π0 by 87.0 and 80.0 percentage points, respectively.

    Source: §I · p. 3

  • Takeaways & Limitations

    The results support structured reasoning with constrained function-level execution as a practical route toward deployable precision medical robotics.

    Source: §IV · p. 7

    Evaluation covers one task with training data generated by the proposed pipeline, and the gate relies on task-specific state-based rules.

    Source: §IV · p. 8

Abstract

from arXiv · show

Precision medical robotics demands adaptive decision-making under strict safety, interpretability, and execution constraints. Although recent Vision-Language-Action (VLA) models show strong multimodal reasoning ability, their continuous action generation paradigm is not well suited for precision medical tasks, where reliable closed-loop operation may also depend on non-action system function calls. To address this gap, we propose MedVLA, a hierarchical framework that couples high-level multimodal reasoning with low-level function-constrained execution. We further introduce a scalable multi-agent pipeline to generate skill-oriented chain-of-thought(CoT) data for structured training. Built on different multimodal large-model backbones, MedVLA consistently improves performance after fine-tuning, demonstrating the effectiveness of the proposed framework across model variants. Under identical initial conditions, we perform 100 closed-loop flexible electrode implantation trials. The results show that MedVLA achieves a 95.0\% task success rate, substantially outperforming representative VLA baselines, including OpenVLA (8\%) and $π_0$ (15\%), in accuracy, stability, and safety. These results indicate that structured reasoning with constrained function-level execution is a practical route toward deployable precision medical robotics.

I. INTRODUCTION

MedVLA addresses the mismatch between open-world VLA action generation and precision medical tasks, which require safety-conscious, state-aware closed-loop decisions. It combines gated function-level execution with structured training data and reports 95.0% success in real-platform closed-loop validation.

I. INTRODUCTION

  • Precision medical tasks demand continuous perception and dynamic correction in narrow, high-risk workspaces with millimeter-scale or smaller operating areas.Robots must jointly consider visual targets, instrument state, task phase, and operation goals during feedback-driven execution.Source: §I · p. 1
  • Existing general-purpose VLA paradigms remain mismatched to precision medical requirements, while costly data collection and limited task-specific datasets constrain structured closed-loop learning.The cited needs include explicit phase recognition and structured workflow representations.Source: §I · p. 1
  • The framework integrates chain-of-thought reasoning, state-based preconditions, and function-call gating to make decisions auditable and constrain unsafe or premature operations.The gate checks physical task state, not only whether a function belongs to the action space.Source: §I · p. 2
  • MedVLA separates high-level judgment from low-level execution to achieve safe closed-loop control over a constrained skill space.Its structured function calls pass through an explicit state-based gate before execution, rather than reaching the executor as open-ended continuous actions.Source: §I · pp. 2–3
  • A skill-oriented multi-agent pipeline constructs approximately 30,000 structured samples with low redundancy and strong generalization.The pipeline is presented as a data foundation for closed-loop decision learning.Source: §I · p. 3
  • In system-level validation on a real platform, the proposed method achieved 95.0% success in complete closed-loop tasks, outperforming OpenVLA and π0 by 87.0 and 80.0 percentage points, respectively.Source: §I · p. 3

A. Task Formulation and Problem Modeling

The paper models precision medical robot tasks as iterative, state-constrained function-call decisions rather than direct regressions from images to continuous actions. Each decision uses visual observations, task intent, and robot state to produce a structured triplet and an allowed next function.

A. Task Formulation and Problem Modeling

  • The task is formulated as iterative state-constrained function calls, not direct regression from visual observations to continuous actions.The loop repeats until termination.Source: A. Task Formulation and Problem Modeling · p. 3
  • At each step, binocular image observations, user task input, and internal robot state jointly inform the system decision.The images capture the surgical scene and local geometry, the input specifies task intent, and state records execution stage and decision variables.Source: A. Task Formulation and Problem Modeling · p. 3
  • The model predicts a structured triplet containing reasoning, an intermediate judgment, and the next function call.This replaces the conventional mapping from image and task input directly to an action.Source: A. Task Formulation and Problem Modeling · p. 3

B. Hierarchical Closed-Loop Control Framework

The control framework links multimodal decision-making to low-level execution through state-aware prompting, function gating, and feedback. Rejected functions trigger a conservative stop and transfer control to a human operator.

B. Hierarchical Closed-Loop Control Framework

  • The high-level module combines robot state with user instruction and maps image observations to an explicit, interpretable decision triplet.The structured output comprises reasoning, an intermediate judgment, and a function call.Source: B. Hierarchical Closed-Loop Control… · p. 3
  • The middle gate checks predicted functions for action-space validity and consistency with current state and domain-specific preconditions before execution.For implantation, checks include needle–hole alignment, vision-system readiness, and workspace initialization.Source: B. Hierarchical Closed-Loop Control… · p. 3
  • A rejected function triggers an emergency stop, hardware self-check, audible alarm, and handover to a human rather than autonomous recovery.The design favors a safe halt when recovery could lead to an uncertain state.Source: B. Hierarchical Closed-Loop Control… · p. 3
  • After execution, updated robot state and visual observations feed the next iteration of the Perceive–Reason–Gate–Execute–Update loop.Source: B. Hierarchical Closed-Loop Control… · p. 3

C. Structured Data Generation via Multi-Agent Collaboration

MedVLA's data pipeline uses collaborating agents to generate skill-oriented structured examples from task materials and initial system states. Redundancy reduction and layered checks produce a validated dataset aligned with state-aware, function-constrained, closed-loop decisions.

C. Structured Data Generation via Multi-Agent Collaboration

  • The pipeline combines task documents, robot capabilities, action rules, and initial states to generate user prompts and structured robot responses.A Human Agent simulates users, while a Robot Agent produces responses to prompts paired with system states.Source: C. Structured Data Generation via… · p. 3
  • Each skill-oriented sample centers on one executable function call, and prompt-state pairs are embedded and deduplicated by cosine similarity.The retained responses follow a structured think–answer–function format.Source: C. Structured Data Generation via… · p. 4, Figure 2 · p. 4
  • Hard-constraint, soft-constraint, and semantic-coherence checks filter generated examples before they enter the validated training dataset.Checks cover function validity and action-space limits, consistency between observations and answers, answer-function alignment, and coherence across the response chain.Source: C. Structured Data Generation via… · p. 4, Figure 2 · p. 4
  • The pipeline is designed to reduce redundancy and enforce constraint consistency, aligning training data with state-aware, function-constrained, closed-loop tasks.Fig. 2 depicts prompt generation, prompt-state embedding and deduplication, structured response generation, and validation.Source: C. Structured Data Generation via… · p. 4, Figure 2 · p. 4

A. Experimental Platform, Setup, and Evaluation Protocol

The evaluation uses a real flexible-electrode implantation platform with micrometer-scale geometric constraints. It combines offline function prediction with real-robot closed-loop execution across multiple model backbones and VLA baselines.

A. Experimental Platform, Setup, and Evaluation Protocol

  • A tungsten needle with an approximately 10 µm tip must pass through a 30 × 50 µm electrode hole on the real implantation platform.The task is evaluated through both offline function prediction and real-robot closed-loop execution.Source: A. Experimental Platform, Setup, and… · p. 4, Figure 3 · p. 5
  • The seven-function decision space groups operations into image regulation, state management, safety control, and action execution.ALIGN NEEDLE and INSERT are the two execution-critical functions directly associated with implantation.Source: A. Experimental Platform, Setup, and… · p. 4
  • Training uses approximately 3,000 real-world samples expanded to about 30,000 through controlled image perturbations, with InternVL2-2B, Qwen2.5-VL-3B, and Llama-3.2-11B fine-tuned.OpenVLA-7B and π0-3.3B are evaluated in closed-loop execution; offline metrics are Exact Match and Macro-F1, while real-robot evaluation uses insertion success rate.Source: A. Experimental Platform, Setup, and… · pp. 4–5

B. Offline Function Prediction Results

The full Qwen2.5-VL-3B model leads the reported offline function-prediction results, while CoT effects vary by function and competitive results across backbones support generality.

B. Offline Function Prediction Results

  • CoT raises ADJUST FOCUS from 65.84% to 96.71% and ALIGN NEEDLE from 48.67% to 72.57%, but INSERT falls from 93.33% to 81.67%.The reported CoT effect is class-dependent.Source: B. Offline Function Prediction Results · p. 5
  • Overall accuracy rises from 85.30% to 95.00% with CoT; at three alignment rounds, closed-loop insertion success is 95% versus 73%.The paper reports that additional alignment calls are less costly than premature insertion.Source: B. Offline Function Prediction Results · p. 5
  • InternVL2-2B and Llama-3.2-11B also achieve competitive results, with Llama highest in overall accuracy and Qwen favored for its smaller size and practical trade-off.Qwen2.5-VL-3B is selected for real-robot experiments.Source: B. Offline Function Prediction Results · p. 5
  • Remaining errors concentrate in ADJUST FOCUS, ALIGN NEEDLE, and INSERT, while four adjustment, recovery, and safety functions reach near-perfect or perfect accuracy.The paper identifies visually ambiguous intermediate judgments and precision-critical execution decisions as the main difficulty.Source: B. Offline Function Prediction Results · pp. 5–6
  • 95.00% accuracy and 0.9604 Macro-F1 are reported for full Qwen2.5-VL-3B, substantially outperforming zero-shot baselines.Table II reports offline function prediction across zero-shot evaluation, cross-backbone fine-tuning, and CoT ablation.Source: B. Offline Function Prediction Results · p. 5, Table II · p. 6

C. Closed-Loop Comparison in Real Execution

In 100 closed-loop trials, the framework achieved a 95.00% success rate with six decision rounds, compared with 8.00% for OpenVLA and 15.00% for π0. The evaluation also examines deployment latency and the gating module’s role in rejecting unsafe outputs.

C. Closed-Loop Comparison in Real Execution

  • 95.00% success with six decision rounds exceeded OpenVLA’s 8.00% and π0’s 15.00% in the unified 100-trial protocol.With the maximum limited to three rounds, the framework achieved 73.00%.Source: C. Closed-Loop Comparison in Real… · p. 6
  • The authors attribute the higher real-execution success to conservative commitment before INSERT, which appears to avoid premature or risky insertion attempts.They note this conservativeness may slightly reduce offline INSERT accuracy.Source: C. Closed-Loop Comparison in Real… · p. 6
  • The authors selected OpenVLA and π0 to represent autoregressive action tokenization and flow-matching continuous control, respectively.They state this comparison isolates hierarchical function-level execution against end-to-end continuous action generation.Source: C. Closed-Loop Comparison in Real… · p. 6
  • The approximately 10-second deployment decision cycle comprises ∼8 seconds of LVLM inference and ∼2 seconds for state update and gating verification.The runtime decision rate is ∼0.1 Hz, distinct from the 5 Hz offline trajectory-sampling rate; the authors consider it acceptable for the current quasi-static task.Source: C. Closed-Loop Comparison in Real… · p. 6
  • Across over 500 test-set inference trials, the gate rejected predicted functions twice; both cases triggered E STOP, a hardware self-check, and an audible alarm before human handoff.A physical ablation disabling the gate was not conducted because arbitrary outputs could reach micrometer-scale hardware.Source: C. Closed-Loop Comparison in Real… · pp. 6–7

D. Real-World Closed-Loop Case Study

A representative deployment using fine-tuned Qwen2.5-VL-3B-Instruct proceeds from binocular microscopic observations through three structured functions to complete electrode-hole insertion.

D. Real-World Closed-Loop Case Study

IV. CONCLUSION

The conclusion presents MedVLA as a hierarchical framework for flexible electrode implantation and reports a 95% closed-loop success rate. It also identifies evaluation scope and task-specific gating rules as boundaries on broader applicability.

IV. CONCLUSION

  • The framework achieves a 95% closed-loop success rate in flexible electrode implantation under micrometer-level constraints.Source: §IV · p. 7
  • New task actions can be supported by extending the multi-agent data pipeline and fine-tuning.Source: §IV · p. 8
  • Evaluation covers one task with pipeline-generated training data, and broader applicability remains bounded by task-specific state-based gating rules.The authors identify independent datasets, alternative surgical tasks, external-operator data, and richer medical-domain constraints as directions for strengthening generalization and clinical applicability.Source: §IV · p. 8
Loading 2609.25756v1…