PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Distribution Matching for State-to-Vision Policy Distillation in Laparoscopic Robotic Manipulation

Mingyu Sheng, Jianan Fan, Dongnan Liu, Guoyan Zheng, Ron Kikinis, Weidong Cai

arXiv:2610.31010v1cs.ROcs.CV

TL;DR

Surgical manipulation policies based on privileged states rely on physical-state information that can be difficult to obtain reliably, while vision-based RL remains challenging. The paper distills a state-based teacher into a vision-based policy through distribution matching. It reports stronger stability and success on complex tasks and superior performance and robustness on relatively complex tasks in SurRoL.

  • Problem

    Privileged-state policies rely on accurate physical states that are difficult to obtain reliably, while vision-based RL for surgical manipulation remains challenging.

    Source: §I · p. 1

  • Method

    The framework trains a privileged state-based teacher and distills its stochastic action distribution to a vision-based student using RGB observations and robot proprioceptive information.

    Source: §I · p. 2, B. State-to-Vision Policy Distillation · p. 4

  • Results

    The method shows stronger stability and higher success rates on complex tasks, with superior performance and robustness on relatively complex SurRoL tasks.

    Source: §I · p. 2, E. Conclusion · p. 7

  • Takeaways & Limitations

    The analysis of distribution-matching objectives offers insights for future vision-based policy learning in surgical robotic manipulation.

    Source: §I · p. 2

Abstract

from arXiv · show

End-to-end control policies trained via Reinforcement Learning (RL) have emerged as an alternative to conventional surgical robotic control approaches, with recent studies exploring state-based policies trained on privileged state observations. However, such approaches rely on precise physical states that are difficult to obtain reliably in surgical environments. Vision-based RL provides an alternative by learning manipulation policies directly from RGB observations, avoiding explicit state estimation and manually engineered perception pipelines while enabling policies to acquire task-relevant visual representations. Nevertheless, obtaining vision-based policies via pure RL methods for surgical manipulation remains challenging. Thus, imitation learning (IL) becomes another solution, yet it requires a large number of expert demonstrations, which are difficult to obtain in minimally invasive surgery. In this work, we propose a framework for state-to-vision policy distillation in laparoscopic robotic manipulation. Specifically, a state-based policy with privileged observations is employed as the teacher, and a vision-based policy learns from RGB observations and robot proprioceptive information, which requires less expert demonstrations than IL and introduces superior performance than RL methods. We conduct extensive experiments on various surgical manipulation tasks in the simulated environment. Experimental results show that, in the context of laparoscopic robotic manipulation, training vision-based policies with general-purpose RL methods is challenging, whereas our proposed state-to-vision policy distillation leads to more consistent and stable performance across tasks.

I. INTRODUCTION

Surgical manipulation policies trained with privileged states face a reliance on accurate physical-state information, while vision-based RL remains difficult and expert demonstrations for imitation learning are scarce. The paper proposes continuous-control state-to-vision distillation and analyzes distribution-matching objectives, reporting stronger stability and success on complex tasks.

  • Privileged-state RL typically assumes accurate, noise-free physical states, and even minor needle-pose deviations can cause task failure.Source: §I · p. 1
  • Vision-based RL struggles with sparse task-relevant pixels and unstable training, while laparoscopic expert demonstrations are expensive and scarce.Source: §I · p. 1
  • The framework distills a privileged state-based teacher into a vision-based student by matching stochastic policy distributions, reducing the demand for expert demonstrations.The student uses visual observations and robot proprioceptive information.Source: §I · p. 2
  • The proposed method shows stronger stability and higher success rates on complex surgical robotic manipulation tasks.The paper also reports a unified first- and second-order gradient analysis of distribution-matching objectives for Gaussian policies.Source: §I · p. 2
  • The work evaluates distillation across surgical manipulation tasks and analyzes gradient scaling, parameter coupling, and curvature for Gaussian-policy objectives.Source: §I · p. 2

II. RELATED WORK

Prior work spans image-based RL, learning-based surgical manipulation, and policy distillation, but the cited approaches leave challenges in sparse visual control, efficient surgical vision policies, and continuous-control distillation. The paper’s workflow first trains a state-based policy, then transfers its behavior to a student by aligning action distributions.

A. Vision-Based Reinforcement Learning

  • Image-based RL methods face difficult visual learning and scarce expert demonstrations in surgical manipulation, where prior methods are challenged compared with humanoid control.Source: A. Vision-Based Reinforcement Learning · p. 2

B. Learning-Based Surgical Robotic Manipulation

  • Surgical manipulation commonly uses privileged object states and a perception-then-control pipeline, while efficient vision-only policies remain an open problem.Some vision-based approaches add depth or segmentation inputs, or use pretrained large models with high computational costs and potential error propagation.Source: B. Learning-Based Surgical Robotic… · p. 2

C. Policy Distillation

  • Policy distillation can reduce reliance on expert demonstrations and provide relatively stable supervised training, but prior work mostly addresses discrete actions rather than continuous laparoscopic control.Source: C. Policy Distillation · p. 2

III. METHODS

  • The workflow trains a state-based policy first, then uses it as teacher to transfer policy knowledge by aligning action distributions; a fresh replay buffer avoids transitions from earlier teacher versions.Source: Figure 1 · p. 3

A. Background

The background presents Soft Actor-Critic as an off-policy method that learns stochastic policies and critics. Its policy is Gaussian in continuous control, with action dimensions treated as conditionally independent and sampled through reparameterization.

  • Soft Actor-Critic learns a stochastic policy to maximize expected cumulative return and trains two critics by minimizing Bellman error.Source: A. Background · p. 3
  • The SAC target value uses the next sampled policy action, target critics, and an entropy term weighted by α.The target critics use exponential moving averages, which the passage says empirically improve off-policy training stability.Source: A. Background · p. 3
  • Continuous-control policies are parameterized as factorized Gaussian distributions, assuming conditional independence across action dimensions for tractable likelihoods and reparameterized sampling.Source: A. Background · p. 3

B. State-to-Vision Policy Distillation

The section formulates state-to-vision distillation as matching stochastic Gaussian teacher and student policies. A frozen state-based teacher guides the visual student through a differentiable loss.

B. State-to-Vision Policy Distillation

  • Gaussian policies let distillation transfer both action tendencies and confidence, rather than only mean actions.The teacher and student explicitly model means and standard deviations, which represent policy uncertainty.Source: B. State-to-Vision Policy Distillation · p. 3
  • The derivation treats one joint at a time because different joints are assumed conditionally independent.Source: B. State-to-Vision Policy Distillation · p. 4
  • The student matches a frozen teacher’s policy distribution, transferring its behavior by minimizing a differentiable discrepancy between the policies.The teacher is state-based and privileged; the student uses visual observations.Source: B. State-to-Vision Policy Distillation · p. 4

C. Distribution Matching Objectives

The section treats transfer between stochastic Gaussian policies as distribution matching and compares divergences and distances as distillation objectives. These objectives differ in their optimization behavior and in how they handle policy uncertainty.

C. Distribution Matching Objectives

MATCHING OBJECTIVES WITH RESPECT TO THE VISUAL POLICY

  • WD accounts for differences in both mean and standard deviation, while MSE considers only the difference between means.For Gaussian distributions, the squared 2-Wasserstein distance has a closed-form expression.Source: MATCHING OBJECTIVES WITH RESPECT TO… · p. 4
  • WD, MSE, and RKL have strictly convex landscapes in student mean and standard deviation, while FKL and JD use standard-deviation-scaled gradients.Source: MATCHING OBJECTIVES WITH RESPECT TO… · p. 4

D. Gradient Analysis

The gradient analysis compares how distribution-matching objectives update the visual policy mean. Some objectives scale those updates according to policy uncertainty, while others do not.

D. Gradient Analysis

  • All objectives drive the visual policy mean toward the state policy mean, while FKL, RKL, and JD scale gradient magnitude using policy standard deviations.FKL, RKL, and JD assign larger magnitudes to dimensions with smaller standard deviations, adapting mean optimization to policy confidence.Source: D. Gradient Analysis · p. 4
  • The positive-definite region of the JD Hessian is broader than the FKL region.Source: Figure 2 · p. 5
  • WD and MSE have the same mean objective, minimizing squared mean difference with gradients independent of teacher and student standard deviations.Under teacher-driven distillation, WD’s variance-matching term has minimal effect on the mean gradient or interaction with the environment.Source: D. Gradient Analysis · p. 5

E. Positive-Definiteness of Different Hessian Matrices

The section uses Hessian curvature to characterize the local optimization landscapes of the matching objectives. RKL, WD, and MSE are positive definite, whereas FKL and JD are not positive definite throughout their domains.

E. Positive-Definiteness of Different Hessian Matrices

  • A positive-definite Hessian indicates positive curvature in every direction and a locally strictly convex landscape around the point considered.Source: E. Positive-Definiteness of Different… · p. 5
  • RKL, WD, and MSE have positive-definite Hessians, with mean and standard-deviation parameters decoupled for these objectives.For RKL, the mean gradient is scaled by a fixed factor determined by teacher uncertainty.Source: E. Positive-Definiteness of Different… · p. 5
  • FKL and JD Hessians are not always positive definite, and their uncertainty-adaptive mean-gradient scaling differs between student-only and joint teacher-student uncertainty.FKL scaling depends on student uncertainty; JD scaling depends on both teacher and student uncertainty.Source: E. Positive-Definiteness of Different… · p. 5
  • The JD Hessian has a broader positive-definite domain than FKL, so JD admits a larger region of local convexity.Source: E. Positive-Definiteness of Different… · p. 5

F. Relaxed Positive-Definiteness Condition

The analysis establishes that JD has a strictly larger positive-definite Hessian region than FKL, giving JD a more relaxed local-convexity condition.

F. Relaxed Positive-Definiteness Condition

  • The positive-definite region of FKL is a proper subset of JD’s, so every point with a positive-definite FKL Hessian also has a positive-definite JD Hessian.The proposition also states that some points have a positive-definite JD Hessian while the FKL Hessian is not positive definite.Source: F. Relaxed Positive-Definiteness… · p. 5
  • At µt = 0, σt = 0.5, µs = 0.75, and σs = 1.0, det(HFKL) = −0.8125 while det(HJD) = 29.9375.The example uses these determinants to show a point where the JD Hessian is positive definite but the FKL Hessian is not.Source: F. Relaxed Positive-Definiteness… · p. 6
  • JD’s larger region of local convexity can yield more favorable local optimization than FKL.The paper says JD does more than symmetrize the divergence and balance mode covering and mode seeking.Source: F. Relaxed Positive-Definiteness… · p. 6

A. Implementation Details

The experiments use a frozen privileged-state teacher to supervise RGB-and-proprioception students, then compare task success across methods and distillation losses. Distillation is relatively stable on complex tasks, and uncertainty-adaptive losses usually outperform non-adaptive ones there.

A. Implementation Details

  • The teacher receives privileged object and robot states, while students use ECM RGB frames and PSM gripper coordinates without other-object states.Teachers are trained with SAC and frozen for student supervision; experiments use five random-seed runs, each evaluated over 100 trials, with the JD loss scaled by 0.5 for gradient-magnitude fairness.Source: A. Implementation Details · p. 6

B. Results Comparison Across Different Tasks

  • Policy distillation maintains about 70% success on complex tasks, showing relatively stronger stability and robustness despite lower performance than ALOHA on GauzeRetrieve and NeedlePick.Pure vision RL performs well on NeedleReach but is challenged on NeedlePick and NeedleRegrasp; ALOHA also degrades on PegTransfer and NeedleRegrasp.Source: B. Results Comparison Across Different… · p. 6

C. Results Comparison Across Different Distillations

  • Uncertainty-adaptive FKL and JD usually outperform non-adaptive RKL and MSE on complex tasks, while simpler tasks show no significant gain from uncertainty adaptation.JD is slightly better than FKL across tasks; RKL reaches 39.8% on NeedlePick, where the authors hypothesize instability may relate to prior-scaled gradients and teacher standard deviation around 0.3.Source: C. Results Comparison Across Different… · pp. 6–7

D. Feature Map Analysis

The feature-map analysis reports activations that track the gripper and target object, while the conclusion describes the framework’s scope and possible architectural extensions.

D. Feature Map Analysis

  • High-activation regions follow the gripper and target object as their positions change, while background regions remain weakly activated.The authors interpret this as task-relevant semantic representation of the gripper-object relative spatial configuration, explaining robustness to object-position changes.Source: D. Feature Map Analysis · p. 7

E. Conclusion

  • In SurRoL, distribution-matching distillation transfers state-policy knowledge to vision policies and shows superior performance and robustness on relatively complex manipulation tasks.The conclusion also reports potential effectiveness of uncertainty adaptation and examines feature maps for task-relevant semantic understanding.Source: E. Conclusion · p. 7
  • The experiments use a simple three-convolutional-layer, three-fully-connected-layer network and RGB frames plus robot posture states, leaving advanced mechanisms such as attention, multi-scales, and mixture-of-experts unused.The paper presents this as indicating potential scalability to enhance performance.Source: E. Conclusion · p. 8

Related papers

Loading 2610.31010v1…