PaperlayerView exact revision on arXiv

Source-linked AI summary

When would Vision-Proprioception Policies Fail in Robotic Manipulation?

Jingxian Lu, Wenke Xia, Yuxuan Wu, Zhiwu Lu, Di Hu

arXiv:2602.12032v1cs.RO

TL;DR

Vision-proprioception policies have shown inconsistent generalization, and this paper investigates when vision is underused. It introduces GAP, which adjusts proprioception’s gradient according to estimated motion-transition probabilities; experiments report superior performance across varied settings.

  • Problem

    Prior studies report inconsistent outcomes when proprioception is added to vision-based manipulation policies, leaving unclear when vision-proprioception policies fail.

    Source: §1 · p. 1

  • Method

    GAP estimates motion-transition probabilities from proprioceptive signals and uses them to reduce proprioception’s gradient magnitude during policy learning.

    Source: §4.2 · p. 5, §4.3 · p. 6

  • Results

    GAP-equipped vision-proprioception policies consistently outperform across the reported tasks, policy architectures, robot setups, and simulated and real-world environments.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings identify motion-transition phases as a key setting for visual underuse and support dynamically balancing vision and proprioception in these policies.

    Source: §1 · p. 2, §7 · p. 10

    All vision-proprioception policies were trained on a single embodiment; the role of proprioception in cross-embodiment datasets remains future work.

    Source: §7 · p. 10

Abstract

from arXiv · show

Proprioceptive information is critical for precise servo control by providing real-time robotic states. Its collaboration with vision is highly expected to enhance performances of the manipulation policy in complex tasks. However, recent studies have reported inconsistent observations on the generalization of vision-proprioception policies. In this work, we investigate this by conducting temporally controlled experiments. We found that during task sub-phases that robot's motion transitions, which require target localization, the vision modality of the vision-proprioception policy plays a limited role. Further analysis reveals that the policy naturally gravitates toward concise proprioceptive signals that offer faster loss reduction when training, thereby dominating the optimization and suppressing the learning of the visual modality during motion-transition phases. To alleviate this, we propose the Gradient Adjustment with Phase-guidance (GAP) algorithm that adaptively modulates the optimization of proprioception, enabling dynamic collaboration within the vision-proprioception policy. Specifically, we leverage proprioception to capture robotic states and estimate the probability of each timestep in the trajectory belonging to motion-transition phases. During policy learning, we apply fine-grained adjustment that reduces the magnitude of proprioception's gradient based on estimated probabilities, leading to robust and generalizable vision-proprioception policies. The comprehensive experiments demonstrate GAP is applicable in both simulated and real-world environments, across one-arm and dual-arm setups, and compatible with both conventional and Vision-Language-Action models. We believe this work can offer valuable insights into the development of vision-proprioception policies in robotic manipulation.

1 Introduction

Prior studies disagree on whether adding proprioception improves manipulation policies. This work finds that vision-proprioception policies underuse vision during motion transitions, proposes GAP to address that imbalance, and reports broad performance gains.

1 Introduction

  • Prior findings on vision-proprioception policies conflict: HPT reported clear gains from combining modalities, whereas Octo observed worse policies with added proprioception.The paper frames this discrepancy as an unresolved question about when such policies fail in manipulation.Source: §1 · p. 1
  • Interventions show that switching from a vision-only to a vision-proprioception policy has little impact during motion-consistent phases but degrades performance during motion transitions.The controlled simulation experiment replaced actions for a 10-timestep period; Figure 1 reports the overall policy comparison as well as the phase intervention.Source: §1 · p. 2, Figure 1 · p. 2
  • During motion transitions, subtle visual cues lead the policy to favor concise proprioceptive signals, dominating optimization and suppressing visual learning.This underuse of vision is consequential when target localization is needed.Source: §1 · p. 2
  • GAP estimates each timestep’s probability of belonging to a motion-transition phase and reduces the magnitude of proprioception’s gradient accordingly.It uses proprioceptive motion signals to segment motion-consistent phases and a temporal network to model transitions.Source: §1 · p. 2
  • Vision-proprioception policies with GAP consistently outperform vision-only policies across the reported settings, including varied tasks, architectures, and one- and dual-arm setups.The evaluations also span simulated and real-world environments and conventional and Vision-Language-Action models.Source: §1 · p. 2

2 Related Works

Prior work reports mixed effects from adding proprioception to manipulation policies and describes modality contributions as varying over time. These observations motivate examining vision and proprioception through a temporal lens.

2.1 Vision-Proprioception Policy in Manipulation.

  • Studies report gains from combining vision and proprioception as well as limited gains or detrimental effects, with proposed explanations including overfitting and causal confusion.The section presents the disagreement as an open issue for understanding modality use in manipulation policies.Source: §2.1 · p. 3

2.2 Modality Temporality.

  • Modality temporality describes how each modality’s contribution changes over a task: vision can locate an object, while proprioception supports precise movement toward it.Prior work has proposed dynamic fusion and modality selection in response to this temporal variation.Source: §2.2 · p. 3

3 Optimization Analysis of Vision-Proprioception Policies

The section formulates vision-proprioception policy learning under behavior cloning and analyzes how joint feature learning can suppress vision. This suppression can impair generalization when target locations vary and require visual localization.

3 Optimization Analysis of Vision-Proprioception Policies

  • The policy uses RGB observations, with proprioception added for vision-proprioception policies; the proprioceptive state includes gripper pose and opening degree.The gripper pose is represented in Cartesian space and orientation, and opening degree ranges from fully closed to fully open.Source: §3 · p. 3
  • The standard joint-learning design extracts vision and proprioception features separately, concatenates them, and feeds them to a policy head under behavior-cloning training.The section focuses its analysis on vanilla mean-squared-error loss for continuous action spaces.Source: §3 · pp. 3–4
  • Subtle visual cues can make proprioceptive signals dominate training-loss reduction, suppressing vision-feature learning during task execution.For example, gripper closing may be visually distinguishable only at the pixel level, while proprioception directly records gripper opening degree.Source: §3 · p. 4
  • When test-time target positions vary, suppressed vision learning can impair generalization during motion transitions that require locating objects.The same proprioceptive overreliance has negligible impact during motion-consistent phases, where ongoing movement benefits from proprioception.Source: §3 · p. 4

4 Method

GAP estimates motion-transition probabilities from proprioception and uses them to adjust proprioceptive optimization. The method aims to preserve visual learning while enabling dynamic collaboration between vision and proprioception.

4 Method

  • GAP uses motion-transition indicators to adjust proprioceptive optimization during vision-proprioception policy learning.The indicators estimate each timestep’s probability of belonging to a transition phase.Source: §4 · p. 4

4.1 Motion Representation of Robot

  • Robot motion is represented by changes in gripper position, orientation, and opening, which support comparisons between phase motion and adjacent motion.The representation is built from proprioceptive signals.Source: §4.1 · p. 4, §4.2 · p. 4

4.2 Motion-Transition Phase Estimation

  • CPD segments demonstrations into motion-consistent phases, while an LSTM uses temporal proprioceptive differences to estimate transition probabilities supervised by those segments.Transition intervals are continuous, so nearby timesteps receive a reduced penalty to capture gradual transitions.Source: §4.2 · pp. 4–5

4.3 Gradient Adjustment for Modality Collaboration

  • The proprioceptive chunk’s backward gradient is modulated according to each timestep’s transition probability, with higher probabilities producing greater modulation.The learning rate and adjustment hyperparameter also control the update.Source: §4.3 · p. 6

5 Experiments

The experiments evaluate GAP across simulated and real-world manipulation, testing whether it improves vision-proprioception policies and generalizes across model and fusion types.

5.1 Experimental Setup

  • The benchmarks include Meta-World and RoboSuite simulations, xArm 6 one-arm tasks, and Cobot Magic dual-arm tasks.Simulation and real-world tasks are evaluated with 100 and 20 rollouts, respectively.Source: §5.1 · p. 6, Figure 3 · p. 7

5.2 Can GAP lead to more robust vision-proprioception policies?

  • GAP-equipped vision-proprioception policies outperform vision-only policies and other methods in simulated and real-world comparisons.MS-Bot's semantic stages offer marginal benefits on transition-heavy tasks; whole-task visual auxiliary loss falls short on threading, while fixed-probability proprioception masking yields minimal improvement.Source: §5.2 · p. 7, Table 1 · p. 7

5.3 Does GAP enhance the utilization of vision modality?

  • GAP reduces vision suppression during motion transitions and improves generalization under out-of-distribution object-position changes.The intervention experiment indicates increased vision utilization after GAP is applied.Source: §5.3 · p. 8, Figure 4 · p. 8, Table 2 · p. 8

5.4 Is GAP compatible with Vision-Language-Action models?

  • 17% average improvement is reported for GAP-equipped Octo-VP†, which also shows stronger generalization than Octo-V.The comparison evaluates fine-tuned Octo with visual input alone or with vision and proprioception.Source: §5.4 · pp. 8–9, Table 3 · p. 9

5.5 Can GAP be applied to various modality fusion approaches?

  • With GAP, Concatenation, Summation, and FiLM policies outperform vision-only policies.Vision-only policies suffice for pick-place and put-hammer-into-drawer but fail on threading and struggle with occluded push-wall targets; FiLM helps on push-wall but can degrade simpler tasks.Source: §5.5 · p. 9, Table 4 · p. 9

6 Visualization

The visualizations show that estimated transition probabilities track motion changes and that GAP changes the policy’s training-loss trajectory.

6.1 Visualization of Motion-Transition Phase Estimation

  • Estimated transition probability ρ rises at grasps and movement shifts, while scaled visual uncertainty falls, indicating an inverse relationship.The visualizations use RGB images from threading alongside the estimated probabilities and visual-uncertainty measure.Source: §6.1 · p. 10, Figure 5 · p. 10

6.2 Visualization of Loss Curves

  • GAP slows loss reduction early and midway through training but ultimately converges to a lower loss.The paper attributes the slower initial and middle-stage loss decrease to adjustments of gradients for proprioceptive parameters.Source: §6.2 · p. 10, Figure 6 · p. 10

7 Conclusion and Limitation

The conclusion identifies suppressed vision during motion transitions as a limitation of vision-proprioception policies and presents GAP as a way to support dynamic collaboration.

7 Conclusion and Limitation

  • The paper concludes that GAP enables dynamic collaboration between vision and proprioception to address visual suppression during motion-transition phases.The authors describe the work as offering insights for developing vision-proprioception policies in robotic manipulation.Source: §7 · p. 10
  • All evaluated vision-proprioception policies were trained on a single embodiment; cross-embodiment proprioception remains future work.The authors identify exploring proprioception in cross-embodiment datasets as a promising direction.Source: §7 · p. 10

A.1 Experiment Settings

Experiments span simulated and real-world manipulation, with distinct demonstration sources and robot setups. GAP’s implementation includes safeguards and architecture-specific adjustments for gradient modulation.

A.1 Experiment Settings

  • Simulation uses Meta-World and RoboSuite tasks, with 100 expert demonstrations for Meta-World and 500 synthesized trajectories for RoboSuite.Meta-World has a 4-dimensional action space; RoboSuite adds gripper orientation and involves longer horizons and richer physical interactions.Source: §A.1 · p. 13
  • Real-world evaluation uses a wrist-camera xArm 6 for one-arm tasks and a four-arm platform with wrist and front cameras for dual-arm tasks.Policies are trained on 50 teleoperated demonstrations; dual-arm policies use each arm’s proprioception and corresponding wrist-camera input alongside the front camera.Source: §A.1 · p. 13
  • OOD tests vary target-object positions from the expert-demonstration training distribution while keeping the gripper’s initial position fixed.Source: §A.1 · p. 13

A.2 Details of GAP

  • GAP adjusts gradients during early training and trains bimanual arm policies independently; for Octo-VP, adjustment is applied before the transformer.The adjustment parameter is kept moderate to avoid unstable updates and potential policy collapse.Source: §A.2 · pp. 13–14

B.1 Ablation on Motion-Transition Phase Estimation

The ablations compare phase-estimation approaches and transition models. GAP’s motion-aware phase estimation outperforms the alternatives described, while temporal modeling is robust to noisy phase estimates.

B.1 Ablation on Motion-Transition Phase Estimation

  • GAP generally outperforms human labeling, HDBSCAN, and CoTPC in phase-estimation comparisons, with its motion-consistent distance supporting more accurate transition prediction.The distance incorporates spatial-position, rotation, and gripper-opening changes; the comparison is reported in Table 6.Source: §B.1 · p. 14, Table 6 · p. 14
  • GAP’s phase estimates have 97% average recall and 73% average precision against human labels, and identify motion transitions that humans may overlook.Ground-truth transition labels are infeasible to obtain for the complex and diverse manipulation tasks studied.Source: §B.1 · p. 15

B.2 Ablation on Temporal Network

  • Smoothing can model transitions but does not provide significant performance improvements over the alternatives tested.Table 7 reports the LSTM-component ablation.Source: §B.2 · p. 15, Table 7 · p. 15
  • Policy performance remains robust under considerable random shifts to CPD output indices, which the authors attribute to the LSTM’s modeling of continuous transitions.The noise-injection comparison is reported in Table 8.Source: §B.2 · p. 15, Table 8 · p. 15

B.3 Ablations on Hyperparameters

The hyperparameter studies find GAP’s advantage persists across tested choices, while adjustment strength and duration have bounds. Additional evaluations report stronger visual features with GAP and continued benefits across policy heads.

B.3 Ablations on Hyperparameters

  • When α is small, displacement consistency contributes more and improves success on displacement-dominant assembly and putting the banana in the box; at α = 2, the policy performs better on threading, where rotation matters.Source: §B.3 · p. 16
  • All three tested hyperparameter settings outperform policies without GAP and vision-only policies.Source: §B.3 · p. 16
  • Performance is not highly sensitive to λ between 0.2 and 0.4, but values that are too small can approach or underperform concatenation, while values that are too large destabilize or collapse training.The default is λ = 0.3; Table 10 reports the ablation.Source: §B.3 · p. 16, Table 10 · p. 16
  • Applying GAP early in training improves performance, but suppressing proprioception gradients for too many epochs degrades it.The study applies GAP during the first 50 of 100 training epochs; Table 11 compares durations.Source: §B.3 · p. 17, Table 11 · p. 17

C.1 Linear-Probing Experiments on the Visual Features

  • With frozen visual features used to predict actions, the branch trained with GAP significantly outperforms the branch trained without GAP in both in-distribution and OOD settings.The comparison covers multiple tasks and is reported in Table 12.Source: §C.1 · p. 17, Table 12 · p. 17

C.2 Experiments with Various Policy Heads

  • With UNet-based diffusion policy heads, policies generally outperform those with MLP heads, and GAP policies still outperform vision-only policies and other methods.The comparisons are reported in Table 13.Source: §C.2 · p. 17, Table 13 · p. 18

C.3 More Out-Of-Distribution (OOD) Evaluations

GAP is evaluated under additional visual distribution shifts and online policy updates. It consistently outperforms tested baselines in the reported OOD conditions and can be applied to on-policy learning across tasks and environments.

C.3 More Out-Of-Distribution (OOD) Evaluations

  • GAP consistently outperforms baseline methods under tested table-color, object-color, lighting, and placement shifts.Placement changes have the greatest reported impact, underscoring the importance of target localization; the evaluations appear in Tables 14 and 15.Source: §C.3 · p. 18, Table 14 · p. 18
  • Under visual OOD scenarios, GAP performs better than vision-only approaches, while proprioception-only benefits under visual disturbance lack precise position perception.Source: §C.3 · p. 18

C.4 Experiments of extending GAP to on-policy updates

  • Online-trained policies are more robust, and GAP applies to on-policy updates across tasks and environments without over-dampening necessary proprioception reliance.The online-update experiments are reported in Table 16.Source: §C.4 · p. 19, Table 16 · p. 19

D Task Descriptions

The simulation evaluations span a wide range of manipulation tasks, including pick-and-place, rotation-sensitive, long-horizon, and contact-rich tasks in one-arm and dual-arm setups. Figure 7 visualizes the evaluated simulation tasks.

D Task Descriptions

  • The task suite covers simple pick-and-place, rotation-sensitive, long-horizon, and contact-rich manipulation across one-arm and dual-arm setups.Figure 7 visualizes the simulation tasks.Source: D Task Descriptions · p. 19, Figure 7 · p. 21
  • Pick-and-place tasks require locating objects and targets, then moving, stacking, or placing objects into containers.Examples include placing a cylinder at a target circle, a cube in a drawer, and a red cube on a green cube.Source: D Task Descriptions · pp. 19–20
  • In push-wall, the cylinder is pushed toward a target hidden from visual observation but fixed across evaluations.Source: D Task Descriptions · p. 19
  • Button pressing, assembly, disassembly, and threading tasks test actions ranging from pressing and fitting parts to aligning a pin with a hole.Threading requires both moving and rotating the pin before insertion.Source: D Task Descriptions · pp. 19–20
  • Longer sequences include sweeping with a rag and opening, loading, and closing a drawer around a hammer.The rag task involves taking it from a shelf and rotating it to sweep the table.Source: D Task Descriptions · pp. 19–20
  • Dual-arm tasks include handing a block between arms, opening a bag before loading a thermos, and coordinating lid lifting with pouring.The handover transfers the block from the right arm to the left arm.Source: D Task Descriptions · p. 20

Related papers

Loading 2602.12032v1…