PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Visual Embodiment Dependence in Visuomotor Policies

Hongjie Fang, Yuxuan Lu, Chenxi Wang, Haoxiang Qin, Shirun Tang, Zihao He, Shangning Xia, Jingjing Chen, Wanxi Liu, Shiquan Wang, Cewu Lu

arXiv:2609.16815v1cs.RO

TL;DR

Visuomotor policies can use visible embodiment cues as brittle shortcuts, especially when configuration correlates with task progress. The paper canonicalizes embodiment geometry into an editable, control-relevant CER and uses configuration-decorrelation augmentation. Canonicalization improves human-to-robot transfer, while augmentation improves recovery on unfamiliar configurations without sacrificing seen-configuration performance.

  • Problem

    The paper asks how visuomotor policies should use visible embodiment information when morphology or configuration cues fail to generalize.

  • Method

    The method replaces original embodiment geometry with a canonical end-effector representation in 3D point clouds and edits CER configurations to decorrelate configuration from behavior.

  • Results

    Embodiment canonicalization improves human-to-robot transfer, reaching 96% success on both task stages in the reported ablation, while configuration-decorrelation improves off-trajectory recovery.

  • Takeaways & Limitations

    Robust visuomotor learning benefits from structuring visual embodiment dependence around shared, control-relevant geometry rather than removing embodiment information.

  • Takeaways & Limitations

    The approach assumes the end effector captures the primary control-relevant embodiment information and depends on reliable embodiment masks.

Abstract

from arXiv · show

Visuomotor policies observe both the task scene and the acting embodiment, allowing embodiment-specific visual cues to influence action prediction. We study this phenomenon as visual embodiment dependence (VED) and show, through cue-conflict interventions across representative policies, that visible robot configuration can become a shortcut to task progress. Rather than eliminating VED, we argue that it should be structured around embodiment information that supports control and generalization. We realize this through embodiment canonicalization in 3D point clouds, replacing the original embodiment with a canonical end-effector representation (CER) that preserves control-relevant geometry while abstracting embodiment-specific morphology. Its editable form further enables configuration-decorrelation augmentation for unfamiliar robot configurations. Experiments show that embodiment canonicalization substantially improves human-to-robot policy transfer without robot demonstrations, while simply removing the embodiment is insufficient without preserving control-relevant geometry. We further find that CER itself can become a configuration shortcut when robot configuration becomes decoupled from task progress; configuration-decorrelation augmentation mitigates this failure mode and restores robust recovery without sacrificing performance on seen configurations. Together, these results show that robust visuomotor learning benefits from structuring, rather than removing, visual embodiment information. Project website: https://tonyfang.net/ved

I. INTRODUCTION

Visual embodiment dependence (VED) describes how visible robot appearance, geometry, and configuration influence visuomotor predictions. The paper argues for structuring VED around control-relevant information rather than eliminating embodiment cues.

  • VED arises because visual observations expose both the task scene and the acting embodiment to visuomotor policies.
  • The paper distinguishes morphology dependence, which limits transfer across embodiments, from configuration dependence, which causes errors when configuration and task progress decouple.
  • Configuration dependence occurs when visible robot configuration becomes a proxy for task progress, causing predictions to follow embodiment-implied actions under cue conflict.
  • Embodiment canonicalization replaces embodiment-specific geometry with a canonical end-effector representation that preserves control-relevant geometry and supports human-to-robot transfer.
  • Across human-to-robot transfer and novel-configuration settings, canonicalization and configuration-decorrelation augmentation improve performance and recovery robustness without robot demonstrations.

B. Cross-Embodiment Policy Learning

The paper studies configuration dependence within a fixed embodiment by creating observations that conflict task-state and visible-configuration cues. Its diagnostic controls and progress-based metric quantify whether policies follow embodiment-implied behavior.

  • C. Generalization and Robustness in Visuomotor Policies: Configuration-decorrelation augmentation targets recovery mismatches by diversifying configuration-behavior associations when visible configuration no longer tracks task progress.
  • B. Configuration Dependence Analysis: The evaluation spans simulated Pick Place Can and real-world Rotate Plate, with multiple policies and physically consistent cue-conflict observations.
  • B. Configuration Dependence Analysis: Cue-conflict observations preserve one task stage while substituting another stage’s embodiment configuration to test which visual cue governs action prediction.
  • A. Cue-Conflict Diagnostic: The diagnostic uses absolute actions and removes proprioceptive state, ensuring visible configuration is the policy’s only configuration signal.
  • A. Cue-Conflict Diagnostic: Higher EFR indicates stronger reliance on visible embodiment configuration than task-state evidence under cue conflict.

B. Configuration Dependence Analysis

Configuration dependence is strongly state-dependent: policies follow visible embodiment cues in some task-stage conflicts but not others. This supports treating configuration as a potential shortcut to task progress within a fixed embodiment.

  • Across BC-RNN, DP, ACT, and RISE, configuration-following behavior is pronounced at selected task states rather than uniformly across stages.
  • In Pick Place Can, BC-RNN and DP follow embodiment cues strongly during approach/transport conflicts, while the tendency largely disappears during grasp/pre-grasp conflicts.
  • A similar phase-dependent pattern appears for ACT and RISE in real-world Rotate Plate.
  • These results indicate that visible configuration can act as a shortcut to task progress and bias predictions toward embodiment-implied actions.

IV. METHOD

The method operates in 3D point clouds by isolating the embodiment, replacing it with a canonical end-effector representation, and using editable canonical geometry for augmentation.

  • B. Embodiment Canonicalization: The pipeline removes embodiment points from the scene point cloud before inserting the canonical end-effector representation.
  • B. Embodiment Canonicalization: CER preserves control-relevant end-effector geometry while abstracting embodiment-specific morphology.
  • B. Embodiment Canonicalization: For gripper manipulation, CER uses a canonical gripper whose geometry encodes opening width and orientation from the gripper pose.
  • B. Embodiment Canonicalization: The same CER design can be instantiated from robot gripper poses or human hand landmarks, enabling a shared representation across demonstrations.
  • B. Embodiment Canonicalization: The formulation can extend beyond grippers by replacing the canonical template with geometry for another end effector and its task-space pose mapping.

C. Configuration-Decorrelation Augmentation

Configuration-decorrelation augmentation edits canonical end-effector configurations in eligible 3D observations and pairs them with recoverable trajectories. This targets brittle generalization while preserving control performance on seen configurations.

  • C. Configuration-Decorrelation Augmentation: Configuration-decorrelation augmentation perturbs CER only in recovery-eligible states, where the underlying task remains unchanged and behavior remains recoverable.Free-space approach and pre-grasp states are included; contact-constrained states are excluded when perturbing the gripper would alter the scene.
  • C. Configuration-Decorrelation Augmentation: Perturbed configurations are paired with interpolated recovery trajectories back to the demonstrated pose before continuing along the original demonstration.Valid perturbations must satisfy workspace and collision constraints, and augmentation is applied online with probability paug = 0.5.
  • A. Human-to-Robot Policy Transfer: Human-to-robot transfer experiments use four manipulation tasks and 50 human demonstrations per task, with no robot demonstrations during training.The tasks span pick-and-place, pushing, articulated-object manipulation, and orientation-intensive manipulation, using controlled RISE variants.
  • A. Human-to-Robot Policy Transfer: Embodiment canonicalization achieves the best overall performance across all four transfer tasks, increasing average success from 20% to 91%, versus 31% for Mask Only and 64% for CER Only.The result indicates that transfer benefits from jointly suppressing morphology-specific cues and preserving control-relevant geometry.
  • C. Configuration-Decorrelation Augmentation: Configuration-decorrelation augmentation restores robustness when CER becomes brittle under configuration shifts, without sacrificing performance on seen configurations.The evaluation covers Rotate Plate and Pour Balls, which involve substantial end-effector rotation and visible configuration variation.
  • A. Human-to-Robot Policy Transfer: Masking alone can improve grasping but not sustained or precise control, whereas CER supplies geometry needed for constrained opening and accurate pouring.On Open Drawer, Mask Only raises grasping from 44% to 96% but opening only from 32% to 36%; Pour Balls has little tolerance for grasp-position or orientation errors.

B. Generalization to Novel Robot Configurations

The study evaluates generalization to robot configurations decoupled from task progress and shows that configuration-decorrelation augmentation improves recovery, while canonicalization preserves control-relevant geometry.

  • Configuration-decorrelation augmentation consistently improves robustness to unfamiliar robot configurations across Rotate Plate and Pour Balls.It exposes policies to off-trajectory CER configurations and corresponding recovery behaviors, improving pouring completion while preserving seen-configuration performance.
  • Full CER achieves 96% success on both Collect Block task stages, while masking alone or adding CER alone is insufficient.Richer end-effector geometries outperform coarse position- or pose-based representations.
  • The full CER matches an oracle representation containing complete robot morphology, indicating that compact canonical geometry preserves control-relevant information.The oracle is constructed from replayed robot trajectories with full robot geometry.
  • Removing deployment-time masking reduces Collect Block success from 96% to 92% while restoring inference latency near the vanilla level.When robot URDF and camera calibration are available, rendered masking maintains 96% success with 9 ms additional latency.
  • Training-time masking reduces sensitivity to original morphology while preserving substantial dependence on CER geometry.Controlled interventions remove either embodiment representation while keeping the task scene fixed.

VI. CONCLUSION

The conclusion argues that visual embodiment dependence should be structured around control-relevant information rather than removed. It identifies canonicalization and configuration-decorrelation as mechanisms for transfer and robust configuration generalization, within stated scope limits.

  • Embodiment canonicalization and configuration-decorrelation augmentation structure visual embodiment dependence to improve human-to-robot transfer and unfamiliar-configuration robustness.
  • The work assumes that the end effector captures the primary control-relevant embodiment information and depends on reliable masks.
  • Future work includes extending CER to dexterous hands and other embodiments, using more diverse human data, and supporting 2D or other modalities.

APPENDIX I IMPLEMENTATION DETAILS

The implementation uses a common RISE-based point-cloud pipeline with fixed preprocessing and training settings. CER is instantiated as a lightweight, pose- and opening-width-dependent pseudo-gripper appended to the scene point cloud.

  • All policy variants use RISE with the same architecture, objective, optimization schedule, and preprocessing, differing only in embodiment processing and augmentation.
  • RGB-D observations become colored 3D point clouds, voxelized at 5 mm and cropped to a fixed camera-frame workspace.Policies train for 1000 epochs with AdamW at a 3 × 10^-4 learning rate and predict 20-step action chunks from single-step observations.
  • SAM2 supplies embodiment masks, which are dilated with a kernel size of 10 before corresponding depth points are removed.When robot URDF and camera calibration are available, deployment masks can be rendered directly from robot state.
  • CER is a lightweight 3D pseudo-gripper composed of two finger modules and one proximal gripper-body module.Each finger is a 2 cm × 1 cm × 6 cm rectangular volume, while the proximal module spans the gripper range with 1 cm thickness.
  • CER preserves gripper opening width by symmetrically translating finger modules and instantiates the template at end-effector pose T_t as C(T_t, w_t) = T_tC_0(w_t).The resulting points receive uniform black RGB features and are appended to the task-scene point cloud.

D. Human Demonstration Processing

Human demonstrations are collected without robot demonstrations, retargeted into robot end-effector space, and processed consistently across methods. Configuration augmentation perturbs only recovery-eligible states and reconnects generated recoveries to the demonstrations.

  • The study collects 50 human demonstrations per task and uses no robot demonstrations for human-to-robot transfer training.Human hand motion is retargeted to robot end-effector space with temporal filtering and orientation regularization.
  • Configuration-decorrelation augmentation is restricted to recovery-eligible states where end-effector perturbations leave the task state unchanged.Free-moving and pre-grasp states are included, while contact-constrained states are excluded.
  • Eligible samples receive bounded rigid-body perturbations in translation and rotations within 60°.The perturbed pose is denoted eT_t = ΔT_tT_t.
  • Augmented observations replace the original CER with the perturbed CER while keeping the task scene unchanged, then generate collision- and workspace-constrained recoveries back to T_t.Translation uses linear interpolation and orientation uses SLERP, with velocity limits of 0.02 m/step and 0.15 rad/step.
  • After the gripper reaches T_t, the recovery trajectory reconnects to the original demonstration, with augmentation sampled online at p_aug = 0.5.

APPENDIX III ADDITIONAL DETAILS OF THE VED DIAGNOSTIC

The diagnostic constructs physically consistent cue-conflict observations and evaluates action discrepancies with an end-effector distance that combines translation and rotation. A full-robot oracle representation matches the compact CER on both reported stages.

  • Cue-Conflict Construction: Cue-conflict observations pair task state S_i with embodiment configuration E_j, using simulation re-rendering or physically reproduced real-world scenes.Each selected stage pair contributes N = 20 observations, and the same observations are evaluated by all policies.
  • Action Distance: The action distance combines Euclidean translation error with geodesic rotation error, scaling rotation by λ_R = 0.06 m/rad to approximate gripper-length displacement.The gripper command is excluded from the diagnostic distance.
  • Full-robot oracle: 96% success on both Pick and Place stages matches the compact CER design, despite the oracle retaining complete robot morphology.The oracle replaces CER with captured full-robot geometry while keeping the task scene and action-processing pipeline unchanged.

APPENDIX V REPRESENTATION SENSITIVITY ANALYSIS

The representation-sensitivity analysis compares controlled variants on identical held-out observations using cosine distances from RISE's final Transformer readout. Training-time masking reduces sensitivity to original morphology while preserving substantial sensitivity to CER.

  • Evaluation Setup: The analysis evaluates 42 held-out robot observations from Collect Block and Pour Balls without retraining or result-based frame selection.The observations include 17 frames from three Collect Block rollouts and 25 frames from three Pour Balls rollouts.
  • Controlled Variants: Four variants—Full, Mask, Full+CER, and Mask+CER—share the same RGB-D frame, task scene, robot state, and CER pose before point-cloud preprocessing.This controls the visual and task inputs while isolating representation changes.
  • Measurement: Representation changes are measured as cosine distances from the final Transformer readout in RISE immediately before the diffusion action decoder.The study defines sensitivities to original morphology and CER using these controlled interventions.
  • Representation Sensitivity Analysis: Training-time masking reduces morphology sensitivity by approximately 32% overall, with a paired difference of −0.00129 and a 95% CI of [−0.00178, −0.00082].The same trend appears on both tasks, while sensitivity to CER remains substantial.
Loading 2609.16815v1…