PaperlayerView exact revision on arXiv

Source-linked AI summary

Enabling a Unified Cross-Domain Representation for Two-Finger Gripper Manipulation via Interaction-Centric Modeling

Guanlin Li, Shifeng Bao, Yihan Zhao, Haitao Shen, Haoyang Li, Chen Zhao, Tong Yang, Jie Tang, Jing Zhang

arXiv:2609.31207v1cs.ROcs.CV

TL;DR

Cross-embodiment imitation learning is challenged by visual representations that entangle task semantics with hardware-specific geometry. This paper uses a parameterized gripper abstraction and interaction-centric hybrid representation; it reports competitive simulation performance and zero-shot cross-embodiment and cross-viewpoint transfer. The authors also note deployment constraints including manual camera recalibration and strict workspace matching.

  • Problem

    The paper identifies embodiment entanglement in visual observations as a hypothesized bottleneck to cross-embodiment imitation learning.

    Source: §I · p. 1

  • Method

    A parameterized two-finger gripper abstraction grounds an interaction triplet and hybrid global APF and local point-cloud features for action prediction.

    Source: §I · p. 1, §IV · p. 3

  • Results

    With 50 demonstrations and 30M parameters, the model matches in-domain simulation performance and reports zero-shot transfer across embodiments and viewpoints under compounded perturbations.

    Source: §I · pp. 1–2

  • Takeaways & Limitations

    The authors conclude that their interaction-centric framework yields an in-domain imitation-learning system with OOD generalization, including zero-shot sim-to-real transfer to unseen heterogeneous robots.

    Source: §VI · p. 7

    Deployment requires manual camera recalibration for viewpoint changes and strict workspace matching across embodiments, with occasional inverse-kinematics failures.

    Source: §VI · p. 7

Abstract

from arXiv · show

Achieving robust cross-embodiment generalization in imitation learning demands overcoming a critical representation flaw that inextricably entangles task semantics with hardware-specific visual geometry. We propose an interaction-centric framework that leverages the shared structure of two-finger grippers via a parameterized universal gripper abstraction, yielding a canonical gripper-frame representation. Given language and RGB-D observations, a VLM infers the subtask and grounds an interaction triplet (gripper, held, target), while SAM~2.1 tracks masks to reduce VLM queries. We design concise hybrid features that combine target/collision artificial potential fields for global guidance with segmented gripper-frame point clouds for local geometry, and use a Flow-Matching Transformer to predict smooth 7-DoF action chunks. Experiments in simulation and real-world tasks demonstrate that ours is the first imitation learning approach to simultaneously achieve competitive benchmark scores and extreme cross-embodiment/cross-viewpoint zero-shot sim-to-real transfer to completely distinct, heterogeneous robot platforms.

I. INTRODUCTION

The introduction frames embodiment entanglement in visual observations as a bottleneck for cross-embodiment imitation learning. It proposes an interaction-centric, canonical gripper representation and a hybrid policy, reporting competitive simulation performance and zero-shot transfer across embodiments and viewpoints.

I. INTRODUCTION

  • The authors identify embodiment entanglement in visual observations as a bottleneck to cross-embodiment transfer and call for a structured bridge between perception and motor execution.They frame this as a hypothesis about the embodiment representation itself.Source: §I · p. 1
  • The framework decouples direct observation-to-action learning by projecting heterogeneous observations into a shared representation that joins language and perception around interaction.The resulting interface is described as embodiment-agnostic and feeds action generation.Source: §I · p. 1
  • A parameterized, appearance-agnostic two-finger gripper abstraction uses intrinsic dimensions to project different robots into a canonical space.The interaction triplet comprises the active gripper, any held object, and the next target; a keypoint envelope captures fingertip contacts and collision boundaries.Source: §I · p. 1
  • The pipeline combines VLM instruction parsing and triplet tracking with global APF guidance, local gripper-frame point clouds, and a Flow-Matching Transformer for 7-DoF trajectories.The interaction geometry is illustrated in the real-world Jaka setting.Source: §I · p. 1, Figure 1 · p. 2
  • The introduction presents interaction-centric representations as a basis for cross-embodiment imitation-learning training and pre-training.Source: §I · p. 1
  • Using only 50 demonstrations and 30M parameters, the model matches in-domain simulation performance while reporting zero-shot transfer across embodiments and viewpoints under compounded perturbations.The reported transfer spans sim-to-real, dual-arm to single-arm, camera shifts, and object variations.Source: §I · pp. 1–2

II. RELATED WORK

The related-work section contrasts target-focused object-centric control with interaction-centric representations. It situates the paper among keypoint approaches while distinguishing its gripper geometry and interaction-triplet formulation.

A. Object-Centric Representation Learning

  • Object-centric approaches use visual detection to focus control on target objects, whereas the paper locates its contribution at the representation level of interaction-centric methods.It distinguishes this from UMI, which captures interactions at the physical interface level.Source: A. Object-Centric Representation… · p. 2

B. Keypoint-Based Representation Learning

  • Prior work combines semantic object and end-effector keypoints for cross-embodied imitation learning; this paper instead decouples gripper keypoints from direct 3D object representations.The authors say this avoids cumbersome object annotations while retaining data efficiency and cross-embodied generalization.Source: B. Keypoint-Based Representation… · p. 2

III. PRELIMINARIES

The preliminaries specify language-conditioned manipulation with calibrated RGB-D observations and a policy that predicts action chunks. The method overview frames tasks through a gripper-held-target interaction triplet and compact gripper-frame context.

III. PRELIMINARIES

  • The setup considers language-conditioned manipulation with one or two robot arms, each equipped with a two-finger gripper, using fixed calibrated RGB-D observations.The gripper morphology is represented by length, width, and height; its proprioceptive state is the 7-DoF end-effector pose.Source: §III · p. 2
  • The policy maps each observation to a horizon-H action chunk executed at a finer control rate.Each 7-DoF action is an end-effector pose increment plus a gripper open/close command.Source: §III · p. 2

IV. METHOD

  • The method represents each timestep with the active gripper, any held object, and the target, plus compressed environment context in the gripper frame.The overview depicts the pipeline ending in smooth horizon-H 7-DoF action trajectories.Source: §IV · p. 3, Figure 2 · p. 3

A. Perception Module

The perception module parses instructions and localizes the interaction triplet, using SAM 2.1 to segment and track masks. The representation module maps observations into the gripper frame and combines global spatial priors with local interaction geometry.

A. Perception Module

  • At refresh steps, a VLM parses the instruction and localizes triplet regions; SAM 2.1 refines bounding boxes into masks and tracks them between refreshes.The resulting masks are forwarded to the representation module.Source: A. Perception Module · p. 3

B. Representation Module

  • The representation module transforms RGB-D point clouds into the gripper frame and combines APFs for global spatial priors with local point-cloud features for triplet geometry.The embodiment is parameterized by gripper length, width, and height.Source: B. Representation Module · p. 3

1) APF representation for global environment:

The APF representation gives the gripper coarse global spatial guidance using separate collision and target fields. Eight gripper-frame keypoints parameterized by gripper dimensions encode these fields for the policy.

1) APF representation for global environment:

  • Separate collision and target APFs provide coarse workspace guidance, with repulsive effects for collisions and attractive effects for targets.The formulation is training-free and inherits VLM priors for OOD generalization.Source: 1) APF representation for global… · p. 3
  • The collision field uses robot-body and held-object voxels, while the target field distinguishes target from non-target voxels.These voxel sets are specified as inputs to their respective APF weightings.Source: 1) APF representation for global… · p. 3
  • Eight gripper-frame keypoints, parameterized by length, width, and height, define the locations used to compute APF vectors.The keypoints include base corners and fingertips.Source: 1) APF representation for global… · p. 4
  • Each keypoint’s APF vector is encoded as a unit direction and log-scaled magnitude, producing a (16, 4) tensor across eight keypoints and two fields.The cited passages also state that weight normalization yields zero total weight, stabilizing policy learning.Source: 1) APF representation for global… · p. 4

2) PCD representation for local geometry:

The local-geometry representation complements global APF context with segmented point clouds for contact-rich manipulation. A two-stage sampling procedure reduces each cloud to 256 points for policy learning.

2) PCD representation for local geometry:

C. Action Module

The Action Module conditions a Flow-Matching Transformer on subtask text and per-effector perception zones to predict horizon-H 7-DoF action chunks. Its per-effector formulation supports deployment across dual-arm and single-arm robots.

C. Action Module

  • A Flow-Matching Transformer predicts horizon-H 7-DoF action chunks from global subtask text and embodiment-specific per-effector perception zones.Each zone combines the interaction-centric representation and proprioceptive state.Source: C. Action Module · p. 4
  • For single-arm deployment, the inactive arm’s perception zone is fixed at its initial state while the active arm’s zone follows the current gripper.In dual-arm settings, the model conditions on global text and two independent gripper zones.Source: C. Action Module · p. 4
  • Shared encoders process the held and target point clouds, gripper APF, and world-frame pose before modality tokens and noisy action tokens enter an eight-layer causal Transformer denoiser.The denoiser has 512 hidden dimensions and about 30M parameters.Source: C. Action Module · p. 4
  • Simulation and real-world teleoperation demonstrations pair processed perception and representation outputs with demonstrated actions to supervise the flow-matching policy.Source: C. Action Module · p. 4

D. Deployment Efficiency

Across 25 RoboTwin2.0 tasks, the method achieves the highest reported Randomized success rate, while a single VLM call triggers a five-second action sequence. The evaluation also describes task-selection constraints and real-world deployment settings.

D. Deployment Efficiency

  • One VLM call triggers five SAM calls and 100 actions over five seconds at 20 Hz, amortizing VLM latency for real-time deployment.The speed assessment assumes 200–500 ms for a locally deployed open-source VLM and 10 ms for non-VLM components.Source: D. Deployment Efficiency · pp. 4–5
  • Real-world evaluation covers four tasks under in-domain and Position, Object, and Camera OOD settings, plus direct simulation-trained deployment on a heterogeneous physical robot.Source: Figure 4 · p. 5

A. Simulation Experiments

  • 46.1% Randomized success across 25 RoboTwin2.0 tasks is highest among methods, while DP3-W falls from 73.1% Clean to 7.5% Randomized.The method’s Clean score is 66.3%, on par with UP-VLA at 66.5%.Source: Table I · p. 5, A. Simulation Experiments · pp. 5–6
  • The evaluation benchmarks against small models trained from scratch and large models fine-tuned from pretrained checkpoints, using 50 Clean demonstrations and 100 Clean and Randomized test trajectories per task.Randomized settings induce OOD shifts through scene randomization.Source: A. Simulation Experiments · p. 5
  • The benchmark excludes tasks with visually ambiguous progress, insufficient viewpoint coverage, or low-contrast objects that cause VLM/SAM parsing failures.These exclusions reflect reliance on VLM priors and a single fixed-view RGB-D stream.Source: A. Simulation Experiments · p. 5
  • Fig. 5 shows six selected RoboTwin2.0 tasks across DOS-W1, Jaka, and simulation rows; Table II distinguishes simulated success rates from physical-robot success counts.Table II also distinguishes near-view from substantially different novel-view perspectives.Source: Figure 5 · p. 6, Table II · p. 6

B. Real-World Experiments

Real-world tests probe position, object, and camera shifts, plus zero-shot transfer from a simulated dual-arm robot to heterogeneous physical platforms. The method retains task success under several shifts, but severe viewpoint changes remain challenging.

Embodiment OOD

  • One policy trained on simulated ALOHA-AgileX transfers zero-shot to Jaka and DOS-W1 across sim-to-real, arm-count, viewpoint, and object shifts.The separate real-world OOD evaluation uses four tasks and 50 demonstrations per task, with position, object, and camera shifts.Source: B. Real-World Experiments · p. 6

Real-world OOD robustness

  • Under Position/Object OOD, ours beats both baselines on every task; under Camera OOD, it retains success on three tasks while both baselines score 0/10.Pouring remains at 0/10 for all approaches under Camera OOD.Source: B. Real-World Experiments · p. 6

Embodiment OOD

  • Despite >90% simulated success on two dynamic tasks, physical deployment retains ∼50% of simulation performance on dynamic tasks for DOS-W1 and near-view Jaka.The simulated results are reported for Shake Bottle and Shake Horiz.Source: B. Real-World Experiments · p. 6
  • Novel-view Jaka Shake Bottle success falls to 9/30, while precise Place Phone remains challenging at 6/30 and 5/30.DOS-W1 generally achieves slightly higher success than Jaka, consistent with its closer workspace proximity to the original robot.Source: B. Real-World Experiments · p. 7

C. Ablation Study

Ablations show complementary roles for local point-cloud geometry, global potential fields, and causal masking, while subtask-level semantic grounding is associated with stronger randomized robustness.

Ablation I: Component Validation

  • Removing PCD drops Clean success from 66.3% to 54.9%; removing APF drops Randomized success from 46.1% to 35.1%.The study reports average Clean and Randomized success rates on 25 tasks.Source: C. Ablation Study · p. 7
  • Removing CM lowers overall success to 54.2% and 36.8% and introduces action-chunk jitter; component failures vary by task type.CM removal disproportionately affects multi-step sequences, missing PCD affects fine-geometry tasks, and missing APF affects cluttered scenes; mismatched dimensions impair precise alignment.Source: C. Ablation Study · p. 7

Ablation II: Generalization Source

  • Randomized success rises from 4.2% with raw point clouds to 6.6% with generic segmentation, then 24.0% and 35.1% with instruction- and subtask-level segmentation.Clean success is 39.2% for raw point clouds and 54.4% for generic segmentation.Source: C. Ablation Study · p. 7

VI. CONCLUSION, LIMITATIONS, AND FUTURE WORK

The paper presents an interaction-centric representation for two-finger manipulation and reports zero-shot transfer to unseen heterogeneous robots. It also identifies perception and deployment constraints and proposes directions for future work.

Conclusion

  • The framework uses an interaction triplet to decouple embodiment from visual states, supporting in-domain imitation learning and zero-shot sim-to-real transfer to unseen heterogeneous robots.The reported transfer includes object and viewpoint variations.Source: §VI · p. 7

Limitations

  • Limitations include single-view RGB input, VLM/SAM localization biases, manual camera recalibration, strict workspace matching, and occasional IK failures.The authors state that strict workspace matching constrains feasible tasks.Source: §VI · p. 7

Future Work

  • Future work targets multi-view perception, more robust foundation models, and learning interaction-centric semantics directly from raw world states.The authors identify interaction-centric semantics as intermediate representations for VLA prediction.Source: §VI · p. 7

Related papers

Loading 2609.31207v1…