PaperlayerView exact revision on arXiv

Source-linked AI summary

Proprioception-Anchored Cross-Modal Pretraining for Zero-Shot Sim-to-Real Contact-Rich Assembly

Yuhan Wang, Yurou Chen, Hongye Jiang, Gaojing Zhang, Wenzhao Lian

arXiv:2609.07534v3cs.ROcs.AI

TL;DR

Contact-rich assembly requires submillimeter alignment and reliable force interpretation, but simulation-to-hardware discrepancies challenge policy transfer. PACE anchors temporal visual and F/T representations to proprioceptive transitions and deploys policies trained on frozen features. Across four tasks, it reports 93.3% average hardware success with a 2.7-percentage-point average sim-to-real decrease.

  • Problem

    Contact-rich assembly requires submillimeter spatial accuracy and force interpretation, while visual, contact-dynamics, and F/T discrepancies can limit simulation-trained policy transfer.

    Source: §I · pp. 1–2

  • Method

    PACE pretrains temporal visual and F/T representations to predict proprioceptive transitions, then freezes the encoder for RL policy training and direct hardware deployment.

    Source: §I · p. 2

  • Results

    Across four assembly tasks, PACE achieves 93.3% average real-world success and a 2.7-percentage-point average sim-to-real decrease, with robustness to visual and F/T perturbations.

    Source: §I · p. 2

  • Takeaways & Limitations

    The results support using a physically grounded modality to anchor more domain-sensitive representations rather than attempting to eliminate every domain gap.

    Source: §VIII · p. 8

    Evaluation covers one robot platform and four quasi-static tasks, with one checkpoint per method–task pair and no transfer-variability assessment across training seeds.

    Source: §VII · p. 7

Abstract

from arXiv · show

Contact-rich assembly remains challenging because it requires submillimeter spatial accuracy and reliable interpretation of forces during sustained contact. Although simulation-based reinforcement learning offers a scalable training paradigm, discrepancies in visual observations, contact dynamics, and force/torque (F/T) measurements often limit policy transfer. We observe that proprioception is comparatively consistent across domains because joint positions are expressed in a shared calibrated coordinate system and joint velocities are computed consistently in simulation and on hardware. Based on this observation, we present PACE (Proprioception-Anchored Cross-Modal Encoder), which supervises temporal visual and F/T representations by predicting proprioceptive state transitions. Static domain-specific factors, including lighting, texture, and sensor bias, contain little information about joint motion; optimizing the proposed objective therefore suppresses their influence on the learned representation while retaining task-relevant motion cues. Policies trained on frozen PACE features are directly deployed on hardware without real-world fine-tuning or object-pose tracking. Across four contact-rich assembly tasks, PACE attains an average real-world success rate of 93.3\% and only a 2.7-percentage-point sim-to-real drop, while remaining robust to perturbations that substantially degrade pose-based and learned-fusion baselines.

I. INTRODUCTION

Contact-rich assembly demands submillimeter alignment and force interpretation, while simulation-to-hardware transfer is challenged by differences in observations, sensing, and contact dynamics. PACE uses proprioceptive transitions to anchor visual and force representations, then evaluates frozen features on hardware.

I. INTRODUCTION

  • Submillimeter clearances make small misalignments consequential, while deployable pose estimates can be brittle and domain adaptation, randomization, or system identification face practical challenges.Lighting variation or calibration drift can produce millimeter-scale errors that exceed assembly tolerances.Source: §I · pp. 1–2
  • PACE uses comparatively consistent proprioceptive state transitions to supervise temporal visual and F/T features, preserving physical motion cues while suppressing domain-specific nuisance factors.The approach treats modality transferability asymmetrically: proprioception anchors representations from more domain-sensitive signals.Source: §I · p. 2
  • PACE combines learnable zero-sum temporal differencing, bidirectional proprioceptive-transition prediction, and auxiliary objectives, then freezes the encoder for policy training and hardware deployment without adaptation.The auxiliary objectives preserve task-relevant spatial and contact information.Source: §I · p. 2
  • 93.3% average real-world success and a 2.7-percentage-point average sim-to-real decrease across four tasks; pose-based and learned-fusion baselines show substantially larger hardware decreases.Controlled ablations identify proprioceptive anchoring as the main contributor to transfer, and perturbation tests show robustness to visual and F/T shifts.Source: §I · p. 2

II. RELATED WORK

Prior assembly and transfer methods use varied sensing, data, and calibration strategies, while representation-learning work offers reusable or dynamics-aware alternatives. PACE instead grounds visual and F/T motion representations in proprioceptive transitions.

A. Contact-Rich Robotic Assembly

  • Classical compliance and force–position methods can work in structured settings but often require task- and geometry-specific design; learning systems variously rely on pose estimates, force feedback, or real-robot data.PACE learns deployable visual and F/T representations from simulated data for direct transfer without an external pose estimator or object tracker.Source: A. Contact-Rich Robotic Assembly · p. 2

B. Sim-to-Real Transfer

  • Domain randomization, adaptation, and system identification address transfer through broader training distributions, real-data feature alignment, or dynamics calibration, but residual errors and task-specific demands remain problematic at tight tolerances.Asymmetric policy learning separates privileged training information from deployable observations, but does not itself ensure cross-domain consistency; PACE uses simulation-only proprioceptive-transition supervision instead.Source: B. Sim-to-Real Transfer · pp. 2–3

C. Representation Learning for Robotic Manipulation

  • General-purpose visual encoders and multimodal fusion do not by themselves target submillimeter spatial reasoning or cross-domain consistency; dynamics-aware and proprioception-based methods provide related representation-learning approaches.PACE uses zero-sum temporal kernels, spatial-alignment and engagement objectives, and masked reconstruction to retain transferable motion and contact information.Source: C. Representation Learning for Robotic… · p. 3

III. PROBLEM FORMULATION

The problem is quasi-static contact-rich assembly under tight positional tolerances, with policies acting from deployable observations. The formulation distinguishes visual, F/T, and physical interaction gaps while treating proprioception as an approximately aligned cross-domain reference.

III. PROBLEM FORMULATION

  • The robot must assemble a grasped object into a target receptacle within a task-specific positional tolerance of 0.1–1.0 mm.Observations include two wrist-camera images, six-axis F/T measurements, joint positions and velocities, and the previous action; the policy outputs end-effector pose increments under impedance control.Source: §III · p. 3
  • The formulation identifies visual observation, F/T sensing, and physical interaction as three sources of sim-to-real discrepancy.Domain randomization is used during large-scale RL training to mitigate the physical interaction gap.Source: §III · p. 3
  • Proprioception is represented as joint positions and velocities in a shared coordinate system, with consistent velocity computation across simulation and hardware despite residual noise, latency, and quantization.This shared reference motivates its use as a cross-domain anchor.Source: §III · p. 3
  • The policy is trained in simulation on encoded features and proprioception, then deployed directly on hardware without real-world policy adaptation.The paper defines this direct deployment as zero-shot sim-to-real transfer.Source: §III · p. 3

IV. METHOD

PACE first pretrains an encoder to predict domain-consistent physical quantities, then freezes it for RL policy training and direct hardware deployment. The expert demonstrations and encoder architecture support this two-stage pipeline.

IV. METHOD

  • The method has two stages: pretrain an encoder to predict domain-consistent physical quantities, then freeze it while training an RL policy for direct hardware deployment.The pipeline includes expert data collection, encoder pretraining, policy learning, and deployment.Source: §IV · p. 3

A. Expert Demonstrations

  • For each task, a privileged-state PPO expert is trained in Isaac Lab with Factory domain randomization, and 300 stochastic-expert trajectories are collected.Recorded demonstrations are partitioned into training and validation sets using an 80:20 split.Source: A. Expert Demonstrations · pp. 3–4
  • The architecture uses a frozen visual backbone with attention pooling and an F/T MLP, then applies stream-specific temporal kernels to preceding features to form forward motion codes.Each stream combines its current feature, motion code, and proprioception to produce a latent representation.Source: Figure 2 · p. 4, A. Expert Demonstrations · p. 4

B. Encoder Architecture

PACE uses learnable temporal differencing to extract motion cues from visual and force streams while suppressing static offsets. The visual stream captures object motion, gripper approach, and contact deflection; the force stream retains contact transients.

B. Encoder Architecture

  • A per-dimension temporal kernel is constrained to sum to zero, removing constant offsets such as static appearance and sensor bias while retaining temporal changes.The kernel is initialized to approximate a first-order difference.Source: B. Encoder Architecture · p. 4
  • The visual stream uses frozen image features and temporal differences to represent object motion, gripper approach, and contact-induced deflection.Learned attention pooling and cross-view fusion combine features from both wrist cameras.Source: B. Encoder Architecture · p. 4
  • The force stream encodes a recent window of six-axis F/T measurements, then applies a separate temporal kernel to suppress static bias while retaining contact transients.Its output is fused with proprioception.Source: B. Encoder Architecture · p. 4

C. Proprioception-Anchored Cross-Modal Pretraining

PACE pretrains visual and force representations by predicting proprioceptive transitions, with cycle consistency and auxiliary objectives for alignment, engagement, and cross-modal reconstruction. The combined objectives aim to retain motion-relevant information while suppressing domain-specific appearance and static sensor offsets.

C. Proprioception-Anchored Cross-Modal Pretraining

  • Predicting the proprioceptive innovation rather than the absolute state prevents copying the earlier state and leaves motion codes constrained by the prediction task.A lightweight MLP predicts the change from the state K steps earlier and adds it to that earlier state.Source: C. Proprioception-Anchored Cross-Modal… · p. 4
  • Forward and backward proprioceptive-transition predictions impose cycle consistency on learned motion codes using a shared prediction head.The backward branch recomputes codes from negated temporal differences, and gradients through the forward prediction are stopped before the backward prediction.Source: C. Proprioception-Anchored Cross-Modal… · p. 4
  • Because appearance and static sensor offsets provide little information about proprioceptive transitions, minimizing the anchoring loss makes temporal representations encode associated physical motion.The visual backbone remains frozen throughout pretraining.Source: C. Proprioception-Anchored Cross-Modal… · p. 4
  • Spatial-alignment and engagement heads use simulation-available state labels to supervise target direction and, for JointAssemble, yaw error, plus held-part engagement.The alignment loss uses cosine loss for planar direction and smooth-L1 loss for yaw when applicable; engagement uses binary cross-entropy.Source: C. Proprioception-Anchored Cross-Modal… · pp. 4–5
  • Masked feature-space reconstruction masks half of the concatenated visual and proprioceptive dimensions, coupling the pathways through a shared multimodal context.The reconstruction target is detached, so gradients update the reconstructor and visual and force pathways but not the target; pretraining combines anchoring, alignment, engagement, and reconstruction losses.Source: C. Proprioception-Anchored Cross-Modal… · p. 5

D. Policy Training and Deployment

The pretrained encoder is frozen and used to provide observations for reinforcement-learning policy training. The frozen encoder and policy are transferred to hardware without parameter updates or real-world training data.

D. Policy Training and Deployment

  • Policies train for 200 Isaac Lab iterations with a frozen encoder, using latent features, proprioception, and previous action for the actor and privileged simulation state for the critic.All policies share the same architecture, hyperparameters, and domain-randomization settings to isolate the observation representation.Source: D. Policy Training and Deployment · p. 5
  • Hardware deployment transfers the frozen encoder and policy without real-world training data or parameter updates, while processing camera and F/T inputs as in simulation.The policy issues incremental end-effector poses through Cartesian impedance control; shared one-time hardware configuration includes camera settings, frame offset, impedance gains, and F/T zeroing.Source: D. Policy Training and Deployment · p. 5

V. EXPERIMENTAL SETUP

The evaluation covers four assembly tasks with different geometric and contact-sensing challenges, using a shared simulation and hardware protocol. Three baselines represent privileged-state control, unadapted sensor-feature learning, and learned multimodal fusion.

A. Tasks and Evaluation

  • Four tasks span fine visual alignment, contact-sensitive gear meshing, translational and yaw alignment, and F/T-sensitive engagement.PegInsert uses a 16-mm peg with approximately 0.1-mm radial clearance; GearMesh uses a 30-mm-diameter gear, while JointAssemble and KnobTighten add yaw alignment and sloped-socket contact demands, respectively.Source: A. Tasks and Evaluation · p. 5
  • Experiments use a Franka FR3 with two wrist RGB cameras and a six-axis wrist F/T sensor; the 15-Hz policy controls translation and yaw through 4 controllable DoF.Roll and pitch are fixed, and success requires reaching the task-specific target-pose tolerance.Source: A. Tasks and Evaluation · p. 5
  • Each method is evaluated across three random seeds and 100 simulated episodes per seed, then the best simulated checkpoint per method-task pair is deployed for 30 hardware trials.Initial gripper and target poses are randomized before each hardware trial, and success is judged from final pose and contact conditions.Source: A. Tasks and Evaluation · p. 5

B. Baselines

  • The baselines compare privileged-state control with external pose estimates, PPO from unadapted sensor features, and behavior-cloning-initialized cross-attention fusion.AugInsert and PACE use the same expert demonstrations for behavior-cloning initialization and encoder pretraining, respectively, and share the PPO budget and hardware protocol.Source: B. Baselines · pp. 5–6

C. Ablations

The ablations examine proprioceptive anchoring, force and visual streams, and masked reconstruction. A cross-domain feature comparison also reports higher PACE similarity than unadapted representations, particularly for F/T signals during contact.

C. Ablations

  • The ablation protocol retains full-model loss weights and training settings unless stated otherwise; removing vision also omits Lalign.Source: C. Ablations · p. 6
  • The anchoring ablation sets Lanchor to zero while retaining spatial alignment, engagement, and reconstruction objectives, isolating transition prediction’s contribution to transfer.Source: C. Ablations · p. 6
  • Removing the force stream tests its contribution to meshing, rotational-mismatch, and sloped-geometry engagement tasks.The force pathway is removed from both encoder pretraining and policy input.Source: C. Ablations · p. 6
  • PACE features show higher cosine similarity than unadapted representations along a matched simulation–hardware trajectory, particularly for F/T during contact.Source: Figure 4 · p. 6
  • Removing the visual stream tests whether force and proprioception alone provide enough spatial information for initial alignment.The visual stream is removed from both encoder pretraining and policy input.Source: C. Ablations · p. 6
  • The reconstruction ablation removes its loss while retaining proprioceptive anchoring and privileged auxiliary supervision, to test nominal transfer and perturbation robustness.Source: C. Ablations · p. 6

VI. RESULTS AND ANALYSIS

PACE transfers strongly across the four assembly tasks and retains performance under controlled GearMesh perturbations. The results also examine simulation performance, cross-domain feature similarity, component ablations, and robustness to real-world shifts.

A. Zero-Shot Sim-to-Real Transfer

  • PACE and Pose-Based PPO have similar average simulated success rates of 96.0% and 96.4%, respectively; vision-free variants reach only 5–9%.The passage reports that replacing privileged poses with estimates produces a much larger hardware decrease for Pose-Based PPO.Source: A. Zero-Shot Sim-to-Real Transfer · p. 6
  • 93.3% average hardware success and a 2.7-pp average transfer decrease make PACE the strongest Table I result, ahead of Pose-Based PPO and AugInsert.Pose-Based PPO reaches 77.5% average hardware success with an 18.9-pp decrease; AugInsert reaches 68.3% with an 18.4-pp decrease.Source: A. Zero-Shot Sim-to-Real Transfer · p. 6, Table I · p. 7
  • Across 20 matched trajectory pairs, PACE visual and F/T features have mean cosine similarities of 0.91 ± 0.08 and 0.64 ± 0.23, respectively.The comparison averages feature similarity over contact phases; it is a diagnostic, while Table I supplies task-level transfer evidence.Source: A. Zero-Shot Sim-to-Real Transfer · pp. 6–7
  • Without proprioceptive anchoring, the average transfer decrease rises from 2.7 to 25.5 pp while simulated success falls by 4.7 pp.Removing force lowers average simulated success by 11.3 pp, with the largest task-level decreases on GearMesh, JointAssemble, and KnobTighten.Source: A. Zero-Shot Sim-to-Real Transfer · p. 7
  • Removing masked reconstruction raises the average transfer decrease from 2.7 to 6.1 pp and reduces simulated success by 4.1 pp.Source: A. Zero-Shot Sim-to-Real Transfer · p. 7

B. Real-World Robustness Under Perturbations

  • GearMesh robustness was tested under lighting change, a 2-cm camera shift, static 2-N F/T bias, and background change, with 20 trials per condition.Figure 5 reports perturbed success and its change from each method’s own unperturbed reference.Source: B. Real-World Robustness Under… · p. 7, Figure 5 · p. 7
  • PACE retains 100% success under lighting and background changes and 90% under camera shift and F/T bias, at most 10 pp below reference.Pose-Based PPO falls from 90% to 10% under camera shift; without masked reconstruction, PACE drops 20–30 pp under every perturbation.Source: B. Real-World Robustness Under… · p. 7, Figure 5 · p. 7

VII. LIMITATIONS AND FUTURE WORK

The evaluation is limited to one robot platform and four quasi-static assembly tasks, and hardware results use one checkpoint per method–task pair. The conclusion presents proprioceptive anchoring as a way to connect domain-sensitive sensing to a stable physical reference.

VII. LIMITATIONS AND FUTURE WORK

  • PACE is evaluated on one robot platform and four quasi-static tasks; one simulated-performance-selected checkpoint per method–task pair leaves transfer variability across training seeds unmeasured.Source: §VII · p. 7
  • The anchor assumes consistent cross-domain proprioceptive semantics, which may weaken with changes in embodiment, low-level control, or proprioceptive sensing.Future work could use end-effector displacement in task space rather than robot-specific joint coordinates.Source: §VII · pp. 7–8

VIII. CONCLUSION

  • Across four Factory and AutoMate tasks, frozen PACE features transfer with less performance degradation than pose-estimate or learned-fusion policies and remain robust to visual and sensor perturbations.Source: §VIII · p. 8
  • The conclusion frames transferable representation learning as anchoring domain-sensitive modalities to a domain-stable physical reference rather than eliminating every domain gap.Its component roles are proprioceptive anchoring for generalization, vision for spatial alignment, F/T sensing for contact-sensitive interactions, and masked reconstruction for robustness to unreliable observations.Source: §VIII · p. 8
Loading 2609.07534v3…