PaperlayerView exact revision on arXiv

Source-linked AI summary

SARI: Phase-Split Sim-Real Co-Training for Contact-Rich Manipulation

Xingxin He, Yuxuan Jiang, Haonan Zhang, Chuhan Cui, Kaile Li, Zhongxing Zheng, Caihao Xu, Ziqi Wang

arXiv:2610.02804v1cs.RO

TL;DR

VLA models often need costly demonstrations for contact-rich tasks, and policies can overfit to demonstrated object placements. SARI trains one policy on simulated approaches and real contact interactions; across five tasks, collection time falls and unseen-placement success exceeds that of the full-task sim-real baselines.

  • Problem

    VLA policies can overfit to demonstrated object placements, while adapting them to new contact-rich tasks commonly requires many demonstrations.

    Source: §I · p. 1

  • Method

    SARI trains one policy on spatially diverse simulated approaches and real contact interactions, using appearance alignment and shared camera-relative actions without phase labels.

    Source: §I · p. 2

  • Results

    34.3% less real-data collection time and 27.5% unseen task completion, versus 0% for all baselines, across five real-world contact-rich manipulation tasks.

    Source: §I · p. 2

  • Takeaways & Limitations

    SARI's phase-specific data allocation and sim-to-real alignment support a single policy's spatial generalization to unseen placements in the evaluated tasks.

    Source: §VI · p. 7

    SARI assumes fixed tabletop scenes, a single-arm Franka robot, and two-stage approach–interaction tasks, and requires manual scene reconstruction, camera calibration, pre-contact-pose selection, and photometric reference selection.

    Source: §VI · p. 7

Abstract

from arXiv · show

Vision-language-action (VLA) models often require costly real-world demonstrations to adapt to contact-rich manipulation tasks, particularly when generalization across object placements is needed. We propose SARI (Simulated Approach, Real Interaction), a phase-split sim-and-real co-training framework built on a simple insight: spatial coverage and contact physics should be acquired from the domains best suited to them. Specifically, free-space approaches require spatial diversity but tolerate modest simulation gaps, making them ideal for synthetic generation; conversely, contact interactions demand accurate physics but vary little across object placements, allowing a few real demonstrations to generalize across the workspace. SARI generates diverse simulated approaches in a photorealistic digital twin while collecting real contact interactions at only a few placements. Post-trained on these phase-segmented demonstrations, a single policy seamlessly stitches simulated approaches with real contact interactions using visual appearance alignment and a shared camera-relative action representation--without explicit phase labels or hand-coded switches. Across five real-world contact-rich manipulation tasks, SARI reduces real-data collection time by 34.3% and achieves 27.5% success at unseen placements, where all full-task sim-real baselines fail completely (0%).

I. INTRODUCTION

SARI divides contact-rich manipulation into simulated free-space approaches and real contact interactions, assigning spatial coverage and contact physics to the domains best suited to them. Across five real-world tasks, it reduces collection time and succeeds at unseen placements where the baselines fail.

I. INTRODUCTION

  • VLA adaptation to contact-rich and long-horizon tasks commonly requires many demonstrations, while placement-specific action sequences can force redundant free-space demonstrations for workspace coverage.Post-trained policies may overfit to demonstrated object placements rather than extrapolate across the workspace [4], [5].Source: §I · p. 1
  • Synthetic-data pipelines can expand workspace coverage, but simulation gaps in friction, compliance, object resistance, and appearance make contact data unreliable; full-task co-training can also dilute simulated coverage.The spatially narrow real-data distribution may override and dilute the broad simulated coverage.Source: §I · pp. 1–2
  • SARI assigns spatially diverse free-space approaches to simulation and contact interactions to a few real demonstrations, matching each phase to the domain best suited to it.Approaches tolerate modest simulation gaps; contact interactions require accurate dynamics but vary little across placements.Source: §I · p. 2
  • 34.3% less real-data collection time and 27.5% unseen task completion, versus 0% for all baselines, across five real-world contact-rich manipulation tasks.SARI uses a Gaussian-Splatting digital twin for simulated approaches across placements and real contact interactions at only a subset.Source: §I · p. 2
  • A single policy stitches the simulated approaches and real contact interactions through visual appearance alignment and shared camera-relative actions, without explicit phase labels or switching heuristics.The policy post-trains on phase-segmented demonstrations.Source: §I · p. 2

II. RELATED WORKS

Prior sim-to-real approaches often rely on full-task trajectories or reliable simulated contact, while SARI assigns free-space coverage to simulation and contact dynamics to real demonstrations. Its task setting transfers a pre-contact pose across placements, but that target does not specify the contact behavior.

II. RELATED WORKS

  • Reconstruction-based synthesis is restricted to appearance-aligned free-space approaches because contact regimes involve friction, compliance, and object resistance that are difficult to model.Related pipelines often require full-task demonstrations or assume synthetic rollouts remain reliable throughout the task [4], [6]–[8],,, [15].Source: §II · p. 2
  • Existing co-training methods generally use full-task simulated and real data or add adaptation machinery, whereas SARI combines simulated approaches and sparse real contact data in one policy without phase labels.The phase allocation assigns workspace-wide coverage to simulation and accurate contact dynamics to real demonstrations.Source: §II · p. 2
  • VLA adaptation typically uses full-task demonstrations, while SARI uses a pretrained VLA, camera-relative actions, and phase-specific sim-real data with standard behavior cloning.The resulting policy is described as supporting generalization to placements without real interaction demonstrations [1], [21]–[27].Source: §II · p. 2

A. Task setting and pre-contact boundary

  • Demonstrations comprise a free-space move from home to the object and a contact segment that presses, pulls, or pushes it; they meet at a pre-contact pose.The pre-contact pose ends the simulated approach and marks the starting region for real contact collection.Source: A. Task setting and pre-contact boundary · pp. 2–3
  • The setup transfers one object-relative pre-contact pose across placements, assuming it remains suitable; fixed wrist-camera mounting preserves a similar local object view, not identical backgrounds or lighting.The geometric consistency also does not imply identical contact dynamics.Source: A. Task setting and pre-contact boundary · p. 3
  • Equation (1) specifies an approach target only; it does not specify the forces or motion needed to complete contact behavior.The approach target therefore does not itself define the interaction policy.Source: A. Task setting and pre-contact boundary · p. 3

B. Segmented, partially real dataset

The training data pair simulated free-space episodes with real contact episodes collected at a subset of placements. Neither source supplies a full home-to-contact demonstration.

B. Segmented, partially real dataset

  • The training set contains simulated free-space episodes from home to the pre-contact pose and real contact episodes beginning near that pose at selected placements.The real placements form a subset of candidate placements.Source: B. Segmented, partially real dataset · p. 3
  • Real contact demonstrations are collected only for placements in the selected subset; simulated episodes cover the approach phase.The real-contact subset is denoted Preal.Source: B. Segmented, partially real dataset · p. 3
  • Both sources use the same task language instruction and action convention, and one policy is post-trained on their union without a phase label or intermediate-goal instruction.Neither source contains a home-to-contact full-task demonstration.Source: B. Segmented, partially real dataset · p. 3

C. Evaluation regimes

Evaluation tests whether contact behaviors transfer to placements without real interaction demonstrations, with real rollouts beginning at home. SARI trains a unified policy by combining simulated approaches and real contact segments through observation alignment and shared camera-relative actions.

C. Evaluation regimes

  • At evaluation, the robot completes each task in one continuous real-world rollout from home; unseen placements have simulated approach coverage but no real interaction demonstrations.The evaluation question is whether contact behaviors generalize to these unseen placements.Source: C. Evaluation regimes · p. 3

IV. METHODS

  • SARI offloads free-space approaches to a workspace digital twin and collects real contact segments from transferred pre-contact poses.Observation alignment and a shared camera-relative action space fuse the two data sources for imitation learning by one policy.Source: §IV · pp. 3–4

A. Cross-domain appearance alignment

The method aligns simulated and real observations through calibrated rendering and color mapping, then expresses both domains’ actions in a shared wrist-camera-relative representation. The shared representation supports chunked policy targets and conversion back to controller commands at rollout.

A. Cross-domain appearance alignment

  • Degree-3 spherical harmonics preserve view-dependent appearance, while per-camera photometric color maps reduce color differences between simulated and real images.The maps are fitted from material-region color statistics and applied to simulated images without changing their actions, states, or episode boundaries.Source: A. Cross-domain appearance alignment · p. 4
  • Generative video translation can fill visual holes and improve alignment, but applying it across demonstrations is computationally expensive; scene rebuilding is more efficient.Source: A. Cross-domain appearance alignment · p. 4

B. Camera-relative action representation

  • Camera-relative actions re-express absolute Cartesian commands as translations and rotations relative to the wrist-camera view, giving both domains shared action semantics.Targets in each chunk use the same starting pose and camera orientation rather than successive step-to-step increments.Source: B. Camera-relative action representation · p. 4
  • The policy predicts chunked camera-relative targets, which are decoded into absolute Cartesian targets for control; the chunk’s anchor pose and camera orientation remain fixed during execution.The anchor is refreshed at the next policy invocation.Source: B. Camera-relative action representation · p. 4
  • A π0.5-DROID checkpoint is midtrained on the camera-relative representation before post-training on segmented task data.The 100,000-step midtraining gives the checkpoint the same action semantics used by the segmented task data.Source: B. Camera-relative action representation · p. 4

C. Segmented data curation

Simulated episodes provide collision-free free-space approaches from the home region to a pre-contact region across candidate placements. Real demonstrations begin from transferred pre-contact poses and cover the remaining contact interaction.

C. Segmented data curation

  • Simulated episodes transfer the reference pre-contact pose across candidate placements and generate collision-free approaches from the deployment home region.They stop at the pre-contact region, with no simulated contact or appended terminal hold.Source: C. Segmented data curation · p. 4
  • The simulated approach images undergo appearance alignment, while the real interaction segment begins at the pre-contact boundary.Source: C. Segmented data curation · p. 4
  • Real contact collection initializes the robot at a transferred pre-contact pose, from which an operator completes the interaction by teleoperation.The construction is applied across tasks using their respective object assets and pre-contact transforms.Source: C. Segmented data curation · p. 4

D. Training

One policy is post-trained on simulated and real segmented demonstrations using standard flow-matching behavior cloning. Training and evaluation configurations match key budgets, while the source sampling ratio is selected separately from the recorded episode count.

D. Training

  • A single π0.5 policy is post-trained on the union of simulated and real data with standard flow-matching behavior cloning, then deployed from home without a hand-coded phase switch.It receives camera observations, robot state, and a task instruction, and predicts action chunks.Source: D. Training · p. 4
  • Methods use matched initialization, post-training compute, and evaluation, while the post-training schedule and prediction and execution horizons are held fixed for each configuration.Source: D. Training · p. 5
  • The sim-real sampling ratio sets how often each data source contributes training samples and is distinct from the number of recorded episodes.Both mixed-data methods receive the same validation budget for selecting the ratio; no additional phase- or distance-based weighting is used.Source: D. Training · p. 5
  • The reported implementation post-trains a 100k-step camera-action DROID checkpoint for 10,000 updates on four H800 GPUs with batch size 128 and action chunk size 16.Source: D. Training · p. 5

V. EXPERIMENTS

The experiments examine whether SARI improves generalization to placements lacking real interaction demonstrations, reduces data-collection effort, and transfers across tasks and physical dynamics. The setup uses five contact-rich tasks with simulated approaches across six placements and real contact data at two.

V. EXPERIMENTS

  • The experiments ask whether SARI improves spatial generalization and reduces real-data effort, transfers across contact-rich tasks, and identifies the alignments needed to stitch the two data sources.Source: §V · p. 5

A. Experiment setup

  • The five tasks cover button pressing, lamp operation, drawer opening, book opening, and tissue pulling, representing spring resistance, friction, articulated lifting, and compliant-material pinching.Source: A. Experiment setup · p. 5
  • Each task has six nominal placements, with real contact data restricted to two and simulated approaches covering all six; a specialist policy is post-trained for each task.Experiments use a Franka Research 3 with a Franka Hand, two exterior cameras, and a wrist camera.Source: A. Experiment setup · p. 5
  • For the lamp testbed, contact training uses P1 and P3 while P2 and P4–P6 are unseen; success requires turning on the lamp within 40 seconds without intervention.All evaluations start from the same home configuration.Source: A. Experiment setup · p. 5

B. Spatial Generalization and Real-Data Efficiency

Phase-split allocation achieved nonzero full-task completion at every unseen placement, unlike full-task sim-real co-training. The evaluation contrasts approach and completion performance across seen and unseen placements.

B. Spatial Generalization and Real-Data Efficiency

  • The comparison used full-task simulated and real demonstrations alongside phase-split data that contained simulated approaches and real contact segments, but no full-task demonstrations.All four methods shared the same camera-action DROID-midtrained checkpoint and post-training compute budget; evaluation separated approach from task completion.Source: B. Spatial Generalization and… · p. 6
  • SARI was the only real-data method with nonzero full-task completion at every unseen placement; full-task co-training completed none.P2 completion was 40.0%; it lies between the seen placements P1 and P3, so the authors conjecture this may reflect spatial interpolation.Source: B. Spatial Generalization and… · p. 6
  • Full Real approached both seen placements in every trial but completed no unseen trials, while Full Sim approached every placement but did not complete real tasks.The Full Real policy typically navigated toward a memorized seen placement when the object was at an unseen location.Source: B. Spatial Generalization and… · p. 6

C. Evaluation Across Contact-Rich Tasks

The phase-split recipe transferred across five contact-rich tasks, with unseen-placement success varying by task. Appearance comparisons found that photometric correction and complete scene geometry matter for aligning simulated observations with real ones.

C. Evaluation Across Contact-Rich Tasks

  • 81.25% on book, 75.0% on button, 43.75% on tissue, 27.5% on lamp, and 25.0% on drawer were unseen-placement success rates.Real contact demonstrations covered P1/P3, while simulated approaches covered all six placements.Source: C. Evaluation Across Contact-Rich Tasks · p. 6

1) Appearance Alignment:

  • Four appearance configurations varied rendering fidelity, color correction, and scene completeness; low-fidelity rendering without color correction failed to bridge the visual domain gap.The configurations included low-fidelity rendering, corrected low-fidelity rendering, corrected high-fidelity rendering with corrupted GS, and corrected high-fidelity rendering with correct GS.Source: 1) Appearance Alignment · p. 7, Figure 5 · p. 7, Table IV · p. 7
  • Photometric color correction was necessary to prevent domain-based behavior, and missing background geometry explicitly hurt policy performance.The rendering comparison included degree-three spherical harmonics versus a low-fidelity rendering truncated to the view-independent term.Source: 1) Appearance Alignment · p. 7

2) Action Representation:

Camera-relative actions performed best among the evaluated action representations for phase stitching. The conclusion reports reduced collection time and unseen-placement completion, while noting limits on scene, robot, task, and setup scope.

2) Action Representation:

  • 50.0% seen and 27.5% unseen success for camera-relative π0.5 exceeded GR00T N1.7's 35.0% seen and 20.0% unseen success.Both joint-space π0.5 configurations failed on every trial.Source: 2) Action Representation · p. 7
  • Among configurations evaluated in the action comparison, camera-relative π0.5 performed best; SARI uses simulated approaches and real interactions, with photometric color alignment and action-space standardization enabling joint VLA post-training.Experiments on five tasks show reduced real-data collection and improved performance. On the lamp task, real demonstration time drops from 28.03 to 18.42 minutes (34.3%), with 11/40 (27.5%) unseen task completion versus 0/40 for all baselines.Source: 2) Action Representation · p. 7, §VI · p. 7

VI. CONCLUSION

  • 34.3% lower lamp-task demonstration time accompanied 11/40 (27.5%) unseen task completion, versus 0/40 for all baselines.The reported real demonstration time fell from 28.03 to 18.42 minutes.Source: §VI · p. 7
  • SARI assumes fixed tabletop scenes, a single-arm Franka, and two-stage tasks; dynamic scenes, moving objects, other embodiments, and automation remain future work.Scene reconstruction, camera calibration, pre-contact-pose selection, and photometric reference selection currently require manual work.Source: §VI · pp. 7–8

Related papers

Loading 2610.02804v1…