PaperlayerView exact revision on arXiv

Source-linked AI summary

MobileVISTA: Generative Data Augmentation for Pose Generalization in Mobile Manipulation

Suzannah Wistreich, Stephen Tian, Isabella Huang, Vitor Campagnolo Guizilini, Sergey Zakharov, Katherine Liu, Jiajun Wu

arXiv:2610.07511v1cs.ROcs.CVcs.LG

TL;DR

Single-pose-trained manipulation policies can struggle when deployment base poses differ, and collecting demonstrations across poses is often impractical. MobileVISTA jointly synthesizes pose-perturbed egocentric observations and retargeted actions from canonical-pose demonstrations. The paper reports improved robustness over baselines in simulation and on a real humanoid, with performance bounded by the tested perturbation range.

  • Problem

    Policies trained on narrow single-pose demonstration distributions may not generalize to deployment base poses, while collecting demonstrations across many poses is often impractical.

    Source: §I · p. 1

  • Method

    MobileVISTA generates pose-perturbed training data from canonical-pose demonstrations by jointly synthesizing egocentric observations and retargeting actions.

    Source: §I · p. 2

  • Results

    MobileVISTA improves robustness over strong baselines in simulation and on a real Galaxea R1 Pro humanoid, with reported benefit scaling with the robot’s share of the egocentric frame.

    Source: §I · p. 2, §V · p. 8

  • Takeaways & Limitations

    The evaluation includes settings where the camera rides the actuated chain and the robot occupies much of the observation, without requiring additional demonstration collection or a trained generative model.

    Source: §I · p. 2, §V · p. 8

    Results are bounded by the sampled 15 cm translation and ±5° yaw range; behavior beyond it is untested.

    Source: §V · p. 8

Abstract

from arXiv · show

Mobile manipulators such as humanoid robots are increasingly deployed in dynamic, unstructured environments to perform dexterous manipulation tasks. However, end-to-end manipulation policies trained to imitate demonstration data collected from a single robot pose are brittle: even centimeter-scale deviations in robot pose at deployment can drive ego-centric observations and end-effector trajectories out of the training distribution, leading to sharp drops in performance. We introduce MobileVISTA, a data generation framework that transforms demonstrations captured at canonical poses into diverse, pose-perturbed training data by jointly (1) augmenting egocentric visual observations and (2) retargeting actions to compensate for base pose changes. Unlike prior methods, which assume a camera rigidly mounted off the actuated chain or non-trivial articulated robot geometry largely out of frame, MobileVISTA targets compatibility with egocentric platforms (e.g., humanoids) where the camera is both influenced by and must observe the robot's kinematic chain as it moves. We study MobileVISTA in simulated tasks spanning humanoid and bimanual embodiments, and on a real Galaxea R1 Pro. We find policies trained on MobileVISTA-augmented data demonstrate improved robustness to previously out-of-distribution poses encountered at test time, without additional demonstration collection or a trained generative model. Additionally, we find MobileVISTA's benefit is largest on tested humanoids, where the camera rides the actuated chain and the robot fills much of the frame. Additional videos and appendix can be found on our website: https://mobilevista.github.io

I. INTRODUCTION

Policies trained from single-pose demonstrations can fail under centimeter-scale base shifts, while existing augmentation methods do not address the combined challenge of an actuated egocentric camera and visible robot geometry. MobileVISTA jointly generates pose-perturbed observations and retargeted actions, and is evaluated against learned novel-view synthesis and other baselines in simulation and on real hardware.

I. INTRODUCTION

  • Even centimeter-scale base pose deviations can push single-pose-trained policies into out-of-distribution states, and collecting demonstrations across many poses is often impractical.Mobile settings also change both egocentric observations and the actions needed to complete a task.Source: §I · p. 1
  • MobileVISTA transforms demonstrations from one canonical base pose into physically consistent, pose-perturbed training data without additional data collection.Its observation synthesis removes the robot, reprojects the scene, and composites the robot rendered at retargeted joint configurations.Source: §I · p. 2
  • Explicit geometry-based reprojection with robot segmentation and re-compositing better preserves scene geometry and yields stronger downstream policy performance than learned novel view synthesis.The reported benefit scales with how much of the egocentric frame the robot occupies; evaluation also reports improvements over baselines under unseen base perturbations.Source: §I · p. 2

II. RELATED WORK

  • The target setting combines a camera on the actuated chain with visible articulated robot geometry, requiring the viewpoint and robot rendering to reflect retargeted actions.Prior methods satisfy at most one of these two setting properties; none of the prior methods in Table I is evaluated on a humanoid.Source: §II · p. 2, Table I · p. 2
  • MobileVISTA is compared directly with learned novel view synthesis; several related methods are not run as baselines because they lack perturbed-pose actions, synthesize no images, or lack a public implementation.The paper instead compares against GEN3C and uses a Fixed Robot Geom. ablation to isolate robot re-rendering.Source: §II · p. 2, §I · p. 2

III. THE MOBILEVISTA FRAMEWORK

MobileVISTA converts single-pose demonstrations into pose-shifted examples by jointly changing egocentric observations and retargeting their corresponding actions. Its visual pipeline reprojects the environment from the action-resolved camera viewpoint and composites the robot in its retargeted configuration.

III. THE MOBILEVISTA FRAMEWORK

  • Each augmented trajectory pairs pose-consistent rendered observations with retargeted actions intended to preserve the demonstrated end-effector trajectory under a sampled base translation and yaw.The translation is sampled within a disk of radius r, and yaw is sampled from U(−5°, 5°); wrist views are left unmodified.Source: §III · p. 3

A. Generating Visual Observations

  • The visual pipeline masks the robot from the original depth image, reprojects the remaining scene from camera extrinsics induced by retargeted actions, and composites a rendering of the robot at its retargeted configuration.Because the camera is on the kinematic chain, its extrinsics are inherited from the joint configuration resolved by inverse kinematics rather than chosen independently.Source: A. Generating Visual Observations · p. 3
  • E2FGVI inpainting fills holes caused by robot removal, scene disocclusion, and incomplete depth before the retargeted robot rendering is composited.The same pipeline is used on real demonstrations, with a robot mask from a rendered URDF depth buffer and intrinsics from camera calibration.Source: A. Generating Visual Observations · p. 3
  • MobileVISTA maintains higher success rates than all baselines as evaluation perturbation magnitude grows, with yaw sampled from U(−5°, 5°).Figure 3 plots success rate against evaluation perturbation radius.Source: Figure 3 · p. 4

B. Retargeting Actions

MobileVISTA retargets actions in end-effector space to compensate for base pose changes, using inverse kinematics when joint-space commands are needed. The experiments examine pose robustness, augmentation components, learned novel-view synthesis, and real-hardware effectiveness.

B. Retargeting Actions

  • Applying the inverse base-pose perturbation to each end-effector goal approximately preserves the demonstrated trajectory; whole-body inverse kinematics converts goals back to joint-space actions when required.Gripper or finger commands remain unchanged.Source: B. Retargeting Actions · p. 4
  • Retargeted trajectories track the original world-frame end-effector path after an initial transient, and all trajectories converged within the sampled 15 cm range.The method assumes task-relevant contact and object dynamics remain approximately unchanged under sampled perturbations.Source: B. Retargeting Actions · p. 4

IV. EXPERIMENTS

  • The experiments ask whether MobileVISTA improves out-of-distribution pose robustness, whether coupled augmentation and robot re-rendering matter, whether it outperforms learned novel-view synthesis, and whether it works on real hardware.Comparisons include canonical-only Diffusion Policy, DP3, fine-tuned π0.5, and a simulator oracle; MobileVISTA and its ablations use the same policy architecture and pipeline.Source: §IV · p. 4

A. Simulated Experimental Setup

The simulated evaluations span humanoid and bimanual tasks, with pose perturbations swept across evaluation radii. Ablations show that joint visual–action augmentation and robot-consistent rendering contribute to pose robustness, although the rendering benefit varies by task.

A. Simulated Experimental Setup

  • Can Sort and Pouring use a Fourier GR-1 humanoid, while Box and Drawer use a bimanual Franka; evaluation sweeps base translation radii from 0 to 15 cm with yaw held at U(−5°, 5°).Can Sort uses an even split of demonstrations across can color; other tasks train on the first 200 demonstrations and evaluate on environment variants.Source: A. Simulated Experimental Setup · pp. 4–5

1) Coupled Visual–Action Augmentation:

  • Augmenting images or actions alone consistently degrades performance across tasks, supporting the need to augment visual observations and actions jointly.The single-modality variants pair augmented observations with canonical actions or canonical observations with retargeted actions.Source: 1) Coupled Visual–Action Augmentation · p. 5

2) Robot-Consistent Rendering After Retargeting:

  • Treating robot geometry as static during scene reprojection causes substantial performance drops in most tasks, supporting explicit rendering of the robot after retargeting.The ablation omits robot segmentation and the rendering overlay, leaving the robot fixed in the environment during reprojection.Source: 2) Robot-Consistent Rendering After… · p. 5, Figure 4 · p. 5
  • MobileVISTA’s average success-rate gap over Fixed Robot Geom. is +0.47 on Pouring, +0.28 on Can Sort, +0.24 on Box, and approximately 0.00 on Drawer.The authors hypothesize that the effect is strongest when the robot occupies more of the view and the camera rides the kinematic chain, but note four tasks cannot separate these properties or explain the Box–Drawer difference.Source: 2) Robot-Consistent Rendering After… · p. 5, Figure 4 · p. 5

C. Quality Analysis of Synthesized Observations

MobileVISTA-synthesized training data outperforms GEN3C across tasks and perturbation magnitudes, and real-world evaluation shows pose robustness on the Galaxea R1 Pro. Observation-quality comparisons and real-robot outcomes also reveal where the gains are strongest and what trade-offs remain.

C. Quality Analysis of Synthesized Observations

  • MobileVISTA-trained policies outperform GEN3C-trained policies across all four tasks and the full range of evaluation perturbations.The comparison links the success-rate curves to the Simulator (Oracle) upper bound.Source: Figure 5 · p. 6
  • On the two humanoid GR-1 tasks, MobileVISTA exceeded GEN3C by 10.2 dB and 8.9 dB, versus 2.9 dB and 3.9 dB on the two Franka tasks.The largest PSNR separation was on humanoid tasks.Source: C. Quality Analysis of Synthesized… · p. 6, Figure 6 · p. 6
  • MobileVISTA generated augmented data 4–7× faster than GEN3C across tasks.Full per-task averages are reported in Appendix Table VI.Source: C. Quality Analysis of Synthesized… · p. 6
  • Unlike static-scene NVS, MobileVISTA re-renders the robot under retargeted actions and composites it with a geometry-consistent background.The paper notes that treating the robot as part of the static scene limits representation of base-conditioned configuration changes.Source: C. Quality Analysis of Synthesized… · p. 6

D. Real-World Evaluation

  • On real Book Stacking, MobileVISTA grasped in 17/25 randomized trials and stacked in 13/25, while no baseline exceeded 2/25 at either stage.Figure 7 shows outcomes distributed across sampled offsets at reval = 15cm.Source: Figure 7 · p. 7, D. Real-World Evaluation · p. 7, Table II · p. 7
  • On Laundry Sort, MobileVISTA sorted at least one item in 18/25 trials and at least two in 12/25, while the strongest baselines reached 14/25 and 3/25, respectively.Source: D. Real-World Evaluation · p. 7

V. CONCLUSION & LIMITATIONS

MobileVISTA jointly synthesizes pose-perturbed egocentric observations and retargeted actions from single-pose demonstrations, and experiments report improved robustness over baselines. The tested scope is bounded by camera, viewpoint, and perturbation-range limitations.

Limitations

  • Results are bounded to 15cm translation and ±5° yaw perturbations; behavior beyond that range is untested.Narrower camera field-of-view may reduce visible context and degrade synthesized-view consistency and policy performance; wrist-mounted cameras were not augmented.Source: §V · p. 8

Conclusion

  • MobileVISTA jointly synthesizes pose-perturbed egocentric observations and retargeted actions from single-pose demonstrations, improving robustness over baselines in simulation and on a real Galaxea R1 Pro.The evaluation includes settings where the camera rides the actuated chain and the robot occupies much of the observation.Source: §V · p. 8

APPENDIX

The appendix describes an observation-synthesis pipeline that segments the robot, reprojects scene geometry to a perturbed camera, fills missing regions, and composites a retargeted robot. It also outlines action retargeting implementations for simulated and real-world embodiments.

1) Robot Model and Segmentation:

  • The pipeline renders a robot silhouette separately, then extracts masks using distinctive skybox colors in simulation or a thresholded, dilated depth-buffer silhouette on the real R1 Pro.Simulation renders at 512×512; real-world data uses native resolution and is upsampled 6× during point-cloud reprojection.Source: 1) Robot Model and Segmentation · p. 9

2) Depth Reprojection and Camera Extrinsics:

  • Masked RGB-D observations are lifted into a world-frame point cloud, then reprojected through camera extrinsics from the robot’s retargeted configuration to create a shifted image and hole mask.The camera intrinsics come from MuJoCo’s vertical field of view in simulation and calibrated ZED2 intrinsics for real data.Source: 2) Depth Reprojection and Camera… · p. 9

3) Inpainting:

  • E2FGVI inpaints regions missing after reprojection, including robot-occupied and newly disoccluded areas.A 3 × 3 morphological closing is applied to the hole mask before temporal-batch inpainting.Source: 3) Inpainting · p. 9

4) Robot Re-Compositing:

  • After inpainting, the robot is re-rendered at its retargeted joint configuration and composited onto the background using a robot-pixel mask.Simulation uses a color-based mask; real-world compositing uses the rendered robot’s depth buffer.Source: 4) Robot Re-Compositing · p. 9

1) Forward and Inverse Kinematics:

  • Simulated end-effector actions are retargeted in closed form, while the real R1 Pro uses whole-body inverse kinematics to solve new joint configurations.The real-robot setup includes joint, velocity, and collision-avoidance limits.Source: 1) Forward and Inverse Kinematics · p. 9

2) End-Effector Goal Perturbation:

MobileVISTA compensates end-effector goals for base perturbations, with embodiment-specific handling for bimanual and real-world setups. Augmented trajectories sample planar offsets and, when enabled, yaw jitter.

2) End-Effector Goal Perturbation:

  • For simulated arms, the inverse base transform updates end-effector positions and orientations while leaving gripper commands unchanged; real R1 Pro IK tracks the original world-frame trajectory after its initial target is transformed.For bimanual Franka, compensation accounts for each arm’s changed base position after yaw about the rig midpoint.Source: 2) End-Effector Goal Perturbation · p. 10

C. Augmentation Sampling

  • Augmented trajectories sample planar translations uniformly within a 15cm disk, with optional yaw jitter from −5° to 5°; no vertical offset is applied.The same 15cm training and evaluation radius is used across simulated and real-world tasks; the paper generates one augmented trajectory per simulated demonstration and ten per real-world demonstration.Source: C. Augmentation Sampling · p. 10

D. Task Definitions

The evaluation covers four simulated tasks across humanoid and bimanual robots, then compares MobileVISTA with ablations and learned novel-view synthesis. MobileVISTA has higher PSNR and faster generation than GEN3C, while ablation results favor MobileVISTA on nearly all tasks.

D. Task Definitions

  • The four tasks are Can Sort and Pouring on a Fourier GR-1 humanoid, and Box and Drawer on a bimanual Franka setup.Can Sort involves a two-arm handover; the other tasks cover sequential pouring, bimanual lid placement, and opening a drawer before placing a mug.Source: D. Task Definitions · p. 10

E. Tabled Per-Task Results

  • MobileVISTA wins across nearly all ablation tasks and perturbation magnitudes; Fixed Robot Geom. is competitive or superior on Drawer at intermediate radii.The authors suggest Drawer may be less sensitive to inaccurate robot re-rendering.Source: E. Tabled Per-Task Results · p. 10, Table IV · p. 11

F. Comparison of MobileVISTA with Learned Novel View Synthesis

G. Policy Learning from GEN3C-Synthesized Views

The study compares policies trained on GEN3C-synthesized views with policies trained on MobileVISTA-synthesized views, using the same action retargeting procedure. MobileVISTA-trained policies outperform across all four tasks and tested perturbation magnitudes.

G. Policy Learning from GEN3C-Synthesized Views

  • The policy comparison isolates visual synthesis by pairing GEN3C views with MobileVISTA's action retargeting procedure.Source: G. Policy Learning from… · p. 12
  • Policies trained on MobileVISTA-synthesized data outperform those trained on GEN3C-synthesized data across all four tasks and evaluation perturbation magnitudes.The comparison pairs GEN3C views with the same action-retargeting procedure and uses the same randomized perturbation regime.Source: G. Policy Learning from… · p. 12
  • The performance gap is consistent with the reconstruction-quality gap reported for the two synthesis methods.Tasks with more similar PSNR generally also show smaller policy-success gaps.Source: G. Policy Learning from… · p. 12

H. Hardware Setup

The hardware evaluation uses a Galaxea R1 Pro with a head-mounted egocentric camera and manually marked base offsets. It tests book stacking and laundry sorting from canonical and randomized poses.

H. Hardware Setup

  • The Galaxea R1 Pro uses a head-mounted ZED2 RGB-D camera for the egocentric view, plus two wrist-mounted RealSense RGB cameras.A floor grid extends 15 cm in each direction from the canonical base pose, with markings checked against ground-truth measurements.Source: H. Hardware Setup · pp. 12–13
  • Table V reports PSNR in dB between synthesized egocentric views and simulator-oracle images across base perturbation magnitudes.Source: Table V · p. 13

I. Data Collection & Real World Task Definitions

  • The real-world tasks are book stacking and sorting four laundry items into two bins, with 50 canonical-pose demonstrations collected for book stacking.Laundry item placement and appearance are not domain-randomized; the task requires longer-horizon bimanual coordination with deformable items.Source: §I · p. 13

J. Evaluation Protocol

  • Each trial samples a base translation within a 15 cm radius, and all four policies are evaluated from the same offset in randomized order.Each policy receives 25 canonical-pose and 25 randomized-pose trials per task.Source: J. Evaluation Protocol · p. 13

K. MobileVISTA and Canonical-2D Diffusion Policy

The paper trains MobileVISTA and Canonical-2D policies with a shared Diffusion Policy configuration, and describes separate settings for DP3 and fine-tuned π0.5. Simulated π0.5 checkpoints are selected using success across three perturbation radii.

K. MobileVISTA and Canonical-2D Diffusion Policy

  • MobileVISTA and Canonical-2D policies use Diffusion Policy with identical hyperparameters across tasks and variants, and all three RGB cameras in simulation and real-world settings.Training runs for 1,400 epochs in simulation, 3,500 for real-world book stacking, and 6,500 for real-world laundry sorting.Source: K. MobileVISTA and Canonical-2D… · p. 13, Table VII · p. 14

L. DP3

  • DP3 uses its official implementation with default hyperparameters, a PointNet point-cloud encoder, and three random seeds per simulated task.Real-world DP3 training is specified by global step count because of compute constraints.Source: L. DP3 · p. 13

M. Fine-Tuned π0.5

  • Fine-tuned π0.5 starts from the released base checkpoint and uses LoRA; simulated checkpoints are selected by mean success across 0, 5, and 10 cm perturbation radii.Real-world fine-tuning uses 22,000 steps for Book Stacking and 41,000 for Laundry, scaled to match relative training budgets.Source: M. Fine-Tuned π0.5 · pp. 13–14, Real-world experiments · p. 14

Related papers

Loading 2610.07511v1…