PaperlayerView exact revision on arXiv

Source-linked AI summary

VOMMI: Collecting and Leveraging Portable Demonstrations for Mobile Manipulation

Yutian Zhang, Xingrui Xiong, Siyuan Ma, Yang Li, Jiawen Wen, Jiaqi Zhai, Liwen Yang, Ce Hao, Haozhen Chi, Yangkun Zhu, Yifan Zhu, Xiaowen Chu, Dong Wei, Qiaojun Yu, Dibo Hou

arXiv:2610.08220v1cs.ROcs.AI

TL;DR

Portable mobile-manipulation demonstrations are difficult to collect with reliable, low-cost motion supervision from RGB, in part because existing approaches can require specialized hardware and VO trajectories can drift. VOMMI connects portable RGB demonstrations to VLA training through trajectory reconstruction and online visual-motion conditioning. Across three real-robot tasks, it improves mean success over OpenPI 0.5 by 8.3 percentage points.

  • Problem

    Portable mobile-manipulation collection faces limited demonstration data, specialized-hardware requirements, and drift-prone VO trajectories that complicate reliable motion supervision from RGB.

    Source: §I · p. 1

  • Method

    VOMMI uses synchronized body and hand RGB views, R2-VO trajectory refinement and local-motion tokens, and a residual adapter that conditions base actions.

    Source: §I · pp. 1–2, 2) Offline Drift Refinement · p. 5

  • Results

    An 8.3-percentage-point mean-success gain over OpenPI 0.5 was reported across the evaluated real-robot tasks.

    Source: B. Main Experiment Results · p. 6

  • Takeaways & Limitations

    Portable VOMMI demonstrations support local-motion prediction and closely match teleoperation references in in-domain tasks, with benefits most pronounced when navigation demands are substantial.

    Source: §VII · p. 8

    Validation is on the M20S platform, and further cross-embodiment evaluation remains necessary.

    Source: §VII · p. 8

Abstract

from arXiv · show

Portable mobile-manipulation demonstrations can help alleviate data scarcity for embodied intelligence, but obtaining reliable, low-cost, and robot-free motion supervision from RGB observations remains challenging. Existing approaches often rely on teleoperation or specialized devices equipped with additional sensing hardware, while directly using estimated visual odometry (VO) trajectories can introduce inconsistencies due to accumulated drift and imperfect motion supervision. We present the Visual-Odometry-Conditioned Mobile Manipulation Interface (VOMMI), a portable demonstration collection and learning framework that connects portable RGB demonstrations to vision-language-action (VLA) post-training through offline trajectory reconstruction and online visual-motion conditioning. VOMMI synchronizes body and hand views to capture navigation context and local object interactions without requiring human-robot kinematic correspondence calibration. R2-VO refines offline demonstration trajectories using sparse geometric anchors and produces causal local-motion tokens over multiple prediction horizons for online policy conditioning. An action-group residual adapter incorporates these tokens only into the base branch. Experiments use a 500-trajectory portable for each task, with 75 trajectories held out for RGB-VO evaluation, and 200 robot demonstrations as references. Our policy, post-trained only on portable demonstrations, achieves 18.2% lower base-velocity error than a policy trained with robot-collected demonstrations, while maintaining comparable end-effector translation accuracy. Offline reconstruction reduces absolute trajectory errors for the body and hand streams by 24.6% on average relative to the best evaluated baseline for each stream. The complete system improves the mean success rate by 8.3 percentage points over OpenPI 0.5 across three real-robot tasks.

I. INTRODUCTION

Mobile-manipulation learning is constrained by scarce demonstrations and portable collection pipelines that can require specialized sensing and demanding trajectory calibration. VOMMI connects body- and hand-view RGB demonstrations to whole-body VLA training using visual motion, without requiring robot-specific kinematic calibration.

I. INTRODUCTION

  • VOMMI links portable body- and hand-view RGB demonstrations to whole-body VLA training, using visual motion cues to condition base actions while keeping manipulation residuals independent.The framework is intended to provide local visual proprioception for navigation-conditioned mobile manipulation, rather than global localization or a general-purpose VO algorithm.Source: §I · pp. 1–2
  • The collection protocol records separate body and hand views without requiring access to the target robot or human–robot kinematic correspondence calibration.The body view captures scene context and locomotion, while the hand view captures local interaction.Source: §I · pp. 1–2
  • R2-VO provides refined RGB motion reconstruction for offline trajectory processing and low-latency local-motion tokens for online VLA inference.The tokens are connected to the base-action residual adapter.Source: §I · p. 2

A. Portable Demonstration Interfaces

B. Robot Learning for Mobile Manipulation

  • VOMMI addresses the lack of explicit recent ego-motion in portable-RGB mobile-manipulation policies through VO observations and separate residual pathways for body and manipulation actions.The stated design does not use metric pose or IMU measurements as policy inputs at deployment.Source: B. Robot Learning for Mobile… · p. 2

C. Visual Body Motion for Policy State

VOMMI frames visual body motion as a compact, causal local state for policy conditioning rather than a globally consistent trajectory. Its data pipeline pairs complementary RGB streams and uses held-out episodes to evaluate RGB-based reconstruction.

C. Visual Body Motion for Policy State

  • VOMMI’s causal local representation reduces sensitivity to constant world-frame offsets while preserving short-term ego-motion cues relevant to whole-body control.The resulting token is a compact visual proprioceptive state for policy conditioning, not a global localization memory.Source: C. Visual Body Motion for Policy State · p. 2

A. Dual-view capture and temporal alignment

  • Body and Hand RGB streams capture scene context and hand–object interaction, then are reconciled through timestamps and trajectory-based compensation.Episodes also store camera intrinsics, TCP and gripper signals, language instructions, and validity masks; collection-time trajectories are offline references, not deployment inputs.Source: A. Dual-view capture and temporal… · p. 2, Figure 2 · p. 3

B. Episode schema and split protocol

  • Each evaluated task has 500 portable episodes, with 75 held out for RGB-VO evaluation and excluded from fitting and model selection.The remaining 425 episodes support model development and downstream VLA training.Source: B. Episode schema and split protocol · p. 3

IV. PROBLEM FORMULATION

The formulation recovers robot-compatible local motion from two RGB streams, using planar Body motion and current-relative Hand motion as complementary targets. Reference trajectories supervise offline construction and evaluation, while deployment uses RGB observations.

IV. PROBLEM FORMULATION

  • The central data-construction problem is recovering robot-compatible local motion from two unsynchronized RGB streams for transferring portable human demonstrations to robot policy learning.Source: §IV · p. 3
  • The Body trajectory is represented as planar motion.Source: §IV · p. 3
  • The Hand trajectory uses current-relative TCP motion in a local SE(3) representation that reduces dependence on the global reference frame.Source: §IV · p. 3
  • Body and Hand motions are estimated from past RGB observations, and the Hand motion is compensated using the planar Body estimate aligned to Hand timestamps.Source: §IV · p. 3
  • Reference trajectories provide privileged supervision for offline construction and evaluation, whereas deployment relies only on RGB observations.Source: §IV · p. 3

A. Unified Observation and Action Space

  • The source-conditioned observation masks unavailable source-specific components; notation distinguishes collection-time reference trajectories from RGB-reconstructed trajectories.Source: A. Unified Observation and Action Space · p. 3
  • The policy predicts a horizon-H action chunk, and the action mask marks valid dimensions so missing values are excluded rather than treated as zeros.The chunk is specified as 50 steps at the 15-Hz policy clock.Source: A. Unified Observation and Action Space · p. 3

V. VOMMI

VOMMI turns portable RGB demonstrations into a whole-body policy representation using offline trajectory refinement and online causal body-motion estimates. Its action interface uses local actions and restricts motion-conditioned updates to body actions.

V. VOMMI

  • VOMMI refines demonstration trajectories offline with sparse geometric anchors and supplies causal body-motion estimates online as multi-horizon tokens that condition body-action residuals.The overview also shows first-valid-pose initialization and planar Body compensation of the Hand stream before forming demonstration targets.Source: §V · p. 3, Figure 3 · p. 4

A. Local Cross-Embodiment Action Representation

  • Body commands are expressed in the current Body frame and end-effector targets in the current TCP frame, providing a common action interface without supervision tied to an arbitrary world origin.A per-sample mask selects source-available action dimensions, and the flow loss is normalized over valid dimensions.Source: A. Local Cross-Embodiment Action… · p. 4
  • Current-frame relative targets cancel constant global translation and yaw offsets when the world-frame error is unchanged; otherwise, only its local error increment enters the target.The same property applies independently to Body and Hand streams, whose tracking frames need not be mutually registered.Source: A. Local Cross-Embodiment Action… · p. 4

B. Body-Hand Alignment and Planar Compensation

Body–Hand alignment uses planar Body motion to compensate Hand trajectories and form body-relative targets. RGB-based Body reconstruction is refined with frozen geometric anchors, while the resulting targets remain local to the relevant frames.

B. Body-Hand Alignment and Planar Compensation

  • Body-relative Hand targets require the Hand TCP and Body pose in a common coordinate system; with verified rigid cross-device registration, the registered poses yield a body-relative TCP pose.Source: B. Body-Hand Alignment and Planar… · p. 4
  • RGB-based Body reconstruction is refined with a frozen VGGT-plus-anchor branch, whose global geometric constraints and sparse anchors mitigate long-term VO drift and recover metric consistency.The resulting canonicalized trajectory is initialized at its first valid timestamp, and its relative increments are invariant under a constant left gauge.Source: B. Body-Hand Alignment and Planar… · p. 4
  • Planar compensation projects Body motion to SE(2), lifts it to SE(3), and uses Body translation and yaw to form the compensated Hand pose.Source: B. Body-Hand Alignment and Planar… · p. 4
  • For RGB source replacement, the canonicalized Body trajectory is aligned to Hand timestamps and compensates Body x/y translation and yaw; the deployment adapter then emits six current-TCP-relative EE dimensions.Source: B. Body-Hand Alignment and Planar… · p. 5

C. R2-VO

R2-VO provides local metric motion from RGB through complementary causal and offline modes that share a frozen motion estimator.

C. R2-VO

  • R2-VO provides local metric motion from RGB through causal and offline branches sharing one frozen motion estimator.The causal branch produces deployment motion tokens; the offline branch refines trajectories during demonstration construction.Source: C. R2-VO · p. 5
  • The causal branch produces motion tokens for deployment.These are distinct from the offline branch's trajectory-refinement role.Source: C. R2-VO · p. 5
  • The offline branch refines trajectories during demonstration construction using the shared frozen estimator.Source: C. R2-VO · p. 5

1) Causal Local Motion Estimation:

The causal estimator predicts one-step Body motion from RGB-derived descriptors and composes predictions into short-horizon motion tokens for online use.

1) Causal Local Motion Estimation:

  • A 206-dimensional visual descriptor expands to 317 dimensions with a seven-frame temporal window, and a 32-tree regressor predicts one-step planar motion.Inference uses past RGB frames, camera intrinsics, and timestamps.Source: 1) Causal Local Motion Estimation · p. 5
  • Atomic motion predictions are resampled to 15 Hz and composed over 2, 3, 6, or 12 frames, spanning 0.13–0.80 seconds.Each step integrates body-frame velocities from an initial pose of (0, 0, 0).Source: 1) Causal Local Motion Estimation · p. 5
  • Each horizon is encoded with position, heading, and uncertainty values, including x, y, sin θ, cos θ, log σx, log σy, and log σθ.Source: 1) Causal Local Motion Estimation · p. 5
  • Four horizon encodings plus a history-validity bit form a 29-dimensional token for causal short-horizon Body motion.Incomplete history is represented by a zero token and zero validity indicator.Source: 1) Causal Local Motion Estimation · p. 5

2) Offline Drift Refinement:

Offline refinement combines causal trajectories with sparse geometric anchors, while separate action-group residual pathways condition base actions on motion tokens but keep manipulation residuals blind to body motion.

2) Offline Drift Refinement:

  • Sparse 1-Hz VGGT-1B anchors refine the integrated causal Body trajectory, preserving short-term VO motion while correcting accumulated drift.A canonicalizer maps VGGT coordinates to the metric initial-camera frame.Source: 2) Offline Drift Refinement · p. 5
  • A weighted Ridge canonicalizer maps first-frame-centered VGGT positions and temporal features to metric reference positions in the initial-camera frame.Its feature map may use linear, time-augmented, or quadratic features for metric translation prediction.Source: 2) Offline Drift Refinement · p. 5
  • For the Hand stream, sparse anchors refine vertical motion offline, while Hand VO predicts complete local TCP SE(3) increments, including rotation, from RGB.TCP-relative pose is the training and evaluation target.Source: 2) Offline Drift Refinement · p. 5
  • Source-replaced demonstrations use the offline Body trajectory interpolated to Hand timestamps for labels, while online inference uses causal local increments.Source: 2) Offline Drift Refinement · p. 5

D. VO-Conditioned Action-Group Adaptation

  • Separate base and manipulation residual branches adapt the action expert; the projected, validity-gated motion token conditions the base residual, not the manipulation residual.The manipulation residual does not receive body motion.Source: D. VO-Conditioned Action-Group… · pp. 5–6
  • Motion-conditioned updates are restricted to the base action group, zero-initialized outputs preserve the pretrained policy, and training combines flow loss with a 0.1-weighted router loss.Router targets derive from base and manipulation activity; flow losses are validity-normalized and use corresponding action masks.Source: D. VO-Conditioned Action-Group… · p. 6

VI. EXPERIMENTS

The experiments evaluate R2-VO reconstruction accuracy, the impact of VO-refined data on rollouts, the role of local motion tokens, and component contributions.

VI. EXPERIMENTS

  • The evaluation examines R2-VO post-processing, VO-conditioned policy performance, and deployment across diverse mobile-manipulation tasks.Source: §VI · p. 6
  • One research question asks how accurately R2-VO reconstructs tracking-derived planar SE(2) and local TCP SE(3) pose references from RGB.Source: §VI · p. 6
  • The experiments ask whether VO-refined data improve real-world rollouts and how local motion tokens affect long-horizon mobile manipulation.Source: §VI · p. 6
  • A further question concerns which components contribute most significantly to R2-VO and VOMMI.Source: §VI · p. 6

A. Experimental Setting

Experiments evaluate R2-VO reconstruction and VOMMI policy performance on three mobile-manipulation tasks, using held-out trajectory references and real-robot trials. VOMMI improves mean task success over OpenPI 0.5, while offline reconstruction and portable-demonstration policy training improve their respective motion and action metrics.

A. Experimental Setting

  • The benchmark covers three tasks and evaluates the M20S over 20 independent trials per condition, using tracking-derived Body and Hand references.It reports trajectory error for post-processing, and task success rate and action deviation for real-world rollouts.Source: A. Experimental Setting · p. 6

B. Main Experiment Results

  • Baseline VO rows are not all 15-Hz runs: CUT3R and MASt3R-SLAM retain native sparse steps from 1-Hz RGB bundles.Ours and DPVO use matched stored-frame spans.Source: B. Main Experiment Results · p. 6
  • At the deployed 15-Hz step, R2-VO lowers mean translation error by 80.4% and 83.8% relative to DPVO and DROID-SLAM, respectively.At 267 ms, it retains 0.878 cm mean error, with reductions of 79.9% and 84.1%; measured latency is compatible with online 15-Hz inference.Source: B. Main Experiment Results · p. 6
  • Offline reconstruction reduces mean absolute Body/Hand trajectory error by 24.6% versus the best baseline for each stream.The offline VGGT+anchor variant gives 5.97 cm ATE and 5.33 cm endpoint error, versus 15.65 cm and 19.29 cm for the integrated causal Body branch.Source: B. Main Experiment Results · p. 6
  • 58.3% mean success for VOMMI versus 50.0% for OpenPI 0.5 across the three tasks, an 8.3-percentage-point gain.The complete-task success rates are reported over 20 trials per task.Source: B. Main Experiment Results · p. 6, Table III · p. 7
  • 18.2% lower base MAE for VOMMI-R than M20-trained, from 0.044 to 0.036 m/s.Compared with VOMMI-A, EE translation and rotation errors fall from 0.569 cm/1.091° to 0.170 cm/0.453°.Source: B. Main Experiment Results · p. 7

C. Ablation Study

The ablation examines how VO conditioning and residual pathways affect action prediction, while the conclusion frames VOMMI as a portable RGB pipeline for whole-body policy learning. Its demonstrated benefits are strongest when navigation demands are substantial, and evaluation so far is limited to the M20S.

C. Ablation Study

  • In one real-robot episode, the VO-conditioned policy retains a positive yaw command when OpenPI has returned to near-zero yaw, showing changed policy dynamics.Figure 6 compares global observations and first-step base actions, highlighting the residual VO turn at t = 3.73 s.Source: C. Ablation Study · p. 7, Figure 6 · p. 7
  • The ablation compares no-VO, base-only VO, all-action VO, and VOMMI residual designs using overall, navigation, and portable-manipulation flow losses.Within-source shuffled VO provides a paired motion-content check for the VO variants.Source: Table V · p. 7, C. Ablation Study · p. 8

VII. LIMITATION AND CONCLUSION

  • VOMMI’s benefits are most pronounced with substantial navigation demands and relatively limited in manipulation-dominated tasks; validation is on M20S, with cross-embodiment evaluation still needed.The conclusion describes portable demonstrations as supporting local-motion prediction and actions that closely match teleoperation references in in-domain real-world tasks.Source: §VII · p. 8
  • The conclusion describes a unified pipeline combining masked observations, causal body-frame motion, planar Body-to-Hand compensation, and action-group residuals that condition base actions on visual motion.Collection-time tracking poses supervise RGB motion estimation and local TCP SE(3) prediction during data construction.Source: §VII · p. 8
Loading 2610.08220v1…