PaperlayerView exact revision on arXiv

Source-linked AI summary

UniWAM Technical Report: Unified Mobile Manipulation via Mixed-Stream World-Action Modeling and Manipulation Anchor Pose Supervision

Wei Xue, Keliang Liu, Mingzhang Cui, Jinhua Xie, Jinjie Wei, Jianan Hou, Jingcheng Lu, Lintao Wang, Kaixiang Qiu, Yizhou Liu, Xinghai Ye, Jinghang Han, Mingcheng Li, Jie Gu, Shunli Wang, Lihua Zhang, Dingkang Yang

arXiv:2609.39388v1cs.ROcs.CV

TL;DR

Mobile manipulation requires reconciling navigation and manipulation’s different visual and action requirements, while explicit manipulation-ready pose supervision is costly to collect. UniWAM combines independently sampled navigation and manipulation streams with MAP supervision. It reduces position and heading errors on MAP-Bench and achieves leading docking and mobile-manipulation results across real-robot tasks, while remaining competitive in manipulation.

  • Problem

    Navigation and manipulation have different visual requirements and action spaces, while existing unified training jointly supervises both behaviors and explicit manipulation-ready pose data are costly to collect.

    Source: §1 · pp. 1–2

  • Method

    UniWAM uses independently sampled navigation and manipulation streams over a shared backbone, with explicit Manipulation Anchor Pose supervision from automatically generated data.

    Source: §1 · p. 2, §5 · p. 23

  • Results

    30.1% lower position error and 44.0% lower heading error versus the strongest baselines on MAP-Bench; across 24 real-robot tasks, UniWAM achieves the most accurate docking and highest mobile-manipulation success while remaining competitive in manipulation.

    Source: §5 · p. 24

  • Takeaways & Limitations

    Image-plane auxiliary supervision grounds the actions of both streams in egocentric imagery.

    Source: §5 · p. 23

    When adjacent objects create nearby action modes, the deterministic sampler can produce an action between them; specifying the target may reduce this ambiguity.

    Source: §4.7 · p. 22

Abstract

from arXiv · show

Mobile manipulation requires precise navigation to a manipulation-ready pose followed by reliable object interaction. These two stages differ in action spaces and visual requirements, which complicates unified policy learning. In addition, collecting diverse real-world navigation data with explicit manipulation-ready pose supervision remains costly and difficult to scale. We introduce UniWAM, a unified mixed-stream world-action model with separate action encoders and output heads for navigation and manipulation, sharing a common backbone. This design supports joint representation learning on independently sampled navigation and manipulation data. UniWAM supports independent inference for either stream and batch-parallel inference for both. We further introduce Manipulation Anchor Pose (MAP) supervision for where to stop and how to orient for manipulation. An automated pipeline constructs MAP-Data from large-scale 3D scenes, yielding over 1.5 million episodes and 7,500 hours. MAP-Data provides per-frame target-object bounding boxes and image-plane MAP coordinates as auxiliary navigation supervision. Together with projected end-effector trajectories for manipulation, these prediction targets provide stream-specific image-plane supervision for action learning from egocentric observations. With large-scale MAP-Data, UniWAM outperforms the strongest external baselines on our MAP-Bench by 30.1\% in position error and 44.0\% in heading error. Across 24 real-robot tasks, UniWAM achieves leading results in MAP navigation and mobile manipulation, with competitive manipulation performance. We have released code, data, and benchmark.

1 Introduction

Mobile manipulation must reconcile navigation and manipulation, which have different visual needs and action spaces, while explicit manipulation-ready terminal-pose data remain costly to collect. UniWAM separates the behaviors into independently sampled streams over a shared backbone, and MAP-Data supplies explicit pose supervision at scale.

1 Introduction

  • Navigation needs a forward view and smooth long-range base actions, whereas manipulation needs detailed local views and faster actions near contact.A unified policy must reconcile these distinct visual requirements and action spaces.Source: §1 · pp. 1–2
  • Jointly supervising both behaviors in complete-trajectory samples spends training on idle commands and prevents direct use of easier-to-collect single-behavior data.Existing policies either share an action representation or separate actions while retaining joint supervision.Source: §1 · p. 2
  • Existing unified models learn the manipulation-ready stopping pose only implicitly from costly complete trajectories, while semantic navigation specifies target proximity alone.Stopping at an unsuitable position or heading can lead to manipulation failure.Source: §1 · p. 2
  • UniWAM gives navigation and manipulation separate streams, visual inputs, action encoders, and output heads over a shared backbone.The streams are sampled independently; either can run alone or both can run in one deployment batch.Source: §1 · pp. 2–3
  • MAP is the base position and heading for ending navigation, and MAP-Data supplies over 1.5 million labeled episodes for explicit supervision.The data include target boxes and image-plane MAP coordinates, and a held-out subset forms leakage-free MAP-Bench.Source: §1 · p. 3
  • 30.1% lower position error and 44.0% lower heading error versus pretrained baselines, with mobile-manipulation success raised by 17.5 points.The evaluation covers MAP-Bench and 24 real-robot tasks.Source: §1 · p. 3

2 Related Work

Prior work includes unified policies that share action representations, world-action models that separate actions but retain shared views and supervision, and methods for selecting manipulation-ready base poses. The paper positions MAP-Data as large-scale supervision for the full approach trajectory, not only a terminal pose or placement score.

Unified mobile-manipulation models

  • Most unified policies share one action representation; ABot-M0.5 and MobileWAM separate actions but still share visual inputs and supervise both behaviors per sample.Cross-embodiment policies separate robots rather than navigation and manipulation phases within one robot.Source: §2 · p. 3

Manipulation-oriented navigation

  • Prior manipulation-oriented navigation methods search reachable placements, learn from affordances or rollouts, or infer poses at test time, but do not supervise the approach trajectory.The paper describes explicit terminal-pose supervision along full approaches as scarce.Source: §2 · pp. 3–4

Unified mobile-manipulation models

  • The comparison framework distinguishes separate views, one behavior per sample, video world modeling, and explicit terminal-pose supervision.These are the dimensions used to compare unified mobile-manipulation models and cross-embodiment policies.Source: Table 1 · p. 4

Manipulation-oriented navigation

  • MAP-Data supervises the full approach with explicit terminal-pose and dense image-plane labels; MoMa-Kitchen's 127K episodes are about one-twelfth its 1.5M episodes.The comparison covers approach supervision, label source, and dataset scale.Source: §2 · p. 4, Table 2 · p. 4

3 Method

The method section covers construction of MAP-Data and MAP-Bench, then describes UniWAM’s stream design and training procedure.

3 Method

  • The method first defines MAP and describes construction of MAP-Data and MAP-Bench.The section then presents UniWAM’s stream interfaces, mixed-stream packing, independent sampling, image-plane supervision, objective, and implementation.Source: §3 · p. 4
  • UniWAM’s method description covers its stream interfaces and mixed-stream batch packing.It also details independent stream sampling and image-plane auxiliary supervision.Source: §3 · p. 4
  • The method section also specifies the training objective and implementation.Source: §3 · p. 4

3.1 MAP: Manipulation Anchor Pose

MAP is a scene-geometry-based base position and heading that specifies where a robot should stop and how it should orient for manipulation. An automated pipeline generates approach trajectories and frame-aligned visual supervision toward these poses, while MAP-Bench measures position and heading errors from held-out starts.

3.1 MAP: Manipulation Anchor Pose

  • MAP specifies the base position and heading for stopping and orienting toward a target, and is computed from scene geometry rather than a particular manipulation policy.Making the terminal pose explicit allows approach trajectories to be generated at scale, yielding dense supervision throughout the approach.Source: §3.1 · p. 4
  • The pipeline curates valid targets, assigns each a MAP and feasible observation poses, then synthesizes approach trajectories with RGB, actions, MAP coordinates, and target boxes.Figure 2 summarizes the three stages and shows MAP-Bench evaluating position and heading errors from held-out initial poses.Source: §3.1 · pp. 4–5, Figure 2 · p. 5
  • Cubic Bézier trajectories connect observation poses to MAPs, with obstructed paths guided by A* and candidates rejected for violating clearance, visibility, curvature, path-length, or endpoint constraints.Retained paths approach from the annotated side and keep the manipulation-facing boundary within the horizontal field of view.Source: §3.1 · pp. 5–6
  • MAP-Data stores synchronized observations, navigation actions, projected MAP coordinates, target boxes, and quality metadata for over 1.5M episodes and 7.5K hours.The frame-aligned labels include scene and target identifiers and each trajectory carries five target-specific instructions.Source: §3.1 · p. 6
  • MAP-Bench evaluates reached position and heading against reference MAPs by distance range, with an Overall score averaging the three ranges.Its split assignment keeps initial poses and near-duplicate views separate, so evaluation tests unseen starting configurations rather than memorized trajectories.Source: §3.1 · p. 6

3.2 UniWAM: Unified Mixed-Stream World-Action Model

UniWAM uses separate navigation and manipulation interfaces and action heads with shared video and action DiTs. Independently sampled stream examples and image-plane auxiliary targets support joint learning while preserving stream-specific supervision.

3.2.1 Stream Interfaces and Action Representations

  • Navigation predicts planar pose increments relative to the chunk-start base pose, while manipulation uses camera-frame end-effector positions and continuous 6D rotations.The dual-arm action has 20 dimensions; a low-level controller converts navigation pose increments into base target velocities.Source: §3.2.1 · pp. 7–8

3.2.2 Mixed-Stream Batch Packing and Attention

  • Navigation and manipulation examples occupy separate batch rows that update shared DiTs, while attention is confined within each row and causal action masks prevent access to later actions.At inference, later actions can condition on clean committed actions in the prefix, and both streams can run continuously in one batch.Source: §3.2.2 · p. 8

3.2.3 Independent Stream Sampling

  • Mobile-manipulation trajectories are split into stream-specific windows, and source-normalized window weights control navigation/manipulation and active/idle proportions during independent sampling.Navigation-only and manipulation-only data contribute examples only to their respective streams; duplicate or temporally adjacent windows are rejected within a batch.Source: §3.2.3 · p. 8

3.2.4 Image-Plane Auxiliary Supervision

  • The action vectors include image-plane targets: target boxes and projected MAP positions for navigation, and projected end-effector tracks with visibility flags for manipulation.The end-effector tracks are generated by calibrated projection and appearance-gated visibility checks; only visible coordinates enter the auxiliary loss.Source: §3.2.4 · pp. 8–9

3.2.5 Training Objective

  • Flow matching trains future video latents, actions, and auxiliary targets, averaging losses over valid positions and features so each sample contributes only to its own stream.The current observation remains clean, while only future video latents are supervised; prefix actions are replaced by clean committed actions and excluded from supervision.Source: §3.2.5 · pp. 9–11

3.2.6 Implementation and Inference

  • Compilation, cached video features, and packed two-stream launches reduce two-stream H200 inference latency from about 220 ms to below 60 ms.The corresponding training setup exceeds 600 samples per second on 16 NVIDIA H200 GPUs.Source: §3.2.6 · p. 11

4 Experiments

The experiments evaluate UniWAM on MAP-Bench and 24 real-robot tasks, examining docking performance, mixed-stream design, data scaling, component contributions, and multi-embodiment co-training. They also present qualitative MAP navigation and image-plane action predictions and identify ambiguity with adjacent-object grasping and small residual base motions during manipulation.

4.1 Experimental Setup

  • The evaluation spans held-out MAP-Bench initial poses and 24 real-robot tasks across four embodiments, with each UniWAM variant serving all tasks and embodiments as one model.UniWAM-Base uses baseline-available data, while UniWAM-Scaled adds MAP-Data and task data through independent stream sampling; real-robot MAP Navigation and Mobile Manipulation trials include docking-error evaluation, and manipulation trials use task-specific success predicates.Source: §4.1 · pp. 11–13

4.2 MAP-Bench Results

  • 30.1% lower Overall position error (9.85 cm vs. GR00T N1.7’s 14.1 cm) and 44.0% lower Overall heading error (2.30° vs. π0.5’s 4.11°) make UniWAM-Scaled best across distance ranges.Without MAP-Data, UniWAM-Base already reduces Overall errors by 11.3% and 21.2%; adding MAP-Data cuts them by a further 21.2% and 29.0% relative to Base. The position gain from MAP-Data grows from 3.0% at short range to 32.4% at long range.Source: §4.2 · pp. 13–14, Table 4 · p. 14

4.3 Real-Robot Results

  • UniWAM-Scaled has the lowest position and heading errors in real-robot MAP Navigation and Mobile Manipulation, the highest Mobile Manipulation success rate, and the second-highest Manipulation success rate.In MAP Navigation it reaches 11.7 cm and 4.25°, 35.4% and 17.5% below the strongest baseline; in Mobile Manipulation it reaches 13.2 cm and 4.56° docking errors and 80.0% success, 17.5 percentage points above π0.5. For Manipulation, π0.5 leads average success at 71.4%, followed by UniWAM-Scaled at 68.9%.Source: Table 6 · p. 15, §4.3 · p. 15
  • Trained only on MAP-Data with no real-robot data, room, or target instance, UniWAM-Sim reaches 18.7 cm and 6.34° average docking errors on real robots.Its position error is lower than that of π0.5 trained on real demonstrations, although its heading error is higher; FastWAM’s position error is 12.1 cm worse than UniWAM-Sim’s.Source: §4.3 · p. 15, Table 8 · p. 17

4.4 Comparison of Unified Architectures

  • With joint sampling, Mixed-Stream achieves 18.6 cm and 5.87° docking errors and 60.0% success, improving on the other architectures’ errors while tying Sequence-Concatenated Streams in success.Independent sampling further reduces errors to 17.1 cm and 5.30° and raises success to 63.1%, the highest among the five variants. Mixed-Stream keeps separate visual inputs, action encoders, and output heads in separate batch rows while sharing the video and action DiTs.Source: §4.4 · pp. 17–18, Table 10 · p. 18

4.5 Ablation Studies

  • Replacing relative-pose targets with velocity commands or removing the causal mask raises Overall MAP-Bench errors and worsens long-range docking.With velocity commands, Overall errors rise from 12.5 cm and 3.24° to 16.3 cm and 4.17°, while long-range position error rises from 21.0 to 30.9 cm. Removing the causal mask raises Overall errors to 16.0 cm and 4.07° and long-range position error from 21.0 to 29.0 cm; the mask prevents later noisy actions from affecting the clean committed prefix.Source: §4.5 · p. 18, Table 11 · p. 19
  • Removing navigation auxiliary targets worsens docking, removing manipulation auxiliary supervision lowers manipulation success, and removing independent sampling worsens all six real-robot metrics.The target box has its largest effect at long range, while projected MAP has a larger effect at short range; removing both raises Overall errors to 16.7 cm and 5.37°. Removing navigation supervision changes MAP Navigation from 15.1 cm and 4.91° to 17.6 cm and 6.84°, while removing manipulation supervision lowers Manipulation success from 44.3% to 39.3%; removing independent sampling raises MAP Navigation errors to 16.8 cm and 5.62° and lowers Mobile Manipulation success to 60.0%.Source: §4.5 · pp. 18–19, Table 11 · p. 19, Table 12 · p. 19

4.6 Qualitative Results

  • On 24 selected MAP-Bench scenes, both UniWAM variants usually follow the reference path and end at the MAP, including long-range scenes where other methods deviate or fail more often.The scenes begin 1.6–4.2 m from the MAP; the last two rows show episodes starting at least 3.5 m away.Source: §4.6 · p. 19, Figure 7 · p. 21

4.5 Ablation Studies

  • 55.2% versus 48.5%: camera-frame end-effector targets raise the unweighted mean success rate over robot-base targets in the reported co-training groups.The comparison covers the reported embodiments and task groups, not transfer to unseen robots.Source: §4.5 · p. 19, Table 14 · p. 22

4.7 Failure Cases

  • During manipulation, the navigation stream can predict small nonzero base motions, which the near-zero rule suppresses for all methods.Source: §4.7 · p. 22
  • When adjacent objects have demonstrated grasps in two nearby action modes, UniWAM’s deterministic ten-step sampler can choose the gap between them instead of either object.The reported color-sorting failure may be reduced by specifying the target object in the instruction.Source: Figure 10 · p. 22, §4.7 · p. 22

4.6 Qualitative Results

  • Across categories, scenes, and viewing distances, predicted target boxes remain tightly localized and stable, while projected MAP points mark where the base should stop.The boxes and MAP coordinates are predicted jointly with actions at each approach step.Source: §4.6 · pp. 19, 22, Figure 8 · p. 23
  • Across two arms, several tasks, and two embodiments, executed gripper positions lie on predicted 2D end-effector tracks, aligning pixel locations with camera-frame actions.Each predicted curve corresponds to an arm’s prior action chunk; circles mark positions after execution.Source: §4.6 · p. 22, Figure 9 · p. 23

5 Conclusion

UniWAM combines independently sampled navigation and manipulation streams with explicit MAP supervision from automatically generated data. It reports improved benchmark errors and strong real-robot docking and mobile-manipulation results, while large-scale real-robot pretraining remains future work.

5 Conclusion

  • UniWAM treats navigation and manipulation as independently sampled streams over shared video and action DiTs, with image-plane auxiliary supervision and over 1.5 million MAP-Data episodes.Source: §5 · p. 23
  • On MAP-Bench, MAP-Data reduces position error by 30.1% and heading error by 44.0% relative to the strongest baselines.Source: §5 · pp. 23–24
  • Across 24 real-robot tasks and four embodiments, one model achieves the most accurate docking and highest mobile-manipulation success while remaining competitive in manipulation.Source: §5 · p. 24
  • Large-scale real-robot pretraining and studying how the mixed-stream design scales with robot data remain future work.Source: §5 · p. 24

Related papers

Loading 2609.39388v1…