Source-linked AI summary
MAVP: Map-Aware Visuomotor Policies for Mobile Manipulation
Jinhe Tang, Ruixiao Dai, Weiming Zhi
TL;DR
Demonstration-trained mobile manipulation can struggle to realise intended base motion, while base positioning affects arm reachability and camera viewpoints. MAVP uses a shared map, predicts explicit base-pose targets, and tracks them with localisation feedback; map-frame pose control improves success across every evaluated task and policy family.
Problem
Demonstration-trained policies can struggle to realise intended base motion, and independently initialised coordinate frames can assign different coordinates to the same workspace pose.
Source: §I · pp. 1–2Method
MAVP reconstructs a shared map from demonstrations and jointly predicts map-frame base-pose targets and arm and gripper actions, with localisation-based feedback tracking.
Source: §I · p. 2Results
Map-frame pose control improves success across every evaluated task and policy family.
Source: F. Policy Architectures (Q7) · p. 7Takeaways & Limitations
Across real-world tasks and multiple policy families, results consistently support shared-map pose targets over unanchored base control and the value of feedback and continued localisation.
Source: §V · p. 8
Abstract
from arXiv · showhide
Successful mobile manipulation requires coordinated base and arm motion while maintaining accurate spatial positioning. However, demonstration-trained policies can struggle to realise the intended base motion reliably, leading to spatial misalignment and subsequent manipulation failures. We present MAVP (Map-Aware Visuomotor Policies), a framework that improves execution reliability by predicting explicit base-pose targets and tracking them using localisation feedback. MAVP reconstructs a static map from teleoperated demonstrations and expresses demonstrated base trajectories in a shared map frame, providing consistent spatial supervision across demonstrations. At execution time, the policy receives RGB observations, joint states, and the robot's current map-frame base pose, and jointly predicts target base poses, arm actions, and gripper actions. A low-level controller tracks the predicted base targets using feedforward motion and pose error feedback, enabling correction of execution deviations. We additionally use pose-noise augmentation during training to improve robustness to errors in the policy's pose input. Across six real-world manipulation tasks and three policy families, MAVP achieves higher task success rates than unanchored velocity control in all tasks. Videos and additional results are available at https://123qwedsa123.github.io/mavp/.
I. INTRODUCTION
MAVP addresses unreliable base motion in demonstration-trained mobile manipulation with shared-map pose targets and localisation-based feedback. The paper also evaluates this framework across real-world tasks, policy families, and system components.
I. INTRODUCTION
- A shared map gives demonstrations a consistent frame for base-pose supervision and target tracking, resolving ambiguity from independently initialised coordinates.Without explicit pose targets, incremental velocity prediction and tracking errors can accumulate into base-pose drift.Source: §I · pp. 1–2
- MAVP conditions on visual observations, joint states, and the current map-frame base pose, jointly predicting base-pose targets and arm and gripper actions.A feedforward-plus-feedback controller tracks base targets so execution deviations can be corrected without predicting each corrective velocity.Source: §I · p. 2
- Pose-noise augmentation perturbs the policy’s pose input during training while retaining the demonstrated action targets.The augmentation is intended to improve tolerance to imperfect localisation.Source: §I · p. 2
- The shared static map is reconstructed from teleoperated demonstrations after robot arms and dynamic regions are masked.It provides a common reference for aligning demonstrations and expressing base states and action targets across episodes.Source: §I · p. 2
- The paper evaluates MAVP’s properties on real-world tasks using different policy classes.Mobile manipulation requires coordinated base and arm motion because base position and orientation affect arm reachability and camera viewpoints.Source: §I · pp. 1–2
II. RELATED WORK
Prior work explores mobile-manipulation coordination and spatial representations; MAVP uses a shared scene map as a reference for learning and execution. Its map-construction pipeline masks robot arms and dynamic regions before reconstructing a static scene map.
II. RELATED WORK
- MAVP combines shared-map base-pose targets with localisation feedback and jointly predicts arm and gripper actions; unlike DAgger and DART, it addresses localisation uncertainty with pose-noise augmentation without corrective demonstrations.Related work also covers base-pose selection, whole-body control, spatial action representations, waypoint methods, and map-based geometric context –, –.Source: §II · p. 2
III. MAVP: MAP-AWARE VISUOMOTOR POLICIES
- MAVP reconstructs a shared static map from demonstrations and uses live observations with visual odometry to estimate the base pose in that map frame.The described setup assumes mapped scene geometry remains static and uses calibrated head-camera extrinsics to transform camera-pose estimates into map-frame base poses.Source: §III · p. 3
A. Shared Map Construction
- The map pipeline samples stereo images from demonstrations and masks robot-arm and dynamic-object regions before static-scene reconstruction.Robot-arm masks are projected using joint configuration, forward kinematics, and calibrated camera parameters; VGGT4D supplies dynamic-region masks.Source: A. Shared Map Construction · p. 3
- Figure 3 depicts shared-map reconstruction from stereo demonstrations after robot arms and dynamic regions are masked.The resulting reference camera poses, landmarks, and image features are used for visual localisation.Source: Figure 3 · p. 3, A. Shared Map Construction · p. 3
- Retained features are matched across frames, demonstrations, and stereo views; COLMAP verifies two-view geometry and reconstructs a sparse map of registered camera poses and triangulated 3D landmarks.Bundle adjustment refines camera poses and landmark positions while holding calibrated intrinsics and stereo geometry fixed.Source: A. Shared Map Construction · p. 3
B. Map-Based Localisation
MAVP estimates the head-camera pose in the shared map, aligns continuous visual odometry to that frame, and transforms the result into a map-frame base pose for policy input and feedback control.
B. Map-Based Localisation
- MAVP masks robot-arm regions in current stereo images and matches query features to map-associated 3D landmarks to localise the head camera.SuperPoint detects keypoints and LightGlue matches them; PnP with RANSAC estimates camera pose from 2D–3D matches subject to inlier, reprojection-error, and stereo-consistency thresholds.Source: B. Map-Based Localisation · p. 3
- Paired map-localised camera poses align the continuous cuVSLAM trajectory from its local tracking frame to the shared map frame.Subsequent map localisations update the alignment, with smoothing to limit pose jumps.Source: B. Map-Based Localisation · p. 4
- The aligned camera pose is transformed using camera-to-base geometry, and its planar position and yaw form the map-frame base pose used by policy and feedback control.The planar pose is (x, y, ψ).Source: B. Map-Based Localisation · p. 4
C. Map-Aware Visuomotor Policies Learning
MAVP uses a shared map frame for the policy’s base-pose input and base-related actions. It trains the policy from demonstrations and augments the pose input with localization-noise perturbations while keeping demonstrated actions unchanged.
C. Map-Aware Visuomotor Policies Learning
- MAVP conditions the policy on the current shared-map-frame base pose and predicts base-pose targets as part of its action sequence.Base motion is represented by target poses in the shared map frame, called map-frame pose control.Source: C. Map-Aware Visuomotor Policies… · p. 4
- The policy is trained by imitation learning to reproduce demonstrated actions from observed states using a policy-specific loss.The demonstration dataset pairs observations and base poses with action sequences.Source: C. Map-Aware Visuomotor Policies… · p. 4
- Training adds an independent Gaussian perturbation to each map-frame base-pose input while leaving the demonstrated action sequence unchanged.The perturbation covariance uses planar-position and yaw fluctuations estimated from localization noise.Source: C. Map-Aware Visuomotor Policies… · p. 4
- Pose-noise augmentation encourages the policy to produce consistent actions under small localization errors.Source: C. Map-Aware Visuomotor Policies… · p. 4
IV. EMPIRICAL EVALUATION
The evaluation covers six real-world mobile manipulation tasks and examines whether map-frame pose control and its components affect execution. It also asks how results vary across policy families and repeated executions.
IV. EMPIRICAL EVALUATION
- The evaluation tests MAVP on six real-world mobile manipulation tasks and examines mapping, policy, and control components.Its questions include the effect of ongoing map localization on repeated task execution.Source: §IV · p. 4
- The evaluation compares map-frame pose control with unanchored velocity control and episode-initialized odometry pose control, and compares pose targets with base velocities.Source: §IV · p. 4
- It tests whether explicitly providing the current map-frame base pose and adding pose-noise augmentation improve policy performance or tolerance to pose-input errors.Source: §IV · p. 4
- The evaluation asks how pose-error feedback affects task success and base-target tracking, and which map-construction components support complete maps and reliable held-out localization.Source: §IV · p. 4
- It also compares MAVP across visuomotor policy families and tests whether ongoing map localization improves repeated task execution.Source: §IV · p. 4
A. Common Experimental Setup
The setup uses a mobile manipulator with head-mounted and wrist cameras, and evaluates six tasks whose success requires completing every listed step. The policies are trained and evaluated with specified optimization settings and rollout counts.
A. Common Experimental Setup
- The robot has an omnidirectional base, a lifting and rotating torso, two 7-DoF arms, and independent grippers, with one head-mounted and two wrist RGB cameras.Source: A. Common Experimental Setup · p. 4, Figure 4 · p. 4
- The six tasks are Drawer Packing, Disassemble and Deliver, Lidded Box Packing, Conveyor Picking, Dual-Drawer Return, and Bag Packing; success requires completing every listed step.Source: A. Common Experimental Setup · p. 5, Table I · p. 5
- The study instantiates ACT, Diffusion Policy, and Flow Matching, trains policies for 30,000 optimization steps, and evaluates each policy with 50 rollouts per task.Each task has 50 teleoperated demonstrations; ACT and the DiT-based DP and FM implementations use different batch sizes, learning rates, and horizons.Source: A. Common Experimental Setup · p. 5
- Success rate is the percentage of successful rollouts, with a rollout succeeding only when every task step is completed.Source: A. Common Experimental Setup · p. 5, Table I · p. 5
B. Map-Frame Pose Control (Q1, Q2, and Q3)
Map-frame pose control improves task completion over velocity and episode-initialised odometry pose control, with the complete configuration performing best on both tasks. Controlled comparisons distinguish the benefits of spatial targets, explicit pose input, and pose-noise augmentation.
B. Map-Frame Pose Control (Q1, Q2, and Q3)
- Map-frame pose control improves task completion over unanchored velocity and episode-initialised odometry pose control, and the complete configuration performs best on both tasks.Trajectory examples contrast velocity-controlled runs that stop short with augmented map-frame pose runs that continue toward manipulation locations.Source: B. Map-Frame Pose Control (Q1, Q2, and… · pp. 5–6, Figure 6 · p. 5
- With the same map-frame pose input and no augmentation, map-frame pose targets outperform velocity commands on both tasks.Pose targets specify where the base should move, while velocity supervision reflects demonstration timing, including acceleration, pauses, and stopping.Source: Figure 5 · p. 5, B. Map-Frame Pose Control (Q1, Q2, and… · p. 6
- Adding the current map-frame base pose improves performance over the otherwise unchanged map-frame pose variant without that input.The pose input locates the base in the target frame, reducing the need to infer its location and heading from vision alone.Source: B. Map-Frame Pose Control (Q1, Q2, and… · p. 6
C. Robustness to Localisation Noise (Q4)
Pose-noise augmentation improves task success across every tested perturbation level on both tasks. Greater perturbations generally reduce reliability, while augmented policies show smaller changes in predicted base targets under offline pose perturbations.
C. Robustness to Localisation Noise (Q4)
- Pose-noise augmentation improves task success at every tested perturbation level on both tasks.The advantage is not confined to evaluation without added noise, although execution generally becomes less reliable as perturbations increase.Source: C. Robustness to Localisation Noise (Q4) · p. 6
- At every nonzero offline pose-noise level, the augmented policy shows smaller mean position and yaw changes in its base-target predictions.The comparison holds reconstructed RGB and proprioceptive observations fixed and measures each checkpoint against its own unperturbed prediction.Source: C. Robustness to Localisation Noise (Q4) · p. 6
D. Contribution of Pose-Error Feedback (Q5)
On Conveyor Picking, pose-error feedback improves task success and positional tracking, with little change in yaw tracking. The ablation holds the augmented policy checkpoint fixed, isolating the contribution of target execution.
D. Contribution of Pose-Error Feedback (Q5)
- With the augmented policy checkpoint fixed, adding pose-error feedback improves task success and positional tracking, while yaw tracking changes little.Feedforward follows target changes but does not correct an offset between the target and the localisation estimate; feedback corrects that offset.Source: D. Contribution of Pose-Error Feedback… · pp. 6–7, Figure 9 · p. 7
E. Geometric Map Construction Ablation (Q6)
The full mapping pipeline reconstructs complete maps for all six tasks. Temporal-only matching fails to produce a complete map, and other geometric ablations reduce deployable localisation queries.
E. Geometric Map Construction Ablation (Q6)
- The full mapping pipeline reconstructs complete maps for all six manipulation tasks, whereas temporal-only matching does not produce a complete map.Other ablations reduce deployable localisation queries; consistent stereo-pair counts can remain high when queries localise within the largest component of an incomplete reconstruction.Source: E. Geometric Map Construction Ablation… · p. 7, Table IV · p. 7
F. Policy Architectures (Q7)
Across ACT, DP, and FM, map-frame pose control improves success on every evaluated task, while ongoing map localisation improves success on both tested tasks. The conclusion highlights shared-map pose targets and localisation feedback, and identifies scalability, uncertainty handling, and adaptation across scenes and tasks as future directions.
F. Policy Architectures (Q7)
- Map-frame pose control improves task success across every evaluated task and all three policy families: ACT, DP, and FM.The comparison evaluates pose inputs, map-frame targets, and localisation feedback jointly.Source: F. Policy Architectures (Q7) · p. 7
G. Ongoing Map Localisation (Q8)
- The localisation comparison retains initial alignment across policy restarts but disables later map queries, whereas ongoing localisation updates alignment during execution.Local odometry and camera-to-base kinematics continue updating pose in the initial-only condition.Source: G. Ongoing Map Localisation (Q8) · pp. 7–8
- Ongoing map localisation achieves higher task success than initial alignment only on both evaluated tasks.Ongoing updates correct local-to-map alignment so policy inputs and controller feedback remain referenced to the target map frame.Source: Table VI · p. 8, G. Ongoing Map Localisation (Q8) · p. 8
V. CONCLUSION AND FUTURE WORK
- Future work targets more scalable mapping and localisation, uncertainty-aware policy and control, and adaptation to diverse scenes and task configurations.The conclusion also points to larger workspaces, longer-horizon behaviours, and more general whole-body mobile manipulation.Source: §V · p. 8