Source-linked AI summary
Beyond Relative Geometry: Metric-Aware Geometry Perception for Robotics
Fengjun Zhong, Congjia Chen, Zhaoxu Liu, Jinyang Du, Yuchen Gong, Enqi Mao, Ruihao Gong, ShuJie Wang, Xianglong Liu, Zhongliang Qiao
TL;DR
Existing reconstruction models recover relative geometry with inconsistent physical scale, weakening alignment between geometric perception and robotic action. MAGP uses metric-conditioned, scale-equivariant reconstruction with flexible sensing inputs and policy integration. Across reconstruction and manipulation benchmarks, it preserves relative accuracy, reduces absolute error, and improves robotic-policy performance.
Problem
Existing reconstruction methods produce relative geometry with view- and scene-dependent scales, preventing consistent alignment between spatial perception and real-world robotic actions.
Method
MAGP is an end-to-end plug-and-play framework that uses metric scale equivariant augmentation, flexible metric conditioning, and geometry tokens to integrate metric reconstruction with embodied policies.
Results
MAGP consistently improves robotic policies across LIBERO, RoboTwin, and zero-shot LIBERO-Plus, with gains up to 6.26% on RoboTwin and absolute error reduced from 2.01m to 0.07m.
Takeaways & Limitations
Metric geometry provides a scale-consistent representation that supports robotic manipulation across heterogeneous sensing configurations.
Takeaways & Limitations
Metric conditioning assumes at least one metric multi-view baseline or one valid depth observation, and the evaluation uses a normalization factor of µ = 10.
Abstract
from arXiv · showhide
Recent embodied models increasingly leverage geometric representations to improve spatial reasoning and robotic manipulation. However, existing reconstruction methods only reconstruct relative geometry with arbitrary scales, causing predicted object dimensions and spatial distances to vary across scenes, viewpoints, and input configurations. This inconsistency prevents geometric perception from being directly aligned with robotic actions defined on the real-world scale. To address this limitation, we propose Metric-Aware Geometry Perception (MAGP), an end-to-end, plug-and-play framework for metric geometry reconstruction that can be seamlessly integrated into robotic policies. At its core, Metric Scale Equivariant Augmentation encourages the model to reconstruct metric geometry from camera parameters and depth observations, ensuring that the reconstructed geometry follows the metric scale specified by observations. Flexible Metric Conditioning further enables MAGP to support arbitrary view counts and combinations of camera and depth inputs, improving robustness to heterogeneous robotic sensing configurations. Together, these designs produce geometrically consistent reconstructions with stable object dimensions and spatial distances across scenes and sensing conditions. Experiments on ETH3D, MegaDepth, and ScanNet++ demonstrate that MAGP maintains strong relative geometry accuracy while reducing the absolute error by over an order of magnitude, from 2.01m to 0.07m. When integrated into multiple robotic policies, MAGP consistently improves performance on LIBERO, RoboTwin, and zero-shot LIBERO-Plus, with gains of up to 6.26% on RoboTwin. These results demonstrate the effectiveness and generalizability of metric geometry for robotic manipulation.
1 Introduction
Existing reconstruction provides relative geometry but lacks a consistent metric scale, limiting alignment between perception and robotic action. MAGP addresses this gap with metric-conditioned reconstruction and improves both geometry and manipulation results.
- Motivation: Relative reconstruction preserves scene structure but produces view-dependent object dimensions and spatial distances.These variations weaken consistency across viewpoints and input configurations.
- Motivation: Inconsistent geometry disrupts shared-coordinate action, causing inaccurate target localization and miscalibrated end-effector motion.Metric geometry instead preserves object dimensions and spatial relationships across views.
- MAGP: MAGP jointly encodes images with optional camera parameters and depth into metric geometry tokens for embodied policies.Metric Scale Equivariant Augmentation grounds scale in metric observations, while Flexible Metric Conditioning supports arbitrary view counts and input combinations.
- MAGP: MAGP combines metric and relative supervision to preserve fine-grained geometry while enforcing a consistent metric scale.This design supports metric reconstruction whenever relevant metric observations are available.
- Results: 84.81% and 87.34% are the depth and point scores achieved with camera parameters and depth observations, alongside absolute errors of 0.11m and 0.06m.In image-only input, the corresponding scores are 74.23% and 84.27%.
- Results: 6.26% is MAGP’s largest reported robotic-policy gain, obtained on RoboTwin, with additional gains on LIBERO and zero-shot LIBERO-Plus.The reported improvements are up to 1.3% on LIBERO and 3.9% on zero-shot LIBERO-Plus.
2 Related Work
Recent feed-forward reconstruction and geometry-aware manipulation models incorporate 3D structure into embodied systems, but their geometric representations remain relative rather than metric.
- 3D Geometry Reconstruction: Feed-forward reconstruction has replaced many optimization-based pipelines with direct inference of scene geometry and camera parameters.DUSt3R and MASt3R use pairwise pointmap regression, while VGGT, CUT3R, and π3 extend unified multi-view modeling.
- Geometry-Aware Robotic Manipulation: Geometry-aware manipulation methods inject 3D observations through visual alignment, depth representations, plug-in features, or dual-stream architectures.Representative systems include 3DS-VLA, DepthVLA, VGGT-DP, and ABot-M0.
- Geometry-Aware Robotic Manipulation: Existing geometry-aware robotic methods improve manipulation performance but capture only relative structure.MAGP differs by recovering metric geometry and directly connecting it to robotic policies.
3 Challenge of Metric Scale 3D Perception
Visual observations leave scene scale ambiguous, so conventional reconstruction predicts geometry up to an unknown factor. Metric reconstruction instead grounds scale in camera translation and depth and requires scale-equivariant behavior.
- Scale Ambiguity: Scenes with different physical dimensions and camera distances can produce indistinguishable images, creating an inherent scale ambiguity.Existing models therefore recover relative geometry whose scale varies with views, camera motion, and visual content.
- Scale Ambiguity: Relative geometry can indicate where to act but cannot reliably determine physical displacement for end-effectors, grippers, or trajectories.This limits its precision for robotic manipulation.
- Metric Evidence: Camera translation supplies a global metric baseline, while depth observations provide local physical-distance anchors along camera rays.Together with camera geometry and correspondences, these observations resolve scene-scale ambiguity.
- Metric Evidence: Metric supervision alone is insufficient because models may ignore camera translation and depth and infer scale from appearance, categories, and scene layout.This produces metric-scale-agnostic predictions that remain unchanged when metric observations are scaled.
- Metric Scale Equivariance: Metric scale equivariance requires scaling camera translation and depth by α > 0 to proportionally scale the reconstructed geometry.This formulation explicitly ties reconstruction scale to the available metric observations.
4 Metric-Aware 3D Perception for Robotics
MAGP is a unified metric-geometry framework that combines flexible sensing, scale-equivariant training, joint supervision, and residual action grounding. It reconstructs scale-consistent scene geometry from heterogeneous visual, camera, and depth inputs for robotic policies.
- Architecture Design: MAGP uses a single transformer to jointly encode multi-view images, optional camera parameters, and depth observations into metric geometry tokens.The unified backbone models view-specific structure and cross-view correspondences while accommodating variable numbers of views.
- Architecture Design: Camera parameters and depth observations are encoded as auxiliary tokens, with missing observations represented by shared learnable tokens.Camera tokens are concatenated with image tokens, while depth tokens are added to image tokens.
- Architecture Design: MAGP applies scene-independent normalization to metric inputs, preserving metric relationships while placing quantities in a stable numerical range.Unlike scenewise normalization, this rescaling does not introduce scene-dependent scale distortion.
- Metric to Action Grounding: Residual cross-attention injects object dimensions, spatial distances, and scene geometry from MAGP tokens into visual-language features for action prediction.Visual-language tokens act as queries, while metric geometry tokens provide keys and values.
- Robust Metric-Aware Geometry Optimization: Metric Scale Equivariant Augmentation jointly rescales camera translation, depth observations, and geometric supervision so identical images correspond to different physical scales.This discourages visual appearance from determining scale and encourages reliance on camera translation and depth constraints.
- Robust Metric-Aware Geometry Optimization: Flexible Metric Conditioning independently varies camera and depth availability and density, unifying camera-conditioned, depth-conditioned, jointly conditioned, and partially observed settings.The model uses available metric observations and complements sparse or missing inputs with remaining observations and multi-view constraints.
- Robust Metric-Aware Geometry Optimization: Joint relative and metric objectives preserve fine-grained geometry and cross-view consistency while supervising reconstruction in metric scale.Metric supervision is applied when a metric multi-view baseline or at least one depth observation provides sufficient scale evidence.
5 Experiments
MAGP is evaluated on 3D reconstruction and robotic manipulation benchmarks, including ablations of metric augmentation, objectives, and normalization. It improves relative and metric reconstruction while consistently benefiting manipulation performance.
- 3D reconstruction: MAGP consistently outperforms VGGT, MapAnything, Depth Anything 3, and VGGT-Ω across observation settings.
- 3D reconstruction: 0.11m and 0.06m absolute errors for depth and points are achieved when camera parameters and depth observations are both provided.
- 3D reconstruction: MAGP maintains consistent metric scale across independently reconstructed subsets, enabling coherent fusion while DA3 produces subset-dependent scales and fragmented geometry.
- Ablations: MSEA enables scale responsiveness to metric observations, while removing it causes substantial degradation and weak metric responsiveness.
- Ablations: Joint relative and metric supervision yields the best overall performance because relative supervision preserves structure while metric supervision anchors scale.
- Robotic manipulation: 98.1% and 98.5% average success rates are achieved on LIBERO for StarVLA-π and GR00T, compared with 96.8% and 97.5%.
- Robotic manipulation: 87.95% and 93.91% success rates are achieved on RoboTwin for GR00T and DM0.5, compared with 81.69% and 92.12%.
6 Conclusion
MAGP is presented as an end-to-end, plug-and-play framework for metric geometry reconstruction in downstream robotic policies. Its training data spans diverse real-world and synthetic environments and sensing configurations.
- MAGP reconstructs metric geometry for downstream robotic policies as an end-to-end, plug-and-play framework.
- Training data covers indoor and outdoor environments, static and dynamic scenes, and monocular, stereo, and multi-view configurations.
A.2 Implementation Details
MAGP is initialized from DA3-Giant and trained with a multi-dataset, two-stage view curriculum. Its lightweight depth encoder adds minimal model complexity.
- MAGP is initialized from pretrained DA3-Giant and trained using a two-stage view curriculum on mixed multi-view reconstruction datasets.
- 0.001B additional parameters are introduced by MAGP through a lightweight depth encoder.
A.4 Qualitative Results
MAGP qualitatively reconstructs diverse indoor and outdoor scenes with detailed geometry and consistent metric scale. Its predictions closely match ground truth without post-hoc scale alignment.
- MAGP predictions closely match ground truth in scene extent, object dimensions, spatial distances, fine structures, and complex layouts.
- Metric-scale agreement is achieved without post-hoc scale alignment, indicating recovery of absolute scene size alongside relative geometry.
B.1 Evaluation Benchmarks and Protocols
The evaluation spans standard manipulation suites, diverse dual-arm tasks, and zero-shot distribution shifts, using repeated rollouts or large-scale variant testing to measure success rates.
- Evaluation metric: Performance is reported using average task success rates across the benchmark evaluations.The protocols combine repeated independent rollouts for standard benchmarks with single evaluations of large-scale LIBERO-Plus variants.
- RoboTwin: RoboTwin evaluates 50 diverse dual-arm manipulation tasks, including transfer, tool use, stacking, articulated-object manipulation, and coordinated operation.Each task is tested over 100 independent rollouts in both clean and randomized settings, totaling 10,000 episodes.
- LIBERO-Plus: LIBERO-Plus tests zero-shot robustness without fine-tuning across 10,030 variants perturbed along seven dimensions.Variants change layouts, viewpoints, robot states, instructions, lighting, backgrounds, and sensor noise, with each variant evaluated once.
B.2 Implementation Details
Policies are trained with large-scale GPU configurations and standard optimization settings, while estimated depth substitutes for unavailable metric sensing data.
- Training setup: LIBERO and RoboTwin policies are trained for 100K and 150K iterations using global batch sizes of 64 and 32, respectively.Training uses 16 H100 GPUs for LIBERO and 32 H100 GPUs for RoboTwin; StarVLA-π and GR00T train from scratch, while DM0.5 is fine-tuned from pretrained weights.
- Metric observations: MoGe-estimated depth from head-camera images is used as the depth observation because the original training data lack metric camera and depth measurements.Optimization uses AdamW with cosine learning-rate decay, and LIBERO-Plus receives no training or fine-tuning.
- Optimization: The optimization configuration uses AdamW with β1 = 0.9, β2 = 0.95, ϵ = 1 × 10−8, weight decay = 1 × 10−8, and an initial learning rate of 1 × 10−5.The learning rate follows cosine decay throughout training.
B.3 Evaluation
MAGP-supported policies show successful manipulation across standard suites, diverse randomized dual-arm tasks, and zero-shot distribution shifts, while qualitative reconstructions preserve metric scene structure.
- LIBERO: LIBERO rollouts show accurate target localization, appropriate motion amplitudes, and successful multistage interactions across all four suites.The suites span spatial relations, object-centric generalization, goal-conditioned manipulation, and long-horizon execution.
- RoboTwin: MAGP enables successful RoboTwin execution in clean and randomized environments with varied object placements, backgrounds, lighting, and scene configurations.The randomized conditions are designed to test robustness under environmental distribution shifts.
- LIBERO-Plus: Zero-shot LIBERO-Plus rollouts retain manipulation capability under substantial changes in visual appearance, object layout, and robot initialization.The learned geometric representation remains reliable beyond the training distribution, particularly under shifts affecting embodiment states and spatial relationships.
- Cross-benchmark result: MAGP is broadly compatible with different robotic policies and manipulation settings across LIBERO, RoboTwin, and zero-shot LIBERO-Plus.The conclusion is based on consistent improvements across all three evaluation settings.
- Geometry reconstruction: Metric geometry reconstructions match ground-truth scene dimensions and spatial extent while preserving fine-grained geometric structures across indoor and outdoor scenes.Identically sized red dashed lines at corresponding structures provide a visual reference for metric consistency.