Source-linked AI summary
RoboBrain 2.5: Depth in Sight, Time in Mind
Huajie Tan, Enshen Zhou, Zhiyu Li, Yijie Xu, Yuheng Ji, Xiansheng Chen, Cheng Chi, Pengwei Wang, Huizhu Jia, Yulong Ao, Mingyu Cao, Sixiang Chen, Zhe Li, Mengzhen Liu, Zixiao Wang, Shanyu Rong, Yaoxu Lyu, Zhongxia Zhao, Peterson Co, Yibo Li, Yi Han, Shaoxuan Xie, Guocai Yao, Songjing Wang, Leiduo Zhang, Xi Yang, Yance Jiao, Donghai Shi, Kunchang Xie, Shaokai Nie, Chunlei Men, Yonghua Lin, Zhongyuan Wang, Tiejun Huang, Shanghang Zhang
TL;DR
Embodied AI models lack metric-grounded spatial precision and dense temporal supervision for reliable physical interaction. RoboBrain 2.5 addresses these gaps with depth-aware 3D manipulation traces and step-aware temporal value estimation, with reported state-of-the-art benchmark performance and stronger zero-shot robustness in contact-rich tasks.
Problem
Embodied AI models often lack metric-grounded spatial precision and dense temporal supervision needed for reliable physical interaction.
Method
RoboBrain 2.5 adds depth-aware coordinate prediction with ordered 3D manipulation traces and visual dense temporal value estimation trained with hop-based supervision.
Results
RoboBrain 2.5 reports state-of-the-art performance on serious benchmarks and superior zero-shot robustness in contact-rich real-world tasks.
Takeaways & Limitations
The framework combines metric-grounded spatial reasoning with execution-aware temporal feedback for more physically grounded embodied intelligence.
Takeaways & Limitations
Naive multi-perspective averaging risks out-of-distribution hallucination and reward hacking during online reinforcement learning, motivating bi-directional consistency checking.
Abstract
from arXiv · showhide
We introduce RoboBrain 2.5, a next-generation embodied AI foundation model that advances general perception, spatial reasoning, and temporal modeling through extensive training on high-quality spatiotemporal supervision. Building upon its predecessor, RoboBrain 2.5 introduces two major capability upgrades. Specifically, it unlocks Precise 3D Spatial Reasoning by shifting from 2D pixel-relative grounding to depth-aware coordinate prediction and absolute metric constraint comprehension, generating complete 3D manipulation traces as ordered keypoint sequences under physical constraints. Complementing this spatial precision, the model establishes Dense Temporal Value Estimation that provides dense, step-aware progress prediction and execution state understanding across varying viewpoints, producing stable feedback signals for downstream learning. Together, these upgrades extend the framework toward more physically grounded and execution-aware embodied intelligence for complex, fine-grained manipulation. The code and checkpoints are available at project website: https://superrobobrain.github.io
1 Introduction
RoboBrain 2.5 addresses reliability gaps in embodied AI by adding precise 3D spatial reasoning and dense temporal value estimation for physically grounded, closed-loop manipulation.
- Motivation: Embodied models often translate language and visual observations into actionable plans but falter during rigorous real-world deployment.Real-world interaction requires physical grounding under metric constraints, occlusions, viewpoint shifts, and continuous self-correction.
- Motivation: 2D or topological grounding lacks absolute depth and scale, preventing guaranteed millimeter-level clearance and collision-free 3D trajectories.These limitations undermine physical compliance during precise interaction.
- Motivation: Open-loop action prediction with sparse success labels cannot monitor intermediate failures such as slippage or regression, making adaptive recovery impossible in long-horizon tasks.The proposed paradigm shift requires dense temporal feedback for closed-loop reliability.
- Depth in Sight: RoboBrain 2.5 extends spatial grounding to depth-aware coordinates and ordered keypoint sequences describing complete manipulation procedures.Its spatial curriculum covers 3D referring, metric measuring, and trace generation under physical constraints.
- Time in Mind: Dense Temporal Value Estimation predicts execution state and step-aware progress from visual observations across viewpoints.Multi-view trajectory modeling and hop-normalized labels produce stable feedback signals for downstream reinforcement learning.
- Synergy and Impact: RoboBrain 2.5 combines metric-grounded spatial precision with execution-aware temporal feedback while retaining general interactive reasoning.The introduction reports state-of-the-art benchmark performance and superior zero-shot robustness in contact-rich real-world tasks.
2 New Feature
The paper develops RoboBrain 2.5 through metric-grounded 3D tracing and dense temporal progress estimation, using structured supervision and multi-perspective feedback to support robust manipulation.
- New capabilities: RoboBrain 2.5 builds on RoboBrain 2.0 and Qwen3-VL with Precise 3D Spatial Reasoning and Dense Temporal Value Estimation.Both capabilities use RGB-based spatial or multi-view temporal observations.
- Precise 3D Spatial Reasoning: Precise 3D Spatial Reasoning represents manipulation as an ordered sequence of points containing image coordinates and absolute depth.The resulting trace acts as a spatial plan and supports referring, measuring, and multi-step tracing under constraints.
- Precise 3D Spatial Reasoning: The decoupled trace representation can omit depth for 2D traces or retain endpoints for referring data, enabling reuse with existing 2D datasets.The formulation supports co-training across complementary tasks and modalities.
- Dense Temporal Value Estimation: Dense Temporal Value Estimation infers fine-grained real-time task progress from visual inputs using segmented trajectories and hop-based labels.The pipeline includes multi-view trajectory segmentation, adaptive sampling, and systematic hop labeling.
- Hop-wise Progress Construction: Global progress is defined over sampled synchronous multi-view states, while hop labels normalize transitions relative to the task span.The hop formulation handles forward progress and regression with separate normalization terms.
- Hop-wise Progress Construction: Hop-based supervision scales changes into [-1, 1] and keeps iteratively reconstructed progress within [0, 1].Sampling balances hop and temporal-distance bins, with additional zero-hop examples reducing bias toward static segments.
- Multi-Perspective Progress Fusion: Multi-perspective fusion combines incremental, forward-anchored, and backward-anchored estimates to balance local precision, initial stability, and goal sensitivity.Incremental estimates accumulate predictions, whereas the anchored views reduce long-trajectory drift and improve sensitivity near completion.
- Bi-directional Consistency Checking: A Gaussian-kernel confidence weight is derived from normalized discrepancies between progress estimates to support reliability-aware temporal feedback.Discrepancies receive greater penalty early in execution, when precise guidance is emphasized.
3 Training Data
RoboBrain 2.5 is trained on a unified, multimodal corpus spanning general perception, spatial reasoning, planning, and temporal value estimation. The data includes metric-grounded 3D supervision and diverse robotic task annotations to support embodied reasoning.
- The unified corpus contains approximately 12.4M samples across General MLLM, Spatial Reasoning, and Temporal Prediction domains.
- Spatial Reasoning Data: The 3D Spatial Reasoning Dataset contains 1.74M samples and 8.08M QA pairs for metric-grounded referring, measuring, and tracing.It supports outputs in units such as centimeters, inches, and meters, using 3D scans and video sources.
- Temporal Prediction Data: ShareRobot Planning contributes 1M QA pairs from 51K instances spanning 102 scenes, 12 robot embodiments, and 107 atomic tasks.
- Temporal Prediction Data: AgiBot Planning provides 9,148 QA pairs across 19 manipulation tasks and 109,378 first-person images with consecutive-frame task progression.
- Temporal Prediction Data: Multi-Robot Planning generates 44,142 samples covering 1,659 collaborative task types across household, supermarket, and restaurant environments.
- Temporal Prediction Data: Dense Value Estimation uses approximately 35M value-estimation samples derived from over 27M raw frames before down-sampling to 3.5M.
4 Training Strategy
RoboBrain 2.5 uses progressive dual-phase training to move from broad multimodal perception and qualitative spatial reasoning toward metric-aware 3D tracing and dense temporal value estimation. The strategy introduces embodied supervision gradually while retaining general interaction and planning abilities.
- Training proceeds in two phases: Foundational Spatiotemporal Learning followed by Specific Spatiotemporal Enhancement.
- Foundational Spatiotemporal Learning: The first phase uses 8.3M samples to establish multimodal semantics, 2D grounding, qualitative 3D understanding, planning, and temporal prediction.
- Specific Spatiotemporal Enhancement: The second phase uses approximately 4.1M samples focused on Metric 3D Spatial Reasoning and Dense Value Estimation.
- Specific Spatiotemporal Enhancement: The training design preserves conversation, 2D grounding, and logical planning abilities while adding fine-grained physical skills for 3D embodied environments.
5 Infrastructure
The infrastructure combines heterogeneous pipeline parallelism, dynamic-shape memory pre-allocation, and cross-accelerator training to support efficient large-scale multimodal training. Reported results include improved pipeline efficiency, stable memory behavior, and close convergence across accelerator platforms.
- Uneven pipeline parallelism places the ViT module at the front and reduces language layers in the first stage to balance computation and mitigate pipeline bubbles.
- Dynamic pre-Allocated Memory: Dynamic sequence lengths create GPU memory fragmentation and can cause out-of-memory failures under the default CUDA caching allocator.
- Dynamic pre-Allocated Memory: Pre-allocating memory at initialization and reusing it across iterations balances memory efficiency, training stability, and throughput.Cache cleanup and re-padding occur only when visual token length exceeds the current maximum.
- Cross-Accelerator Training and Inference: End-to-end training on 1,000 non-NVIDIA accelerators achieved a final convergence gap within 0.62% of NVIDIA-platform behavior.The resulting checkpoints were migrated to NVIDIA platforms, where benchmark performance remained highly consistent.
6 Evaluation Results
RoboBrain-2.5 is evaluated across 2D and 3D spatial reasoning and temporal value estimation benchmarks, with strong results across both NVIDIA and Moore-Threads variants. The evaluations emphasize metric-grounded tracing, cross-view correspondence, and forward/reverse temporal ordering robustness.
- 2D Spatial Reasoning: 94.58 is RoboBrain-2.5’s best CV-Bench accuracy on NVIDIA, while its MTT variant reaches 93.90.Both variants outperform the listed general-purpose and embodied baselines.
- 2D Spatial Reasoning: 76.30 (MTT) and 75.40 (NVIDIA) on CrossPoint substantially surpass all evaluated baselines.CrossPoint measures fine-grained point correspondence across different viewpoints.
- 3D Spatial Reasoning: 64.17 (NVIDIA) on MSMU is the best reported result, while 78.31 (MTT) on Q-Spatial exceeds most listed baselines.These benchmarks evaluate quantitative 3D measuring, object sizes, and distances.
- 3D Spatial Reasoning: TraceSpatial evaluates multi-step metric-grounded traces requiring correct start and end constraints and collision-free trajectories.Table 3 reports 3D Start, 3D End, and Success as fine-grained trace metrics.
- Temporal Value Estimation: Temporal value estimation is evaluated with VOC+ and VOC− across six real-robot, simulation, and human egocentric video sources.VOC− is computed after reversing the video, testing whether predicted value consistently inverts with temporal order.
- Temporal Value Estimation: On DROID, RoboBrain-2.5 (MTT) achieves 93.67 / 89.26, substantially improving over all baselines.The paired values are Forward VOC / Reverse VOC, respectively.
7 Conclusion and Future Works
The conclusion presents RoboBrain-2.5 as an embodied foundation model that combines metric-grounded spatial reasoning with dense temporal feedback. Future work targets unified world-modeling, deployment on mobile and humanoid platforms, and scalable model variants.
- Conclusion: RoboBrain-2.5 combines depth-aware coordinate prediction and collision-free manipulation traces with dense, step-aware progress and regression feedback.The spatial component uses decoupled (u, v, d) representation, while the temporal component uses hop-based labeling and multi-perspective fusion.
- Conclusion: The model is intended to bridge high-level semantic reasoning and low-level physical interaction through metric-grounded spatial precision and dense temporal supervision.The conclusion identifies these as the two core pillars of the upgrade.
- Future Works: Future work will integrate spatiotemporal understanding with image and video prediction to form an embodied world model.The stated goal is to simulate action outcomes before execution for safer and more robust planning.
- Future Works: Future work will validate the models on mobile manipulators and humanoid robots, using spatial reasoning for manipulation generalization and temporal estimation for physical-world reinforcement learning.The proposal specifically targets training-free manipulation generalization and reward signals for efficient RL.
- Future Works: The authors plan model variants at different parameter scales to accommodate computational and latency constraints.The passage specifically mentions lightweight versions optimized for edge deployment.
A Qualitative examples
The qualitative examples focus specifically on RoboBrain 2.5’s 3D spatial reasoning and temporal value estimation. They illustrate these capabilities across various embodied AI tasks and their potential for real-world applications.
- Qualitative Examples: The examples demonstrate 3D spatial reasoning and temporal value estimation rather than repeating capabilities already shown in RoboBrain 2.0.Pointing, affordance, and planning examples are described as similar to those in the predecessor.
A.1 Examples on 3D Spatial Reasoning
RoboBrain 2.5 generates metric-aware 3D manipulation traces for constrained instructions and demonstrates qualitative generalization across environments, objects, and dual-arm tasks.
- Compliance with Fine-Grained Spatial Constraints: RoboBrain 2.5 generates 3D traces that satisfy relative positional and metric constraints in fine-grained manipulation instructions.Examples include ordinal object references, relative placement, and obstacle-aware movement.
- Multi-Step Compositional Reasoning: Ordered keypoint sequences decompose complex manipulation goals into localization, metric estimation, and obstacle-avoiding 3D movement.The described task requires locating an object, estimating a target edge and wall distance, then connecting them with a smooth trace.
- Generalization Across Environments and Objects: Metric-grounded spatial representations support precise traces across kitchen, bedroom, office, and study environments and across varied object types.The model adapts traces to object-specific affordances and differing sizes, shapes, and functions.
- RoboTwin 2.0: On RoboTwin 2.0 AgiLex dual-arm tasks, predicted traces handle distractors, arm-specific instructions, and precise placement goals.Examples span clicking, ranking, handover, hanging, and moving objects.
- TraceSpatial-Bench: TraceSpatial-Bench visualizations compare ground-truth starts and endpoints with 2D projections of predicted 3D traces.The examples visualize predicted traces against red starting-point masks and purple endpoint boxes.
A.2 Examples on Temporal Value Estimation
RoboBrain 2.5 estimates dense progress and instantaneous change across diverse manipulation tasks, with consistency across sampling intervals and recovery-relevant feedback during disturbed execution.
- Dense Value Predictions on Diverse Tasks: RoboBrain 2.5 predicts frame-wise Hop and accumulated Progress across deformable-object, unstructured real-world, and multi-stage simulation tasks.Examples include folding pants, cleaning a table, stacking bowls, and opening a drawer.
- Robustness to Temporal Intervals: Progress estimates remain consistent across sampling intervals of Δt = 10, 25, 50, and 100 frames despite larger visual differences between adjacent frames.The comparison evaluates robustness to temporal granularity and control frequency.
- Progress Estimation Modes: Incremental, forward-anchored, and backward-anchored modes provide complementary perspectives for estimating progress.Incremental prediction accumulates local hop values, while forward anchoring provides an early-execution reference.
- Real-World RL Rollout: During disturbed Insert Block execution, Progress drops sharply after target displacement and the robot subsequently recovers and completes insertion.The rollout used a policy with a success rate over 95%.
B Proof of Bounded Global Progress
The proof shows that recursively applying bounded predicted progress hops preserves reconstructed global progress within [0, 1] at every step.
- Recursive Update: The recursive update uses one rule for nonnegative hops and another for negative hops to update reconstructed global progress.For H ≥ 0, the update scales remaining progress; for H < 0, it scales current progress.
- Inductive Argument: Starting from Φ⋆(s0) = 0 with H ∈ [−1, 1], the induction assumes bounded prior progress and analyzes positive and negative hops separately.The proof establishes the base case at t = 0 before treating both update scenarios.
- Positive Hop: For positive hops, nonnegative terms and bounded prior progress preserve the lower and upper bounds on Φ⋆(st).The argument treats the update as a convex combination.
- Negative Hop: For negative hops, the factor 1 + H remains nonnegative and below one, preserving both bounds on reconstructed progress.This covers the regression case −1 ≤ H < 0.
- Conclusion: Therefore, Φ⋆(st) ∈ [0, 1] for all t.The conclusion follows because the property holds initially and is preserved by both update cases.