Source-linked AI summary
RoboBrain 2.0 Technical Report
BAAI RoboBrain Team, Mingyu Cao, Huajie Tan, Yuheng Ji, Xiansheng Chen, Minglan Lin, Zhiyu Li, Zhou Cao, Pengwei Wang, Enshen Zhou, Yi Han, Yingbo Tang, Xiangqi Xu, Wei Guo, Yaoxu Lyu, Yijie Xu, Jiayu Shi, Mengfei Du, Cheng Chi, Mengdi Zhao, Xiaoshuai Hao, Junkai Zhao, Xiaojie Zhang, Shanyu Rong, Huaihai Lyu, Zhengliang Cai, Yankai Fu, Ning Chen, Bolun Zhang, Lingfeng Zhang, Shuyi Zhang, Dong Liu, Xi Feng, Songjing Wang, Xiaodan Liu, Yance Jiao, Mengsi Lyu, Zhuo Chen, Chenrui He, Yulong Ao, Xue Sun, Zheqi He, Jingshu Zheng, Xi Yang, Donghai Shi, Kunchang Xie, Bochao Zhang, Shaokai Nie, Chunlei Men, Yonghua Lin, Zhongyuan Wang, Tiejun Huang, Shanghang Zhang
TL;DR
Embodied AI needs models that can connect visual understanding with reasoning and planning in physical environments. RoboBrain 2.0 addresses this with a modular vision-language architecture and progressive three-stage training, achieving strong results across spatial and temporal embodied benchmarks. The report positions the model as a foundation for more general embodied AI and releases its checkpoints, code, and benchmark.
Problem
Existing language and vision-language models show strong digital perception, understanding, and reasoning, but embodied AI requires capabilities grounded in physical environments.
Method
RoboBrain 2.0 combines a vision encoder with a decoder-only language model and trains them progressively through foundational spatiotemporal learning, embodied enhancement, and chain-of-thought reasoning.
Results
RoboBrain 2.0 demonstrates strong performance across a wide range of embodied tasks, with the 32B variant achieving leading results on spatial and temporal benchmarks.
Takeaways & Limitations
RoboBrain 2.0 provides a practical foundation for developing more general embodied AI systems by integrating perception, reasoning, and planning.
Takeaways & Limitations
Incorrect cases were identified in a test set, manually screened, and excluded.
Abstract
from arXiv · showhide
We introduce RoboBrain 2.0, our latest generation of embodied vision-language foundation models, designed to unify perception, reasoning, and planning for complex embodied tasks in physical environments. It comes in two variants: a lightweight 7B model and a full-scale 32B model, featuring a heterogeneous architecture with a vision encoder and a language model. Despite its compact size, RoboBrain 2.0 achieves strong performance across a wide spectrum of embodied reasoning tasks. On both spatial and temporal benchmarks, the 32B variant achieves leading results, surpassing prior open-source and proprietary models. In particular, it supports key real-world embodied AI capabilities, including spatial understanding (e.g., affordance prediction, spatial referring, trajectory forecasting) and temporal decision-making (e.g., closed-loop interaction, multi-agent long-horizon planning, and scene graph updating). This report details the model architecture, data construction, multi-stage training strategies, infrastructure and practical applications. We hope RoboBrain 2.0 advances embodied AI research and serves as a practical step toward building generalist embodied agents. The code, checkpoint and benchmark are available at https://superrobobrain.github.io.
1 Introduction
RoboBrain 2.0 addresses the gap between digital and physical intelligence by supporting embodied perception, reasoning, and planning. Its capabilities span spatial and temporal understanding, interactive reasoning, and scene reasoning for physical environments.
- RoboBrain 2.0 targets the gap between digital intelligence and physical intelligence in embodied AI.
- The model supports interactive reasoning with long-horizon planning and closed-loop feedback.
- Its spatial capabilities include precise point and bounding-box prediction from complex instructions.
- Its temporal and scene capabilities include future trajectory estimation and real-time scene-graph construction and updating.
- The report covers RoboBrain 2.0’s architecture, data construction, training strategies, infrastructure, and practical applications.
2 Architecture
RoboBrain 2.0 uses a modular encoder-decoder architecture that combines visual, textual, and structured scene-graph inputs in a unified language-model token stream. Its decoder performs reasoning and produces task-specific outputs such as plans, spatial relations, and coordinates.
- Core architecture: RoboBrain 2.0 combines a tokenizer, vision encoder, MLP projector, and Qwen2.5-VL-initialized language-model backbone.
- Inputs: The model accepts multi-image, long-video, and high-resolution visual inputs alongside complex instructions and structured scene graphs.
- Input fusion: Visual features are projected into the language model’s token space, while textual and structured inputs are tokenized into a unified stream.
- Visual processing: Dynamic-resolution processing, adaptive positional encoding, and windowed attention support high-resolution and multi-view observations.
- Spatiotemporal encoding: Frame-wise tokenization, multi-dimensional RoPE, and view-specific positional identifiers encode temporal and multi-view structure.
- Decoder outputs: The decoder supports concise responses and chain-of-thought reasoning, generating plans, spatial relations, coordinates, grounded expressions, and reasoning traces.
3 Training Data
RoboBrain 2.0 is trained on diverse multimodal, spatial, and temporal data designed for embodied perception, reasoning, and planning. The datasets include large-scale grounding resources, spatial reasoning samples, egocentric planning data, and multi-robot collaboration scenarios.
- Data organization: Training data is categorized into general multimodal understanding, spatial perception, and temporal modeling.
- General multimodal understanding: The general multimodal dataset contains 873K samples spanning standard VQA, region-level queries, OCR-based VQA, and visual dialogues.
- Spatial perception: Spatial datasets support localization, pointing, affordance recognition, and 3D spatial reasoning through curated images, coordinates, and object relations.
- Spatial perception: The Spatial Referring Dataset contains 802K samples targeting a single unambiguous target for precise robotic object identification and localization.
- Temporal modeling: Temporal data includes egocentric next-action prediction, frame-aligned manipulation planning, and long-horizon task planning across varied environments.
- Temporal modeling: Multi-Robot Planning defines 1,659 collaboration task types and produces 44,142 samples across household, supermarket, and restaurant environments.
4 Training Strategy
RoboBrain 2.0 uses progressive three-phase training that moves from foundational spatial and temporal understanding to embodied enhancement and chain-of-thought reasoning. The strategy increases supervision complexity toward dynamic reasoning and actionable planning.
- Overall strategy: The three-phase strategy progresses from foundational spatiotemporal learning through embodied enhancement to chain-of-thought reasoning.
- Stage 1: Foundational Spatiotemporal Learning: Stage 1 develops spatial perception and temporal understanding using dense captioning, localization, image-text documents, video QA, and referring expressions.
- Stage 1: Foundational Spatiotemporal Learning: Stage 1 establishes grounding for objects, spatial relations, motion events, egocentric video streams, and spatially anchored instructions.
- Stage 2: Embodied Enhancement: Stage 2 adds high-resolution, multi-view, and egocentric video data plus navigation and interaction tasks such as 3D affordance estimation and scene-graph construction.
- Stage 3: Chain-of-Thought Reasoning: Stage 3 applies CoT-SFT and reinforcement fine-tuning to long-horizon planning, manipulation prediction, closed-loop interaction, and multi-robot collaboration.
5 Infrastructures
RoboBrain 2.0’s infrastructure addresses the heterogeneous architecture and variable multimodal inputs that complicate distributed training. The system combines parallelism, memory, data-loading, fault-tolerance, post-training, and inference optimizations to improve efficiency and reliability.
- 5.1.1 Multi-Dimensional Hybrid Parallelism: Multimodal heterogeneity complicates distributed training because vision and language modules differ substantially and tokenized samples vary widely in length and modality.FlagScale targets these architectural and data characteristics with specialized distributed-training strategies.
- 5.1.1 Multi-Dimensional Hybrid Parallelism: Non-uniform pipeline parallelism reduces LLM layers in the first stage because the early ViT module has relatively low computational cost.A separate recompute strategy mitigates out-of-memory errors when vision inputs reach 20,000–30,000 tokens.
- 5.1.2 Data Processing and Loading: Preprocessing 320,000 samples takes less than 10 minutes when only JSON is preprocessed and image handling is deferred to TaskEncoder.This keeps image processing aligned with the model’s training preprocessor and avoids inconsistencies from separate image readers.
- 5.1.2 Data Processing and Loading: The optimized system reduces redundant distributed data loading by assigning loading responsibilities according to the data-parallel and hybrid-parallel configuration.Only the GPU processes that need data load their assigned shards, reducing I/O burden on compute nodes.
- 5.1.3 Fault Tolerance: Fault-tolerant mechanisms automatically detect common hardware and software failures, recover jobs, and resume data state from the latest checkpoint.The framework coordinates FlagScale with the platform, while its custom DataLoader supports full data-state recovery.
- 5.1.4 Post-Training and Inference: RLVR post-training uses VeRL’s hybrid-controller architecture, while FlagScale selects inference configurations and mixed-bit quantization targets efficient, accuracy-sensitive deployment.These infrastructure choices address complex post-training workflows and inference latency on heterogeneous accelerators.
6 Evaluation Results
RoboBrain-2.0 is evaluated on spatial and temporal embodied reasoning benchmarks, with both model variants showing strong results across multiple tasks. The 32B model achieves leading performance on many spatial and temporal benchmarks, while the 7B model remains competitive.
- Spatial Reasoning Capability: Evaluation covers nine spatial reasoning benchmarks spanning depth, visual-spatial integration, embodied spatial tasks, referring, affordance, and trajectory prediction.The benchmarks include BLINK, CV-Bench, EmbSpatial, RoboSpatial, RefSpatial-Bench, SAT, VSI-Bench, Where2Place, and ShareRobot-Bench.
- Spatial Reasoning Capability: 83.95 average on BLINK makes RoboBrain-7B-2.0 SOTA, while RoboBrain-32B-2.0 scores 83.63 and surpasses all baselines except the 7B variant.BLINK evaluates depth perception and spatial relation understanding.
- Spatial Reasoning Capability: 72.43 on RoboSpatial makes RoboBrain-32B-2.0 SOTA, while the 7B model scores 54.23 and exceeds all general baselines except the 32B model.RoboSpatial measures spatial reasoning in robot environments, including object localization and manipulation.
- Spatial Reasoning Capability: 54.00 for both Location and Placement makes RoboBrain-32B-2.0 SOTA on RefSpatial-Bench, while the 7B model records 36.00 and 29.00.The benchmark evaluates precise point predictions for spatial referring expressions.
- Temporal Reasoning Capability: 80.33 average on Multi-Robot Planning makes RoboBrain-32B-2.0 SOTA, while RoboBrain-7B-2.0 scores 81.50 and also outperforms all general baselines.The task evaluates coordination across supermarket, restaurant, and household scenarios.
- Temporal Reasoning Capability: 72.16 on RoboBench makes RoboBrain-7B-2.0 SOTA, while RoboBrain-32B-2.0 scores 68.33 and surpasses several general and embodied baselines.RoboBench evaluates planning for mobile manipulation across cross-embodiment, cross-object, and cross-view categories.
7 Conclusion and Future Works
RoboBrain 2.0 unifies perception, reasoning, and planning for complex physical-environment tasks and reports strong generalization across embodied benchmarks. Future work targets tighter VLA and robotics-system integration for broader practical deployment.
- RoboBrain 2.0 combines a vision encoder and decoder-only language model to support unified perception, reasoning, and planning.
- The model uses progressive training spanning foundational spatiotemporal learning, embodied enhancement, and chain-of-thought reasoning.
- RoboBrain 2.0 achieves state-of-the-art results on most public embodied spatial and temporal reasoning benchmarks, including closed-loop interaction and long-horizon planning.
- Future Works: Future work will integrate embodied VLMs into VLA systems to improve action-generation generality and robustness.
- Future Works: Future system-level integration aims to support serverless deployment, adaptation-free skill registration, and low-latency real-time control.
- Future Works: The released checkpoints, training recipes, and evaluation tools support broader embodied-AI research and downstream applications.
8 Contributions and Author List
This section lists contributors associated with product and operations, infrastructure, and system management, and introduces qualitative examples of RoboBrain 2.0’s embodied capabilities.
- Author List: Dong Liu is listed under Product & Operations.
- Author List: Yance Jiao is listed under Infrastructure.
- Author List: Xi Yang is listed under System Management.
- Contributions: The qualitative examples cover spatial reasoning, temporal planning, and interactive reasoning for potential real-world applications.
A.1 Examples for Pointing
RoboBrain 2.0’s pointing examples illustrate spatial referring across simple object identification, spatial relations, attributes, affordances, and free-space references. The appendix also presents affordance and trajectory examples showing functional-part identification and path prediction.
- Pointing: Pointing tasks require identifying objects from spatial instructions, including references such as “the brown box on the shelf.”
- Affordance: Affordance-pointing examples identify actionable object parts according to the requested interaction.
- Pointing: The examples show spatial referring predictions for objects or parts selected by object attributes and spatial relations.
- Pointing: Some pointing examples identify free space indicated by spatial relations alongside the referenced objects.
- Trajectory: Trajectory examples depict blue model-predicted paths composed of key trajectory points for task completion.
A.4 Examples for EgoPlan2
The EgoPlan2 and close-loop examples demonstrate sequential task decomposition and feedback-driven interaction. They depict plans for daily activities, multi-step object manipulation, and iterative navigation or machine operation.
- EgoPlan2: EgoPlan2 asks the model to plan daily activities from an egocentric perspective, such as preparing a cup of coffee.
- EgoPlan2: The coffee-preparation example decomposes the task into locating equipment, fetching ingredients, and brewing actions.
- Examples: The figures depict key reasoning processes, final answers, and planning steps for EgoPlan and close-loop tasks.
- Close-loop Planning: Close-loop interaction requires refining actions from real-time feedback while searching for objects and operating appliances.
- Close-loop Planning: The examples include multi-step tasks that place a pen and pillow, then place a cup, potato, and egg across household locations.
A.6 Examples for Multi-Robot Planning
RoboBrain 2.0 coordinates multiple robots to complete shared tasks across restaurant, household, and supermarket scenarios. Its planning assigns roles, coordinates movement, and supports scene-aware collaboration.
- A.6 Examples for Multi-Robot Planning: RoboBrain 2.0 coordinates multiple robots toward a common goal by assigning tasks and managing their movements.The planning process includes collision avoidance and timely goal completion.
- A.6 Examples for Multi-Robot Planning: Figure 33 showcases restaurant collaboration between Unitree G1 and Agilex robots for burger preparation and delivery.
- A.6 Examples for Multi-Robot Planning: In the household scenario, Realman and Agilex robots execute commands to fetch requested items.
- A.6 Examples for Multi-Robot Planning: In the supermarket scenario, RoboBrain 2.0 analyzes gift dimensions and bag compatibility while coordinating gift placement and bag-opening actions.
A.7 Examples for Synthetic Benchmarks
Synthetic benchmarks and prompting procedures evaluate RoboBrain 2.0’s spatial reasoning, trajectory prediction, temporal action selection, and multi-robot coordination. The examples span visual analysis, planning, and tool-mediated execution.
- A.7 Examples for Synthetic Benchmarks: BLINK evaluates depth perception and spatial relations, while CV-Bench evaluates 3D spatial understanding.RoboBrain 2.0 is reported to achieve high accuracy on BLINK and accurate 3D-scene reasoning on CV-Bench.
- A.7 Examples for Synthetic Benchmarks: The benchmark examples include identifying points in images according to visual criteria and returning selected coordinates.The point-identification process analyzes object shape, size, and spatial relationships before pinpointing coordinates.
- A.7 Examples for Synthetic Benchmarks: Trajectory examples require predicting key points for reaching targets while accounting for obstacles and smooth end-effector control.The process includes object analysis, capability assessment, contextual alignment, and path verification.
- A.7 Examples for Synthetic Benchmarks: Temporal examples determine the next action from image sequences and answer questions using conversation history and prior visual inputs.These procedures analyze task progress, the current scene, and available action options.
- A.7 Examples for Synthetic Benchmarks: Multi-robot planning is divided into global task decomposition and agent-based tool-calling for coordinating actions toward a common goal.Tool calls produce actions whose execution returns an observation for subsequent reasoning.
B.4 Spatial Understanding: Freeform Q&A -- General Spatial Analysis
General spatial analysis answers questions about spatial relationships or action outcomes from one or more images. The process focuses visual analysis on the relevant scene elements or projected actions.
- B.4 Spatial Understanding: Freeform Q&A -- General Spatial Analysis: The model answers spatial-relation and action-outcome questions using one or more images.It examines relevant scene elements or projects actions toward a target trajectory.