Source-linked AI summary
RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete
Yuheng Ji, Huajie Tan, Jiayu Shi, Xiaoshuai Hao, Yuan Zhang, Hengyuan Zhang, Pengwei Wang, Mengdi Zhao, Yao Mu, Pengju An, Xinda Xue, Qinghang Su, Huaihai Lyu, Xiaolong Zheng, Jiaming Liu, Zhongyuan Wang, Shanghang Zhang
TL;DR
Long-horizon robotic manipulation requires planning, affordance perception, and trajectory prediction that existing MLLMs lack. The paper introduces ShareRobot and RoboBrain, combining multidimensional robotic supervision with a multimodal, multistage model; RoboBrain achieves state-of-the-art performance across robotic benchmarks.
Problem
Existing MLLMs lack planning, affordance perception, and trajectory prediction capabilities for long-horizon robotic manipulation, partly because fine-grained robotic-operation datasets are scarce.
Method
The paper builds ShareRobot with labels for task planning, object affordance, and end-effector trajectories, then trains RoboBrain with robotic and general multimodal data, multistage training, long videos, and high-resolution images.
Results
RoboBrain achieves state-of-the-art performance across various robotic tasks and outperforms baseline models across OpenEQA, ShareRobot, and RoboVQA.
Takeaways & Limitations
ShareRobot and RoboBrain advance robotic capabilities from abstract instruction understanding toward concrete manipulation planning, affordance perception, and trajectory prediction.
Abstract
from arXiv · showhide
Recent advancements in Multimodal Large Language Models (MLLMs) have shown remarkable capabilities across various multimodal contexts. However, their application in robotic scenarios, particularly for long-horizon manipulation tasks, reveals significant limitations. These limitations arise from the current MLLMs lacking three essential robotic brain capabilities: Planning Capability, which involves decomposing complex manipulation instructions into manageable sub-tasks; Affordance Perception, the ability to recognize and interpret the affordances of interactive objects; and Trajectory Prediction, the foresight to anticipate the complete manipulation trajectory necessary for successful execution. To enhance the robotic brain's core capabilities from abstract to concrete, we introduce ShareRobot, a high-quality heterogeneous dataset that labels multi-dimensional information such as task planning, object affordance, and end-effector trajectory. ShareRobot's diversity and accuracy have been meticulously refined by three human annotators. Building on this dataset, we developed RoboBrain, an MLLM-based model that combines robotic and general multi-modal data, utilizes a multi-stage training strategy, and incorporates long videos and high-resolution images to improve its robotic manipulation capabilities. Extensive experiments demonstrate that RoboBrain achieves state-of-the-art performance across various robotic tasks, highlighting its potential to advance robotic brain capabilities.
1. Introduction
Existing MLLMs perform well on general multimodal tasks but remain limited in long-horizon robotic manipulation. The paper attributes this gap to missing planning, affordance perception, and trajectory prediction capabilities, and introduces ShareRobot and RoboBrain to address it.
- Research Gap: Current MLLMs lack planning, affordance perception, and trajectory prediction capabilities needed for long-horizon manipulation.These capabilities cover decomposing instructions, recognizing interactive object regions, and anticipating manipulation trajectories.
- Contributions: The paper introduces ShareRobot, a large-scale, fine-grained robotic dataset labeling task planning, object affordance, and end-effector trajectories.Three annotators review and refine the dataset’s labels.
- Contributions: RoboBrain combines robotic and general multimodal data with multi-stage training, long videos, and high-resolution images.The design targets visual perception, historical-frame memory, and robotic manipulation planning.
- Results: RoboBrain outperforms existing models across multiple robotic benchmarks, including RoboVQA and OpenEQA.The experiments evaluate whether the proposed dataset and model improve robotic manipulation capabilities.
2. Related Work
Prior robotic MLLM research addresses planning, action generation, and multimodal understanding, while manipulation datasets increasingly emphasize diverse interactions and embodiments. The paper positions ShareRobot as addressing the remaining need for fine-grained affordance and trajectory supervision.
- Datasets for Manipulation Planning: ShareRobot extends this direction by labeling planning, affordances, and trajectories for robotic operation tasks.Its generation process converts demonstrations into fine-grained supervision for manipulation planning.
- MLLMs for Robotic Manipulation Planning: Existing robotic MLLMs address planning, action sequencing, and multimodal observations but still lack mechanisms for complex atomic-task execution.The related work identifies affordance perception and trajectory prediction as remaining needs.
- Datasets for Manipulation Planning: Earlier manipulation datasets mainly annotate hand-object interactions such as grasping and pushing.Later datasets broaden multimodal coverage and cross-embodiment generalization.
3. ShareRobot Dataset
ShareRobot is a large-scale, fine-grained, multidimensional dataset built from selected robotic demonstrations. It combines detailed planning instructions with affordance and trajectory labels across diverse scenes, embodiments, and atomic tasks.
- Overview: ShareRobot provides fine-grained low-level planning instructions linked to individual frames, alongside object affordance and end-effector trajectory labels.The dataset is designed to transform abstract task descriptions into concrete actions.
- Diversity: ShareRobot covers 102 scenes, 12 embodiments, and 107 types of atomic tasks.The source distribution includes common actions such as pick, move, reach, lift, and place.
- Data Selection: Videos are filtered for resolution, accurate descriptions, successful execution, sufficient length, visible objects and end-effectors, and clear trajectories.Videos with fewer than 30 frames or resolution below 128 pixels are excluded.
- Data Labeling: Planning labels are generated from 30 extracted frames and high-level descriptions, then reviewed by three annotators and expanded into question templates.Affordance regions use bounding boxes, while trajectory labels contain at least three gripper coordinates.
4. RoboBrain Model
RoboBrain uses a multimodal foundation model and specialized affordance and trajectory modules, trained through general and robotic phases. Its pipeline plans tasks before executing affordance perception and trajectory prediction, and it achieves state-of-the-art benchmark performance.
- Model Architecture: RoboBrain consists of a planning foundation model, an A-LoRA affordance module, and a T-LoRA trajectory module.The model first generates detailed plans, then splits them into sub-task descriptions for downstream prediction.
- Model Architecture: The foundational model uses a vision encoder, projector, and LLM to process image or video inputs and generate language responses.The implementation uses SigLIP, a 2-layer MLP, and Qwen2.5-7B-Instruct within a LLaVA-based architecture.
- Affordance Perception: Affordances are represented as bounding boxes for regions where the human hand contacts objects.The coordinates specify the top-left and bottom-right corners of each affordance region.
- Trajectory Prediction: Trajectory waypoints are 2D coordinates representing end-effector or hand movement through an episode.The T-LoRA module predicts visual traces from sequences of waypoint coordinates.
5. Experiment
RoboBrain is evaluated on planning, affordance prediction, and trajectory prediction, with results showing strong performance across robotic benchmarks and trajectory metrics.
- Planning Task: RoboBrain outperformed all baseline models across RoboVQA, OpenEQA, and ShareRobot planning benchmarks.The baselines included GPT-4V, Claude3, LLaVA-1.5, LLaVA-OneVision-7b, Qwen2-VL-7b, and RoboMamba.
- Planning Task: 18.75 BLEU-4 points separated RoboBrain from the second-place model on RoboVQA.The result highlights performance on decomposing complex long-range task planning.
- Trajectory Prediction: 42.9%, 94.2%, and 31.6% reductions in DFD, HD, and RMSE, respectively, were achieved by the most effective trajectory model versus baseline.The final model integrated start points, waypoint sampling, end points, and special tokens.
- Robot Brain Task: RoboBrain uses human instructions and visual inputs to plan future steps while outputting affordances and trajectories through multi-turn interaction.Figure 6 depicts action plans and assessments based on real-time image feedback, with trajectory prediction for each step.
6. Conclusion
The paper introduces ShareRobot and RoboBrain to connect abstract instruction understanding with concrete robotic manipulation capabilities.
- Conclusion: ShareRobot labels task planning, object affordance, and end-effector trajectory as multidimensional robotic data.The dataset is presented as high quality and heterogeneous.
- Conclusion: RoboBrain integrates robotic and general multimodal data, uses multistage training, and leverages long videos and high-resolution images.The model is designed to enhance robotic manipulation capabilities.
- Conclusion: Extensive experiments report state-of-the-art performance across various robotic tasks.The conclusion frames this result as potential progress for robotic capabilities.
A. Details of Models and Training
RoboBrain uses a LLaVA-based architecture with a visual encoder, projector, and language model, trained through staged procedures that include parameter-efficient adaptation.
- Model Setting: RoboBrain consists of a visual encoder, projector, and large language model within the LLaVA framework.These are the model’s three main components.
- Model Setting: The visual encoder is SigLIP siglip-so400m-patch14-384, while the projector is a two-layer MLP and the language model is Qwen2.5-7B-Instruct.SigLIP processes 384x384 images into visual tokens projected into the text-embedding dimensions.
- Training Strategy: Stage 4 uses rank-64 LoRA modules in the projector and language-model feed-forward layers to acquire affordance and trajectory capabilities.All parameters except the LoRA modules are frozen during training.
- Training Strategy: Training follows a staged strategy based on LLaVA-OneVision and expands the robotic training phase.Experiments were conducted on servers equipped with 8×A800 GPUs.
B. Details of Training Dataset
The training data combines general multimodal sources with robotic data, while supplementary experiments examine data composition, architectures, and benchmark performance.
- Training Dataset: The training corpus includes LCS-558K, Image-4M, SI-3.2M, and OV-1.6M multimodal datasets.These sources provide visual-language alignment, diverse images, curated multimodal samples, and video data.
- Training Dataset: The actual data used amounts to 3.1M because certain datasets were unavailable.This is a stated difference between intended and available training data.
- Experiments: The supplementary experiments analyze ShareRobot’s training impact, robotic-data proportions, architectures, pretrained MLLMs, language-model backbones, and training-stage ablations.These studies compare RoboBrain with baselines on general and robotic benchmarks.
- General Benchmarks: General benchmark evaluation covers chart, diagram, document, visual perception, multidomain reasoning, and real-world understanding tasks.The evaluation uses the LMMs-Eval toolkit and multiple image benchmarks.
- General Benchmarks: RoboBrain surpasses LLaVA-OV-7B and GPT-4V on the reported structured OCR benchmarks, with exceptions on DocVQA and OCRBench.It performs slightly lower than GPT-4V on DocVQA and slightly behind LLaVA-OV-7B on OCRBench.
C.2. More Results on Robotic Benchmarks.
RoboBrain is evaluated on robotic benchmarks covering planning, affordance, trajectory, and environment understanding. It outperforms the reported baselines across these assessments, including approximately 30% higher RoboVQA performance than the second-best model.
- Benchmark setup: RoboBrain is evaluated on RoboVQA, OpenEQA, and the ShareRobot test set against general-purpose and robotic MLLM baselines.The baselines include GPT-4V, LLaVA-OV-7B, Qwen2-VL-7B, and RoboMamba.
- RoboVQA: Approximately 30% higher performance than the second-best model is achieved by RoboBrain on RoboVQA.RoboVQA contains long-horizon planning and robotics VQA tasks evaluated with average BLEU-1–BLEU-4 performance.
- OpenEQA: RoboBrain outperforms GPT-4V overall on OpenEQA’s embodied question-answering evaluation.OpenEQA evaluates environment understanding using over 1,600 human-generated questions from more than 180 real-world scenes.
- ShareRobot benchmark: RoboBrain outperforms all baselines on the ShareRobot benchmark, demonstrating planning capabilities across diverse scenes and embodiments.The evaluation uses GPT-4o scoring across planning, future prediction, affordance, and success-related task categories.
- Data contribution: Including ShareRobot data improves RoboBrain’s performance compared with training scenarios without ShareRobot.The ablation identifies ShareRobot as contributing to enhanced planning capability.
C.4. Effectiveness of Robot Data Proportion
The experiments examine how robotic-to-general data proportions, ShareRobot inclusion, model backbones, and staged training affect RoboBrain. A 4:6 robotic-to-general ratio balances robotic and general benchmark performance, while staged training improves planning, affordance, and trajectory capabilities.
- Robot data proportion: The evaluated robotic-to-general ratios range from 3:7 through 7:3, including 4:6, 5:5, and 6:4 configurations.These configurations correspond to robotic-data proportions from 30% to 70%.
- Robot data proportion: A 4:6 ratio of robotic to general data is an effective training choice, balancing performance on robotic and general benchmarks.The comparison keeps the total training dataset at 200K samples while varying the data proportions.
- ShareRobot ablation: ShareRobot data produces significant performance improvements across tested MLLM architectures during stage 3 supervised fine-tuning.The architecture comparison includes LLaVA-OV-7B, OpenVLA-7B, and Qwen2VL-7B using the same training-data proportions.
- ShareRobot ablation: Different LLM backbones benefit from fine-tuning on ShareRobot data.The backbone comparison uses four distinct LLMs.
- Staged training: Stages 1–3 consistently improve planning performance, while stage 4 enhances affordance and trajectory capabilities.The results are reported in the staged-training evaluation summarized in Table 9.
D.1. Visualization on Planning
Visualizations show RoboBrain generating detailed manipulation plans from visual observations and task instructions, while also exposing failures in object recognition and noisy-scene perception. The examples further illustrate affordance localization and executable trajectory prediction, including trajectory failure cases requiring physical constraints and world knowledge.
- Planning examples: RoboBrain generates plans for watering plants, placing a pot in a drawer, clustering colored blocks, and cleaning a desk.The first three examples are categorized as good cases and the cleaning example as a bad case.
- Planning examples: The model uses environmental information and interactive-object states from first- or third-person images to plan varied manipulation tasks.For colored-block clustering, it counts blocks and assigns them to four designated table corners.
- Planning limitations: Planning failures include misidentifying a tissue as a disinfectant wipe, omitting tissue extraction, and failing to prioritize spilled coffee.The authors attribute the primary cause possibly to color similarity between the desk and spilled coffee.
- Planning limitations: RoboBrain demonstrates robust planning in most tested cases despite a small number of unreasonable bad cases.The authors describe this planning performance as a foundation for executing long-horizon manipulation tasks.
- Affordance perception: Affordance visualizations show reasonable regions predicted from human instructions and visual information, including identifying a closed bottle cap for drinking.The examples are presented as evidence of understanding abstract instructions.
- Trajectory prediction: Trajectory predictions align with ground-truth structure and remain executable, often producing smoother and potentially more efficient paths.The visualizations report generalization across scenarios, while failures involve cups, articulated fridge doors, and deformable clothing.
E. Details of ShareRobot Dataset
ShareRobot combines diverse long-horizon descriptions, low-level instructions, and templated planning questions with visual examples of RoboBrain’s planning, affordance, and trajectory capabilities.
- Data Generation: The labeling workflow uses prompts for Gemini and templates to generate question-answer pairs during data generation.The paper presents these prompts and templates as part of the dataset-generation process.
- Dataset Contents: ShareRobot contains 10,290 long-horizon high-level descriptions and 28,181 low-level instructions.The dataset lists frequent examples for both description levels.
- Embodied Planning: Additional planning visualizations include good embodied-planning cases and one bad case.The paper directs readers to Sec. D.1 for more detailed analysis.
- Affordance Perception: Additional visualizations show RoboBrain predicting affordance areas from instructions and visual information, while also exposing failure cases.Red bounding boxes denote predicted affordance areas; the fourth row contains failures.
- Trajectory Prediction: Trajectory visualizations compare ground-truth red-to-purple curves with predicted green-to-blue curves for end-effector manipulation.The examples are intended to show trajectory generation from robot observations and task instructions.
- Planning Templates: Its planning data uses 10 question types, each with five templates to diversify question formulations.The templates ask for the next action needed to achieve a long-horizon objective.