Source-linked AI summary
RynnBrain: Open Embodied Foundation Models
Ronghao Dang, Jiayan Guo, Bohan Hou, Sicong Leng, Kehan Li, Xin Li, Jiangpin Liu, Yunxuan Mao, Zhikai Wang, Yuqian Yuan, Minghao Zhu, Xiao Lin, Yang Bai, Qian Jiang, Yaxi Zhao, Minghua Zeng, Junlong Gao, Yuming Jiang, Jun Cen, Siteng Huang, Liuyi Wang, Wenqiao Zhang, Chengju Liu, Jianfei Yang, Shijian Lu, Deli Zhao
TL;DR
Embodied intelligence needs models that generalize across environments, tasks, and interaction regimes while reasoning over physical spatial structure and fine-grained spatio-temporal episodes. RynnBrain addresses this gap with an open embodied foundation-model family and specialized variants, achieving strong performance across embodied benchmarks and domains.
Problem
Embodied intelligence requires behavioral and cognitive generalization across environments, tasks, and interaction regimes, while existing reasoning paradigms and benchmarks inadequately capture physical spatial structure and fine-grained spatio-temporal understanding.
Method
RynnBrain unifies egocentric understanding, spatio-temporal localization, physically grounded reasoning, and physics-aware planning across 2B, 8B, and 30B-A3B MoE models, using dedicated data pipelines and specialized post-trained variants.
Results
Across 28 benchmarks, RynnBrain consistently performs as a capable open embodied foundation model at all model scales, while four specialized variants achieve state-of-the-art performance in their respective domains.
Takeaways & Limitations
The results demonstrate the value of RynnBrain pretraining for a wide range of embodied tasks and support its use as a foundation for specialized embodied models.
Takeaways & Limitations
In vision-language navigation, high object-navigation success but lower success on terminal stopping indicates coarse navigation proficiency without sufficient stopping precision.
Abstract
from arXiv · showhide
Despite rapid progress in multimodal foundation models, embodied intelligence community still lacks a unified, physically grounded foundation model that integrates perception, reasoning, and planning within real-world spatial-temporal dynamics. We introduce RynnBrain, an open-source spatiotemporal foundation model for embodied intelligence. RynnBrain strengthens four core capabilities in a unified framework: comprehensive egocentric understanding, diverse spatiotemporal localization, physically grounded reasoning, and physics-aware planning. The RynnBrain family comprises three foundation model scales (2B, 8B, and 30B-A3B MoE) and four post-trained variants tailored for downstream embodied tasks (i.e., RynnBrain-Nav, RynnBrain-Plan, and RynnBrain-VLA) or complex spatial reasoning tasks (i.e., RynnBrain-CoP). In terms of extensive evaluations on 20 embodied benchmarks and 8 general vision understanding benchmarks, our RynnBrain foundation models largely outperform existing embodied foundation models by a significant margin. The post-trained model suite further substantiates two key potentials of the RynnBrain foundation model: (i) enabling physically grounded reasoning and planning, and (ii) serving as a strong pretrained backbone that can be efficiently adapted to diverse embodied tasks.
1 Introduction
RynnBrain is proposed as a physically grounded embodied foundation model designed to unify broad semantic understanding with spatial-temporal reasoning and planning. Its foundation models and post-trained variants target comprehensive egocentric cognition, localization, grounded reasoning, and downstream embodied tasks.
- Embodied intelligence requires agents to generalize behavior and cognition across environments, tasks, and interaction regimes.
- Existing VLMs lack intrinsic grounding in physical dynamics, while action-centric embodied models can sacrifice semantic abstraction and broad multimodal generalization.
- RynnBrain unifies egocentric understanding, spatio-temporal localization, physically grounded reasoning, and physics-aware planning in a physically grounded foundation model.
- Its localization capabilities cover objects, target areas, affordances, and trajectories, while its egocentric understanding includes spatial and temporal comprehension, OCR, counting, and fine-grained video understanding.
- The model family includes 2B, 8B, and 30B-A3B MoE scales, with post-trained variants for chain-of-point reasoning, navigation, manipulation planning, and vision-language-action modeling.
- A human–model collaborative data flywheel scales the training corpus beyond 20 million samples while targeting OCR, localization, planning, and physically grounded reasoning.
2 Overview
RynnBrain uses a decoder-only vision–language architecture with multimodal inputs and aligned outputs for embodied understanding and planning. Its infrastructure also addresses long-tailed sequence lengths through online load balancing and per-sample loss reduction.
- RynnBrain combines a vision encoder, vision-language projector, and Qwen3-VL-initialized language backbone in a decoder-only architecture.
- Its multimodal training data spans short localization and perception tasks and long-form captioning and reasoning tasks, creating high-variance, long-tailed sequence lengths.
- Online load balancing redistributes samples across data-parallel workers to minimize the maximum cumulative sequence length per worker.
- The model handles single-view images, multi-view images, videos, and language instructions, producing text, regions, trajectories, and pointing signals in a shared output space.
- Per-sample loss reduction avoids the extra all-gather required by global token-count normalization, eliminating synchronization overhead and improving efficiency.
- Training uses ZeRO-1 and gradient checkpointing for the 2B and 8B models, while the released frameworks are based on HuggingFace Transformers.
3 Physics-Aware Spatio-temporal Pretraining
RynnBrain pretraining unifies multimodal spatio-temporal representations with physically grounded outputs and diverse task data. Its datasets cover general multimodal understanding, spatial reasoning, counting, OCR, grasp poses, and planning guidance.
- Unified Spatio-temporal Representation: RynnBrain represents images and videos as frame sequences, using temporal positional embeddings to capture order, motion, and trajectory dynamics.
- Physically Grounded Output Space: Discrete coordinate tokens encode bounding boxes, points, and trajectory waypoints in [0, 1000], converting spatial prediction into classification.The same autoregressive mechanism used for language generation produces these spatial outputs.
- Pretraining Data: The pretraining corpus spans images, videos, and text across short-response localization tasks and long-form multimodal reasoning.The mixture includes publicly available multimodal datasets and specialized object-understanding data with spatial annotations.
- Spatial Understanding: Over 2.5M spatial instruction samples cover general spatial understanding and fine-grained object-centric reasoning.Calibrated 3D scenes support questions about metric distances, relative positions, heights, and other spatial relationships.
- OCR: Approximately 1M OCR samples provide scene-text recognition and grounding from egocentric videos.The OCR data include normal video QA and area-prediction QA with frame indices and normalized bounding coordinates.
- Manipulation Data: 1.3M grasp-pose samples derived from 945K images teach orientation-aware, spatially grounded grasp-pose prediction for manipulation.
4 Physically Grounded Chain-of-Point Reasoning
RynnBrain-CoP grounds textual reasoning in explicit spatial references from egocentric video. Its specialized data and physically anchored rewards are designed to support more reliable embodied reasoning.
- Motivation and Approach: Chain-of-Point reasoning interleaves textual inference with spatial grounding over egocentric video streams.Intermediate reasoning steps are anchored to concrete physical references rather than remaining purely textual.
- Data Construction: CoP training data enrich spatio-temporal localization samples with a Thinking field that connects high-level task understanding to low-level spatial localization.
- Data Construction: Entities in generated reasoning chains are linked to selected video frames and annotated with representative points or 2D bounding boxes.The annotations are inserted back into the reasoning text using structured object/area, frame, and coordinate markup.
- Data Construction: The resulting CoT-style dataset continually anchors internal reasoning to specific visual evidence in physical space.The paper presents this grounding as supporting transparent and hallucination-resistant reasoning for embodied scenarios.
- Training Recipe: GRPO aligns the model with physically grounded reasoning by estimating baselines from group scores of multiple outputs instead of using a value-function critic.This design reduces memory usage and training complexity relative to standard PPO.
- Reward Design: Task-specific rewards evaluate trajectory shape, affordance-point set similarity, and area-point accuracy using physically grounded spatial criteria.The curated reinforcement-learning dataset contains 30K samples after difficulty-aware filtering and inclusion of key-frame failures.
5 Post-training for Embodied Tasks
RynnBrain is adapted to embodied tasks through specialized post-training for navigation, planning, and physical robot control. These variants combine multimodal observations, conversational history, spatial grounding, and action prediction to support embodied execution.
- Vision-Language Navigation: RynnBrain-Nav fine-tunes the pretrained foundation model on vision-language navigation data and deploys it as an agent that generates actions from observations and instructions.The navigation formulation uses visual observations, language instructions, and a discrete action space containing forward, left, right, and stop actions.
- Vision-Language Navigation: The navigation training format interleaves observation-action pairs in multi-turn conversations, training the model to predict each next action from the current observation and conversational history.Each trajectory is represented as an image-text sequence such as {o0, a0, o1, a1, . . . , on, an}.
- Vision-Language Navigation: The navigation dataset combines 450K Habitat-generated clips from R2R, R2R-EnvDrop, and RxR with 300K additional ScaleVLN samples, followed by multi-turn DAgger collection.The 450K clips span 60 Matterport3D environments and are intended to increase scene diversity and generalization.
- Manipulation Planning: Long-horizon manipulation adaptation uses multi-turn dialogues as an explicit memory buffer, preserving historical reasoning results during interaction.This post-training direction addresses the need to maintain effective memory when adapting planning capabilities to complex manipulation tasks.
- Vision-Language-Action Modeling: RynnBrain-VLA bridges planning and physical execution by using a flow-matching framework in which a single-stream Diffusion Transformer predicts executable action chunks.Linear projections align input noise, timestamp embeddings, and output actions with the VLM hidden size; actions are placed at the sequence end to enable KV caching.
- Vision-Language-Action Modeling: RynnBrain-VLA is built on RynnBrain-2B and uses fine-grained object references and precise spatial localization while maintaining low inference latency.Its control training uses six pick-and-place tasks involving three objects, collected through manual teleoperation on a Franka Emika arm.
6 Evaluation
RynnBrain is evaluated across embodied cognition, spatial grounding, general visual understanding, physically grounded reasoning, navigation, and hierarchical planning. Results show strong performance across diverse tasks, while navigation exposes precision and scaling limitations.
- Embodied Cognition Capability: RynnBrain improves over prior models across embodied cognition benchmarks, including gains of 8.7% on VSI-Bench and 25.1% on RynnBrain-Spatial.It also improves by 10.5% on EgoTaskQA, 6.6% on Open-X VQA, and 20.2% on RynnBrain-Object.
- Embodied Location Capability: RynnBrain-8B leads most spatial location benchmarks, achieving 59.2 on RefSpatial-Bench, 44.7 on ShareRobot-Affordance, and strong grasp-pose results.At 30B-A3B, it further reaches 33.6 on Cornell-Grasp, 14.5 on VMRD-Grasp, 83.9 on RynnBrain-Grounding, and 90.5 on RynnBrain-Affordance.
- General Visual Understanding: RynnBrain maintains Qwen3-VL’s general visual understanding while achieving state-of-the-art results on AI2D, MVBench, and InfoVQA.This preserves broad visual capability alongside embodied cognition strengths.
- Physically Grounded Reasoning: 73.8 average score makes RynnBrain-CoP-8B outperform MiMo-Embodied-7B at 65.8, Gemini-3-Pro at 65.1, and RoboBrain2.0-32B at 57.7.The model also reaches 90.3 on affordance prediction, 59.6 on area prediction, and 71.2 on trajectory prediction.
- Vision-Language Navigation: RynnBrain-Nav-8B achieves top-ranked SR of 58.6% on R2R and 56.1% on RxR, with lowest NE of 4.92 on both benchmarks.On R2R, its lower SR than OS indicates coarse navigation strength but insufficient precision for terminal stopping.
- Vision-Language Navigation: Dense models improve from 2B to 8B, but the 30B MoE model does not outperform 8B dense models during initial navigation training.This suggests MoE sparsity was not fully leveraged by the VLN task or requires different training strategies.
- Hierarchical Planning: RynnBrain-Plan-30B-A3B exceeds baselines on hard in-distribution planning, maintaining over 75% completion on Hard Desk Organization while baselines remain near 0%.On unseen Table Bussing, it reaches near 100% task progress across difficulty levels, whereas Qwen3-VL remains below 10% on Hard.
- Hierarchical Planning: Removing multi-turn interaction history sharply degrades RynnBrain-Plan, with the single-turn variant completing only Easy tasks at low success rates.The ablation highlights the importance of temporal context for long-horizon planning.
7 Conclusion and Future Works
RynnBrain is presented as a capable open embodied foundation-model suite spanning cognition, localization, reasoning, planning, and downstream adaptation. The work also introduces RynnBrain-Bench and releases the model family under Apache 2.0 to support broader embodied research.
- RynnBrain expands embodied foundation models across egocentric cognition, spatio-temporal localization, physically grounded reasoning, and physics-aware planning.
- Across 28 benchmarks and all model scales, RynnBrain is described as a highly capable and well-rounded open embodied foundation model.
- Four post-trained variants—RynnBrain-CoP, RynnBrain-Nav, RynnBrain-Plan, and RynnBrain-VLA—achieve state-of-the-art performance in their respective domains.
- RynnBrain-Bench evaluates fine-grained spatiotemporal cognition and localization through episode-level video understanding, precise grounding, and physically meaningful pointing behaviors.
- The authors envision RynnBrain as a core foundation in holistic embodied-agent stacks containing brain, cerebellum, memory, and sensorimotor-interface components.
- The full model family is openly released under the Apache 2.0 license to support broader embodied scenarios and progress toward general embodied intelligence.
B.1 Examples for Embodied Cognition
RynnBrain demonstrates broad embodied cognition in visual examples, spanning object properties, spatial relations, grounded perception, OCR, and higher-level spatial reasoning.
- RynnBrain supports embodied cognition abilities from estimating object size and distance to reasoning about directions, counts, OCR, and 3D spatial structure.
B.2 Examples for Embodied Location
RynnBrain demonstrates embodied location understanding by grounding language and spatial references in physical coordinates and generating task-relevant trajectories.
- RynnBrain localizes objects from relative positions, identifies graspable items through functional cues, and generates trajectories for tasks such as moving or cleaning regions.
B.3 Examples for General Visual Understanding
RynnBrain also demonstrates general visual understanding for both video comprehension and structured image content.
- RynnBrain handles video comprehension and image-level understanding of structured content, including charts and documents.
B.4 Examples for physically Grounded Reasoning
RynnBrain demonstrates physically grounded reasoning and long-horizon planning across navigation and manipulation settings.
- RynnBrain-CoP demonstrates physically grounded reasoning.
- RynnBrain-Plan demonstrates robust long-horizon navigation planning.
B.6 Examples for Manipulation Planning
The examples present RynnBrain’s embodied cognition, navigation, and manipulation-planning capabilities across increasingly complex tabletop and real-environment tasks.
- Planning Manipulation: RynnBrain-Plan handles long-horizon planning and fine-grained manipulation tasks through precise grounding.Examples include one-step offline planning and multi-step online planning.
- Task Variations: Object-classification settings require sorting fruit by category and arranging it spatially, sometimes while leaving bowl contents unmoved.
- Embodied Cognition: Embodied cognition covers object size, direction, distance, counting, OCR, and higher-level spatial reasoning over egocentric views and 3D shapes.
- Grounded Spatial Reasoning: Grounded spatial reasoning maps linguistic descriptions to 3D locations and actions for objects, areas, affordances, trajectories, and grasp poses.The examples cover both video- and image-based location tasks.
- General Visual Understanding: RynnBrain preserves general visual understanding for video comprehension and image-based charts and documents.
- Manipulation Planning: Manipulation-planning tasks include desk organization, table bussing, tableware distribution, and object classification.The task settings vary in difficulty and include out-of-distribution table bussing.
- Task Variations: Task difficulty is varied through requirements such as ordered pen placement, increasing tableware or object counts, and constraints on unmoved fruits.
- Navigation: RynnBrain-Nav examples cover visual-language navigation in real indoor environments and on R2R-CE and RxR-CE.
C Prompts Details
The appendix documents reproducible training and evaluation prompts, benchmark-specific output formats, and fixed inference settings.
- C.1 Training QA Prompts: Training and evaluation prompts are listed in Tables 9 and 10 using unified Python f-string templates.
- C.2 Evaluation and Inference Prompts: Continuation markers and table headings organize the multi-page benchmark and training-prompt listings.
- C.2 Evaluation and Inference Prompts: Navigation prompts request action sequences using TURN LEFT, TURN RIGHT, MOVE FORWARD, and STOP.
- C.2 Evaluation and Inference Prompts: Planning prompts require a single complete sentence embedding a selected frame and integer coordinates in the normalized [0, 1000] range.
- C.2 Evaluation and Inference Prompts: General vision benchmarks specify concise answer formats, including single words, phrases, or multiple-choice letters.
- C.2 Evaluation and Inference Prompts: Spatial benchmarks use task-specific formats for points, bounding boxes, areas, affordances, trajectories, and grasp poses.
- C.2 Evaluation and Inference Prompts: RynnBrain trajectory prompts first identify the starting frame and then output up to 10 normalized key points.
- C.3 Hyper-parameters for Evaluation: Evaluation disables sampling and sets explicit image and video resolution and frame-sampling limits.Videos are sampled at 2 FPS and capped at 512 frames when necessary.