PaperlayerView exact revision on arXiv

Source-linked AI summary

Gemini Robotics 1.5: Pushing the Frontier of Generalist Robots with Advanced Embodied Reasoning, Thinking, and Motion Transfer

Gemini Robotics Team, Abbas Abdolmaleki, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montserrat Gonzalez Arenas, Ashwin Balakrishna, Nathan Batchelor, Alex Bewley, Jeff Bingham, Michael Bloesch, Konstantinos Bousmalis, Philemon Brakel, Anthony Brohan, Thomas Buschmann, Arunkumar Byravan, Serkan Cabi, Ken Caluwaerts, Federico Casarini, Christine Chan, Oscar Chang, London Chappellet-Volpini, Jose Enrique Chen, Xi Chen, Hao-Tien Lewis Chiang, Krzysztof Choromanski, Adrian Collister, David B. D'Ambrosio, Sudeep Dasari, Todor Davchev, Meet Kirankumar Dave, Coline Devin, Norman Di Palo, Tianli Ding, Carl Doersch, Adil Dostmohamed, Yilun Du, Debidatta Dwibedi, Sathish Thoppay Egambaram, Michael Elabd, Tom Erez, Xiaolin Fang, Claudio Fantacci, Cody Fong, Erik Frey, Chuyuan Fu, Ruiqi Gao, Marissa Giustina, Keerthana Gopalakrishnan, Laura Graesser, Oliver Groth, Agrim Gupta, Roland Hafner, Steven Hansen, Leonard Hasenclever, Sam Haves, Nicolas Heess, Brandon Hernaez, Alex Hofer, Jasmine Hsu, Lu Huang, Sandy H. Huang, Atil Iscen, Mithun George Jacob, Deepali Jain, Sally Jesmonth, Abhishek Jindal, Ryan Julian, Dmitry Kalashnikov, M. Emre Karagozler, Stefani Karp, Matija Kecman, J. Chase Kew, Donnie Kim, Frank Kim, Junkyung Kim, Thomas Kipf, Sean Kirmani, Ksenia Konyushkova, Li Yang Ku, Yuheng Kuang, Thomas Lampe, Antoine Laurens, Tuan Anh Le, Isabel Leal, Alex X. Lee, Tsang-Wei Edward Lee, Guy Lever, Jacky Liang, Li-Heng Lin, Fangchen Liu, Shangbang Long, Caden Lu, Sharath Maddineni, Anirudha Majumdar, Kevis-Kokitsi Maninis, Andrew Marmon, Sergio Martinez, Assaf Hurwitz Michaely, Niko Milonopoulos, Joss Moore, Robert Moreno, Michael Neunert, Francesco Nori, Joy Ortiz, Kenneth Oslund, Carolina Parada, Emilio Parisotto, Amaris Paryag, Acorn Pooley, Thomas Power, Alessio Quaglino, Haroon Qureshi, Rajkumar Vasudeva Raju, Helen Ran, Dushyant Rao, Kanishka Rao, Isaac Reid, David Rendleman, Krista Reymann, Miguel Rivas, Francesco Romano, Yulia Rubanova, Peter Pastor Sampedro, Pannag R Sanketi, Dhruv Shah, Mohit Sharma, Kathryn Shea, Mohit Shridhar, Charles Shu, Vikas Sindhwani, Sumeet Singh, Radu Soricut, Rachel Sterneck, Ian Storz, Razvan Surdulescu, Jie Tan, Jonathan Tompson, Saran Tunyasuvunakool, Jake Varley, Grace Vesom, Giulia Vezzani, Maria Bauza Villalonga, Oriol Vinyals, René Wagner, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Chengda Wu, Markus Wulfmeier, Fei Xia, Ted Xiao, Annie Xie, Jinyu Xie, Peng Xu, Sichun Xu, Ying Xu, Zhuo Xu, Jimmy Yan, Sherry Yang, Skye Yang, Yuxiang Yang, Hiu Hong Yu, Wenhao Yu, Wentao Yuan, Yuan Yuan, Jingwei Zhang, Tingnan Zhang, Zhiyuan Zhang, Allan Zhou, Guangyao Zhou, Yuxiang Zhou

arXiv:2510.03342v3cs.RO

TL;DR

General-purpose robots need physical-world understanding, advanced reasoning, and general dexterous control. The report introduces a multi-embodiment VLA, an embodied-reasoning VLM, and an agentic framework combining Motion Transfer with embodied thinking. The family advances robot generalization, complex multi-step task handling, and embodied reasoning, while real-world evaluation remains necessary.

  • Problem

    General-purpose robots require physical-world understanding, advanced reasoning, and general and dexterous control.

  • Method

    The report combines a multi-embodiment VLA using Motion Transfer and embodied thinking with an embodied-reasoning VLM in an agentic framework.

  • Results

    The model family advances robot generalization, complex multi-step task handling, and embodied reasoning across a broad suite of embodied intelligence benchmarks.

  • Takeaways & Limitations

    The combined models offer a supported path toward general-purpose robots that perceive, reason, plan, and act in the physical world.

  • Takeaways & Limitations

    More than 90% of development evaluation episodes used simulation, so real-world evaluation remains necessary to determine model quality.

Abstract

from arXiv · show

General-purpose robots need a deep understanding of the physical world, advanced reasoning, and general and dexterous control. This report introduces the latest generation of the Gemini Robotics model family: Gemini Robotics 1.5, a multi-embodiment Vision-Language-Action (VLA) model, and Gemini Robotics-ER 1.5, a state-of-the-art Embodied Reasoning (ER) model. We are bringing together three major innovations. First, Gemini Robotics 1.5 features a novel architecture and a Motion Transfer (MT) mechanism, which enables it to learn from heterogeneous, multi-embodiment robot data and makes the VLA more general. Second, Gemini Robotics 1.5 interleaves actions with a multi-level internal reasoning process in natural language. This enables the robot to "think before acting" and notably improves its ability to decompose and execute complex, multi-step tasks, and also makes the robot's behavior more interpretable to the user. Third, Gemini Robotics-ER 1.5 establishes a new state-of-the-art for embodied reasoning, i.e., for reasoning capabilities that are critical for robots, such as visual and spatial understanding, task planning, and progress estimation. Together, this family of models takes us a step towards an era of physical agents-enabling robots to perceive, think and then act so they can solve complex multi-step tasks.

1. Introduction

The Gemini Robotics 1.5 family combines a multi-embodiment VLA with an embodied-reasoning VLM and an agentic framework. Its innovations target cross-robot generalization, action-linked reasoning, and complex task execution.

  • Gemini Robotics 1.5 uses a novel architecture and Motion Transfer mechanism to learn from heterogeneous robot data across embodiments.The model is designed to form a unified understanding of motion and physical interactions.
  • The Thinking VLA interleaves physical movements with language-based reasoning that decomposes instructions and supports task recovery.It can simplify complex instructions, detect success or failure, and propose recovery behaviors.
  • Gemini Robotics-ER 1.5 provides embodied reasoning for visuo-spatial-temporal understanding required by robotic applications.It retains general frontier-model capabilities while targeting embodied intelligence benchmarks.
  • The VLA and embodied-reasoning VLM can be combined in an agentic framework for long-horizon task execution, multimodal interaction, tool use, and safety reasoning.The framework coordinates complex planning and adaptive orchestration with explicit reasoning about safety violations.

2. Method Overview

The system combines an embodied-reasoning VLM orchestrator with a VLA action model, supported by multi-embodiment training and simulation-heavy evaluation. Embodied thinking decomposes tasks into progressively finer action segments before execution.

  • Agentic System Architecture: The agentic system uses GR-ER 1.5 as an orchestrator and Gemini Robotics 1.5 as the action model.The orchestrator handles task flow and planning, while the action model translates instructions into robot actions.
  • Embodied thinking: Embodied thinking lets the VLM plan complex tasks and the VLA decompose instructions into shorter segments or primitive motions.The process can proceed from a high-level instruction to seconds-long movement segments and then to commands such as moving a gripper or closing it.
  • Motion Transfer: Motion Transfer enables learning from different robots and data sources while forming a unified understanding of motion and physical interactions.The training data includes ALOHA, Bi-arm Franka, and Apollo humanoid robot data alongside public text, image, and video datasets.
  • Evaluation: The report evaluates comparisons with interleaved A/B/n tests on the same real-robot work cell to reduce variance from robot and environmental differences.The development process also uses MuJoCo simulation aligned with real scenes to accelerate evaluation iteration.
  • Evaluation: Over 90% of development evaluation episodes were conducted in simulation, while real-world evaluation remained necessary for determining model quality.Simulation expanded evaluation breadth and reduced the volume of tests required on real hardware.

3. Gemini Robotics 1.5 is a general multi-embodiment Vision-Language-Action Model

Gemini Robotics 1.5 is a general multi-embodiment VLA that controls diverse robots without embodiment- or task-specific post-training. Its Motion Transfer recipe, cross-embodiment data, and thinking-based action decomposition improve generalization, transfer, and multi-step task performance.

  • Generalization: GR 1.5 consistently outperforms baselines across visual, instruction, action, and task generalization categories.On ALOHA, gains are especially substantial for instruction, action, and task generalization; comparisons with earlier specialists are not fully apples-to-apples.
  • Motion Transfer: Motion Transfer amplifies the benefit of adding data from other embodiments and enables stronger positive transfer of skills between robots.The effect varies with platform data availability: external data helps most for data-scarce humanoids, while MT is less pronounced across larger embodiment gaps.
  • Cross-embodiment transfer: GR 1.5 achieves zero-shot skill transfer on tasks seen only by another robot embodiment, with success-rate evidence indicating completed execution rather than partial progress.Single-embodiment models perform poorly on this benchmark, whereas cross-embodiment data combined with MT performs significantly better.
  • Thinking and acting: Enabling thinking improves progress on longer multi-step tasks by decomposing high-level instructions into short-horizon language commands before mapping them to robot actions.The two-stage decomposition leverages visual-linguistic reasoning for the first step and a simpler action mapping for the second.
  • Thinking and acting: Thinking traces improve interpretability and support situational awareness, automatic subtask transitions, and recovery behaviors during execution.The Apollo rollout illustrates switching objectives after grasp success and responding when an object slips.

4. Gemini Robotics-ER 1.5 is a generalist embodied reasoning model

Gemini Robotics-ER 1.5 is presented as a generalist embodied reasoning model combining broad frontier-model capabilities with strong spatial and physical-world understanding. It achieves state-of-the-art performance across embodied reasoning benchmarks, supports complex robotic capabilities, scales with inference-time thinking, and improves long-horizon agentic execution.

  • Embodied reasoning covers grounded spatial, temporal, physical, causal, and affordance understanding required for robotic applications.
  • 4.1. Generality: GR-ER 1.5 combines strong embodied reasoning with broad frontier-model generality, evaluated across 15 embodied reasoning benchmarks and general-purpose benchmarks.The embodied reasoning score weights spatial reasoning and question-answering benchmarks equally.
  • 4.2. Frontier capabilities for Embodied Reasoning: GR-ER 1.5 achieves state-of-the-art complex-pointing results and significantly outperforms GR-ER, Gemini 2.5, and GPT-5 across pointing, spatial, steerable, and point-to-count capabilities.Complex pointing includes spatially constrained pointing, instruction-guided point modification, and counting that uses points as an intermediate reasoning step.
  • 4.3. Thinking: GR-ER 1.5 improves embodied reasoning as its thinking-token budget grows, with image and video question answering benefiting more from longer traces than pointing tasks.The model can automatically modulate thinking-token usage, and its inference-time scaling is stronger than Gemini 2.5 Flash’s.
  • 4.3. Thinking: In long-horizon tasks, the GR 1.5 agent frequently approaches 80% progress scores, compared with up to 44% for the Thinking VLA.The system’s failures include orchestrator errors, success-detection errors, and VLA failures; GR-ER 1.5 as orchestrator improves all reported categories over Gemini 2.5 Flash.
  • 4.3. Thinking: The agentic architecture assigns high-level planning and orchestration to GR-ER 1.5, while a VLA handles action execution, improving reliability on complex, long-horizon tasks.The paper identifies the combination of robust low-level control and high-level embodied reasoning as central to capable physical agents.

6. Responsible Development and Safety

The report presents a layered safety and alignment approach spanning semantic reasoning, human dialogue, pre-action safety thinking, physical safeguards, and continuous adversarial evaluation.

  • The safety approach combines semantic reasoning, respectful dialogue, thinking about safety before acting, and low-level physical safeguards such as collision avoidance.
  • The report links strong compliance with safety policies to socially appropriate speech, gestures, and actions in human-robot interactions.
  • ASIMOV-2.0 expands semantic action-safety evaluation with improved injury coverage, video modalities, and embodied reasoning tasks involving physical constraints.
  • GR-ER 1.5 improves recognition of risks, understanding of action consequences, and intervention timing, achieving state-of-the-art results on safety-constrained pointing when fine-tuned for safety thinking.
  • Auto-Red-Teaming dynamically stress-tests models through adversarial prompt, scene, and rollout attacks using Attacker, Target, and Autorater models.
  • Auto-red-teaming results indicate stronger GR-ER 1.5 robustness with Thinking, reliable critique and correction by AutoRaters, and mitigation of hallucination vulnerabilities through generated training data.

7. Discussion

The discussion frames Gemini Robotics 1.5 as combining embodied reasoning, generalist action control, and cross-embodiment learning, while identifying data scarcity and dexterity as remaining challenges.

  • The report identifies three contributions: Thinking VLA, learning across robot embodiments, and state-of-the-art embodied reasoning.
  • Thinking VLA: Thinking VLA uses a multi-level internal monologue to improve handling of complex, multi-step tasks.
  • Learning across different robot embodiments: Learning from heterogeneous datasets enables skill transfer between robot platforms and addresses the data scarcity problem in robot learning.
  • Combining embodied thinking with a general VLA and embodied reasoning supports task decomposition, execution, perception, semantic understanding, and planning.
  • Future work targets scalable video data sources to mitigate data scarcity and new architectures or reinforcement learning to improve dexterity without sacrificing generality.

8. Contributions and Acknowledgments

The supplied material documents the model card, model inputs and outputs, training data, preprocessing, hardware and software, and pointers to evaluation sections.

  • Gemini Robotics-ER 1.5 is a Vision-Language Model, while Gemini Robotics 1.5 is a Vision-Language-Action model for manipulation across tasks, scenes, and robots.
  • Both models accept text and images; GR-ER 1.5 outputs text, while GR 1.5 outputs continuous robot actions and optional thinking text.
  • Training uses images, text, robot sensor data, and robot action data.
  • Preprocessing includes multi-stage safety and quality filtering, sensitive-data filtering, and synthetic captions paired with original captions.
  • The model card identifies TPU v4, v5p, and v6e hardware and JAX with ML Pathways as software infrastructure.
  • Evaluation procedures and results are directed to the report’s embodied reasoning, robotics, and safety sections.

B.1. Rank consistency between evaluations in simulation and on real robots

The supplemental evaluations use physics-based simulation to scale testing, with rank consistency between simulation and real-robot results supporting faster iteration; the benchmarks also broaden generalization coverage.

  • B.1. Rank consistency between evaluations in simulation and on real robots: Simulation and real-robot evaluations show rank-consistent success rates across tasks, enabling faster iteration on architectures, objectives, and experiment design.
  • Generalization benchmark: The expanded benchmark adds five action-generalization tasks and twelve entirely new task-generalization tasks to prior coverage.
  • Task generalization: Task-generalization evaluations use unseen instructions, objects, and initial conditions across twelve tasks and multiple scenes.
  • Bi-arm Franka benchmark: The Bi-arm Franka benchmark contains 44 tasks, including 20 in-distribution tasks and 24 variants spanning instruction, visual, and action generalization.
  • Bi-arm Franka benchmark: The Bi-arm Franka scenes cover workbench manipulation, computer-and-cable interaction, and the NIST Assembly Board 2.
  • Apollo humanoid benchmark: The Apollo humanoid benchmark evaluates instruction, visual, and action generalization using the same task-generalization tasks as the ALOHA benchmark.

B.2.3.2 Qualitative Results

Gemini Robotics 1.5 demonstrates qualitative generalization on humanoid manipulation and is evaluated for Motion Transfer across robot embodiments. The benchmarks test whether skills learned on one robot transfer to another, including tasks involving motions absent from the target robot’s data.

  • Humanoid generalization: GR 1.5 grasps novel objects, places them into unseen receptacles, and manipulates objects on surfaces at unseen heights.Different grasping strategies emerge automatically depending on the objects being manipulated.
  • Humanoid generalization: Humanoid control is challenging because it requires high-degree-of-freedom whole-body control, multi-finger dexterity, and active perception under limited workspace visibility.
  • Motion Transfer evaluation: Motion Transfer benchmarks test tasks on Robot A using data collected only on Robot B, and vice versa.The evaluation spans the Bi-arm Franka, ALOHA, and humanoid robots.
  • Motion Transfer evaluation: The cross-embodiment tasks include motions such as opening drawers and closing a pear-shaped organizer that are unavailable in Bi-arm Franka data.Additional packing tasks test whether Motion Transfer also improves performance on easier tasks.

B.3.3. ALOHA benchmark →Humanoid robot

The ALOHA-to-humanoid benchmark evaluates transfer of ALOHA skills that are not covered by humanoid action data. The broader evaluation also measures ordered and goal-oriented multi-step behavior across three robot platforms.

  • Cross-embodiment evaluation: The cross-embodiment evaluation includes Bi-arm Franka → ALOHA, ALOHA → Bi-arm Franka, Apollo humanoid → Bi-arm Franka, and ALOHA → Humanoid tasks.
  • Multi-step benchmarks: Multi-step benchmarks combine individual tasks into compound instructions or abstract, goal-oriented instructions.Compound instructions typically require a specified order to receive a full score of 1.0.
  • Multi-step benchmarks: The multi-step benchmarks cover ALOHA, Bi-arm Franka, and Apollo humanoid robots, with task visuals and progress-score definitions provided for each platform.
  • Results reporting: The reported results section provides success rates for all results in Section 3, including comparisons involving Gemini Robotics 1.5 and baseline models.

B.5.2. Success rate for data and model ablation

The ablation compares Gemini Robotics 1.5 against variants that remove Motion Transfer or alter the training data and recipe. A separate ablation measures success rates with and without thinking during inference.

  • Dataset and training ablations: The ablation includes a multi-embodiment Gemini Robotics 1.5 variant trained without the Motion Transfer recipe.
  • Thinking ablation: The thinking ablation compares multi-step benchmark success rates with and without enabling thinking during inference.

C.1. Evaluation Details: Generality

Generality evaluation measures Gemini Robotics-ER 1.5 and other multimodal models on embodied reasoning benchmarks. The evaluation includes aggregated and per-benchmark results, plus a detailed complex-pointing breakdown and sampled reasoning traces.

  • Benchmark evaluation: Gemini Robotics-ER 1.5 is evaluated against other multimodal models on a mix of 15 academic embodied reasoning benchmarks.The benchmarks assess the model’s overall embodied reasoning performance through aggregated and individual results.
  • Benchmark evaluation: Text-based VQA benchmarks are graded with Gemini 2.5 Flash for both multiple-choice and freeform response formats.
  • Reasoning traces: Sampled thoughts from GR-ER 1.5 are presented for embodied reasoning tasks.

D.1. Long-horizon benchmarks

The long-horizon benchmarks evaluate the GR-ER 1.5 and VLA combination as an autonomous agent on ALOHA and Bi-arm Franka tasks. These tasks require physical understanding, tool use, planning, execution, and error recovery, with progress scored across subtasks.

  • The benchmarks combine GR-ER 1.5 with the VLA to evaluate an autonomous agent.
  • Progress is scored as the sum of points earned across each subtask for both robot benchmarks.
  • The ALOHA benchmark contains four long-horizon tasks.
  • The Bi-arm Franka benchmark also contains four long-horizon tasks.
  • Figure 41 reports success rates for the results in Section 5.
  • Tasks require advanced real-world understanding, tool use, long-horizon planning, execution, and error recovery.
Loading 2510.03342v3…