PaperlayerView exact revision on arXiv

Source-linked AI summary

Gemini Robotics: Bringing AI into the Physical World

Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montserrat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, Steven Bohez, Konstantinos Bousmalis, Anthony Brohan, Thomas Buschmann, Arunkumar Byravan, Serkan Cabi, Ken Caluwaerts, Federico Casarini, Oscar Chang, Jose Enrique Chen, Xi Chen, Hao-Tien Lewis Chiang, Krzysztof Choromanski, David D'Ambrosio, Sudeep Dasari, Todor Davchev, Coline Devin, Norman Di Palo, Tianli Ding, Adil Dostmohamed, Danny Driess, Yilun Du, Debidatta Dwibedi, Michael Elabd, Claudio Fantacci, Cody Fong, Erik Frey, Chuyuan Fu, Marissa Giustina, Keerthana Gopalakrishnan, Laura Graesser, Leonard Hasenclever, Nicolas Heess, Brandon Hernaez, Alexander Herzog, R. Alex Hofer, Jan Humplik, Atil Iscen, Mithun George Jacob, Deepali Jain, Ryan Julian, Dmitry Kalashnikov, M. Emre Karagozler, Stefani Karp, Chase Kew, Jerad Kirkland, Sean Kirmani, Yuheng Kuang, Thomas Lampe, Antoine Laurens, Isabel Leal, Alex X. Lee, Tsang-Wei Edward Lee, Jacky Liang, Yixin Lin, Sharath Maddineni, Anirudha Majumdar, Assaf Hurwitz Michaely, Robert Moreno, Michael Neunert, Francesco Nori, Carolina Parada, Emilio Parisotto, Peter Pastor, Acorn Pooley, Kanishka Rao, Krista Reymann, Dorsa Sadigh, Stefano Saliceti, Pannag Sanketi, Pierre Sermanet, Dhruv Shah, Mohit Sharma, Kathryn Shea, Charles Shu, Vikas Sindhwani, Sumeet Singh, Radu Soricut, Jost Tobias Springenberg, Rachel Sterneck, Razvan Surdulescu, Jie Tan, Jonathan Tompson, Vincent Vanhoucke, Jake Varley, Grace Vesom, Giulia Vezzani, Oriol Vinyals, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Fei Xia, Ted Xiao, Annie Xie, Jinyu Xie, Peng Xu, Sichun Xu, Ying Xu, Zhuo Xu, Yuxiang Yang, Rui Yao, Sergey Yaroshenko, Wenhao Yu, Wentao Yuan, Jingwei Zhang, Tingnan Zhang, Allan Zhou, Yuxiang Zhou

arXiv:2503.20020v1cs.RO

TL;DR

General-purpose AI models remain difficult to translate from digital understanding into robust physical interaction. This report introduces Gemini Robotics-ER for embodied reasoning and Gemini Robotics for direct robot control, reporting broad manipulation capabilities, adaptation across tasks and embodiments, and safety evaluation alongside remaining precision and reasoning limitations.

  • Problem

    Translating general multimodal AI capabilities into robust embodied reasoning and physical robot interaction remains a significant challenge.

    Source: §1 · p. 1

  • Method

    The report combines Gemini 2.0, embodied-reasoning training and benchmarking, robot action data, and optional specialization into Gemini Robotics-ER and the Gemini Robotics VLA.

    Source: §1 · p. 2, §2.2 · p. 6

  • Results

    Gemini Robotics demonstrates diverse dexterous manipulation, open-vocabulary following, zero-shot generalization, and adaptation to new tasks and robot embodiments.

    Source: §6 · p. 28

  • Takeaways & Limitations

    The Gemini Robotics family provides an initial foundation for connecting embodied reasoning with active robot control across varied physical tasks and embodiments.

    Source: §6 · p. 28

    Gemini 2.0 and Gemini Robotics-ER can struggle with long-video spatial grounding and precise numerical predictions, while Gemini Robotics still requires further work on multi-step dexterous scenarios.

    Source: §6 · p. 28

Abstract

from arXiv · show

Recent advancements in large multimodal models have led to the emergence of remarkable generalist capabilities in digital domains, yet their translation to physical agents such as robots remains a significant challenge. This report introduces a new family of AI models purposefully designed for robotics and built upon the foundation of Gemini 2.0. We present Gemini Robotics, an advanced Vision-Language-Action (VLA) generalist model capable of directly controlling robots. Gemini Robotics executes smooth and reactive movements to tackle a wide range of complex manipulation tasks while also being robust to variations in object types and positions, handling unseen environments as well as following diverse, open vocabulary instructions. We show that with additional fine-tuning, Gemini Robotics can be specialized to new capabilities including solving long-horizon, highly dexterous tasks, learning new short-horizon tasks from as few as 100 demonstrations and adapting to completely novel robot embodiments. This is made possible because Gemini Robotics builds on top of the Gemini Robotics-ER model, the second model we introduce in this work. Gemini Robotics-ER (Embodied Reasoning) extends Gemini's multimodal reasoning capabilities into the physical world, with enhanced spatial and temporal understanding. This enables capabilities relevant to robotics including object detection, pointing, trajectory and grasp prediction, as well as multi-view correspondence and 3D bounding box predictions. We show how this novel combination can support a variety of robotics applications. We also discuss and address important safety considerations related to this new class of robotics foundation models. The Gemini Robotics family marks a substantial step towards developing general-purpose robots that realizes AI's potential in the physical world.

1. Introduction

The report asks how frontier multimodal models can acquire embodied reasoning and action capabilities for general, dexterous interaction with the physical world. It introduces Gemini Robotics-ER and Gemini Robotics, combining spatial-temporal understanding with robot control, specialization, and safety-oriented development.

1. Introduction

  • The report frames grounded embodied reasoning, combined with real-world action data, as a route toward robots that understand and act in the physical world with greater generality.This objective addresses the gap between passive perception and active physical interaction.Source: §1 · pp. 1–4
  • Gemini Robotics builds on Gemini 2.0 to connect multimodal understanding with embodied reasoning and physical robot action.The approach requires both understanding geometric and temporal-spatial details and grounding that reasoning in contact physics, dynamics, and real-world interaction.Source: §1 · p. 2
  • Gemini Robotics-ER provides embodied VLM capabilities spanning 3D perception, pointing, trajectory and grasp prediction, and robot state or affordance reasoning.These capabilities support robotics applications through zero-shot robot code generation and few-shot in-context learning.Source: §1 · p. 3
  • Gemini Robotics is a generalist Vision-Language-Action model for high-frequency dexterous control, diverse manipulation tasks, language following, and distribution-shift generalization.The model integrates robot action data and is presented as robust across variations in instructions, visuals, motions, tasks, and embodiments.Source: §1 · p. 3
  • Optional specialization adapts Gemini Robotics to extreme dexterity, difficult generalization settings, and completely new robot embodiments.The report also discusses responsible development, including societal risks and potential mitigation for robotics foundation models.Source: §1 · p. 3
  • The report introduces ERQA, an open-source benchmark designed to evaluate embodied reasoning beyond atomic visual capabilities.ERQA includes questions requiring multi-frame object registration, affordance reasoning, and understanding of 3D relationships.Source: §1 · p. 3

2. Embodied Reasoning with Gemini 2.0

This section examines how Gemini 2.0 enables embodied reasoning grounded in real-world objects, spatial concepts, and robotics tasks, spanning embodied perception and robot control. It also covers Gemini Robotics’ dexterous manipulation, precise language following, and generalization across visual, instructional, and action variations.

2.1. Embodied Reasoning Question Answering (ERQA) Benchmark

  • Gemini 2.0 models achieve a new state-of-the-art on ERQA, RealworldQA, and BLINK within their respective model classes, with ERQA the most challenging benchmark.ERQA contains 400 questions spanning spatial, trajectory, action, state-estimation, pointing, multi-view, and task reasoning, and 28% use multiple images.Source: §2.1 · pp. 4–5, Table 1 · p. 5
  • Chain-of-Thought prompting improves Gemini 2.0 performance on ERQA: Flash exceeds Pro Experimental without CoT, while CoT further improves Pro Experimental by grounding spatial observations into step-by-step reasoning.Reasoning traces show the model precisely grounding spatial understanding in images before performing complex embodied reasoning.Source: §2.1 · pp. 5–6, Table 2 · p. 5, Figure 5 · p. 6

2.2. Gemini 2.0’s Embodied Reasoning Capabilities

  • Gemini 2.0 and Gemini Robotics-ER unify open-vocabulary object detection, pointing, trajectory prediction, and top-down grasp prediction, enabling semantic generalization beyond specialized expert models.Queries can target objects, parts, affordances, free space, and spatial concepts; Gemini Robotics-ER extends pointing to grasp poses represented by y, x, and rotation angle θ.Source: §2.2 · pp. 6–9
  • Gemini 2.0 significantly outperforms state-of-the-art VLMs on three pointing benchmarks, while Gemini Robotics-ER surpasses Molmo on two of the three subtasks.The evaluated benchmarks are Paco-LVIS, Pixmo-Point, and Where2place.Source: §2.2 · p. 8, Table 3 · p. 8
  • Gemini Robotics-ER achieves a new state-of-the-art on SUN-RGBD 3D object detection, with performance comparable to existing expert models while supporting open-vocabulary queries rather than closed categories.Gemini 2.0 also performs multi-view point correspondence across substantially different views and predicts metric 3D bounding boxes from monocular images.Source: §2.2 · p. 9, Table 4 · p. 10

2.3. Gemini 2.0 Enables Zero and Few-Shot Robot Control

  • Gemini 2.0 can control robots without robot-action training by combining perception, state estimation, spatial reasoning, planning, and control in one model.In-context demonstrations further improve dexterous and bimanual behaviors, but zero-shot real-world performance remains limited for precise grasps and tasks such as dress folding.Source: §2.3 · pp. 11–13
  • On ALOHA 2 manipulation tasks, Gemini Robotics-ER averaged 53% success versus 27% for Gemini 2.0 Flash; with 10 demonstrations, it reached 65% in simulation and real-world control.Demonstrations improved performance, especially on dexterous tasks, by conditioning the model to output more precise bimanual trajectories.Source: §2.3 · pp. 12–13, Table 5 · p. 12

3.2. Gemini Robotics can solve diverse dexterous manipulation tasks out of the box

  • Gemini Robotics achieves success rates above 80% on half of 20 diverse dexterous tasks out of the box and is the only method with non-zero success on several challenging tasks.It particularly excels at deformable-object manipulation, while baselines struggle; the hardest tasks motivate later specialization for longer-horizon dexterous behavior.Source: §3.2 · pp. 15–16, Figure 16 · p. 16

3.3. Gemini Robotics can closely follow language instructions

  • Gemini Robotics more effectively follows precise language instructions than either baseline, especially in challenging scenes containing novel objects and fine-grained descriptions.The comparison covers 25 instructions across five diverse scenes, including novel scenes, objects, and receptacles; the PaliGemma-based π0 re-implement struggles with descriptive attributes and unseen objects.Source: §3.3 · pp. 16–17, Figure 17 · p. 17

3.4. Gemini Robotics brings Gemini’s generalization to the physical world

  • The generalization evaluation tests invariance to visual changes, instruction paraphrases and typos, languages and specificity, and changed initial conditions or object instances.These axes measure whether the model preserves task behavior despite scene changes, equivalent language formulations, and novel movement conditions or objects.Source: §3.4 · p. 17
  • Gemini Robotics consistently outperforms baselines across visual, instruction, and action variations, achieving non-zero performance when baselines fail catastrophically, including instructions in a new language.The benchmark comprises 85 tasks: 20% within the training distribution, 28% visual generalization, 28% instruction generalization, and 24% action generalization.Source: §3.4 · pp. 17, 19

4. Specializing and Adapting Gemini Robotics for Dexterity, Reasoning, and New Embodiments

Gemini Robotics is specialized beyond its generalist capabilities for long-horizon dexterity, enhanced reasoning, rapid task learning, and new robot embodiments. Across these settings, fine-tuning improves challenging-task performance, generalization, data efficiency, and embodiment transfer.

4. Specializing and Adapting Gemini Robotics for Dexterity, Reasoning, and New Embodiments

  • The section tests four deployment-relevant extensions: long-horizon specialization, reasoning-based generalization, rapid adaptation to new tasks, and adaptation to new embodiments.Long-horizon and reasoning experiments target future model improvements, while rapid task and embodiment adaptation target practical deployment.Source: §4 · p. 19

4.1. Long-horizon dexterity

  • Specialized Gemini Robotics models average 79% success across six long-horizon tasks, including 100% on lunch-box packing and four of six unseen hand-drawn spelling sketches.Baselines cannot consistently recognize and spell the images, while models trained from scratch achieve 0% across the evaluated tasks.Source: §4.1 · pp. 21–22

4.2. Enhanced reasoning and generalization

  • Reasoning-enhanced Gemini Robotics substantially improves success on out-of-distribution tasks requiring single-step reasoning, semantic knowledge, and spatial understanding.The model also produces intermediate trajectory traces resembling human-interpretable embodied reasoning traces.Source: §4.2 · pp. 22–24

4.3. Fast adaptation to new tasks

  • Fine-tuning reaches above 70% success on 7 of 8 new tasks with at most 100 demonstrations, including 100% success on two tasks.Baselines remain competitive on easier tasks but struggle on difficult origami and lunch-box tasks with limited demonstrations.Source: §4.3 · p. 24, Figure 26 · p. 24

4.4. Adaptation to new embodiments

  • After fine-tuning, Gemini Robotics transfers to Franka and Apollo robots, achieving 63% average success on considered Franka tasks and substantially outperforming diffusion baselines under disturbances.The adapted model is on par or slightly better on in-distribution tasks and retains robustness across visual and action variations.Source: §4.4 · p. 25, Figure 27 · p. 25

5. Responsible Development and Safety

The report treats robotics safety as encompassing semantic action safety alongside inherited content-safety controls and traditional low-level physical safeguards. Evaluations indicate strong semantic safety understanding, with constitutional methods and post-training improving robustness, while the authors characterize these findings as initial assurances.

Content safety

  • 96% rejection of bias-inducing pointing queries, versus 20% for the baseline, after supervised fine-tuning of Gemini 2.0 and Gemini Robotics-ER.The training teaches the models when not to generalize beyond image evidence for pointing outputs.Source: §5 · p. 26

Safety framework

  • Robotics safety requires semantic action safeguards for open-domain physical constraints in addition to traditional collision, force, and control mechanisms.Examples include avoiding unsafe object placement, allergen exposure, and improper handling of fragile objects.Source: §5 · p. 26

Safety evaluation

  • Gemini 2.0 Flash and Gemini Robotics-ER perform similarly on visual and real-world-injury safety QA, while constitutional methods improve performance and mitigate adversarial-prompt degradation.The alignment metric is binary classification accuracy against human safety assessments.Source: §5 · pp. 27–28, Figure 29 · p. 27
  • ASIMOV safety data combines visual and text-only question-answering instances, including scenarios derived from real-world injury records.Gemini Robotics-ER models are post-trained on these instances.Source: §5 · p. 27

Responsible development

  • The authors report only initial assurances that non-robotics safety standards extend to embodied models and emphasize continued mitigation, monitoring, and societal-impact management.The report identifies safety and wider societal implications as ongoing responsibilities for deployment.Source: §5 · p. 28

6. Discussion

The discussion presents Gemini Robotics-ER and Gemini Robotics as complementary models for transferring Gemini 2.0’s reasoning into physical interaction. The models support embodied reasoning, dexterous manipulation, open-vocabulary and zero-shot generalization, and adaptation across tasks and embodiments, while important capability limitations remain.

Gemini Robotics-ER

  • Gemini Robotics-ER advances spatial understanding, trajectory prediction, multi-view correspondence, and precise pointing, validated on a new open-source embodied-reasoning benchmark.The resulting model supports zero-shot and few-shot adaptation for perception, planning, and robot-control code generation.Source: §6 · p. 28

Gemini Robotics

  • Gemini Robotics performs diverse dexterous manipulation, follows open-vocabulary instructions, and exhibits strong zero-shot generalization.The report attributes this success to embodied reasoning, robotics-specific training data, and a low-latency control architecture.Source: §6 · p. 28

Specialization and adaptation

  • Fine-tuning enables Gemini Robotics to specialize to new tasks and embodiments, achieve extreme dexterity, and generalize in challenging scenarios.These adaptations are presented as optional mechanisms for translating foundational capabilities into real-world applications.Source: §6 · p. 28

Limitations and future work

  • Gemini 2.0 may struggle with spatial grounding across long videos, and its point and box predictions may lack precision for fine-grained robot control.The authors identify these as capability limitations requiring improvement.Source: §6 · p. 28
  • Future work targets multi-step dexterous reasoning, simulation-based data generation, real-world transfer, and reduced data requirements for adapting to new robot types.The longer-term goal is zero-shot cross-embodiment transfer to novel robotic platforms.Source: §6 · p. 29

Conclusion

  • The authors frame Gemini Robotics as a step toward general-purpose autonomous AI that could broaden robotics applications while requiring continued safety and societal-impact consideration.The conclusion emphasizes responsible development as the technology matures.Source: §6 · p. 29

7. Contributions and Acknowledgments

The report’s acknowledgments list the large Gemini Robotics Team and the broader set of authors who contributed to the work. The credited contributors span the team roster presented across the report’s author sections.

Authors

  • The author list includes contributors from Saminda Abeyruwan through Jan Humplik.This passage presents the first segment of the report’s author roster.Source: Authors · p. 36
  • The author list continues from Atil Iscen through Jonathan Tompson, followed by Vincent Vanhoucke through Peng Xu.These passages complete the remaining author segments supplied for the acknowledgments section.Source: Authors · pp. 36–37

A. Model Card

The model card documents Gemini Robotics-ER and Gemini Robotics, their inputs and outputs, training data, spatial conventions, robot-task settings, and the zero-shot control procedure. It also records the benchmark and evaluation sections covering their reported results.

A. Model Card

  • Gemini Robotics-ER is a vision-language model for world understanding, while Gemini Robotics is a vision-language-action model for general-purpose manipulation across tasks, scenes, and robots.Both models are documented as state-of-the-art systems in the model card.Source: A. Model Card · p. 38
  • Both models take text and images as inputs and are trained on images, text, robot sensor data, and action data.Gemini Robotics-ER outputs text such as coordinates, while Gemini Robotics outputs text describing robot actions.Source: A. Model Card · p. 38
  • The model card describes safety and quality filtering, including sensitive-data filtering and synthetic captions paired with each image.Synthetic captions are generated using Gemini and FlexCap models to expose additional image details during training.Source: A. Model Card · p. 38

B.1. Spatial Understanding Conventions and Prompts

  • Spatial outputs use normalized image coordinates, point tuples, 3D box parameters with Euler angles, and top-down grasp points with rotation angles.Two-dimensional coordinates are normalized as integers from 0 to 1000, while grasp rotations range from −90 to 90 degrees.Source: §B.1 · p. 39

B.2. Pointing Benchmark Comparisons

  • The pointing benchmark measures the percentage of predicted points inside ground-truth masks, using circular approximations for Pixmo-Point and parsed instruction-based outputs for comparison models.The supplied passage specifies a radius-25 circular approximation for Pixmo-Point masks.Source: §B.2 · p. 39

B.3.1. ALOHA 2 Robot Task Descriptions

  • ALOHA 2 evaluations include randomized simulated and real-world manipulation tasks such as banana handover, fruit placement, dress folding, and wiping.The task descriptions specify success conditions for representative simulated and real tasks.Source: §B.3.1 · p. 39, §B.3.1.1 · p. 40, §B.3.1.2 · pp. 40–41

B.3.2. System Prompt for Gemini during zero-shot robot control

  • The zero-shot control prompt has Gemini describe the scene, plan concise execution steps, issue robot-API code, inspect the resulting state, and re-plan after failures until completion.The prompt also enforces reachability, visibility, grasp, clearance, and collision-avoidance guidelines.Source: §B.3.2 · pp. 41–42
  • The documented robot workspace uses x, y, and z world axes, a 0.80-by-0.40-meter table, and distinct reachable x ranges for the left and right arms.The API exposes object detection, grasp-pose prediction, gripper control, movement, state description, and reset operations.Source: §B.3.2 · pp. 41–43

B.3.3. Sample output from Gemini during zero-shot robot control

The appendix illustrates Gemini’s zero-shot robot-control behavior through examples of planning, grasping, lifting, and error recovery. It also defines evaluation metrics that combine binary task success with continuous progress.

B.3.3. Sample output from Gemini during zero-shot robot control

  • Gemini produces planning, grasping, and lifting outputs while carrying out robot-control tasks.The examples distinguish planning, code, analysis, and environment feedback in the model’s interaction trace.Source: §B.3.3 · p. 44, Figure 32 · p. 44, Figure 33 · p. 45
  • Gemini can detect errors and retry during robot-control tasks.Source: Figure 34 · p. 46

C.1. Evaluation procedure

  • Evaluations use repeated randomized A/B trials and pairwise t-tests to reduce environmental and operator bias.Each task is run multiple times for each model, with models evaluated back-to-back in random order.Source: §C.1 · p. 47
  • Performance is measured by binary success and a 0–1 progress score capturing partial completion on difficult tasks.The progress metric complements success/failure labels for long-horizon, dexterous, and generalization tasks.Source: §C.1 · p. 47

C.1.1. Evaluation tasks to test out-of-the-box in-distribution performance

The evaluation suite tests in-distribution performance, instruction following, and generalization across language, visual conditions, object instances, and positions. These tasks vary instructions and scenes beyond the training distribution to probe robustness.

C.1.1. Evaluation tasks to test out-of-the-box in-distribution performance

C.1.2. Evaluation tasks for instruction following analysis

C.1.3.1 Visual and Instruction generalization tasks

  • Instruction generalization uses typos, Spanish translations, rephrasing, and descriptive modifiers across four lunch-bag tasks.Source: §C.1.3.1 · p. 50, Figure 37 · p. 50
  • Generalization tests vary distractors, backgrounds, lighting, object positions, and target-object color, shape, or size.Visual variations are outside training data, while action generalization spans six tasks and two variation axes.Source: §C.1.3.1 · p. 50, §C.1.3.2 · p. 51, Figure 38 · p. 51

C.1.3.3 Task and progress definition

The appendix defines continuous progress scores for task-specific partial completion and documents the baseline implementations used for comparison. Figure 40 reports that Gemini Robotics outperforms baselines across the tested generalization variations.

C.1.3.3 Task and progress definition

  • Progress scores assign graded credit to intermediate task states, complementing binary success rates.Examples include partial zipper opening, placing subsets of blocks, and incomplete object manipulation.Source: §C.1.3.3 · pp. 51–53
  • Gemini Robotics consistently outperforms baselines across instruction, visual, and action variations, retaining non-zero performance where baselines catastrophically fail.Source: Figure 40 · p. 53
  • The task definitions cover manipulation behaviors including tightening bottle caps, opening drawers, folding dresses, handovers, and compartment placement.Source: §C.1.3.3 · p. 53

C.2. Baselines

  • The baseline suite includes a 𝜋0 re-implementation, multi-task diffusion, and single-task diffusion policies.The 𝜋0 re-implementation is trained on the same data mixture as Gemini Robotics and is reported because it outperforms the openpi checkpoint in internal evaluations.Source: §C.2 · pp. 54–55, Figure 41 · p. 54

D.1.1. Evaluation procedure

The evaluation procedure uses repeated trials and task-specific progress scores to assess dexterous and long-horizon behavior. Additional reasoning evaluations test indirect instructions involving semantic matching and object properties unseen during training.

D.1.1. Evaluation procedure

  • Evaluations primarily use 20 trials per task and model, with 12 trials for the spelling game split between printed cards and hand-drawn sketches.Source: §D.1.1 · p. 55
  • Progress scores provide fine-grained comparisons across six dexterous tasks beyond success rates.Figure 42 reports average task progress for new, dexterous, and long-horizon tasks.Source: §D.1.1 · pp. 55–56, Figure 42 · p. 55
  • Task-specific progress scales grade partial completion for card play, salad preparation, lunch-box packing, and spelling.Source: §D.1.1 · p. 56

D.2.1. Evaluation procedure

  • Reasoning-enhanced and vanilla models are evaluated over 100 trials across eight tasks covering one-step reasoning, semantic generalization, and spatial understanding.One-step tasks require indirect interpretation such as matching colors, sorting by properties, or selecting the correct item among distractors.Source: §D.2.1 · p. 56, §D.2.1.1 · pp. 56–57

D.2.1.2 Semantic Generalization Tasks

The evaluation tests whether Gemini Robotics handles unseen semantic and spatial concepts in natural-language manipulation instructions. Tasks require identifying novel objects or attributes and applying relative or absolute spatial relationships.

D.2.1.2 Semantic Generalization Tasks

  • Semantic generalization tasks require selecting targets using concepts unseen during training, including “Japanese fish delicacy” and “full.”The sushi must be distinguished from distractors, and the full bowl from empty bowls.Source: §D.2.1.2 · p. 57

D.2.1.3 Spatial Understanding Tasks

  • Spatial generalization tasks evaluate relative and absolute relationships, including “smallest” and “top left” or “bottom left.”The tasks combine novel object descriptions with spatial concepts that are unseen, rare, or common in training.Source: §D.2.1.3 · p. 57

D.3.1. Tasks and evaluation details

The fast-adaptation evaluation measures how Gemini Robotics learns short-horizon tasks from limited demonstrations. It uses unseen tasks from prior dexterous-task segments and reports success-rate curves across demonstration budgets.

D.3.1. Tasks and evaluation details

  • The adaptation test fine-tunes a Section 3 checkpoint that has never seen demonstrations from the evaluated tasks, providing a fair test of new-task adaptation.The tasks are shorter segments sampled from the long-horizon dexterous tasks in Section 4.1.Source: §D.3.1 · pp. 57–59
  • Gemini Robotics is evaluated on eight short-horizon tasks, including card handling, food preparation, container packing, zipping, and an origami fold.The tasks range from drawing and playing cards to sealing a container and making the first origami fold.Source: Figure 43 · p. 58, §D.3.1 · p. 59
  • Success rate is measured after 5, 20, and 100 demonstration episodes using 10 trials per point, without a progress score for these short-horizon tasks.Figure 26 reports curves for Gemini Robotics and baselines as demonstration data increases.Source: §D.3.1 · p. 59

D.4.1. Tasks description

The bi-arm Franka evaluation covers four dexterous manipulation tasks and defines graded progress scores for partial and complete task execution. It also varies scene appearance to test visual generalization.

D.4.1. Tasks description

  • The bi-arm Franka evaluation covers tape hanging, plug insertion, round-belt assembly, and timing-belt assembly tasks.These tasks involve handovers, stabilization, flexible-belt manipulation, and coordinated forceful actions.Source: §D.4.1 · pp. 59–60, Figure 44 · p. 60
  • Progress scores assign graded credit for intermediate achievements such as grasping, handover, alignment, insertion, and complete placement.For example, plug insertion ranges from 0.2 for grasping the plug to 1.0 for insertion, illumination, and arm withdrawal.Source: §D.4.1 · pp. 59–60
  • Timing-belt assembly requires coordinated bi-arm action and roughly 40N of force on a blue handle to secure the belt.Its score distinguishes partial placement, grasping, and successful insertion on both wheels.Source: §D.4.1 · pp. 60–61

D.4.2.1 Visual generalization tasks

  • Visual generalization changes scene appearance by adding novel distractors, altering backgrounds, and changing lighting conditions.The benchmark runs 20 trials per task with initial conditions based on training data.Source: §D.4.2.1 · p. 61, §D.4.2 · p. 61

D.4.2.2 Action generalization tasks

Action generalization evaluates Gemini Robotics when object locations and object instances differ from training. Figure 47 reports success-rate comparisons with a diffusion baseline across the generalization benchmark.

D.4.2.2 Action generalization tasks

  • Action generalization changes object positions and uses instances with different appearances, shapes, or physical properties.These variations test whether the model can act beyond the configurations seen in training.Source: §D.4.2.2 · p. 61
  • Figure 47 breaks down success-rate metrics across the benchmark and reports that Gemini Robotics significantly outperforms the diffusion baseline.The figure complements the progress-score results reported in Figure 28.Source: §D.4.2.2 · p. 61, Figure 47 · p. 64

Related papers

Loading 2503.20020v1…