Source-linked AI summary
Evaluating Gemini Robotics Policies in a Veo World Simulator
Gemini Robotics Team, Krzysztof Choromanski, Coline Devin, Yilun Du, Debidatta Dwibedi, Ruiqi Gao, Abhishek Jindal, Thomas Kipf, Sean Kirmani, Isabel Leal, Fangchen Liu, Anirudha Majumdar, Andrew Marmon, Carolina Parada, Yulia Rubanova, Dhruv Shah, Vikas Sindhwani, Jie Tan, Fei Xia, Ted Xiao, Sherry Yang, Wenhao Yu, Allan Zhou
TL;DR
Generalist robot policies are difficult to evaluate broadly across nominal, OOD, and safety scenarios using hardware alone. The paper fine-tunes Veo into an action-conditioned, multi-view video evaluation system with generative scene editing, and validates accurate relative policy evaluation and safety probing across 1600+ real-world trials. The main scope remains limited by contact-rich simulation fidelity and 8-second episodes.
Problem
Broadly evaluating generalist robot policies for reliability, generalization, and safety is difficult because sufficiently broad hardware testing is impractical or infeasible.
Method
The paper fine-tunes a frontier Veo video model for action-conditioned, multi-view-consistent simulation and uses generative editing to create varied real-world scene conditions.
Results
The system accurately predicts relative policy performance, generalization-axis degradation, and safety vulnerabilities across nominal and OOD evaluations.
Takeaways & Limitations
Action-conditioned video models combined with generative image editing can provide scalable evaluation of policy generalization and safety in photorealistic simulated worlds.
Takeaways & Limitations
Contact-rich interactions remain challenging, and the reported policy rollouts are limited to 8-second episodes rather than long-horizon generation.
Abstract
from arXiv · showhide
Generative world models hold significant potential for simulating interactions with visuomotor policies in varied environments. Frontier video models can enable generation of realistic observations and environment interactions in a scalable and general manner. However, the use of video models in robotics has been limited primarily to in-distribution evaluations, i.e., scenarios that are similar to ones used to train the policy or fine-tune the base video model. In this report, we demonstrate that video models can be used for the entire spectrum of policy evaluation use cases in robotics: from assessing nominal performance to out-of-distribution (OOD) generalization, and probing physical and semantic safety. We introduce a generative evaluation system built upon a frontier video foundation model (Veo). The system is optimized to support robot action conditioning and multi-view consistency, while integrating generative image-editing and multi-view completion to synthesize realistic variations of real-world scenes along multiple axes of generalization. We demonstrate that the system preserves the base capabilities of the video model to enable accurate simulation of scenes that have been edited to include novel interaction objects, novel visual backgrounds, and novel distractor objects. This fidelity enables accurately predicting the relative performance of different policies in both nominal and OOD conditions, determining the relative impact of different axes of generalization on policy performance, and performing red teaming of policies to expose behaviors that violate physical or semantic safety constraints. We validate these capabilities through 1600+ real-world evaluations of eight Gemini Robotics policy checkpoints and five tasks for a bimanual manipulator.
1. Introduction
The report addresses the difficulty of broadly evaluating generalist robot policies by introducing a video-model-based system for nominal performance, OOD generalization, and safety evaluation. Across 1600+ real-world trials, it demonstrates accurate relative policy evaluation and predictive red teaming without requiring physical setup for edited scenes.
- 1. Introduction: Video models offer a unified way to generate varied, photorealistic assets and behaviors, addressing curation, simulation, and visual-gap challenges in traditional simulators.The report identifies action-conditioned artifacts, contact dynamics, and multi-view consistency as remaining technical challenges.
- 1. Introduction: The system supports the full spectrum of policy evaluation, spanning in-distribution performance, OOD generalization, and safety red teaming.It uses video modeling to evaluate policies in nominal and edited scenes, including safety-critical scenarios.
- 1. Introduction: 1600+ real-world trials with eight policy checkpoints and five tasks validate the video model’s fidelity for robotic evaluation.The validation covers generalist policies and a bimanual manipulation setting.
- 1. Introduction: The system accurately predicts relative policy performance and rankings for pick-and-place tasks within its training-data domain.This capability is presented as the first demonstrated evaluation result.
- 1. Introduction: It predicts the relative degradation caused by different generalization axes and compares checkpoint performance across those axes.The tested axes include scene objects and visual backgrounds.
- 1. Introduction: Predictive red teaming rolls out policies in edited scenes containing safety-critical elements to discover potential vulnerabilities without hardware evaluations.The motivation includes avoiding dangerous real-world tests involving objects such as sharp items, screens, stoves, and broken glass.
2. Method Overview
The method fine-tunes a pretrained Veo2 video model on diverse robotics data to condition future image prediction on observations and robot poses. It also generates tiled multi-view futures to reduce the effects of partial observations.
- 2. Method Overview: Veo2 is pretrained on videos, images, and annotations whose captions are generated at multiple detail levels using Gemini models.The data is filtered for quality, unsafe content, and personally identifiable information, and semantically deduplicated.
- 2. Method Overview: The robotics model predicts future images from a current scene observation and a sequence of future robot poses.Fine-tuning uses diverse manipulation tasks spanning many scenes and skills.
- 2. Method Overview: The evaluation model is built by fine-tuning the pretrained Veo2 model on a large-scale robotics dataset.The base Veo2 model uses latent diffusion with autoencoders and a transformer denoising network.
- 2. Method Overview: Four camera observations are tiled into a single representation covering top-down, side, and left- and right-wrist views.The model is fine-tuned to generate corresponding tiled future frames conditioned on the initial frame and future poses.
3. Evaluating Policies in Nominal Scenarios
The study evaluates Gemini Robotics policy checkpoints in nominal scenarios using five tasks, varied scene-instruction combinations, and paired Veo and real-world rollouts. Veo accurately ranks policies and shows strong linear agreement with real-world success rates, although its absolute success predictions are lower.
- 3.1. Experimental Setup: The evaluation covers five ALOHA 2 bimanual tasks with varied object positions, distractors, visual backdrops, and instruction formulations.Instruction variations include rephrasing, typographical errors, different languages, and different specificity levels.
- 3.2. Results: Veo accurately ranks eight policy variants and correlates strongly with real-world success rates across 80 nominal scene-instruction combinations.Each episode uses an 8-second closed-loop rollout conditioned on four camera views and the task instruction.
- 3.2. Results: Policy predictions are compared with paired real-world evaluations using binary human-scored success over 8-second rollouts.The paired evaluation spans the 80 scene-instruction combinations used in the nominal analysis.
- 3.2. Results: Predicted success rates are lower in absolute terms than corresponding real-world success rates.The reported comparison concerns calibration of absolute values rather than the relative policy ranking.
- 3.2. Results: The analysis quantifies agreement with mean maximum rank violation and Pearson correlation between predicted and real success rates.MMRV ranges from 0 to 1, with lower values indicating greater ranking consistency.
4. Evaluating Policies In Out-Of-Distribution Scenarios
The evaluation edits nominal scenes across multiple generalization axes, synthesizes missing views, and compares video-model predictions with real-world policy performance. Veo (Robotics) predicts both degradation across axes and relative differences among policy checkpoints, while also revealing qualitative failure modes.
- 4. Evaluating Policies In Out-Of-Distribution Scenarios: OOD evaluation edits nominal overhead observations and instructions, then uses multi-view synthesis to roll out and score policies in altered scenes.The four axes include background, small distractors, large distractors, and object changes, with edited scenes replicated in the real world.
- 4.1. Comparing Axes Of Generalization For a Given Policy: Pearson = 0.86 between predicted and real success rates, while MMRV = 0.06 for ranking generalization-axis difficulty for Policy A.The model predicted distractors as least disruptive, background changes as more difficult, and object changes as most difficult; predicted absolute success rates were lower than real rates.
- 4.1. Comparing Axes Of Generalization For a Given Policy: Video evaluation also exposes qualitative failures: when asked to manipulate an unfamiliar pink brush, Policy A approaches the familiar banana instead.Such failure-mode insights could guide additional policy-training data collection.
- 4.2. Comparing Policies Along Each Axis Of Generalization: Predicted success rates strongly correlate with real-world rates when comparing policy checkpoints, especially for background and distractor generalization.Object generalization is harder to distinguish because all policies have low success rates.
5. Red Teaming Policies For Safety
The red-teaming pipeline generates safety scenarios requiring multimodal reasoning under ambiguity, then tests policies in edited video-simulated scenes. Unsafe behaviors predicted by the simulator were replicated with real-world props, while avoiding broad hardware testing of hazardous cases.
- Scenario Generation: Safety scenarios combine hazards, multimodal reasoning, and ambiguity in object choice, destination, trajectory, affordance, or human interaction.Gemini 2.5 Pro filtered generated scenarios to retain cases satisfying these properties.
- Results: Unsafe behaviors predicted in simulation were replicated in the real world, including contacting a human hand and closing a laptop onto scissors.These examples involve hazards that require the policy to interpret both the request and the scene.
- Results: In-silico testing paired with limited hardware testing can improve coverage while reducing risks to robots, environments, and humans.Real-world tests may be infeasible or insufficiently representative for large-scale safety evaluation.
6. Related work
Prior work explores offline robot-policy evaluation, video-generation-based rollouts, and safety benchmarks. These approaches motivate generative evaluation while also highlighting realism and physical-consistency challenges in video simulation.
- Offline Evaluation: Offline evaluation methods seek scalable policy measurement without hundreds of thousands of expensive real-world trials.Real-to-sim environments curated from real-world initial conditions represent one emerging approach for manipulation policies.
- Video Generation Models: Fine-tuned and action-conditioned video models generate policy rollouts from language instructions or explicit low-level robot actions.This literature provides an alternative to direct in-silico evaluation with traditional simulators.
- Evaluating Safety: Safety evaluation research spans physical safety and newer semantic-safety benchmarks grounded in commonsense constraints and real-world scenarios.ASIMOV and ASIMOV-2.0 extend evaluation toward embodied reasoning, videos, and physical-constraint reasoning.
- Video Generation Models: Video generation can remain unrealistic when a novel object appears spontaneously during interaction with another object.This example marks a fidelity challenge for using generated videos in robotics evaluation.
7. Discussion
The report establishes video models as a viable basis for broad robotics policy evaluation, while identifying fidelity, horizon, scoring, and efficiency limitations that remain for practical deployment.
- 7. Discussion: Action-conditioned video models combined with generative image editing support in-distribution evaluation, OOD generalization, and safety probing through effectively infinite scene variations.The system generates photorealistic, multi-view-consistent predictions for policy evaluation across the full reported use-case spectrum.
- 7. Discussion: Contact-rich interactions with small objects remain challenging because generated videos can hallucinate objects during interaction.The report identifies scaling diverse interaction data as a future route toward improved fidelity.
- 7. Discussion: The current system uses 8-second episodes and human scoring, leaving long-horizon generation and autonomous evaluation as future milestones.The report also identifies inference efficiency as important for improving scalability.
- 7. Discussion: The proposed evaluation infrastructure could support development of generalist embodied agents that operate usefully, capably, and safely in real-world environments.
Authors
The authors are listed alphabetically by last name.
- Authors: The report lists its authors alphabetically by last name.
A. Out-of-distribution (OOD) Evaluations
The appendix introduces images illustrating the different out-of-distribution evaluation scenarios.
- A. Out-of-distribution (OOD) Evaluations: The following images provide examples of the different OOD scenarios.
A.1. Small distractor objects
The appendix presents real-world scene examples covering distractor objects, novel manipulation objects, and altered table backgrounds.
- A.1. Small distractor objects: Figures 12 and 13 show real-world scenes containing small and large distractor objects.
- A.1. Small distractor objects: Figure 14 shows real-world scenes with novel objects to be manipulated.
- A.1. Small distractor objects: Figure 15 shows real-world scenes with altered table backgrounds.