PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Gemini Robotics Policies in a Veo World Simulator

Gemini Robotics Team, Krzysztof Choromanski, Coline Devin, Yilun Du, Debidatta Dwibedi, Ruiqi Gao, Abhishek Jindal, Thomas Kipf, Sean Kirmani, Isabel Leal, Fangchen Liu, Anirudha Majumdar, Andrew Marmon, Carolina Parada, Yulia Rubanova, Dhruv Shah, Vikas Sindhwani, Jie Tan, Fei Xia, Ted Xiao, Sherry Yang, Wenhao Yu, Allan Zhou

arXiv:2512.10675v2cs.ROcs.AIcs.CVcs.LG

TL;DR

Broadly evaluating robot policies for reliability, generalization, and safety is difficult because hardware coverage is impractical. This paper builds an action-conditioned, multi-view Veo evaluation system with edited scene variations and reports accurate assessments of policy rankings, generalization degradation, and potential safety vulnerabilities.

  • Problem

    Evaluating generalist robot policies across reliability, generalization, and safety is challenging because broad hardware evaluations are impractical and safety evaluations may be infeasible.

    Source: §1 · p. 1

  • Method

    The system uses action-conditioned, multi-view video simulation with generative editing to create varied scenes for in-distribution, OOD, and safety evaluation.

    Source: §1 · p. 2

  • Results

    The system accurately predicts policy rankings in in-domain pick-and-place tasks, relative degradation across generalization axes, and potential safety vulnerabilities from edited-scene rollouts.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The report demonstrates a path toward scalable evaluation of robot-policy generalization and safety in video-simulated worlds.

    Source: §1 · p. 3

    Simulating contact-rich interactions, particularly with small objects, remains challenging.

    Source: §7 · p. 12

Abstract

from arXiv · show

Generative world models hold significant potential for simulating interactions with visuomotor policies in varied environments. Frontier video models can enable generation of realistic observations and environment interactions in a scalable and general manner. However, the use of video models in robotics has been limited primarily to in-distribution evaluations, i.e., scenarios that are similar to ones used to train the policy or fine-tune the base video model. In this report, we demonstrate that video models can be used for the entire spectrum of policy evaluation use cases in robotics: from assessing nominal performance to out-of-distribution (OOD) generalization, and probing physical and semantic safety. We introduce a generative evaluation system built upon a frontier video foundation model (Veo). The system is optimized to support robot action conditioning and multi-view consistency, while integrating generative image-editing and multi-view completion to synthesize realistic variations of real-world scenes along multiple axes of generalization. We demonstrate that the system preserves the base capabilities of the video model to enable accurate simulation of scenes that have been edited to include novel interaction objects, novel visual backgrounds, and novel distractor objects. This fidelity enables accurately predicting the relative performance of different policies in both nominal and OOD conditions, determining the relative impact of different axes of generalization on policy performance, and performing red teaming of policies to expose behaviors that violate physical or semantic safety constraints. We validate these capabilities through 1600+ real-world evaluations of eight Gemini Robotics policy checkpoints and five tasks for a bimanual manipulator.

1. Introduction

The report presents video-model simulation as a scalable way to evaluate generalist robot policies across nominal performance, out-of-distribution generalization, and safety. It builds on Veo with action-conditioned, multi-view-consistent simulation and edited scenes, and validates its predictions against real-world trials.

Introduction

  • Broad hardware evaluation of generalist policies is impractical, and safety evaluation may be infeasible; probing semantic safety can require hazardous real-world setups.The long-tail constraints include avoiding placing plastic on a stove or leaving broken glass on the floor, which could endanger people, the robot, or its environment.Source: §1 · p. 1
  • Traditional physics-based simulators require curated realistic assets, can struggle to simulate them accurately, and may need months of expert work to close the visual gap.The cited challenges include non-rigid objects and labor-intensive processes such as careful green screening (Li et al., 2024; Liu et al., 2023; Pumacay et al., 2024; Badithela et al. (2025); Li et al. (2024)).Source: §1 · pp. 1–2
  • Video models offer a unified route to simulating diverse assets and complex behaviors, but closed-loop action conditioning, contact dynamics, and multi-view consistency remain challenges.The report describes this as an early stage for video modeling in robotics and a path toward scalable evaluation of policy generalization and safety.Source: §1 · pp. 2–3
  • The system combines action-conditioned, multi-view-consistent video simulation with generative editing to create realistic scenes containing novel objects, backdrops, and safety-critical elements.It is designed to remain responsive to fine-grained robot control while supporting the full spectrum of policy evaluation, from in-distribution testing through OOD generalization and safety red teaming.Source: §1 · p. 2
  • Across 1600+ real-world trials, the system predicts in-domain policy rankings, relative degradation across generalization axes, and potential safety vulnerabilities.Evaluations covered eight policy checkpoints and five tasks; the reported in-domain ranking results concern pick-and-place tasks.Source: §1 · pp. 2–3

2. Method Overview

The system fine-tunes Veo2 for robot-conditioned video prediction and tiled multi-view generation, using a latent-diffusion model pretrained on curated video, image, and annotation data.

Model Architecture

  • Veo2 uses latent diffusion: autoencoders compress spatio-temporal data, then a transformer denoiser iteratively refines random noise into video (van den Oord and Roman, 2024; Veo Team, 2025).Source: §2 · p. 3

Training Data & Curation

  • Veo pretraining uses videos, images, and annotated captions, with quality and safety filtering, removal of personally identifiable information, and semantic deduplication (Veo Team, 2025; GeminiTeam et al., 2025).Source: §2 · p. 3

Action Conditioning

  • Robot-specific fine-tuning conditions Veo2 on a scene image and future robot poses to predict corresponding future images.The fine-tuning data spans diverse manipulation tasks and scenes.Source: §2 · p. 3

Multi-View Generation

  • To mitigate partial observations, Veo2 generates tiled future frames from top-down, side, and left- and right-wrist camera views, conditioned on the initial frame and future robot poses.Source: §2 · p. 3

3. Evaluating Policies in Nominal Scenarios

The section evaluates policies in nominal, in-distribution scenarios and finds that Veo (Robotics) predicts their relative performance, while using varied tasks, scenes, and instructions to test them.

3.1. Experimental Setup

  • The evaluation uses five ALOHA 2 bimanual tasks, varying object positions, distractors, backdrops, and instructions across 80 scene-instruction combinations scored for binary success.Instruction variations include rephrasing, typographical errors, another language, and different levels of specificity.Source: §3.1 · p. 4
  • The evaluated policies are end-to-end vision-language-action models based on Gemini Robotics On-Device, trained on expert demonstrations collected from ALOHA 2 robots over 12 months.The policies predict continuous actions at 50 Hz in 1-second chunks.Source: §3.1 · p. 4

3.2. Results

  • Veo (Robotics) accurately ranks eight GROD policy variants and shows strong linear correlation with real-world success, although predicted success rates are lower.Real-world and predicted outcomes were compared across 80 scene-instruction combinations using 8-second rollouts scored by human evaluators.Source: §3.2 · p. 5, Figure 4 · p. 5

4. Evaluating Policies In Out-Of-Distribution Scenarios

Veo (Robotics) predicted how different generalization axes affected policy performance and correlated with real-world success rates when comparing policies across those axes. Evaluations also revealed instruction-following failures on unfamiliar objects.

4. Evaluating Policies In Out-Of-Distribution Scenarios

  • OOD tests edited nominal overhead scenes across background, small and large distractor, and novel-object axes, with instructions updated for novel objects and matched real-world scenes used for validation.Edited observations were expanded into multi-view inputs with fine-tuned Veo2; Veo (Robotics) rollouts used the observations and language instruction and were scored for success or failure. Five checkpoints were evaluated.Source: §4 · pp. 6–7

4.1. Comparing Axes Of Generalization For a Given Policy

  • Policy A’s predicted ordering of generalization difficulty matched real-world evaluations (MMRV of 0.06), and predicted versus real success rates had a Pearson correlation of 0.86.Both evaluations ranked small and large distractors as least impactful, background changes as more impactful, and object changes as most impactful; predicted absolute success rates were lower than real rates.Source: Figure 8 · p. 8, §4.1 · p. 8
  • In object-generalization failures, Policy A often approached a familiar object instead of the unfamiliar instructed object.In the example, it approached the banana rather than the pink brush, suggesting a qualitative failure insight that could guide additional data collection.Source: §4.1 · p. 8

4.2. Comparing Policies Along Each Axis Of Generalization

  • Predicted success rates were strongly correlated with real-world rates, especially for background and distractor conditions; low object-condition success made policies harder to distinguish.Each plot compares policies for one axis of generalization.Source: Figure 9 · p. 8, §4.2 · pp. 8–9

5. Red Teaming Policies For Safety

The Veo (Robotics) model supports policy red teaming by generating safety-relevant scenes where rollout is needed to assess behavior, exposing unsafe actions that were also observed with real-world props.

Red Teaming Policies For Safety

  • Safety red teaming (Majumdar et al., 2025) uses synthetically edited scenes to find potentially unsafe policy behavior when scene semantics matter and policy rollout is necessary to assess safety.Instructions whose safety can be determined without rollout can instead be filtered by a policy-agnostic safety layer.Source: §5 · p. 9

Scenario Generation

  • Scenarios pair a user request with an image and are critic-filtered to include a hazard, require joint image-and-request reasoning, and contain ambiguity.Gemini 2.5 Pro served as the critic for scenarios in an ALOHA tabletop environment.Source: §5 · p. 10
  • Scenario ambiguity covers imprecise object choice or destination, safe grasp or trajectory requirements, and human presence that makes incorrect actions risky.Examples include choosing among objects when one is unsafe, avoiding an edge or person when placing an object, grasping a knife by its handle, and keeping a cup upright.Source: §5 · p. 10

Results

  • Fig. 10 surfaces Policy A unsafe behavior: reaching for a red block contacts a human hand, and closing a laptop over scissors may break its screen; real-prop tests reproduced both.The scenarios target risks that require policy rollout to assess.Source: §5 · p. 10
  • In-silico safety testing can reveal unsafe behavior without risking robots, environments, or people, while small-scale hardware tests can complement its realism and coverage limits.The combined approach can also test mitigation strategies.Source: §5 · p. 10

6. Related work

Prior work explores simulation and video generation for scalable robot-policy evaluation, while safety benchmarks are largely non-interactive. This report uses action-conditioned video simulation to study distribution shifts and closed-loop safety.

Offline Evaluation

  • Physics simulation and curated real-to-sim environments offer alternatives to costly real-world policy trials, but manipulation real-to-sim remains nascent and physics simulation is difficult to tune and scale.Simulation benchmarks standardize tasks, initial conditions, success criteria, and expert trajectories (Liu et al., 2023; Pumacay et al., 2024; Wang et al., 2025); real-to-sim evaluation has shown predictive signal in autonomous driving (Dosovitskiy et al., 2017).Source: §6 · pp. 10–11

Video Generation Models

  • Action-conditioned video models can generate policy rollouts from language or robot actions, and prior work uses them to evaluate real-data-trained policies on in-distribution tasks.This report extends evaluation to visual and semantic shifts and safety red-teaming; unlike edited-first-frame anomaly scoring (Majumdar et al., 2025), it simulates full episodes with an action-conditioned video model.Source: §6 · p. 11

Evaluating Safety

  • Existing semantic-safety benchmarks assess text, images, or videos non-interactively, whereas this report evaluates closed-loop policy safety as robot actions affect later time steps.Prior benchmarks include text-only and multimodal assessments and ASIMOV datasets grounded in real-world scenes and injury reports, later expanded with videos and physical-constraint reasoning (Sermanet et al., 2025; Jindal et al., 2025).Source: §6 · p. 11

7. Discussion

Action-conditioned video models trained on large-scale robotics data can support policy evaluation across in-distribution, out-of-distribution, and safety scenarios using photorealistic, multi-view-consistent simulations of varied scenes. Key next steps include improving interaction fidelity, long-horizon generation, automated scoring, and inference efficiency.

  • The system supports policy evaluation across in-distribution, out-of-distribution, and safety scenarios with photorealistic, multi-view-consistent simulations and effectively infinite scene variations.It combines video models trained on large-scale robotics datasets with generative image editing.Source: §7 · p. 12
  • Current limitations include artifacts in contact-rich interactions, 8-second rollouts, human-scored videos, and the need for more efficient video generation.The report identifies long-horizon (e.g., 1+ minutes) multi-view-consistent generation, VLM-based automated scoring, and optimized architectures as development priorities; latent-action models offer a path to longer generation (Bruce et al., 2024), while optimized architectures may improve inference efficiency (Hafner et al., 2025).Source: §7 · p. 12
  • The work positions richly varied simulated worlds as infrastructure for developing generalist embodied agents that operate usefully, capably, and safely in real-world environments.The report describes the potential impact as evaluating robots in an infinitely rich and varied proxy of the world.Source: §7 · p. 12

Appendix · A. Out-of-distribution (OOD) Evaluations

The appendix provides images illustrating examples of different out-of-distribution scenarios.

A. Out-of-distribution (OOD) Evaluations

A.1. Small distractor objects

Figure 12 presents real-world scenes containing small distractor objects.

A.1. Small distractor objects

  • Figure 12 shows real-world scenes with small distractor objects.Source: Figure 12 · p. 16

A.2. Large distractor objects · A.3. Novel objects to be manipulated · A.4. Table Background

The appendix illustrates real-world scenes with large distractors, novel objects to be manipulated, and altered table backgrounds.

A.2. Large distractor objects

  • Figure 13 shows real-world scenes with large distractor objects.Source: Figure 13 · p. 17

A.3. Novel objects to be manipulated

  • Figure 14 shows real-world scenes with novel objects to be manipulated.Source: Figure 14 · p. 18

A.4. Table Background

  • Figure 15 shows real-world scenes with altered table backgrounds.Source: Figure 15 · p. 19

Related papers

Loading 2512.10675v2…