PaperlayerView exact revision on arXiv

Source-linked AI summary

SIMA 2: A Generalist Embodied Agent for Virtual Worlds

SIMA team, Adrian Bolton, Alexander Lerchner, Alexandra Cordell, Alexandre Moufarek, Andrew Bolt, Andrew Lampinen, Anna Mitenkova, Arne Olav Hallingstad, Bojan Vujatovic, Bonnie Li, Cong Lu, Daan Wierstra, Daniel P. Sawyer, Daniel Slater, David Reichert, Davide Vercelli, Demis Hassabis, Drew A. Hudson, Duncan Williams, Ed Hirst, Fabio Pardo, Felix Hill, Frederic Besse, Hannah Openshaw, Harris Chan, Hubert Soyer, Jane X. Wang, Jeff Clune, John Agapiou, John Reid, Joseph Marino, Junkyung Kim, Karol Gregor, Kaustubh Sridhar, Kay McKinney, Laura Kampis, Lei M. Zhang, Loic Matthey, Luyu Wang, Maria Abi Raad, Maria Loks-Thompson, Martin Engelcke, Matija Kecman, Matthew Jackson, Maxime Gazeau, Ollie Purkiss, Oscar Knagg, Peter Stys, Piermaria Mendolicchio, Raia Hadsell, Rosemary Ke, Ryan Faulkner, Sarah Chakera, Satinder Singh Baveja, Shane Legg, Sheleem Kashem, Tayfun Terzi, Thomas Keck, Tim Harley, Tim Scholtes, Tyson Roberts, Volodymyr Mnih, Yulan Liu, Zhengdong Wang, Zoubin Ghahramani

arXiv:2512.04797v1cs.AIcs.RO

TL;DR

SIMA 2 targets the gap between passive foundation-model intelligence and active embodied behavior. Built on Gemini, it reasons, converses, and acts across diverse 3D worlds, substantially improves over SIMA 1, generalizes to unseen environments, and self-improves from generated experience. The paper supports a path toward versatile interactive agents while identifying important limits in long-horizon reasoning, memory, and precise control.

  • Problem

    Foundation models are strong at reasoning and language but remain disembodied, with deficits in the sensorimotor skills required for embodied interaction.

    Source: §1 · p. 2

  • Method

    SIMA 2 fine-tunes a Gemini foundation model with multimodal gameplay and bridge data so it can reason about goals, converse, and generate embodied actions.

    Source: §1 · p. 2, §3.3 · p. 7, §3.3.2 · p. 9

  • Results

    SIMA 2 reasons and acts across diverse 3D worlds, generalizes to unseen and photorealistic environments, and improves through self-generated experience.

    Source: §1 · p. 2, §5 · p. 26

  • Takeaways & Limitations

    The work supports generalist interactive agents that integrate reasoning with action and can continue learning in new environments.

    Source: §1 · p. 2, §5 · p. 26

    SIMA 2 remains limited on very long-horizon tasks, has short interaction memory, and struggles with precise low-level actions and robust visual understanding.

    Source: §5 · p. 26

Abstract

from arXiv · show

We introduce SIMA 2, a generalist embodied agent that understands and acts in a wide variety of 3D virtual worlds. Built upon a Gemini foundation model, SIMA 2 represents a significant step toward active, goal-directed interaction within an embodied environment. Unlike prior work (e.g., SIMA 1) limited to simple language commands, SIMA 2 acts as an interactive partner, capable of reasoning about high-level goals, conversing with the user, and handling complex instructions given through language and images. Across a diverse portfolio of games, SIMA 2 substantially closes the gap with human performance and demonstrates robust generalization to previously unseen environments, all while retaining the base model's core reasoning capabilities. Furthermore, we demonstrate a capacity for open-ended self-improvement: by leveraging Gemini to generate tasks and provide rewards, SIMA 2 can autonomously learn new skills from scratch in a new environment. This work validates a path toward creating versatile and continuously learning agents for both virtual and, eventually, physical worlds.

1. Introduction

SIMA 2 addresses the gap between strong disembodied reasoning and weak embodied action by combining Gemini-based reasoning with interaction in diverse 3D worlds. It advances beyond SIMA 1 toward conversational, goal-directed behavior, broad generalization, and self-improvement.

Motivation

  • Foundation models excel at high-level reasoning but remain fundamentally disembodied and weak at low-level sensorimotor skills.This reflects a modern form of Moravec’s Paradox.Source: §1 · p. 2
  • Embodied agents must perceive environments, act to change them, and learn from the resulting consequences.Source: §1 · p. 2

Prior Work

  • SIMA 1 generalized language instruction across diverse 3D games but remained limited to short commands, lacked dialogue and action reasoning, and was brittle in new situations.Source: §1 · p. 2

SIMA 2

  • SIMA 2 integrates Gemini to reason about high-level goals, formulate multi-step plans, converse with users, and act as an interactive companion.Source: §1 · p. 2
  • SIMA 2 generalizes across unseen environments, including photorealistic Genie 3 worlds, and can learn new skills from its own experience.Source: §1 · p. 2

Conclusion

  • The results support incorporating embodied intelligence and agentic control into foundation models as a path toward general-purpose interactive agents and eventual physical-world applications.Source: §1 · p. 2

2. Background & Related Works

The background positions games and simulations as scalable testbeds for embodied-agent research, while recent foundation models, world models, and open-ended learning address generalization, perception, and continual improvement. SIMA 2 combines these directions in complex 3D worlds.

Games & Simulation Driving Agent Research

  • Games and simulations provide increasingly complex environments for developing agents, from Atari and 2D platforms to 3D, multi-agent, and open-world games.Source: §2 · pp. 2–3
  • Generalist game agents increasingly target multiple skills, long-horizon goals, reasoning, imitation, and visually rich environments.Source: §2 · p. 3

World Models

  • World models predict outcomes from observations and actions and support planning, exploration, and offline learning in increasingly complex environments.Source: §2 · p. 3
  • Genie provides conditional world generation from text or initial observations, enabling SIMA 2 to interact with photorealistic environments unseen during training.Source: §2 · p. 3

Foundation Models in Embodied Agents

  • Embodied agents evolved from models trained from scratch toward pretrained visual, language, and vision-language-action models that generalize to novel objects and scenes.Source: §2 · p. 4

Open-Ended Self-Improvement

  • Open-ended self-improvement requires solving both which tasks to pursue and how to measure progress, and SIMA 2 uses foundation models for task setting and trajectory scoring.Source: §2 · pp. 4–5

3. Methods

SIMA 2 is trained and evaluated across diverse 3D environments through a human-like visual-action interface, multimodal gameplay data, bridge annotations, and reward-driven learning. Its evaluation suite emphasizes held-out generalization and increasingly challenging automatic and human assessments.

3.1.1. Held-Out Environments

  • SIMA 2 trains on research environments and commercial games, then tests generalization in entirely held-out environments such as ASKA, MineDojo, The Gunk, and Genie 3 worlds.Held-out settings introduce new visuals, menus, and game mechanics.Source: §3.1.1 · pp. 5–6, §3.1 · p. 5

3.2. Agent-Environment Interface

  • The agent receives RGB video and language history, then emits structured text parsed into keyboard, mouse, dialogue, and reasoning outputs without privileged environment state.The action space includes keyboard keys, mouse clicks, and discretized mouse movements.Source: §3.2 · p. 7

3.3. Data, Agent & Training

  • A Gemini Flash-Lite core is fine-tuned on gameplay and non-gameplay data to preserve vision, dialogue, reasoning, and promptability while learning embodied actions.Source: §3.3 · p. 7
  • Human trajectories teach low-level control, while synthetic bridge data links high-level instructions and dialogue with internal reasoning and low-level actions.Source: §3.3 · pp. 7–8, §3.3.1 · p. 8

3.3.1. Human Data

  • Human data combines free-form post-hoc annotations with setter-solver interactions, predefined game tasks, success judgments, trajectory comparisons, and filtered gameplay spans.Source: §3.3.1 · pp. 8–9

3.3.2. Bridge Data

  • Bridge data uses Gemini Pro to add causally consistent reasoning and dialogue, including error correction, instruction chaining, visual question answering, memory, and long-horizon behavior.Source: §3.3.2 · p. 9

3.3.3. Reinforcement Learning

  • Online reinforcement learning uses verifiable task tuples, trajectory-based rewards, answer correctness, and shaped rewards, but is restricted to training environments and excludes ASKA and MineDojo.Source: §3.3.3 · pp. 9–10

3.4. Evaluations

  • The evaluation suite combines ground-truth, programmatic, and human assessments, with stricter persistence, action-budget, and sequential-task requirements than SIMA 1.Source: §3.4 · pp. 10–11

4.1. New Capabilities

SIMA 2 extends SIMA 1 with Gemini-enabled dialogue, internal reasoning, complex language understanding, and multimodal prompting. These capabilities support embodied information seeking, novel instruction following, and performance approaching human levels.

New Capabilities

  • SIMA 2 uses Gemini’s broad vision and language understanding to overcome SIMA 1’s limitations, enabling dialogue, multimodal prompting, internal reasoning, and broader, more complex tasks.Vision, language, and action occupy a single stream of tokens, lifting SIMA 1’s input and output constraints.Source: §4.1 · p. 11

Embodied Dialogue

  • Embodied dialogue lets SIMA 2 confirm requests, report completion, ask clarifying questions, and act in the world to answer information-seeking queries.In one example, it navigates to egg-shaped objects and reports their material after inspecting on-screen text.Source: §4.1 · p. 12

Basic Reasoning

  • Internal reasoning enables SIMA 2 to convert indirect descriptions into actions, such as identifying a red house from the phrase “colored like a ripe tomato.”The paper states that this reasoning also supports appropriate actions in entirely novel environments.Source: §4.1 · p. 12

Complex Instructions

  • SIMA 2 generalizes to multilingual, emoji-based, and multi-step instructions, including navigating sequentially through rooms while reporting progress.These capabilities extend beyond the annotated English embodied data used for training and exceed SIMA 1’s described scope.Source: §4.1 · p. 13

Multi-modal Prompting

  • Image prompting allows users to specify locations, paths, or objects with sketches, enabling SIMA 2 to identify a drawn tree and appropriately chop one down.The broader multimodal interface supports more intuitive interaction than language-only instructions.Source: §4.1 · p. 13, Figure 5 · p. 13

Performance

  • SIMA 2 effectively doubles SIMA 1’s average success rate and approaches human performance under both unrestricted and agent time-restricted evaluations.The comparison averages performance over training environments for human and automatic evaluations.Source: Figure 6 · p. 14

4.2. Embodied Task Performance

SIMA 2 substantially improves embodied task performance over SIMA 1 in training environments and generalizes to entirely unseen worlds, including novel visuals, menus, mechanics, and photorealistic settings.

4.2.1. Performance in Training Environments

  • SIMA 2 effectively doubles SIMA 1’s success rate and nearly closes the gap with human performance across human- and automatically evaluated training tasks.The improvement is consistent across both evaluation types and across environments.Source: §4.2.1 · p. 14, Figure 8 · p. 15, Figure 9 · p. 15
  • SIMA 2 improves over SIMA 1 across all training environments, particularly in visually diverse games requiring menu interaction and challenging navigation.Human comparisons use both timeout-restricted and unrestricted baselines where applicable.Source: Figure 8 · p. 15, Figure 9 · p. 15, §4.2.1 · p. 16
  • SIMA 2 substantially improves across eight skill categories, approaching human-level performance in interaction and object management while still struggling with combat and resource gathering.Combat remains difficult partly because tasks require precise motor control and rapid decisions.Source: Figure 7 · p. 14, §4.2.1 · p. 16

4.2.2. Performance in Held-Out Environments

  • SIMA 2 significantly outperforms SIMA 1 by over 10% in both entirely held-out environments, ASKA and MineDojo.In MineDojo, SIMA 2 completes tasks in 26 of 50 categories, whereas SIMA 1 completes only two task types.Source: §4.2.2 · pp. 16, 18, Figure 10 · p. 16
  • In held-out environments, SIMA 2 generalizes beyond navigation by recognizing on-screen events and performing interaction, resource-gathering, combat, and menu-use tasks.Its dialogue indicates that recognized events help drive behavior in unfamiliar settings.Source: Figure 11 · p. 17, §4.2.2 · p. 18
  • Direct SIMA 2–human comparisons in held-out environments should not be over-interpreted because humans and agents exhibit different failure modes.Humans more often fail from time limits, while agents more often fail from suboptimal exploration.Source: §4.2.2 · p. 18
  • SIMA 2 progresses through novel mechanics in The Gunk, completing the first 15–20 minutes and reaching the Campsite checkpoint through manual instruction.It handles unfamiliar terrain, a suction device, object scanning, ledges, gaps, and gunk-clearing tasks using on-screen cues.Source: §4.2.2 · pp. 18–19, Figure 12 · p. 19
  • In Genie 3’s naturalistic and urban photorealistic environments, SIMA 2 navigates to particular points of interest despite training purely in research and video-game environments.This provides a proof-of-concept for generalization to more realistic environments.Source: §4.2.2 · p. 19

4.3. Comparison to Baseline Gemini Models

The comparison evaluates whether embodied fine-tuning preserves Gemini’s general reasoning abilities while enabling virtual-world interaction. Baseline Gemini models perform poorly on embodied tasks, whereas SIMA 2 retains most general capabilities with only minor regression.

Gemini Acting in Virtual Embodied Environments

  • Baseline Gemini Flash-Lite and Pro achieve only 3.2% and 7.0% success, respectively, across ten training domains without specialized embodied training.Prompt engineering was used to support action and text formatting, but competent embodied interaction still required explicit training.Source: §4.3 · p. 20

Retaining Language and Reasoning Capabilities

  • SIMA 2 exhibits only a minor regression on language and reasoning benchmarks relative to baseline Gemini, with no significant additional regression from RL beyond SFT.The benchmarks assess coding, mathematics, and scientific reasoning.Source: §4.3 · p. 21, Table 1 · p. 21
  • The results indicate that high embodied competence can coexist with retained foundation-model reasoning capabilities.SIMA 2 bridges high-level reasoning and low-level control without sacrificing the measured core capabilities.Source: §4.3 · p. 21

4.4. Gemini Instructing SIMA 2

A hierarchical Gemini Pro–SIMA 2 system adds slower, higher-level reasoning and recurrent memory to SIMA 2’s embodied control. It successfully follows complex multimodal instructions by decomposing and tracking a multi-step visual task.

Gemini Instructing SIMA 2

  • Gemini Pro periodically reasons over recent video and issues natural-language instructions to SIMA 2, while text summaries provide recurrent memory for longer-horizon behavior.The architecture combines a slower reasoning model with SIMA 2’s real-time embodied interaction.Source: §4.4 · p. 21

Complex Multi-modal Instruction Following

  • The combined agent successfully parses a complex diagram, decomposes campfire construction into steps, tracks progress, and communicates current and intended actions.The task integrates visual understanding, reasoning, memory, and embodied control.Source: §4.4 · p. 22, Figure 14 · p. 22

Gemini Instructing SIMA 2

  • Using SIMA 2 as a base allows newer Gemini versions to provide increasingly advanced embodied behavior through composition.This separates general embodied interaction from improvements in higher-level reasoning.Source: §4.4 · p. 22

4.5. Self-Improvement

SIMA 2 self-improves from self-generated experience using Gemini as both task setter and reward model, improving skills in ASKA and generalizing improvements from urban to held-out natural Genie 3 environments.

4.5. Self-Improvement

  • Gemini dynamically creates achievable tasks and scores trajectories from pixels and language goals without internal game-state rewards.The task setter adapts task distributions toward weaker skills, while the reward model assigns rubric-based scores from 0 to 100.Source: §4.5 · p. 23

4.5.1. ASKA

  • Self-generated experience raised ASKA performance above human reference scores and above the success threshold across all fixed tasks.The initial agent succeeded on fewer than a quarter of tasks, while later self-improved agents exceeded the threshold on every task.Source: §4.5.1 · p. 24, Figure 15 · p. 24
  • In ASKA, self-generated training enabled new navigation and interaction skills and substantially deeper technology-tree progression than SIMA 2 alone.The self-improved agent learned to reach new objects, extinguish a campfire, and build a shelter within one hour.Source: §4.5.1 · pp. 24–25

4.5.2. Genie 3

  • The Genie 3 results provide preliminary evidence for open-ended skill acquisition across increasingly diverse photorealistic environments.The authors frame this as an initial working example of learning within a universal world model.Source: §4.5.2 · pp. 25–26
  • SIMA 2 improved across nearly all train tasks, often by 25 points or more, and these gains extended to the majority of held-out tasks in natural environments.The method generated trajectories, scored them with a Gemini-based reward model, and trained on self-generated experience; test tasks used entirely different environments.Source: §4.5.2 · p. 25

5. Discussion

The discussion presents SIMA 2 as a generalist embodied agent that reasons, converses, and acts across virtual and photorealistic worlds, while emphasizing substantial current limitations.

5. Discussion

  • SIMA 2 combines reasoning, dialogue, and goal-directed action across diverse 3D worlds, including previously unseen and Genie 3 photorealistic environments.It also improves in new environments using entirely self-generated experience.Source: §5 · p. 26
  • The authors position self-improvement as a possible route toward transferring virtual embodied capabilities to physical applications such as robotics.This is presented as a promising direction rather than a demonstrated transfer result.Source: §5 · p. 26
  • SIMA 2 remains limited on very long-horizon tasks, extensive multi-step reasoning, goal verification, and low-latency interaction with short memory.Precise keyboard-and-mouse control and robust visual understanding of complex 3D scenes also remain open challenges.Source: §5 · pp. 26–27
  • The system is presented as a limited research preview developed with responsible-development collaboration and early access for a small cohort.This approach is intended to gather feedback and perspectives while risks and mitigations are explored.Source: §5 · p. 27

A. Embodied Skill Categories

The supplied material identifies Table 2 as presenting embodied skill categories.

A. Embodied Skill Categories

  • Table 2 is titled “Skill Categories,” but the supplied passage provides no category contents or comparisons.No further claims about the categories are supported by the available text.Source: Table 2 · p. 37

B. Additional Results Combining Gemini Pro & SIMA 2

Combining Gemini Pro with SIMA 2 enables behavior that uses prior user instructions and user answers to guide reasoning, action, and exploration.

B. Additional Results Combining Gemini Pro & SIMA 2

  • The combined agent uses past user instructions to modify current behavior or guide exploration rather than responding only to the immediate task.The examples test memory and reasoning beyond ordinary instruction following.Source: B. Additional Results Combining Gemini… · p. 38

Abstract Reasoning

Active Exploration

C. Additional Self-Improvement Results

In the previously unseen environment ASKA, SIMA 2 acquires new object-recognition and action skills through self-generated experience.

C. Additional Self-Improvement Results

  • SIMA 2 learns to recognize a novel rain collector and extinguish a campfire after multiple self-improvement iterations in ASKA.The skills are acquired through training on self-generated experience.Source: Figure 20 · p. 39
  • ASKA’s technology tree identifies the minimal progression needed to summon the first villager from a new game.Source: Figure 21 · p. 40
Loading 2512.04797v1…