Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agents

1 Introduction

• Environment-specific, predefined action sets tightly couple agents to tasks and limit generalization, particularly to new open-world tasks.The introduction identifies action-space definition as a core bottleneck for scalable generalist agents.Source: §1 · p. 3
• The proposed action space replaces application-specific high-level instructions with low-level keyboard and mouse inputs usable across GUI environments.The authors present this native human-interaction paradigm as a foundation for large-scale cross-domain pretraining.Source: §1 · p. 3
• The training process combines continual pretraining and post-training, using Sparse Thinking to interleave reasoning and action at critical decision points.Post-training strategies target instruction adherence, dynamic context learning, efficient reasoning, and long-term memory.Source: §1 · p. 3
• Post-training incorporates trajectories from code generation, GUI automation, and research tasks to broaden the agent beyond game playing.The stated aim is a more versatile general computer user.Source: §1 · p. 3
• Approximately 2x performance improvement in Minecraft versus previous state-of-the-art expert models followed large-scale general computer-use pretraining.The unified action space did not initially outperform GUI-based agents when trained on less than 10B data; the authors identify scalability as its advantage.Source: §1 · p. 3
• Game-TARS outperformed Gemini-2.5-Pro, GPT-5, and Claude-4-Sonnet across diverse-domain tasks, including FPS-game, 3D simulator, and online web-game benchmarks.The introduction names Vizdoom, Miniworld, and online web games among the evaluations.Source: §1 · p. 3

2 Continual Pre-Training for Generalist Game Agent

• The policy generates reasoning and actions from observations and an internal memory state that summarizes prior interactions.A complete trajectory records successive reasoning, action, and observation steps.Source: §2 · p. 5

2.1 A Scalable and Generalist Action Space for Computer-Use Agents

• Grounding actions in keyboard and mouse inputs avoids environment-specific action APIs and supports one time-aligned action space across games and desktop applications.The authors present this Human-Native Interaction paradigm as a foundation for cross-domain continual pre-training.Source: §2.1 · pp. 5–6

2.2 Native Sparse ReAct Pretraining via Thinking Aloud

• Sparse ReAct trajectories place reasoning at selected moments, with required opening planning and closing summaries rather than reasoning at every rapid action step.The collection protocol asks annotators to verbalize thoughts at least three key moments per minute.Source: §2.2 · pp. 6–7
• Synchronized screen, input, and audio capture is processed into structured trajectories, while cursor-based visual anchors correct timestamp bias between mouse actions and observation frames.Thought timestamps are also relocated toward their semantically relevant subsequent actions.Source: §2.2 · pp. 7–8, Figure 3 · p. 7

2.3 Continual Pre-training with Decaying Loss Function

• Over 90% of actions repeat the previous action; history-aware exponential weighting reduces repeated-action loss while retaining no-ops that may represent necessary waiting.New or changed actions receive full weight, and each subsequent repetition is down-weighted.Source: §2.3 · p. 8

3 Post-Training for Generalist Game Agent

• Post-training targets instruction following, sparse thinking, and long-term memory after continual pre-training on over 20k game hours (about 500B tokens).The paper presents this stage as enhancing execution capabilities and interactive intelligence.Source: §3 · p. 8

3.1 Instruction Following

• Randomly substituting key bindings trains the agent to follow the current action-space instructions instead of relying on memorized action frequencies.Inverse-dynamics prediction adds an auxiliary task: infer the action that caused a state transition from current and next observations.Source: §3.1 · p. 9

3.2 Multimodal Prompts Elicit In-Context Learning

• Multimodal action-effect examples help the agent infer context-dependent action meanings, while trajectory history supports adaptation to environment-specific continuous-action scales.Training also random-scales mouse movements, capped at 180 in absolute value, so the agent must infer scale from historical feedback.Source: §3.2 · p. 10

3.3 Sparse Thinking via Reinforcement Fine-Tuning

• Sparse-thinking fine-tuning retains reasoning only when it leads to the demonstrated action and targets steps where an action-only policy fails.This addresses abstract reasoning and no-op actions in think-aloud data that weaken the learned link between reasoning and action.Source: §3.3 · p. 11

3.4 Long Episodic Memory Compression with Sparse Thought

• A two-tier memory keeps recent observations and ReAct pairs in context while preserving sparse thought summaries for evicted history.The short-term context is typically 32k tokens; summary memory can retain task context across over 2000 interaction steps, and training sequences reach 128k tokens.Source: §3.4 · p. 12, Figure 6 · p. 12

3.5 Post-Training with Cross-Source Agentic Trajectories

• Post-training combines game data with code, GUI, and research-task trajectories to develop broader problem-solving capabilities on top of game interaction skills.The paper describes code data as supporting structured reasoning, GUI data as supporting multi-step application tasks, and research data as supporting planning and knowledge integration.Source: §3.5 · pp. 12–13

4.1 Training Details

• Training used a unified single-stage continual-pretraining blend of over 500B tokens, followed by about 20B curated post-training tokens.The dataset covers over 500 games; the paper reports that an earlier staged training strategy had harmed final performance.Source: §4.1 · pp. 13–14

4.2.1 Experimental Setups

• The unseen-game evaluation spans Minecraft, Poki web games, Vizdoom, and MiniWorld, selected to test varied tasks, interaction modes, and environments.Minecraft tasks use over 800 MCU tasks and manually curated initial environments; the web, FPS, and simulator benchmarks test unfamiliar settings.Source: §4.2.1 · p. 14

4.2.2 Evaluation on Open-World Minecraft

• Game-TARS-MoE-mini achieved the highest success rates across Minecraft task categories, including 55.0% on GUI, 72.0% on embodied, and 66.0% on combat tasks.The paper reports that it outperformed Minecraft-only expert baselines; the dense Game-TARS model used game data alone.Source: §4.2.2 · p. 15, Table 3 · p. 15

4.2.3 Evaluation on Unseen Benchmarks and Simulators

• Game-TARS-mini performed near human or better across most Poki games, beat fresh humans in Jump-Only and Airplane Flying, and outperformed baselines on almost all Vizdoom maps.It also completed tasks across nearly all tested MiniWorld maps without prior MiniWorld training exposure.Source: Figure 7 · p. 16, Figure 8 · p. 16, Figure 9 · p. 16

4.3.1 Scaling Experiments on Training Datasets and Inference Steps

• GUI actions perform relatively well with limited data, but unified actions need more data initially and approach GUI-action upper-bound performance after large-scale pretraining.The authors attribute the difference partly to unified actions not being natively supported by pretrained VLMs; GUI actions require extensive preprocessing and do not naturally transfer across heterogeneous games.Source: §4.3.1 · p. 18
• Adding 200B tokens from other games raised Minecraft success from 42% to 65%.The authors identify consistent cross-game representation as enabling aggregation of heterogeneous data.Source: §4.3.1 · p. 18
• Multimodal and cross-domain agent data further raised performance to 72%, and unified actions sustained gains through cross-game and cross-domain scaling.These data included captioning, VQA, video understanding, and trajectories from other domains.Source: §4.3.1 · p. 18

4.3.2 Ablation on sparse thinking

• Reasoning benefits vary by task: it helps planning-intensive 2048, while excessive thinking can slow Doom and trigger hallucinated reasoning loops.The agent selectively reasons at critical points and reuses established reasoning during routine steps; its qualitative trajectories show different patterns across Minecraft, 2048, and VizDoom.Source: §4.3.2 · pp. 18–19
• Sparse thinking uses 37 tokens per step versus 68 for Greedy Thinking, with 63% Minecraft success and 924 points in 2048.The reported strategy achieves state-of-the-art performance across most benchmarks while reducing redundant computation.Source: §4.3.2 · p. 19

4.3.3 Validation on Continual Pre-Training Loss

• Decaying loss improved online success across embodied, combat, and GUI tasks, shortened trajectories, and raised behavioral diversity from 60–80% to nearly 100%.Combat success increased from 14.5% to 22.2%; the authors suggest decaying loss helps mitigate repetitive behavior.Source: §4.3.3 · p. 19
• Decaying loss yielded 39% non-repetitive accuracy versus 12% for the baseline, despite lower overall accuracy of 47% versus 59%.The authors interpret the baseline’s higher overall accuracy as exploiting repeated-action bias; both loss approaches reached comparable final training losses without a stable-convergence penalty.Source: §4.3.3 · pp. 19–20

5.1 Game Agents

• DQN, AlphaGo, and AlphaStar used game-specific action spaces, while VPT’s keyboard-mouse pretraining was confined to Minecraft [4].Voyager used high-level APIs, creating a semantic gap between sequencing functions and acquiring sensorimotor skills [33] [66] [71] [83] [86].Source: §5.1 · p. 20

5.2 Generalist Agents

• Gato serialized multimodal tasks but retained domain-specific action tokenization [53]; early SIMA focused on short-horizon tasks and limited in-domain games [51].The passage presents these as limitations in pursuing a unified generalist interface.Source: §5.2 · p. 21

5.3 Broader AI Agents

• GUI and web agents often ground high-level plans in UI elements, whereas Game-TARS uses low-level keyboard-mouse outputs; cross-domain post-training data complements its sensorimotor skills.The cited broader domains include software engineering and flexible tool integration.Source: §5.3 · p. 21

• The agent uses a scalable native keyboard-mouse action space and continual pretraining on more than 500B tokens across game trajectories, AI-agent trajectories, and multimodal data.Source: §6 · p. 21
• Game-TARS achieves 2x SOTA success rate in Minecraft, nears human-level generalization in unseen web games, and outperforms GPT-5, Gemini-2.5-Pro, and Claude-4-Sonnet on FPS benchmarks.Source: §6 · p. 21
• Game-TARS uses Sparse Thinking to balance reasoning depth with inference cost, and its scaling results support simple action representations with large-scale pretraining as a path toward generalist computer-use agents.Source: §6 · p. 21

• The section lists the paper’s authors.Source: Contributions · p. 21
• The listed affiliations are ByteDance Seed, Peking University, and M-A-P.Source: Contributions · p. 21
• The acknowledgment thanks Muyao Li for discussions and Cheng Yuan and Hao Chen for efforts on the Game-TARS website.Source: Contributions · p. 21

• Game-TARS avoids creeper-explosion damage by retreating and uses jumping strikes to increase damage.Source: Figure 11 · p. 27
• Game-TARS uses tools and interacts with objects in the environment.Source: Figure 12 · p. 27
• Game-TARS tracks foxes over time, remembering their movement trajectory while navigating and exploring to pursue them.Source: Figure 13 · p. 27
• Game-TARS sustains a long press to break obsidian blocks, which require 10 seconds of pressing.Source: Figure 14 · p. 27
• Game-TARS switches between and interacts with embodied 3D and 2D GUI interfaces using unified actions.Source: Figure 15 · p. 27

• Table 6 presents a Game-TARS rollout trajectory in the Poki Plane game, with some timestamps omitted.Source: Table 6 · p. 28
• Table 7 presents a Game-TARS rollout trajectory in Temple Run, with some timestamps omitted.Source: Table 7 · p. 29
• Table 8 presents a Game-TARS rollout trajectory in Poki Race, with some timestamps omitted.Source: Table 8 · p. 30

B.1 Prompts of Game-TARS on Minecraft

• The Minecraft prompt frames the task around instructions, action history, and screenshots, with a next-action objective and a Thought/Action response format.Source: §B.1 · p. 31
• Minecraft actions use mouse clicks and movement, keyboard presses, and no-op waiting, with controls mapped to in-game and GUI operations.Source: §B.1 · p. 31
• Minecraft prompts request a brief plan in Chinese when thought is needed, but permit action-only output otherwise; prior thoughts accumulate in the conversation history.Source: §B.1 · p. 31

B.2 Prompts of Game-TARS on Unseen Games

• The Drive Freedom prompt maps arrow keys to acceleration, braking, and steering, with no-op available for waiting.Source: §B.2 · p. 32
• The Plane game prompt gives driving instructions to maintain speed, steer, avoid traffic and roadside obstacles, and stay on the road while pursuing the fastest route to the destination.Source: §B.2 · p. 32
• Temple Run prompts use screenshots and action history, with controls for jumping, sliding, changing tracks, or waiting; tutorial guidance prioritizes survival, resource collection, score, and exploration.Source: §B.2 · p. 33
• Across unseen-game prompts, the response format allows a Thought/Action output when thought is needed or an action-only response for low-level steps.Source: §B.2 · pp. 32–34
• The Jump Only prompt restricts each step to one key and requires timed left/right jumps with no-op waits between jumps to avoid obstacles and reach the finish.Source: §B.2 · p. 34