Source-linked AI summary
NitroGen: An Open Foundation Model for Generalist Gaming Agents
Loïc Magne, Anas Awadalla, Guanzhi Wang, Yinzhen Xu, Joshua Belofsky, Fengyuan Hu, Joohwan Kim, Ludwig Schmidt, Georgia Gkioxari, Jan Kautz, Yisong Yue, Yejin Choi, Yuke Zhu, Linxi "Jim" Fan
TL;DR
Generalist gaming agents lack large, diverse labeled action data and open frameworks for training and evaluation. NitroGen combines an internet-scale video-action dataset, a universal multi-game simulator and benchmark, and vision-action behavior-cloning pre-training. It achieves strong performance across varied games and up to 52% relative improvement in success rates when transferred to unseen games.
Problem
Generalist gaming-agent research lacks large, diverse labeled action datasets and open-source frameworks for training and evaluating agents across games.
Method
NitroGen extracts actions from publicly available gameplay videos, evaluates agents through a universal simulator and multi-game benchmark, and pre-trains a vision-action model with behavior cloning.
Results
Up to 52% relative improvement in success rates over training from scratch is obtained when fine-tuning pre-trained NitroGen weights on unseen games with fixed data and compute.
Takeaways & Limitations
NitroGen provides an open foundation for training and evaluating generalist gaming agents and for studying transfer across heterogeneous game environments.
Takeaways & Limitations
NitroGen is a short-context reactive system-1 model that cannot plan over long horizons or follow language instructions, and its data is biased toward action games and gamepad-played titles.
Abstract
from arXiv · showhide
We introduce NitroGen, a vision-action foundation model for generalist gaming agents that is trained on 40,000 hours of gameplay videos across more than 1,000 games. We incorporate three key ingredients: 1) an internet-scale video-action dataset constructed by automatically extracting player actions from publicly available gameplay videos, 2) a multi-game benchmark environment that can measure cross-game generalization, and 3) a unified vision-action model trained with large-scale behavior cloning. NitroGen exhibits strong competence across diverse domains, including combat encounters in 3D action games, high-precision control in 2D platformers, and exploration in procedurally generated worlds. It transfers effectively to unseen games, achieving up to 52% relative improvement in task success rates over models trained from scratch. We release the dataset, evaluation suite, and model weights to advance research on generalist embodied agents.
1. Introduction
NitroGen addresses the lack of large, diverse, labeled action data and open evaluation frameworks for generalist gaming agents. It combines an internet-scale video-action dataset, a multi-game simulator and benchmark, and behavior-cloning pre-training to support cross-game generalization.
- Motivation: NitroGen targets embodied-agent generalization by using video games as visually rich interactive environments spanning diverse tasks and temporal horizons.Prior approaches relied on hand-crafted game APIs or complicated perception modules, while open-source frameworks for training and evaluating generalist gaming agents remained limited.
- Contributions: 40,000 hours of publicly available videos across more than 1,000 games form an action-labeled dataset for large-scale training.Input overlays expose player commands in real time, and an annotation model extracts frame-level actions without costly manual data collection.
- Contributions: The multi-game evaluation suite contains 30 tasks from 10 commercial games, covering combat, navigation, decision-making, platforming, exploration, and puzzle-solving.A universal Gymnasium API wraps commercial games for evaluating diverse agent capabilities.
- Contributions: NitroGen trains a vision-action transformer with large-scale behavior cloning and demonstrates a multi-game policy using noisy internet data.Post-training on unseen games tests whether pre-training improves transfer under fixed data and compute budgets.
- Results and resources: Up to 52% relative improvement in success rates is achieved over training from scratch when fine-tuning from pre-trained NitroGen weights with fixed data and compute.The dataset, simulator, and pre-trained weights are released as open-source resources for generalist embodied-agent research.
2. Approach
NitroGen combines an action-labeled internet-scale video dataset, a unified multi-game benchmark, and a vision-action model trained through behavior cloning. Its evaluation spans diverse game genres, visual styles, tasks, and standardized controller actions.
- Components: NitroGen comprises an internet-scale video dataset with action labels, a multi-game benchmark, and a vision-action model pretrained through large-scale behavior cloning.The dataset and benchmark support training and evaluating a shared policy across game titles.
- Dataset: 40,000 hours of gameplay videos span more than 1,000 games after filtering, with coverage distributed across games and genres.The dataset includes 846 games with over one hour of data, 91 with over 100 hours, and 15 exceeding 1,000 hours.
- Dataset: Gamepad actions are recovered through template matching, segmentation-based parsing, and quality filtering from videos containing real-time input overlays.The parsing model processes consecutive frames and predicts joystick positions on an 11 × 11 grid plus binary button states.
- Evaluation suite: A universal simulator wraps commercial game titles with a Gymnasium API and enables frame-by-frame interaction by controlling simulation time.The shared interface uses RGB observations, 16 binary gamepad-button actions, and 4 continuous joystick values.
- Evaluation suite: The evaluation suite covers 10 games and 30 tasks across 2D and 3D visual styles, including combat, navigation, and game-specific challenges.It includes five 2D games, five 3D games, 11 combat tasks, 10 navigation tasks, and 9 game-specific tasks.
- Foundation model: NitroGen uses a flow-matching vision-action transformer that generates future action chunks conditioned on RGB observations.The architecture adapts GR00T N1 by removing language and state encoders and retaining a single action head.
3. Experiments
Experiments evaluate action extraction, zero-shot behavior across games, and transfer to held-out environments. NitroGen performs non-trivial tasks across diverse games, while pretraining benefits vary by game and task type.
- Action extraction: The action extraction pipeline achieves an average joystick R2 of 0.84 and average button accuracy of 0.96 across popular controller families.Both metrics are computed against ground-truth controller inputs from recorded gameplay.
- Pretraining: Without game-specific fine-tuning, NitroGen achieves non-trivial success rates across games with different visual styles, genres, and task structures.Performance extends to both fixed-layout tasks that may be memorized and procedurally generated tasks requiring adaptation.
- Pretraining: The dataset contains noise from delayed or inaccurately parsed actions, creator-specific video artifacts, and varying controller configurations, yet supports robust policy training.These sources of noise include livestream overlays and differences in sensitivity settings or button mappings.
- Transfer learning: 10% average relative improvement is achieved on the isometric roguelike, while the 3D action-RPG achieves 25% after fine-tuning from NitroGen pretraining.The comparison uses held-out games and matches models trained from scratch with the same architecture, data, and compute budget.
- Transfer learning: 52% relative improvement occurs for combat tasks and 25% for navigation on the 3D action-RPG, while game-specific tasks improve by 5%.The results indicate stronger transfer for common gameplay patterns than for mechanics specific to the new game.
4. Limitations and future work
NitroGen’s scope is constrained by its short-context, reactive design and by dataset coverage that favors action games and gamepad-based play. These limitations bound long-horizon planning and generalization to strategy or simulation genres.
- Design limitations: NitroGen cannot plan over long horizons or follow language instructions because it reacts only to its short visual context.The authors describe it as a fast-reacting system-1 sensory model and propose language-following and reinforcement-learning post-training as future enhancements.
- Dataset bias: The dataset is biased toward action games and titles typically played with a gamepad.Keyboard-only games and games requiring complex manipulation are less represented.
- Dataset bias: This collection bias may limit generalization to strategy and simulation games that depend more on planning and keyboard input.
5. Related works
Prior gaming-agent and embodied-AI approaches use reinforcement learning, structured language-model interfaces, or behavior cloning, each with important data, simulator, or interface constraints. NitroGen advances behavior cloning by scaling it with action-labeled internet gameplay videos.
- Gaming agents: Gaming-agent research spans reinforcement learning, language-model methods with structured APIs, and pixel- or state-based behavior cloning.These directions include systems ranging from Atari and AlphaGo to Voyager, MineRL, VPT, SIMA, GATO, and related agents.
- Gaming agents: Reinforcement-learning and vision-based methods often require engineered rewards, specialized simulators, or environment-specific training.
- Gaming agents: Language-model gaming agents depend on hand-crafted programmatic interfaces for exposing game state and control.
- Gaming agents: Existing gaming behavior-cloning systems rely on datasets bootstrapped from human demonstrations or reinforcement-learning-generated data.
- Large-scale action datasets: NitroGen addresses action-data scale by using input overlays in public gameplay videos, avoiding the costly, limited, and less organically diverse data collection described for robotics.
6. Conclusion
NitroGen scales foundation pre-training for video-game agents with an internet-scale video-action dataset and demonstrates a multi-game policy with positive transfer to new environments. The authors present it as a starting point for more powerful general-purpose agents.
- Conclusion: NitroGen uses publicly available gameplay data to scale foundation pre-training for video-game agents and yield a generalist policy.
- Conclusion: The approach trains a multi-game policy and shows positive signs of generalization in fine-tuning experiments.
- Conclusion: NitroGen lowers the barrier to training agents on new environments and serves as a starting point for more powerful general-purpose agents.
A.1. Training objective
The training objective corrupts ground-truth action chunks with Gaussian noise at a sampled timestep and trains the model to predict the corresponding conditional velocity field. The loss compares the predicted velocity with the action-noise displacement.
- Training objective: A ground-truth action chunk is combined with Gaussian noise along a flow-matching timestep to construct a noisy action.
- Training objective: The model minimizes conditional flow-matching loss by predicting the velocity field associated with the noisy action.
- Training objective: The loss conditions prediction on the noisy action, image encoding, and timestep, with the DiT and image encoder providing the model components.
A.2. Inference
At inference, NitroGen starts from Gaussian noise and iteratively denoises the action for k steps using Euler integration. The implementation uses 16 denoising steps because additional steps provide no measurable improvement.
- Inference initializes the action from Gaussian noise and iteratively denoises it for k steps using Euler integration.
- 16 denoising steps are used because additional steps yield no measurable improvement.
B.1. Synchronous inference
The synchronous inference setup freezes the game while the model predicts each next action. Replay experiments show that visual divergence occurs over time in both paused and real-time conditions, supporting error accumulation rather than pausing as the cause.
- The Gymnasium API freezes the game while the model predicts the next action.
- Replayed sequences diverge after about one minute with continuous actions and about three minutes with discrete actions.
- Divergence is the same with and without frequent pausing, supporting error accumulation as its explanation.