PaperlayerView exact revision on arXiv

Source-linked AI summary

Player-1: Learning to Play from Synthetic Experience

Mingyu Ouyang, Yiran Wang, Siyuan Hu, Xingyilang Yin, Huiqiang Sun, Guangzhi Wang, Junfu Pu, Cheng Cheng, Honghao Chen, Ying Shan, Mike Zheng Shou, Hwee Tou Ng

arXiv:2610.20584v1cs.CVcs.AIcs.HC

TL;DR

Game agents have limited coverage of dynamic states in demonstrations, while scalable interactive training environments are difficult to obtain. PLAYER-1 combines supervised initialization with multi-environment RL in synthetic environments, improving game-control performance beyond SFT and generalizing beyond its training suite.

  • Problem

    Game-agent demonstrations cover only a limited portion of open-ended task states, and scalable environments for interactive training remain difficult to obtain.

    Source: §1 · pp. 1–2

  • Method

    PLAYER-1 initializes a shared game-control action space from curated gameplay trajectories, then trains with multi-environment RL in sixteen synthetic environments using verifiable rewards and randomized observations and control dynamics.

    Source: §1 · pp. 2–3

  • Results

    Player-1-9B and Player-1-35B-A3B improve MineDojo-300 success over SFT by 27.8 and 17.7 percentage points, respectively, with further gains beyond the RL training suite.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    The findings support synthetic experience as an effective resource for learning transferable game control without online training in each target game.

    Source: §6 · p. 10

Abstract

from arXiv · show

Computer-use agents have demonstrated strong capabilities in interpreting screens and executing keyboard and mouse actions. However, games, with their visually rich worlds and diverse mechanics, remain challenging and offer a demanding next step toward general digital control. Current game-agent learning pipelines often rely on large-scale demonstrations with limited coverage of dynamic game states, while scalable environments for interactive training remain difficult to obtain. We introduce Player-1, a post-training framework for learning game control beyond demonstrations through synthetic experience. We extend the computer-use action space with relative pointer motion, sustained inputs, and composite actions. For supervised initialization, we fine-tune computer-use models on 6,044 curated gameplay trajectories that pair recorded actions with task instructions and visual reasoning. We construct Player-1-Gym, sixteen synthetic 2D and 3D environments designed around control demands shared across games. Programmatically verifiable rewards and randomized observations and control dynamics support multi-environment reinforcement learning with PPO. The resulting Player-1-9B and Player-1-35B-A3B improve MineDojo-300 success over their SFT policies by 27.8 and 17.7 percentage points, respectively. Player-1-9B reaches 38.33% success compared with 12.63% for the strongest evaluated baseline, with further gains over SFT on ViZDoom-10 and GameWorld. Further analyses show that broader synthetic environment coverage improves transfer, while the use of composite actions that combine multiple keyboard and mouse inputs increases during RL. We demonstrate that learning from synthetic experience substantially improves game control beyond supervised initialization, with gains that generalize beyond the RL training suite.

1 INTRODUCTION

Game-control agents face limited demonstration coverage and a shortage of scalable interactive training environments. PLAYER-1 combines supervised initialization with synthetic multi-environment reinforcement learning, reporting gains beyond SFT and the RL training suite.

1 INTRODUCTION

  • Demonstrations cover only part of the states in open-ended tasks, scalable interactive training environments are difficult to obtain, and games share recurring control demands.The challenges include heterogeneous demonstration inputs, rewards across diverse commercial games, and combining experience across environments with different dynamics and reward scales.Source: §1 · pp. 1–2
  • On MineDojo-300, Player-1-9B rises from 10.54% SFT success to 38.33%, above the 12.63% strongest evaluated baseline; it also improves on ViZDoom-10 and GameWorld.The external games are outside the RL training suite.Source: §1 · pp. 2–3
  • PLAYER-1 uses 6,044 curated gameplay trajectories for supervised initialization, then learns transferable game-control capabilities from synthetic experience through a shared action space supporting relative pointer motion, sustained inputs, and composite actions.Source: §1 · p. 3
  • PLAYER-1-GYM provides sixteen synthetic 2D and 3D environments with programmatically verifiable rewards and randomized observations and control dynamics for multi-environment reinforcement learning.The environments target recurring control demands across games.Source: §1 · pp. 2–3
  • Player-1-9B and Player-1-35B-A3B gain 27.8 and 17.7 percentage points in MineDojo-300 success, with broader environment coverage improving transfer and composite-action use increasing during RL.The reported gains extend beyond the RL training suite to ViZDoom and GameWorld.Source: §1 · p. 3

2 RELATED WORK

Related work spans computer-use agents, game agents and environments, and post-training with verifiable rewards. PLAYER-1 combines supervised initialization and synthetic experience organized around shared control demands.

Generalist and Computer-Use Agents

Game Agents, Development, and Environments

  • Game-agent research includes planning systems, learned visual policies and VLA models, joint video–action models, and reinforcement learning for self-improvement and long-horizon control.Related work also covers gameplay benchmarks, game-development benchmarks, code world models, and game-engine-generated trajectory data.Source: §2 · p. 3

Agent Post-Training with Verifiable Rewards

  • Post-training with verifiable rewards supports learning from action outcomes, while environment synthesis expands training coverage; PLAYER-1 organizes synthetic environments around shared control demands.SIMA 2 is described as combining supervised initialization with online RLVR and transfer across Genie 3 environments (SIMA Team, 2025).Source: §2 · pp. 3–4

3 METHOD

PLAYER-1 connects supervised initialization from curated gameplay demonstrations to PPO training on synthetic environments, using an extended action space for dynamic game control. PLAYER-1-GYM supplies varied tasks and verifiable rewards for learning beyond demonstrations.

3.1 EXTENDING THE COMPUTER-USE ACTION SPACE

  • The action space adds relative pointer motion and sustained inputs, while composite actions execute compatible keyboard or mouse inputs concurrently within one control step.These controls support coordinated movement and combat, such as running jumps or aiming while firing.Source: §3.1 · p. 4

3.2 SUPERVISED INITIALIZATION FOR GAME CONTROL

  • 6,044 curated gameplay trajectories are mapped to the shared action space and annotated with task instructions and per-turn visual reasoning for supervised initialization.SFT predicts reasoning and actions from the instruction, screenshot, and retained interaction history.Source: §3.2 · p. 5

3.3 PLAYER-1-GYM: LEARNING FROM SYNTHETIC EXPERIENCE

  • Sixteen synthetic environments span five shared game-control families, with game-state scoring that provides dense, verifiable step-level rewards without exposing privileged state.The environments use different mechanics, viewpoints, and layouts; observations and control dynamics are randomized during training.Source: §3.3 · pp. 5–6
  • PPO jointly optimizes the SFT-initialized policy across randomized environments, with per-environment advantage normalization to limit reward-scale effects on policy updates.Return-to-go targets accumulate later rewards so subsequent outcomes can inform earlier control decisions.Source: §3.3 · p. 6

4 EXPERIMENTS

The experiments evaluate SFT-initialized models against baselines and test transfer beyond the training suite. Synthetic-experience training improves MineDojo success and performance on ViZDoom-10 and GameWorld, with an illustrated sequence in an unseen commercial game.

4.1 EXPERIMENTAL SETUP

  • Both model sizes receive two SFT epochs on the full demonstration dataset, followed by 200 PPO training steps across all sixteen PLAYER-1-GYM environments.Evaluation includes MineDojo-300, ViZDoom-10, and GameWorld, with comparisons against game-specific and computer-use models.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

  • On MineDojo-300, interactive learning raises 9B success to 38.33%, a 27.79-percentage-point gain over SFT’s 10.54%; results use eight trials per task.Tasks completed in at least four of eight trials increased from 20 to 136.Source: §4.2 · p. 7, Table 2 · p. 7
  • Player-1-9B raises ViZDoom-10 scaled return from 0.4361 after SFT to 0.6922, while GameWorld progress rises from 31.43% to 34.45%.Player-1-35B-A3B reaches 0.7236 overall scaled return on ViZDoom-10, above GameWAM's 0.3938.Source: §4.2 · p. 7, Table 3 · p. 7, Table 4 · p. 7
  • In unseen Watch Dogs 2, Player-1-9B demonstrates on-foot navigation, car approach, vehicle entry, and city driving with coordinated movement and camera control.The game was unseen during both SFT and RL.Source: Figure 4 · p. 8, §4.2 · p. 8

5 ANALYSIS

Across the 16-environment RL suite, SFT-initialized policies improve, while the analysis also tests environment breadth and tracks rising composite-action use.

5.1 LEARNING DYNAMICS

  • At step 100, SFT→RL reaches 51.67% Minecraft success versus 16.67% for Base→RL, a 35.00-percentage-point gap.Source: Figure 5 · p. 8, §5.1 · p. 8
  • Both models’ mean episode returns and game scores rise during RL, with the largest gains early and 35B-A3B finishing slightly higher.Source: §5.1 · p. 8, Figure 5 · p. 8
  • Both models make broad net gains across environments, but learning varies: 2D aiming improves rapidly before leveling off, while 2D circuit-driving gains emerge later.Source: §5.1 · pp. 8–9, Figure 6 · p. 9

5.2 ENVIRONMENT SCALING

  • The environment-scaling comparison holds the budget at 25,600 episodes over 100 RL steps while varying the suite from 2 to 16 environments.Each configuration uses 256 episodes per step, allocated equally across environments; the supplied passages do not report the comparison’s outcome.Source: Figure 7 · p. 9, §5.2 · p. 9

5.3 EVOLUTION OF ACTION COMPOSITION

  • Composite-action use rises from 64.7% at RL start to 85.4% at 50 steps and 93.2% at 100 steps, without an explicit composition reward.The authors suggest task-driven RL reinforces coordinated inputs such as aiming while firing or moving while adjusting the camera.Source: Figure 8 · p. 9, §5.3 · p. 9

6 CONCLUSION

The conclusion presents synthetic experience as a route to game-control gains beyond SFT that transfer beyond the RL suite. The appendices specify action semantics, camera response, and interaction history used in evaluation.

6 CONCLUSION

  • Synthetic-experience training advances control beyond SFT, generalizes beyond the RL suite, and shows improved transfer with larger training suites.The conclusion frames synthetic experience as an effective resource for transferable game control without online training in each target game.Source: §6 · pp. 9–10

A.1 COORDINATES AND COMPOSITE ACTIONS

  • Composite inputs execute concurrently within one step, using only bindings supported by each environment.The coordinate convention places the origin at the upper-left, with positive displacement rightward and downward.Source: Table 5 · p. 13, §A.1 · p. 13

A.2 EXECUTION AND VISUAL FEEDBACK

  • Relative mouse motion requests displacement rather than a universal camera angle; camera rotation depends on game-specific gain.The seven mouse-controlled 3D environments use a nominal gain of 0.00196 radians per observation pixel, with control randomization modifying effective gain.Source: §A.2 · p. 13

A.3 OBSERVATION AND INTERACTION HISTORY

  • SFT, RL, and evaluation retain two recent visual–action turns and two earlier action-only turns, replacing earlier screenshots and reasoning with omission markers.The current screenshot and executed historical actions remain available.Source: §A.3 · p. 13

B.1 SOURCES AND STATISTICS

The appendix describes the demonstration collection, its conversion into a shared action format, and the annotation process used to pair goals and reasoning with gameplay.

B.1 SOURCES AND STATISTICS

  • The collection contains 6,044 trajectories and 295,097 turns from 2,544 recording sessions and 172 games, totaling an estimated 16.4 sampled control hours.Quality selection combines automated validity, activity, and repetition checks with source-level and manual review; the final set includes 5,360 gameplay and 684 UI trajectories.Source: §B.1 · p. 13, Figure 9 · p. 14

B.2 ACTION STANDARDIZATION

  • Recorded controls are aligned to a shared 200-ms action space, with duration-based handling of presses and source-specific calibration of relative mouse motion.UI trajectories preserve operation boundaries, while relative inputs are mapped to pixel-equivalent displacements using GUI-pointer calibration.Source: §B.2 · p. 14, Table 6 · p. 14

B.3 GOAL AND REASONING ANNOTATION

  • Reusable game profiles, trajectory storyboards, and action timelines supply goals; per-turn reasoning annotations also use the goal, fixed action target, and current and past frames.Source: §B.3 · pp. 14–15

C.1 TASKS AND VERIFIABLE REWARDS

The appendix specifies verifiable tasks and randomized environments, then describes action-balanced supervised fine-tuning and multi-environment PPO training.

C.1 TASKS AND VERIFIABLE REWARDS

  • Sixteen environments span aiming, combat, driving, navigation, and UI interaction, with cumulative scores derived from privileged game state while policies receive screenshots and task/control text.Rewards are score increments; episodes end on native completion or failure, or at a turn cap if unfinished.Source: §C.1 · p. 15, Table 7 · p. 15, Table 8 · p. 15

C.2 RANDOMIZATION AND SAMPLING

  • Appearance, camera, scene, and control-response randomization vary supported observations and dynamics, while collision rules, gravity, attack timing, bindings, and reward definitions remain fixed.Appearance and camera are sampled at reset and fixed within an episode; control-response values are likewise held constant through the episode.Source: §C.2 · p. 16, Table 9 · p. 17, Figure 10 · p. 18

D.1 SUPERVISED FINE-TUNING

  • SFT uses token-level cross-entropy on current-turn reasoning and actions, with action-balanced sampling to strengthen supervision for infrequent control actions.Sampling draws from overlapping control-action pools and assigns the Other pool a lower weight of 0.1.Source: §D.1 · p. 16

D.2 MULTI-ENVIRONMENT REINFORCEMENT LEARNING

  • RL uses synchronous PPO with a warmed-up dense critic; penalized step rewards accumulate into undiscounted return-to-go targets over complete trajectories.Invalid actions execute a wait fallback, reasoning-token penalties are applied, and there is no additional terminal-failure penalty or task-wise reward rescaling.Source: §D.2 · pp. 16, 18

D.3 TRAINING IMPLEMENTATION

Both model sizes share the post-training settings except for actor architecture. The scaling protocol matches tasks and trial seeds across environment subsets and uses equal episode allocations.

D.3 TRAINING IMPLEMENTATION

  • SFT uses 32 NVIDIA H200 GPUs for approximately 24 hours, while a 200-step RL run uses 64 H200 GPUs for approximately 36 hours.Browser rollouts support up to 512 concurrent environments; both durations include validation.Source: §D.3 · pp. 18–19, Table 10 · p. 19
  • Browser rollouts use 32 workers with up to 16 isolated Playwright contexts each, reusing browser processes and contexts across episodes.This provides capacity for 512 concurrent environments.Source: §D.3 · p. 19

D.4 ENVIRONMENT-SCALING PROTOCOL

  • The 9B scaling runs share Minecraft development tasks and trial seeds, with SFT as the zero-environment reference; each run uses 100 RL steps and 25,600 episodes allocated equally across environments.Interaction lengths and compute may vary across episodes.Source: §D.4 · p. 19, Table 11 · p. 19

E EVALUATION AND RESULTS

The evaluation describes baseline inference settings and reporting protocols for Player-1-Gym and three external benchmarks. It also estimates the cost of full-budget evaluation and limits the focus to representative released policies.

E EVALUATION AND RESULTS

  • The evaluation panels cover 100 Minecraft tasks, ten ViZDoom maps, and 170 GameWorld tasks, while full protocols allocate up to 3.14 million decision turns per model.The external benchmarks use environments outside Player-1-Gym.Source: E EVALUATION AND RESULTS · p. 19, Table 12 · p. 20
  • At the estimated full-budget evaluation cost of approximately USD$110,000 per model, the study focuses mainly on representative released policies rather than proprietary API models.The estimate uses observed input and output token lengths as a workload proxy; actual costs depend on model-specific tokenization, reasoning, and episode lengths.Source: E EVALUATION AND RESULTS · p. 20

E.1 BASELINE INFERENCE AND ACTION ADAPTERS

  • Baseline inference uses model-specific histories, action chunking, screenshot preprocessing, and adapters that map native actions to benchmark controls.For example, NitroGen executes three native actions per benchmark decision, while GameWAM executes eight of its predicted 16 actions before replanning.Source: §E.1 · p. 20

E.2 PLAYER-1-GYM IN-SUITE VALIDATION

  • In-suite validation uses the seeded panel and training randomization, averaging trial returns within each environment and equally across all sixteen environments.Episode return combines native game-score increments with invalid-action and length penalties.Source: §E.2 · p. 20

E.3 MINEDOJO-300

  • MineDojo-300 contains 200 Combat and 100 Harvest tasks, with success averaged equally across tasks and 95% intervals measuring within-panel task-sampling uncertainty.Success is verified from inventory acquisition for Harvest or target kills for Combat; the evaluation retains eight trials per task for bootstrap intervals.Source: §E.3 · pp. 20–21

E.4 VIZDOOM-10

  • ViZDoom-10 evaluates ten maps using fixed reward scales and 50 seeds per map, with observation geometry, timing, and camera constraints matched within each backbone.The maps use differing decision limits, from 150 decisions in Basic to 2,100 in Deathmatch, subject to earlier termination.Source: §E.4 · p. 21, Table 13 · p. 21

E.5 GAMEWORLD

  • GameWorld spans 170 tasks across 34 browser games; success measures completion trials and progress measures achieved fraction toward each target, averaged equally across tasks.Policies receive screenshots, rules, and instructions while evaluator state is hidden.Source: §E.5 · p. 22

Related papers

Loading 2610.20584v1…