PaperlayerView exact revision on arXiv

Source-linked AI summary

Lumine: An Open Recipe for Building Generalist Agents in 3D Open Worlds

Weihao Tan, Xiangyang Li, Yunhao Fang, Heyuan Yao, Shi Yan, Hao Luo, Tenglong Ao, Huihui Li, Hongbin Ren, Bairen Yi, Yujia Qin, Bo An, Libin Liu, Guang Shi

arXiv:2511.08892v1cs.AI

TL;DR

Open-world agents must handle diverse, long-horizon tasks, but many existing agents remain specialized for constrained environments. Lumine combines pixel-based perception, language-grounded reasoning, and keyboard-mouse control in a staged training recipe; it completes long missions in Genshin Impact and transfers to other games without fine-tuning.

  • Problem

    Agents that master closed environments and single objectives can remain brittle, with limited transfer and adaptability to open-world ambiguity and diversity.

    Source: §1 · p. 3

  • Method

    Lumine is a vision-language-model-based agent trained through a staged curriculum to perceive raw pixels, generate keyboard-mouse actions, and reason when needed.

    Source: §1 · p. 3

  • Results

    Lumine completes Mondstadt’s Act I with expert human-level efficiency, performs comparably on Acts II and III, and completes missions in Wuthering Waves and Honkai: Star Rail without fine-tuning.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The authors present Lumine as an open recipe for transforming a small open-source VLM into a real-time agent for diverse tasks and hours-long missions in complex 3D open worlds.

    Source: §9 · p. 31

    In cross-game play, Lumine can misread on-screen key prompts and use Genshin-specific names for entities in Wuthering Waves.

    Source: §8.3.2 · p. 30

Abstract

from arXiv · show

We introduce Lumine, the first open recipe for developing generalist agents capable of completing hours-long complex missions in real time within challenging 3D open-world environments. Lumine adopts a human-like interaction paradigm that unifies perception, reasoning, and action in an end-to-end manner, powered by a vision-language model. It processes raw pixels at 5 Hz to produce precise 30 Hz keyboard-mouse actions and adaptively invokes reasoning only when necessary. Trained in Genshin Impact, Lumine successfully completes the entire five-hour Mondstadt main storyline on par with human-level efficiency and follows natural language instructions to perform a broad spectrum of tasks in both 3D open-world exploration and 2D GUI manipulation across collection, combat, puzzle-solving, and NPC interaction. In addition to its in-domain performance, Lumine demonstrates strong zero-shot cross-game generalization. Without any fine-tuning, it accomplishes 100-minute missions in Wuthering Waves and the full five-hour first chapter of Honkai: Star Rail. These promising results highlight Lumine's effectiveness across distinct worlds and interaction dynamics, marking a concrete step toward generalist agents in open-ended environments.

1 Introduction

Lumine addresses the gap between specialized agents for closed tasks and generalist behavior in open worlds through a language-grounded recipe for unified perception, reasoning, and action. Its reported results span short instruction-following tasks, long Genshin Impact missions, and zero-shot transfer to other games.

1 Introduction

  • Prior agents have achieved strong performance in constrained games but remain tied to closed environments, single objectives, and explicitly shaped rewards, limiting transfer and adaptability.Source: §1 · p. 3
  • The introduction identifies language as a potential shared medium for grounding knowledge across modalities and domains, while noting that existing language-grounded interactive agents have not reached human-level accuracy and efficiency.Source: §1 · p. 3
  • The paper frames generalist open-world agents around six challenges: scalable environments, multimodal perception, high-level planning, low-level control, memory, and real-time inference.Source: §1 · p. 3
  • The recipe combines raw-pixel perception at 5 Hz with keyboard-and-mouse action generation at 30 Hz and reasoning that is invoked when needed.It builds on Qwen2-VL-7B-Base and uses a three-stage curriculum: 1,731 hours of gameplay, 200 hours of instruction data, and 15 hours of reasoning data.Source: §1 · p. 3
  • Lumine retains up to 20 recent steps as short-term memory and reasoning steps as long-term memory; end-to-end optimization reduces overall latency by 25.3×.Source: §1 · p. 4
  • During pretraining, capabilities emerge in sequence: object interaction, basic combat and GUI manipulation, then game-specific mechanisms and navigation.The authors report that structured visuomotor competence can arise from large-scale human-gameplay imitation without explicit supervision.Source: §1 · p. 4
  • Lumine follows instructions on short-horizon tasks lasting 10 seconds to several minutes, with a success rate exceeding 80% and generalization to unseen objectives and scenarios.Source: §1 · p. 4
  • Lumine completes Act I of Mondstadt’s main storyline with expert human-level efficiency and performs comparably on Acts II and III, which were excluded from reasoning training.It also completes a 100-minute Wuthering Waves mission and the five-hour first chapter of Honkai: Star Rail without fine-tuning.Source: §1 · p. 4

2 Related Work

Related work contrasts specialized agents and prompt-based systems with the demands of open-ended, real-time interaction. Lumine’s design draws on language grounding, flexible reasoning, contextual memory, and human-style input while targeting long-horizon tasks.

2 Related Work

  • Traditional game agents optimize policies for closed objectives, while prompt-based language and vision-language agents can reason over long tasks but struggle with precise actions, fine visual patterns, and latency.The paper positions Lumine as a data-driven approach intended to combine instruction-following versatility with long-horizon autonomy.Source: §2 · pp. 4–5
  • Table 1 compares game agents by environment open-endedness, longest task duration, vision and text understanding, instruction following, reasoning, real-time inference, and interface.Source: §2 · p. 4, Table 1 · p. 5
  • Existing game environments often use fixed maps and distinct observation-action conventions, whereas commercial games offer richer dynamics but commonly lack API access.Human-like screen, keyboard, and mouse interaction is presented as a way to reach arbitrary PC games.Source: §2 · p. 5
  • Commercial game missions provide long-horizon objectives, while VLA models typically handle tasks lasting seconds to minutes and prompt-based agents have demonstrated hour-scale play.Lumine aims to combine VLA instruction-following with prompt-based long-horizon autonomy.Source: §2 · p. 5
  • Lumine adopts hybrid thinking: it can generate explicit reasoning when needed or directly produce actions, rather than reasoning at every control step.The related-work discussion contrasts this with hierarchical systems that separate reasoning and action, and ReAct-style systems that reason at each step.Source: §2 · p. 6
  • Lumine treats context as memory by retaining recent observations and using reasoning to summarize the past and plan future goals, without specialized memory modules.Source: §2 · p. 6
  • Human-style pixel, keyboard, and mouse control is suited to games because relative mouse movement and key holds or releases can carry action meaning that coarse GUI inputs miss.Code-like action formats are also described as inefficient for these interactions.Source: §2 · p. 6
  • Lumine combines action chunking, traditional inference optimization, and efficient action modeling to target real-time inference in gaming environments.The discussion distinguishes game actions, whose keys and mouse movements have interpretable semantic intent, from low-level robotic actions.Source: §2 · p. 6

3 Environment

The paper selects Genshin Impact as a representative commercial game for developing agents in a content-rich environment. Its open world, varied tasks, and built-in guidance are balanced against practical barriers to autonomous interaction and scalable training.

3 Environment

  • Commercial games offer high-quality graphics, expansive worlds, and complex systems, but their closed-source nature leaves them largely unexplored as agent-development environments.Genshin Impact is selected as a representative case for a general and scalable solution.Source: §3 · p. 7
  • Genshin Impact offers extensive traversal across diverse terrain and regions, with movement such as climbing, gliding, swimming, sailing, and diving.Fully traversing and understanding the world typically requires weeks or months of continuous exploration.Source: §3 · p. 7
  • Its main-story missions and side quests span hundreds of hours, while character and team progression can unfold over weeks or months.These extended tasks contrast with short-term objectives in traditional research benchmarks.Source: §3 · pp. 7–8
  • The game combines exploration, combat, navigation, NPC dialogue, quests, progression, resource management, and puzzles involving environmental clues and multi-stage reasoning.It also includes 2D and 3D mini-games and events, offering heterogeneous tasks within one virtual world.Source: §3 · p. 8
  • On-screen tutorials introduce new mechanics and remain available for review, while a gradual difficulty curve provides a staged learning curriculum.Source: §3 · p. 8
  • The game’s large global player base and active online community provide potential sources of gameplay data, guides, walkthroughs, and annotations.Source: §3 · p. 8
  • Commercial games generally require screen pixels and keyboard-mouse input, impose latency constraints with asynchronous actions, and hide internal states and rewards.GPU rendering also limits scalability to thousands of parallel rollouts, making reinforcement learning particularly difficult.Source: §3 · pp. 8–9

4 The Lumine Model

Lumine is a 7B-parameter vision-language model that turns pixel observations and interaction history into executable keyboard and mouse actions. It combines optional intermediate reasoning with a compact action representation for closed-loop control.

Model Overview

  • Lumine processes raw pixels and historical context to generate executable keyboard and mouse actions, with interpretable reasoning for closed-loop visual decision-making.It builds on Qwen2-VL-7B-Base [77].Source: §4 · p. 9, Figure 3 · p. 9

Hybrid Thinking

  • At each step, Lumine may generate reasoning before predicting an action, conditioning both on visual observations and prior reasoning and actions.When it does not reason explicitly, the reasoning variable is null.Source: §4 · pp. 9–10

Keyboard and Mouse Modelling

  • Lumine unifies keyboard and mouse control in a concise language-space representation, avoiding changes to the model architecture or vocabulary.The action format encodes mouse displacement and scrolling alongside key presses.Source: §4 · p. 10
  • Six 33 ms action chunks form a 200 ms window, yielding 30 Hz control while supporting varied key combinations and mouse-button actions.Consecutive chunks retain held keys; omitted keys are released.Source: §4 · pp. 10–11

Training Recipe

  • The three-stage recipe progresses from image–action pre-training to language-grounded instruction following and then reasoning-conditioned action generation.These stages produce Lumine-Base, Lumine-Instruct, and Lumine-Thinking, respectively.Source: Figure 4 · p. 11

5 Data Curation and Training Recipe

Lumine’s training recipe builds from human gameplay demonstrations, first teaching action primitives, then language grounding, and finally explicit reasoning for long-horizon missions.

5 Data Curation and Training Recipe

  • The three training stages move from reactive control on raw observations to language-grounded short tasks and explicit reasoning for complex, long-horizon decisions.The stages yield Lumine-Base, Lumine-Instruct, and Lumine-Thinking.Source: §5 · pp. 11–12

5.1 Pre-Training

  • 2424 hours of synchronized gameplay recordings provide image–action data; rule-based filtering removes 95% of idle actions and camera-jitter-dominated clips, leaving 1731 hours for pre-training.Pre-training also mixes in approximately 20% multimodal web data to retain general knowledge.Source: §5 · p. 12, §5.1 · p. 12

5.2 Instruction Following

  • Instruction fine-tuning uses a label-then-augment pipeline to ground action prediction in language, producing 200 hours of curated instruction-following data.The pipeline begins with 165 hours of human annotations and uses classifier labeling plus GPT-4.1 instruction generation and verification.Source: §5.2 · pp. 12–13

5.3 Reasoning

  • Human-annotated inner monologues from Mondstadt Act I provide 15 hours of reasoning data and 15K traces for fine-tuning Lumine-Instruct into Lumine-Thinking.The traces average 3.2 seconds apart; action filtering is omitted so the model can learn to wait at decision points.Source: §5.3 · p. 13

5.4 Training Details

  • Training covers history and non-history settings; history samples use multi-turn image–action context, while reasoning trajectories end at a new thought or after 20 frames.Pre-training history runs continued to improve across epochs, whereas non-history runs tended to overfit after epoch two.Source: §5.4 · pp. 13–14

6 Inference

Lumine combines context management with communication, decoding, and infrastructure optimizations to meet real-time control demands. The system reduces latency, but stale visual inputs and reasoning pauses remain practical constraints.

6.1 Context Management

  • Context management retains a 20-image–action-pair history window, discards the oldest pair first, and flushes the context when new reasoning is generated.The system prompt and prior reasoning remain available to guide action generation.Source: §6.1 · p. 14

6.2 Real-Time Optimization

  • 25.3× overall speedup over the baseline; without reasoning, first-action delay is about 110 ms, and the longest action chunk takes 12 ms.The reported thresholds are 200 ms before action generation and 33 ms per action chunk.Source: Figure 7 · p. 15, §6.2 · p. 16
  • Streaming lets Lumine execute each of six 33 ms action chunks as soon as it is produced, rather than waiting for the full action sequence.The 5 Hz control cycle requires completing capture, inference, and execution within 200 ms.Source: §6.2 · p. 15
  • Prefill reuses historical KV cache and applies StreamingLLM when dropping old turns; clearing context at new reasoning alleviates its observed long-horizon performance degradation.Decoding uses compact actions and draft-model-less speculative decoding with stage-specific delimiters.Source: §6.2 · pp. 15–16
  • Infrastructure optimizations include tensor parallelism across four GPUs, W8A8 quantization, tuned kernels, a fused CUDA graph, and GPU-based image preprocessing.The all-reduce kernel and speculative decoding further reduce communication and CPU latency.Source: §6.2 · p. 16
  • Visual inputs are roughly 200 ms old, and reasoning steps can exceed 200 ms and briefly idle the agent, though their low frequency caused no noticeable visual impact in observed gameplay.The authors note stale input may cause side effects in highly time-sensitive scenarios.Source: §6.2 · p. 16

7 Benchmark

The benchmark evaluates language-conditioned open-world gameplay across four task categories and three difficulty levels, using human assessment under specified game conditions.

Benchmark Overview

  • The benchmark comprises 141 language-conditioned tasks across collection, combat, NPC interaction, and puzzle-solving categories.Each category includes simple, hard, and unseen tasks.Source: §7 · p. 16, Figure 8 · p. 17

Task Categories

  • Tasks test varied demands, including spatial navigation and object recognition, enemy-specific combat, NPC dialogue and GUI manipulation, and observation-based puzzle solving.Combat tasks may also require retrieving a chest after defeating enemies.Source: §7 · pp. 16–17

Task Difficulty

  • Simple and hard tasks are set in Mondstadt, while unseen tasks use new regions and entities in Liyue or Inazuma; unseen puzzles are excluded because they depend on region-specific mechanics.Simple tasks are generally short-horizon; hard tasks add distractors, hidden targets, vertical navigation, or elite enemies.Source: §7 · p. 17

Evaluation Setting

  • Agents receive textual objectives at predefined locations, and humans evaluate performance; tasks are generally run five times under specified character and equipment conditions.The benchmark compares Lumine with GPT-5, Gemini 2.5 Pro, Grok4, Doubao1.6-Vision, and Qwen3-VL-235B-A22B-Thinking through the Cradle framework.Source: §7 · pp. 17–18

8 Experimental Results

The experiments evaluate Lumine’s capability growth from pretraining and instruction following through long-horizon reasoning, as well as its transfer to unseen missions and games. They report task completion and reasoning accuracy alongside efficiency limits and persistent failure modes.

8.1 Scaling Results

  • Beyond 1,200 training hours, 2B benchmark performance degrades despite declining loss, while 7B maintains improvements and lower loss, motivating the choice of 7B as the primary base.The reported scaling comparison covers 2B and 7B models during the first pretraining epoch without history.Source: §8.1 · p. 18, Figure 9 · p. 19
  • Core skills emerge at different data scales: object interaction before 100 hours, combat and GUI manipulation around 1,000, and game mechanics and navigation after 1,800, with navigation still imperfect.Navigation strengths include following roads and quest markers, avoiding obstacles, stopping at edges, and gliding during falls; remaining issues include hesitation, inefficient stamina use, and mistimed actions.Source: §8.1 · pp. 19–20

8.2.1 Non-History Performance

  • Lumine-Instruct-NonHis gains 61% over its base model and exceeds 80% success in all four simple-task categories, doubling base-model success on combat tasks.It also improves NPC Interaction and Puzzle performance by more than 50% and Collection by 23%, with more consistent goal-conditioned behavior.Source: §8.2.1 · pp. 20–21, Figure 10 · p. 20
  • On hard tasks, Lumine-Instruct-NonHis struggles with combat, off-ground collection, and puzzles; models trained only on instruction data generally perform worse, especially on hard tasks.The instruction-only ablation performs slightly better on NPC Interaction, suggesting pretraining can bias models toward passing NPCs without engaging them.Source: §8.2.1 · p. 21
  • On unseen tasks, NPC Interaction remains comparable to in-domain performance, while Collection and Combat decline because new items and enemy attack patterns are unfamiliar.In-context learning also helps: detailed instructions, prior knowledge, or task decomposition can enable completion of tasks that previously had low or zero success.Source: §8.2.1 · pp. 21–22, Figure 13 · p. 22

8.3.1 Performance in Genshin Impact

  • Lumine-Thinking completed the Mondstadt act in 56 minutes, versus 78 minutes for fresh human players and 53 minutes for expert players, and substantially outperformed instruct models across the five subtasks.The history-trained model had one failure, caused by falling from a platform and failing to generate intermediate reasoning to invoke quest guidance; it later timed out while trying to recover.Source: Figure 17 · p. 24, §8.3.1 · p. 25, Table 5 · p. 25

8.2.2 Benefits of History

  • History helps real-time navigation and GUI control under asynchronous delays and reduces visual target-tracking errors, but history models show more low-level control errors and modality conflicts.Other common failures include inconsistent instruction following and spatial misjudgments; across models, multimodal understanding accounts for nearly half of errors.Source: §8.2.2 · pp. 23–24, §8.3.1 · p. 25

8.3.1 Performance in Genshin Impact

  • 593 history-setting reasoning instances had an 8.8% error rate, versus 14.0% across 708 non-history instances; error types were similar, but linguistic collapse appeared occasionally only without history.Common errors included perceptual misrepresentation, premature completion, causal misjudgment, self-contradiction, and instruction unfollowing; low-quality reasoning did not necessarily cause task failure.Source: §8.3.1 · pp. 25–26, Figure 18 · p. 26
  • 4.7 hours versus 3.6 hours for expert humans: Lumine completed Acts II and III, which were in pre-training data but excluded from reasoning data, and transferred reasoning to unfamiliar mechanisms.It recognized a moving stone platform despite a different layout and background, but reasoning was less reliable for unfamiliar mechanisms and puzzles.Source: §8.3.1 · pp. 26–27, Figure 19 · p. 27, Figure 20 · p. 28
  • In Genshin Impact, weak fast-travel use and unreliable minimap understanding cause inefficient navigation, while absent health-recovery patterns leave the party without active healing.Lumine often follows quest markers on foot, can stray when markers disappear, and does not use Statues of The Seven or food to recover health.Source: §8.3.1 · pp. 27–28
  • A four-second, 20-frame memory span contributes to marker-driven oscillation and abandoned detours, while combat remains less efficient than expert play, especially in multi-wave encounters and boss fights.Combat shortcomings include skill coordination, aiming accuracy, dodging timing, and mechanics understanding.Source: §8.3.1 · p. 28
  • In fully unseen Liyue, Lumine reached Liyue Harbor and an Adeptus dwelling, then completed the mission after a quest-tracking error led to about two hours of aimless exploration.The mountainous terrain, new characters, enemies, and puzzles made Liyue a challenging out-of-domain environment.Source: Figure 19 · p. 27, §8.3.1 · pp. 28–29

8.3.2 Generalization to New Games

  • 107 minutes versus 101 minutes for fresh human players: Lumine completed Wuthering Waves’ first two storyline missions while handling exploration, combat, and 2D GUI tasks.The game was unseen and its release postdated Qwen2-VL’s knowledge cutoff, reducing the likelihood of prior-game knowledge from pretraining.Source: §8.3.2 · p. 30
  • 7 hours versus 4.7 hours for fresh human players: Lumine completed Honkai: Star Rail’s first chapter, defeated the Doomsday Beast, cleared the tutorial, and reached Jarilo-VI.Despite the game’s turn-based combat and hub-based exploration, it demonstrated reasonable navigation, NPC interaction, and GUI manipulation.Source: §8.3.2 · p. 30
  • Across new games, transferable navigation, combat, and GUI skills coexist with domain-bias and perception failures that can stall progress or prevent longer-task completion.In Wuthering Waves, OCR confuses F with E and reasoning imports Genshin names; in Honkai: Star Rail, jump-based route attempts and misread turn-based controls cause repeated failures; in Black Myth: Wukong, invisible walls, hidden UI, and health-bar confusion impede play.Source: §8.3.2 · pp. 30–31

9 Discussion and Future Work

Lumine combines an open, end-to-end recipe with a prototype agent that completes long missions and transfers across games. The discussion identifies data scale, long-term memory, and inference efficiency as limitations, while outlining possible game-development and entertainment uses.

Discussion and Future Work

  • Lumine’s zero-shot transfer to unseen missions and games suggests transferable meta-skills, including 3D navigation and 2D manipulation.The recipe spans environment selection, data collection and preprocessing, interaction frequency, model design, training, and inference optimization.Source: §9 · p. 31

Scaling

  • The pretraining data covered only Mondstadt, while reasoning data covered only the first hour, leaving scope to scale both datasets across games and domains.The authors identify this scaling as potential to enhance generalization and robustness.Source: §9 · p. 31

Long-Term Memory

  • The current memory uses previous reasoning steps for long-term memory and a 20-frame context window for short-term memory, but remains inadequate for complex, long-horizon missions.The authors propose more sophisticated retrieval and management over thousands of turns.Source: §9 · p. 32

Online Learning

  • Offline data is efficient, but the authors say it limits Lumine’s ability to surpass human-level performance; online reinforcement learning is proposed for exploration and self-improvement.The proposed direction builds on the existing Lumine model.Source: §9 · p. 32

Real-Time Inference

  • Lumine uses tensor parallelism across multiple GPUs for real-time inference but still has delay, motivating more efficient strategies to reduce computational overhead.The authors say improved efficiency could also speed reinforcement-learning rollouts.Source: §9 · p. 32

Practical Applications

  • The authors describe potential uses in game testing and interactive entertainment, including quality assurance, usability evaluation, and AI-driven assistants or streaming.They say these applications could help developers reduce manual testing costs and improve development efficiency.Source: §9 · p. 32

Ethics Statement

The authors state that the work follows ethical standards for AI research involving human gameplay data and virtual environments. They describe participant consent and data use, adherence to game terms, and a call for community dialogue on responsible governance.

Ethics Statement

  • The authors state that the study adheres to ethical standards for research involving AI systems, human gameplay data, and virtual environments.They also say the work complies with the terms of service of the games used.Source: Ethics Statement · p. 32
  • Gameplay recordings came from consenting adults compensated at fair market rates, who were told recordings would be used solely for AI-agent research..Source: Ethics Statement · p. 32
  • The authors say Lumine is for academic research, not cheating or commercial competitive advantage, and oppose uses undermining fair play or player experience.They acknowledge potential misuse as a game cheat or unauthorized automation tool.Source: Ethics Statement · p. 32
  • The authors call for collaboration among researchers, developers, and players on governance models balancing innovation, creativity, and fairness.They propose shared ethical standards and technological safeguards as part of responsible development.Source: Ethics Statement · p. 32

Contributions

The collection appendix describes gameplay annotation and participant recruitment, including the language of the recorded data and baseline eligibility requirements.

A Gameplay Collection

  • Gameplay annotators were native Chinese speakers playing the Chinese version of Genshin Impact, so curated instructions and reasoning annotations were produced in Chinese..Source: A Gameplay Collection · p. 39

A.1 Participant Recruitment

  • Recruitment sought annotators with strong gameplay ability and comprehension, including Genshin Impact accounts at Adventure Rank 45 or higher or frequent experience with PC-based 3D games..Source: §A.1 · p. 39
  • Eligibility required participants to be 18–40 years old, hold an associate degree or higher, and use devices meeting minimum specifications: Windows 10 or Windows 11, Intel Core i5-12400 or better, and 16 GB or higher memory.Source: §A.1 · p. 39

A.2 Annotation Tasks

The appendix details annotation tasks, standardized gameplay recording, and conversion of recordings into frame-action data. It describes complementary mouse-input logging and processing choices for overworld and GUI scenes.

A.2 Annotation Tasks

  • Annotators started with new accounts and were asked to complete Mondstadt Acts I–III and reach over 80% exploration in each region, with about 30 hours of gameplay per annotator on average.The protocol constrained characters, settings, region, tutorials, gameplay habits, use of guides, and spending.Source: §A.2 · p. 39
  • After standardized training and quality control, 70 qualified participants yielded 2,424 hours of raw gameplay footage.The collection ran from March to May 2025 across Genshin Impact versions 5.4–5.6.Source: §A.2 · pp. 39–40

A.3 Gameplay Recording

  • Recording software standardized participants’ display settings because hardware and software variation could distort mouse movement and introduce display artifacts.Gameplay video was captured at 1080p and 60 fps; keyboard events and absolute mouse coordinates were logged alongside relative mouse movement.Source: §A.3 · p. 40

A.4 Post-Process

  • The processing pipeline aligns input events with video frames and reconciles absolute cursor positions with relative mouse displacements into frame-action pairs.Because recording startup could misalign video and inputs by about 800 ms to 2 seconds, timestamps were used to synchronize them.Source: §A.4 · p. 41
  • For overworld scenes, mouse motion is represented by summed relative movements; for GUI scenes, the pipeline uses consecutive-frame absolute-position differences and template matching to select the method.Keyboard states are reconstructed from events, and mouse movements are discretized in 5-pixel X and 4-pixel Y units; the input-token mapping is described in Table 6.Source: §A.4 · p. 41, Table 6 · p. 42
  • Replaying processed data usually reproduced game events over a few seconds, but drift emerged beyond ten seconds or during large camera rotations.The authors attribute drift to stochastic game mechanics such as camera collision-avoidance adjustments.Source: §A.4 · p. 41

B Instruction Following Data Curation

The instruction-following data combines a hierarchical taxonomy and human annotation with classifier-assisted labeling and GPT-4.1 instruction generation. A 2B classifier was selected for production after both tested model sizes exceeded 80% average precision with only a marginal performance difference.

B Instruction Following Data Curation

  • A three-level taxonomy organizes gameplay into Game Scene, Game Content, and fine-grained activities, with 38 distinct categories spanning Overworld, Domain, and GUI Interface tasks.The taxonomy differentiates broad scenes and activities, with examples including specific puzzle-solving tasks.Source: B Instruction Following Data Curation · p. 42, Figure 23 · p. 43
  • Human annotators exhaustively label gameplay segments and timestamps in 20-second clips, producing 165 hours of fine-grained classification data through iterative quality checks.Clips with classification or timestamp errors are returned for re-annotation.Source: B Instruction Following Data Curation · p. 42, Figure 23 · p. 43
  • The classifier is trained on five-frame samples drawn from 165 hours of 720p gameplay footage labeled frame by frame, with category down-sampling to reduce imbalance.The balanced set is split 90%/10% into training and test data.Source: B Instruction Following Data Curation · p. 42
  • Both classifier sizes achieve over 80% average precision with a marginal performance difference, so the 2B model is chosen for production throughput and efficiency.Table 7 compares accuracy across model sizes.Source: B Instruction Following Data Curation · p. 43, Table 7 · p. 43
  • Classifier-labeled gameplay transitions are used to generate context-aware instructions, with GPT-4.1 verifying samples before filtering yields 200 hours of instruction-following data.Category labels guide instruction generation around task-context transitions.Source: B Instruction Following Data Curation · p. 43

C Reasoning Data Curation

Reasoning data is built from experienced players’ first-person accounts of decisions during gameplay. Keyframes and annotation rules structure each account as a temporally grounded sequence of thoughts, scene interpretation, and planned action.

C Reasoning Data Curation

  • Experienced Genshin Impact players annotate the first hour of gameplay by identifying key decision frames and describing the player’s thought process in first person.Each 10-second segment pairs sampled visual frames with raw action logs.Source: C Reasoning Data Curation · p. 44
  • Keyframes mark situations with multiple options, noticeable changes requiring a response, or a completed or invalidated line of reasoning that calls for a new goal.Examples include a fork in the road, a mission completion, or a new instruction.Source: C Reasoning Data Curation · p. 44
  • Each keyframe annotation follows a three-part structure: summarize the previous step, analyze essential elements of the current scene, and plan the next move.Source: C Reasoning Data Curation · p. 44
  • Annotation rules favor natural, intention-oriented thoughts that guide the next 5–20 seconds, precede the corresponding action, and focus combat notes on significant mechanics.The rules aim to form a coherent chain of thoughts linking what the player sees with what they decide.Source: C Reasoning Data Curation · p. 44
  • The main character is consistently named the Traveler rather than by the player’s chosen name.Source: C Reasoning Data Curation · p. 44

D Benchmark Introduction

The benchmark presents representative tasks from four categories, distinguished by difficulty, and the appendix also supplies system prompts for instruction-following and reasoning. The tasks cover collection, combat, NPC interaction, and puzzles, while the prompts specify short action sequences and when reasoning is requested.

D Benchmark Introduction

  • The benchmark contains 141 tasks across Collection, Combat, NPC Interaction, and Puzzle, with representative examples marked as simple, unseen, or hard.The examples are a subset, categorized by difficulty and visually distinguished with background colors.Source: D Benchmark Introduction · p. 45

D.1 Collection

  • Collection has 62 tasks: 21 simple, 31 unseen, and 10 hard, each presented with a starting point and instruction.Source: §D.1 · p. 45

D.2 Combat

  • Combat has 21 tasks—8 simple, 7 unseen, and 6 hard—all using the instruction to defeat nearby monsters and open the chest.Source: §D.2 · p. 46

D.3 NPC Interaction

  • NPC Interaction has 21 tasks: 7 simple, 8 unseen, and 6 hard.Source: §D.3 · p. 47

D.4 Puzzle

  • Puzzle has 23 tasks, with 9 simple and 14 hard examples, including opening thorn-wrapped chests, activating elemental monuments, and using a pinwheel to reach a chest.Source: §D.4 · p. 48

E Prompt

  • The system prompts provide English translations of the original Chinese and ask for six 33 ms action steps over the next 200 ms, with reasoning only when needed in the reasoning prompt.The action format encodes mouse movement and keyboard groups; the instruction-following prompt also plans six steps.Source: E Prompt · pp. 49, 51

Related papers

Loading 2511.08892v1…