PaperlayerView exact revision on arXiv

Source-linked AI summary

GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay

Yiran Wang, Xingyilang Yin, Junfu Pu, Guangzhi Wang, Kaifeng Li, Mingyu Ouyang, Huiqiang Sun, Lingen Li, Cheng Cheng, Wangbo Yu, Honghao Chen, Xiaodong Cun, Chi-Man Pun, Zhiguo Cao, Ying Shan

arXiv:2609.25001v1cs.CVcs.AI

TL;DR

Gameplay benchmarks lack broad game coverage, comprehensive instructions, and reproducible evaluations across model types. GameHorizon combines automated multi-horizon annotation, aligned gameplay data, and offline and stepwise online testing; its evaluations reveal a hierarchy of task difficulty and model capabilities.

  • Problem

    Existing gameplay datasets and benchmarks have narrow game coverage, incomplete language instructions, or high-variance online evaluations, limiting systematic comparisons across planning, instruction following, and action execution.

    Source: §1 · p. 2

  • Method

    GameHorizon combines an automated three-level instruction pipeline, a dataset aligning gameplay videos, actions, and instructions, and reproducible offline and stepwise online benchmarks.

    Source: §1 · pp. 2–3

  • Results

    7.2 percentage points: medium- and long-horizon instructions improve future-action planning over vision-only input, amid a hierarchy of task difficulty and model capabilities across 47 models.

    Source: §1 · p. 3

  • Takeaways & Limitations

    GameHorizon can serve as a unified yardstick for systematically examining gameplay performance across horizons and model families.

    Source: §5 · p. 14

Abstract

from arXiv · show

Modern video games provide a measurable testbed for AI models, combining abilities of visual understanding, instruction decomposition, goal planning, and precise action control over multiple temporal horizons. Existing datasets and benchmarks, however, either cover a narrow range of games, lack language instructions, or rely on high-variance online rollouts. To address these challenges, we introduce GameHorizon, a unified data and evaluation suite that measures gameplay capabilities at different horizons for diverse model families. GameHorizon Suite consists of three components. First, GameHorizon-Annotator is a scalable and automated annotation pipeline for multi-horizon instructions. Second, utilizing the pipeline, we construct GameHorizon-Data, the first large-scale AAA gameplay dataset with temporally aligned videos, player actions, and multi-horizon instructions. It comprises 5,000 hours of recordings from 21 games, collected by 100 human expert players. Third, we build GameHorizon-Bench with reproducible offline and stepwise online testing. The offline track enables reproducible evaluation using thousands of standardized questions organized into three primary tasks and a series of diagnostic variants, while the online track tests whether offline scores reflect actual gameplay capabilities and localizes failures to specific steps within long-horizon gameplay. Based on our GameHorizon Suite, we evaluate 47 models through more than one million model invocations, revealing a meaningful hierarchy of task difficulty and pronounced differences in model capabilities. Our work can provide a standardized yardstick for evaluating gameplay capabilities across horizons and model families. We will release our dataset, annotator, and benchmark to facilitate future research.

1 Introduction

GameHorizon addresses gaps in gameplay data and evaluation with a unified suite spanning multiple temporal horizons and diverse games. Its automated instruction pipeline, aligned dataset, and reproducible offline and stepwise online benchmark support evaluation across model types.

1 Introduction

  • Existing resources have narrow game coverage, incomplete language-instruction annotations, or small, harness-dependent online evaluations that limit reproducibility and failure diagnosis.Aggregate success rates can obscure whether a model misidentifies actions, infers goals incorrectly, or fails to map goals to controls.Source: §1 · p. 2
  • GameHorizon-Annotator automatically builds a three-level instruction pyramid, abstracting fine-grained instructions into short-horizon operations, medium-horizon goals, and long-horizon strategies.The pipeline operates bottom-up.Source: §1 · p. 2
  • GameHorizon-Data aligns videos, player actions, and multi-horizon instructions across 5,000 hours of gameplay collected from 100 expert players in 21 game titles.The introduction describes it as the first publicly available dataset with this alignment.Source: §1 · p. 2
  • GameHorizon-Bench combines reproducible offline evaluation with stepwise online testing, covering action, instruction-decomposition, and cross-horizon-consistency tasks.Offline evaluation uses thousands of standardized questions; online subtasks and environment resets support verification and failure localization.Source: §1 · pp. 2–3
  • Across 47 models, results show a hierarchy of task difficulty and capability gaps; planning is harder than current-action choice, while longer-horizon instructions improve future-action planning by 7.2 percentage points over vision-only input.The paper also reports a positive association between online success rates and offline scores.Source: §1 · p. 3

2 Related Work

Earlier gameplay datasets and benchmarks are limited by narrow game coverage, sparse instructions, and small or hard-to-reproduce online evaluations. GameHorizon offers a unified evaluation across multiple horizons and model types.

2 Related Work

  • Prior datasets often cover one game or simplified mini-games, lack comprehensive language instructions, or rely on too few online rollouts for confident, reproducible comparisons.The resulting benchmarks may also collapse distinct failure modes into aggregate success rates.Source: §2 · p. 3
  • Dedicated game agents tend to prioritize high-frequency action control over long-horizon reasoning, while general-purpose models have not been systematically evaluated on action execution.Evaluations using bespoke harnesses remain difficult to compare across models.Source: §2 · pp. 3–4
  • GameHorizon provides a standardized yardstick for evaluating different models on goal planning, instruction following, and executable actions across multiple horizons.Its unified suite includes diverse AAA games and reproducible offline-online benchmarks.Source: §2 · pp. 3–4

3 GameHorizon Suite

GameHorizon combines human-recorded gameplay, automated multi-horizon instruction annotation, and reproducible offline and stepwise online evaluation. Its suite is designed to assess gameplay across temporal horizons and model families.

3.1 Overview

  • GameHorizon records synchronized 2K video and timestamped keyboard-mouse actions from 100 experienced players, forming the basis for its aligned dataset.The dataset contains 5,000 hours across 21 titles, with 6,184,036 instructions and every frame aligned to instructions at all three horizons. It jointly provides large-scale AAA gameplay, direct human actions, and dense multi-horizon instructions.Source: §3.1 · pp. 4–5, §3.3 · p. 8

3.2 GameHorizon-Annotator

  • The bottom-up annotator segments action-aware L1 clips, merges coherent clips into L2 goals and L3 strategies, and generates instructions using level-specific inputs.Actions ground L1 and L2 instructions in player controls, while L3 omits action traces to focus on high-level strategies.Source: §3.2 · p. 6

3.4 GameHorizon-Bench

  • The offline track uses thousands of standardized MCQs across T1 action choice, T2 goal decomposition, T3 cross-horizon consistency, and ten diagnostic variants.Standardized inputs and independence from game environments and agent harnesses support reproducible comparisons across model families.Source: §3.4 · pp. 8–9, Figure 4 · p. 8
  • Diagnostic variants compare action perception with future planning, top-down decomposition with bottom-up abstraction, and instruction-only with instruction-and-action options.These comparisons probe how instructions affect decisions and whether bottlenecks arise from goal planning or action decoding.Source: §3.4 · p. 9
  • The online track tests whether offline scores are positively associated with gameplay capabilities using order-dependent causal and order-flexible thematic tasks.Tasks comprise 2–6 verifiable subtasks; after a failure, resetting to the corresponding success state lets evaluation continue and localizes errors to specific stages.Source: §3.4 · p. 9

4 Experiments

Experiments compare model families using standardized offline questions and stepwise online gameplay tasks. Results distinguish model capabilities, expose differences across task types, and show that long-horizon gameplay remains difficult.

4.1 Evaluation Protocols and Details

  • The offline track uses standardized four-option questions across 47 models, while online testing samples 12 models and evaluates causal and thematic tasks through 62 short-horizon subtasks.The offline benchmark contains 5,000 questions; each online subtask has a maximum budget of 400 model calls.Source: §4.1 · p. 10

4.2 Offline Primary Tasks

  • 64.7% mean accuracy exceeds the 25% random baseline, while T1, T2, and T3 score 57.3%, 65.1%, and 71.6%, respectively.Model accuracy spans 44.6%–80.2%; T1 is hardest under strict action-prediction criteria, while T3 is easiest with rich multi-horizon information.Source: §4.2 · pp. 10–11, Table 4 · p. 11
  • General-purpose VLMs lead the model-family results; UMMs and GUI agents score lower, coding agents are competitive, and dedicated game agents approach random on unseen AAA games.JARVIS-VLA scores 54.2% on in-domain Minecraft data but 28.1% on unseen AAA games.Source: §4.2 · pp. 11–12
  • Thinking improves some models but reduces accuracy for others, with gains linked to stronger goal decomposition and losses associated with hallucinated reasoning.GLM-5V-Turbo and Doubao-Seed-2.1-Pro gain 5.6 and 9.3 percentage points, while UI-TARS-1.5-7B and Qwen2.5-VL-7B lose 8.2 and 5.9 points.Source: §4.2 · p. 12, Table 6 · p. 12

4.3 Offline Variant Tasks

  • Action perception outperforms future-action planning, while short-horizon instructions particularly improve perception and multi-horizon instructions further improve planning.The reported perception–planning gaps are 9.9, 23.8, and 18.2 percentage points across the three input settings; multi-horizon instructions add 7.0 points for planning.Source: §4.3 · p. 13
  • Bottom-up abstraction scores 98.4% versus 69.5% for top-down decomposition, while adding action sequences to cross-horizon consistency reduces accuracy by 0.6 points.The authors use bottom-up abstraction for annotation and top-down decomposition as a challenging evaluation task.Source: §4.3 · pp. 13–14, Table 8 · p. 13

4.4 Stepwise Online Track

  • Online performance aligns with offline rankings: models in offline Tiers 1–4 occupy online ranks 1–3, 4–6, 7–9, and 10–12, respectively.The online track tests 12 models, with three selected from each offline tier.Source: §4.4 · p. 14, Table 9 · p. 14
  • Long-horizon tasks remain difficult: GPT-6-Astra achieves 45.0% success, while ten of twelve models achieve at most 10.0%; stepwise resets localize failures to specific operations.The protocol can distinguish lacking knowledge of a procedure from performing one of its steps incorrectly.Source: §4.4 · p. 14

5 Conclusion

The conclusion presents GameHorizon as a multi-horizon suite for diverse AAA games and model families, combining an aligned dataset with reproducible evaluation and stepwise diagnosis.

5 Conclusion

  • GameHorizon-Annotator automatically builds an instruction pyramid of short-horizon operations, medium-horizon goals, and long-horizon strategies.The conclusion describes the pipeline as automatic.Source: §5 · p. 14
  • GameHorizon-Data aligns gameplay videos, actions, and multi-horizon instructions across 21 AAA game titles.The conclusion identifies it as a large-scale dataset.Source: §5 · p. 14
  • GameHorizon-Bench combines reproducible offline evaluation with stepwise online testing for standardized comparisons and fine-grained failure diagnosis.The authors describe the suite as a data foundation and unified yardstick for examining gameplay across model families and temporal horizons.Source: §5 · p. 14

Appendix

The appendix lists further annotation-pipeline details and additional benchmark results.

Appendix

  • The appendix includes further details on GameHorizon-Annotator, including annotation prompts, segmentation, and keybinds.Source: Appendix · p. 18
  • It also includes additional GameHorizon-Bench results, such as qualitative examples from the online track.Source: Appendix · p. 18

A.1 Annotation Prompts

The annotation prompts build instructions at three horizons from sampled gameplay frames and, for shorter horizons, aligned actions and existing instructions.

A.1 Annotation Prompts

  • The short-horizon prompt uses sampled frames and aligned keyboard-mouse actions to describe the current operation with critical visual details.Game-specific keybinds help interpret the controls in their in-game context.Source: §A.1 · p. 18
  • The medium-horizon prompt adds short-horizon instructions and summarizes the player’s goal without excessive local details.It also supplies game-specific keybinds.Source: §A.1 · p. 18
  • The long-horizon prompt uses sampled frames and medium-horizon goals, omitting actions and keybinds to capture overall strategy.Across horizons, prompts include video metadata and request a natural-language instruction without additional explanation.Source: §A.1 · p. 18

A.2 Action-Aware Video Segmentation

Action-aware segmentation maps raw inputs to gameplay semantics, builds and cleans action segments, then uses a VLM to refine ambiguous or vision-dependent cases. The pipeline preserves concurrent evidence and produces continuous action events.

A.2 Action-Aware Video Segmentation

  • The four-stage pipeline calibrates keyboard-mouse semantics, classifies primitive actions, applies rule-based coarse segmentation, and refines segments with a VLM.It removes temporal gaps and overlaps to produce segments with continuous action events.Source: §A.2 · p. 18
  • Ambiguous inputs retain multiple candidate actions, while primitive actions are classified as input-deterministic, input-ambiguous, or vision-dependent.When actions coincide in one frame, priority rules select a primary label and preserve remaining actions as concurrent evidence.Source: §A.2 · p. 18
  • Frame-aligned input logs recover held keys, discrete triggers, mouse-button states, and camera movements before segments are merged, smoothed, or divided.Consecutive frames with identical labels are merged, short noisy segments smoothed, and overly long segments divided.Source: §A.2 · p. 18
  • A VLM refines ambiguous or vision-dependent segments using video, candidate labels, action semantics, and trigger timestamps; PySceneDetect is also compatible.The text notes that visual-similarity boundary detection can fragment continuous actions during large camera movements.Source: §A.2 · p. 18

A.3 Game-Specific Keybinds

Game-specific keybinds map raw keyboard and mouse inputs to in-game action semantics; Apex Legends and Cyberpunk 2077 provide representative examples.

A.3 Game-Specific Keybinds

  • Game-specific keybinds map raw keyboard and mouse inputs to their corresponding in-game action semantics.The section references Apex Legends and Cyberpunk 2077 as representative examples.Source: §A.3 · p. 18
  • Table 10 presents the keybinds for Apex Legends.Multiple candidate semantics for an ambiguous input are separated by slashes.Source: §A.3 · p. 18
  • Table 11 presents the keybinds for Cyberpunk 2077.Multiple candidate semantics for an ambiguous input are separated by slashes.Source: §A.3 · p. 18

B More Results on GameHorizon-Bench

Additional benchmark materials describe horizon-specific annotation prompts, online task execution, and examples of long-horizon tasks and their constituent subtasks.

B More Results on GameHorizon-Bench

  • The appendix provides game-specific keybind tables for Apex Legends and Cyberpunk 2077, with slash-separated candidate meanings for ambiguous inputs.The mappings connect keyboard and mouse inputs to in-game action semantics.Source: Table 10 · p. 19, Table 11 · p. 19
  • Short-horizon prompts combine sampled frames and aligned keyboard-mouse actions to describe current operations with object and spatial detail.Medium-horizon prompts add constituent short-horizon instructions and summarize sustained goals; long-horizon prompts use frames and medium-horizon goals to capture strategy without actions or keybinds.Source: Figure 6 · p. 20, Figure 7 · p. 21, Figure 8 · p. 22
  • A long-horizon task passes only if every subtask succeeds; after a failure, the environment resets to the corresponding success state so evaluation can continue.Figure 9 shows Gemini 3.6 Flash rollouts with arrows marking its actual execution order.Source: B More Results on GameHorizon-Bench · p. 18, Figure 9 · p. 23
  • The task examples include thematic subtasks that share a theme but can occur in any order, such as completing an exterior wall using a finished house as reference.Another example groups six block placements to complete the target house’s wall.Source: Figure 9 · p. 23, B More Results on GameHorizon-Bench · p. 24
  • The online track includes 20 long-horizon tasks and 62 short-horizon subtasks, divided into fixed-order causal tasks and any-order thematic tasks.The examples include a causal sequence of mining iron, smelting it, crafting a sword, and using it to defeat a zombie.Source: Table 12 · p. 23, B More Results on GameHorizon-Bench · p. 24

Related papers

Loading 2609.25001v1…