PaperlayerView exact revision on arXiv

Source-linked AI summary

PlaySuite: A Large-Scale Benchmark for Interactive Visual Intelligence

Dheeraj Varghese, Anna Vettoruzzo, Walter Simoncini, Michelle Lorena Acevedo Callejas, Mohammad Mahdi Derakhshani, Kristof Meding, Joaquin Vanschoren, Cees G. M. Snoek

arXiv:2610.07127v1cs.CVcs.AI

TL;DR

Static benchmarks leave open how well multimodal models act competently in dynamic environments over extended time horizons. PlaySuite combines a large open-game corpus, a unified interaction framework, and milestone-based judging to evaluate fourteen open models. Models make meaningful progress on only a minority of games, with recurring interaction failures.

  • Problem

    Many multimodal benchmarks assess perception and reasoning on static inputs, leaving open how well models translate those capabilities into goal-directed interaction over time.

    Source: §1 · p. 2

  • Method

    PlaySuite evaluates agents across thousands of games using a unified closed-loop harness and a Video-LLM judge that maps observable gameplay milestones to progress levels.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Across fourteen open models, meaningful gameplay progress occurs on only a minority of games, with recurring difficulties in spatial grounding, control selection, action execution, and adaptation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    PlaySuite provides an open, extensible testbed for measuring interactive behavior across diverse game environments.

    Source: §1 · p. 2, §6 · p. 9

    The evaluation measures zero-shot generalization rather than fine-tuned or reinforcement-learning performance, excludes real-time reaction speed, and does not directly measure strategy, reasoning quality, or game-mechanic mastery.

    Source: §6 · p. 9

Abstract

from arXiv · show

Recent advances in multimodal foundation models yield strong performance on static perception and reasoning benchmarks, yet such evaluations largely overlook a central aspect of intelligence: acting competently in dynamic environments over extended time horizons. We introduce PlaySuite, a large-scale benchmark for evaluating interactive visual intelligence across more than 5K open-source video games curated from PyWeek and itch.io. Spanning diverse genres and engines, including Pygame, HTML5, Godot, and Unity, these independent games are largely out-of-distribution for current models, reducing the likelihood that success can be achieved by retrieving memorized walkthroughs or web-scale training artifacts. To enable scalable evaluation across heterogeneous titles, we develop a unified closed-loop interaction framework optimized for HPC clusters alongside a Video-LLM-as-a-judge protocol that maps observable gameplay milestones to standardized progress levels. We evaluate fourteen recent open models spanning vision-language models, computer-use agents, and vision-language-action models. Our results yield strong evidence of a perception-action gap: despite strong reasoning capabilities, current models struggle to make sustained progress and exhibit recurring failures in spatial grounding, action execution, and self-correction. PlaySuite provides a reproducible and extensible testbed for measuring progress from visual perception to goal-directed interaction, and a foundation for developing models that can act, adapt, and generalize in dynamic visual environments.

1 Introduction

Static perception and reasoning benchmarks leave open how well multimodal models act toward goals in dynamic environments. PlaySuite evaluates that gap across thousands of games and finds that current models make meaningful progress on only a minority of them.

1 Introduction

  • Games combine computer use, spatial reasoning, temporal dynamics, and goal-directed behavior, while prior demonstrations and benchmarks cover relatively few environments.This limited coverage raises questions about how broadly demonstrated capabilities generalize.Source: §1 · p. 2
  • PlaySuite spans 5,734 distinct video games and offers over 100 times more game environments than prior multi-game benchmarks, drawn from a larger pool of approximately 17.5K launchable games validated by its pipeline.Source: §1 · p. 2
  • PlaySuite combines an engine-spanning interaction layer with milestone-based Video-LLM judging to enable scalable, comparable evaluation without game-specific rewards.The judge maps observable gameplay milestones to progress levels.Source: §1 · p. 2
  • Current models make meaningful gameplay progress on only a minority of games, with recurring difficulties in spatial grounding, control selection, action execution, and adaptation over time.The evaluation covers fourteen open models spanning vision-language models, computer-use agents, and vision-language-action models.Source: §1 · p. 2
  • The benchmark is released with environment wrappers, evaluation protocols, and high-performance cluster tooling, and is designed to grow as new environments emerge.Its three contributions are a heterogeneous game benchmark, unified HPC-optimized infrastructure, and milestone-based evaluation.Source: §1 · p. 2

2 Related Work

Prior work has moved from language-model planning toward multimodal perception-action systems and interactive benchmarks. PlaySuite positions itself as a large-scale, unified assessment across diverse game mechanics, addressing limitations in the breadth and engineering requirements of existing game benchmarks.

Foundation models in interactive environments

  • Multimodal systems now ground instructions in pixels or predict control tokens, but still struggle with effective action, long-horizon planning, and robust spatial reasoning.Earlier LLM work emphasized reasoning-chain planning and recursive feedback for adapting to environmental shifts.Source: §2 · p. 3

Game-based benchmarks

  • Game benchmarks have progressed from 2D or grid-based settings to 3D environments, but many remain single-title or small bespoke suites with hand-crafted rewards.The 3D settings add perspective, depth, and occlusion, while earlier 2D environments did not capture real-world spatial nuance.Source: §2 · p. 3
  • PlaySuite synthesizes modular harnesses and judge-based assessment into a unified evaluation across a large variety of game mechanics.The related-work comparison highlights scale, infrastructure, and dynamic evaluation.Source: §2 · p. 3

3 PlaySuite Construction

PlaySuite assembles a broad collection of open-source games and a common interaction and judging protocol for evaluating agents across heterogeneous environments. Its milestone-based scores provide a standardized measure of observable gameplay progress.

Corpus

  • The benchmark curates 5,734 open-source games from PyWeek and itch.io, spanning diverse 2D and 3D dynamics and heterogeneous game engines.Games are executed in isolated sandboxes with virtual displays for scalable cluster evaluation.Source: §3 · pp. 3–4, Figure 2 · p. 4

Filtering and taxonomy

  • Games are filtered for community ratings, platform compatibility, playability, and safety, while a nine-genre taxonomy combines itch.io tags with LLM categorization of PyWeek titles.A custom triage pipeline quarantines high-risk archives and runs other games in network-isolated sandboxes.Source: §3 · p. 4

Evaluation pipeline

  • Agents receive a unified prompt, raw pixel observations, and optional interaction history, then must infer game mechanics and objectives without supplied solution strategies.Their structured output contains a reasoning trace and an action that must generalize across thousands of environments.Source: §3 · p. 4

Milestone-based evaluation

  • Observable events such as score increases, new areas, and objective completion map to five ordered progress levels, from None to Completed.Partial progress requires reaching a new area or increasing score; completion requires explicit visual evidence of game completion.Source: §3 · p. 4

Judge validation

  • The default Qwen3.5-9B judge agrees with human ratings within one level in 90.4% of cases, versus 93.0% for human–human pairs.Qwen3.5-27B reaches 91.6% within one level, while Qwen3.5-9B is selected for practicality at scale.Source: §3 · p. 5

4 PlaySuite Experimental Architecture

PlaySuite standardizes the game interface and model prompting while separating game execution from inference on HPC clusters. Its shared inference worker batches requests across concurrent workers to improve evaluation throughput.

Input and output configuration

  • Prompts provide game metadata, observations, and optional history; models must infer undocumented controls, and the template directs them to change strategy when actions yield no visible progress.Model-specific formats and coordinate conventions are translated through adapters into the shared keyboard-and-mouse interface.Source: §4 · p. 5

Temporal context and history

  • A rolling history of prior observations and actions is intended to support temporal reasoning and long-horizon planning, with lengths of 0, 3, 5, and 10 evaluated.The history is intended to help models capture relationships between actions and environmental transitions.Source: §4 · p. 5

Baseline models

  • The evaluation compares 14 open-weight models across general-purpose VLMs, computer-use agents, and vision-language-action models.Proprietary systems are excluded; model architectures are included to compare how different modeling priors handle zero-shot game interaction.Source: §4 · pp. 5–6

Batched multi-worker execution

  • Eight concurrent workers raise Qwen3-VL-8B throughput on one H100 from 6.7 to 42.4 evaluations per GPU-hour.A shared vLLM worker batches frame-and-prompt requests while game workers run independently in sandboxed virtual displays.Source: §4 · p. 6, Figure 3 · p. 6
  • The shared architecture uses one model copy in GPU memory and separates CPU-bound game execution from GPU-bound inference.Game workers send requests over a Unix socket, keeping game processes network-isolated.Source: §4 · p. 6

5 PlaySuite Benchmark Results

PlaySuite results show that current models make meaningful progress on only a minority of games, with outcomes varying by history condition and genre. Additional analyses find modest gains from faster control, mixed effects from prompting and history, and greater stability for broad model rankings than for fine-grained comparisons.

Overall performance

  • CUAs remain competitive with similarly sized VLMs, while evaluated VLAs generally trail the strongest VLM and CUA models.Across the benchmark, meaningful progress remains limited to a minority of games.Source: §5 · p. 6
  • Qwen3-Omni leads no-history mean Progress Score at 0.84 on itch.io and 1.26 on PyWeek, while Gemma-4-26B leads with history at 1.02 and 1.37, respectively.Qwen3-Omni reaches at least Partial progress in 19.5% of itch.io runs and 37.0% of PyWeek runs without history.Source: §5 · p. 6, Table 2 · p. 7

Performance across game genres

  • Among genres with at least 60 games, Simulation and Strategy yield the highest mean progress, while Adventure and Puzzle yield the lowest; genre rankings correlate 0.88–0.99 with overall rankings.The genre-specific model-ranking correlation is Spearman ρ = 0.88 to 0.99.Source: Table 3 · p. 7, §5 · p. 7

Qualitative failure modes

  • Observed failures span misread entities, inaccurate clicks, unsupported controls, repetitive loops, passive waiting, poor instruction following, and state mismanagement.In Goob, Gemma-3 misplaces the coin and repeatedly presses “d” despite recognizing the arrow-key instruction and making no progress.Source: §5 · p. 7, Figure 4 · p. 8

Effect of action frequency

  • On PyWeek, raising interaction frequency from 0.33 Hz to 3 Hz increases mean Progress Score from 0.862 to 0.926, with no clear evidence that frequency changes the effect of history.Source: §5 · p. 7

How many games are needed?

  • With 200 sampled itch.io games, median rank correlation is 0.937, but the full-corpus top-three set appears in only 26.0% of samples; Figure 5 contrasts sampled and full-corpus rankings.Broad model ordering stabilizes with a few hundred games, while finer comparisons require larger samples.Source: Figure 5 · p. 8, §5 · p. 8

Impact of prompting strategies and temporal history

  • Neither explicit strategy traces nor few-shot examples improve average performance, and few-shot prompting reduces progress on both subsets; longer history also lowers average progress.Across history lengths k ∈ {0, 3, 5, 10}, no history yields the highest average progress on both corpora, although individual models can benefit from history.Source: §5 · p. 8, Figure 6 · p. 9

Cost-performance analysis

  • Qwen3-Omni improves mean progress over Qwen2.5-VL but requires 105.5 versus 59.9 H100 GPU-hours per 1,000 games.Source: §5 · p. 9

6 Conclusion

PlaySuite evaluates interactive visual intelligence across a large distribution of open-source games, where agents must turn perception and reasoning into gameplay actions. Across fourteen open models, meaningful progress remains limited to a minority of games, and the evaluation does not measure several aspects of real-time or strategic performance.

Conclusion

  • PlaySuite evaluates fourteen open VLMs, CUAs, and VLAs across heterogeneous open-source games, finding meaningful progress on only a minority and recurring perception-action failures.Observed issues include spatial grounding, control selection, action execution, passive waiting, and effective use of temporal context.Source: §6 · p. 9
  • The benchmark currently validates approximately 17.5K executable, safe, reproducible environments, of which 5,734 are used in this evaluation.The paper frames this broad environment distribution as a way to study generalization beyond demonstrations concentrated in a few individual games.Source: §6 · p. 9

Limitations

  • Zero-shot evaluation measures out-of-the-box generalization, not maximum performance after fine-tuning or RL, and the protocol excludes model-inference latency from game dynamics.Milestone-based scoring does not directly measure strategic efficiency, reasoning quality, or mastery of game mechanics; public games and metadata also leave contamination risk unresolved.Source: §6 · p. 9

A.1 Data curation methodology and technical validation

The curation pipeline filters candidate games for quality, launchability, technical viability, and safety before producing the evaluation collections. Platform-specific pruning leaves a stable PyWeek set and a substantially larger itch.io collection.

Data curation methodology and technical validation

  • The initial pool contained 1,460 PyWeek games and 1.7 million itch.io games, with a community-rating quality gate and platform-specific eligibility checks.Itch.io candidates had to be free, fully released, rated by at least five users, Linux- or web-compatible, keyboard- or mouse-controllable, and launch successfully.Source: §A.1 · p. 15
  • PyWeek pruning removed deprecated-Python titles, restricted the set to Pygame, and used automated dry-runs to remove runtime bugs and broken assets, leaving 104 stable games.An intermediate technical-pruning stage left 376 games before the Pygame restriction and dry-runs.Source: §A.1 · p. 15
  • The itch.io audit combined NSFW tag filtering with archive-risk triage and sandboxing, producing a final collection of 5,630 games.High-risk archives were quarantined for manual review; lower-risk or clean archives were executed in a network-isolated sandbox, and unreliable games were excluded after technical validation.Source: §A.1 · p. 15

A.2 Game genre taxonomy and categorization

PlaySuite consolidates platform metadata and game-AI taxonomies into nine canonical genres designed to reflect both computational challenges and source-platform mechanics. It uses different classification procedures for itch.io and PyWeek games.

Game genre taxonomy and categorization

  • The taxonomy reconciles 17 itch.io tags with game-AI literature and industry tagging, grouping games into nine genres with descriptions that ground automated classification.The design aims to reflect computational challenges such as reactive control versus long-term reasoning, as well as source-platform mechanics.Source: §A.2 · p. 16
  • The nine categories include Action, Adventure, Puzzle, and Shooter, each defined by its characteristic gameplay objective or activity.Source: §A.2 · p. 16
  • The remaining categories are Simulation, Survival, Strategy, Fighting, and Racing, each described by its characteristic activity or player objective.Source: §A.2 · p. 16
  • For itch.io, a rule-based merge maps original tags to canonical genres and resolves multiple tags by choosing the more specific category; games without genre metadata are classified by an LLM.Specificity is defined by the inverse number of original itch.io tags mapped to a PlaySuite genre.Source: §A.2 · p. 17, Table 6 · p. 17
  • For PyWeek, Qwen2.5-14B-Instruct infers a primary genre from game descriptions, README files, and diary entries using the taxonomy descriptions and examples.Using the same qualitative category descriptions as prompts is intended to align PyWeek classifications semantically with itch.io.Source: §A.2 · p. 17

A.3 Licensing and Terms of Use

PlaySuite states that it respects the source platforms’ and developers’ rights and distribution terms. Its PyWeek and itch.io games are distributed or used under the conditions described for each source.

A.3 Licensing and Terms of Use

  • PyWeek entries retain their original files; the benchmark is distributed free under the submission license, unless a game’s own license applies.The stated license permits redistribution, copying, and running without modification, and distribution of unmodified screenshots without charge.Source: §A.3 · p. 17
  • The itch.io corpus is limited to publicly distributed, free-to-play titles, whose binaries and assets are not modified.The framework uses the files for non-commercial research and benchmarking and provides metadata and links to original game pages.Source: §A.3 · p. 17
  • The evaluation set spans nine canonical genres across 5,630 itch.io games and 104 PyWeek games.The figure presents the distribution across the nine genres but the supplied passage gives no genre-level counts.Source: Figure 8 · p. 19

A.4 Model configurations

The paper evaluates 14 baselines and documents their model families, parameter scales, checkpoints, prompts, and output parsers.

A.4 Model configurations

  • The evaluated baselines are summarized by model family, parameter scale, Hugging Face checkpoint, and reference.The table covers 14 models.Source: §A.4 · p. 19, Table 7 · p. 19
  • Table 7 identifies the checkpoint and reference associated with each baseline.The table also lists its family and parameter scale.Source: Table 7 · p. 19, §A.4 · p. 19
  • Prompting and output parsing are part of each evaluated model configuration.The model-specific assignments are summarized in Table 8.Source: §A.4 · p. 19

A.5 Prompt Templates

The appendix details the benchmark’s prompts, execution and judging setup, and additional analyses. Its measurements report throughput gains, limited benefits from added history or prompt structure, and differences across models and genres.

A.7.3 Human and cross-judge validation

  • Qwen3.5-9B agrees within one progress level with human labels in 90.4% of comparisons; human-only Krippendorff’s α is 0.624.Including the judge as a seventh rater yields α = 0.605; the validation uses 60 gameplay videos and six blinded annotators.Source: §A.7.3 · p. 22, Table 10 · p. 22

A.8 Benchmark execution efficiency

  • 90 s to 8.3 s per-game launch overhead and 6.7 to 42.4 evaluations per GPU-hour with eight workers summarize the harness optimizations.Readiness checks and browser/display reuse cut launch overhead; shared vLLM serving across workers raises throughput, while higher concurrency risks safety timeouts.Source: Figure 10 · p. 23, §A.8 · p. 23
  • 160 to 370 videos per GPU-hour summarizes judge throughput after optimization; end-to-end throughput is approximately 277 itch.io videos per GPU-hour.The controlled-set gain comes from overlapping video loading and decoding with inference and increasing batched-token capacity, without changing comparable-run milestone levels.Source: Figure 10 · p. 23, §A.8 · p. 23

A.9.1 Performance across game genres

  • Adding k = 3 history reduces mean progress in every reported genre on both subsets; itch.io progress is higher in Simulation and Strategy than Adventure and Puzzle.PyWeek shows greater model variation across genres, with small counts in several genres limiting interpretation.Source: §A.9.1 · p. 23, Table 11 · p. 24, Figure 11 · p. 24

A.9.3 Impact of prompting strategies

  • Neither prompt addition improves average performance: few-shot examples reduce PyWeek progress from 0.789 to 0.720 and itch.io progress from 0.538 to 0.504.An explicit Strategy field reduces PyWeek progress from 0.785 to 0.747 and leaves itch.io nearly unchanged at 0.538 to 0.535; the ablation covers eight VLMs.Source: §A.9.3 · p. 25, Figure 14 · p. 26

A.9.2 Effect of action frequency

  • On PyWeek, raising action frequency from 0.33 Hz to 3 Hz raises mean progress from 0.942 to 1.036 without history and 0.778 to 0.823 with k = 3 history.The improvement is 0.049 points larger without history, but the reported interval does not show clear evidence of an interaction; per-model history effects correlate at r = 0.96.Source: §A.9.2 · pp. 23, 25, Figure 13 · p. 26

A.9.4 Impact of temporal history

  • No history has the highest mean progress on both corpora: PyWeek falls from 0.946 at k = 0 to 0.740 at k = 10, while itch.io falls from 0.616 to 0.511.Intermediate history lengths do not improve on k = 0 in the reported means.Source: §A.9.4 · p. 25, Figure 15 · p. 27

A.9.5 Cost–performance analysis

  • Qwen2.5-VL scores 0.822 at 59.9 GPU-hours per 1,000 games, while Qwen3-Omni scores 0.841 at 105.5 GPU-hours; the failure example shows Qwen2.5-VL waiting after mistaking an uneventful opening for completion.The cost measure includes startup, idle allocation, retries, and failed or interrupted jobs; the appendix also supplies prompt templates and control instructions for the evaluated agents.Source: §A.5 · pp. 19–20, Figure 16 · p. 27, §A.9.5 · p. 27

Related papers

Loading 2610.07127v1…