PaperlayerView exact revision on arXiv

Source-linked AI summary

GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents

Mingyu Ouyang, Siyuan Hu, Kevin Qinghong Lin, Hwee Tou Ng, Mike Zheng Shou

arXiv:2604.07429v1cs.CVcs.AIcs.HC

TL;DR

GameWorld addresses the lack of standardized, verifiable evaluation for multimodal game agents facing heterogeneous interfaces and demanding interactive tasks. It benchmarks two normalized agent interfaces across diverse browser games using deterministic state-based metrics and extensive robustness analyses. Results show meaningful partial progress but remain far from reliable task completion and human-level performance, motivating improvements in action grounding, trajectory memory, and latency robustness.

  • Problem

    Multimodal game-agent evaluation is hindered by heterogeneous action interfaces and heuristic verification despite games demanding fine-grained perception, long-horizon planning, and precise control.

  • Method

    GameWorld evaluates Computer-Use and Generalist agents across browser games using a shared runtime, deterministic Semantic Action Parsing, and outcome-based state-verifiable metrics.

  • Results

    Across 34 games, 170 tasks, and 18 model–interface pairs, agents often make meaningful partial progress but remain far from reliable task completion and human-level performance.

  • Takeaways & Limitations

    GameWorld provides a standardized, verifiable, and reproducible foundation for studying multimodal agents in complex, open-ended interactive environments.

  • Takeaways & Limitations

    Designing unique instruction sets for each new environment tightly couples the action space to the task and constrains scalability.

Abstract

from arXiv · show

Towards an embodied generalist for real-world interaction, Multimodal Large Language Model (MLLM) agents still suffer from challenging latency, sparse feedback, and irreversible mistakes. Video games offer an ideal testbed with rich visual observations and closed-loop interaction, demanding fine-grained perception, long-horizon planning, and precise control. However, systematically evaluating these capabilities is currently hindered by heterogeneous action interfaces and heuristic verification. To this end, we introduce GameWorld, a benchmark designed for standardized and verifiable evaluation of MLLMs as generalist game agents in browser environments. Two game agent interfaces are studied: (i) computer-use agents that directly emit keyboard and mouse controls, and (ii) generalist multimodal agents that act in a semantic action space via deterministic Semantic Action Parsing. GameWorld contains 34 diverse games and 170 tasks, each paired with state-verifiable metrics for outcome-based evaluation. The results across 18 model-interface pairs suggest that even the best performing agent is far from achieving human capabilities on video games. Extensive experiments of repeated full-benchmark reruns demonstrate the robustness of the benchmark, while further studies on real-time interaction, context-memory sensitivity, and action validity expose more challenges ahead for game agents. Together, by offering a standardized, verifiable, and reproducible evaluation framework, GameWorld lays a robust foundation for advancing research on multimodal game agents and beyond. The project page is at https://gameworld-bench.github.io.

1 Introduction

GameWorld addresses the difficulty of evaluating multimodal game agents by combining diverse browser games with standardized interfaces and outcome-based state verification. Its analyses indicate that agents make partial progress but remain far from reliable, human-level task completion.

  • Games provide a demanding testbed because agents must interpret changing visual scenes, commit to consequential actions, and recover from mistakes over many steps.Browser games are lightweight, mechanically diverse, and easy to reset.
  • GameWorld evaluates multimodal game agents across 34 browser games, five genres, and 170 diverse tasks.
  • Repeated reruns, real-time evaluation, context-memory analysis, and action-validity diagnostics assess benchmark robustness and interface-conditioned behavior.
  • Current agents often make meaningful partial progress but remain far from reliable task completion and human-level performance.
  • The benchmark standardizes two interfaces—Computer-Use Agents and Generalist Multimodal Agents—under a shared executable action space.Generalist agents use deterministic Semantic Action Parsing, while Computer-Use Agents emit keyboard and mouse controls.
  • Outcome-based metrics computed from serialized gameAPI state provide deterministic task success and normalized progress without perceptual noise.

2 Game Agent

GameWorld standardizes heterogeneous game-agent outputs by defining two interfaces and normalizing them into a shared executable control space. The harness further controls prompting, memory, and tool interfaces while exposing latency as a challenge in real-time play.

  • GameWorld defines Computer-Use and Generalist interfaces to standardize heterogeneous model outputs across game-agent implementations.
  • Both interfaces are normalized into atomic human-computer interaction events under a shared runtime contract.The executor-level events include mouse and keyboard actions, scrolling, and waiting.
  • Computer-Use Agents must emit exact coordinates and key sequences, making the interface sensitive to inference latency in real-time settings.Evaluation enforces one executable action per step and rejects controls outside the permitted game interface.
  • Semantic Action Parsing deterministically maps Generalist semantic actions to fixed low-level interaction commands.
  • The shared agent harness standardizes structured prompts, short- and long-term memory, and model-specific tool interfaces for sustained gameplay.The fixed prompt template contains game rules, role and controls, task instruction, and output format; rolling memory records recent interaction rounds.
  • Longer reasoning can improve visual inspection and decision making but also introduces latency that may harm game-agent performance.

3 GameWorld Benchmark

GameWorld is a browser-based benchmark designed to address limited game diversity, heterogeneous action interfaces, latency confounds, and noisy evaluation. It combines diverse tasks, a latency-controlling sandbox, and state-verifiable outcome metrics across game-agent settings.

  • Benchmark Design: Existing game benchmarks often cover few games and narrow genres, while heuristic, OCR, or VLM-based evaluation introduces verification noise.
  • Games and Tasks: GameWorld organizes 34 browser games and 170 task instructions across Runner, Arcade, Platformer, Puzzle, and Simulation genres.The genres target reactive control, spatial navigation, logical reasoning, long-horizon planning, resource management, and 3D spatial reasoning.
  • Games and Tasks: Each task pairs an open-ended natural-language instruction with a quantitative target, fixed step budget, and verifiable evaluator.Agents receive no intermediate guidance and must act autonomously from visual observations.
  • Games and Tasks: Success Rate measures target completion as SR ∈{0, 1}, while Progress measures normalized advancement toward the objective as PG ∈[0, 1].Progress provides partial credit for incomplete runs.
  • Browser-Based Sandbox Environment: The sandbox pauses game execution during model inference so scores reflect decision quality rather than response latency.It also supports isolated browser instances, readiness gating, configurable game speed, and deterministic seeds.
  • Outcome-Based State-Verifiable Evaluation: Outcome-based evaluation reads serialized gameAPI state to compute deterministic success and progress signals without perceptual noise.The evaluator uses task-relevant fields such as score, level, coordinates, lives, coins, and checkpoints, with task-level stopping and reset rules.

4 Experiments

GameWorld evaluates 18 model–interface pairs using a shared paused protocol, deterministic state-based metrics, and repeated analyses of performance, robustness, capabilities, real-time interaction, memory, and action validity. Agents make partial progress but remain substantially below human performance, with persistent challenges in long-horizon planning, timing, latency, and reliable action execution.

  • Experimental Setup: 18 model–interface pairs, comprising 8 Computer-Use Agents and 10 Generalist Agents, are evaluated across the benchmark.The study covers 13 base models under the two interfaces.
  • Experimental Setup: Each task is evaluated under a shared paused protocol with one action per step, fixed action durations, and a maximum budget of 100 actions.Pausing separates decision quality from response speed during the main evaluation.
  • Main Results: Overall PG peaks at 41.9 for Gemini-3-Flash-Preview among Generalist agents and 39.8 for Seed-1.8 among Computer-Use Agents, while SR remains only 12.4–21.2%.The gap between progress and success indicates frequent partial advancement without full task completion.
  • Main Results: Simulation games are broadly difficult, whereas Runner games yield the highest progress for many models.Simulation difficulty is associated with open-ended objectives and longer-horizon state tracking.
  • Main Results: Human players reach 55.3 SR / 64.1 PG under the same action budget, leaving the best current agents far below novice performance.The comparison highlights unresolved challenges in robust control, long-horizon planning, and reliable completion.
  • Benchmark Robustness: Repeated reruns produce stable aggregate measurements with low single-digit PG variation, although larger variance remains in a limited set of difficult control-sensitive games.Hextris, Cubefield, Wordle, and World’s Hardest Game 2 show visibly larger per-game variation.
  • Real-Time Interaction: GameWorld-RT keeps the environment running during inference, coupling response latency with action relevance; the faster 30B model is still challenged despite the 235B model’s slightly higher progress.Success rates remain very low across the four real-time settings.
  • Action Validity and Memory: Under long interactive contexts, weaker models more often forget the available action space and emit non-executable or non-permitted tool calls.Invalid Action Rate measures failures involving parsing, role constraints, or parser checks and serves as an instruction-following proxy.

5 Case Study

The case studies compare interfaces and expose distinct failure modes in game-agent interaction. They show that semantic control, long-horizon completion, and precise timing each create different demands beyond locally plausible actions.

  • Interface Comparison: Matched Mario trajectories isolate interface effects: CUA emits low-level keyboard and mouse actions, while Generalist agents produce richer semantic plans and actions.The backbone and game environment are held constant in the comparison.
  • Long-Horizon Simulation: Minecraft Clone reaches 90% progress yet fails to complete the resource-collection target within the step limit.The trajectory illustrates missing closure rather than instruction-following failure.
  • Real-Time Timing: In Flappy Bird, visually similar consecutive frames require alternating waiting and flapping, so a slightly early or late action can determine progress.The example demonstrates why timing control is mechanically decisive under tight constraints.

6 Related Work

Related work evaluates game agents through modular game benchmarks, long-horizon play, execution-based interaction, and generalist agents operating across diverse digital worlds. These efforts motivate standardized interfaces and interactive evaluation for multimodal agents.

  • Interactive Evaluation: OSWorld demonstrates execution-based evaluation for computer-use tasks, complementing game-specific efforts with interactive environment assessment.This line of work evaluates agents through real interaction rather than static outputs.
  • Generalist Agents: Generalist digital-world agents use interfaces ranging from native keyboard-mouse inputs to multimodal control across many games.Game-TARS and Jarvis-VLA exemplify agents acting through shared or direct interaction interfaces.

7 Conclusion

GameWorld evaluates multimodal game agents through standardized, verifiable browser-game tasks and shows that current agents often make partial progress without reliable completion. Its analyses identify interface-conditioned weaknesses and motivate improvements in reasoning, action grounding, trajectory memory, and latency robustness.

  • Repeated evaluation, real-time, context-memory, and action-validity analyses expose weaknesses across interfaces under standardized outcome-based evaluation.
  • Stronger agents will require better reasoning, more reliable action grounding, more useful trajectory memory, and greater robustness to latency.
  • Each new environment requires a unique instruction set, tightly coupling the action space to the task and constraining scalability.Automating Semantic Action Parsing production and alignment through MLLM-powered exploration remains future work.
  • Across 34 games, 170 tasks, and 18 model–interface pairs, agents often make meaningful partial progress yet remain far from reliable task completion and human-level performance.

A Benchmark Runtime

The benchmark runtime composes reusable game, task, and model configurations into standalone or expanded suite runs. This decomposition supports flexible evaluation while isolating concurrent executions.

  • A run preset combines <game_id>, <task_id>, and <model_spec> into a concrete runtime configuration.The three components are resolved independently before composition.
  • <game_id> supplies game rules, roles, low-level controls, and semantic actions, while <task_id> supplies instructions, evaluators, target metrics, budgets, and URL suffixes.
  • <model_spec> supplies model identifiers, provider overrides, prompt templates, and output-format prompts.
  • Changing a task or swapping models does not require duplicating the underlying game configuration.The decomposition keeps game definitions, tasks, and model profiles reusable and flexible.
  • Suite files expand benchmark cases across games, tasks, and models, then group child runs into repeat waves with bounded parallelism and isolated execution.

B Observation-Action-Evaluation Loop

Each interaction round converts the current game observation into an executable action, applies it, and records a verifiable state snapshot. Evaluation then combines terminal, budget, target, and task-specific stopping conditions, with optional reset-on-fail continuation.

  • The runtime wraps each model in an Agent and coordinates it with a browser GameEnv and Evaluator for task-progress tracking.
  • Each round captures a screenshot, optionally pauses inference, obtains the model response, resumes gameplay, parses an executable action, executes it, and captures game state.
  • Stopping or resetting can be triggered by terminal status, step-budget exhaustion, reaching the target score, or a task-specific end-field rule.
  • The evaluator receives the state snapshot, step index, maximum budget, target threshold, and accumulated metrics to determine progress and termination.
  • When continue_on_fail is enabled, terminal failure resets the task, assigns a new episode ID, waits for reinitialization, and continues under the same global step budget.

C Browser Sandbox and Game API

The browser sandbox launches each game in Chromium with controlled environment behavior and prepares it for interaction through readiness checks. It also configures deterministic and presentation-preserving browser instrumentation.

  • The GameLauncher serves the game locally, while the browser manager launches Chromium with a fixed viewport and disabled background throttling.
  • Initialization injects dynamic speed-control and deterministic-randomness scripts by overriding JavaScript headers.
  • Screenshots use the Chrome DevTools Protocol instead of standard page screenshots to avoid visible flashing in headed mode.
  • The environment starts by opening the game in Chromium and waiting until the readiness gate makes it actionable.

C.2 Readiness Gate

The readiness gate ensures that evaluation begins only when a game is actionable, filtering transient loading and menu phases before decisions and after resets.

  • The runtime waits for an actionable game state before the first action and after every reset.The default actionable statuses are ready and playing.
  • Table 12 lists the status values consumed by the readiness gate and evaluator.

C.3 Verifiable State: Game API Schema

GameWorld standardizes verifiable evaluation through a serializable gameAPI that exposes common state fields while preserving game-specific details. The runtime also normalizes, validates, and executes actions through shared interfaces and maintains configurable multimodal interaction history.

  • Every game exposes window.gameAPI methods init(config), reset(options), and getState() for verifiable state access.The returned state includes identifiers, timing, lifecycle status, terminal metadata, structured game state, task metrics, and raw details.
  • The common Game API contract remains stable across games even though game-specific fields vary.
  • The example schema separates structured game_state, comparable metrics, and optional raw game-specific details.The example includes score, progress, player state, counters, and terminal information.
  • The client records each interaction as user_prompt, screenshot, reasoning, and action, then reinjects a filtered recent history at inference time.Text and screenshots form an interleaved multimodal memory context.
  • Executable actions are normalized before Playwright into shared mouse, keyboard, and timing operations.The runtime-facing schema includes click, click_hold, drag, mouse_move, scroll, type, press_key, press_keys, and wait.
  • The runtime translates normalized actions into Playwright primitives while supporting parser compatibility, legality checks, and one-action-per-step execution.
  • Legality validation is role-aware: generalist actions are resolved through semantic-control mappings before low-level checks, while computer-use actions are validated directly.
  • Generalist agents emit semantic controls whose identifiers and arguments are resolved through a registry-built, alias-aware map before execution.

E Prompt Templates and Game Prompt Blocks

GameWorld uses fixed prompt templates and per-game prompt blocks to align agent instructions with each game's rules, role, task, and executable controls. The library provides distinct generalist and computer-use formats while specifying game-specific objectives and legal actions.

  • E Prompt Templates and Game Prompt Blocks: The shared prompt template contains Game Rules, Role and Controls, Task Instruction, and Output Format blocks assembled at runtime.
  • E.1 Prompt Assembly: Shared Templates: Generalist role blocks render semantic action lists from registered controls, keeping prompts synchronized with the executable action space.
  • E.1.1 Generalist Agent Template: The generalist template instructs agents to act frame-by-frame through registered semantic controls rather than direct keyboard or mouse actions.
  • E.1.2 Computer-Use Agent Template: The computer-use template instructs agents to observe the screen, follow game rules, and execute actions frame-by-frame using computer-use controls.
  • E.2 Per-Game Prompt Library: Per-game prompt blocks include exact game rules, role descriptions, control prompts, semantic action lists, and five task prompts for each benchmark game.
  • E.2 Per-Game Prompt Library: The 2048 prompt defines tile-combination objectives and directional moves, while its role prompt requires exactly one action per step.
  • E.2 Per-Game Prompt Library: Other game blocks encode game-specific mechanics such as maze movement, precision jumping, brick breaking, paddle control, and platform-adventure rules.

F Costs and Licensing Considerations

GameWorld's benchmark costs are reported alongside licensing boundaries for the included browser games. Users must obtain lawful access and permissions, while the full evaluation cost is estimated from model traces and pricing assumptions.

  • Game access and distribution must comply with applicable licenses and permissions.
  • Users are responsible for purchasing or obtaining lawful access and necessary permissions for benchmarked games and downstream uses.
  • Table 13 reports average per-step input and output tokens and estimated costs for evaluating all 170 tasks per model.
  • 815.19 USD is the reported total cost for evaluating all 170 tasks across the listed models.The calculation excludes open-weight models and uses pricing estimates recorded on March 7, 2026.
Loading 2604.07429v1…