GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents

• Video games test visual perception, strategic planning, precise timing, sustained action, and mistake recovery over long horizons, while browser games are diverse, lightweight, and easy to reset.Source: §1 · p. 3
• Existing game benchmarks improve realism and scale but remain hindered by heterogeneous action interfaces, real-time latency coupling, and heuristic or non-reproducible evaluation methods.Prior efforts include LMGame-Bench, BALROG, VideoGame-Bench, Orak, and OSWorld.Source: §1 · p. 3
• GameWorld evaluates 34 browser games across five genres and 170 tasks using a latency-decoupled sandbox and two interfaces: raw computer controls and deterministic semantic actions.The benchmark studies 18 model–interface pairs under a shared runtime.Source: §1 · p. 3
• Analyses show stable aggregate measurements, partial progress but unreliable completion, foundation-model-aligned strengths and weaknesses, real-time latency challenges, and distinct interface trade-offs.The suite includes repeated reruns, GameWorld-RT, curriculum profiling, context-memory sensitivity, and action-validity diagnostics.Source: §1 · pp. 3–4
• GameWorld computes deterministic task success and normalized progress from serialized gameAPI state, avoiding perceptual noise and enabling reproducible outcome-based evaluation.Source: §1 · pp. 3–4

2.1 Agent Interfaces

• GameWorld evaluates Computer-Use Agents and Generalist Multimodal Agents through a shared runtime, verifier, and unified atomic-event control space.Computer-use agents emit low-level controls, whereas generalist agents act semantically through deterministic parsing; the protocol also supports interface-aware analyses of robustness, real-time interaction, memory, and action validity.Source: §2 · p. 4, §2.1 · p. 5, Table 1 · p. 5

2.2 Computer-Use Agents: Low-Level Controls

• Computer-Use Agents emit exact keyboard and mouse commands, with one executable action per step and rejection of controls outside each game’s permitted interface.The interface requires precise coordinates and key sequences, making it especially sensitive to inference latency in real-time settings.Source: §2.2 · p. 5

2.3 Generalist Agents: Semantic Action Parsing

• Semantic Action Parsing deterministically maps each semantic action to one fixed low-level command, while Action Atomicity disallows multi-command macros.This places generalist agents under the same runtime and verifier as computer-use agents and removes parser-side stochasticity from interface comparisons.Source: §2.3 · p. 5

2.4 Agent Harnesses

• A shared agent harness standardizes structured prompts, memory, and model-specific tool interfaces because foundation models alone are insufficient for sustained gameplay.The harness provides common components across models while preserving implementation-specific details in the appendices.Source: §2.4 · p. 6

2.4.1 Structured Prompt

• A fixed four-part prompt template controls cross-model comparisons by keeping structure constant while varying game-specific rules, roles, objectives, and output formats.Its components are #Game Rules, #Role and Controls, #Task Instruction, and #Output Format.Source: §2.4.1 · p. 6

2.4.2 Context Memory

• Rolling memory prepends recent user_prompt → screenshot → reasoning → action sequences as Action History, helping agents avoid repeated failures and maintain step-to-step consistency.The memory provides short-horizon trajectory context, with its performance effects evaluated separately in Section 4.5.2.Source: §2.4.2 · p. 6

2.4.3 Reasoning

• Deliberate textual and visual reasoning can improve perception and long-horizon subgoal maintenance, but its additional time may harm game-agent performance through latency.Image zooming and cropping are examples of tool-assisted operations that can support close visual inspection.Source: §2.4.3 · p. 6

2.4.4 Customized Function Calling

• The harness registers semantic actions and computer-use primitives through each provider’s native function-calling interface while enforcing uniform legality and action-resolution rules.This preserves native model-specific agentic capabilities without changing the harness-level protocol.Source: §2.4.4 · p. 6

3.1 Benchmark Design

• GameWorld addresses benchmark diversity, runtime-latency confounds, and noisy evaluation through a five-genre curated benchmark, a latency-decoupling sandbox, and state-verifiable outcome metrics.Source: §3.1 · pp. 6–7

3.2 Games and Tasks

• The benchmark spans 34 browser-based games and 170 instructions across Runner, Arcade, Platformer, Puzzle, and Simulation genres, each testing distinct control, planning, or reasoning capabilities.Tasks provide natural-language goals, quantitative targets, fixed step budgets, and no intermediate guidance; Success Rate gives binary credit while Progress provides partial credit.Source: §3.2 · p. 7

3.3 Game Information

• The 34-game collection ranges from sparse board-state puzzles and continuous real-time control to open-ended simulations, with varied 2D and 3D presentations motivating a unified interface.Examples include 2048 and Minesweeper, Temple Run 2 and Pac-Man, and Monkey Mart and Minecraft Clone.Source: §3.3 · p. 7

3.4 Browser-Based Sandbox Environment

• The sandbox pauses games during inference to make scores reflect decisions rather than response speed, while also supporting real-time evaluation for studying latency effects.It runs isolated browser instances through a screenshot-query-action loop and supports readiness gating, configurable speed, and deterministic seeds for reproducibility.Source: §3.4 · p. 8

3.5 Outcome-Based State-Verifiable Evaluation

• State-verifiable evaluation reads serialized gameAPI fields such as scores, levels, coordinates, lives, coins, and checkpoints, producing deterministic signals without perceptual noise.GameWorld instruments 233 task-relevant state fields across 34 games and computes Success Rate and normalized Progress from configured task scores.Source: §3.5 · pp. 11–12

4.1 Experiment Setup

• The experiments evaluate 13 base models in 18 model–interface pairs under a shared paused protocol, with one command per step, fixed execution durations, a 100-action budget, and state-verifiable metrics.The study covers 8 Computer-Use Agents and 10 Generalist Agents across the benchmark.Source: §4.1 · pp. 12–13

4.2 Main Results

• Among Generalist agents, Gemini-3-Flash-Preview achieves the best overall PG at 41.9, while Computer-Use Agent Seed-1.8 leads at 39.8; current agents remain far below the Novice Player’s 64.1 PG.Overall SR remains 12.4–21.2%, indicating partial progress often fails to meet full task targets; the Novice Player achieves 55.3 SR / 64.1 PG under the same action budget.Source: §4.2 · p. 14

4.3 Benchmark Robustness Under Repeated Evaluation

• Repeated full-benchmark reruns show stable aggregate performance, with overall PG standard deviations remaining in the low single digits across four Qwen model–interface settings.Per-game variance is concentrated in difficult or control-sensitive games such as Hextris, Cubefield, Wordle, and World’s Hardest Game 2.Source: §4.3 · p. 15, Table 7 · p. 15

4.4 Capability-Aligned Curriculum Analysis

• Under the capability-aligned curriculum, Generalist and Computer-Use agents show similar performance, peaking at Levels 4 and 2 and dropping sharply at Levels 1 and 5, with long-horizon tasks and timing grounding remaining bottlenecks.Source: §4.4 · p. 17

4.5.1 GameWorld-RT: Real-Time Benchmark

• GameWorld-RT couples reasoning speed with action timing because the environment continues evolving during inference, yet success rates remain very low across all four tested settings.The 30B backbone is substantially faster, whereas the 235B backbone achieves slightly higher progress; real-time results should not be directly compared with paused evaluation.Source: §4.5.1 · pp. 17–18, Table 8 · p. 17

4.5.2 Context-Memory Sensitivity

• Increasing memory raises Generalist input tokens from about 1.3k to 3.1k and latency from 5.5 to 8.6 seconds per step, while CUA costs rise from 1.9k to 5.6k tokens and 7.2 to 12.8 seconds.Generalist PG rises modestly with more memory, but CUA performance steadily declines because low-level histories lack semantic information and accumulate distracting details.Source: Table 9 · p. 17, §4.5.2 · p. 18

4.5.3 Action Validity and Instruction Following

• Invalid Action Rate measures the fraction of actions failing parsing, role constraints, or legality checks; under long contexts, weaker models more often emit non-executable or impermissible tool calls.Invalid actions are divided into No-Tool-Call and Out-of-Space categories.Source: §4.5.3 · pp. 18–19, Table 10 · p. 18, Table 11 · p. 18

4.5.4 Failure Modes and Analysis

• The failure analysis identifies instruction-following, perception, fine-grained action, and long-horizon memory as distinct causes of unsuccessful gameplay.These include visual misreading, mistimed or imprecise execution, invalid or irrelevant actions, and repeated ineffective loops caused by lost historical context.Source: §4.5.4 · p. 19

5.1 Game Agent Interface Comparison: Generalist Agent vs. CUA

• Matched Mario trajectories isolate how CUA’s low-level controls differ from the Generalist agent’s richer semantic action interface under the same model and environment.Figure 6 attributes trajectory divergence to the control interface rather than the model backbone.Source: Figure 6 · p. 20, §5.1 · p. 20

5.2 Long-Horizon Simulation: Minecraft Resource Collection

• 90% progress was reached in Minecraft resource collection, but the agent still failed to finish the target within the step limit.The failure reflects missing closure rather than instruction-following, despite repeated mining toward the target.Source: Figure 7 · p. 21, §5.2 · p. 22

5.3 Real-Time Reaction and Timing Control: Flappy Bird

• In Flappy Bird, visually similar consecutive frames require alternating waiting and flapping, so slightly early or late actions can determine whether progress occurs.This case demonstrates the difficulty of real-time reaction and timing control under tight interaction constraints.Source: Figure 8 · p. 21, §5.3 · p. 22

6.1 Computer-Use Benchmarks with Online Environments

• GameWorld transfers standardized action spaces, interactive environments, parallel instances, and outcome-based verification from computer-use benchmarks to game-agent evaluation.WebArena [86] and OSWorld emphasize outcome-based evaluation, while GameWorld applies these principles to interactive game environments.Source: §6.1 · p. 22

6.2 Video Game Benchmarks for LLM and MLLM Agents

• Game benchmarks increasingly target reliable evaluation through long-horizon reasoning, real-time latency, story completion, transfer, collaboration, and automated verification, while heuristic progress assessment remains challenging.VideoGameBench isolates inference latency with a paused track, FlashAdventure [3] introduces CUA-as-a-Judge, and GameVerse combines semantic and GUI control but still relies on heuristic VLM-based progress quantification.Source: §6.2 · p. 22

6.3 Game Agents and Scalable Infrastructure

• GameWorld provides a comprehensive benchmark with diverse games and tasks under a unified, verifiable protocol as game agents mature across keyboard-mouse, vision-language, universal-API, and simulated-world interfaces.Prior systems include Game-TARS, Jarvis-VLA [41], NitroGen, Lumine [64], SIMA, and SIMA 2, motivating standardized cross-interface measurement.Source: §6.3 · p. 23

7 Conclusion

• Under a shared runtime and verifier, interface-conditioned weaknesses emerge in real-time interaction, context-memory sensitivity, and action validity.Source: §7 · p. 23
• Stronger game agents will require better reasoning together with more reliable action grounding, more useful trajectory memory, and greater robustness to latency.Source: §7 · p. 23
• Designing unique instruction sets for each new environment tightly couples the action space to tasks and constrains benchmark scalability.Source: §7 · p. 23

Appendix

• Automating Semantic Action Parsing production and alignment through MLLM-powered agent exploration remains future work, while licensing, compliance, and cost details appear in Appendix F.Source: §7 · p. 23

7 Conclusion

• Across 34 games, 170 tasks, and 18 model–interface pairs, current agents remain far from reliable task completion and human-level performance despite meaningful partial progress.Source: §7 · p. 23
• GameWorld establishes standardized, outcome-based, state-verifiable evaluation as a reproducible foundation for studying multimodal agents in complex browser environments.Source: §7 · p. 23

A.1 Preset Configuration

• Standalone runs use a registry preset supplied through the –config command-line flag.The preset syntax is <game_id>+<task_id>+<model_spec>.Source: §A.1 · p. 24
• The three preset components are resolved independently and composed into one concrete runtime configuration.This separates game definitions, task settings, and model profiles before execution.Source: §A.1 · p. 24
• <game_id> supplies game rules, role definitions, low-level control constraints, and semantic action definitions.Source: §A.1 · p. 24
• <task_id> supplies the task instruction, evaluator configuration, target metrics, maximum step budget, and optional URL suffixes.Source: §A.1 · p. 24
• <model_spec> supplies model identifiers, provider-specific overrides, prompt templates, and output-format prompts.Because these components are reusable, changing tasks or swapping models does not require duplicating the underlying game configuration.Source: §A.1 · p. 24

A.2 Suite Runner

• The suite runner expands YAML-defined games, tasks, and models into explicit runs, groups them into repeat waves, and launches up to –max-parallel child processes per wave.Each child receives a dedicated port, unique session ID, isolated run directory, and standalone-run syntax; per-model summaries support monitoring and aggregate analysis.Source: §A.2 · p. 24

B.1 Runtime Coordinator

• The Runtime Coordinator wraps each MLLM in an Agent and instantiates the browser GameEnv and Evaluator for task tracking.Each Agent includes an ID, model type, client, role-specific controls, semantic-control map, and step counter.Source: §B.1 · p. 24
• Each interaction round captures the game screen, optionally pauses during inference, obtains a prompted model response, and resumes the game if needed.The sequence uses the current screenshot and assembled prompt template as model input.Source: §B.1 · p. 24
• The runtime parses the raw response into an executable action payload, executes it in the browser, captures a verifiable state snapshot, and evaluates progress.It then checks whether stopping or resetting conditions apply.Source: §B.1 · p. 24

B.2 Evaluator and Reset-on-Fail

• The Evaluator combines the current state snapshot, global step index, maximum step budget, target threshold, and accumulated metrics to determine run status.Its evaluation uses four stop or reset signals: terminal status, fixed-budget exhaustion, target-score attainment, and task-specific end-field rules.Source: §B.2 · p. 25
• The evaluator can terminate a run when the game is terminal, the fixed step budget is exhausted, the target score is reached, or an end-field rule fires.These conditions provide the evaluator’s explicit stopping criteria.Source: §B.2 · p. 25
• When continue_on_fail is enabled, terminal failure triggers gameAPI.reset(), a new episode ID, reinitialization, and continuation under the same global step budget.The runtime therefore does not end the run immediately after a resettable failure.Source: §B.2 · p. 25

C.1 Browser Management

• Chromium runs with a fixed viewport and background-throttling behaviors disabled.The browser manager launches Chromium after the local game server is started.Source: §C.1 · p. 25
• JavaScript headers inject dynamic speed control and deterministic randomness into the browser environment.Source: §C.1 · p. 25
• Screenshots use the Chrome DevTools Protocol to avoid visible flashing in headed mode.Source: §C.1 · p. 25
• The GameLauncher starts a local HTTP server that serves the game HTML.Source: §C.1 · p. 25
• Initialization starts the environment and opens the game in Chromium.Source: §C.1 · p. 25
• The system waits until the game becomes actionable under the readiness gate described in Section C.2.Source: §C.1 · p. 25

Readiness condition

• The runtime waits for an actionable game state before the first agent action and after every reset.Source: §C.2 · p. 25

Default statuses

• The default actionable statuses are `ready` and `playing`.Source: §C.2 · p. 25

Status values

• Table 12 documents common status values consumed by the readiness gate and evaluator.Source: Table 12 · p. 25, §C.2 · p. 25

C.3 Verifiable State: Game API Schema

• Each game exposes window.gameAPI with init(config), reset(options), and getState(), returning lifecycle metadata, structured state, task metrics, and raw details for outcome verification.The shared top-level contract remains constant across games, while game-specific fields may vary.Source: §C.3 · p. 25
• The API separates structured game_state, compact comparable metrics, and optional raw game-specific details, enabling standardized evaluation alongside further analysis.The Mario example includes progress, player state, level information, counters, lives, and timing fields.Source: §C.3 · p. 26

D.1 Rolling Memory

• The base client stores each interaction as user_prompt, screenshot, reasoning, and action, then reinjects filtered recent rounds as interleaved text-and-image history.Memory behavior is controlled by memory_rounds, memory_format, and memory_include_fields.Source: §D.1 · p. 26

D.2 Low-Level Action and Validation

• Runtime actions are normalized into mouse, keyboard, and wait operations before Playwright execution, with one normalized action handled per step.The normalized layer supports parser compatibility and legality checks before translation into Playwright primitives.Source: §D.2 · p. 27

D Agent Details

• Together, the agent details support a shared execution pipeline in which semantic generalist actions and direct computer-use actions converge on normalized low-level controls.This convergence preserves distinct interfaces while applying common execution and validation machinery.Source: §D.2 · p. 27, §D.3 · p. 27

D.2 Low-Level Action and Validation

• Legality is role-aware: generalist semantic controls are checked against registered mappings before low-level validation, while computer-use actions are validated directly against role-defined controls.Invalid or disallowed actions are logged and ignored, with key aliases normalized without expanding the legal action space.Source: §D.2 · p. 27

D.3 Semantic Action Parsing

• Generalist agents emit semantic control payloads rather than raw inputs; the runtime resolves aliases and identifiers, merges arguments, expands cell bindings, and routes valid results through shared execution.Unknown control identifiers become invalid no-ops at execution time.Source: §D.3 · p. 27

E Prompt Templates and Game Prompt Blocks

• The runtime assembles each final model prompt from game rules, role and controls, task instructions, and model-specific output-format blocks.These four components form the shared prompt structure used at runtime.Source: E Prompt Templates and Game Prompt… · p. 27

E.1 Prompt Assembly: Shared Templates

• Generalist and computer-use agents use templates with a fixed section order shared across both interfaces.The shared order organizes the game rules, role and controls, task instruction, and output format sections.Source: §E.1 · p. 27
• The role block specifies textual keyboard-and-mouse controls for computer-use agents but renders executable semantic_controls entries for generalist agents.The semantic action list is generated from the registry, keeping it synchronized with the executable action space.Source: §E.1 · p. 28

E.1.1 Generalist Agent Template

• The generalist template instructs frame-by-frame play through the registered semantic control list rather than direct keyboard or mouse actions.Its conditional sections include game rules, semantic role and controls, task instructions, and model output format.Source: §E.1.1 · p. 28

E.1.2 Computer-Use Agent Template

• The computer-use template likewise instructs frame-by-frame play, but provides a computer-use role-and-controls block alongside the game rules, task instructions, and output format.Unlike the generalist template, it does not restrict actions to the registered semantic control list.Source: §E.1.2 · p. 28

E.2 Per-Game Prompt Library

• The per-game prompt library records exact game rules, registry-loaded role prompts, interface-specific controls, semantic action lists, and five task prompts for each benchmark game.The 2048 entry, for example, specifies tile-sliding rules and distinct keyboard controls and semantic actions such as move_up and move_right.Source: §E.2 · pp. 28–29
• The library covers diverse interaction patterns, including platforming, maze navigation, precision jumping, brick breaking, jetpack movement, runner survival, and timing-based ball launches.Examples include Another Gentleman’s Adventure, Astray, Boxel Rebound, Breakout, Captain Callisto, Chrome Dino, and Core Ball, each with corresponding game objectives and legal controls.Source: §E.2 · pp. 29–31
• Each prompt block defines a game objective, rules, agent role, legal computer-use actions, and semantic controls requiring exactly one action per step.The library explicitly labels legal action spaces and registered semantic controls for each game.Source: §E.2 · p. 32
• Cubefield maps left, right, and wait controls to avoiding cubes while the ship moves automatically and collisions end the game.Its semantic actions are wait, move_left, and move_right.Source: §E.2 · p. 32
• Doodle Jump, Edge Surf, Flappy Bird, GeoDash, and Google Snake encode survival or score objectives with movement, timing, turning, or boost actions.Their prompts include platform hazards, lives and items, gravity and pipes, auto-running jumps, or continuous directional movement toward apples.Source: §E.2 · pp. 32–34
• Across the library, semantic actions normalize keyboard combinations and game-specific abilities into concise commands such as flap, jump, move, turn, rotate, boost, and accelerate.Examples include jump_left and jump_right for both Fireboy and Watergirl, boost requiring a boost item, and directional Snake controls.Source: §E.2 · pp. 33–35
• Fireboy and Watergirl use separate role prompts and controls, requiring each character to reach its exit while avoiding character-specific lethal liquids.Watergirl avoids red lava, Fireboy avoids blue water, and both can move, jump, or jump diagonally through semantic actions.Source: §E.2 · pp. 33–34
• Hextris specifies matching three or more same-colored blocks, boundary prevention, rotation, and accelerated falling through left, right, down, and semantic equivalents.The semantic controls are wait, rotate_left, rotate_right, and accelerate.Source: §E.2 · p. 36
• Mario’s prompt defines reaching the flagpole, collecting coins, defeating enemies, limited lives and time, hazards, and power-ups within a platforming task.The rules specify three lives, enemy-contact failures, jump-based attacks, question-block rewards, and mushroom growth.Source: §E.2 · p. 36
• Per-game prompts pair role instructions with explicit legal computer-use controls and exactly one semantic action per step.The library repeatedly separates computer-use prompts from registered semantic controls, whose actions are constrained to game-specific lists.Source: §E.2 · p. 36
• Minecraft Clone prompts encode resource gathering, movement, camera control, mining, block placement, and hotbar selection across keyboard, mouse, and semantic actions.Semantic actions include movement, camera turns, mining the targeted block, placing blocks, and selecting hotbar slots.Source: §E.2 · pp. 36–37
• Minesweeper prompts specify revealing safe cells, flagging mines, using adjacent mine counts, and ending the game when a mine is clicked.The semantic interface addresses cells by identifiers such as cell="a1".."i9".Source: §E.2 · p. 37
• Monkey Mart prompts coordinate harvesting, stocking, serving customers, collecting money, assistants, and upgrades through movement and waiting actions.The rules require interleaving harvesting, stocking, and serving, while the assistant only helps with corn collection and restocking.Source: §E.2 · p. 38
• Platform-game prompts tailor actions to each game’s hazards and movement physics, including landing on platforms in NS-Shaft, wall-assisted jumps and ground smashes in OVO, and automatic flaps with gliding in Restless Wing Syndrome.Their semantic action lists expose corresponding controls such as left/right movement, jumping, smashing, sliding, and directional gliding.Source: §E.2 · pp. 38–40
• Pac-Man prompts define pellet collection and ghost avoidance while providing directional movement through the maze in both control interfaces.The rules also advise using corridors to bait ghosts and saving power pellets for difficult sections.Source: §E.2 · p. 39
• Rocket League 2D prompts map scoring and defense to driving, jumping, and boosting controls for a side-view car-soccer task.The computer-use prompt uses A/D, W, and Space, while the semantic action list begins the corresponding registered interface.Source: §E.2 · p. 40
• Each game prompt block defines the game objective and rules alongside a player-role instruction tailored to the game’s mechanics.Examples cover survival, timing, platforming, combat, word guessing, and maze navigation objectives.Source: §E.2 · pp. 41–44
• Computer-use prompts explicitly restrict controls to legal actions, mapping keyboard inputs to movement, jumping, waiting, dropping blocks, rotating, soft or hard dropping, swapping pieces, and starting or retrying.Source: §E.2 · pp. 41–42
• Generalist prompts register semantic actions and require exactly one action per step, abstracting game-specific controls into names such as jump, drop_block, rotate_cw, and shoot.The semantic vocabularies are adapted to each game, including turn and slide actions for Temple Run 2 and movement variants for maze games.Source: §E.2 · pp. 41–42
• The library covers distinct interaction patterns in endless-runner, falling-block puzzle, and five-letter text-entry games, including movement, jumping/sliding, rotation/dropping/holding, and typed guesses with feedback.Source: §E.2 · pp. 41–44
• Wolfenstein 3D prompts combine navigation, turning, shooting, health and ammunition management, and door opening, with corresponding keyboard and semantic controls.The semantic action list includes move_forward, move_backward, turn_left, turn_right, shoot, and open_door.Source: §E.2 · p. 43
• World’s Hardest Game prompts encode collecting items, reaching a green exit, and avoiding enemies moving on fixed paths through directional controls and semantic movement actions.The two listed variants use the same four directional computer controls and semantic actions, with the second variant referring to collecting items.Source: §E.2 · pp. 44–45

E.3 Model Output-Format Blocks

• The next subsection states that it lists the exact output_format block from each registered model specification.This marks the transition from per-game prompt definitions to model-specific output-format specifications.Source: §E.3 · p. 45
• Claude computer-use prompts require exactly one action or key combination per step, prohibit self-directed screenshots, and disallow free-form text outside tool calls.Source: §E.3 · pp. 45–46
• Claude generalist prompts require exactly one tool call per step, a registered action identifier, and a short rationale labeled reasoning, without free-form text.Source: §E.3 · p. 45
• Qwen3-VL-235B-A22B and Qwen3-VL-30B-A3B computer-use prompts require a brief <think> block followed by one JSON-only <tool_call> block using the computer_use tool.Source: §E.3 · pp. 45–46
• The corresponding Qwen3-VL-235B-A22B and Qwen3-VL-30B-A3B generalist prompts use the same brief <think> and single <tool_call> structure and forbid other text.Source: §E.3 · p. 46
• Qwen3-VL-Plus computer-use prompts require one computer_use action or key combination per step and restrict output to tool-call blocks.Source: §E.3 · p. 46
• Qwen3-VL-Plus generalist prompts require one registered action tool per step, a short reasoning rationale, and no free-form text.Source: §E.3 · p. 46
• The explicit action syntax supports single-key and multi-key hotkeys, left clicks, right clicks, and wait/observe operations, with examples such as hotkey(key=’w d’).Source: §E.3 · pp. 46–47

Licensing Considerations

• Users must obtain lawful access and any necessary permissions, and the project grants no rights beyond applicable licenses, terms of service, and law.This applies to evaluation, dataset creation, model development, downstream use, reproduction, distribution, modification, and commercial use.Source: F Costs and Licensing Considerations · p. 47

Cost Summary

• Table 13 estimates benchmark costs from average per-step input and output tokens, including cached input tokens when supported, and measured average task costs from trace logs.Underlying model pricing estimates use a snapshot recorded on March 7, 2026.Source: Table 13 · p. 47, F Costs and Licensing Considerations · pp. 47–48
• 815.19 USD is the final estimated cost to evaluate all 170 tasks across the listed models, excluding open-weight models.Excluded models include Qwen3-VL-235B-A22, Qwen3-VL-30B-A3B, and UI-TARS-1.5-7B.Source: Table 13 · p. 47, F Costs and Licensing Considerations · p. 48