FlashAdventure: A Benchmark for GUI Agents Solving Full Story Arcs in Diverse Adventure Games

1 Introduction

• Adventure games combine non-standard layouts with inventory management, branching dialogue, context-sensitive actions, and narrative reasoning, while prior tasks often target specific missions or action sequences.The paper identifies diversity and full-story completion as unresolved evaluation gaps.Source: §1 · pp. 1–2
• FlashAdventure evaluates agents from start to finish across 34 adventure games, using compact Flash games to make self-contained story arcs practical to assess.The observation-behavior gap is the delay between observing information and being able to act on it.Source: §1 · p. 2
• The paper introduces CUA-as-a-Judge to verify predefined game milestones and COAST to maintain and apply long-term clue memory during problem solving.COAST proactively maintains clue memory and applies it during problem solving.Source: §1 · p. 2
• GUI agents struggle to complete full story arcs because of weak planning, limited perception in non-standard layouts, and a lack of lateral thinking.The reported results also identify a significant gap between agents and human performance.Source: §1 · p. 2

2 Related Work

• Standard GUI benchmarks often use standardized templates, restricting the interaction behaviors and visual structures agents encounter.The related-work discussion also describes structured-data, vision-only, and hybrid input approaches.Source: §2 · pp. 2–3
• API-based benchmarks provide precise, immediate feedback but require accessible game APIs and substantial setup, such as modding or reverse engineering.Their interaction paradigm relies on APIs rather than GUI manipulation.Source: §2 · p. 3
• Screenshot-based benchmarks use visual inputs and clicks or keystrokes, but often rely on inaccessible buy-to-play AAA games and manual evaluation.These constraints limit accessibility, reproducibility, and broader assessment.Source: §2 · p. 3
• Existing game benchmarks include at most 12 games and generally do not test complete story arcs; room-escape benchmarks have short human playtimes that limit evaluation of long-term observation-behavior gaps.The paper attributes this limitation to room escapes being designed for research rather than adapted from entertainment games.Source: §2 · p. 3

3. The FlashAdventure Benchmark

• The benchmark comprises 34 diverse Flash-based classic adventure games for evaluating agents’ ability to solve full story arcs.The games were selected from the FlashPoint Archive and span free-to-play subgenres, favor reasoning over reaction speed, and have validated walkthroughs and defined milestones.Source: §3 · p. 4, §3.2 · p. 4

3.1 Problem Formulation

• Gameplay is formulated as a POMDP in which agents act on rendered RGB frames, while a separate milestone reward function is not directly observable to them.The agent policy conditions on the task query and interaction history to produce low-level GUI actions; the episode reward indicates final task success.Source: §3.1 · p. 4

3.3 Human GamePlay

• Human participants completed full story arcs with an average of 1,142 steps, 26 minutes, and a 97.1% success rate.The average observation-behavior gap was 251.1 steps, with observed item-to-reward gaps of 99 and 421 steps.Source: §3.3 · p. 4

3.4 Evaluation

• The benchmark uses human-reviewed progression milestones, with an average of 6.7 discrete milestones across 27 non-simulation games and normalized HUD scores for seven simulations.Three authors independently reviewed walkthroughs and finalized milestones by consensus; success rate, milestone completion rate, and steps are the evaluation metrics.Source: §3.4 · p. 5

3.5 CUA-as-a-Judge

• 94.00% accuracy, 0.9912 Spearman correlation, and 0.9999 Pearson correlation show CUA-as-a-Judge agrees closely with human judgments across 300 episodes.The judge performed particularly well on step-by-step milestone verification but had limitations with counting-based milestones requiring multiple items to be assessed at once.Source: §3.5 · p. 5
• The judge resumes from the game’s final state, interacts with the environment to verify milestones sequentially, and scores completion as verified milestones divided by the total.Verification stops at the first unsatisfied milestone; in some cases, milestones can instead be checked in one pass by counting cumulative game states.Source: §3.5 · p. 5

4 Approach

• COAST uses long-term clue memory to guide agents through a Seek-Map-Solve cycle for complex tasks.The cycle repeatedly updates memory and filters resolved goals until the step limit is reached.Source: §4 · pp. 5–6
• The Clue Seeker explores for Nseek steps and stores potential clues, while the Clue Mapper links promising clues to observations and generates plausible subtasks.The mapper forms a candidate goal set containing clue, observation, and subtask combinations.Source: §4 · p. 6
• The Problem Solver executes proposed subtasks, and completed goals are stored in a resolved set to avoid duplication.COAST continues cycling through the stages until the step limit is reached.Source: §4 · p. 6

5.1 Baseline Agents

• The five baseline agents span end-to-end and modular designs, with Cradle and Agent S2 separating functions into modules.Cradle has six modules; Agent S2 further separates grounding and planning and refines action sketches as observations arrive.Source: §5.1 · p. 7

5.2 Experimental Settings

• The evaluation tested seven agents, with a maximum of 1,000 action steps per agent.The agents include end-to-end systems integrating perception, planning, and action, as well as modular systems that separate components.Source: §5.2 · p. 7, §5.1 · p. 7

5.3 Experimental Results

• Agents’ full-arc failures involved repetitive actions, weak planning, poor perception of non-standard layouts, and inflexible lateral thinking.Limited memory can prevent agents from turning relevant clues into subtask plans; some agents focus on a goal, such as a door, while missing nearby clues.Source: §5.3 · p. 7
• Across 34 games, agents had near-zero success and lower milestone completion than humans, while COAST led baselines by 5.88 percentage points in success rate and up to 2.78 percentage points in milestone completion.The comparison covers all 34 games; COAST does not consistently outperform Claude-3.7-Sonnet Computer Use in visual novels.Source: §5.3 · pp. 7–8, Table 2 · p. 8
• Figure 4 compares average milestone completion across subgenres; COAST does not consistently outperform Claude-3.7-Sonnet Computer Use in visual novels.The authors attribute the smaller advantage there to a smaller observation-behavior gap and less benefit from clue-based reasoning.Source: §5.3 · p. 8, Figure 4 · p. 8

5.4 Ablation Study

• The full Seeker, Mapper, and Solver configuration performed best overall; Seeker alone was poor, and Seeker plus Solver had suboptimal milestone completion in clue-rich games.The Mapper connects raw observations and clues into subtask plans.Source: §5.4 · p. 8, Table 3 · p. 8

5.5 Failure Analysis and Mitigation

• COAST mitigated planning and lateral-thinking failures in examples, but results were mixed for resource management and it did not mitigate visual-perception failures.It used prior failed attempts to locate hidden objects and bridged long-term gaps in Sherlock Holmes 2 and Camping Room Escape.Source: §5.5 · pp. 8–9

6.1 Analysis on Contaminations

• The contamination analysis sampled a representative game per subgenre and asked Claude-3.7-Sonnet and GPT-4o questions about in-game situations to test for pretrained game-specific knowledge.The questions targeted actions, interactions, and problem-solving steps and assessed whether models could answer without genuine reasoning.Source: §6.1 · p. 9
• Claude-3.7-Sonnet showed no signs of contamination, while GPT-4o exhibited contamination and hallucination in 3 of 10 questions.In one case, GPT-4o knew game-specific content but failed to perform the corresponding gameplay action; its gameplay performance lagged behind Claude-3.7-Sonnet.Source: §6.1 · p. 9

6.2 Additional Hint Injection

• Injected hints enabled completion of all milestones in Sherlock Holmes: The Tea Shop Murder Mystery in 758 turns, but did not help clear the first milestone in Computer Office Escape.Without hints, the agent achieved one milestone after 1,000 turns in the mystery game; the authors note hints do not overcome complex pattern reasoning in unstructured environments.Source: §6.2 · p. 9

7 Conclusion

• FlashAdventure comprises 34 diverse Flash-based adventure games for evaluating GUI agents’ full-story-arc completion.The benchmark targets completion from the start to finish.Source: §7 · p. 9
• CUA-as-a-Judge automatically verifies milestones and closely aligns with human judgments.The evaluator is an automatic milestone-verification agent.Source: §7 · p. 9
• Current state-of-the-art GUI agents struggle with the planning, perception, and lateral thinking required for full-game completion.COAST manages long-term clue memory and generates subtasks, significantly improving performance and bridging the observation-behavior gap.Source: §7 · p. 9
• A substantial gap remains between agent and human performance, warranting further improvement.The conclusion calls for further improvements.Source: §7 · p. 9

Limitations

• Manual milestone definition remains a limitation, although the lightweight process takes 10–15 minutes per game.Annotators review a walkthrough, select initial milestones, and discard those not indispensable to the story; narrative flow charts are suggested as a future way to derive milestones.Source: Limitations · p. 10
• CUA-as-a-Judge is less applicable to fast-paced games when character movement directly affects game state or narrative outcomes.The approach fits FlashAdventure games without strict time constraints or movement-dependent narrative progression.Source: Limitations · p. 10
• COAST has no memory-management mechanism, which may create scalability issues as action steps increase.The authors suggest summarization, retrieval, or forgetting mechanisms as possible remedies.Source: Limitations · p. 10
• Running all games for 1,000 steps with proprietary APIs incurs substantial cost and can hinder reproduction and large-scale experimentation.The paper distinguishes this cost from a limitation inherent to the benchmark, pointing instead to the need for more efficient open-source GUI agents.Source: Limitations · p. 10

Ethics Statement

• The games remain their creators’ intellectual property; use is non-commercial and academic, with rights holders contacted and games accessed through authorized platforms.The games are used for evaluation rather than training, are not redistributed or modified, and gameplay relies on screenshots without reverse-engineering or modding low-level code.Source: Ethics Statement · p. 10
• Human gameplay demonstrations involved 13 adults who gave informed consent before the study.Participants were compensated fairly, could withdraw without penalty, and gameplay data collection respected privacy and autonomy.Source: Ethics Statement · p. 10

A.1 Related Work

• FlashAdventure bridges text-adventure research and video games, retaining Jericho-like long-horizon and combinatorial complexity in multimodal interactive settings.The connection links symbolic reasoning and narrative progression with GUI-agent evaluation in video-game worlds.Source: §A.1 · p. 14

A.2 Problem Formulation

• The appendix points to the complete action set used by GUI agents and labels its unified action-space table.The passage identifies Table 5 as the complete action set.Source: §A.2 · p. 14, Table 5 · p. 14

A.3 Game Selection

• The game-selection appendix lists games by subgenre and provides annotated human walkthroughs for one selected game per subgenre.Examples span point-and-click mystery, hidden object, room escape, visual novel, and simulation games.Source: §A.3 · p. 14, Table 6 · p. 14

A.4 Analysis on Game Diversity

• The five subgenres have distinct demands: mystery/detective games use NPC interactions to gather clues, room escapes emphasize puzzles, hidden objects require visual search, and visual novels and simulations center on resource management.Visual novels optimize a single metric, whereas simulations balance multiple factors; hidden object games tend to have the shortest playtime, under five minutes.Source: §A.4 · pp. 14–16
• Even games with thematically similar narratives can have considerably different structures, and excelling in one game does not necessarily predict success in another.Across mystery/detective and room escape games, the three human players’ rankings and normalized scores varied substantially across games.Source: §A.4 · pp. 16–17
• Room Escape normalized-score standard deviations were A=314, B=361, and C=964, while Mystery/Detective Player A outperformed others most often but still showed large variation.Rankings and normalized scores also shifted substantially from game to game across both subgenres.Source: §A.4 · p. 17

A.7 CUA-as-a-Judge

• CUA-as-a-Judge checks milestones one by one in 11 games, stopping at the first unmet milestone, while 23 games allow all milestones to be assessed in one pass using cumulative state.Progress in step-by-step games is reported as Km/Nm; cumulative-state games are scored by total milestones completed.Source: §A.7 · pp. 20–21
• 94.00% accuracy, 0.9912 Spearman correlation, and 0.9999 Pearson correlation show high agreement between CUA-as-a-Judge and human judgments across 300 evaluation samples.The evaluation sampled 8 or 9 cases for each of the 34 games.Source: §A.7 · p. 21

B.1 Implementation Details

• Experiments were run once because of high API costs; COAST uses Claude-3.7-Sonnet components with genre-specific seeker and solver settings and K = 5 for the Clue Mapper.The seeker/solver settings are (5,2) for visual novels and simulation games, and (15,5) for other genres.Source: §B.1 · p. 22

B.1.1 Detailed Prompts

• COAST’s prompts have the Clue Seeker extract screen clues and episodic memories, then the Clue Mapper connect clues to past events and propose a concrete next action.The mapper is prompted to use abductive and lateral reasoning, while grounding matches in prior gameplay.Source: §B.1.1 · pp. 23–24
• The agent pairs discovered clues with related episodic memories, then selects a goal and action intended to advance gameplay.Goal selection should use clue metadata such as type, interactability, and usage hint.Source: §B.1.1 · pp. 24–25
• The prompt records each step’s action and observation, place, and episodic memory, with optional mapping details explaining the clue-related goal, reasoning, and result.The response must follow the specified JSON format or it will be discarded.Source: §B.1.1 · pp. 24–25
• A clue counts as a success only when it solves a specific puzzle or problem and produces a meaningful in-game change; otherwise, the result is a fail.Examples of meaningful changes include unlocking an area or triggering story progression.Source: §B.1.1 · p. 25

• The full results cover all 34 games using OpenAI CUA, Claude-3.7-Sonnet Computer-Use, COAST, and human baselines; Table 14 reports additional GUI agents.Source: §B.2 · p. 25
• Overall, despite having the smallest parameter size, UI-TARS-1.5-7B achieves a higher milestone completion rate than proprietary LLM-based models GPT-4o + UGround-V1-7B + Cradle and Claude-3.7-Sonnet + Claude-3.7-Sonnet + Agent S2.Source: §B.2 · p. 25
• Agent S2 has the lowest performance among the compared agents on FlashAdventure, despite performing well in desktop GUI tasks.The authors highlight a gap between everyday digital interactions and the complexities of adventure games.Source: §B.2 · p. 25

B.2.1 Multi-Run Experiments

• Across three trials, Claude-3.7 Computer-Use + COAST achieves higher SR and MCR than baseline agents, consistent with single-run results.Table 15 presents the multi-run results; Table 2 contains the single-run results.Source: §B.2.1 · p. 25, Table 13 · p. 26
• In Sort the Court, both agents complete the full story arc, while COAST reaches a near-perfect MCR of 95.5%.The authors suggest this may reflect stronger performance in NPC conversations and resource management.Source: §B.2.1 · p. 25
• MCR can exceed 100% in visual novel and simulation games because it is normalized against human performance; 101.2% in Sort the Court indicates performance on par with or above a human player.Source: §B.2.1 · p. 25

B.2.2 Fair Comparison between COAST and Baselines

• The control combined COAST’s module prompts for Claude-3.7 Computer-Use, but this prompt-optimized baseline was slightly worse than the naive baseline and missed milestones that COAST solved in mystery/detective and room escape games.These findings support the authors’ claim that COAST’s gains stem from its structured Seek-Map-Solve framework rather than better prompting.Source: §B.2.2 · p. 26

C.1 Analysis on Contaminations

• The contamination check used 1–3 questions per game across five games and assessed answer accuracy, conclusion accuracy, and behavior match; GPT-4o showed partial or imprecise contamination.In Question 2, GPT-4o recognized the boy and newspaper but hallucinated alternative ways to obtain the newspaper.Source: §C.1 · pp. 26–27

C.2 Additional Hint Injection

• Conditional walkthrough hints took Sherlock Holmes: The Tea Shop Murder Mystery from one milestone in 1,000 turns to all milestones within 758, but Computer Office Escape failed its first milestone despite hints every 50 turns.Hints were decomposed from complete walkthroughs into subtasks aligned with each game's milestone structure and injected only after a predefined number of steps without progress.Source: §C.2 · pp. 27–28

C.3 Large Reasoning Models

• o4-mini achieved 0 milestones in Sherlock Holmes: The Tea Shop Murder Mystery and 2 in Camping Room Escape, showing no improvement over GPT-4o despite more structured reasoning.The comparison replaced GPT-4o with o4-mini while retaining UGround-V1-7B and Cradle.Source: §C.3 · p. 31