Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games

• Existing benchmarks often use text-only or 2D-grid games, insufficiently assess agentic modules, and lack fine-tuning datasets for gameplay agents.Source: §1 · p. 1
• Orak evaluates LLM agents on 12 popular games spanning six major genres and diverse cognitive demands.Source: §1 · pp. 1–2
• MCP provides a plug-and-play interface for sequentially retrieving game states, invoking agentic modules, and executing game steps across environments.Source: §1 · p. 2
• The released expert gameplay trajectories encode when and how to use agentic strategies across genres, supporting more resource-efficient and effective agents.Source: §1 · p. 2
• Experiments with 15 LLMs find proprietary models generally outperform open-source models, while fine-tuning transfers gameplay meta-knowledge across games and to unseen math and web tasks.Source: §1 · p. 2

• Earlier game-playing studies progressed from text-based environments to 2D-grid games and then to complex titles such as Minecraft, Civilization, Pokémon, and StarCraft.Source: §2 · p. 2
• Game-playing approaches often rely on manually customized workflows for each game, limiting their usability for general gaming agents.Source: §2 · p. 3
• Existing benchmarks cover selected genres and settings but generally lack full genre coverage, in-depth agentic-module studies, and alignment methods for versatile game agents.Source: §2 · p. 3, Table 1 · p. 3
• Agent fine-tuning methods use expert demonstrations or learned agentic interactions, while game agents face large, dynamic, partially observable state spaces and diverse behavior patterns.Source: §2 · p. 3

3 ORAK

• Orak’s evaluation loop configures a game, backbone LLM, and agentic strategy, retrieves observations, applies actions, and records scores until termination or a step limit.Source: §3 · p. 3, Figure 2 · p. 4
• MCP packages game environments and agentic modules as callable components, enabling users to replace backbone models or customize single or sequential agentic strategies.Source: §3 · p. 3, Figure 2 · p. 4, §3.2 · p. 6

3.1 GAME ENVIRONMENTS

• The 12 environments span diverse genres and require capabilities including rule following, logical and spatial reasoning, long-text understanding, planning, error handling, and odds handling.Source: Figure 3 · p. 4, §3.1 · p. 4
• Capability levels are rated from 1 to 3 by eight human participants, with the reported value taken as the moderate rating.Source: §3.1 · p. 4
• Each game specifies a structured state, action space, and evaluation task or metric, including combat stages, storyline flags, profit, win rate, floors reached, and item collection.Source: §3.1 · p. 5

3.2 SUBMISSION GUIDELINE TO BENCHMARK LEADERBOARD

• Participants may submit pretrained or fine-tuned LLMs and VLMs, or new agentic modules and strategies that follow a consistent structure across games.Source: §3.2 · p. 6

• Expert LLM trajectories from all 12 Orak games form the fine-tuning dataset for adapting pretrained models into gaming agents.The trajectories capture how agentic strategies are used across diverse game genres.Source: §4 · p. 6
• Each trajectory records sequential game-step inferences, including an agentic module, its prompt, the game state, and the model response.Source: §4 · p. 6
• Approximately 11k selected samples use high-scoring reflection-planning-action trajectories, with about 300 samples per agent module across the 12 games.The data are selected after collecting more than 1,000 inference sequences per game and are augmented through paraphrasing.Source: §4 · p. 6
• The dataset is mainly designed for supervised fine-tuning, while dynamic environment extraction for reinforcement-learning fine-tuning remains future work.Source: §4 · p. 6

5.1 EXPERIMENT SETUP

• Orak evaluates 15 LLMs with normalized game scores averaged over 3–20 trials, using game-specific default agentic strategies selected from zero-shot, reflection, reflection-planning, and skill-management agents.Source: §5.1 · p. 7

5.2 LLM GAMEPLAY PERFORMANCE

• Gemini-2.5-pro ranks first in 5 of 12 games with a best average ranking of 3.5, while proprietary LLMs generally outperform open-source models across Orak.Small open-source models often score nearly zero on complex games, whereas larger open-source models improve on moderately difficult games but remain below proprietary models.Source: §5.2 · p. 7

5.3 LLM ARENA

• In competitive arenas, Minitron-8B leads Street Fighter III Elo ratings, while Claude-3.7-Sonnet leads StarCraft II and GPT-4o and Minitron-8B form the second group.Street Fighter III uses three-round pairwise matches with the same character; StarCraft II uses single-round pairwise matches with the same race.Source: §5.3 · pp. 7–8, Figure 4 · p. 8

5.4 ABLATION STUDY FOR AGENTIC MODULES

• GPT-4o improves consistently with added agentic modules, reaching average ranking 2.2 with reflection-planning, whereas Llama-3.2-3B ranks best with reflection at 5.6 rather than reflection-planning at 6.1.The results indicate that added prompt complexity can hinder smaller models’ decision-making accuracy.Source: §5.4 · p. 8, Table 4 · p. 8

5.5 EFFECT OF VISUAL INPUT

• Image-only input substantially worsens average ranks, while combining text and images has mixed effects, including Claude gaining 16.6 points in Street Fighter III and GPT-4o losing 31.8 in Ace Attorney.In Ace Attorney, GPT-4o’s average rank falls from 2.9 with text-only input to 5.0 with both modalities.Source: §5.5 · pp. 8–9, Table 5 · p. 9, Table 6 · p. 9

5.6 EFFECT OF FINE-TUNING

• Fine-tuning improves unseen-scenario performance in 3 of 5 intra-game evaluations and significantly improves OOD games such as Super Mario and 2048, but smaller models show limited spatial or strategic generalization.Fine-tuning can improve valid-action generation, yet it does not overcome the spatial demands of Baba Is You or strategic demands of new StarCraft II maps.Source: §5.6 · p. 9, Table 7 · p. 9
• Fine-tuning sometimes transfers to Math500 and WebShop, with LLaMA-3.2-3B improving WebShop-E from 0.0% to 8.4% and WebShop-H from 0.0% to 12.6%, unlike Llama-3.2-1B.Source: §5.6 · pp. 9–10
• High-score data outperform low-score or mixed data, while real-time StarCraft II exposes severe latency sensitivity: all models fail hard mode, and only GPT-4o remains perfect on easy mode.Reflection-planning response latencies are 19.6 s for GPT-4o-mini, 27.2 s for GPT-4o, and 99.2 s for Gemini-2.5-pro.Source: §5.6 · p. 10, §5.7 · p. 10, Table 8 · p. 10

6 DISCUSSION

• Orak benchmarks and trains LLM agents across diverse video games, enabling comprehensive assessment of capabilities needed across most game genres.The authors position its comprehensive game set as a foundation for progress toward versatile, high-performing gaming agents.Source: §6 · p. 10
• Its MCP-powered plug-and-play interface supports consistent evaluation of rapidly evolving LLMs across varied agentic modules.The interface is described as user-friendly and suitable for systematic evaluation across the benchmark’s game set.Source: §6 · p. 10
• The released fine-tuning dataset contains trajectories from top-performing LLMs and can transform pre-trained LLMs into gaming agents.The dataset covers game interactions and is intended to support adaptation of general-purpose models for gameplay.Source: §6 · p. 10
• Orak preprocesses game states into structured text that hides gameplay-irrelevant information, but full uncurated in-game states remain future work.This design favors LLM reasoning and decision-making and may inform deployment of resource-efficient LLMs; cost and licensing are discussed in Appendix U.Source: §6 · p. 10

A GAME GENRE CATEGORIZATION

• Orak organizes 12 games across six major genres, allowing structured analysis of agent cognition and gameplay dynamics across diverse games.Some games receive secondary genre labels because of hybrid mechanics.Source: A GAME GENRE CATEGORIZATION · pp. 17–18, Table 9 · p. 17

B REQUIRED LLM CAPABILITIES FOR GAMEPLAY

• Strategic games require five distinct LLM capabilities at level 3, while different genres emphasize spatial reasoning, long-term planning, logical reasoning, rule following, or long-text understanding.Figure 3 describes genre-specific capability profiles, including lower spatial-reasoning demands for Stardew Valley’s money-earning evaluation task.Source: B REQUIRED LLM CAPABILITIES FOR GAMEPLAY · p. 18

C.1 GAME DESCRIPTION FOR STREET FIGHTER III

• Street Fighter III uses Diambra Arena to expose screenshots and game state while accepting discrete controller inputs, with YOLOv11 adding relative character-position categories.The environment provides health, timer, super-bar, stun, and frame information; detected positions are mapped to very close, close, or far.Source: §C.1 · p. 19
• A higher-level action space abstracts 18 frame-level actions into semantically meaningful multi-frame commands, typically yielding about 20 character-dependent actions.It combines character-agnostic actions with character-specific moves such as Ken’s Hadouken and Chun-Li’s Kikkoken.Source: §C.1 · pp. 19–20

C.2 GAMEPLAY PROMPT FOR STREET FIGHTER III

• The reflection-planning agent iteratively plans a subtask, infers an action, and evaluates success through reflection using current state, prior actions, and state transitions.The implementation also supports reflection, planning, and zero-shot agents.Source: §C.2 · pp. 20–21

C.3 EVALUATION METRIC FOR STREET FIGHTER III

• Single-agent evaluation measures progression through 10 stages, whereas multi-agent evaluation uses three-game pairwise matches to calculate win rates and Elo ratings.The resulting win-rate matrix and Elo scores are presented for competitive comparison.Source: §C.3 · pp. 21–22
• Elo ratings are estimated by gradient ascent on the Bradley–Terry log-likelihood, shifted to a mean of 1,500, with head-to-head wins breaking rating ties.Expected win probabilities increase with the rating difference between agents.Source: §C.3 · p. 22

C.4 EXPERIMENTAL CONFIGURATION FOR STREET FIGHTER III

• Experiments evaluate six open-source LLMs using temperature 0.0 and repetition penalty 1.0, with gameplay scores, agentic-module ablations, and modality comparisons reported in Tables 10–12.The listed models are Llama-3.2-1B/3B, Qwen-2.5-3B/7B, and Minitron-4B/8B.Source: §C.4 · p. 22, Table 11 · p. 23, Table 10 · p. 23
• The Street Fighter III environment permits up to 10,000 game steps, although best-of-three stages typically finish within 100–200 steps because rounds time out or end when a player’s health reaches zero.Episodes generally do not reach the configured maximum because of in-game time constraints.Source: §C.4 · p. 23

C.5 RESULT FOR STREET FIGHTER III

• Commercial LLMs generally outperform open-source models in Street Fighter III, while Minitron-8B leads the multi-agent arena despite weaker single-agent results.Single-agent evaluation found GPT-4o and o3-mini among the strongest models, whereas Minitron-8B achieved the highest Elo in pairwise matches.Source: §C.5 · pp. 23–24
• Image inputs hurt GPT-4o in Street Fighter III but improve Gemini and Claude, showing that visual information is not uniformly useful across models.The study expected images to help with spatial information, but models differed in their ability to exploit the visual data.Source: §C.5 · p. 23
• Targeted fine-tuning on Ken gameplay enables strong Chun-Li performance, allowing Llama-3.2-1B to follow the required format and Llama-3.2-3B to surpass GPT-4o.Chun-Li evaluation represents an out-of-distribution setting because the action spaces differ from Ken’s.Source: §C.5 · p. 23, Table 13 · p. 24

D.1 GAME DESCRIPTION FOR SUPER MARIO

• Super Mario converts RGB frames into object coordinates, then restricts control to seven jump levels while the agent selects jumps to reach the flag safely.Template matching identifies objects and their locations, which are passed as text; jump levels 0–6 discretize Mario’s jumping behavior.Source: §D.1 · p. 24, §D.2 · p. 25, §D.3 · p. 25

D.5 RESULT FOR SUPER MARIO

• 38.0 is Gemini-2.5-pro’s top Super Mario score, ahead of o3-mini at 34.9, while models commonly misestimate mid-air parabolic trajectories.Llama-3.2-3B and Minitron-8B score 31.8 and 31.3, respectively, and are comparable to Claude-3.7-sonnet at 31.7.Source: §D.5 · p. 26, Table 14 · p. 26
• Reflection improves obstacle handling in Super Mario, with GPT-4o reaching 34.1 using reflection plus planning and Llama-3.2-3B reaching 32.4 with reflection alone.Reflection lets agents revise low jump decisions when blocked by obstacles such as warp pipes.Source: §D.5 · p. 26, Table 15 · p. 26
• Image-only observations underperform text-only inputs across Super Mario models, whereas combined inputs help Gemini-2.5-pro but reduce GPT-4o and Claude-3.7-sonnet performance.The results suggest multimodal input helps when a model can integrate complementary visual and textual information effectively.Source: §D.5 · p. 26, Table 16 · p. 26

E.1 GAME DESCRIPTION FOR ACE ATTORNEY

• Ace Attorney combines dialogue progression, multiple-choice answers, and evidence-based cross-examination, with subtasks scored using correctness rewards and step counts normalized against fixed benchmarks.Source: §E.1 · pp. 26–27, §E.3 · p. 28

E.5 RESULT FOR ACE ATTORNEY

• o3-mini leads Ace Attorney at 91.7, ahead of GPT-4o at 85.3, while open-source models score substantially lower and smaller variants often fail to advance.Qwen-2.5-3B completes the first cross-examination in one of three trials, whereas low-scoring models commonly misinterpret details, repeat errors, select irrelevant options, or violate the required format.Source: §E.5 · p. 28
• Reflection and planning have game- and model-dependent effects: reflection helps GPT-4o in Ace Attorney, combined reflection-planning helps Llama-3.2-3B in Her Story, and reflection-planning gives GPT-4o the best Pokémon Red score at 38.9.In contrast, GPT-4o performs best zero-shot in Her Story and Darkest Dungeon, where added modules provide no improvement or can reduce performance; smaller models gain more modestly.Source: §E.5 · pp. 28–29, §F.5 · p. 31, §G.5 · p. 35

F.5 RESULT FOR HER STORY

• Text-only input suffices for Ace Attorney and generally outperforms multimodal input in Her Story, whereas visual information improves or preserves performance for Pokémon Red and offers minimal gains in Darkest Dungeon.For Pokémon Red, GPT-4o rises from 38.9 text-only to 41.7 with both modalities; in Her Story, static visuals add little because useful interface information is already textual and video-only cues are unavailable.Source: §E.5 · p. 29, §F.5 · p. 31, §G.5 · p. 35

F.1 GAME DESCRIPTION FOR HER STORY

• Her Story abstracts point-and-click interaction into Search [keyword] and Play Video [i], converts logged database states into textual observations, and scores agents by distinct clips viewed from an archive of 271.Experiments use three trials, a 400-step interaction limit, temperature 0.3, and repetition penalty 1.Source: §F.1 · p. 29, §F.3 · p. 30, §F.4 · p. 30

F.5 RESULT FOR HER STORY

• Gemini-2.5-pro achieves the highest Her Story score at 67.3, followed by Deepseek-R1 at 66.9 and o3-mini at 66.3, while every evaluated open-source model scores below 10.GPT-4o-mini scores 21.1 because it repeatedly searches already-used keywords, and other open-source models fail to play unseen clips, repeat keywords, or issue sentence-like queries.Source: §F.5 · p. 31, Table 20 · p. 31

G.1 GAME DESCRIPTION FOR POK´EMON RED

• Pokémon Red exposes emulator-extracted map, position, party, opponent, and inventory state as text, supports button-level and higher-level movement, dialogue, battle, and item tools, and scores the percentage of 12 storyline flags achieved.Darkest Dungeon similarly supplies structured combat state and evaluates a composite score from cleared combats, hero survival, and remaining stress; Minecraft uses Mineflayer textual observations and JavaScript actions to craft eight target items in a fixed world.Source: §G.1 · p. 32, §G.3 · p. 32, §H.1 · p. 36

G.5 RESULT FOR POK´EMON RED

• Gemini-2.5-pro leads Pokémon Red at 83.3, while all evaluated open-source models and o3-mini score 0.0 because they struggle to ground actions in the game state.GPT-4o scores 38.9; o3-mini repeatedly moves downward based on an incorrect assumption about the exit, illustrating failures to adapt pre-existing reasoning to the environment.Source: §G.5 · p. 35, Table 23 · p. 35

H.5 RESULT FOR DARKEST DUNGEON

• Gemini-2.5-pro achieves the highest Darkest Dungeon score at 93.7, narrowly ahead of GPT-4o at 93.4 and Deepseek-R1 at 91.7, while Qwen-2.5-7B is the strongest open-source model at 88.8.Llama-3.2-1B and Minitron-4B score 0.0, while several small models issue invalid commands, overuse swaps, and become stuck after the Crusader is repositioned by the ‘Surprised!’ status.Source: §H.5 · p. 37, Table 26 · p. 37

I.1 GAME DESCRIPTION FOR MINECRAFT

• Minecraft agents use Mineflayer APIs for movement, mining, crafting, placement, smelting, combat, and chest-item transfers.The available primitives include bounded exploration, resource collection, item crafting, block placement, smelting, mob hunting, and chest retrieval or deposits.Source: §I.1 · p. 38

I.2 GAMEPLAY PROMPT FOR MINECRAFT

• The zero-shot Minecraft prompt combines task guidance, JavaScript control primitives, textual observations, and an expected reasoning-and-code response format.The user prompt supplies the current Mineflayer game state, from which the LLM infers JavaScript actions for the target task.Source: §I.2 · p. 38, Figure 18 · p. 39

I.3 EVALUATION METRIC FOR MINECRAFT

• Crafting success is scored as 100 when the target item appears in the inventory and 0 otherwise, checked at each game step.The metric uses Mineflayer inventory access and marks an episode successful only when the crafted item is presented.Source: §I.3 · p. 39

I.4 EXPERIMENTAL CONFIGURATION FOR MINECRAFT

• Experiments use temperature 1, repetition penalty 1, a 100-step interaction limit, and three repetitions per target item, reporting mean score and standard deviation.These settings apply across all evaluated LLMs.Source: §I.4 · p. 39

I.5 RESULT FOR MINECRAFT

• 75.0: o3-mini, Gemini-2.5-pro, and Claude-3.7-sonnet each crafted six of eight target items, whereas all six open-source models scored 0.0.All proprietary models failed to craft the enchanting table and nether portal; five open-source models generated no compatible Mineflayer code.Source: §I.5 · p. 39, Table 29 · p. 40
• Reflection improved GPT-4o, but reflection-planning scored 50.0 versus 71.0 for skill management.Skill management stores successful code actions and retrieves them when needed, suggesting a stronger role in Minecraft performance.Source: §I.5 · p. 40, Table 30 · p. 40

J.1 GAME DESCRIPTION FOR STARDEW VALLEY

• The Stardew Valley task maximizes profit over 13 in-game days under a 50-energy limit, requiring crop selection, scheduling, and resource management.The agent starts with 200 gold and is evaluated against a 956-gold human-expert profit baseline.Source: §J.1 · p. 40, §J.3 · p. 41
• Stardew Valley uses eight high-level actions mapped to predefined keyboard sequences, with JSON game states serialized into natural-language observations.The setup supports reproducibility through SMAPI, custom actions, and a shared save point.Source: §J.1 · pp. 40–41

J.2 GAMEPLAY PROMPT FOR STARDEW VALLEY

• The Stardew Valley zero-shot prompt constrains outputs to predefined Python-coded actions while supplying crop statistics, energy rules, and current game state.It permits up to two actions and includes prior executed actions to avoid repeated failures.Source: §J.2 · p. 41, Figure 19 · p. 42

J.5 RESULT FOR STARDEW VALLEY

• GPT-4o performs best in Stardew Valley, while open-source models fail to earn money because of invalid actions or poor crop scheduling.None of the models fully follow the optimal parsnip strategy under the imposed energy constraints.Source: §J.5 · pp. 42–43, Table 34 · p. 42
• Planning significantly improves GPT-4o in Stardew Valley, whereas Llama-3.2-3B fails to profit across four agent configurations.The task’s crop-selection and scheduling demands make planning especially relevant.Source: §J.5 · p. 43
• Vision-only input performs poorly for proprietary models, and adding images lowers GPT-4o and Claude-3.7-Sonnet performance but improves Gemini-2.5-pro.Screenshots contain redundant content and obscure structured cues such as watered plants and the current day.Source: Table 34 · p. 42, §J.5 · p. 43

K.1 GAME DESCRIPTION FOR STARCRAFT II

• StarCraft II uses a partially observable Protoss-versus-Zerg setting on Ancient Cistern LE, with observations covering resources, buildings, units, research, actions, and enemy information.The environment supports comparisons across races, maps, and difficulty settings.Source: §K.1 · p. 43

K.2 GAMEPLAY PROMPT FOR STARCRAFT II

• The agent selects from 72 high-level Protoss commands and outputs five actions executed sequentially, with prompts requiring resource-aware analysis and cost checks.The format guides strategic suggestions, reasoning, and immediately executable commands.Source: §K.1 · p. 43, §K.2 · p. 44

K.3 EVALUATION METRIC FOR STARCRAFT II

• Single-agent StarCraft II uses win rate over up to four matches, while multi-agent play scores army-supply differences and converts them into Elo ratings.The multi-agent score captures both victory and winning margin.Source: §K.3 · pp. 44–45

K.5 RESULT FOR STARCRAFT II

• GPT-4o and Gemini-2.5-pro won all four StarCraft II matches, while most proprietary models exceeded 50% win rate and small open models repeated simple actions.Successful proprietary models allocated resources over time in line with game progression.Source: §K.5 · p. 45, Table 35 · p. 46
• Planning is critical for GPT-4o in StarCraft II, whereas reflection alone performs worse than zero-shot and Llama-3.2-3B remains at 0% win rate across ablations.The result links long-term strategy maintenance to the game’s real-time setting.Source: §K.5 · p. 45, Table 36 · p. 46
• Adding image input does not improve GPT-4o, Gemini, or Claude in StarCraft II and can reduce performance relative to textual observations alone.The reported interpretation is that image input may make multimodal reasoning harder.Source: §K.5 · p. 45, Table 37 · p. 46
• Claude achieves the highest Elo in multi-agent StarCraft II despite a 50% single-agent win rate, while Minitron-8B earns the second-highest Elo despite 0% single-agent wins.The authors note that one evaluation episode may bias the observed rankings.Source: §K.5 · p. 45

L.1 GAME DESCRIPTION FOR SLAY THE SPIRE

• Slay the Spire tasks the LLM with card play and reward selection while rule-based policies handle navigation, events, and potions.Runs use a fixed seed and pre-unlock all cards for consistency and reproducibility.Source: §L.1 · pp. 45–46

L.2 GAMEPLAY PROMPT FOR SLAY THE SPIRE

• Slay the Spire represents states textually and permits PLAY, END, CHOOSE, and SKIP actions, with prompts exposing combat mechanics and detailed game state.The evaluation targets strategic choices in a stochastic, multi-step environment.Source: §L.1 · p. 46, §L.2 · p. 46

L.3 EVALUATION METRIC FOR SLAY THE SPIRE

• The Slay the Spire score combines cleared floors with bonus points for defeating three major bosses.The final-boss objective is floor 50, and each boss contributes 50/3 bonus points.Source: §L.3 · p. 46

L.5 RESULT FOR SLAY THE SPIRE

• Gemini-2.5-pro achieves the highest Slay the Spire score, defeating the second boss in two of three runs, while most open-source models make near-zero progress.The stochastic game produces high variance across proprietary models.Source: §L.5 · p. 47, Table 38 · p. 48
• GPT-4o benefits substantially from reasoning modules, with reflection scoring highest and planning and reflection-planning outperforming zero-shot.Planning is less effective because opponent intent is only partially observable and difficult to predict several turns ahead.Source: §L.5 · pp. 47–48, Table 39 · p. 48
• Adding visual input does not improve proprietary-model performance in Slay the Spire and often degrades it because screenshots omit detailed card, relic, and power information.The results favor high-quality textual representations for this information-dense environment.Source: §L.5 · p. 48, Table 40 · p. 48

M.1 GAME DESCRIPTION FOR BABA IS YOU

• Baba Is You requires discovering and manipulating text-tile rules so a controllable object can touch a winning object.The default evaluation uses Level 1, and the game state includes object coordinates and manually extracted active rules.Source: §M.1 · pp. 48–49

M.2 GAMEPLAY PROMPT FOR BABA IS YOU

• The Baba Is You prompt supplies map dimensions, object locations, active rules, and pushing mechanics before requesting directional action sequences.The action space consists of left, right, up, and down moves.Source: §M.1 · p. 49, §M.2 · p. 49, Figure 23 · p. 49

M.3 EVALUATION METRIC FOR BABA IS YOU

• Baba Is You uses hierarchical scoring: clearing the level earns 100 points, breaking ‘Wall Is Stop’ earns 20, and also creating a winning rule earns 40.Full completion overrides subgoal rewards.Source: §M.3 · pp. 49–50, Table 41 · p. 50

M.5 RESULT FOR BABA IS YOU

• Models often fail to infer that breaking ‘Wall Is Stop’ is necessary for constructing the winning condition, making their successful rule-breaking largely unintentional.Source: §M.5 · p. 50
• 20.0 was the score for both Llama-3.2-3B and GPT-4o across all tested agentic configurations, matching the zero-shot baseline.The ablation found no measurable improvement from reflection, planning, or their combination.Source: §M.5 · p. 50
• The task remains challenging because models frequently fail to account for pushing mechanics, sending text tiles in unintended directions and misaligning rule formation.Source: §M.5 · p. 50

N.1 GAME DESCRIPTION FOR 2048

• 2048 is a single-player sliding-tile puzzle on a 4×4 grid with four directional actions; progress toward creating the 2048 tile is measured by final score, normalized to 100 against an estimated 20,000-point target.Source: §N.1 · p. 51, §N.3 · p. 51

N.2 GAMEPLAY PROMPT FOR 2048

• The 2048 zero-shot prompt supplies game rules, prior and current board states, scores, and the previous action so the agent can select its next move.Source: §N.1 · p. 51, §N.2 · p. 51

N.4 EXPERIMENTAL CONFIGURATION FOR 2048

• The 2048 experiments used up to 10,000 steps, but episodes ended earlier after board lock-up or five consecutive invalid moves.Source: §N.4 · p. 51

N.5 RESULT FOR 2048

• Open-source models often misread the textual 2D board and repeated invalid actions, while GPT-4o, Gemini-2.5-pro, and Claude-3.7 detected some invalid moves but still approached random-agent performance.Source: §N.5 · pp. 52–53
• o3-mini produced a 512 tile in 4 of 5 runs, while o3 produced a 1024 tile in 2 of 5 runs; other models typically reached only 64 or 128.Source: Table 44 · p. 52, §N.5 · p. 53
• Reasoning models implicitly discovered human-like spatial organization from basic rules alone, without explicit instructions such as cornering or staircase strategies.o3’s runs showed emergent spatial organization resembling experienced human play.Source: §N.5 · p. 53, Figure 25 · p. 53
• For Llama-3.2-3B and GPT-4o, adding Reflection or Planning individually or jointly produced no significant gains, indicating external scaffolding alone was insufficient.Source: Table 45 · p. 52, §N.5 · p. 53
• o3-mini performed better with zero-shot prompting than with Reflection or Reflection-Planning, plausibly because extra modules duplicated its internal planning or propagated intermediate errors.Source: §N.5 · p. 53, Figure 26 · p. 54
• GPT-4o performed worse with image inputs, whereas Gemini-2.5 Pro and Claude 3.7 performed better with images than text, revealing non-uniform modality strengths.Source: Table 46 · p. 52, §N.5 · pp. 53–54

O DETAILS FOR DATA AUGMENTATION

• Each system prompt was paraphrased into 10 semantic-preserving variants, expanding every original example to 11 versions while keeping the dynamic user prompt fixed.The user prompt remained unchanged because modifying state-dependent content could introduce inconsistencies or hallucinations.Source: O DETAILS FOR DATA AUGMENTATION · p. 55
• Prompt augmentation increased syntactic and lexical diversity while preserving semantic fidelity, and small augmentation counts such as 3 or 10 were effective before gains diminished.Source: O DETAILS FOR DATA AUGMENTATION · p. 55, Table 47 · p. 55

P.1 ASYNCHRONOUS INFERENCES FOR MULTI-AGENT ENVIRONMENT

• Asynchronous execution lets each agent choose a new action immediately after its current action ends, supporting overlapping variable-duration actions without synchronization barriers.This design targets real-time games such as Street Fighter III and StarCraft II.Source: §P.1 · p. 55

P.2 FINE-TUNING

• Supervised fine-tuning adapted Llama-3.2-1B and 3B using 105,502 data points, with length filtering removing some Pokémon and Minecraft examples.Source: §P.2 · p. 56

P.3 UNSEEN SCENARIOS

• Unseen scenarios changed characters, maps, parties, seeds, or levels while holding other game settings constant to test intra-game generalization.Examples include Chun-Li replacing Ken, Babylon LE replacing Ancient Cistern LE, and Baba Is You Level 2 replacing training stages.Source: §P.3 · p. 56

Q ANALYSIS OF THE GAP BETWEEN IN-GAME AND OOD-GAMES IN FINE-TUNING

• Fine-tuning on Baba Is You improved OOD performance most for same-genre puzzle games such as 2048, while gains were mixed for games with different dynamics and action spaces.A diverse ten-game fine-tuning set instead improved Super Mario consistently, supporting broader cross-genre generalization.Source: Q ANALYSIS OF THE GAP BETWEEN IN-GAME… · p. 57, Table 49 · p. 57

R UNIFIED PERFORMANCE COMPARISON

• Text-only LLM agents achieved the best result for each game in the unified LLM/VLM comparison.Source: R UNIFIED PERFORMANCE COMPARISON · p. 57, Table 50 · p. 58

S STATISTICAL SIGNIFICANCE ANALYSIS

• GPT-4o showed significant or deterministic differences from Llama-3.2-3B in 10 of 12 games and from Qwen-2.5-72B in 7 of 12 games.Source: S STATISTICAL SIGNIFICANCE ANALYSIS · p. 57, Table 52 · p. 59
• Across-game paired tests found GPT-4o significantly outperformed both Llama-3.2-1B and Qwen-2.5-72B.Source: S STATISTICAL SIGNIFICANCE ANALYSIS · p. 57, Table 53 · p. 59

T DISCUSSION FOR INTEGRATING ORAK WITH MULTI-TURN RL FINE-TUNING FRAMEWORKS

• Orak can support multi-turn RL fine-tuning through reward discounting and turn-level rewards, which smooth credit assignment when intermediate feedback is available.The discount factor satisfies 0 < γ < 1, while turn-level rewards use λ to balance intermediate and final scores.Source: T DISCUSSION FOR INTEGRATING ORAK WITH… · pp. 58–59
• Dynamic data extraction repeatedly rolls out policies, estimates rewards, and retains the better policy based on validation accuracy to support progressive experience collection.Source: T DISCUSSION FOR INTEGRATING ORAK WITH… · p. 59

Cost Consideration

• Six games require one-time purchases costing typically $9.99–$24.99, while proprietary LLM API calls create the larger recurring expense.The authors regard the benchmark as accessible for sustained research despite these costs.Source: U LIMITATIONS AND FUTURE WORK · p. 59

License Issue

• Commercial game executables and assets are not distributed, and derived models are restricted to non-commercial research use.Users must purchase the commercial games themselves for evaluation.Source: U LIMITATIONS AND FUTURE WORK · p. 59

Real-time Gameplay

• Real-time games are paused during LLM inference, so current evaluation removes responsiveness constraints that matter for practical gameplay.Latency-aware evaluation protocols are left for future work.Source: U LIMITATIONS AND FUTURE WORK · p. 59

Study on RL-based Fine-tuning

• Orak does not explore RL-based fine-tuning, although gameplay feedback could support context-aware rewards for dynamically adaptive strategic reasoning.The authors identify this as a future direction for multi-agent settings and related real-world applications.Source: U LIMITATIONS AND FUTURE WORK · pp. 59–60

Support for Diverse Modalities

• Orak supports visual evaluation but omits audio, despite sound cues conveying danger, attacks, spatial awareness, and threat proximity in many games.Audio-enabled benchmarking and Speech-Language Model evaluation remain open challenges.Source: U LIMITATIONS AND FUTURE WORK · p. 60

Societal Impact

• Minecraft and Stardew Valley could support systematic comparisons of human and LLM trajectories and studies of emergent social behavior among multiple agents.The authors present these simulation environments as testbeds for measuring long-horizon behavior and social impact.Source: U LIMITATIONS AND FUTURE WORK · p. 60