PaperlayerView exact revision on arXiv

Source-linked AI summary

BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games

Davide Paglieri, Bartłomiej Cupiał, Samuel Coward, Ulyana Piterbarg, Maciej Wolczyk, Akbir Khan, Eduardo Pignatelli, Łukasz Kuciński, Lerrel Pinto, Rob Fergus, Jakob Nicolaus Foerster, Jack Parker-Holder, Tim Rocktäschel

arXiv:2411.13543v2cs.AI

TL;DR

Agentic models still struggle with complex, dynamic tasks requiring long-horizon interaction, spatial reasoning, and exploration, while existing evaluation methods for these abilities are limited. BALROG evaluates LLMs and VLMs across six challenging games using fine-grained metrics. Models show partial success on easier games but struggle on harder tasks, and several perform worse with visual observations.

  • Problem

    Evaluations of agentic models have limited coverage of long-horizon interaction, despite real-world tasks requiring extended decision-making, spatial reasoning, and exploration.

    Source: §1 · pp. 1–2

  • Method

    BALROG combines six reinforcement-learning game environments to evaluate LLMs and VLMs, using fine-grained performance metrics and zero-shot baselines across language-only and vision-language modalities.

    Source: §1 · pp. 2–3

  • Results

    Models show partial success on easier games but struggle on harder tasks, and multimodal models often perform worse when environment images are provided.

    Source: §1 · p. 3

  • Takeaways & Limitations

    BALROG enables fine-grained evaluation of long-term planning and spatial reasoning across a difficult benchmark, with code and the benchmark opened for external submissions.

    Source: §1 · p. 3

    Image-processing costs constrained evaluation to one image per observation, with the remainder of the history supplied as text.

    Source: §6 · p. 10

Abstract

from arXiv · show

Large Language Models (LLMs) and Vision Language Models (VLMs) possess extensive knowledge and exhibit promising reasoning abilities, however, they still struggle to perform well in complex, dynamic environments. Real-world tasks require handling intricate interactions, advanced spatial reasoning, long-term planning, and continuous exploration of new strategies-areas in which we lack effective methodologies for comprehensively evaluating these capabilities. To address this gap, we introduce BALROG, a novel benchmark designed to assess the agentic capabilities of LLMs and VLMs through a diverse set of challenging games. Our benchmark incorporates a range of existing reinforcement learning environments with varying levels of difficulty, including tasks that are solvable by non-expert humans in seconds to extremely challenging ones that may take years to master (e.g., the NetHack Learning Environment). We devise fine-grained metrics to measure performance and conduct an extensive evaluation of several popular open-source and closed-source LLMs and VLMs. Our findings indicate that while current models achieve partial success in the easier games, they struggle significantly with more challenging tasks. Notably, we observe severe deficiencies in vision-based decision-making, as several models perform worse when visual representations of the environments are provided. We release BALROG as an open and user-friendly benchmark to facilitate future research and development in the agentic community. Code and Leaderboard at balrogai.com.

1 INTRODUCTION

BALROG addresses the need to evaluate agentic models on long-horizon interaction, beyond benchmarks limited to a few dozen rounds. Its six-game testbed reveals partial success on easier tasks but major difficulties on harder games and in vision-based decision-making.

Motivation

  • Where many current agentic benchmarks involve only a few dozen interaction rounds, realistic tasks can require orders of magnitude more.The introduction names office tasks, internet navigation, and GitHub issue resolution as examples of shorter-horizon benchmarks.Source: §1 · pp. 1–2

Benchmark

  • BALROG combines six procedurally generated reinforcement-learning games, ranging from comparatively accessible tasks to NetHack, into a benchmark for long-context LLMs and VLMs.The environments use lightweight simulators, and repeated encounters with the same instance are unlikely.Source: §1 · p. 2

Analysis

  • The evaluation examines spatial reasoning, systematic exploration, long-term planning, and discovery of environment dynamics, including a knowing-doing gap.The analysis finds that models may possess relevant knowledge but fail to use it during play.Source: §1 · pp. 2–3

Results

  • Models show fair performance on easier games but make little progress on MiniHack and NetHack, with multimodal models often performing worse when given environment images.The authors also report that some models know game facts from pretraining but fail to apply them in practice.Source: §1 · p. 3

Contributions

  • BALROG provides fine-grained model evaluation and a data-informed progression system for NetHack, alongside an open toolkit supporting inference-time prompting strategies.The toolkit supports strategies such as chain-of-thought and few-shot learning; the paper's baseline evaluations are zero-shot.Source: §1 · p. 3

2 BALROG

BALROG brings long-context models into challenging reinforcement-learning environments and separates model evaluation from inference-time prompting strategies. Its suite spans different games and task demands, and accepts both new models and agentic strategies.

2 BALROG

  • BALROG evaluates whether long-context models can act agentically in sequential decision tasks and decouples prompting strategies from underlying models to support rapid prototyping and consistent evaluation.The benchmark is intended to help researchers understand whether such models can automate complex activities.Source: §2 · p. 3

2.1 ENVIRONMENTS

  • The suite includes BabyAI navigation, Crafter survival and crafting, text-based TextWorld exploration, and Baba Is AI puzzles that alter object interactions.BabyAI tasks are described in natural language; TextWorld is not a grid-world.Source: §2.1 · pp. 3–4
  • The benchmark's environments span different skills, time horizons, and task complexities, positioning it for evaluation of harder, longer-horizon interactive settings.The evaluated tasks cover a range of difficulties.Source: Table 1 · p. 4, §2.1 · p. 4
  • MiniHack tasks assess exploration, navigation, planning, and resource management, while NetHack demands long-term strategy, short-term tactics, and accurate credit assignment.A winning NetHack game can involve hundreds of thousands of steps, and human players may take years to master it.Source: §2.1 · p. 4

2.2 SUBMITTING TO THE BENCHMARK LEADERBOARD

  • Submissions may introduce natural-language action-generating models, evaluated zero-shot by default, or inference-time strategies that extend beyond direct zero-shot action prediction.The benchmark distinguishes new models from agentic strategies for improving an existing model.Source: §2.2 · p. 4

3 ZERO-SHOT EVALUATION PROTOCOL

The evaluation establishes zero-shot baselines by prompting models to choose actions from interaction histories. It specifies scoring, invalid-action handling, observation formats, and the models evaluated.

3 ZERO-SHOT EVALUATION PROTOCOL

  • The protocol measures baseline zero-shot performance, prompting agents at each timestep to select a natural-language action from their interaction history.Successful behavior requires interpreting game rules and action options and producing valid actions.Source: §3 · p. 5, §3.1 · p. 5, Figure 2 · p. 6

3.1 EVALUATION SETTING

  • Invalid or hallucinated actions receive feedback, trigger a default fallback action, and are logged so interaction can continue and errors can be analyzed.Fallbacks include a do-nothing action or a standard move such as north.Source: §3.1 · p. 5
  • Task scores range from 0 to 100; some environments use completion scores of 0 or 100, while others score achievement proportions.For NetHack, BALROG proposes a data-informed progression metric because the game's scoring system does not adequately reflect actual progression.Source: §3.1 · p. 5

3.2 OBSERVATIONS

  • Prompts introduce game rules and available actions, then provide chat-formatted observation-action histories; environments are evaluated with language-only and, where applicable, vision-language observations.Vision-language observations pair a current-state image with a natural-language description; TextWorld has no visual component.Source: §3.2 · pp. 5–6
  • MiniHack and NetHack language observations include ASCII maps, and experiments use a history length of 16 observations while allowing benchmark participants to change it.The general prompt is not fine-tuned to a specific LLM.Source: §3.2 · pp. 5–6

3.3 MODELS

  • The evaluation covers closed- and open-source LLMs and VLMs, with o1-preview tested only on NetHack because of budget constraints.The model set includes Gemini, GPT-4o, Claude 3.5, and Llama 3.1 and 3.2 variants.Source: §3.3 · p. 6

4 RESULTS

Across environments, language-only performance is led by Claude 3.5 Sonnet, while vision input lowers GPT-4o and Llama 3.2 performance. Qualitative analyses reveal persistent weaknesses in exploration, planning, and aligning actions with known consequences.

4 RESULTS

  • GPT-4o and Claude 3.5 Sonnet perform best on BabyAI, Crafter, and Baba Is AI; Llama models lead on Baba Is AI language-only, while Gemini models complete no TextWorld tasks after safety flags.The TextWorld prompts contained no actual safety concerns.Source: §4 · p. 6
  • No model solves MiniHack quest or Boxoban tasks, and o1-preview achieves just 1.5% average game progression on NetHack.The reported NetHack result is for the best-performing model, o1-preview.Source: §4 · p. 7
  • 32.64% average progression: Claude 3.5 Sonnet is best overall in language-only evaluation; GPT-4o and Llama 3.2 decline with images, unlike Gemini-1.5-Pro and Claude 3.5 Sonnet.Table 2 gives the language-only aggregate; Table 3 presents vision-language results.Source: §4 · p. 7, Table 2 · p. 7, Table 3 · p. 7

4.1 QUALITATIVE ANALYSIS

  • Agents struggle with spatial tasks: they fail to place objects adjacently in BabyAI and often become cornered during corridor combat in NetHack and MiniHack.Source: §4.1 · p. 7
  • In TextWorld’s Coin Collector, agents revisit rooms while missing others; similar inefficient exploration appears in MiniHack maze-like quests.Source: §4.1 · p. 7
  • MiniHack and NLE show near-zero performance, and no successful trajectory appears in Boxoban, where each step risks irreversible failure.Source: §4.1 · p. 7
  • Models struggle to turn discoveries into strategies, including using paralysis potions or crossing lava with tools, and fail to learn Crafter’s long-term shelter-building survival skill.Source: §4.1 · p. 8
  • Models often take actions they know are dangerous, including exiting NetHack early or eating rotten food despite recognizing the risks.Source: §4.1 · p. 8

5 RELATED WORK

Prior LLM benchmarks span static language, reasoning, and coding tasks, alongside short-horizon interactive tasks. BALROG uses games with varied difficulty, including NetHack, to extend agentic evaluation; the paper notes that accessible large-scale evaluation can be hindered by some environments’ computational and multimodal requirements.

5 RELATED WORK

  • Static benchmarks assess language, reasoning, and coding, but generally lack sequential decision-making, adaptation to changing environments, and sustained interaction.The paper characterizes these benchmarks as single-turn or short-context and notes that they have saturated rapidly.Source: §5 · p. 8
  • Existing interactive benchmarks cover office tasks, web navigation, GitHub issues, and multi-agent settings, while prior game benchmarks emphasize problem-solving, spatial reasoning, and clear rules.Source: §5 · p. 8
  • BALROG spans games of varied difficulty, including NetHack, which takes humans years to master; some alternative open-ended environments are less practical for accessible large-scale benchmarking because of computational demands and multimodal inputs.Source: §5 · pp. 8–9
  • BALROG currently evaluates single-agent foundational capabilities; collaboration and coordination remain possible directions for future extensions.Source: §5 · p. 9

6 OPEN RESEARCH PROBLEMS

BALROG supports research on inference-time methods for long-context agentic tasks, but current techniques and models face practical and computational limits. The discussion highlights variable VLM performance, constrained visual histories, and unresolved limits on complex reasoning.

In-Context Learning and Few-Shot Prompting

  • Current methods that improve reasoning on shorter-context problems can be infeasible to apply naively to BALROG’s extremely long-context tasks.The authors present this as an open challenge for developing inference-time methods for agentic capabilities.Source: §6 · p. 9
  • Naive few-shot prompting is computationally expensive on BALROG; one NetHack demonstration can require upwards of 700,000 input tokens.The authors found these experiments infeasible at the time and suggest retrieval-augmented selection of relevant demonstration parts as a possible direction.Source: §6 · p. 9

Advanced Reasoning Strategies

  • BALROG supports studying chain-of-thought, self-refinement, planning, multi-agent collaboration, tool use, memory, reinforcement learning, and self-improvement for long-context reasoning.The authors identify these as future-work directions; memory or reinforcement learning could help models apply knowledge in practical, long-horizon tasks.Source: §6 · p. 9

Limitations of Current Vision-Language Models

  • VLM performance varies: Llama 3.2 struggles to integrate visual information into coherent decisions, while Sonnet 3.5 performs more strongly in VLM mode.The authors suggest differences in training objectives and datasets may contribute to this variation; Sonnet 3.5’s stronger performance is attributed in part to computer-use training.Source: §6 · pp. 9–10
  • Grid-based observations differ from many VLMs’ natural image-text training data, and image-processing cost limited evaluation to one image per observation, with the remaining history in text.The authors note that this constraint may hinder some models, while Claude 3.5 Sonnet remains robust under these conditions.Source: §6 · p. 10

Computational Limitations of Large Language Models

  • LLMs perform well on simple tasks such as navigation and object manipulation but struggle with complex tasks that may require non-trivial, general-purpose computation.Possible sources include difficulty retrieving relevant computational circuits, inference-time budget limits, or representational expressivity; the authors call for further research into causes.Source: §6 · p. 10

7 CONCLUSION

BALROG evaluates LLM and VLM agentic capabilities across diverse, challenging long-horizon tasks. Its evaluations reveal shortcomings in vision-based decision-making and long-term planning, using procedurally generated environments intended to support rigorous comparisons.

7 CONCLUSION

  • BALROG assesses LLM and VLM agentic capabilities across diverse, challenging, long-horizon tasks.The conclusion describes its evaluation framework as using fast, procedurally generated environments to prevent test-set leakage.Source: §7 · p. 10
  • The benchmark reveals shortcomings in vision-based decision-making and long-term planning, with a gap between model performance and human-level capabilities.The authors connect these deficiencies, identified through qualitative analysis, to challenges in real-world scenarios.Source: §7 · p. 10
  • Procedurally generated environments are intended to prevent test-set leakage and support rigorous, fair comparisons.Source: §7 · p. 10

ETHICS STATEMENT

The authors view experimentation in simulated environments as important for building safe agentic systems. They emphasize interpretability and alignment with human intentions as concerns for agent behavior.

ETHICS STATEMENT

  • The paper argues that simulated environments, where agent behavior is easy to interpret, are crucial for building safe agentic systems.Source: ETHICS STATEMENT · p. 10
  • The authors identify ensuring that an agent’s behavior is aligned with human intentions as an important question.Source: ETHICS STATEMENT · p. 10
  • The paper frames its work as a benchmark for evaluating LLM agentic capabilities in simulated environments.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The paper shares its evaluation code and points to appendices for environment evaluation schemes. Its BabyAI material describes the environment and the textual observations used for five BabyAI-Text tasks.

REPRODUCIBILITY STATEMENT

  • The authors share the evaluation code in supplementary materials and provide full descriptions of environment evaluation schemes in Appendices A to F.Source: REPRODUCIBILITY STATEMENT · p. 11

A BABY AI

  • BabyAI is a 2D grid-world research platform for grounded language learning and instruction following, with agents navigating rooms and interacting with objects.The paper uses it as a historically relevant environment expected to be relatively easy to solve.Source: A BABY AI · p. 18

A.1 BABYAI-TEXT

  • The evaluation uses five BabyAI-Text tasks, whose observations are provided as textual descriptions rather than symbolic representations.Source: §A.1 · p. 18

A.2 BABYAI RESULTS

BabyAI results show GPT-4o leading, with Llama 3.1 70B close behind; adding vision generally reduces performance, while Gemini-1.5-Pro remains stable.

A.2 BABYAI RESULTS

  • GPT-4o leads BabyAI, closely followed by Llama 3.1 70B; adding vision reduces performance for the models except Gemini-1.5-Pro, whose performance remains stable.Results use 25 seeds for each of five tasks.Source: §A.2 · p. 18, Table 4 · p. 18, Table 5 · p. 18

A.3 OBSERVATIONS

  • BabyAI observations pair natural-language action instructions with descriptions of nearby objects and their relative positions.The example prompt describes actions such as turning, moving forward, picking up, dropping, and toggling objects.Source: §A.3 · p. 19

B CRAFTER

  • Crafter is an open-source, Minecraft-inspired 2D survival game designed for research on generalization, exploration, and long-term reasoning.It combines resource gathering, crafting, and combat in procedurally generated environments.Source: B CRAFTER · p. 19

B.1 CRAFTER RESULTS

Crafter results differ by modality and model: GPT-4o leads in language-only mode, while Gemini-1.5-Pro leads with vision. Rendering Crafter in 3D does not improve VLM performance.

B.1 CRAFTER RESULTS

  • GPT-4o leads in language-only Crafter, Gemini-1.5-Pro leads in vision-language mode, and Llama 3.2 90B falls below its 11B counterpart when images are added.Standard errors use 10 seeds.Source: §B.1 · p. 20, Table 6 · p. 20, Table 7 · p. 20

B.2 OBSERVATIONS

  • Crafter prompts list actions for movement, gathering, survival, crafting, and combat, alongside 22 achievements for agents to pursue.The example also supplies status, inventory, and nearby-object observations.Source: §B.2 · p. 21

B.3 ANALYSIS OF VLM PERFORMANCE IN THE CRAFTER ENVIRONMENT

  • Rendering Crafter with Minecraft 3D models and textures did not improve VLM performance; some results were slightly worse than with the original 2D textures.The authors say further experimentation is needed before drawing stronger conclusions about why VLMs underperform.Source: §B.3 · p. 22

C TEXTWORLD

  • TextWorld experiments use three interactive-fiction games whose difficulty can vary with room counts, obstacles, and distractor rooms.The task-generation rules follow Lu et al. (2024).Source: C TEXTWORLD · pp. 22–23

C.1 TREASURE HUNTER

  • Treasure Hunter uses a 20-room maze at maximum difficulty, with locked doors and containers; tasks solvable optimally in 20 steps or fewer are excluded.The solution description is removed.Source: §C.1 · p. 23

C.2 THE COOKING GAME

  • The Cooking Game spans 13 rooms and up to five ingredients, requiring agents to navigate, process food with tools, and cook it using specified methods.The setup enables all additional challenging options.Source: §C.2 · p. 23

C.3 COIN COLLECTOR

  • Coin Collector has 40 rooms, possible distractor rooms, and a 20-step optimal route; its solution description is removed to increase difficulty.The task tests navigation, exploration, and efficient decision-making.Source: §C.3 · p. 23

C.4 TEXTWORLD RESULTS

TextWorld results show GPT-4o ahead by more than twofold over its closest competitor, while Coin Collector remains difficult and Gemini APIs could not complete evaluation. The following material describes the cooking-game prompt and Baba Is AI benchmark setup.

C.4 TEXTWORLD RESULTS

  • GPT-4o achieves more than twice the average progression of its closest competitor, Llama 3.1 70B; it solves Coin Collector only once in 20 attempts.Gemini APIs often flagged TextWorld inputs as unsafe, preventing full evaluation; those models were marked at 0% progression.Source: §C.4 · p. 23, Table 9 · p. 24

C.5 OBSERVATIONS

  • The Cooking Game prompt gives agents 80 steps without restarting and specifies the sequence for finding a recipe, preparing ingredients, cooking, and eating the meal.It names the BBQ for grilling, stove for frying, and oven for roasting.Source: §C.5 · p. 25

D BABA IS AI

  • Baba Is AI tests rule manipulation across 40 puzzle levels, requiring agents to combine previously seen rules in novel ways.Players rearrange textual rules to change object properties and goals.Source: D BABA IS AI · pp. 25–26, Figure 6 · p. 26

D.1 BABA IS AI LANGUAGE WRAPPER

  • The Baba Is AI language wrapper describes active rules and object positions relative to the player; it withholds solutions and grid boundaries in text-only experiments..Source: §D.1 · p. 26

D.2 BABA IS AI RESULTS

The supplied sections report Baba Is AI results and describe MiniHack and NetHack tasks, interfaces, scoring, and knowledge-use analysis. They portray NetHack progression as difficult to achieve and measure.

D.2 BABA IS AI RESULTS

  • Llama models lead Baba Is AI language-only results, with Llama 3.2 90B exceeding GPT-4o by 10%; adding vision hurts performance except for Gemini-1.5-Pro.The results use five seeds for each of 40 tasks.Source: §D.2 · p. 26, Table 10 · p. 26, Table 11 · p. 26

D.3 OBSERVATIONS

  • Baba Is AI observations present active rules, object positions, and movement actions, letting agents inspect and rearrange rule blocks.The prompt advises identifying rules in the form “[Subject] IS [Property].”Source: §D.3 · p. 27

E MINIHACK

  • MiniHack covers navigation, combat, questing, and Boxoban puzzles; only GPT-4o and Gemini-1.5-Pro complete some Corridor and CorridorBattle tasks, while other models solve none.The tasks include mazes, lava crossings, monsters, and pushing four boulders onto goal locations.Source: E MINIHACK · pp. 27–28

E.1 OBSERVATIONS

  • MiniHack observations expose a broad action set and state information, including movement, combat, inventory, and nearby terrain or monsters.The example quest prompt asks agents to explore, fight, and reach stairs down.Source: §E.1 · pp. 30–31

F NETHACK LEARNING ENVIRONMENT

  • NetHack requires descending procedurally generated dungeons, gathering equipment, and eventually retrieving the Amulet of Yendor; a winning run may involve hundreds of thousands of steps.The language wrapper translates NLE observations into readable text and converts text actions into discrete NLE actions.Source: F NETHACK LEARNING ENVIRONMENT · p. 31, §F.1 · p. 32

F.2 NEW NETHACK PROGRESSION SYSTEM

  • The NetHack progression system estimates human win probability from dungeon or experience level, assigning 0% to level 1 and 100% to ascension.Agent progress is the higher progression reached through dungeon level or experience level, based on human-played games.Source: §F.2 · pp. 32–33

F.3 NETHACK RESULTS

  • o1-preview achieves the highest NetHack progression among tested models, but the paper reports it remains far from significant progress.The best individual run, by Gemini-1.5-Pro in vision-language mode, reached dungeon level 3 and experience level 4.Source: §F.3 · p. 33, Table 14 · p. 33, Table 15 · p. 33

F.4 OBSERVATION

  • NetHack prompts retain the game message and language observation across turns but omit agent statistics and inventory to limit context growth.Language mode receives an ASCII map, while vision-language mode receives an RGB tiles map.Source: §F.4 · pp. 33–34
Loading 2411.13543v2…