PaperlayerView exact revision on arXiv

Source-linked AI summary

SmartPlay: A Benchmark for LLMs as Intelligent Agents

Yue Wu, Xuan Tang, Tom M. Mitchell, Yuanzhi Li

arXiv:2310.01557v5cs.LGcs.AI

TL;DR

Existing LLM benchmarks do not capture key characteristics of intelligent agents, including reasoning about randomness, spatial reasoning, and error handling. SmartPlay evaluates LLM agents across games and capabilities, and its results show substantial remaining gaps from human performance on challenging tasks.

  • Problem

    Existing benchmarks fail to capture key characteristics of intelligent agents, including understanding randomness, spatial reasoning, and error handling.

    Source: §6.1 · p. 9

  • Method

    SmartPlay evaluates LLM agents across six distinct games, with up to 20 settings and varied challenges spanning multiple dimensions of agent capabilities.

    Source: §1 · p. 2

  • Results

    10% gap on 3DiskHanoi; GPT-4 variants also trail human baselines on Minecraft creative tasks and Crafter.

    Source: §5.1 · p. 7

  • Takeaways & Limitations

    SmartPlay provides a standardized evaluation pipeline and a way to identify research gaps in applying LLMs as agents.

    Source: §1 · p. 2

Abstract

from arXiv · show

Recent large language models (LLMs) have demonstrated great potential toward intelligent agents and next-gen automation, but there currently lacks a systematic benchmark for evaluating LLMs' abilities as agents. We introduce SmartPlay: both a challenging benchmark and a methodology for evaluating LLMs as agents. SmartPlay consists of 6 different games, including Rock-Paper-Scissors, Tower of Hanoi, Minecraft. Each game features a unique setting, providing up to 20 evaluation settings and infinite environment variations. Each game in SmartPlay uniquely challenges a subset of 9 important capabilities of an intelligent LLM agent, including reasoning with object dependencies, planning ahead, spatial reasoning, learning from history, and understanding randomness. The distinction between the set of capabilities each game test allows us to analyze each capability separately. SmartPlay serves not only as a rigorous testing ground for evaluating the overall performance of LLM agents but also as a road-map for identifying gaps in current methodologies. We release our benchmark at github.com/Microsoft/SmartPlay

1 INTRODUCTION

SmartPlay addresses the lack of standardized evaluation for LLMs acting in environments with a benchmark of games that tests diverse agent capabilities. It provides varied settings and an automated evaluation pipeline with defined objectives and metrics.

1 INTRODUCTION

  • Existing benchmarks emphasize static knowledge, reasoning, or conversational behavior, leaving LLM capabilities in agent–environment interaction without systematic evaluation.The introduction identifies the absence of a standardized benchmark as a key reason these capabilities have not been systematically studied.Source: §1 · p. 2
  • Intelligent LLM agents must plan over long horizons, reason about probabilities and space, and learn from interactions or mistakes in unseen situations.These are the four challenges the paper says prior benchmarks do not capture.Source: §1 · p. 2
  • Procedurally generated games can resist evaluation-data contamination, while LLMs struggle to memoize intermediate states even in 3-disk Tower of Hanoi.Games also involve problem-solving, odds, spatial reasoning, changing difficulty, and quantifiable outcomes (Pell, 2011; Genesereth et al., 2005; Whiteson et al., 2010; Schaul et al., 2011; Bellemare et al., 2013; Cˆot´e et al., 2019; Hafner, 2021; Guss et al., 2021; Fan et al., 2022).Source: §1 · p. 2
  • SmartPlay comprises six games, offering up to 20 settings and infinite environment variations that range from simple rule-following to long-term planning and multi-hop dependencies.Its settings include both deterministic and stochastic environments and span skills from basic text understanding to 3D spatial reasoning.Source: §1 · p. 2
  • SmartPlay defines game objectives and metrics including completion rate, reward, and score, enabling automated standardized evaluation that the authors use to compare recent LLMs and identify research gaps.The authors describe the benchmark as a reachable near-term goal that may nevertheless require new breakthroughs.Source: §1 · p. 2

2 CAPABILITIES NECESSARY FOR INTELLIGENT AGENTS

The section defines nine abilities important for intelligent LLM agents and operationalizes them through graded evaluation dimensions.

2 CAPABILITIES NECESSARY FOR INTELLIGENT AGENTS

  • Long-text understanding has 4 degrees based on document length and syntactic variation, from few fixed lines through few fixed paragraphs and syntactic variations to longer than 1 page (500 words); reasoning involves multi-hop deduction about in-game object interactions or action dependencies.Source: §2 · p. 2
  • Instruction following measures compliance with environment or user rules, while planning measures long-horizon goal pursuit and prioritization among concurrent objectives.Rule levels range from a single rule to 5+ rules; planning levels distinguish fewer than 5 steps, 5+ steps, and concurrent objectives.Source: §2 · p. 2
  • Generalization is assessed across increasing game variability, from fixed environments to procedurally generated worlds.The middle level uses a fixed game world with random objectives.Source: §2 · p. 3
  • Understanding the odds is evaluated by how central randomness is, from absent to a core game mechanism.The three levels are no randomness, randomness present in the game, and randomness as the core mechanism.Source: §2 · p. 3
  • Learning from interactions measures knowledge acquired through play, while error handling tests recovery from mistakes through rollback or reasoning and replanning.Learning levels range from no learning to 5+ unique interactions; error handling ranges from not required to reasoning and replanning to correct an error.Source: §2 · p. 3
  • Spatial reasoning evaluates understanding of and navigation through game worlds across increasing dimensionality.Its levels are 0–1D, 2D, and 3D.Source: §2 · p. 3

3 GAMES IN SMARTPLAY

SmartPlay uses distinct games to probe different agent capabilities and also tests how well agents integrate skills, with Crafter combining multiple demands in one environment.

3.1 RESEARCH CHALLENGES

  • The games separate capability profiles, from Bandits’ odds to Hanoi’s planning and Minecraft’s 3D reasoning, while Crafter integrates long texts, interactions, objectives, and error handling.The benchmark is intended to help identify capability gaps and guide future research.Source: §3.1 · p. 3

3.3 ROCK PAPER SCISSORS

  • Rock Paper Scissors combines odds and multiple-rule understanding with long-text comprehension, instruction following, and learning opponent behavior from interaction.The agent infers a hidden opponent strategy from past outcomes; action scores are revealed in the manual.Source: §3.1 · p. 3, §3.3 · p. 4

3.4 TOWER OF HANOI

  • Tower of Hanoi tests object-dependency reasoning, strategic planning, and rule-following, while mistakes may require agents to undo several moves.Agents must move one disk at a time without placing a larger disk on a smaller one, aiming to transfer all disks in the fewest moves.Source: §3.1 · p. 3, §3.4 · pp. 4–5

3.6 CRAFTER

  • Crafter (Hafner, 2021) is a comprehensive integrated test requiring reusable skills, navigation, crafting, survival, planning, and mistake handling in a procedurally generated world.Its context string omits some crafting requirements, such as needing 2 woods for a crafting table, so agents must learn through interaction.Source: §3.1 · p. 3, §3.6 · p. 5

3.2 TWO ARMED BANDITS

  • Two-armed Bandits tests odds, randomness, and learning from interaction by requiring agents to balance exploration against exploiting the better-returning arm over 50 rounds.The agent receives two slot machines with hidden reward probabilities and wins or loses the associated payoff on each round.Source: §3.2 · p. 3, §3.1 · p. 3

3.5 MESSENGER

  • MESSENGER (Hanjie et al., 2021) tests manual-to-entity synonym mapping and message delivery, with levels adding long-text understanding, generalization, reasoning, 2D navigation, and distractions.Agents must identify message and goal objects, navigate while avoiding an enemy, and deliver the message to the goal.Source: §3.5 · p. 5

3.7 MINECRAFT

  • In Minecraft creative “find” tasks, LLM agents need biome domain knowledge, connect visual observations described in text with that knowledge, and navigate 3D environments; challenges include planning, spatial reasoning, and generalization.Source: §3.7 · p. 6

4 USING SMARTPLAY

SmartPlay provides a unified text-based Gym interface with fixed game settings, recommends averaging results across repeated runs, and evaluates performance using reward, completion rate, and game-specific scores.

4.1 ENVIRONMENT INTERFACE AND EVALUATION PROTOCOL

  • Table 1 specifies each game's recommended inputs, manuals, action space, rollout length, history length, and trial numbers.Input, manual, action space, rollout length, and trial numbers should not be modified; history length and trial numbers may be increased for future needs.Source: Table 1 · p. 6, §4.1 · p. 6
  • SmartPlay uses a unified Gym interface with text observations, manuals, history, and categorical actions; repeated runs and average metrics are recommended for random games.The manual, action space, rollout length, and trial numbers are fixed; history length and trial numbers may be adjusted if changes are explicitly reported.Source: §4.1 · p. 6

4.2 EVALUATION METRICS

  • SmartPlay reports reward, completion rate, and a game-specific score to summarize agent performance.Reward follows prior RL-game definitions for Bandits, Rock Paper Scissors, Messenger, and Crafter. Completion rate applies to Hanoi, Messenger, and Minecraft. Scores count optimal-action matches for Bandits and Rock Paper Scissors, disks moved to the goal peg in Hanoi, Messenger reward, cumulative Crafter achievements, and completion of Minecraft's “find” objective.Source: §4.2 · p. 6

5 EXPERIMENTAL RESULTS

Experiments use direct step-by-step prompting with manual, history, and current observations, then select an executable action. Average scores compare nine LLMs across seven selected settings; GPT-4 variants lead other models but remain below human baselines on challenging tasks.

5.1 QUANTITATIVE ANALYSIS

  • GPT-4 variants outperform other LLMs but trail human baselines by 10% on 3DiskHanoi, 40% on Minecraft creative tasks, and 70% on Crafter.The reported gaps highlight substantial remaining room for improvement on planning, reasoning, and spatially demanding benchmarks.Source: Table 2 · p. 7, §5.1 · p. 7
  • Other proprietary models trail GPT-4, while open-source models fall further behind, especially on challenging tasks.Claude, Bard, and text-davinci-003 lag GPT-4 by more than 20% across all games except Minecraft, and GPT-4 variants score 3 times higher on Crafter. Open-source models achieve less than half GPT-4 performance on Bandit and Rock-Paper-Scissors and 1/8 on harder tasks; fine-tuned Vicuna-13b performs worse than base LLAMA-13b.Source: §5.1 · p. 7
  • Model capability profiles differ: GPT-4 variants have weaknesses, while other proprietary and open-source models show distinct strengths and losses.GPT-0614 is slightly worse at planning and reasoning than the other GPT-4 variant, and GPT-4 variants score lower on learning from interactions, error handling, and spatial reasoning. Claude exceeds Bard particularly in planning, reasoning, and instruction following; text-davinci-003 favors interaction learning and randomness but is weaker in instruction following, planning, and reasoning. LLAMA-2-13b improves planning, reasoning, and error handling over LLAMA-1-13b but is worse at learning from randomness and interactions; Vicuna-13b loses reasoning, planning, long-text understanding, and error handling after fine-tuning.Source: §5.1 · p. 8

5.2 QUALITATIVE ANALYSIS

  • Agents show interaction learning in bandit and Rock-Paper-Scissors settings, but GPT-4 variants also recover from a Crafter crafting mistake.In Bandits and Rock Paper Scissors, proprietary models initially explore, then exploit biased opponents based on prior observations; in Crafter, GPT-4 variants attempt to build a crafting table with 1 wood and recover after failing with 2 woods.Source: §5.2 · p. 8
  • Despite training familiarity with Tower of Hanoi’s starting configuration, most LLMs become confused after a few moves from intermediate states.The paper suggests these states may be uncommon in training and argues that games may be more robust to dataset contamination.Source: §5.2 · p. 8
  • LLMs struggle with spatial navigation in Minecraft, sometimes reversing direction and undoing their exploration.Observed behavior includes repeated moves north followed by repeated moves south.Source: §5.2 · p. 8

6 RELATED WORKS

Prior LLM evaluations struggle to capture interaction and intelligent-agent capabilities, while SmartPlay draws on game benchmarks and text-game methods to test a broader range of skills.

6.1 LLM EVALUATION

6.2 USING GAMES TO EVALUATE GENERALIST AGENTS

6.3 CREATING/CONVERTING TO TEXT GAMES

  • SmartPlay uses procedural text-game rules and directional object observations to test language understanding and planning without low-level controllers that can trivialize converted environments.Messenger uses procedural rule generation (Hanjie et al., 2021); for converted 2D/3D games, SmartPlay describes objects by their directional relationship to the agent, following Wu et al. (2023c).Source: §6.3 · p. 9

7 CONCLUSION · A RESEARCH CHALLENGES

SmartPlay is a benchmark and methodology for evaluating LLMs as agents, spanning basic abilities and capabilities often underrepresented in existing benchmarks. The conclusion presents it as an extensible, continuously improved suite intended to support research on more capable and reliable agents.

7 CONCLUSION

  • SmartPlay is introduced as both a challenging benchmark and a methodology for evaluating LLMs’ performance as agents.Source: §7 · p. 9
  • Its initial release includes Two-armed Bandits, Rock Paper Scissors, Messenger (Hanjie et al., 2021), Crafter (Hafner, 2021), and Minecraft creative navigation tasks (Fan et al., 2022).Source: §7 · p. 9
  • SmartPlay benchmarks basic abilities such as instruction following and in-context reasoning, alongside planning, randomness understanding, 2D/3D spatial reasoning, and error handling.The latter capabilities are often underrepresented in existing LLM benchmarks.Source: §7 · p. 9
  • The authors argue that language models should move beyond fluent language to become intelligent agents able to interact with the world and human users.Source: §7 · p. 9
  • The authors hope SmartPlay will catalyze research on building more capable and reliable LLM agents.Source: §7 · p. 9
  • SmartPlay provides guidelines for adding games to the suite and is intended to be continuously improved with up-to-date challenges for next-generation LLMs.Source: §7 · p. 9

A RESEARCH CHALLENGES

  • Table 3 associates research challenges with six games; it includes only Messenger L2+ because MessengerL1 does not cover multi-hop reasoning.Source: Table 3 · p. 14

B MINECRAFT VISUAL DESCRIPTOR

The Minecraft visual descriptor converts difficult-to-comprehend block-level observations into directional text by grouping connected blocks and locating each group relative to the agent.

B MINECRAFT VISUAL DESCRIPTOR

C EXAMPLE INPUTS … C.5 CRAFTER

The example inputs pair game instructions with sample observations, ranging from simple reward feedback to games with object, objective, and action dependencies. They illustrate what agents must pursue, avoid, and satisfy in each setting.

C.1 BANDITS

  • Bandits frames play as choosing between two slot machines to maximize earnings, with each observation reporting the selected machine and its reward.The manual describes the casino objective; the observation gives the selected machine i and reward r_i.Source: §C.1 · p. 15

C.2 ROCK PAPER SCISSORS Example input

  • Rock Paper Scissors defines the three choices, win and loss scoring, and draws, then illustrates a winning round and its score.A loss gives the negative of the winner’s score; matching choices score 0.Source: §C.2 · p. 15

C.3 HANOI

  • Hanoi requires moving the full disk stack from rod A to C one disk at a time, without placing a larger disk on a smaller one.The example starts with disks [2,1,0] on A and none on B or C; the stated goal places the stack on C.Source: §C.3 · pp. 15–16
  • Messenger asks the agent to bring a message to the goal while avoiding enemies; encountering an enemy or reaching the goal first loses with reward -1.The example guidance recommends matching seen objects to entity descriptions, including synonyms, to determine whether interaction is good or bad.Source: §C.3 · p. 16

C.5 CRAFTER

  • Crafter’s manual presents survival and progression tasks, including gathering resources, crafting tools, defending against monsters, and collecting diamonds to progress.The listed chain includes wood and stone tools, a furnace and iron tools, then diamond collection; achievements provide rewards.Source: §C.5 · p. 16
  • Crafter’s objective example makes dependencies explicit, such as requiring a wood pickaxe to collect stone and a stone pickaxe to collect iron.Collecting diamond requires making an iron pickaxe; making that pickaxe requires a furnace, coal, and iron.Source: §C.5 · p. 17
  • Crafter’s action list specifies prerequisites for movement, tool use, crafting, placing objects, and sleeping, while Noop is always applicable.For example, making an iron pickaxe requires a nearby table and furnace plus wood, coal, and iron in inventory.Source: §C.5 · pp. 17–18

C.6 MINECRAFT

The Minecraft setting asks the agent to find a forest biome without crafting. Its observations convey nearby blocks through distances, directions, and the screen space objects occupy, which varies with proximity.

C.6 MINECRAFT

  • The goal is to find a forest biome without crafting anything.Source: §C.6 · p. 18
  • Objects of the same size take up more of the field of view when closer to the agent.Source: §C.6 · p. 18
  • Observations describe surrounding blocks by distance and direction, and may report the percentage of the screen each object occupies.Source: §C.6 · p. 18

D ADDITIONAL EXPERIMENTAL RESULTS … D.3 RAW SCORES

The additional results describe a human baseline, define a normalized human score using raw game scores, human baselines, and minimum possible scores, and present a raw-score comparison of different LLMs across seven games.

D.1 HUMAN BASELINE

  • Three players, including the authors, who were very familiar with the environments and API played through the SmartPlay interface.Source: §D.1 · p. 18
  • The human-baseline trial schedule varied by game: 3 rounds each for Bandit and RPS, 1 for Hanoi, Crafter, and Minecraft, and 5 for each Messenger level.Source: §D.1 · p. 18
  • The reported human-baseline score is the final average across all trials and players.Source: §D.1 · p. 18

D.2 NORMALIZED HUMAN SCORE

  • The normalized human score is computed from an LLM’s raw game score, the human baseline for that game, and its minimum possible score.Source: §D.2 · p. 18

D.3 RAW SCORES

  • Table 4 compares different LLMs by average score across the listed games.The caption names BanditTwoArmedHighLowFixed-v0, RockPaperScissorBasic-v0, Hanoi3Disk-v0, MessengerL1v0, MessengerL2-v0, Crafter-v0, and MinedojoCreative0-v0.Source: Table 4 · p. 19
Loading 2310.01557v5…