Source-linked AI summary
Benchmarking the Spectrum of Agent Capabilities
Danijar Hafner
TL;DR
Crafter addresses the challenge of evaluating diverse agent abilities, including generalization, exploration, representation learning, and long-term reasoning. It provides reward-based and intrinsic-objective benchmarks in one visual environment, showing measurable progress while remaining difficult for current methods.
Problem
Crafter targets the challenge of evaluating generalization, wide and deep exploration, representation learning, long-term reasoning, and credit assignment together.
Method
Crafter uses a procedurally generated visual survival game with reward-based and intrinsic-objective protocols, evaluating achievement success rates and their geometric-mean aggregate score.
Results
Current methods make progress but remain far below expert performance: DreamerV2 scores 10.0% versus expert human players at 50.5%.
Takeaways & Limitations
Crafter is a single-environment benchmark of appropriate difficulty for guiding research on agents learning from extrinsic rewards or intrinsic objectives.
Takeaways & Limitations
Crafter may require future extensions with new enemies, resources, items, and achievements as research progresses.
Abstract
from arXiv · showhide
Evaluating the general abilities of intelligent agents requires complex simulation environments. Existing benchmarks typically evaluate only one narrow task per environment, requiring researchers to perform expensive training runs on many different environments. We introduce Crafter, an open world survival game with visual inputs that evaluates a wide range of general abilities within a single environment. Agents either learn from the provided reward signal or through intrinsic objectives and are evaluated by semantically meaningful achievements that can be unlocked during each episode, such as discovering resources and crafting tools. Consistently unlocking all achievements requires strong generalization, deep exploration, and long-term reasoning. We experimentally verify that Crafter is of appropriate difficulty to drive future research and provide baselines scores of reward agents and unsupervised agents. Furthermore, we observe sophisticated behaviors emerging from maximizing the reward signal, such as building tunnel systems, bridges, houses, and plantations. We hope that Crafter will accelerate research progress by quickly evaluating a wide spectrum of abilities.
1 INTRODUCTION
Crafter is an open-world survival benchmark that uses visual inputs and semantically meaningful achievements to evaluate diverse agent abilities in one procedurally generated environment.
- Crafter presents randomly generated 2D worlds with forests, lakes, mountains, and caves, where players forage, shelter, defend, collect materials, and craft tools.
- Procedural generation tests generalization, the technology tree tests broad and deep exploration, and image observations, repeated subtasks, and sparse rewards test representation learning and long-term reasoning.
- The visual interface displays procedurally generated terrain, resources, and creatures while agents learn from image inputs and pursue achievement unlocking.
- Agents are evaluated through episode achievements representing meaningful behavioral milestones, providing insight into the ability spectrum of reward and unsupervised agents.
- Crafter evaluates many abilities within one environment, reducing computation relative to benchmark suites requiring separate training runs and potentially improving representativeness for new domains.
2 RELATED WORK
Related benchmarks span diverse games and evaluation settings, but differ in complexity, computational cost, input modality, and whether they emphasize generalization or domain-specific knowledge.
- Minecraft: Minecraft offers diverse modifiable worlds, resources, tools, and monsters, but is too complex for current methods and has unclear evaluation, slow execution, and difficult setup.
- Atari: Atari contains around 54 games, but its recommended evaluation protocol requires over 2000 GPU days for five seeds and 200M steps per game.
- ProcGen: ProcGen uses procedural generation and randomized textures across 16 games, yet still requires training on each game and focuses each game on a narrow agent ability.
- NetHack: NetHack uses symbolic text inputs and demands extensive domain-specific knowledge, with 1B environment steps used in its original paper.
- Crafter: Crafter instead generates diverse complex worlds from simple rules and emphasizes generalization over memorization of facts.
3 CRAFTER BENCHMARK
Crafter is a visually observed open-world survival benchmark that evaluates diverse agent abilities through procedurally generated worlds, survival mechanics, crafting, and semantically meaningful achievements. Its geometric-mean score rewards broad exploration while exposing challenges in generalization, long-term reasoning, and efficient learning.
- Environment and evaluation: Crafter evaluates diverse agent abilities in a single open-world survival environment with visual inputs and semantically meaningful achievements.Agents forage, maintain survival needs, defend against monsters, collect materials, and build tools.
- Environment and evaluation: Each episode generates a unique world with varied terrain and resources, making memorized action sequences ineffective and requiring behavioral generalization.The worlds include grasslands, forests, lakes, mountains, caves, ores, lava, and randomly placed objects or creatures.
- Environment and evaluation: The interface provides 64 × 64 × 3 color images and 17 categorical actions, while privileged world-state information is restricted to debugging and research uses.The observation includes a local map view and the player’s inventory state, which agents must read from the image.
- Environment and evaluation: The benchmark grants agents 1M environment steps and evaluates both individual achievement success rates and an aggregate score.It includes reward and intrinsic-objective settings with otherwise identical evaluation protocols.
- Achievements and scoring: Crafter’s 22 achievements span independent and dependent behaviors, testing breadth and depth of exploration across collecting, crafting, survival, and combat.Several achievements form a technology tree and require repeated subtasks or long-term planning.
- Achievements and scoring: The aggregate score is the geometric mean of achievement success rates, emphasizing difficult achievements and favoring agents that explore broadly.Unlike reward, the score accounts for achievement difficulty without requiring those difficulties to be known beforehand; its ranking can differ from reward-based evaluation.
4 EXPERIMENTS
Experiments establish Crafter baselines for reward-driven and unsupervised agents, showing measurable progress but substantial gaps from expert performance and deeper achievements. Qualitative training also reveals sophisticated survival, construction, and resource-management behaviors.
- Experimental setup: Agents receive 1M environment steps, and experiments evaluate individual achievement success rates plus an aggregate score across reward and no-reward settings.Unsupervised agents were trained for 1M steps in under 24 hours on one GPU, with 10 random seeds per method.
- Benchmark with rewards: DreamerV2 scores 10.0%, versus PPO at 4.6%, Rainbow at 4.3%, and human experts at 50.5%.Current methods make learning progress but remain far below expert performance.
- Benchmark with rewards: Reward-trained agents learn easier achievements and some stone tools and furnaces, but none reliably collect iron or diamonds or achieve high success across many achievements.DreamerV2 also discovers growing and harvesting plants more frequently than the other reported reward agents.
- Unsupervised benchmark: Plan2Explore scores 2.1% and RND 2.0%, ahead of random at 1.6%, while occasionally reaching coal, furnaces, and stone swords.These intrinsic objectives produce meaningful behaviors but remain far below optimal or reward-trained performance.
- Emergent behaviors: After 50M steps, DreamerV2 exhibits tunnels, bridges, arrow dodging, self-built shelters, and plantations that support food supply.The behaviors were investigated qualitatively on individual occurrences rather than summarized by a separate quantitative benchmark.
- Human experts: Across 100 human-expert episodes, 5 episodes unlock all 22 achievements.Experts received game instructions and several hours of practice before recording.
5 DISCUSSION
The discussion concludes that Crafter evaluates diverse general agent abilities in one visual environment and is difficult enough to guide research with extrinsic or intrinsic objectives. The benchmark can also be extended as research advances.
- Future work: Future versions may add enemies, resources, items, and achievements as research progresses.The Python implementation is described as easy to extend.
- Discussion: Crafter evaluates a variety of general agent abilities through visual inputs within a single environment.Agents are assessed by how frequently they unlock semantically meaningful achievement milestones.
- Discussion: The authors conclude that Crafter has appropriate difficulty for future research on both reward learning and intrinsic objectives.The conclusion is based on experiments with agents trained with and without rewards.
A SUCCESS RATES WITH REWARDS
Table A.1 reports achievement success rates for Crafter agents trained with rewards under a 1M-environment-step budget, together with an aggregate geometric-mean score.
- Success rates with rewards: Success rates are fractions of training episodes unlocking each achievement at least once within 1M environment steps.The score is computed separately for each seed before averaging over seeds.
- Success rates with rewards: The aggregate score is the geometric mean of success rates across all achievements.This aggregation accounts for achievement difficulty without requiring those difficulties to be known beforehand.
B SUCCESS RATES WITHOUT REWARDS
Table B.1 reports achievement success rates for Crafter agents trained without rewards under a 1M-environment-step budget, along with an aggregate geometric-mean score.
- Success rates without rewards: Success rates measure the fraction of training episodes unlocking each achievement at least once within 1M environment steps.The score is computed separately for each seed before averaging over seeds.
- Success rates without rewards: The aggregate score is the geometric mean of success rates across all achievements.This summarizes performance across the achievement set while accounting for varying difficulty.
C SUCCESS RATES OF HUMAN EXPERTS
Human-expert success rates are computed from 100 recorded Crafter games, and their overall score is estimated using five groups treated as separate seeds.
- Human-expert achievement success rates equal the fraction of 100 recorded games in which each achievement was unlocked at least once.
- The human-expert score is computed by randomly splitting the 100 games into five groups treated as different seeds.
- The human-expert evaluation follows the same scoring procedure used for artificial agents.
D EPISODE REWARD
Crafter episode reward is visualized with shaded standard deviation, but final performance should be reported using the achievement-based score rather than reward alone.
- 22 is the optimal achievable episode reward in Crafter.
- The Crafter score is the geometric mean of success rates across all achievements and accounts for their different difficulties.
- Reward visualizations can aid debugging, whereas final Crafter performance should be reported with the score.
E TEXTURES
Crafter’s visual worlds contain varied materials, resources, objects, and creatures, with purpose-built textures supporting survival and interaction.
- Crafter worlds include several materials, resources, objects, and creatures.
- Players can collect resources, maintain food and water supplies, craft pickaxes and swords, and defend themselves.
- Crafter uses textures specifically created for the game.
F ACTION SPACE
Crafter uses a flat 17-action categorical space for movement, resource collection, object placement, crafting, and interaction, alongside achievement-count visualizations for several agents.
- Action space: 17 categorical actions let agents move, collect materials, place objects, craft objects, and interact with objects in front of the player.
- Action space: Actions whose requirements are unsatisfied have no effect.
- Achievement visualizations: Supplementary figures visualize achievement counts for Rainbow and PPO with shaded minimum and maximum values.
- Achievement visualizations: Supplementary figures visualize achievement counts for DreamerV2 and random actions with shaded minimum and maximum values.
- Achievement visualizations: Supplementary figures visualize achievement counts for unsupervised RND and Plan2Explore with shaded minimum and maximum values.