PaperlayerView exact revision on arXiv

Source-linked AI summary

MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge

Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, Anima Anandkumar

arXiv:2206.08853v2cs.LGcs.AIcs.CLcs.CV

TL;DR

Existing embodied agents often train in narrow, isolated settings and therefore do not address broad open-ended task learning. MineDojo combines a diverse Minecraft task suite, internet-scale multimodal knowledge, and a video-language learned reward; its agent solves varied tasks, with success-rate gains of up to 73% in some cases. The framework and resources are released to support generally capable embodied-agent research.

  • Problem

    Existing embodied agents typically learn tabula rasa in isolated environments with limited objectives, producing specialists that generalize narrowly across tasks.

  • Method

    MineDojo combines a Minecraft benchmark, internet-scale multimodal knowledge, and MINECLIP, a video-language model used as an open-vocabulary reward for reinforcement learning.

  • Results

    MineDojo’s agent solves the majority of 12 experimental tasks, matches competitive performance against engineered dense rewards, and improves success rates by up to 73% in some cases.

  • Takeaways & Limitations

    MineDojo provides an open framework and released resources for developing and evaluating open-ended, generally capable embodied agents.

  • Takeaways & Limitations

    MINECLIP has 150M parameters and is substantially more expensive to query than hard-coded reward functions, requiring acceleration choices for reinforcement-learning training.

Abstract

from arXiv · show

Autonomous agents have made great strides in specialist domains like Atari games and Go. However, they typically learn tabula rasa in isolated environments with limited and manually conceived objectives, thus failing to generalize across a wide spectrum of tasks and capabilities. Inspired by how humans continually learn and adapt in the open world, we advocate a trinity of ingredients for building generalist agents: 1) an environment that supports a multitude of tasks and goals, 2) a large-scale database of multimodal knowledge, and 3) a flexible and scalable agent architecture. We introduce MineDojo, a new framework built on the popular Minecraft game that features a simulation suite with thousands of diverse open-ended tasks and an internet-scale knowledge base with Minecraft videos, tutorials, wiki pages, and forum discussions. Using MineDojo's data, we propose a novel agent learning algorithm that leverages large pre-trained video-language models as a learned reward function. Our agent is able to solve a variety of open-ended tasks specified in free-form language without any manually designed dense shaping reward. We open-source the simulation suite, knowledge bases, algorithm implementation, and pretrained models (https://minedojo.org) to promote research towards the goal of generally capable embodied agents.

1 Introduction

MineDojo addresses the narrowness of conventional embodied-agent training with a Minecraft framework combining open-ended tasks, internet-scale multimodal knowledge, and scalable agent learning. Its learned video-language reward enables an agent to solve varied language-specified tasks without manually designed dense rewards.

  • Conventional embodied agents often train tabula rasa in isolated, limited environments and remain specialists that generalize narrowly.
  • MineDojo organizes generalist-agent development around diverse open-ended environments, large-scale prior knowledge, and flexible, scalable architectures.
  • Minecraft provides a procedurally generated world with varied terrains, resources, crafting, construction, and discovery, without a fixed reward or storyline.
  • The framework includes thousands of natural-language-prompted tasks and an internet-scale corpus of 730K+ YouTube videos, 6K+ Wiki pages, and 340K+ Reddit posts.
  • MINECLIP uses YouTube video-text associations as an open-vocabulary multitask reward, solving the majority of 12 experimental tasks and improving success rates by up to 73%.It achieves competitive performance with engineered dense-shaping rewards and can handle open-ended tasks without special modifications.
  • MineDojo releases its simulator, knowledge bases, implementations, pretrained checkpoints, and task-curation tools to support research on generally capable embodied agents.

2 MINEDOJO Simulator & Benchmark Suite

MineDojo’s simulator supports a large benchmark spanning automatically assessable programmatic tasks and creative tasks requiring learned evaluation. Task variation, web mining, and GPT-3 generation expand coverage across capabilities and difficulty levels.

  • Simulator and benchmark: MineDojo standardizes task specifications, world settings, and observation/action spaces through simulator APIs, enabling a benchmark with thousands of tasks.
  • Programmatic tasks: Programmatic tasks cover survival, harvesting, technology progression, and combat, with template variations changing terrain, inventory, quantity, and difficulty.
  • Task categories: Programmatic tasks use simulator state for automatic assessment, while creative tasks lack straightforward success criteria and require a learned evaluation metric.
  • Creative tasks: Creative-task definitions expand from 216 manually authored tasks to 1,560 through systematic approaches, making the suite three orders of magnitude larger than BASALT’s four tasks.
  • Creative-task expansion: YouTube tutorial mining contributes 1,042 creative task ideas, while GPT-3 contributes 302 additional tasks after de-duplication.
  • Evaluation: The benchmark includes 64 curated core tasks, evenly split between programmatic and creative tasks and between standard and difficult settings.
  • Evaluation: Researchers are recommended to run 100 evaluation episodes per task and report percentage success rates, using ground-truth or learned evaluation as appropriate.

3 Internet-scale Knowledge Base

MineDojo uses the open web as a large-scale source of multimodal Minecraft knowledge, replacing costly task-specific reward engineering and human demonstrations with data from millions of players.

  • Reward functions are difficult or infeasible to craft for MineDojo’s diverse, open-ended task suite, while large-scale expert demonstrations would also be costly and infeasible.
  • The knowledge base harvests Minecraft expertise through web scraping and filtering of videos, Wiki pages, and Reddit discussions.The pipeline also filters low-quality and toxic content.
  • 730K+ narrated YouTube videos provide 33 years of footage and 2.2B English transcript words.The videos capture creative activities and sophisticated missions that can take hours to complete.
  • The Minecraft Wiki contributes 6,735 pages containing multimodal tables, recipes, illustrations, and step-by-step tutorials.The pages are highly unstructured and designed for human rather than AI consumption.
  • Reddit contributes 340K+ posts and 6.6M comments covering task-solving questions, player achievements, and tips for users across expertise levels.The paper does not use Reddit data for training in Section 5, but identifies future uses for grounded instruction generation.

4 Agent Learning with Large-scale Pre-training

MineDojo trains language-conditioned agents with MINECLIP, a video-language model whose learned correlation supplies rewards for open-ended tasks and evaluation for creative tasks.

  • MineDojo provides an open-ended task suite and internet knowledge base, while this work demonstrates a proof-of-concept single language-prompted agent on complex Minecraft tasks.
  • MINECLIP is competitive with manual rewards on Programmatic tasks and significantly outperforms CLIPOpenAI across all Creative tasks.Programmatic success is precise, whereas Creative success is estimated by MineCLIP; results average 3 seeds and 200 episodes per seed.
  • MINECLIP maps a language goal and video snippet to a scalar reward by learning correlations between positive video-text pairs with an InfoNCE objective.
  • MINECLIP uses separate text and video encoders, with the video encoder aggregating 16 consecutive frames into one video embedding.
  • MINECLIP[attn] uses two transformer layers to capture temporal information, whereas MINECLIP[avg] uses faster average pooling without temporal ordering.
  • MINECLIP rewards train a raw-pixel, language-conditioned policy with PPO, and prompts are handled zero-shot without further finetuning.

5 Experiments

The experiments evaluate MINECLIP across diverse programmatic and creative tasks, visual shifts, multitask training, and novel-task adaptation. Results show competitiveness with manual rewards, stronger visual robustness, mixed task transfer, and improved adaptation after finetuning.

  • Evaluation setup: The evaluation covers 8 Programmatic and 4 Creative tasks selected for diverse skills and Minecraft entities.The tasks are grouped into Animal-Zoo, Mob-Combat, and Creative categories.
  • Main results: MINECLIP-guided agents achieve performance comparable to manual rewards on Programmatic tasks and outperform Sparse-only, while CLIPOpenAI achieves no success.The comparison with manual reward is not statistically significant (p-value 0.3991 ≫0.05).
  • Visual generalization: MINECLIP agents are more robust than the baseline across 27 unseen combinations of weather, lighting, and terrain.All methods incur performance drops, but MINECLIP agents show smaller relative degradation across tasks.
  • Multitask learning: A single 12-task agent improves on 6 tasks, degrades on 4, and remains roughly unchanged on 2, indicating both positive and negative task transfer.The comparison uses percentage success rates averaged over 3 seeds, with 200 test episodes per seed.
  • Multitask learning: The difference between the single 12-task agent and separately trained task-group agents is not statistically significant (p-value 0.3720 ≫0.05).The result comes from a paired Student’s t-test.
  • Novel-task generalization: Zero-shot performance on “harvest spider string” and “hunt pig” is weak, but finetuning with the MINECLIP reward substantially improves performance.The pretrained multitask agent significantly outperforms baselines using only around 5% of the total environment samples.

6 Related work

Related work spans open-ended environments, Minecraft platforms, internet-scale multimodal datasets, and large-scale embodied-agent pre-training. MineDojo distinguishes itself by combining Minecraft’s flexible world with internet-scale domain knowledge for generally capable agents.

  • Open-ended environments: Minecraft provides a procedurally generated 3D world with flexible mechanics supporting many activities and open-ended agent learning.Unlike many prior open-ended environments, the approach uses Minecraft’s abundant online gameplay data to learn open-vocabulary reward models.
  • Minecraft platforms: MineDojo builds on Malmo and MineRL while expanding the simulator and benchmarking capabilities for generally capable embodied agents.MineRL provided a codebase and human play trajectories for Minecraft research benchmarks.
  • Minecraft platforms: Existing Minecraft benchmarks emphasize dialogue, human evaluation, structure building, or rapid experimentation, whereas MineDojo centers on internet-scale knowledge.The comparison includes CraftAssist, IGLU, BASALT, EvoCraft, and Crafter.
  • Multimodal knowledge bases: Large multimodal datasets have supported representation learning, but many are not specifically targeted at embodied agents.The related work contrasts general datasets such as Common Crawl, the Pile, LAION, YouTube-8M, and HowTo100M with agent-centric collections.
  • Large-scale pre-training: Embodied-agent pre-training research includes new architectures, better representations, better policies, and direct reuse of pretrained language models.Examples include Decision Transformer, GATO, Unified-IO, VIMA, R3M, CLIPort, AlphaStar, SayCan, and VPT.

7 Conclusion

MineDojo combines a large Minecraft task suite, internet-scale multimodal knowledge, and MINECLIP, a language-conditioned reward model trained on in-the-wild YouTube videos. The framework and associated resources are released to support open-ended embodied-agent research.

  • Framework: MineDojo provides thousands of Programmatic and Creative tasks together with videos, wiki pages, and forum discussions.Programmatic tasks have simulator-state success criteria, while Creative tasks motivate a learned evaluation protocol.
  • Learning and evaluation: MINECLIP uses in-the-wild YouTube videos to provide a language-conditioned reward function and agrees well with human evaluation for Creative tasks.The conclusion presents MINECLIP as an example of the research possibilities enabled by MineDojo.
  • Simulator design: The simulator supports unified observation and action spaces intended for multitask and continually learning agents.The observation space includes RGB imagery and additional state information such as inventory, location, health, and surrounding blocks.
  • Simulator design: The framework includes a compound action space combining one movement action with an optional functional action.Functional actions can take arguments, such as crafting, or no arguments, such as attacking.
  • Simulator design: MineDojo environments can be customized by terrain, weather, lighting, entity and material spawning, and special structures.Supported examples include desert, jungle, taiga, iced plain, ocean monument, desert temple, and End city.

C.2 Creative Tasks

MINEDOJO constructs Creative tasks by combining manual brainstorming, YouTube tutorial mining, and GPT-3 generation. Its annotation pipeline extracts task goals from videos, while GPT-3 supplies detailed guidance for selected tasks.

  • Creative tasks are generated through manual brainstorming, YouTube tutorial mining, and GPT-3 queries.
  • A three-stage YouTube pipeline filters videos, obtains human acceptance decisions, and refines accepted titles, timestamps, and descriptions into task goals.The final annotation stage uses the full video and description in a labeling interface.
  • 1,042 task ideas were extracted from experienced Minecraft players, including automated mining and vertically growing cactus.
  • GPT-3-davinci generates detailed, step-by-step guidance for a subset of task goals.Examples cover crafting a gold pickaxe and sailing with a sheep in a boat.
  • Playthrough: Playthrough asks an initially barehanded agent to defeat the Ender dragon in a freshly created world.The task is separately categorized because it is uniquely difficult despite being programmatically verifiable.
  • Playthrough: The Playthrough mission requires extensive preparation, exploration, agility, and trial-and-error over a horizon of hundreds of thousands of steps.

D.1 YouTube Videos and Transcripts

The YouTube and transcript database supplies large-scale, time-aligned multimodal Minecraft knowledge, while Wiki and Reddit collections add structured game information and community discussion. The released data preserves access through metadata, IDs, and reconstruction tools but omits some raw content.

  • YouTube Videos and Transcripts: The YouTube collection process searches Minecraft-related channels, gathers their video IDs, and filters videos by Minecraft category.
  • YouTube Videos and Transcripts: Videos are filtered using transcript availability, view count, aspect ratio, duration, age restriction, and toxicity probability.A video is deleted when any Detoxify toxicity category exceeds 0.5.
  • Data access: The released YouTube and Reddit resources omit raw videos, transcripts, and post contents, requiring researchers to obtain or reconstruct them externally.YouTube provides URLs and metadata, while Reddit reconstruction uses an official API license key.
  • Wiki: The Wiki collection scrapes 6,735 multimodal pages containing text, images, tables, diagrams, screenshots, animations, and sprites.Bounding boxes and table headers are retained to preserve page structure and enable reconstruction.
  • Reddit: The Reddit database covers r/Minecraft discussions, showcases, and advice, with posts, comments, metadata, and reconstructed comment relationships.Researchers receive post IDs and metadata plus a PRAW-based download function requiring an official API license key.

E.3 Training

MINECLIP converts video observations and language goals into reward signals through alternative DIRECT and DELTA formulations. The choice depends on whether the task involves moving entities or static objects.

  • MINECLIP is adapted into DIRECT and DELTA scalar reward formulations.
  • DIRECT: DIRECT uses the probability that an observation video matches the task goal against negative goal descriptions.For example, “shear a sheep” is contrasted with prompts such as “milk a cow” or “hunt a sheep”.
  • Reward formulations: DIRECT performs strongly for tasks involving moving creatures but can be suboptimal for static-object tasks.
  • DELTA: DELTA defines reward as rt = PG,t−PG,t−1 and provides better-shaped reward for tasks involving static entities.The formulation addresses agents that attend to an object without moving closer or interacting.

F.3 RL Training

MINEDOJO trains policies with PPO and supplements on-policy learning with self-imitation from high-quality past trajectories. This combination improves the stability, performance, and sample efficiency of reinforcement learning.

  • PPO: PPO serves as the on-policy reinforcement-learning backbone and constrains deviation between new and previous policies.The surrogate objective uses an advantage estimator and deviation hyperparameter ϵ.
  • Self-Imitation Learning: Self-imitation learning stores successful or high-return trajectories generated during PPO rollouts in a replay buffer.
  • Self-Imitation Learning: Trajectory sampling prioritizes successful episodes while retaining unsuccessful episodes with probabilities proportional to episodic return.
  • Results: Adding self-imitation dramatically improves RL training stability, performance, and sample efficiency in MINEDOJO.

G.1 Task Details

The section describes MineDojo’s task groups, observation and action spaces, training procedures, and evaluation of MINECLIP on Creative tasks. It also introduces action smoothing and staged multi-task training to address practical RL challenges.

  • Task Groups: MineDojo experiments use three four-task groups: Animal Zoo, Mob Combat, and Creative.Animal Zoo covers hunting or harvesting animals, Mob Combat covers four hostile monsters, and Creative tasks lack manual dense rewards and code-defined success criteria.
  • Task Groups: Creative tasks such as finding Nether Portals or oceans are difficult to evaluate automatically because randomly generated worlds lack exact structure locations.The authors classify these tasks as Creative despite their apparent similarity to object navigation.
  • Task Setup: The experimental policy observes RGB frames, compass, GPS, and voxels, and uses 89 discrete movement, camera, use, and attack actions.The current experiments use a trimmed observation and action space because crafting and inventory management are not required.
  • RL Training: Action jitter degrades exploration and produces non-smooth behavior, so training jointly optimizes PPO with an action-smoothing loss over consecutive action distributions.The smoothing loss uses KL divergence within a sliding window of policies.
  • RL Training: Hardware limits require sequential multi-stage training, with task splits, carried-over self-imitation buffers, and a reset policy head at stage two for selected groups.Animal-Zoo and Creative tasks are split into two stages of two parallel tasks each.
  • Evaluation: MINECLIP evaluation combines human labels from 200 trajectories with trajectory-level scores and K-means clustering, while complex-task tests report F1 agreement with human labels.The additional evaluation covers five semantically complex Creative tasks and finds generally positive agreement, though not perfect.
  • Evaluation: The authors characterize MINECLIP as a proof-of-concept whose evaluation may improve with more training data and parameters, while human judgment remains important.This qualification limits the interpretation of automated Creative-task evaluation.

H Limitations and Potential Societal Impact

The section documents limitations and responsible-use considerations for MineDojo’s internet-derived knowledge base. Key constraints include missing control actions, reliance on external resources, and residual harmful content or bias despite filtering.

  • Data Limitations: The YouTube data contains video observations but no control actions, preventing direct use by imitation-learning methods that require observation-action pairs.The authors present reward learning through MINECLIP as a way to sidestep this limitation and identify Learning-from-Observation as complementary future work.
  • Societal Impact: Internet-scraped videos and Reddit posts may retain undesirable biases and toxicity despite automated filtering, and the pretrained foundation models may contain harms.The authors provide a datasheet and describe responsible-usage documentation.
  • Composition: The released database is not fully self-contained because YouTube videos and transcripts must be acquired separately, while Reddit reconstruction requires an official API license key.For YouTube, the release provides URLs rather than raw MP4 files because of legal concerns.
  • Maintenance: The dataset was collected, curated, and filtered by MineDojo coauthors between December 2021 and May 2022, with updates posted on the project website.Older versions remain permanently accessible on Zenodo, and contributors can extend the dataset through the project website.
  • Intended Use: The knowledge base is intended primarily for research on open-ended, generally capable embodied agents but also supports video understanding, document understanding, language modeling, and multimodal learning.The dataset documentation states that the entire database is intended for pre-training and has no recommended splits.
  • Responsible Use: The authors oppose research that intentionally generates harmful or toxic content using the YouTube, Wiki, and Reddit data.This restriction is stated as a prohibited use rather than a claim that all harmful content has been removed.
Loading 2206.08853v2…