The MineRL BASALT Competition on Learning from Human Feedback

• The competition targets tasks without crisp, easily defined reward functions and uses human evaluation of agents trained to perform English-described tasks.Source: §1 · p. 2

1.1 Background and impact

• Hand-coded rewards can be optimized in unintended ways, while LfHF treats rewards as evidence about designer intent and incorporates other feedback such as demonstrations or comparisons.The CoastRunners agent collected power-ups without finishing the race.Source: §1.1 · p. 3
• Open-world benchmarks are intended to reduce cases where the environment itself signals the desired behavior; the competition aims to build toward more capable Minecraft agents and increasingly difficult tasks.Source: §1.1 · pp. 3–4

1.1.1 Performing fuzzy tasks

• LfHF can address tasks whose success depends on context, such as judging salient details in summaries, and the paper notes that human summaries and comparisons have produced strong summarization results [Stiennon et al., 2020].Source: §1.1.1 · p. 4

1.1.2 Making AI governable

• LfHF could support governance by transferring a reasonable person's conception of properties such as fairness or safety, though the authors acknowledge alternative solutions may exist.Source: §1.1.2 · p. 5

1.1.3 Value alignment

• Learning objectives from human feedback is presented as a natural approach to value alignment, although people may reasonably disagree about predictions of user well-being.Source: §1.1.3 · p. 5

1.1.4 Malicious use

• LfHF is described as omni-use: it could support beneficial or harmful AI applications, including language models fine-tuned to produce targeted hate speech.Source: §1.1.4 · p. 6

1.2 Novelty

• MineRL BASALT is, to the authors’ knowledge, the first competition to test learning from human feedback in a sequential decision-making setting.The authors hope to make it an annual competition, with task difficulty increasing alongside algorithm capabilities.Source: §1.2 · p. 6
• Unlike MineRL Diamond’s focus on environment sample efficiency, BASALT focuses on solving tasks and on sample efficiency of human feedback.MineRL Diamond uses a fixed, limited budget of compute and environment interactions.Source: §1.2 · p. 6
• BASALT’s tasks are explicitly designed without reward functions, unlike MineRL Diamond’s tasks, which include them.Source: §1.2 · p. 6
• BASALT permits any participant approach, while MineRL Diamond restricts approaches to prevent baking in domain knowledge.The only BASALT restriction is against extracting additional information from the Minecraft simulator beyond what is provided.Source: §1.2 · p. 6
• BASALT allows any type of human feedback available from contractors with Minecraft knowledge, whereas MineRL Diamond allows only a specific set of human demonstrations.Contractors may receive less than 30 minutes of additional instruction.Source: §1.2 · p. 6

1.3 Data

• Each task has a human-demonstration dataset of trajectories sampled at every Minecraft game tick, with state observations and recorded actions.States include first-person RGB frames and game-state features; actions include keyboard and mouse inputs, GUI interactions, and crafting.Source: §1.3 · p. 7
• Participants may ignore the demonstrations and use another feedback approach, though the authors expect winning submissions to use the datasets.Source: §1.3 · p. 7
• Human-generated comparisons of agent videos are also used to score agents.Source: §1.3 · p. 7
• The demonstration-collection and agent-evaluation procedures were approved under UC Berkeley IRB 2021-01-13955.Source: §1.3 · p. 7

1.4 Tasks and application scenarios

• The competition uses four tasks so that performance across them can encourage general-purpose techniques rather than task-specific hard-coding.Training code need not be identical across tasks.Source: §1.4 · pp. 7–8
• FindCave requires an agent spawned in a plains biome to explore the surrounding area and find a naturally generated cave.Source: §1.4 · p. 8
• MakeWaterfall asks an agent in an extreme hills biome to create a waterfall and take a scenic photograph, represented by throwing a snowball.Judges assess both whether a waterfall was created and the photograph’s aesthetic quality; agents receive two water buckets and movement tools.Source: §1.4 · p. 8
• CreateVillageAnimalPen requires building a pen beside a village house and corralling a pair of breedable animals into it.Agents receive building materials and food for luring animals.Source: §1.4 · p. 8
• BuildVillageHouse requires constructing a new house in the style of the village houses without damaging the village.Agents receive materials for building in different village types.Source: §1.4 · p. 8

1.5 Metrics

• Agents are evaluated by human raters who compare videos of two agents performing a task in the same test environment.Test environments use the training-environment generation procedure with different Minecraft world seeds.Source: §1.5 · p. 8
• TrueSkill scores are computed from human comparisons, with matches selected for expected information gain and continued until rankings are sufficiently stable.Source: §1.5 · pp. 8–9
• Final scores average each participant’s within-task normalized TrueSkill scores, so no single task unduly drives the total while score differences within tasks still matter.Scores are normalized to mean 0 and standard deviation 1 within each task.Source: §1.5 · p. 9
• Exact replication is not guaranteed because different humans may judge the same matches differently, although the authors expect reruns to produce similar, consistent results.The videos shown to raters are replicable to the extent that participant training procedures are replicable.Source: §1.5 · pp. 9–10
• Raters may overlook valued factors or overweight less important features, and follow-up questions about specific performance features only partly explain low scores.Raters may also rely on implicit features the questions do not cover.Source: §1.5 · p. 10
• Human evaluations can have relatively high variance; calibration is intended to reduce it, and the authors plan to use enough matches to overcome remaining variance.Source: §1.5 · p. 10

Baselines

• On MakeWaterfall, behavioral cloning approaches waterfalls but typically does not create them, because water placement is rare in the demonstrations.The baseline is provided as a starting point, but the authors note significant room for improvement.Source: §1.6 · p. 10

Starting kit

• The starter kit includes a reward-free Gym environment and demonstration-download code for each of the four tasks.It also provides a submission template and a ready-to-submit behavioral-cloning implementation.Source: §1.6 · p. 10
• Participants can use MineRL Diamond environments with reward signals to simulate human feedback and quickly test human-in-the-loop algorithms.The paper gives simulated comparisons between trajectories as one example.Source: §1.6 · p. 11

Tutorial and documentation

• The website provides competition guidance and updates, plus references to prior learning-from-human-feedback approaches.These references are intended to give participants ideas for tackling the competition.Source: §1.7 · p. 11

2.1.1 Submission protocol

• Submissions include trained agents and training code, with an English-language feedback interface and instructions required when training uses live human feedback.The feedback interface must be usable by people with minimal training (<30m); training code is generally checked for rule compliance rather than run.Source: §2.1.1 · p. 11

2.1.2 Evaluating submissions

• Final evaluation uses human pairwise comparisons of agent videos and costs about $100 per submission, so it is planned only at competition end.A public leaderboard supplies interim comparisons; organizers may restrict final evaluation to top-ranked submissions if evaluating every entry is prohibitively expensive.Source: §2.1.2 · p. 12

2.1.3 Preventing cheating

• Organizers validate leading entries by inspecting and rerunning training code, then checking whether retrained agents behave similarly to submitted agents.Winners are determined by submitted-agent scores; retraining is for detecting rule violations, not rescoring.Source: §2.1.3 · p. 12

2.2 Rules

• Participants may use any approach, but cannot extract additional Minecraft simulator information beyond what the Gym environment provides.The rules also encourage external resources and feedback, require adherence to compute and human-feedback limits, and require attribution and open-sourcing.Source: §2.2 · p. 13

2.3 Schedule

• The planned schedule runs from competition launch on Jul 7 to submissions closing Oct 15, evaluation completion Oct 29, validation completion Nov 12, and presentation Dec 13.The presentation is scheduled for NeurIPS 2021.Source: §2.3 · p. 14

2.4 Competition promotion

• Promotion combines MineRL channels, outreach to affinity groups and technical mailing lists, and planned media and social-media coverage.Organizers also coordinated promotion with MineRL Diamond and planned outreach to groups including WiML, LXAI, BAI, and Queer in AI.Source: §2.4 · pp. 14–15

3.1.1 Organizers

• The organizer roster includes researchers and competition contributors, including Rohin Shah, whose research focuses on preference learning and value alignment, and organizers with prior MineRL competition experience.The roster also includes organizers whose biographies describe work on underrepresented-group participation, explainable reinforcement learning, and learning from human feedback.Source: §3.1.1 · pp. 15–17

3.1.2 Advisors

• The listed advisors include Sergio Guadarrama, a TensorFlow and TF-Agents developer, and Katja Hofmann, whose research focuses on reinforcement learning in games and Project Malmo.Hofmann’s stated long-term goal is AI systems that collaborate with people.Source: §3.1.2 · pp. 18–19

3.1.3 Partners and sponsors

• The organizers report AIcrowd collaboration and sponsorship from Microsoft and OpenAI, while noting discussions with other potential partners.One additional potential partner had provided a soft commitment.Source: §3.1.3 · p. 19

3.2 Resources provided by organizers

• Organizer-provided resources include a Discord community, hoped-for compute grants for teams lacking resources, and at least $12,000 in prizes.Compute grants remained dependent on ongoing sponsor discussions.Source: §3.2 · p. 19

3.3 Support requested

• The organizers request no support beyond a time slot to present the competition results.Source: §3.3 · p. 19