PaperlayerView exact revision on arXiv

Source-linked AI summary

Skill Reinforcement Learning and Planning for Open-World Long-Horizon Tasks

Haoqi Yuan, Chi Zhang, Hongcheng Wang, Feiyang Xie, Penglin Cai, Hao Dong, Zongqing Lu

arXiv:2303.16563v2cs.LGcs.AI

TL;DR

Learning long-horizon tasks in open-world environments with RL is extremely inefficient without human demonstrations. Plan4MC trains fine-grained skills, including a Finding-skill, and plans over an LLM-informed skill graph; it reports completing 40 diverse Minecraft tasks and outperforming demonstration-free RL methods in sample efficiency.

  • Problem

    Without human demonstrations, reinforcement learning is extremely sample-inefficient for long-horizon tasks in large open-world environments.

    Source: §1 · p. 1

  • Method

    Plan4MC trains fine-grained skills with RL and intrinsic rewards, uses a Finding-skill to improve initialization, and plans over an LLM-informed skill graph.

    Source: §1 · pp. 2–3

  • Results

    Plan4MC accomplishes all 40 diverse Minecraft tasks and is more sample-efficient than existing demonstration-free RL methods on Minecraft Tech Tree tasks.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Experiments on 40 challenging Minecraft tasks verify Plan4MC’s advantages over various baselines for diverse long-horizon open-world tasks.

    Source: §7 · p. 9

    The Finding-skill is not goal-aware during exploration, making its goal-reaching policy sub-optimal.

    Source: §7 · p. 9

Abstract

from arXiv · show

We study building multi-task agents in open-world environments. Without human demonstrations, learning to accomplish long-horizon tasks in a large open-world environment with reinforcement learning (RL) is extremely inefficient. To tackle this challenge, we convert the multi-task learning problem into learning basic skills and planning over the skills. Using the popular open-world game Minecraft as the testbed, we propose three types of fine-grained basic skills, and use RL with intrinsic rewards to acquire skills. A novel Finding-skill that performs exploration to find diverse items provides better initialization for other skills, improving the sample efficiency for skill learning. In skill planning, we leverage the prior knowledge in Large Language Models to find the relationships between skills and build a skill graph. When the agent is solving a task, our skill search algorithm walks on the skill graph and generates the proper skill plans for the agent. In experiments, our method accomplishes 40 diverse Minecraft tasks, where many tasks require sequentially executing for more than 10 skills. Our method outperforms baselines by a large margin and is the most sample-efficient demonstration-free RL method to solve Minecraft Tech Tree tasks. The project's website and code can be found at https://sites.google.com/view/plan4mc.

1 Introduction

Plan4MC addresses the difficulty of learning long-horizon Minecraft tasks by training fine-grained skills and planning over them. It uses a Finding-skill to improve skill-learning initialization and a skill graph for planning; experiments report success across 40 diverse tasks and advantages over demonstration-free RL baselines.

1 Introduction

  • Minecraft’s vast world, distant resources, partial visual observations, and many sub-goals make open-world long-horizon RL difficult.Mining cobblestone can require more than 10 sub-goals and thousands of environmental steps.Source: §1 · pp. 1–2
  • A Finding-skill explores for items and provides better initialization for other skills, improving their sample efficiency.It is implemented as a hierarchical policy that maximizes the area traversed by the agent.Source: §1 · p. 2
  • An LLM generates skill relationships for a pre-built dependency graph, while an interactive search algorithm plans executable skill sequences for each task.The LLM is used before task execution rather than to generate the online plan.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • Plan4MC accomplishes all tasks in the Minecraft Tech Tree and is much more sample-efficient than existing demonstration-free RL methods.Tasks require planning and execution for 2~30 basic skills and take thousands of environmental steps.Source: §1 · pp. 2–3
  • The framework decomposes long-horizon tasks into sequences of fine-grained skills, whose policies are trained with RL and intrinsic rewards.The skills include Finding-skills, Manipulation-skills, and Crafting-skills.Source: §1 · pp. 2–3, Figure 1 · p. 2

2 Preliminaries

The preliminaries formulate Minecraft tasks as partially observed problems and describe a hierarchical agent that plans over reusable skills. They also contrast demonstration-based skill learning with RL’s difficulty and motivate fine-grained skills plus Finding-skill initialization.

2.1 Problem Formulation

  • A task is a goal paired with initial conditions; the agent receives partial observations and sparse completion rewards in a POMDP.The initial condition specifies starting tools and conditions, while the goal is the target entity to acquire.Source: §2.1 · p. 3
  • The hierarchical agent trains low-level policies for individual skills and uses a high-level planner to convert each task into a skill sequence.Skills can be reused across complex tasks, supporting memory and learning efficiency.Source: §2.1 · p. 3

2.2 Skills in Minecraft

  • Without demonstrations, RL struggles to learn Minecraft skills, while imitation learning commonly relies on gameplay demonstrations or large labeled video datasets.PPO with sparse rewards fails on “milk a cow” and “shear a sheep” even when target mobs are within 10 blocks.Source: §2.2 · p. 3
  • RL performs comparably to imitation learning when task-relevant entities start nearby, motivating fine-grained skills and a Finding-skill that locates targets before task-specific harvesting.The example decomposes “milk a cow” into “find a cow” and “harvest milk_bucket.”Source: §2.2 · p. 4

3 Learning Basic Skills with Reinforcement Learning

The section decomposes Minecraft tasks into three fine-grained skill types and trains them with reinforcement learning. A hierarchical Finding-skill explores for targets, while Manipulation- and Crafting-skills handle obtaining and crafting items.

3 Learning Basic Skills with Reinforcement Learning

  • The three basic skill types are Finding-skills, which locate targets; Manipulation-skills, which obtain materials; and Crafting-skills, which craft advanced items or tools.Together, these skills are intended to compose Minecraft tasks.Source: §3 · p. 4

3.1 Learning to Find with a Hierarchical Policy

  • A target-free hierarchical policy addresses the long-horizon challenge of finding items by exploring the world’s surface for diverse targets.The high-level policy chooses goals from historical locations and is rewarded for reaching new 10 × 10 grid areas; the low-level policy is rewarded for reducing distance to its goal.Source: §3.1 · p. 4
  • The low-level goal-reaching policy is pretrained with randomly generated goals using DQN, then held fixed while PPO trains the high-level policy.At test time, exploration continues until lidar detects the target, after which the low-level policy approaches it.Source: §3.1 · p. 4

3.2 Manipulation and Crafting

  • Finding-skills can initialize Manipulation-skills by bringing required targets nearby, either during training or by reaching them with the Finding-skill.The latter approach resembles Go-Explore and is suitable for environments without commands to initialize targets nearby.Source: §3.2 · p. 5
  • Manipulation-skills use MineCLIP similarity rewards and PPO with self-imitation learning, while Crafting-skills require only one MineDojo action.Training maximizes a weighted sum of intrinsic rewards and sparse extrinsic success reward.Source: §3.2 · p. 5

4 Solving Minecraft Tasks via Skill Planning

Plan4MC uses an LLM-generated skill dependency graph and searches it to construct task plans. The search checks skill prerequisites and effects against the agent’s current items so the resulting sequence is executable and target-reaching.

4 Solving Minecraft Tasks via Skill Planning

  • The skill graph is generated in advance with an LLM, enabling the agent to search for skill sequences during task execution.The graph structures skill relationships, including item consumption and requirements.Source: §4 · p. 5, §4.1 · pp. 5–6

4.1 Constructing Skill Graph with Large Language Models

  • The LLM generates structured information for skills, representing required conditions, consumed items, and obtained items as relationships in a directed graph.The graph contains hundreds of nodes; GPT-3.5 is prompted with demonstrations and explanations of the format.Source: §4.1 · pp. 5–6

4.2 Skill Search Algorithm

  • A depth-first search starts at the target and traverses toward parent skills, maintaining the available items and appending skills once their conditions are met or they have no parents.After each skill execution, the agent updates its condition from inventory and the last skill, then searches again; this accommodates possible skill failures.Source: §4.2 · p. 6

5 Experiments

Across four Minecraft task sets, Plan4MC outperforms the reported baselines, while Finding-skills improve performance on difficult tasks and interactive replanning supports long-horizon execution. The evaluation covers 40 tasks with varied planning demands and also tests skill-learning and planning variants.

5 Experiments

  • The experiments evaluate basic-skill implementation, a task suite, and skill-learning and planning results against baselines and ablations.Basic skills are pretrained in MineDojo programmatic tasks; Manipulation-skills use targets initialized close to the agent for simplicity.Source: §5 · p. 6, §5.1 · p. 6

5.2 Task Setup

  • The benchmark contains 40 tasks across four groups, requiring an average of 25 planned skills and up to 121; task episodes range from 3,000 to 12,000 maximum steps.Success rates are averaged over 30 test episodes per task.Source: §5.2 · pp. 6–7

5.3 Skill Learning

  • Plan4MC’s skill-learning comparison uses MineAgent and a variant that skips Finding-skills, while the planning comparison includes interactive LLM, zero-shot, and half-length episode variants.MineAgent receives 7M training steps, equal to the total used to train Plan4MC’s basic skills.Source: §5.3 · p. 7, §5.4 · p. 8
  • Plan4MC outperforms both baselines across the four task sets; MineAgent fails on Cut-Trees, Mine-Stones, and Mine-Ores, while Plan4MC without Finding-skills underperforms Plan4MC across tasks.MineAgent is comparable on some easier Interact-Mobs tasks; the gap on Mine-Ores is small because that task set does not use Finding-skills.Source: §5.3 · p. 7, Table 2 · p. 7
  • With Finding-skills, conditional Manipulation-skill success is 0.40 versus 0.25 without them, and the largest finding-step drops occur for cobblestones and logs.These items are harder to find than mobs in the tested environment.Source: §5.3 · p. 7, Figure 3 · p. 8
  • Most Manipulation-skills have slightly lower test than training success rates because of the domain gap, while this decrease does not occur when items or mobs are initialized at larger distances.Finding-skills can provide better test initialization by reaching targets within three units.Source: §5.3 · pp. 7–8, Table 3 · p. 8

5.4 Skill Planning

  • Interactive LLM planning is comparable on Interact-Mobs, but performs worse on long-horizon Mine-Stones and Mine-Ores; zero-shot planning performs much worse across all task sets.Halving episode length causes the least performance decrease relative to Plan4MC.Source: §5.4 · p. 9

6 Related Work

Prior Minecraft research spans hierarchical reinforcement learning, imitation learning, multi-task learning, skill discovery, LLM planning, and video pretraining. This work positions itself as reinforcement learning and planning for Minecraft, using fine-grained skills and a skill graph with search.

Minecraft

  • Minecraft research has used benchmark environments and datasets, studied ObtainDiamond with hierarchical RL and imitation learning, and explored multi-task learning, skill discovery, LLM planning, and video pretraining.The paper places its contribution within reinforcement learning and planning in Minecraft.Source: §6 · p. 9

Learning Skills in Minecraft

  • Prior skill-learning approaches include imitation learning and VPT-based pretraining, while demonstration-free MineAgent and CLIP4MC use RL but acquire only a small skill set.Unsupervised skill discovery learns navigation behaviors; this work instead learns fine-grained basic skills with intrinsic rewards.Source: §6 · p. 9

Planning with Large Language Models

  • LLM planning methods include skill affordances, demonstration selection, interaction feedback, and plan-generation components; this work uses an LLM-generated skill graph with search to eliminate planning mistakes.Source: §6 · p. 9

7 Conclusion and Discussion

Plan4MC combines learned skills with graph-based planning to address long-horizon Minecraft tasks, while its Finding-skill and skill-search process are demonstrated on exploration and crafting tasks. The authors identify goal-unaware Finding-skill exploration as a limitation and describe the approach as extendable to similar open-world environments.

7 Conclusion and Discussion

  • Plan4MC’s framework combines fine-grained RL skills, a general Finding-skill for skill-learning initialization, and graph-based planning; experiments cover 40 challenging Minecraft tasks.The planner uses LLM prior knowledge to build a skill graph and skill search to generate plans.Source: §7 · p. 9
  • The authors state that Plan4MC, implemented in Minecraft, is extendable to similar open-world environments.Source: §7 · p. 9
  • The Finding-skill does not know its goal during exploration, making its goal-reaching policy suboptimal; correcting outputs when the LLM lacks domain knowledge remains open.Source: §7 · p. 9

A The Necessity of Learning the Finding-skill

B Pipeline Demonstration

C Algorithms

  • The algorithm sketches show skill search recursively expanding a target’s parent skills while tracking item quantities and building a skill sequence.Source: C Algorithms · p. 15

D Details in Training Basic Skills

This section details the intrinsic rewards and training configurations used for basic skills, then describes how ChatGPT-generated skill information supports graph construction. The generated skills are checked against the Minecraft database before use.

D Details in Training Basic Skills

  • MineCLIP rewards encourage prompt-matching behavior, while distance, attack, and depth rewards supplement them for combat, mining, and underground tasks.The added rewards encourage chasing and attacking mobs, staying close to target blocks, and mining deeper before returning to the ground.Source: D Details in Training Basic Skills · p. 16
  • Finding-skill intrinsic rewards combine visited-area state counts with goal navigation that rewards movement toward a sampled goal and looking in its direction.The high-level policy tracks visited grids; the low-level policy is encouraged to reach the goal position.Source: D Details in Training Basic Skills · p. 16
  • For combat, valid centered attacks receive a larger reward than valid off-center attacks; mining distance rewards are modified to encourage staying close to blocks.The attack reward values are 90 for a valid centered attack and 1 for a valid attack with the target off-center.Source: D Details in Training Basic Skills · p. 16
  • The training tables list skill configurations and selected policy information; Finding- and Mining-skill models are selected by smoothed return, while other skills use smoothed success rate.Table 5 defines Max Steps, Training Steps, and initialization distance; Table 6 reports selected policies’ success rates.Source: Table 5 · p. 17, Table 6 · p. 17, D Details in Training Basic Skills · p. 17

E LLM Prompt Design

  • ChatGPT generates skill relationships from demonstrated formats, and the authors corrected 6 mistakes among 55 generated skills using the Minecraft database.The examples distinguish consumed items, required items, equipment, and obtained items.Source: E LLM Prompt Design · pp. 18, 20

F Task Setup

Evaluation tasks use randomized starts and, for mob-interaction tasks, randomized mob placements; the Mine-Ores set also uses a modified breaking-speed setting.

F Task Setup

  • At each test reset, the agent is randomly transported by up to 500 units to ensure an unseen environment.Source: F Task Setup · p. 20
  • For tasks involving mobs, cows and sheep spawn at distances of up to 30, farther than in basic-skill training.Source: F Task Setup · p. 20
  • Mine-Ores tasks use a breaking-speed multiplier of 10; the other three task sets use the default simulator.Source: F Task Setup · p. 20

G Experiment Results for All the Tasks

Table 9 reports success rates for all methods across all evaluation tasks, grouped into four task sets.

G Experiment Results for All the Tasks

H Training Manipulation-skills without Nearby Items

Plan4MC-go-explore retrains Manipulation-skills without target items initialized nearby, using a pre-trained Finding-skill to locate them. The resulting skills retain comparable training success and generalize well to unseen environments.

H Training Manipulation-skills without Nearby Items

  • A pre-trained Finding-skill explores for the target item before the policy collects data for RL training without nearby item initialization.The previous and new approaches are denoted Plan4MC and Plan4MC-go-explore, respectively.Source: H Training Manipulation-skills without… · pp. 20–21
  • All Go-Explore-trained skills avoid failure and achieve success rates comparable to the previous skills across 100 training epochs.Plan4MC-go-explore outperforms Plan4MC for Milk and Wool, where the agent can start closer to target mobs.Source: H Training Manipulation-skills without… · p. 21
  • Plan4MC-go-explore outperforms Plan4MC on three of four task sets when Manipulation-skills are trained without nearby target items.The results support generalization of the Go-Explore-trained skills to unseen environments.Source: H Training Manipulation-skills without… · p. 21
  • Plan4MC-go-explore has a small generalization gap for the first four mob-related skills, while Harvest Log is harder to train because it often finds previously cut trees.For Harvest Log, test success exceeds training success.Source: H Training Manipulation-skills without… · p. 21

I Discussion on the Generalization of Plan4MC

Plan4MC combines RL skill acquisition with LLM-assisted skill planning, and its skill decomposition is proposed as applicable beyond Minecraft. Experiments report sample-efficiency gains from object-based intrinsic rewards, while graph-based planning incorporates LLM knowledge to improve planning accuracy.

I Discussion on the Generalization of Plan4MC

  • Plan4MC combines LLM-assisted planning and RL skill acquisition, a pipeline described as applicable to other open-world domains.The cited examples include fine-grained navigation, arm positioning, and object manipulation skills for indoor robotics.Source: I Discussion on the Generalization of… · p. 22
  • Object-based intrinsic rewards improve sample efficiency in Minecraft skill learning; both MineCLIP reward and distance reward positively affect Harvest Milk training.The paper identifies vision-language models, object detectors, and distance estimation as possible bases for reward design.Source: I Discussion on the Generalization of… · p. 22, Figure 7 · p. 24
  • The paper presents graph-based planning that incorporates LLM knowledge as an extension of LLM-based planners, improving planning accuracy.The approach is described as extendable to settings where agent state can be abstracted by text or entities.Source: I Discussion on the Generalization of… · p. 22
Loading 2303.16563v2…