Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft

• Uniformly sampling tasks can waste computation when few are currently learnable, motivating curricula that infer learnability during training.Source: §1 · p. 2
• A goal-independent exploration bonus for tasks unrelated to the current goal substantially improves performance over uniform sampling.Source: §1 · p. 3
• Dynamically removing bonuses from reliably obtainable items adds across-training diversity pressure and further improves performance.Source: §1 · p. 3
• Learning-progress sampling tracks the expanding skill frontier; tracking performance drops as well as gains avoids forgetting seen with improvement-only sampling.Source: §1 · p. 3
• The bidirectional learning-progress curriculum discovers the most target items, followed by dynamic, fixed, then no exploration bonus; learning progress further increases performance.Source: Figure 1 · p. 3, §1 · p. 3

2.1 Learning progress curriculum

• Learning-progress curricula prioritize tasks with changing success probabilities, tracking both increases and decreases to revisit tasks that may be forgotten.Unlike fixed success-probability thresholds, progress can indicate learnability despite task- or environment-dependent limits on achievable success rates.Source: §2.1 · pp. 5–6

2.2 Learning progress inference

• The estimator contrasts fast and slow success-probability averages; an intermediate measurement timescale balances noise against lag, while low-probability reweighting prioritizes unfinished tasks.The method also concentrates 90% of sampling weight on roughly 20% of tasks with the largest reweighted progress.Source: §2.2 · pp. 6–7

2.3 Curricula over goal-conditioned tasks

• A goal-independent, within-episode bonus rewards newly collected items, while the dynamic version removes reliably obtainable items to limit distraction from harder goals.Items leave the exploration set above a goal-conditioned success probability of 0.1 and can return if performance drops below that threshold.Source: §2.3 · pp. 7–8

2.4 "Simon Says" task

• For goal-conditioned Minecraft tasks, the agent selects from 107 tech-tree items, has 5 minutes (1500 time steps) to complete each, and starts another task without resetting the world or respawning after success or failure.Episodes last at most 30 minutes (9000 time steps); a new episode starts in a new environment, with a 50% chance of retaining the prior inventory.Source: §2.4 · p. 8

3.1 Uniform sampling without exploration bonus

• The agent learned to obtain only 17 of 107 tasks, with learning defined as success probability of at least 5%; learning plateaued early, and it discovered only a subset of surface-obtainable items.The target set also contained six additional items later realized to be impossible for the agent.Source: §3.1 · pp. 8–9

3.2 Uniform sampling with fixed exploration bonus

• The fixed exploration bonus increases explored items at convergence from 17 to 43, extending discovery beyond surface items to underground mining and some iron tools.The bonus is applied over all items, including those already known to the agent, which can distract it from harder goals.Source: §3.2 · p. 9, §3.3 · p. 9

3.3 Uniform sampling with dynamic exploration bonus

• The dynamic exploration bonus raises explored items at convergence to about 70 by rewarding hard items the agent rarely obtains.The bonus is removed for items whose goal-conditioned success probability exceeds 0.1; discovered items include all surface and stone items, most iron items, and most redstone items.Source: §3.3 · p. 9

3.4 Learning progress curriculum

• The bidirectional curriculum discovers 82 items, while the unidirectional curriculum discovers 79 and undergoes unstable cycles of forgetting and rediscovery.The bidirectional treatment reaches half of the golden items and a few diamond items; the unidirectional treatment’s cycles arise from catastrophic forgetting.Source: §3.4 · p. 10

3.5 Mitigating catastrophic forgetting by tracking bidirectional learning progress

• Bidirectional progress resamples tasks when performance declines, and sporadic revisiting of old tasks is reported as sufficient to mitigate forgetting more efficiently than uniform resampling.Unidirectional progress does not increase sampling of declining tasks, allowing extended focus on surface items and loss of underground skills.Source: §3.5 · p. 10

Automated curriculum learning

• Threshold-based methods select tasks by success probability or reward, while regret-based methods favor tasks with high gaps between maximum return and average return [6] [7] [8] [16] [17] [18].The paper notes that static thresholds can be difficult or impossible to tune.Source: §4 · p. 11
• Under environmental stochasticity, regret can favor less-learnable tasks when unusually lucky returns inflate the known maximum [17] [18].The authors present learning-progress curricula as having potential to address this issue.Source: §4 · p. 11

Learning progress and automated curricula

• Learning-progress curricula extend earlier uses of progress as curiosity for novel-state exploration [22] to task selection, and this work studies reliable measurement and combination with dynamic exploration.The authors identify these as their novel contributions alongside applying progress curricula to hard problems requiring RL at scale.Source: §4 · p. 11

Exploration bonuses

• Static within-episode item bonuses encourage obtaining new items, whereas dynamic bonuses encourage obtaining different items across episodes [3].The cited Curiosity Search work also reports the cost of repeatedly exploring the whole environment for deep exploration.Source: §4 · p. 11
• The bidirectional curriculum resamples goals when success declines and restores items to the bonus set when success falls, helping mitigate detachment and reducing derailment risk.The paper says it does not address derailment as explicitly; removing learned items from the bonus set reduces the incentive for exploration when prerequisites are lacking.Source: §4 · p. 11

5 Discussion, Conclusion, and Future Work

• 17 to 43, then 70 and 82 learned tasks as static bonuses, dynamically restricted bonuses, and bidirectional learning-progress sampling are added in sequence.The curriculum’s task sampling follows the agent’s learning curve and focuses on the expanding frontier of its skills.Source: §5 · p. 12
• Bidirectional learning progress mitigates catastrophic forgetting by resampling tasks at risk of being forgotten, unlike the unidirectional curriculum observed to forget tasks.It tracks both task improvements and deteriorations in performance.Source: §5 · p. 12
• The current curriculum assumes explicit task labels, and faithfully estimating progress becomes computationally expensive when task counts grow.Suggested extensions include clustering tasks and estimating progress over large task spaces with function approximation or environment generators.Source: §5 · p. 12

A.1 Optimal ∆t for estimating learning progress:

• The estimator uses samples at t and t−∆t to form a difference quotient for the derivative of mean success probability.It is unbiased for the finite-difference quotient, but not for the derivative itself at finite ∆t.Source: §A.1 · pp. 13–14
• The optimal ∆t balances variance and curvature bias: more samples or greater curvature favor a smaller interval, while greater average variance favors a larger one.The optimum exists only when the mean’s second derivative is nonzero.Source: §A.1 · p. 14
• The expected squared error combines the squared bias and variance, with the bias determined to first order by the mean’s second derivative.The approximation neglects terms of third and higher order in ∆t when ∆t is assumed small.Source: §A.1 · p. 14

A.2 Reweighting of learning progress towards small success probabilities

• For each task, the bidirectional learning-progress measure is the absolute difference between reweighted fast and slow success-probability estimates.The estimates are exponential moving averages with different time scales.Source: §A.2 · p. 15
• Reweighting magnifies probability differences at low success levels: the gap from 0.1 to 0.2 becomes larger than the gap from 0.6 to 0.7.Figure 6 illustrates these two probability intervals with dotted red and blue lines, respectively.Source: §A.2 · p. 15, Figure 6 · p. 15
• The reweighting is applied to probabilities before learning progress is calculated.This is the stated scope of Figure 6.Source: Figure 6 · p. 15, §A.2 · p. 15

A.3 Conversion of reweighted learning progress measure to sampling weight

• Z-scored reweighted learning progress is passed through a sigmoid and the resulting weights are normalized into sampling probabilities.The sigmoid is centered on the 90% quantile of the normal distribution.Source: §A.3 · pp. 15–16
• The sigmoid's saturation for large learning progress prevents sampling from focusing on only one or two tasks.Under Gaussian-distributed reweighted learning progress, the algorithm would focus on the top 10% of tasks.Source: §A.3 · pp. 15–16, Figure 7 · p. 16
• Because reweighted learning progress often deviates from Gaussianity, the curriculum may sample a larger or smaller share than the intended top 10%.Source: §A.3 · p. 16

A.4.1 Environment

• The agent observes 64x64 RGB frames, game-state features, the current goal item, and the exploration-set items.Features include inventory, health, food, saturation, experience, and the items in hand.Source: §A.4.1 · p. 16
• Actions include movement and combat controls, camera control, and special craft, equip, and place actions masked when unavailable in the current game state.Source: §A.4.1 · p. 16
• Most actions repeat for four environment steps, while a channel-attack action repeats attacks for a tool-dependent duration to make mining easier to learn.At 20 frames per second, the four-step frame skip lets the policy choose a new action every 200 milliseconds.Source: §A.4.1 · pp. 16–17

A.4.2 Policy and Optimization details

• The policy and value function share an enlarged IMPALA convolutional network followed by a 512-unit fully connected layer.Source: §A.4.2 · p. 17
• Visual and feature embeddings are summed, processed by an LSTM and another 512-unit layer, then sent to factorial policy and value-function heads.Inventory features are embedded linearly; other features use one-hot encoding.Source: §A.4.2 · p. 17
• Optimization uses Proximal Policy Optimization and General Advantage Estimation.Source: §A.4.2 · p. 17