Discovering Hierarchical Achievements in Reinforcement Learning via Contrastive Learning

1 Introduction

• Hierarchical-achievement discovery in procedurally generated environments requires generalization and long-term reasoning because achievements are reusable subtasks with dependencies.The environment provides sparse rewards for novel achievements, each of which can contribute to completing more complex achievements.Source: §1 · p. 1
• Prior model-based and hierarchical approaches can require large models, substantial exploratory or expert data, or advance knowledge of the achievement set.These requirements limit practicality in open-world scenarios where the number of achievements cannot be predetermined.Source: §1 · p. 2
• PPO, optimized with recent implementation practices, outperforms prior model-based and hierarchical methods and shows uncertain ability to predict the next achievement.Achievement distillation builds on this predictive ability to strengthen next-achievement prediction.Source: §1 · p. 2
• PPO with achievement distillation achieves state-of-the-art Crafter performance, unlocking all 22 achievements in 1M environment steps with 4% of the previous state-of-the-art model’s parameters.The approach distills achievement information from policy-update episodes into the encoder using contrastive learning.Source: §1 · p. 2

2.1 Markov decision processes with hierarchical achievements

• An achievement graph is a directed acyclic graph whose edges encode dependencies; agents receive reward for a newly unlocked achievement and must infer unlocked achievements without prior graph knowledge.The objective is a generalizable policy that maximizes expected return across environments.Source: §2.1 · p. 2

2.2 Crafter environment

• Crafter is a partially observable, procedurally generated survival game with 22 prerequisite-linked achievements, including crafting tools and collecting diamonds.World layouts, terrain, resources, and enemy locations vary by integer seed.Source: §2.2 · p. 3

2.3 Proximal policy optimization

• PPO collects episodes, updates policy and value networks, and shares an image encoder whose representations can also be trained with auxiliary objectives.Policy and value optimization alone may yield suboptimal encoder representations in procedurally generated environments.Source: §2.3 · p. 3

3.1 PPO is a strong baseline for hierarchical achievements

• PPO is considered less often than model-based or hierarchical methods because it lacks an explicit long-term planning component.The section tests PPO without pretraining, using recent implementation practices.Source: §3.1 · p. 4
• After 1M environment steps training modified PPO on Crafter, the achievement-unlocking score rose from 8.17 to 15.60, surpassing DreamerV3’s score of 14.77.Source: §3.1 · p. 4

3.2 Representation analysis of PPO

• 44.9% accuracy in 22-way classification shows PPO representations contain next-achievement information, but prediction confidence is low, with a 0.240 median.The analysis trains a linear classifier on frozen encoder representations and evaluates it on held-out states.Source: §3.2 · p. 4, Figure 3 · p. 4
• The low-confidence predictions motivate alternative, unsupervised guidance because the agent does not have access to achievement labels.The section proposes exploring methods to guide next-achievement prediction rather than supervised learning.Source: §3.2 · p. 4

4 Contrastive learning for achievement distillation

• Achievement distillation combines within-episode prediction and cross-episode achievement matching as self-supervised contrastive objectives alongside reinforcement learning.It transfers achievement information from episodes collected during policy updates into the encoder.Source: §4 · p. 5

4.1 Intra-trajectory achievement prediction

• Within an episode, the next achievement is the positive for a state-action pair, while another pair from that episode is the negative; policy and value regularizers preserve network outputs.The achievement representation is formed from the residual between consecutive state representations, while the state-action representation combines state and action information.Source: §4.1 · pp. 5–6

4.2 Cross-trajectory achievement matching

• Across episodes, partial optimal transport soft-matches achievement sequences, then thresholding at 0.5 yields conservative matches for contrastive learning.The method uses hard matching to exclude achievements without a counterpart and encourages each achievement to match at most one other achievement.Source: §4.2 · p. 6

4.3 Achievement representation as memory

• The previous achievement is concatenated with the latent state for policy and value inputs, and with state-action information for next-achievement prediction.The latter representation is interpreted as learning forward dynamics in achievement space.Source: §4.3 · pp. 6–7

4.4 Integration with RL training

• Training alternates PPO policy phases that collect episodes into a buffer with auxiliary phases that optimize contrastive objectives and policy-value regularizers on buffered episodes.The auxiliary phase is called achievement distillation.Source: §4.4 · p. 7

5.1 Experimental setup

• Achievement distillation is evaluated on Crafter for 1M environment steps using 10 runs; success rates cover 22 achievements and are summarized by their geometric mean.Episode reward is averaged over episodes in the most recent 100K environment steps.Source: §5.1 · p. 7

5.2 Crafter results

• 21.79% score and 12.60 reward: achievement distillation outperforms all scratch-trained baselines, with higher score than MuZero + SPR and comparable rewards.The comparison uses Crafter scores and rewards.Source: Table 1 · p. 8, §5.2 · p. 8

5.3 Model size analysis

• With 4% of DreamerV3’s parameters, the method performs better; a 1M-parameter variant still outperforms baselines with a 17.07% score.Most of the method’s parameter increase over PPO comes from auxiliary-learning networks not used during inference.Source: §5.3 · p. 8

5.4 Representation analysis of achievement distillation

• 73.6% classification accuracy—28.7 percentage points above PPO—and a median prediction confidence of 0.752 indicate stronger next-achievement predictions.Source: §5.4 · p. 9

5.5 Ablation studies

• In ablations, intra-trajectory achievement prediction contributes most, while cross-trajectory achievement matching and memory also improve performance.Source: Table 2 · p. 9, §5.5 · p. 9

5.6 Extension to value-based algorithms

• With QR-DQN, achievement distillation raises the Crafter score from 4.14 to 8.07, indicating effectiveness with an off-policy value-based algorithm.Source: §5.6 · p. 9

5.7 Application to other environments

• On Heist and MiniGrid, the method improves PPO scores from 29.6 to 71.0 and from 3.33 to 8.04, respectively.Heist varies wall and background colors; MiniGrid tests a deeper achievement graph.Source: §5.7 · p. 9

• Model-based approaches use world models or future-state prediction, but DreamerV3 exceeds 200M parameters and MuZero + SPR uses 150M pretraining environment steps.Source: §6 · p. 10
• Hierarchical methods include HAL’s next-achievement classifier and SEA’s graph planner, but rely on achievement information or expert data and have limited sample-efficient evaluation.HAL does not scale to high-dimensional Crafter, while SEA has not been tested in a sample-efficient regime.Source: §6 · p. 10
• Model-free achievement discovery has fewer studies; LSTM-SPCNN improves PPO modestly but requires 135M parameters, and representation learning for this problem remains little explored.Prior SPR results on Crafter were not promising.Source: §6 · p. 10
• Prior optimal-transport work mainly matches agent behavior to offline expert data for imitation learning, whereas this work uses it for online generalizable achievement representations.Source: §6 · p. 10

7 Conclusion

• Achievement distillation discovers hierarchical achievement structure without an explicit long-term planning component, with state-of-the-art Crafter performance using fewer parameters and data.The method distills achievement information from episodes collected during policy updates into the encoder and integrates with PPO.Source: §7 · p. 10
• The method uses minimal achievement information: the agent receives a reward when a new achievement is unlocked.The authors have not evaluated transfer to an unsupervised agent without rewards.Source: §7 · p. 10
• The paper identifies reward-free unsupervised achievement recognition, potentially combined with curiosity-driven exploration, as future work.The proposed future direction is a representation-learning method that distinguishes achievements without supervision.Source: §7 · p. 10

A Representation analysis

• The analysis dataset contains 215,578 states collected using an expert policy trained with the method for 1M environment steps.The expert policy used a different seed from the policies used for evaluation; 50,000 states were sampled for training and 10,000 for testing.Source: A Representation analysis · p. 14
• A linear classifier is trained on frozen encoder representations to predict the next achievement, with labels 0 through 21 for Crafter's possible achievements.The classifier is optimized for 500 epochs using Adam with a learning rate of 1e-3.Source: A Representation analysis · p. 14
• The evaluation measures test-set classification accuracy and the prediction probability assigned to the ground-truth achievement.The probability measure represents confidence in the correct label.Source: A Representation analysis · p. 14

B Examples of cross-trajectory achievement matching

• Figure 7 compares cosine distances between achievement representations from two episodes; the same achievements have lower distances with the proposed method than with PPO.The example uses achievement sequences represented by each method and matched across two expert-policy episodes.Source: B Examples of cross-trajectory… · p. 14, Figure 7 · p. 14
• The proposed method matches identical achievements across episodes, whereas PPO's hard matching includes inaccurate pairs and misses an achievement.The example shows PPO matching “Defeat zombie” with “Eat cow” and failing to match “Collect wood”; the proposed method avoids matches for achievements absent from the other episode.Source: B Examples of cross-trajectory… · p. 14, Figure 8 · p. 15, Figure 9 · p. 15
• The appendix evaluates matching on two different episodes, comparing the proposed partial-transport-and-thresholding algorithm with Hungarian bipartite matching.The episode pairs are collected using an expert policy for the matching examples and a method-trained policy for the early-training comparison.Source: B Examples of cross-trajectory… · pp. 14, 16
• The proposed matching algorithm avoids an incorrect “Defeat zombie”–“Collect sapling” match made by the Hungarian algorithm in the example.The compared hard matches are computed from episodes collected using a method-trained policy for up to 100 epochs.Source: B Examples of cross-trajectory… · p. 16, Figure 10 · p. 16

C.1 Computational resources

• The experiments run on cluster nodes with two AMD EPYC 7402 CPUs, 500GB of RAM, and eight NVIDIA RTX 3090 GPUs.PyTorch is the primary deep learning framework.Source: §C.1 · p. 17

C.2 Implementation details and hyperparameters

• PPO encodes 64 × 64 × 3 images with an IMPALA-style ResNet and feeds its latent vector to separate policy and value heads.The flattened encoder output passes through dense layers of sizes 256 and 1024; the policy head outputs 17 actions and the value head a scalar.Source: §C.2 · p. 17, Figure 11 · p. 19
• PPO training uses fewer mini-batches and rollout epochs than the cited Stanić et al. setting, and normalizes value targets with an EWMA.The mini-batches per epoch decrease from 32 to 8, rollout epochs from 4 to 3, and EWMA decay is 0.99; the auxiliary schedule selects 8 policy phases and 6 auxiliary epochs.Source: §C.2 · pp. 17, 19, 20
• Reproduction settings differ: DreamerV3 uses XL size and training ratio 512 [21]; MuZero + SPR results are replicated from its paper [53]; SEA uses original Crafter for 1M steps, unlike its modified-Crafter 500M-step setup [57].Source: §C.2 · pp. 17–18
• Achievement distillation represents an achievement using the difference between encoder latent states and combines state, action, and prior-achievement information for state-action representations.The state representation is combined with action using a FiLM layer; the resulting vector is concatenated with an achievement representation and passed through an MLP.Source: §C.2 · p. 19
• The auxiliary objectives use policy and value regularizer coefficients of 1.0 and partial optimal transport with entropic regularizer coefficient 0.05, set without hyperparameter search.The phase counts were searched over 4, 8, and 16 policy phases and 1, 3, and 6 auxiliary epochs.Source: §C.2 · pp. 19–20

C.3 Environment details

• Crafter observations are agent-centric 64 × 64 × 3 images, and the environment has 17 discrete actions and 22 achievements with specific requirements.Observations include inventory and health, food, water, and energy levels; achievements can depend on resources, tools, and prior achievements.Source: §C.3 · pp. 20–21

D Extension to value-based algorithms

• Achievement contrastive learning significantly improves QR-DQN performance.The method is applied to the Q-network encoder before each target Q-network update.Source: D Extension to value-based algorithms · p. 21
• QR-DQN uses the Rainbow open-source implementation, original hyperparameters, and a ResNet encoder matching the PPO implementation.Source: D Extension to value-based algorithms · p. 21
• Table 10 lists the default QR-DQN hyperparameter values.Source: Table 10 · p. 21

E.1 Procgen Heist

• Procgen Heist requires collecting matching-color keys to open blue, green, and red locks before stealing a gem; these actions define achievements.Opening each lock earns 2 reward, and stealing a gem earns 10 reward.Source: §E.1 · p. 21

E.2 MiniGrid

• Custom MiniGrid has six rooms, 7×7 local observations, and 10 achievements, with a reward of 1 for each newly unlocked achievement.The task requires sequentially unlocking doors, finding keys, and reaching the green square.Source: Figure 14 · p. 22, §E.2 · p. 22

E.1 Procgen Heist

• The method outperforms PPO by a significant margin throughout Procgen Heist training on gem-pilfering success rate and episode reward.Agents train for 25M environment steps in hard difficulty mode.Source: §E.1 · p. 22, Figure 15 · p. 23

E.2 MiniGrid

• In custom MiniGrid, the method outperforms PPO and shows reduced variance across training seeds.Evaluation follows the Crafter protocol after 1M environment steps.Source: §E.2 · p. 22, Figure 16 · p. 23