PaperlayerView exact revision on arXiv

Source-linked AI summary

Playing hard exploration games by watching YouTube

Yusuf Aytar, Tobias Pfaff, David Budden, Tom Le Paine, Ziyu Wang, Nando de Freitas

arXiv:1805.11592v2cs.LGcs.AIcs.CVstat.ML

TL;DR

Sparse-reward exploration remains difficult, while standard imitation learning typically requires controlled demonstrations with aligned observations and action and reward trajectories. The paper learns a domain-invariant representation from noisy, unaligned audio-visual videos using temporal and cross-modal self-supervision, then derives an imitation reward from one YouTube video. The resulting agent convincingly exceeds human-level performance on three hard-exploration Atari games, even without environment rewards.

  • Problem

    Hard-exploration reinforcement-learning tasks have sparse rewards, and conventional demonstrations require controlled observation conditions plus action and reward trajectories.

  • Method

    The method learns a common representation from unaligned videos using temporal and cross-modal self-supervision, then embeds one YouTube video to construct an imitation reward.

  • Results

    The agent convincingly exceeds human-level performance on MONTEZUMA’S REVENGE, PITFALL! and PRIVATE EYE, even without an environment reward signal.

  • Takeaways & Limitations

    One-shot imitation from noisy, unaligned YouTube footage can guide agents through hard-exploration games without demonstrator action trajectories or environment rewards.

  • Takeaways & Limitations

    The demonstrations remain constrained by domain variation and use only keypoint-based affine preprocessing to align game screens from the first frame.

Abstract

from arXiv · show

Deep reinforcement learning methods traditionally struggle with tasks where environment rewards are particularly sparse. One successful method of guiding exploration in these domains is to imitate trajectories provided by a human demonstrator. However, these demonstrations are typically collected under artificial conditions, i.e. with access to the agent's exact environment setup and the demonstrator's action and reward trajectories. Here we propose a two-stage method that overcomes these limitations by relying on noisy, unaligned footage without access to such data. First, we learn to map unaligned videos from multiple sources to a common representation using self-supervised objectives constructed over both time and modality (i.e. vision and sound). Second, we embed a single YouTube video in this representation to construct a reward function that encourages an agent to imitate human gameplay. This method of one-shot imitation allows our agent to convincingly exceed human-level performance on the infamously hard exploration games Montezuma's Revenge, Pitfall! and Private Eye for the first time, even if the agent is not presented with any environment rewards.

1 Introduction

Hard-exploration Atari games remain difficult because sparse rewards make naive exploration practically impossible. The paper addresses limitations of conventional imitation learning by learning from unaligned YouTube demonstrations without action or reward trajectories.

  • The hard-exploration problem: Sparse rewards make hard-exploration games such as MONTEZUMA’S REVENGE difficult for existing reinforcement-learning agents.Naive ϵ-greedy exploration is practically impossible because possible action trajectories grow exponentially between rewards.
  • Limitations of existing imitation: Conventional imitation methods assume no domain gap between agent and demonstrator observations and access to exact action and reward sequences.These assumptions typically require demonstrations collected under artificial conditions with a specialized training stack.
  • Proposed approach: The proposed method learns a common representation across demonstrations using self-supervised temporal and cross-modal classification without frame-by-frame alignment or annotations.The paper also introduces cycle-consistency as an unsupervised measure of embedding quality.
  • Proposed approach: A single YouTube video supplies an imitation reward that guides an agent without requiring the demonstrator’s action trajectory.The reward is constructed using the learned embedding.
  • Results: The agent convincingly exceeds human-level performance on MONTEZUMA’S REVENGE, PITFALL! and PRIVATE EYE, even without environment rewards.This is presented as the first such result on these three hard-exploration games.

2 Related Work

Prior imitation methods generally rely on controlled demonstrations, while domain-alignment methods address simpler or differently constrained settings. This work targets noisy, unaligned YouTube gameplay and combines cross-domain representation learning with imitation from missing action and reward sequences.

  • Imitation learning: Existing imitation methods such as DQfD use demonstrations collected without domain gaps and with full action and reward visibility.These methods have shown promising results in sparse-reward game-playing and robotics.
  • Domain alignment: Frame-aligned domain methods such as CCA, DCTW and TCN are unsuitable when different demonstrations produce nonaligned observation sequences.YouTube Atari videos can contain substantially different sequences because demonstrators take different actions.
  • Domain alignment: Shared auxiliary objectives have aligned substantially different domains, while domain-adaptive meta-learning has addressed robotic domain gaps using first-person and third-person demonstrations.The paper distinguishes its setting from these approaches through unaligned YouTube gameplay and missing action or reward trajectories.
  • Self-supervised representation learning: Single-view TCN avoids paired training data, but this work uses temporal classification instead of triplet-based ranking.The paper states that this removes the need to tune local-neighborhood and ranking-margin hyperparameters.
  • Imitation component: The imitation component learns a reward function explaining demonstrator behavior, relating the method to inverse reinforcement learning and few-shot imitation.Its setting is more complex because of the domain gap and absent demonstrator action and reward sequences.

3 Closing the domain gap

The paper learns an embedding for noisy, unaligned demonstrations by combining temporal and audio-visual self-supervision. It evaluates embedding quality with cycle-consistency and uses the representation to guide imitation from YouTube gameplay.

  • Representation learning: The method learns a common representation across demonstrations to generalize to agent observations without explicit exposure to the Atari environment.Self-supervision is used because the demonstrations are not pre-aligned.
  • Temporal distance classification: Temporal distance classification predicts the time interval between two frames, encouraging features that capture visual transformations and environment dynamics.The method uses six temporal-distance categories ranging from [0] to [21 −200].
  • Cross-modal temporal distance classification: Audio contributes self-supervised information because Atari sound corresponds to salient events such as jumping, obtaining items and collecting points.Correlating audio and visual observations is intended to emphasize important game events in the visual abstraction.
  • Cross-modal temporal distance classification: Cross-modal temporal distance classification predicts the temporal distance between a video frame and an audio snippet.The final embedding minimizes L = Ltdc + λLcmc, combining temporal and cross-modal losses.
  • Model selection through cycle-consistency: Cycle-consistency evaluates whether nearest-neighbor matches across embedding sequences return to the original point along two cycle paths.The measure uses Euclidean distance in the embedding space and a third sequence U.

4 One-shot imitation from YouTube footage

The method turns a single YouTube gameplay video into an imitation reward by placing checkpoints along its learned embedding trajectory. It evaluates the embedding through cycle-consistency and enforces soft-order checkpoint visitation.

  • One-shot imitation from YouTube footage: A single YouTube video is embedded, and checkpoints placed along its trajectory provide a reward signal for a standard reinforcement learning agent.Checkpoints are generated every N = 16 frames along the embedded trajectory.
  • Reward construction: The reward requires checkpoints to be visited in soft order, allowing the next checkpoint within a window after the last collected checkpoint.The experiments set ∆t = 1 and α = 0.5, except for pixel-only embeddings, where α = 0.92 performed best.
  • Embedding evaluation: Combining temporal distance classification and cross-modal classification produces the best 2- and 3-cycle-consistency, especially in deeper embedding layers.The combined method avoids performance loss using FC1 or FC2.

5 Implementation Details

The implementation uses separate visual and audio embedding networks, shared-form architecture classifiers for temporal and cross-modal tasks, sampled temporal-distance training pairs, and an IMPALA-based training setup.

  • Embedding networks: The visual encoder processes four cropped RGB frames through convolutional and residual layers, then produces an l2-normalized embedding.Its input is a 128x128x3x4 tensor formed from randomly cropped 140x140 images.
  • Embedding networks: The audio encoder uses 1D convolutions over STFT-derived frequency inputs and outputs an l2-normalized embedding.The input is a width-137 sample with 256 frequency channels, and the network uses channels (32, 64, 128, 256).
  • Self-supervised classifiers: Temporal and cross-modal classifiers use the same shallow architecture but maintain separate weights because they solve different problems.Element-wise multiplication combines the input vectors before a 2-layer MLP with widths (1024, 6).
  • Training procedure: Training pairs are sampled from demonstration sequences by selecting a temporal-distance interval, a distance within it, and two frames separated by that distance.The demonstrations come from three training videos, with intervals [0],,, [3 −4], [5 −20], and [21 −200].
  • Training procedure: The model is optimized with Adam at learning rate 10^-4, batch size 32, for 200,000 steps.These are the stated optimization settings.
  • Reinforcement learning setup: The imitation agent uses IMPALA with 100 actors and adds imitation rewards to optional environment rewards.The network modification computes distances between the agent and the next two checkpoints.

6 Analysis and Experiments

Experiments evaluate embeddings for aligning unaligned gameplay sequences and guiding agents through three hard-exploration Atari games using YouTube-derived imitation rewards.

  • Experimental setup: Four YouTube videos per game—three for training and one for testing—were used across Montezuma’s Revenge, Pitfall! and Private Eye.Videos lasted 3–10 minutes, and none were collected using the specific Arcade Learning Environment.
  • Embedding space evaluation: The combined TDC and CMC embedding provides the best cycle-consistency results, outperforming pixel distance, TCN and L3-Net.TDC and CMC individually improve over TCN and L3-Net, especially at deeper abstraction levels.
  • Embedding space evaluation: TDC and TDC+CMC produce globally consistent trajectories, whereas pixel-space alignment fails and purely cross-modal embeddings remain scattered.The combined temporal and cross-modal objectives yield the strongest alignment and trajectory continuity.
  • Embedding space evaluation: The learned representation emphasizes players, enemies, inventory states and key items while largely ignoring distractors and domain-specific artifacts.Audio in CMC places additional emphasis on important objects and their inventory locations.
  • Hard-exploration game performance: The combined TDC+CMC imitation reward achieves the best game results, including when no environment reward is provided.Without imitation reward, the pure RL agent collects no sparse rewards in Montezuma’s Revenge and Pitfall!, while pixel-space imitation reaches 17k points only in Private Eye.
  • Hard-exploration game performance: Across expert trajectories, all three agents clear Montezuma’s Revenge’s first level, and domain shift is reported as not a significant performance factor.Expert 1 achieves the highest score despite having the biggest domain shift.
  • Hard-exploration game performance: The method is reported as the first to exceed human-level performance on all three games, solve the first level of Montezuma’s Revenge and Private Eye, and substantially outperform state of the art on Pitfall!.Evaluation uses the mean over 200 episodes with random 1-to-30 no-op initialization.

7 Conclusion

The paper guides exploration by learning a domain-invariant representation from unaligned YouTube videos and embedding checkpoints from one video as imitation rewards. Combined with IMPALA, the approach reaches human-level performance in three difficult exploration games.

  • Method: The method learns a domain-invariant representation across noisy, unaligned audio-visual demonstrations using self-supervised temporal and cross-modal objectives.It then uses embedded checkpoints from a single YouTube video to guide exploration.
  • Evaluation: Figure 8 compares TDC+CMC with and without environment reward against pixel-space imitation and IMPALA without demonstrations.The curves are interpreted relative to previously published maximum reward and average-human score.
  • Conclusion: The agent is reported to achieve the first human-level performance on Montezuma’s Revenge, Pitfall! and Private Eye.Table 1 compares the best policy, averaged over 200 evaluation episodes, with previously published results.

8 Supplementary Material

Supplementary experiments examine YouTube-video preprocessing, learned embedding alignment, feature-space policy learning, and the agent’s gameplay behavior. The combined temporal and cross-modal embedding produces aligned, continuous trajectories, while conditioning on the next checkpoint speeds learning.

  • Video data and preprocessing: Four visually diverse YouTube gameplay videos were selected for each Atari game, with an additional out-of-distribution validation video for Montezuma’s Revenge.Training videos were limited to 9000 frames, while the full durations were used to create checkpoints.
  • Gameplay behavior: The agent learns to optimize jump timing over an enemy while continuing to follow the expert trajectory in Montezuma’s Revenge.The demonstration illustrates how the agent interprets expert gameplay footage.
  • Learning from features: Replacing IMPALA’s convolutional layers with the frozen embedding reduces trainable parameters by about 60%.The resulting policy uses two linear layers of width 256, taking a 1024-dimensional embedding as input; imitation-reward generation remains unchanged.
  • Learning from features: Feature-space learning underperforms the full method but still significantly outperforms the state-of-the-art.Conditioning on the next checkpoint’s embedding provides a richer similarity signal and significantly speeds learning.
  • Visualization of the learnt embedding space: The combined temporal and cross-modal objectives yield very good alignment and continuity across trajectories traversing Montezuma’s Revenge’s first room.Purely cross-modal embeddings produce scattered, disjoint trajectories, while TCN reduces the state space to a single curve despite the trajectory’s return path.
  • Visualization of the learnt embedding space: Small timing differences between mostly shared expert paths appear as divergences in the combined embedding space.An animation temporally aligns four expert videos, revealing these trajectory differences.
Loading 1805.11592v2…