PaperlayerView exact revision on arXiv

Source-linked AI summary

Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos

Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, Jeff Clune

arXiv:2206.11795v1cs.LGcs.AI

TL;DR

Sequential decision domains have abundant online video but lack the action labels needed for behavioral-prior pretraining. VPT uses a small labeled set to train an inverse dynamics model that labels large-scale Minecraft videos, producing a behavioral prior with zero-shot abilities and harder-task performance after fine-tuning.

  • Problem

    Sequential decision domains contain abundant unlabeled online video but lack the action labels needed to train behavioral priors at internet scale.

  • Method

    VPT trains an inverse dynamics model on a small labeled contractor dataset, uses it to label filtered online Minecraft videos, and behaviorally clones the inferred actions.

  • Results

    VPT shows nontrivial zero-shot behavior and, after behavioral-cloning and reinforcement-learning fine-tuning, achieves over 80% iron-pickaxe reliability, almost 20% diamond-collection reliability, and 2.5% diamond-pickaxe reliability.

  • Takeaways & Limitations

    The results support using freely available unlabeled web video to learn behavioral priors for sequential decision tasks and to improve exploration through reinforcement-learning fine-tuning.

  • Takeaways & Limitations

    The reported experiments are in Minecraft only, and the models condition on past observations without accepting specific task instructions.

Abstract

from arXiv · show

Pretraining on noisy, internet-scale datasets has been heavily studied as a technique for training models with broad, general capabilities for text, images, and other modalities. However, for many sequential decision domains such as robotics, video games, and computer use, publicly available data does not contain the labels required to train behavioral priors in the same way. We extend the internet-scale pretraining paradigm to sequential decision domains through semi-supervised imitation learning wherein agents learn to act by watching online unlabeled videos. Specifically, we show that with a small amount of labeled data we can train an inverse dynamics model accurate enough to label a huge unlabeled source of online data -- here, online videos of people playing Minecraft -- from which we can then train a general behavioral prior. Despite using the native human interface (mouse and keyboard at 20Hz), we show that this behavioral prior has nontrivial zero-shot capabilities and that it can be fine-tuned, with both imitation learning and reinforcement learning, to hard-exploration tasks that are impossible to learn from scratch via reinforcement learning. For many tasks our models exhibit human-level performance, and we are the first to report computer agents that can craft diamond tools, which can take proficient humans upwards of 20 minutes (24,000 environment actions) of gameplay to accomplish.

1 Introduction

VPT addresses the lack of action labels in abundant online video for sequential decision domains by using semi-supervised imitation learning. In Minecraft, the approach enables zero-shot behavior and progressively harder skills through behavioral-cloning and reinforcement-learning fine-tuning.

  • Motivation: Online data for sequential decision domains is abundant but usually consists of unlabeled video, making behavioral-prior training harder than in language or vision.When labeled demonstrations are unavailable, reinforcement learning is often sample inefficient and expensive for hard-exploration problems.
  • Approach: VPT uses a small labeled dataset to train an inverse dynamics model that labels large-scale online Minecraft videos for behavioral-prior pretraining.The method extends internet-scale pretraining to sequential decision domains through semi-supervised imitation learning.
  • Experimental setting: The study uses Minecraft’s native mouse-and-keyboard interface at 20 frames per second, including human-style GUI interactions such as crafting and item dragging.This setting is intended to model human behavior and avoid domain shift or hand-engineered interfaces.
  • Results: The foundation model achieves nontrivial zero-shot behavior, including crafting planks and crafting tables—tasks described as impossible to learn with reinforcement learning alone.A proficient human requires a median of 50 seconds or approximately 970 consecutive actions for these tasks.
  • Results: Behavioral-cloning and reinforcement-learning fine-tuning extend performance into harder technology-tree tasks, culminating in diamond-tool crafting.Diamond-tool crafting requires a proficient human a median of more than 20 minutes or approximately 24,000 actions.

2 Preliminaries and Related Work

The related work contrasts imitation from labeled trajectories with methods for unlabeled demonstrations and positions VPT in Minecraft’s full, human action space. Prior Minecraft studies generally used custom worlds or restricted task settings.

  • Imitation learning: Imitation learning models action distributions from observation-action trajectories and requires causal policies that condition only on current and past observations.Labeled imitation learning has been applied to aerial vehicles, self-driving cars, board games, and video games.
  • Unlabeled demonstrations: When demonstrations lack action labels, prior approaches include adversarial imitation, latent-action learning, waypoint matching, and goal-conditioned policies.These methods use different proxies or objectives to imitate behavior from unlabeled demonstrations.
  • Minecraft research: Minecraft research has covered navigation, block placing, instruction following, combat, hill climbing, curriculum learning, and diamond mining across custom or large generated environments.The domain’s popularity and mechanics have attracted substantial reinforcement-learning research.
  • Positioning: VPT differs by operating in Minecraft’s full, unmodified human action space, including drag-and-drop inventory management and item crafting.The paper states that no published prior work operated in this complete human action space.

3 Methods

VPT trains an inverse dynamics model on a small labeled contractor dataset, uses it to label filtered online Minecraft videos, and behaviorally clones those inferred actions. The resulting foundation model supports zero-shot behavior and later fine-tuning.

  • Inverse dynamics: VPT first trains an inverse dynamics model on a small amount of labeled contractor data to infer actions from observation trajectories.Unlike a causal imitation policy, the inverse dynamics model may use both past and future observations when predicting an action.
  • Data filtering: Online Minecraft videos are filtered to retain clean survival-mode segments without visual artifacts or incompatible gameplay settings.A contractor-labeled dataset of 8,800 sampled images trains the clean-versus-unclean filtering model.
  • Foundation-model training: The foundation model is trained with standard behavioral cloning by minimizing the negative log-likelihood of actions predicted by the inverse dynamics model on clean data.The policy conditions on observations through the current timestep while the inverse dynamics model supplies action labels.
  • Capabilities: The resulting foundation model exhibits nontrivial zero-shot behavior and can be fine-tuned with both imitation learning and reinforcement learning for more complex skills.These capabilities are reported as outcomes of the foundation-model training pipeline.

4 Results

VPT uses a compact labeled dataset to train an inverse dynamics model that pseudo-labels roughly 70k hours of Minecraft video, enabling a behavioral prior with zero-shot skills and progressively stronger BC and RL fine-tuning. The resulting agents reach difficult technology-tree milestones, including diamond-pickaxe crafting, while performance depends on data scale, IDM quality, and preserving pretrained skills during RL.

  • 4.1 Performance of the Inverse Dynamics Model: 90.6% keypress accuracy and 0.97 mouse-movement R2 were achieved by an IDM trained on 1962 hours of labeled data.The IDM was evaluated on held-out contractor-labeled data.
  • 4.1 Performance of the Inverse Dynamics Model: IDMs were two orders of magnitude more data efficient than behavioral-cloning models and improved more quickly with additional data.This supports using labeled contractor data to train an IDM rather than directly training a foundation model.
  • 4.2 VPT Foundation Model Training and Zero-Shot Performance: The VPT foundation model learned from roughly 70k hours of IDM-labeled web video and developed nontrivial zero-shot behavior, including crafting tables.Contractors averaged 5.44 crafting tables per 60 minutes, compared with 0.19 for the foundation model.
  • 4.3 Fine-Tuning with Behavioral Cloning: BC fine-tuning specialized behavior and enabled wooden and stone tools, with contractor_house producing 213x more crafting tables and 59x more crafting overall.The contractor_house model also collected cobblestone and crafted stone tools, while earlygame_keyword improved existing skills without new behaviors.
  • 4.4 Fine-Tuning with Reinforcement Learning: Three-phase training achieved over 80% reliability on iron pickaxes, almost 20% on diamonds, and 2.5% on diamond pickaxes; the latter was the first reported nonzero success rate for this task.The model was human-level for crafting iron pickaxes and mining diamonds, though below humans for obtaining diamond pickaxes.
  • 4.4 Fine-Tuning with Reinforcement Learning: Removing the KL loss caused RL progress to stall, consistent with catastrophic forgetting of skills needed to advance beyond early items.The no-KL treatment obtained logs, planks, sticks, and crafting tables but did not progress further.

5 Discussion and Conclusion

VPT extends internet-scale pretraining to sequential decision domains by learning behavioral priors directly from unlabeled online video. The models show zero-shot behavior, achieve diamond-tool crafting after RL fine-tuning, and leave steerability, scaling, evaluation, and societal-impact questions for future work.

  • The models condition only on past observations and therefore cannot be asked to perform specific tasks.Preliminary closed-caption conditioning made the models weakly steerable.
  • Loss was not consistently correlated with downstream evaluation metrics, making progress slow and hard-won.The paper also identifies potential future risks from emulating inappropriate behavior as internet pretraining expands.
  • VPT learns behavioral priors directly from freely available internet-scale data for sequential decision domains.The paper contrasts directly learned action priors with representational priors from generative video modeling or contrastive methods.
  • The foundation model exhibited impressive zero-shot behavior before task-specific fine-tuning.Earlier results include crafting tables and other increasingly difficult Minecraft tasks.

A.1 Initial Unclean Dataset Curation

The initial dataset pipeline curates Minecraft Survival Mode footage from online videos using metadata filtering, human frame labels, automated classification, and temporal segmentation. It produces clean video segments and an early-game subset aligned with the evaluation setting.

  • Initial Dataset Curation: The curation targets single-player Minecraft Survival Mode footage resembling the evaluation environment, preferably version 1.16 with mouse-and-keyboard input.The scope excludes clips with substantial visual artifacts and focuses on computer gameplay rather than controller-based play.
  • Automated Filtering: The classifier uses CLIP image embeddings and an RBF-kernel SVM to identify clean frames.It is trained from 8,800 randomly sampled labeled images.
  • Human Labeling: Workers label sampled frames as clean Survival Mode, Survival Mode with artifacts, or None of the Above.Survival Mode is identified by health, hunger, and item-hotbar indicators; creative mode is excluded.
  • Automated Filtering: Videos are sampled at 3 frames per second, retained when at least 80% of frames are clean, median-filtered, and segmented into clean clips lasting at least 5 seconds.The resulting dataset is called web_clean.
  • Early-Game Subset: The early_game dataset is a roughly 3,000-hour web_clean subset selected by metadata matching fresh-world or first-episode indicators.Only the first 5 minutes of each matching video are retained.

B Contractor Data

Contractor data records Minecraft actions and video in an environment matched to evaluation, supporting inverse-dynamics training, human benchmarks, and targeted task datasets. The action space largely follows human controls but restricts arbitrary text entry and discretizes mouse movement.

  • Data Collection: A custom recorder captures contractor actions and game video using the same underlying engine as RL rollouts and evaluations.Contractors were recruited through UpWork and paid $20 per hour before platform fees and taxes.
  • Task Datasets: Contractors initially played Survival Mode freely, then completed targeted tasks including treechop, house building, and diamond-pickaxe acquisition.The targeted tasks provide behavior distributions and human completion-time statistics.
  • Task Datasets: The IDM training data includes general gameplay and treechop trajectories totaling 1,962 hours, while late-collected contractor_house data is excluded.The IDM can use any action-labeled data because it infers actions from video.
  • Environment and Actions: The environment exposes raw human-view pixels, including the hotbar, health indicators, and hand animation, with nearly human-like keyboard, mouse, and click actions.The client and server run at the same frequency, allowing simulation slower or faster than real time.
  • Environment and Actions: The agent cannot type arbitrary letters into the crafting recipe-book search bar, although it can browse the book with the mouse.This creates a narrower action space than the human interface for recipe lookup.
  • Environment and Actions: Foveated discrete mouse bins support both fine-grained inventory interaction and coarser mining and navigation movements.Preliminary experiments found this design improved crafting performance.

D.1 IDM Architecture

The inverse dynamics model processes a temporal window of image frames, combines temporal convolution with per-frame visual processing and unmasked transformer blocks, and predicts keyboard and mouse actions.

  • Input and Temporal Processing: The IDM takes 128 consecutive 128×128×3 image frames and predicts the action at each frame.The model has approximately 0.5 billion trainable weights.
  • Input and Temporal Processing: A 3-D convolution with temporal kernel width 5 incorporates neighboring temporal information at the model’s first layer.The paper compares this design with removing the temporal convolution on the default IDM dataset.
  • Visual Processing: The temporal convolution is followed by a ResNet image-processing network whose frames are then independently passed through dense layers.The ResNet stacks have widths W = {64, 128, 128}.
  • Transformer Processing: Four subsequent unmasked residual transformer blocks process the resulting frame representations.These blocks use unmasked attention, so the architecture is non-causal within the input video window.
  • Action Prediction: Independent action heads predict binary keypress states and 11-way horizontal and vertical mouse-movement categories.Mouse movements are discretized along both axes.

D.2 IDM Training

IDM and behavioral-cloning training use distinct causal requirements, data treatments, and action-space designs to model Minecraft behavior. Null-action filtering and hierarchical action modeling address distributional problems observed during training and rollout.

  • Experimental Scope: Training results come from one run for IDM, behavioral cloning, and reinforcement learning because the experiments were computationally expensive.The authors cite low training variance and sweep trends as partial mitigation.
  • Architecture: The IDM predicts actions using future frames, while behavioral cloning is causal and cannot see future observations.IDM pseudo-labeling uses sliding windows and retains center-frame predictions to avoid boundary effects.
  • Null Action Filtering: Null actions comprise 35% of human actions, but unfiltered behavioral-cloning models often produce more than 95% null actions.Filtering consecutive null-action groups generally increases crafting rates.
  • Null Action Filtering: Filtering groups of 3 or more null actions performed slightly better than filtering all null actions or groups of 21.Compute constraints prevented retraining all experiments with the later-selected setting.
  • Joint Hierarchical Action Space: Factored action spaces independently predict keys, whereas hierarchical actions jointly model keypress dependencies and whether the mouse moves.The full joint space is impractically large, motivating the hierarchical decomposition.
  • Joint Hierarchical Action Space: Factored-action models sample far more null actions than hierarchical models despite comparable environment statistics and zero null actions in the filtered training data.Independent key predictions are not conditioned on other keypresses.

E.4 Foundation Model Training

Foundation-model training uses IDM-generated pseudo-labels, while behavioral-cloning fine-tuning uses either focused pseudo-labeled videos or contractor data with ground-truth labels.

  • Foundation Model Training: Foundation-model training is similar to IDM training but replaces action labels with IDM-generated pseudo-labels.Its hyperparameters are listed in Table 4.
  • Behavioral-Cloning Fine-Tuning: Behavioral-cloning fine-tuning uses either the focused early_game dataset with pseudo-labels or contractor_house data with ground-truth labels.Its hyperparameters are listed in Table 5.

G.1 Reinforcement Learning Fine-Tuning Training Details

RL fine-tuning uses PPG, shaped item rewards, and a decaying KL penalty to preserve pretrained behavior while optimizing sparse Minecraft rewards.

  • RL Algorithm: PPG fine-tunes the policy using separate optimization phases for policy, value function, and shared representation.PPG extends PPO with additional value-function optimization to improve sample efficiency.
  • Policy Preservation: A KL divergence loss compares the trainable policy with the frozen pretrained policy to reduce catastrophic forgetting.The KL coefficient ρ weights this auxiliary loss relative to other losses.
  • Policy Preservation: The KL penalty replaces entropy maximization because uniform-random exploration is infeasible in the large, sparse-reward diamond-pickaxe task.The pretrained action distribution is used as an exploration prior instead.
  • Hyperparameters: Table 6 specifies the RL hyperparameters, including a special lower learning rate for early-game fine-tuning without KL loss.That treatment used 3 × 10^-6 instead of the standard 2 × 10^-5.
  • Policy Preservation: The KL coefficient starts relatively high and decays after each iteration, protecting early skills while allowing eventual reward optimization.This schedule balances preservation of pretrained behavior with divergence from it when necessary.
  • Reward Design: Rewards are tiered from 1 for wooden and stone items to 8 for diamond, with quantities normalized to prevent bulk items dominating.Items later in the technology tree receive higher base rewards.
  • Reward Design: The reward remains sparse and sometimes deceptive because thousands of actions may separate rewards, even after prerequisites are acquired.Finding diamonds can require more than 10,000 actions after crafting an iron pickaxe.
  • Compute: RL training collected roughly 1.4 million episodes totaling 16.8 billion frames over approximately 6 days on 80 GPUs and 56,719 CPUs.The experiments used about 4,000 optimization iterations.

G.2 Reinforcement Learning Fine-Tuning Additional Data

Additional RL experiments compare initialization choices and test the role of KL regularization in retaining and extending pretrained Minecraft skills.

  • KL Ablation: Without KL loss, RL fine-tuning from the early-game model learns only logs, planks, sticks, and crafting tables.The authors attribute the restriction to catastrophic forgetting of skills not initially rewarded.
  • Initialization Comparison: House-building initialization improves reward faster initially, but early-game initialization surpasses it after 800,000 episodes and begins smelting iron ingots.The early-game model was therefore selected for the main experiments.

H Foundation Model Scaling

Scaling produces different outcomes before and after fine-tuning: smaller models perform better zero-shot, whereas the 0.5B model performs best after contractor_house fine-tuning.

  • Zero-shot performance: The 71M model achieves the best zero-shot environment performance despite having the highest web_clean loss.It also shows non-zero wooden tool crafting, while the 248M model appears better at crafting than the 0.5B model.
  • Interpretation: Larger models have better web_clean validation loss but worse contractor-dataset loss and zero-shot performance.The authors hypothesize that larger models overfocus on visual peculiarities in web data, while fine-tuning helps shift their features to the game engine.
  • Evaluation setup: Figure 18 compares web_clean loss, contractor-dataset loss, and zero-shot environment performance across 71M, 248M, and 0.5B models.The 71M and 248M models use 15 epochs, while the 0.5B model uses 30 epochs.

I Text Conditioning

The paper explores conditioning a VPT agent on closed-caption text to make behavior steerable. Text conditioning increases targeted movement and collection behaviors, but crafting control remains weak and not practically useful.

  • Motivation: Closed captions could provide intent signals for conditioning VPT agents toward goals such as exploring, collecting resources, or crafting items.The dataset contains about 17k hours of videos with associated closed captions.
  • Method: The text-conditioned model is trained by associating caption text from each 30-second video chunk with every frame in that chunk.The model is a fine-tuned 220M-parameter VPT foundation model.
  • Behavioral effects: Text conditioning increases travel for exploration and water prompts and preferentially increases collection of seeds, wood, and dirt.Figure 20 measures five-minute episodes across conditioning variants; error bars are 95% confidence intervals over 1,000 episodes per variant.
  • Evaluation: Figure 20 plots distance traveled, wheat seeds collected, oak logs collected, and dirt collected by conditioning variant.Non-overlapping bars indicate statistically significant differences at p < 0.05.
  • Limitations: Conditioning does not reliably steer flower gathering, hunting, or crafting toward a requested item.Agents often follow the unconditional Minecraft crafting prior instead, such as crafting a stone pickaxe when asked for a wooden one.
  • Limitations: The resulting natural-language-conditioned agent is somewhat steerable but still too weak for practical use.The authors identify more research, data, and training compute as possible directions for improvement.
Loading 2206.11795v1…