PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Policy Adaptation during Deployment

Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyà, Pieter Abbeel, Alexei A. Efros, Lerrel Pinto, Xiaolong Wang

arXiv:2007.04309v3cs.LGcs.CVcs.ROstat.ML

TL;DR

Vision-based RL policies often fail to generalize from their training environment to unseen deployment environments, where reward-based fine-tuning may be impractical. The paper proposes self-supervised Policy Adaptation during Deployment to update pretrained policies online without rewards, improving generalization across simulated and real-world changes.

  • Problem

    Vision-based RL policies may not generalize to new environments, while deployment-time reward supervision is often impractical to engineer.

  • Method

    Policy Adaptation during Deployment updates a policy's shared feature extractor during deployment using gradients from a self-supervised prediction task instead of rewards.

  • Results

    PAD improves generalization across diverse simulated and real-world environmental changes, including 19 out of 22 DeepMind Control test environments and all considered CRLMaze environments.

  • Takeaways & Limitations

    Online self-supervision can adapt pretrained vision-based policies to their actual deployment environments without requiring reward signals.

  • Takeaways & Limitations

    Forward dynamics self-supervision can admit trivial constant-feature solutions, motivating care in selecting the auxiliary task.

Abstract

from arXiv · show

In most real world scenarios, a policy trained by reinforcement learning in one environment needs to be deployed in another, potentially quite different environment. However, generalization across different environments is known to be hard. A natural solution would be to keep training after deployment in the new environment, but this cannot be done if the new environment offers no reward signal. Our work explores the use of self-supervision to allow the policy to continue training after deployment without using any rewards. While previous methods explicitly anticipate changes in the new environment, we assume no prior knowledge of those changes yet still obtain significant improvements. Empirical evaluations are performed on diverse simulation environments from DeepMind Control suite and ViZDoom, as well as real robotic manipulation tasks in continuously changing environments, taking observations from an uncalibrated camera. Our method improves generalization in 31 out of 36 environments across various tasks and outperforms domain randomization on a majority of environments.

1 INTRODUCTION

Vision-based reinforcement-learning policies often fail to generalize across unseen environments, while reward-based deployment fine-tuning is frequently impractical. PAD instead uses self-supervision to adapt a pretrained policy online without deployment rewards, improving generalization across simulated and real settings.

  • Vision-based RL policies may not generalize from training environments to unseen environments, especially with high-dimensional image inputs.
  • Domain randomization trains policies to be invariant to anticipated environmental variation, but its training coverage may not match the test environment.
  • Reward-based fine-tuning during deployment is often impractical because crafting dense deployment rewards can require substantial engineering.
  • PAD adapts a pretrained policy to an unknown environment without rewards by adding self-supervision to the policy’s training objectives.
  • 19 out of 22 DeepMind Control test environments improved, while all considered CRLMaze environments improved under PAD.

2 RELATED WORK

Prior work studies robust policies, domain adaptation, self-supervised representation learning, and test-time training, but PAD targets truly unseen environmental changes in reinforcement learning. Its deployment procedure uses observations collected online rather than a fixed policy.

  • Self-supervised learning creates training labels from input data through auxiliary tasks such as rotation prediction, jigsaw solving, and tracking.
  • Domain adaptation commonly assumes access to target-domain data, whereas domain generalization assumes domains arise from a shared meta-distribution.
  • PAD focuses on generalization to environmental changes that cannot be anticipated during training.
  • Related image-recognition benchmarks test models on corruption types unknown during training, motivating analogous unseen-change evaluations in RL.
  • During deployment, PAD collects observations online and optimizes only the self-supervised objective rather than keeping the policy fixed.

3 METHOD

PAD shares feature representations between a policy and a self-supervised prediction head, then updates the feature extractor online using observations from the deployment environment. Training jointly optimizes reinforcement-learning and self-supervised objectives, while deployment removes the reward-dependent objective.

  • PAD can be implemented on top of any policy network and standard on-policy or off-policy RL algorithm minimized by stochastic gradient descent.
  • The architecture factorizes the policy into a feature extractor and action head, with a self-supervised head operating on extracted features.
  • Inverse dynamics predicts the intervening action from states before and after a transition, using feature representations rather than raw observations.
  • For continuous actions, inverse-dynamics loss uses mean squared error; for discrete actions, it uses cross-entropy on a soft-max action distribution.
  • PAD uses inverse dynamics instead of forward dynamics in feature space to avoid trivial constant-feature solutions.
  • Rotation prediction classifies whether an image was rotated by 0, 90, 180, or 270 degrees.
  • Before deployment, training minimizes the RL objective plus a weighted self-supervised objective; during deployment, only the self-supervised loss updates the policy’s feature extractor.
  • At each deployment iteration, the method observes a state, updates self-supervised parameters, and selects an action using the adapted policy.

4 EXPERIMENTS

Experiments test PAD on unseen visual environments across DMControl, CRLMaze, and robotic manipulation, using task-specific self-supervision during reward-free deployment. PAD generally improves generalization, with benefits depending on the environment changes and auxiliary task.

  • Experimental setup: PAD is evaluated without test-time rewards on single-environment training followed by deployment across unseen environments.Experiments cover DMControl, CRLMaze, and robotic manipulation, with visual changes and, in robotics, dynamics changes.
  • DeepMind Control: PAD improves randomized-color generalization in all considered DMControl tasks and outperforms domain randomization in 6 out of 9 tasks.Despite overlap between domain-randomization training and evaluation domains, domain randomization performs no better than vanilla SAC on most tasks.
  • DeepMind Control: PAD improves generalization in 6 out of 7 long-horizon tasks without degrading policy performance, while online learning provides substantially more benefit than offline adaptation.Performance on the original training environment remains virtually unchanged.
  • DeepMind Control: PAD outperforms all baselines on 7 out of 8 video-background tasks, reaching a 104% improvement over domain randomization on Finger, spin.The authors conjecture that domain randomization struggles because randomized-color training does not cover video image statistics.
  • CRLMaze and auxiliary tasks: Rotation prediction is more suitable for scene-understanding navigation, whereas inverse dynamics is more useful for motor-control tasks.PAD with rotation prediction improves all considered CRLMaze test environments, while inverse dynamics does not; the paper leaves auxiliary-task selection automation for future work.
  • Robotic manipulation tasks: PAD improves robotic push generalization by as much as 24% under cloth dynamics and 28% when all three simulated environmental changes are combined.The method is especially effective when dynamics changes are non-trivial; mount-position changes alone affect PAD and SAC+IDM relatively little.

5 CONCLUSION

The paper proposes adapting pretrained vision-based policies to unknown deployment environments without rewards, using self-supervision during deployment. It reports improved generalization across diverse simulated and real-world changes, while identifying auxiliary-task selection as an unresolved dependency.

  • PAD adapts pretrained vision-based policies to new environments without reward signals by using self-supervised learning during deployment.
  • The framework targets environmental changes that previous methods must anticipate when learning invariant policies.
  • Empirical evaluations cover diverse simulated and real-world environmental changes across multiple tasks.
  • The framework currently relies on prior knowledge to select self-supervised tasks for policy adaptation.
  • The authors position PAD as an initial step toward agents that learn with and without rewards before and during deployment.

A PERFORMANCE ON THE TRAINING ENVIRONMENT

On unchanged training environments, PAD preserves performance rather than producing the generalization gains observed in novel environments. The evaluation covers DMControl and CRLMaze, including comparisons with a blind SAC baseline in DMControl.

  • PAD performance is virtually unchanged when evaluated in the same environment used for training.The authors conjecture that continued training has little influence because the algorithm is already adapted to the training distribution.
  • The training-environment evaluation includes both DeepMind Control and CRLMaze results.
  • DMControl comparisons additionally include a blind SAC agent operating only from previous actions.The blind agent indicates how much a task benefits from visual information.
  • Table 7 reports episodic return for nine DMControl tasks using means and standard deviations over 10 seeds.
  • Table 8 reports CRLMaze training-environment episodic return using means and standard errors over 10 seeds.

B LEARNING CURVES ON DEEPMIND CONTROL

On three DMControl tasks, SAC with and without the inverse-dynamics self-supervision has similar learning behavior, while domain randomization is less sample-efficient and can converge sub-optimally.

  • Learning curves compare SAC, SAC trained with domain randomization, and SAC+IDM on three DMControl tasks.
  • All methods are trained until convergence for 500,000 frames, and episodic return is averaged across 10 seeds with 95% confidence intervals.
  • SAC and SAC+IDM exhibit similar sample efficiency and final performance across the three tasks shown.
  • Domain randomization consistently shows worse sample efficiency and larger variation between random seeds.
  • Domain randomization converges to sub-optimal performance in two of the three tasks shown.

C KEEPING πs FIXED DURING POLICY ADAPTATION

Fixing the self-supervised task head at test time produces a negligible difference from updating it, so the default implementation updates both the task head and feature extractor.

  • Updating or fixing the self-supervised task head πs at test time makes a negligible difference in randomized-color generalization.
  • The default PAD variant optimizes both πe and πs using the self-supervised objective during adaptation.
  • The comparison uses three DMControl tasks and reports episodic return over 10 seeds.

D COMPARISON TO ADAPTATION WITH REWARDS

The authors compare PAD with naïve reward-based fine-tuning that assumes target-environment data and rewards are available before deployment.

  • PAD is compared with naïve fine-tuning using transitions and rewards collected from the target environment before deployment.The comparison uses datasets of 1, 10, or 100 episodes collected with the learned policy while its parameters remain fixed.

E ADDITIONAL ROBOTIC MANIPULATION SAMPLES

The robotic manipulation evaluation includes real-robot reach samples and comparisons involving target environments with randomized colors.

  • The reach task includes samples from training and test environments, with agents trained in simulation and deployed on a real robot.Samples from the push task are presented separately in Figure 4.
  • Table 10 reports episodic return in test environments with randomized colors across 10 seeds.It compares IDM (PAD) with naïve fine-tuning using 1, 10, or 100 target-environment episodes before deployment.
  • Reach moves a robot gripper to a red disc using observations from an uncalibrated camera.Agents are trained in setting (a) and evaluated in settings (b–d) on a real robot.

F IMPLEMENTATION DETAILS

The experiments use convolutional visual policies with shared features, algorithm-specific RL implementations, random-crop augmentation, and self-supervised deployment-time updates.

  • Implementation scope: The implementation covers DMControl, CRLMaze for ViZDoom, and robotic manipulation, with code made publicly available.The robotic manipulation implementation closely follows DMControl.
  • Architecture: The policy and self-supervised prediction share a feature extractor, while the policy head outputs the action distribution.Figure 7 illustrates the network architecture used for DMControl, CRLMaze, and robotic manipulation.
  • Architecture: Observations use stacked rendered frames cropped from 100 × 100 to 84 × 84, producing 9×84×84 inputs when k = 3.The shared extractor has 8 convolutional layers for DMControl and robotic manipulation, or 6 for CRLMaze.
  • Architecture: The network uses 32 filters in convolutional layers and hidden size 1024 in fully connected layers.
  • Learning algorithm: SAC is used for DMControl and robotic manipulation, whereas A2C is used for the discrete-action CRLMaze task.SAC policies output Gaussian action distributions for continuous action spaces, while A2C learns a soft-max distribution for CRLMaze.
  • Data augmentation: Random cropping regularizes learning while preserving spatio-temporal patterns across stacked frames.For inverse dynamics, consecutive observations receive different crops while frames within each observation share one crop.
  • Policy adaptation during deployment: During deployment, agents receive no reward signal and update the policy using a self-supervised objective on observations from the environment.The evaluation measures episodic return after training in one environment and testing across environments with distinct changes.
  • Policy adaptation during deployment: PAD uses a batch size of 32 randomly cropped observations, including augmented copies of the most recent observation.The method combines random-crop augmentation with batch-based adaptation because both improved test-time learning empirically.
Loading 2007.04309v3…