Source-linked AI summary
Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning
Kimin Lee, Kibok Lee, Jinwoo Shin, Honglak Lee
TL;DR
Deep RL agents struggle to generalize to semantically similar environments with unseen visual patterns, especially from high-dimensional observations. The paper randomizes input observations with re-initialized convolutional networks and evaluates the approach across three tasks, where it outperforms regularization and data augmentation baselines.
Problem
Deep RL agents often fail to generalize to unseen environments with semantically similar content, particularly when observations are high-dimensional images.
Method
The method re-initializes a random convolutional network during training to perturb low-level visual features and uses Monte Carlo inference to reduce test-time variance.
Results
The method significantly reduces the generalization gap across CoinRun, DeepMind Lab, and Surreal robotics control, including a 39.8% to 58.7% success-rate improvement over PPO with cutout on CoinRun.
Takeaways & Limitations
Randomized observations encourage invariant and robust representations that support adaptation to unseen visual environments across 2D and 3D RL tasks.
Takeaways & Limitations
The paper focuses on high-dimensional state spaces such as images and notes that the method is also applicable to value-based methods.
Abstract
from arXiv · showhide
Deep reinforcement learning (RL) agents often fail to generalize to unseen environments (yet semantically similar to trained agents), particularly when they are trained on high-dimensional state spaces, such as images. In this paper, we propose a simple technique to improve a generalization ability of deep RL agents by introducing a randomized (convolutional) neural network that randomly perturbs input observations. It enables trained agents to adapt to new domains by learning robust features invariant across varied and randomized environments. Furthermore, we consider an inference method based on the Monte Carlo approximation to reduce the variance induced by this randomization. We demonstrate the superiority of our method across 2D CoinRun, 3D DeepMind Lab exploration and 3D robotics control tasks: it significantly outperforms various regularization and data augmentation methods for the same purpose.
1 INTRODUCTION
Deep RL agents often fail to generalize across semantically similar environments with unseen visual patterns, motivating a simple randomization-based approach. The method perturbs observations during training and outperforms conventional regularization and augmentation across three tasks.
- Deep RL agents can fail on unseen but semantically similar environments, especially when visual changes affect high-dimensional observations such as images.
- Prior work uses regularization and data augmentation, while simulator-based rendering randomization improves generalization but requires a physics simulator.
- The proposed method re-initializes random convolutional-network parameters each iteration to perturb textures, colors, and shapes in input observations.
- The method trains agents on randomized inputs and includes Monte Carlo inference to reduce test-time variance from the randomization.
- 39.8% to 58.7% success rates were achieved against PPO with cutout on 2D CoinRun, alongside higher total scores on DeepMind Lab and Surreal robotics control.
2 RELATED WORK
Related work studies RL generalization through environment splits, regularization, random networks, and transfer learning. These approaches address different aspects of generalization and adaptation in deep RL.
- RL generalization has been evaluated by separating training and test environments using random seeds or distinct video-game level sets.
- Regularization is a major direction for improving the generalization ability of deep RL algorithms.
- Random networks have been used in deep RL for intrinsic exploration rewards and ensemble-based approaches.
- Transfer learning aims to improve target-task performance by transferring knowledge from a source task, but fine-tuning pretrained models has not been beneficial in deep RL.
3 NETWORK RANDOMIZATION TECHNIQUE FOR GENERALIZATION
The method trains deep RL agents on observations transformed by reinitialized random networks, encouraging representations invariant to varied low-level visual features. It also uses feature matching and Monte Carlo inference to improve robustness and reduce variance.
- Randomized input observations: The agent receives randomized input bs = f(s; φ), with random-network parameters reinitialized each iteration to expose it to varied visual patterns.The transformations preserve semantics while perturbing low-level features such as textures, colors, and shapes.
- Feature matching: Feature matching combines hidden features from clean and randomized observations to learn more invariant features against input changes.The hidden representation is taken from the policy network’s penultimate layer.
- Random-network design: A single-layer CNN random network produces outputs with the same dimension as the input, enabling observation perturbations without changing input size.Its parameters are initialized from a mixture that includes identity and normal-distribution components.
- Random-network design: The mixture uses clean inputs with probability α because training only on randomized inputs can complicate training.The Xavier normal component maintains input variance, and the authors report that it stabilizes training.
- Monte Carlo inference: At inference, the policy is treated as stochastic over random-network parameters and approximated by aggregating decisions from M random inputs.This Monte Carlo approximation is intended to estimate the expectation over the random-network prior more accurately.
4 EXPERIMENTS
Experiments evaluate generalization to unseen visual environments across CoinRun, DeepMind Lab, and Surreal robotics control. The proposed method is analyzed through performance, representation quality, activation maps, and comparisons with domain randomization.
- Experimental setup: Experiments measure trained-agent performance in unseen environments with different backgrounds, objects, and floors across three RL tasks.The tasks are 2D CoinRun, 3D DeepMind Lab exploration, and 3D robotics control.
- CoinRun: All baseline agents fail to generalize from one seen CoinRun level to unseen backgrounds, while the proposed method is most effective among evaluated methods.Regularization has little impact; cutout and color jitter provide only slight improvements.
- CoinRun: Increasing Monte Carlo samples improves performance and reduces variance, with gains saturating around ten samples.The experiments use ten samples in subsequent evaluations.
- Representation analysis: The proposed method improves hidden-representation alignment with unseen trajectories, as measured by cycle-consistency relative to vanilla PPO.The analysis uses t-SNE projections of human-demonstrator trajectories and quantitative cycle-consistency measurements.
- CoinRun: 39.8% to 58.7%: success rates improve when comparing PPO with cutout augmentation against the proposed method in large-scale CoinRun.The authors report that the proposed method outperforms all baseline methods by a large margin.
- DeepMind Lab and robotics: The proposed method achieves high scores in both seen and unseen DeepMind Lab environments, whereas baseline agents fail to adapt to unseen environments.This evaluation tests generalization under high-dimensional, complex 3D observations.
- DeepMind Lab and robotics: The method significantly improves unseen-environment performance in Surreal robotics control while maintaining seen-environment performance and outperforms domain-randomized baselines trained on more diverse environments.The domain-randomization comparison uses more finite seen environments for baseline training, while the proposed agent remains trained in a single environment.
5 CONCLUSION
The paper targets generalization to semantically similar environments with unseen visual patterns by randomizing the first CNN layer. It argues that diverse inputs encourage invariant, robust representations with potential relevance beyond the evaluated setting.
- The method randomizes the first CNN layer to perturb low-level features such as textures, colors, and shapes.
- Diverse randomized observations encourage agents to learn invariant and robust representations for unseen visual patterns.
- The proposed representations may also support adversarial defense, sim-to-real transfer, transfer learning, and online adaptation.
B ROBUSTNESS AGAINST ADVERSARIAL ATTACKS
The paper evaluates whether its randomized-network method improves robustness to adversarial visual perturbations. It reports improved robustness against FGSM attacks with ε = 0.01, associating this result with more robust hidden representations.
- Adversarially perturbed observations can cause DNN-based policies to make incorrect test-time decisions.
- The method is evaluated for robustness against adversarial attacks because randomized training is expected to improve resistance to such perturbations.
- The proposed method improves robustness against FGSM attacks with ε = 0.01, implying more robust hidden representations.
C DETAILS FOR TRAINING AGENTS USING PPO
The experiments train policies with PPO, compare several image augmentations, and examine randomized-network placement across architectures and environment designs. Training and evaluation vary visual themes, layouts, and task-specific action settings.
- Policy optimization: PPO uses γ = 0.999, GAE λ = 0.95, entropy bonus 0.01, 256-step rollouts, three epochs, eight mini-batches, and Adam with learning rate 0.0005.
- Data augmentation methods: Compared augmentations include cutout, grayout, inversion, and color jitter, which modify input images through masking or channel and pixel transformations.
- Random-network placement: In large-scale CoinRun, unseen-environment performance decreases when the random network is placed in higher layers.
- Small-scale CoinRun: Small-scale CoinRun fixes a map layout with two moving obstacles and changes background style to measure performance.
- Large-scale CoinRun: Large-scale CoinRun trains on 500 levels using half of the available themes and evaluates on 1,000 levels with unseen themes.
- DeepMind Lab: DeepMind Lab varies ten floor and ten wall styles, while its action configuration includes eight agent actions and a native seven-action set.
H EXPERIMENTS ON DOGS AND CATS DATABASE
The dogs-and-cats experiment constructs a color-biased dataset in which animal color correlates with class during training but reverses across test and validation examples. Models are trained without ImageNet-pretrained ResNet-18 and evaluated using validation-selected hyperparameters.
- Dataset: The original database contains 25,000 training images and 12,500 testing images of dogs and cats.
- Dataset: Training images contain bright dogs and dark cats, whereas test and validation sets contain dark dogs and bright cats.
- Training and evaluation: The experiments report training and test accuracies with validation-selected hyperparameters and avoid ImageNet-pretrained ResNet-18.
I EXPERIMENTAL RESULTS ON SURREAL ROBOT MANIPULATION
The paper evaluates its method on Surreal robot manipulation using a hybrid CNN-LSTM policy in a distributed PPO setup. Figure 13 compares learning curves for regularization and data augmentation techniques across three runs.
- The Block Lifting task rewards the Sawyer robot for successfully lifting a randomly placed block from a table.
- The Surreal experiment uses a hybrid CNN-LSTM architecture as the policy network within a distributed reinforcement-learning framework.
- Figure 13 reports learning curves for regularization and data augmentation, with means and standard deviations computed across three runs.
J EXTENSION TO DOMAINS WITH DIFFERENT DYNAMICS
The paper extends evaluation to CartPole and Hopper, where agents must generalize across unseen dynamics using proprioceptive inputs. Randomization improves unseen-environment performance while remaining comparable to seen-environment performance.
- The extension tests dynamics generalization on CartPole and Hopper using proprioceptive features such as positions and velocities.
- Figure 15 displays mean performance and standard deviation across three runs for seen and unseen CartPole and Hopper environments.
- CartPole and Hopper use training dynamics that differ from testing dynamics, including disjoint parameter ranges or mass values.
- The proposed randomization improves performance in unseen environments while achieving performance comparable to seen environments.
K FAILURE CASE OF OUR METHODS
The paper examines color- or texture-conditioned tasks as a potential failure case for observation randomization. It finds that performance can remain effective when other distinguishing structure is available or relational color information is preserved.
- The method could fail when good and bad objects share the same shape and differ only in color.
- In modified CoinRun, the method works when agents can use additional object factors beyond color to distinguish good from bad objects.
- Color-matching tasks remain compatible with the method because matching objects retain the same color relationship after randomization.
L ABLATION STUDY FOR FRACTION OF CLEAN SAMPLES
The ablation varies the fraction of clean samples used during training and examines both performance and hidden representations. On large-scale CoinRun, the best unseen performance occurs with a clean-sample fraction of 0.1.
- The best unseen performance on large-scale CoinRun is achieved when the fraction of clean samples during training is 0.1.
- Algorithm 1 samples random-network parameters, runs policies on randomized observations, estimates advantages, and optimizes the random-network objective.
- Figure 17 visualizes hidden representations for PPO, several baselines, and the proposed method using t-SNE, with colors denoting environments.