PaperlayerView exact revision on arXiv

Source-linked AI summary

Observational Overfitting in Reinforcement Learning

Xingyou Song, Yiding Jiang, Stephen Tu, Yilun Du, Behnam Neyshabur

arXiv:1912.02975v2cs.LGcs.AIstat.ML

TL;DR

Model-free RL agents can overfit to spurious observation features rather than latent task dynamics, complicating generalization analysis. The paper fixes underlying MDP dynamics while varying observation functions, then studies this observational overfitting across theoretical and empirical settings. The experiments identify architectural implicit regularization as a recurring property, while noting limits of margin-based generalization measures and current norm bounds.

  • Problem

    RL generalization studies confound observation-based overfitting with dynamics, policy, and optimization effects, leaving observational causes insufficiently isolated.

  • Method

    The paper fixes an underlying MDP and constructs parameterized observation functions that combine invariant task features with parameter-dependent features across LQR, Projected-Gym, and CoinRun settings.

  • Results

    Across LQR, Projected-Gym, and CoinRun, the paper reports implicit regularization effects, including improved generalization from MLP overparametrization in CoinRun.

  • Takeaways & Limitations

    Observational overfitting provides a controlled framework for studying how RL architectures generalize when spurious features vary while task dynamics remain fixed.

  • Takeaways & Limitations

    State-action margin distributions are not fully correlated with total-reward generalization, and current norm measures are too loose for the RL case.

Abstract

from arXiv · show

A major component of overfitting in model-free reinforcement learning (RL) involves the case where the agent may mistakenly correlate reward with certain spurious features from the observations generated by the Markov Decision Process (MDP). We provide a general framework for analyzing this scenario, which we use to design multiple synthetic benchmarks from only modifying the observation space of an MDP. When an agent overfits to different observation spaces even if the underlying MDP dynamics is fixed, we term this observational overfitting. Our experiments expose intriguing properties especially with regards to implicit regularization, and also corroborate results from previous works in RL generalization and supervised learning (SL).

1 Introduction

The paper isolates observational overfitting as a distinct source of RL generalization failure and develops a framework for studying it while holding underlying MDP dynamics fixed. Its experiments examine implicit regularization across synthetic and established RL settings.

  • 1 Introduction: Prior RL generalization work spans theoretical analyses and empirical benchmarks, but lacks a middle ground for complex MDPs with nonlinear policies and comparable benchmarks.Existing studies also mix confounds including dynamics, exploration, determinism, and optimization choices.
  • 1 Introduction: The paper identifies observational overfitting as overfitting to observation properties irrelevant to the latent MDP dynamics, while fixing one underlying MDP’s dynamics.This isolates observation effects from other RL generalization confounds.
  • 1 Introduction: The paper introduces a parametric framework applicable to any underlying MDP and studies observational overfitting with LQR, neural policies in Gym, and CoinRun.The framework supports both theoretical analysis and empirical evaluation across changing observation spaces.
  • 1 Introduction: Across these settings, the paper’s primary novel result is that architectural implicit regularization occurs under observational overfitting in RL.The appendix further analyzes convex one-step LQR and reports a necessary generalization gap under Gaussian policy initialization and gradient descent.

2 Motivation and Related Work

The paper motivates observational overfitting through visual RL examples and formalizes it by varying observation functions while preserving base MDP dynamics. The resulting framework separates invariant task features from parameter-dependent, potentially spurious features and focuses analysis on architectural implicit regularization.

  • 2 Motivation and Related Work: Blackening Sonic’s scoreboard increased mean test reward from 1052 to 1141 for NatureCNN and from 1130 to 1250 for IMPALA.The reported increase was 10% for both policies, with standard deviation 40 between runs.
  • 2 Motivation and Related Work: Saliency maps showed that Sonic agents used the timer and background objects because both were correlated with progress, despite not necessarily generalizing across levels.The sidescroller background moves backward as the character moves forward, creating a progress correlation.
  • 2.3 Setup: The formal construction fixes a base MDP and varies only an observation function φθ, turning each parameterized environment into a base MDP combined with a POMDP-like observation mapping.The agent receives a high-dimensional observation rather than the underlying state.
  • 2.3 Setup: The (f, g)-scheme combines invariant features f with parameter-dependent features gθ, where the latter represent changing backgrounds or textures that can induce overfitting.A combination function h produces the final observation from these two projections.
  • 2.4 Architecture and Implicit Regularization: The setup assumes a static concatenation function h, allowing explicit regularization to remove access to gθ, so the analysis focuses on implicit architectural regularization.Architectural comparisons are made with fixed hyperparameters to isolate effects attributable to architecture.

3 Experiments

Across LQR and projected Gym experiments, overparameterization and architecture choices affect RL generalization through implicit regularization, while standard supervised-learning bounds often predict it poorly. The results also show that generalization depends strongly on environment dynamics and convolutional inductive biases.

  • 3.1 Overparamterized LQR: Increasing noise dimension raises the LQR generalization gap, with experiments suggesting scaling by O(√d_noise), while gradient descent cannot remove the associated overfitting component.The bound is reported to remain tight as noise dimension increases.
  • 3.1 Overparamterized LQR: Overparameterization reduces the LQR generalization gap and end-to-end policy norms without changing expressiveness, suggesting an implicit bias toward simpler policies.This effect occurs with stacked linear layers, so it does not require additional nonlinear representation power.
  • 3.1 Overparamterized LQR: Standard supervised-learning bounds have little predictive power for RL generalization: spectral, Frobenius-product, and Fischer-Rao terms become ineffective or too loose, while nuclear-norm products help only for width.The naive spectral bound diverges with two intermediate layers, and weight-counting sums are too loose.
  • 3.2 Projected Gym Environments: Projected Gym results vary substantially by environment and architecture: Tanh depth, ReLU width, and residual ReLU layers can improve generalization, but Tanh stacking may cause vanishing gradients and poor training.Residual layers improve generalization while stabilizing training, whereas the smoothness and complexity of reward functions differ across environments.
  • 3.3 Deconvolutional Projections: On synthetic Gym-Deconv, NatureCNN, IMPALA, and IMPALA-LARGE preserve the same generalization ranking as on CoinRun, despite evaluation using purely numeric observations.This suggests convolutional generalization quality is not limited to real-world or human-prior visual data.
  • 3.3 Deconvolutional Projections: Memorization is reduced by these architectures, with IMPALA-LARGE memorizing less than IMPALA despite greater capacity; specific limits include 30 versus 50 levels for NatureCNN and 2 versus 5 for IMPALA.The authors interpret residual blocks as implicit regularizers, while attention-only RMC performs poorly on the task.
  • 3.3 Deconvolutional Projections: Raw policy norms poorly predict generalization because softmax does not penalize arbitrarily large logits, and margin distributions are not fully correlated with total-reward performance.The paper notes that this correlation strengthens under Lipschitz assumptions on state-action transitions.

4 Conclusion

The paper isolates observational overfitting across analytical, principled synthetic, and large-scale visual settings, finding that overparameterization can improve RL generalization while current supervised-learning-inspired bounds remain inadequate.

  • Conclusion: The study examines observational overfitting through LQR, Projected-Gym, and CoinRun settings, including nonlinear policies and moving relevant features.These settings cover linear policies, MLPs and convolutional networks, and large-scale visual inputs.
  • Conclusion: Overparameterization improves generalization in the CoinRun setting with MLP policies, extending the observed effect beyond the synthetic control environments.
  • Conclusion: Current network-policy bounds adapted from supervised learning cannot explain the observed generalization benefits of overparameterization in RL.The paper identifies this mismatch as an important direction for further theory.
  • Conclusion: Extending the framework to time-dependent feature movement and memory-based recurrent policies is identified as beneficial future work.The paper connects moving-object settings to both RL generalization and video classification.
  • Conclusion: Whether overparameterization effects persist for off-policy methods such as Q-learning and evolution-strategy methods remains unclear.

A.1.1 LQR

The appendix reports that explicit norm-based regularization reduces generalization gaps, with first-layer regularization especially effective in the synthetic tasks.

  • A.1.1 LQR: Figure A1 organizes results by depth, width, and noise dimension, including singular values and train/test losses.
  • A.1.1 LQR: Explicit norm-based regularization reduces generalization gaps in the LQR-related synthetic experiments.
  • A.1.1 LQR: First-layer weight regularization substantially improves generalization, possibly because it biases the layer toward viewing the output of the task’s feature map.
  • A.1.1 LQR: Figure A2 presents the effect of explicit regularization on layer norms.

A.2 Large ImageNet Models for CoinRun

The appendix documents the large ImageNet-model configurations used for CoinRun and provides smoothed training and testing curves for these architectures.

  • A.2 Large ImageNet Models for CoinRun: RMC32x32 downsamples CoinRun images to 32 × 32, uses all pixels for attention, and employs coordinate embeddings with specified memory settings.
  • A.2 Large ImageNet Models for CoinRun: ShakeShake training excludes its auxiliary loss, using only the pure network.
  • A.2 Large ImageNet Models for CoinRun: VGG-A is described as a similar but slightly smaller architecture than VGG-16.
  • A.2 Large ImageNet Models for CoinRun: The appendix provides smoothed training and testing curves for the large architectures evaluated on CoinRun.

A.3.1 Exact infinite LQR

The appendix states the infinite-horizon LQR cost and policy notation, then identifies hyperparameters for the LQR experiments.

  • A.3.1 Exact infinite LQR: The infinite-horizon LQR policy is written as u_t = −Kx_t, with cost C(K) represented using the solution P_K of a Lyapunov equation.
  • A.3.1 Exact infinite LQR: Vectorizing the LQR matrix equation and using the Kronecker product converts the cost calculation into a linear regression problem with an exact TensorFlow-implementable solution.
  • A.3.1 Exact infinite LQR: Table 2 lists the hyperparameters used for the LQR experiments.

A.3.2 Projection Method

The projection method simulates environment-specific network weights by processing each batch entry with the same architecture but different weights indexed by environment.

  • Batch matrix multiplication assigns different network weights to each environment while preserving a shared architecture.The environment's numeric ID indexes the corresponding weights from global memory.

A.3.3 ImageNet Models

The supervised-learning network configurations are documented through external TensorFlow Models and DeepMind Sonnet repositories.

  • Supervised-learning tasks use network implementations referenced from TensorFlow Models and DeepMind Sonnet repositories.The cited components include Slim network factories and relational memory modules.

A.3.4 PPO Parameters

This material specifies PPO2 and CoinRun implementation choices while developing an LQR analysis of observational overfitting and gradient-descent generalization. The analysis links finite-sample error and initialization effects to Hessian rank and implicit regularization.

  • A.3.4 PPO Parameters: The experimental setup uses PPO2 hyperparameters and retains CoinRun defaults except for minibatches, while omitting recurrent additions.Velocity can be presented in the image instead of using recurrent temporal processing.
  • A.4 Theoretical (LQR): The LQR analysis restricts policies to a bounded stable set so that the infinite-horizon cost remains finite.The restriction imposes bounds on policy norm and the closed-loop transition norm.
  • A.4.2 Gradient Dynamics in 1-Step LQR: Non-degenerate initialization components in the Hessian null space remain unchanged under gradient descent, preventing convergence to a minimizer confined to the Hessian span.This mechanism is described for degenerate Hessians and isotropic Gaussian initialization.
  • A.4.2.1 Finite Sample Generalization Gap: As the number of samples m increases, the matrix rank increases, allowing gradient descent to recover the minimizer more accurately.The supplied discussion attributes improved recovery to the increased rank of the relevant matrix.
  • A.4.2.1 Finite Sample Generalization Gap: The one-step LQR analysis computes the exact gradient-descent error and relates its scaling to Hessian rank rather than a standard 1/√m concentration argument.The supplied figure passage states that the proof relies specifically on Hessian rank as m increases.
  • A.4.2.1 Finite Sample Generalization Gap: Gradient descent on finite samples has a generalization gap whose contributions arise from the minimum-norm stationary point and full-rank initialization.The theorem defines the limiting point K∞, while the decomposition identifies E1 and E2 as the two error sources.
Loading 1912.02975v2…