PaperlayerView exact revision on arXiv

Source-linked AI summary

Identity Crisis: Memorization and Generalization under Extreme Overparameterization

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Michael C. Mozer, Yoram Singer

arXiv:1902.04698v4stat.MLcs.AIcs.LG

TL;DR

The paper investigates what determines generalization in overparameterized networks, using identity mapping and single-example training to examine memorization and generalization. It compares network architectures and finds markedly different inductive biases, including cases where CNNs generalize while FCNs do not.

  • Problem

    The source of inductive bias in highly overparameterized networks remains unclear, despite models often exceeding the number of training examples.

    Source: §1 · p. 1

  • Method

    The authors train linear and nonlinear fully connected and convolutional networks on identity mapping, comparing architecture and other training choices in a setting that permits visualization.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Architectures exhibit different inductive biases: with 60k MNIST digits, only the CNN reasonably approximates identity outside the hull of hand-printed digits, while single-example experiments reveal a continuum from constant memorization to identity generalization.

    Source: §1 · p. 2, §4 · p. 9

  • Takeaways & Limitations

    The single-example identity setup permits rich characterization and visualization of how architectural choices shape memorization and generalization.

    Source: §1 · p. 2, §4 · p. 9

    The study restricts its learning task to identity mapping, a choice that sidesteps the problem's Bayes and hypothesis-class approximation errors.

    Source: §4 · p. 9

Abstract

from arXiv · show

We study the interplay between memorization and generalization of overparameterized networks in the extreme case of a single training example and an identity-mapping task. We examine fully-connected and convolutional networks (FCN and CNN), both linear and nonlinear, initialized randomly and then trained to minimize the reconstruction error. The trained networks stereotypically take one of two forms: the constant function (memorization) and the identity function (generalization). We formally characterize generalization in single-layer FCNs and CNNs. We show empirically that different architectures exhibit strikingly different inductive biases. For example, CNNs of up to 10 layers are able to generalize from a single example, whereas FCNs cannot learn the identity function reliably from 60k examples. Deeper CNNs often fail, but nonetheless do astonishing work to memorize the training output: because CNN biases are location invariant, the model must progressively grow an output pattern from the image boundaries via the coordination of many layers. Our work helps to quantify and visualize the sensitivity of inductive biases to architectural choices such as depth, kernel width, and number of channels.

1 Introduction

The paper asks what drives generalization in overparameterized networks, then uses the identity task to visualize architecture-dependent inductive bias. Its comparisons show that architectures differ in generalization even with many examples, and that single-example training reveals a continuum between memorization and generalization.

1 Introduction

  • Overparameterization motivates the question of inductive bias because model size can exceed the training set, while existing analyses do not identify its root source.Guarantees that large networks fit training data do not distinguish structured labels from randomly shuffled labels, which can only be memorized.Source: §1 · p. 1
  • The figure separates three training examples from test examples while comparing predictions from three architectures trained on 60k MNIST examples.The caption specifies the boundary between training and test examples; the accompanying prose reports the architecture comparison.Source: Figure 1 · p. 2, §1 · p. 2
  • The paper studies identity mapping as a restrictive, visualizable task for comparing architecture, optimization, initialization, and hyperparameter effects on inductive bias.The task requires transmitting every input feature and supports analysis of model behavior and hidden representations.Source: §1 · p. 2
  • Only the CNN approximates the identity function reasonably well outside the hull of hand-printed digits, although all three architectures fit training examples and transfer to novel digits and blends.Figure 1 compares a 20-layer CNN, a 10-layer ReLU FCN, and a 1-layer FCN trained on 60k MNIST digits.Source: §1 · p. 2, Figure 1 · p. 2
  • With one training example, model behavior is architecture-dependent, spanning memorization of a constant function to generalization via the identity function.The experiments examine how depth, initialization, and hyperparameters affect where models fall on this continuum.Source: §1 · p. 2

2 Related work

Prior work examined optimization and convergence, generalization under structural assumptions, implicit regularization, and memorization. This paper situates its broader study of inductive bias alongside work on CNN memorization with few examples.

2 Related work

3 Experiments

The experiments compare how fully connected and convolutional networks trained by gradient descent on one identity-mapping example behave on unseen inputs. Their inductive biases vary with architecture, depth, input scale, and other training choices.

3.1 Fully connected networks

  • One-layer FCNs produce training-output-like predictions near the training example but random outputs when its perpendicular component dominates.Deeper linear and ReLU FCNs instead show a strong bias toward the constant function, despite similar representational power in multilayer linear networks.Source: §3.1 · p. 4

3.2 Convolutional networks

  • The CNN error bound worsens with more parameters, while full rank in the span of local image patches guarantees learning the global identity map.The rank depends on training-image contents; increasing channels also increases parameter count, so the bound reflects a trade-off.Source: §3.2 · p. 5
  • Shallow CNNs learn an approximate identity map from one example, intermediate-depth CNNs behave like edge detectors, and very deep CNNs bias toward the constant function.The one-layer theorem guarantees identity learning only under a bound involving network parameters and the rank of local image patches.Source: §3.2 · pp. 5–6
  • CNN layer representations lose and then recover information, but very deep networks can learn a constant map when the input-output layer gap is large.The averaging-based classifier does not capture all preserved information, so its error is only a proxy for representation quality.Source: §3.2 · p. 6, Figure 5 · p. 6

3.3 Robustness to changes in input scale

  • A 5-layer CNN’s identity mapping generally holds for larger-than-trained inputs but is disturbed on smaller inputs; a 20-layer CNN’s constant map holds over a smaller size range.As input size increases, the deeper CNN’s generated pattern is defined relative to the input-map corners and grows inward from boundaries.Source: §3.3 · pp. 6–7, Figure 7 · p. 7

3.4 Varying other factors during training

  • Training on smaller images increases CNN bias toward the constant function at a given depth.The paper relates this to smaller images providing fewer pixel-to-pixel mapping constraints.Source: §3.4 · p. 7, Figure 8 · p. 7
  • Larger convolution filters produce blurrier predictions, and filters covering the entire input array create a strong bias toward the constant function.Even large filters differ from FCNs because shared convolution filters are repeatedly applied across spatial locations.Source: §3.4 · p. 8, Figure 9 · p. 8
  • Three-channel CNNs often lose image-center content despite reconstructing the training image, while more channels do not rescue 20-layer CNNs from learning a constant output.The authors suggest poor initialization as a possible reason for weak identity learning with very few channels; 1024-channel networks show no apparent overfitting.Source: §3.4 · pp. 8–9

4 Conclusions

The paper studies memorization and generalization under single-example overparameterization using the identity task to make trained behavior easier to visualize and interpret. Its results show that architectures have substantially different inductive biases; future work will extend the study to other domains and architectures.

4 Conclusions

  • Different architectures exhibit vastly different inductive biases toward memorization and generalization in the single-example, identity-task setting.Restricting the task sidesteps underlying Bayes error and hypothesis-class approximation error, while facilitating visualization and interpretation.Source: §4 · p. 9
  • Future work will extend the study to domains such as natural language processing and recurrent neural networks, and seek more qualitative relationships between inductive bias and architecture configurations.Source: §4 · p. 9

A Experiment details and hyper-parameters

The experiments use a single MNIST digit for training and evaluate identity mapping across architectures and test inputs. The appendix gives explicit constructions showing that the studied network types can represent the identity, while noting that training need not find those constructions.

A Experiment details and hyper-parameters

  • The default experiment uses one seeded MNIST digit for training and tests on MNIST, Fashion-MNIST, structured patterns, and random images.Images are normalized to [0, 1] and standardized using the full MNIST training-set mean 0.1307 and standard deviation 0.3081.Source: A Experiment details and… · p. 12
  • Training minimizes mean squared error with vanilla SGD for 2,000,000 steps, using learning rate 0.01, momentum 0.9, and no weight decay.The learning rate decays by a factor of 0.2 at 30%, 60%, and 80% of training.Source: A Experiment details and… · p. 12

B.1 Linear models

  • A linear network represents the identity by setting its weight matrix to the identity; in multilayer linear networks, hidden dimensions must be at least the input dimension for this construction.Source: §B.1 · p. 12

B.2 Multi-layer ReLU networks

  • A one-hidden-layer ReLU network can preserve positive and negative input parts separately using a hidden dimension twice the input dimension, then reconstruct the input.For deeper ReLU networks, the bottom layer splits the signs, the top merges them, and intermediate layers pass the nonnegative representation through.Source: §B.2 · p. 12

B.3 Convolutional networks

  • A convolutional layer can represent identity by using only the center kernel slice, equivalent to a 1 × 1 convolution that preserves each multichannel pixel.For multilayer convolutional ReLU networks, the construction uses twice as many hidden channels to split and then reconstruct positive and negative inputs.Source: §B.3 · p. 13

C Proof of Theorem 1

For a one-layer linear network trained on one example, gradient descent selects a particular global minimizer despite the non-unique empirical-risk solutions. Its predictions combine the training-example projection with a random projection of the orthogonal residual.

C Proof of Theorem 1

  • Although the one-example linear model has non-unique empirical-risk minimizers, random initialization and gradient descent yield a unique global minimizer.The one-layer optimization problem is convex, so under standard convex-optimization conditions gradient descent converges to that minimizer.Source: C Proof of Theorem 1 · pp. 13–14
  • The learned prediction combines the component of a test input parallel to the training example with a random projection of its orthogonal component.The red term is the projection onto the training example, and the blue term is the residual projection onto the orthogonal subspace.Source: C Proof of Theorem 1 · pp. 13–14

D Proof of Theorem 2

The proof analyzes one-layer convolutional identity learning through local image patches and derives prediction behavior under over- and underparameterization. Its bounds depend on patch-subspace rank, initialization, and parameter count.

D Proof of Theorem 2

  • In the overparameterized case, the proof assumes local patches are linearly independent and uses their span to express predictions for test pixels.The patch collection has one location per image pixel, and each target pixel occurs in its corresponding local patch.Source: D Proof of Theorem 2 · pp. 14–15
  • The prediction-error analysis depends on the rank and nullity of the projection onto the local-patch subspace, with pixel values assumed bounded by B.The proof also assumes i.i.d. Gaussian initialization; a chi-square tail bound controls the probability that a projected initialization component is large.Source: D Proof of Theorem 2 · pp. 15–16
  • The cited note states that the convolutional network has K_H K_W C^2 parameters, specifying its parameter count without stating any error-bound dependence or other effects.Source: D Proof of Theorem 2 · p. 16
  • In the underparameterized case with no redundant features, the proof states that the prediction is exact and the error is zero.This case applies when the number of patch locations exceeds K_H × K_W × C × C.Source: D Proof of Theorem 2 · p. 16

E Full results of fully connected multi-layer networks

Fully connected networks show that depth can matter more than parameter count, while jointly training both layers changes two-layer ReLU predictions from speckled noise toward the training image. With increasing depth, ReLU FCNs increasingly favor constant outputs, while CNN visualizations show distinct outcomes across depths.

E.1 Fully connected linear networks

  • Increasing depth affects linear FCN inductive bias more than width: a 3-hidden-layer network with 784 units behaves less like the convex case than a 1-hidden-layer network with 2,048 units.The wider network has 3.2M parameters, compared with 2.5M for the deeper network.Source: §E.1 · p. 17

E.2 Two-layer fully connected ReLU networks

  • Training both layers of a two-layer ReLU network shifts unseen-image predictions toward the training image, whereas training only the bottom layer tends toward speckled noise.When both layers are trained, the upper-layer weights depend on the bottom-layer initialization, so the fixed-randomness argument for orthogonal test examples no longer applies.Source: §E.2 · pp. 18–19

E.3 Nonlinear multi-layer fully connected networks

  • With increasing depth, ReLU FCNs bias toward the constant function sooner than linear FCNs, while neither reliably matches the identity on test images nearly orthogonal to the training example.Shallow networks instead produce random predictions on nearly orthogonal examples.Source: §E.3 · p. 19

F Visualization of the intermediate layer representations for CNNs

  • For CNN intermediate-layer visualizations, the authors flatten each layer's channel-by-height-by-width tensor, apply SVD, and display the top singular vector as a one-channel image.The visualizations are used to compare layer representations with inputs and outputs.Source: F Visualization of the intermediate… · p. 20
  • CNN intermediate activations distinguish depth-dependent outcomes: a 7-layer network approximately renders the identity, a 14-layer network fails at identity but recovers meaningful higher-layer information, and a 20-layer network constructs the training digit from scratch.The 20-layer network's lower layers receive updates, despite its output fitting the constant function.Source: F Visualization of the intermediate… · p. 20

G Measuring the change in weights of layers post training

The authors use relative changes in layer weights as a proxy for training updates and compare this measure across network depths. CNN lower-layer changes shrink with depth but remain nonzero, while very deep ReLU FCNs can have near-zero changes in lower layers even when shallower ReLU FCNs already favor constant outputs.

G Measuring the change in weights of layers post training

  • CNN lower-layer relative weight changes shrink as depth increases but remain nonzero, consistent with updates distinguishing even the 20-layer model's lower layers from random initialization.The distance proxy compares each layer's initialized and trained weight tensors.Source: G Measuring the change in weights of… · p. 21
  • In an 11-hidden-layer ReLU FCN, the lower five layers have near-zero weight changes, although ReLU FCNs begin favoring the constant function at only three hidden layers.The reported pattern indicates that constant-function bias can appear without the measured lower-layer changes being near zero.Source: G Measuring the change in weights of… · p. 21

H Full results on robustness of inductive biases

The appendix extends CNN tests across input sizes and isolates upper subnetworks to examine how their predictions behave. The 20-layer CNN's constant output persists across a wide range of input sizes, and its upper layers progressively construct the training digit.

H.1 Testing on different input image sizes

  • A trained 20-layer CNN's constant map persists across a wide range of input sizes, with boundary-padding artifacts apparently serving as cues to grow the digit pattern inward.The authors hypothesize that the last several layers realize this construction because shared filters and biases otherwise provide little spatial-location information.Source: §H.1 · p. 23

H.2 The upper subnetworks

  • When lower layers are skipped, the top-one layer's behavior changes clearly at 15 layers, where the networks begin shifting from edge-detector bias toward constant-function bias.Intermediate layers can be given test patterns directly for these visualizations.Source: §H.2 · p. 23
  • For the 20-layer CNN that learns the constant map, visualizing only the upper subnetworks shows that the upper half already begins constructing the digit “7” from nowhere.The figures compare the final two layers and progressively larger groups of upper layers.Source: §H.2 · p. 23

I Full results on varying different factors when training CNNs

CNN inductive bias varies with filter size, channel count, initialization, and optimizer, while an MSE-based appendix comparison remains broadly consistent with the main-text correlation plots. Large filters blur predictions and eventually favor constant outputs; restricted channels and optimization choices also alter learned behavior.

I Full results on varying different factors when training CNNs

  • As convolution filters grow from 5 × 5 to 57 × 57, predictions become blurrier, and filters covering the whole 28 × 28 input begin favoring the constant function.The authors note that repeated spatial application of shared filters remains, so very large filters do not make CNN computation trivially equivalent to FCNs.Source: I Full results on varying different… · p. 26
  • A 5-layer CNN with only three channels per convolution layer fails to approximate the identity function.The visual and correlation results agree on this failure.Source: I Full results on varying different… · p. 28
  • With three channels, CNNs deeper than eight layers fail to converge, while 5- and 7-layer models favor edge- or contour-finding and 6- and 8-layer models show different biases.The authors suggest as a potential reason that few channels may leave initialization “unlucky” cases insufficiently smoothed; 128-channel networks appear smoother at initialization.Source: I Full results on varying different… · p. 28
  • All tested initialization schemes favor identity functions in shallow networks, but Kaiming initialization produces heavy test-prediction artifacts and is less interpretable for deep networks.Xavier initialization behaves similarly to the default for deep constant-function bias, though more layers are needed for a visually good identity function.Source: I Full results on varying different… · p. 28
  • Different optimizers find different global minimizers with drastically different inductive biases on test examples.Some optimizers require smaller base learning rates to avoid parameter explosion during training.Source: I Full results on varying different… · p. 28

J Correlation vs MSE

  • MSE plots are overall consistent with corresponding correlation plots, but correlation displays patterns more clearly and has a fixed [0, 1] range.Source: J Correlation vs MSE · p. 31

K Robustness of observations to random seeds

The reported CNN patterns persist across random seeds and different single training images, while additional experiments examine more training examples, parameter counts, and residual FCNs.

K Robustness of observations to random seeds

L Learning with multiple examples

  • With 60k examples, deep CNNs learn identity, while FCNs learn it only on digit inputs and fail to reconstruct meaningful results on other test patterns.With two or three examples, networks prone to constant-map overfitting still memorize the training digits and predict mixtures of them on test inputs.Source: Figure 38 · p. 33, Figure 39 · p. 34, L Learning with multiple examples · p. 35

M Parameter counts for neural networks used in this paper

N Residual Networks

  • Identity skip connections bias FCNs toward the identity map, but prediction noise increases with depth despite the explicit skips.Each two-layer block adds an identity skip; the residual networks use 784 hidden dimensions to preserve matching input-output shapes.Source: N Residual Networks · p. 36

O Experiments on CIFAR-10 Images

On two CIFAR-10 training images, the observed architecture-dependent patterns resemble those on MNIST: shallow CNNs generalize, while deeper CNNs and FCNs show different forms of memorization.

O Experiments on CIFAR-10 Images

Related papers

Loading 1902.04698v4…