PaperlayerView exact revision on arXiv

Source-linked AI summary

An analytic theory of generalization dynamics and transfer learning in deep linear networks

Andrew K. Lampinen, Surya Ganguli

arXiv:1809.10374v2stat.MLcs.LG

TL;DR

Deep learning lacks tight analytic explanations for good generalization and quantitative accounts of transfer across related tasks. The paper develops an analytic theory for deep linear networks and finds that early generalization is governed mainly by task structure, while transfer depends on task SNRs and feature alignment.

  • Problem

    Existing generalization bounds are exceedingly loose, and no analytic theory quantitatively predicts how knowledge transfer depends on relationships between tasks.

  • Method

    The paper develops an analytic theory of training and test-error dynamics in deep linear networks within and across tasks.

  • Results

    The theory finds that networks learn important task structure first, while transfer depends sensitively but computably on task SNRs and feature-space alignments.

  • Takeaways & Limitations

    Tight generalization theories should account for task structure, and auxiliary tasks may be selected using analytically predicted transfer relationships.

  • Takeaways & Limitations

    The main results assume a student-teacher setting with noisy low-rank teachers and, in the main exposition, one hidden layer and orthonormal training inputs.

Abstract

from arXiv · show

Much attention has been devoted recently to the generalization puzzle in deep learning: large, deep networks can generalize well, but existing theories bounding generalization error are exceedingly loose, and thus cannot explain this striking performance. Furthermore, a major hope is that knowledge may transfer across tasks, so that multi-task learning can improve generalization on individual tasks. However we lack analytic theories that can quantitatively predict how the degree of knowledge transfer depends on the relationship between the tasks. We develop an analytic theory of the nonlinear dynamics of generalization in deep linear networks, both within and across tasks. In particular, our theory provides analytic solutions to the training and testing error of deep networks as a function of training time, number of examples, network size and initialization, and the task structure and SNR. Our theory reveals that deep networks progressively learn the most important task structure first, so that generalization error at the early stopping time primarily depends on task structure and is independent of network size. This suggests any tight bound on generalization error must take into account task structure, and explains observations about real data being learned faster than random data. Intriguingly our theory also reveals the existence of a learning algorithm that proveably out-performs neural network training through gradient descent. Finally, for transfer learning, our theory reveals that knowledge transfer depends sensitively, but computably, on the SNRs and input feature alignments of pairs of tasks.

1 INTRODUCTION

The paper develops an analytic theory of generalization and transfer in deep linear networks to address loose generalization bounds and quantify cross-task knowledge transfer.

  • 1 INTRODUCTION: The theory analytically characterizes training and test error as functions of time, examples, architecture, initialization, task structure, and SNR.It is developed for deep linear networks, whose learning dynamics exhibit nonlinear phenomena despite linear input-output maps.
  • 1 INTRODUCTION: Small-initialization networks learn important task structure first, making early-stopping test error depend mainly on task structure and SNR rather than architecture.This holds when the architecture is expressive enough to attain small training error.
  • 1 INTRODUCTION: The theory identifies a non-gradient-descent learning algorithm that provably outperforms gradient-descent training in the linear case.
  • 1 INTRODUCTION: For transfer learning, knowledge transfer depends sensitively but computably on the SNRs and feature-space alignments of task pairs.The framework extends beyond single-output settings that had prevented analysis of transfer learning in related work.

2 THEORETICAL FRAMEWORK

The framework models noisy low-rank teacher networks and student networks, expressing training and test errors through their evolving singular modes and the teacher and data structure.

  • 2.1 AN ENSEMBLE OF LOW-RANK NOISY TEACHERS: The framework uses low-rank noisy teacher networks to generate training data for potentially more complex student networks.Teacher singular values represent signal-to-noise ratios after setting the noise scale to one.
  • 2.1 AN ENSEMBLE OF LOW-RANK NOISY TEACHERS: A teacher computes a composite linear map y = Wx from input to output through input-hidden and hidden-output weight matrices.The teacher map is analyzed through the singular value decomposition of W.
  • 2.1 AN ENSEMBLE OF LOW-RANK NOISY TEACHERS: Teacher outputs are generated from orthonormal inputs with additive Gaussian noise, producing training statistics that guide student learning.The noise matrix has independently drawn entries, while the input covariance is assumed white.
  • 2. THEORETICAL FRAMEWORK: The main exposition assumes one hidden layer and a full orthonormal basis of training inputs, while the theory also extends to arbitrary depth and white inputs with P ≠ N1.
  • 2.2 STUDENT TRAINING AND TEST ERROR: The student may have more hidden units than the teacher, and its training and test errors are defined on noisy training data and new isotropic Gaussian inputs, respectively.The two errors can be expressed through the student, training-data, and teacher singular decompositions.
  • 2.2 STUDENT TRAINING AND TEST ERROR: Error dynamics depend on how the student’s time-evolving singular modes align with fixed training-data and teacher singular modes.This alignment distinguishes training-aligned learning from alignment with the teacher’s task structure.

3 SINGLE TASK GENERALIZATION DYNAMICS: THEORY AND EXPERIMENT

The theory explains generalization dynamics through singular-mode learning: stronger task modes are learned first, while training and test errors evolve through mode-dependent transitions, plateaus, and overfitting. Theory matches simulations across shallow and deep networks, and structured data learns faster because its signal singular values are encountered earlier than those of randomized data.

  • 3.2 HOW THE TEACHER IS BURIED IN THE TRAINING DATA: A RANDOM MATRIX ANALYSIS: Teacher modes cross a phase transition at s = A1/4: above it, noisy singular values separate from the MP sea and align with teacher singular vectors.Below the threshold, the mode is not imprinted in the noisy training data and the singular-vector overlap remains zero.
  • 3.3 PUTTING IT ALL TOGETHER: AN ANALYTIC THEORY OF GENERALIZATION DYNAMICS: The complete theory expresses training and test errors through task structure, student architecture, initialization, and training time.It incorporates teacher rank, SNRs, aspect ratio, student rank, depth, initialization scale, and time.
  • 3.1 STUDENT TRAINING DYNAMICS AND TRAINING-ALIGNED (TA) NETWORKS: Learning proceeds as a detection wave that moves from large to small training-data singular values, producing sequential error drops across teacher modes.Rank-1 teachers show one early drop at sufficient SNR, whereas rank-3 teachers show several drops as modes are learned.
  • 3.3 PUTTING IT ALL TOGETHER: AN ANALYTIC THEORY OF GENERALIZATION DYNAMICS: Theory and simulations closely match for training and test errors, minimum generalization errors, and optimal stopping times in single-hidden-layer students.Random students show a small alignment delay, but their optimal test errors match those of training-aligned students.
  • 3.4 NUMERICAL TESTS OF THE THEORY OF NEURAL NETWORK GENERALIZATION DYNAMICS: For five-layer students, theory again matches training-aligned simulations, while random initialization produces a larger alignment lag but comparable optimal stopping errors.The deeper-network extension retains the qualitative learning dynamics while slowing alignment for randomly initialized networks.
  • 3.5 RANDOMIZED DATA VS. REAL DATA: A LEARNING TIME PUZZLE: Randomized data learns more slowly because signal singular values are diluted across many modes, delaying the detection wave relative to structured data.Randomization preserves basic statistics but destroys the outlier signal singular values associated with the teacher.

4 A THEORY FOR THE TRANSFER OF KNOWLEDGE ACROSS MULTIPLE TASKS

The paper defines transfer benefit by comparing optimal early-stopping errors for separate versus jointly trained tasks, then derives its dependence on task SNR and input-feature alignment. Transfer can help weak tasks but can interfere with already strong tasks unless alignment is high.

  • Transfer benefit: The transfer benefit T_A←B compares task A's optimal test error when trained jointly with B against training A alone.Positive benefit means joint learning lowers task A's optimal test error; negative benefit means it raises it.
  • Task relationship: The theory expresses transfer through the teachers' singular structures and a similarity matrix Q describing alignment between their relevant input feature subspaces.The derivation considers teacher SVDs and uses Q to capture input-feature similarity.
  • Task relationship: Transfer depends on relevant input features rather than on the tasks' output responses.The transfer benefit is independent of the output singular vectors in this setting.
  • Transfer regimes: Knowledge can flow from a high-SNR task to a low-SNR task, with greater transfer as task alignment increases.Even weak alignment can reveal task-A signals that would otherwise sink into the noise spectrum.
  • Transfer regimes: For a high-SNR target task, transfer is beneficial only when the auxiliary task is very well aligned; otherwise interference occurs.The required alignment rises with target SNR, while the magnitude of benefit or interference decreases.

5 DISCUSSION

The discussion argues that deep linear networks analytically reproduce major generalization puzzles and clarify how task structure, training dynamics, and transfer shape learning. It also identifies non-gradient training and task selection as directions enabled by the theory.

  • Discussion: The theory shows that deep linear networks learn important task structure first, making early-stopping generalization depend more on task structure than network size.This motivates tighter bounds that incorporate task structure rather than architecture alone.
  • Discussion: The analytic framework explains why structured data are learned faster than randomized data and identifies a non-gradient method that outperforms gradient descent in the linear case.These results connect learning dynamics to observed differences between structured and random data.
  • Implications: The paper proposes theory-driven selection of auxiliary tasks to maximize transfer, alongside searches for tighter task-structure-aware bounds and non-gradient algorithms.These are presented as directions motivated by the analytic results.
  • Learning dynamics: The appendix analyzes mode dynamics by integrating the strength equation for arbitrary integer depth, with numerical inversion required for the five-layer case.The five-layer setting corresponds to three hidden layers.
  • Learning dynamics: Randomly initialized networks lag teacher-aligned networks because their modes must first align with the data modes.The lag is larger in deeper networks because alignment is substantially slower and may remain incomplete when singular values begin increasing.

C TRAIN AND TEST ERRORS AFTER A PROJECTION

The appendix generalizes the train and test error formulas to losses evaluated after applying a projection operator. It rewrites the projected errors using trace expressions and the student, data, and teacher singular-value decompositions.

  • Projected errors: Projected train and test errors evaluate network loss after applying a projection operator P to the outputs.This extension is needed for transfer learning and losses restricted to subsets of outputs.
  • Projected errors: The projected training error is rewritten as a trace expression involving the teacher, projection, student estimate, and training-data covariance.The displayed formula gives the corresponding quadratic trace terms.
  • Projected errors: The projected test error is likewise expressed through trace terms for the teacher and student under the output projection.The appendix then transforms both errors using the cyclic property of the trace.
  • Derivation: The derivation expresses the projected errors in terms of the student, training-data, and teacher singular-value decompositions.The relevant decompositions are introduced before the final transformed expressions.

D TRANSFER LEARNING DERIVATIONS & DETAILS

The transfer theorem reduces multi-task transfer to the relationship between task input modes and singular values. Output-mode orientation does not affect transfer, while shared input structure can improve one task by suppressing task-specific distinctions.

  • Transfer theorem: The transfer theorem identifies the transfer benefit as computable from the tasks' singular structures and their input-mode similarity.The similarity matrix Q represents overlap between the low-dimensional input feature spaces important to the tasks.
  • Output-mode independence: The transfer benefit is unaffected by the output singular vectors of the two tasks.Orthogonal transformations of output modes leave transfer unchanged, so output responses do not determine transfer here.
  • Mode alignment: Figure 9 shows that deeper randomly initialized networks align their modes more slowly, with alignment sometimes incomplete as singular values begin increasing.The figure compares one- and three-hidden-layer networks across rank-one and rank-three teachers.
  • Input-mode dependence: Transfer effects are driven by the relationship between the tasks' input singular modes and their singular values.The derivation shows that the relevant input-feature alignment, weighted by signal strengths, determines transfer.
  • Rank-one interpretation: In the rank-one case, the combined task structure separates into shared and differing modes, with alignment q controlling their relationship.The sign of q is irrelevant because output-mode orthogonal transformations do not affect transfer.
  • Interference: Joint learning can strengthen shared structure while pushing task-specific distinctions toward noise, so knowledge transfer comes at the cost of ignoring differences between tasks.Poor alignment therefore creates interference by changing the effective input singular dimensions of the target task.

E NON-GRADIENT TRAINING ALGORITHM

The non-gradient shrinkage algorithm predicts optimal stopping error for a rank-1 teacher, matching empirical results from gradient-descent training.

  • The predicted optimal stopping error closely matches the error achieved by gradient descent for a rank-1 teacher.Figure 11 reports the comparison between the non-gradient prediction and empirical gradient-descent performance.

F TRANSFER RESULTS GENERALIZE TO NON-LINEAR NETWORKS

Transfer patterns from linear networks qualitatively generalize to nonlinear teachers and students. Benefits depend on task SNR and alignment, while strong-SNR tasks can experience interference unless alignment is high.

  • Transfer patterns qualitatively generalize from rank-1 linear teachers to higher-rank nonlinear teachers and students.The nonlinear experiments use leaky-ReLU teacher and student networks and report results qualitatively similar to the linear case.
  • Low-SNR tasks improve with support from aligned tasks, especially when the supporting task has moderately higher SNR.The benefit is strongest for aligned auxiliary tasks and can enable learning signals that would otherwise be lost.
  • Moderate-SNR tasks benefit from well-aligned tasks but can suffer interference from poorly aligned tasks, with effects amplified by SNR.The reported transfer benefit varies with both alignment and the signal strengths of the two tasks.
  • High-SNR tasks show little transfer effect, and any effect is negative unless the auxiliary task is very well aligned.The high-SNR case has a different scale and is comparatively insensitive to other tasks except under close alignment.

G VARYING THE NUMBER OF TRAINING EXAMPLES

Varying the number of training examples changes both effective signal strength and learning speed in oversampled data, while undersampled data introduces projection and frozen-subspace effects.

  • The analysis varies P using noisy teacher outputs, unit-norm inputs, and approximately white input covariance, treating oversampled and undersampled regimes separately.The construction uses orthogonalized inputs as an analytically convenient approximation to iid unit-norm Gaussian inputs.
  • When P ≥ N1, minimum generalization error is determined by SNR, whereas for P < N1, SNR-dependent projection and noise effects separate the curves.The undersampled regime retains dependence on initial SNR because the training inputs do not span the full input space.
  • In the oversampled regime, increasing data density boosts effective SNR and speeds learning by a multiplicative factor of D.The test-error curves for D > 1 can be obtained from the D = 1 theory using an SNR boost and a learning-time rescaling.
  • In the undersampled regime, learning is confined to the training-input subspace, leaving an untrained complement that contributes to generalization error.The projection operator onto the P-dimensional input span determines the learned component, while the remaining subspace is frozen.

G.3 COMPARISON OF THEORY AND EXPERIMENT FOR UNDER AND OVER SAMPLED MEASUREMENT REGIMES

Theory and simulation agree across undersampled and oversampled regimes, while the number of examples improves minimum error without necessarily improving infinite-time error.

  • The theory matches empirical simulations across varying P in both oversampled and undersampled measurement regimes.The comparison covers the regimes where the data density D is above or below one.
  • Minimum generalization error improves monotonically with P, but asymptotic error need not because undersampled training leaves a frozen subspace.Modes outside the training-data span are not overfit when P < N1, so increasing examples affects finite- and infinite-time behavior differently.
  • Approximately unit-norm Gaussian inputs produce results similar to the orthogonalized inputs used in the theory, with slightly higher optimal stopping error.This supports the relevance of the orthogonal-input analysis beyond its exact analytical setting.

H LESS THAN FULL RANK STUDENTS

The theory remains exact for TA networks of any rank, and reducing student rank leaves optimal stopping generalization error essentially unchanged while delaying optimal stopping for random networks.

  • H LESS THAN FULL RANK STUDENTS: The theory remains exact for TA networks of any rank, despite assuming full-rank students in the main text.
  • H LESS THAN FULL RANK STUDENTS: Reducing random-network rank increases the lag in optimal stopping time relative to TA networks.Lower-rank random networks have less initial projection onto random modes, requiring more alignment.
  • H LESS THAN FULL RANK STUDENTS: Optimal stopping generalization errors are almost identical across random and aligned initializations and different student ranks.Figure 14 compares minimum generalization errors across initialization types and student ranks.
Loading 1809.10374v2…