On the Theory of Transfer Learning: The Importance of Task Diversity

Nilesh Tripuraneni; Michael I. Jordan; Chi Jin

Source-linked AI summary

On the Theory of Transfer Learning: The Importance of Task Diversity

Nilesh Tripuraneni, Michael I. Jordan, Chi Jin

arXiv:2006.11650v2Jun 20, 2020cs.LGstat.ML

TL;DR

The paper asks how shared representations can be learned across tasks and used to reduce the data required for a new task. It develops a two-stage ERM framework with a general task-diversity notion and Gaussian-complexity chain rule, showing that diverse tasks support representation learning and low-complexity adaptation. The framework applies across several multitask models, while its guarantees rely on assumptions such as common design and realizability.

Problem
The paper addresses the limited theoretical understanding of how many samples are needed to learn a shared representation across tasks and improve prediction on a new task.
Method
The paper analyzes a two-stage ERM procedure that learns a shared representation from t training tasks and then learns a new task using that representation and fresh samples.
Results
For diverse training tasks, representation-learning complexity scales as C(H) + t C(F), while learning a new task with an accurate representation scales only with C(F).
Takeaways & Limitations
Task diversity and common design enable fast transfer-learning convergence across all samples, with guarantees covering general losses, tasks, and features.
Takeaways & Limitations
The stated results assume common design and realizability, whose relaxation is identified as future work; one index-model example also incurs a bias term ˜𝜖 for robust transfer beyond F_t.

Abstract

from arXiv · show

We provide new statistical guarantees for transfer learning via representation learning--when transfer is achieved by learning a feature representation shared across different tasks. This enables learning on new tasks using far less data than is required to learn them in isolation. Formally, we consider $t+1$ tasks parameterized by functions of the form $f_j \circ h$ in a general function class $\mathcal{F} \circ \mathcal{H}$, where each $f_j$ is a task-specific function in $\mathcal{F}$ and $h$ is the shared representation in $\mathcal{H}$. Letting $C(\cdot)$ denote the complexity measure of the function class, we show that for diverse training tasks (1) the sample complexity needed to learn the shared representation across the first $t$ training tasks scales as $C(\mathcal{H}) + t C(\mathcal{F})$, despite no explicit access to a signal from the feature representation and (2) with an accurate estimate of the representation, the sample complexity needed to learn a new task scales only with $C(\mathcal{F})$. Our results depend upon a new general notion of task diversity--applicable to models with general tasks, features, and losses--as well as a novel chain rule for Gaussian complexities. Finally, we exhibit the utility of our general framework in several models of importance in the literature.

1 Introduction

The paper develops statistical guarantees for transfer learning through shared representations across diverse tasks, addressing how much data is needed to learn the representation and adapt to a new task. Its framework applies to general losses, tasks, and features, and is illustrated across several multitask models.

Motivation: Transfer learning studies how shared representations can reduce the data needed to learn new tasks compared with learning them in isolation.The motivating model assumes differing tasks share a common low-dimensional representation while retaining task-specific parameters.
Method: The proposed two-stage ERM first learns a shared representation from t training tasks with n samples each, then learns a new task from m fresh samples.The training phase uses all nt training samples, while the test phase combines the learned representation with new-task data.
Results: With abundant training-task data, learning a new task requires m ≈ C(F) fresh samples, whereas isolated learning requires m ≈ C(F ◦ H).The transfer advantage follows when learning both task and representation is substantially more complex than learning the task-specific component alone.
Framework: The framework introduces a problem-agnostic notion of task diversity for general losses, tasks, and features, combined with a common-design assumption.These ingredients yield guarantees whose convergence rate decays with all samples in the transfer-learning problem.
Theory: The analysis decouples the complexity of learning task-specific functions from that of learning the shared representation through a novel chain rule for Gaussian complexities.The resulting modularity permits separate sample-complexity bounds for task and feature functions.
Applications: The framework is applied to multitask logistic regression, deep neural network regression, and robust regression for single-index models.The examples demonstrate end-to-end transfer-learning guarantees across several model classes.

2 Preliminaries

The paper formalizes transfer learning with a shared nonlinear representation and task-specific mappings, using a two-phase ERM procedure. Gaussian complexity measures function-class complexity, while the transfer risk evaluates prediction on a new task.

Shared-representation model: Tasks share a representation h in H and use task-specific mappings f_j in F, with a common marginal distribution over inputs.The true representation and task mappings are assumed to belong to H and F, respectively.
Two-phase procedure: Training learns the shared representation from t tasks with n samples each, using all nt training samples.The test phase uses m fresh samples from a new task together with the learned representation.
Two-phase procedure: The training estimator jointly minimizes empirical risk over the representation class H and t task functions in F.The learned representation is the representation component of the joint training-phase ERM.
Two-phase procedure: The new-task estimator minimizes empirical risk over F while holding the learned representation fixed.Its performance is measured by expected excess prediction risk on a fresh example from the new task.
Model complexity: Gaussian complexity measures how strongly functions in a class can correlate with random noise on sampled data.The paper defines both empirical and population Gaussian complexities for vector-valued function classes.

3 Main Results

The main results analyze representation learning, transfer to new tasks, and the role of diverse training tasks. They provide generalization guarantees whose complexity terms separate shared representations from task-specific mappings.

3.1 Learning shared representations: Theorem 1 bounds the task-averaged distance between the learned and true representations under regularity and realizability assumptions.The bound uses the Gaussian complexity of the composed class F⊗t ◦ H plus a lower-order noise term.
3.1 Learning shared representations: The representation-learning complexity decomposes into contributions from H and F through a novel chain rule for Gaussian complexities.This decomposition enables separate bounds for feature and task-function classes.
3.2 Transferring to new tasks: Theorem 2 separates new-task excess risk into representation-transfer error and the complexity of learning the new task-specific mapping.The representation term captures bias from using an estimated representation, while the second term depends on learning f_0 from m samples.
3.3 Task diversity and end-to-end transfer learning guarantees: Task diversity ensures that small task-averaged representation error controls worst-case representation error for the new-task class up to additive error ε.The definition applies beyond the linear-function, quadratic-loss setting and recovers earlier special-case notions.
3.3 Task diversity and end-to-end transfer learning guarantees: Theorem 3 bounds end-to-end transfer risk for (ν, ε)-diverse training tasks under the paper’s regularity and realizability assumptions.The bound is driven by three Gaussian-complexity terms.
3.3 Task diversity and end-to-end transfer learning guarantees: When n and t are sufficiently large and m is small, transfer risk scales with C(F)/m rather than the isolation baseline ˜O((C(H) + C(F))/m).The shared-representation complexity is learned from the training tasks, improving few-shot learning relative to learning both H and F from new-task data.

4 Applications

The framework is instantiated across logistic regression, deep neural-network regression, and index models, yielding transfer guarantees under task-diversity assumptions. These applications show savings from learning shared representations, while also exposing model-specific scope and complexity limitations.

Multitask Logistic Regression: Logistic regression uses linear task maps and a low-dimensional projection representation, with task diversity quantified by feature-space conditioning.The diversity parameter is ˜ν = σr(A), and well-conditioned task vectors span the r-dimensional representation space.
Multitask Logistic Regression: For logistic regression, the transfer guarantee improves over a naive O(d/m) bound when r ≪ d.The result also admits sharper empirical, data-dependent quantities when the data are close to a low-dimensional subspace.
Multitask Deep Neural Network Regression: Deep neural-network regression uses linear task maps over a depth-K vector-valued network representation, with Gaussian complexities bounded using network norm and width parameters.The framework combines these complexity bounds with task diversity to produce an end-to-end transfer guarantee.
Multitask Deep Neural Network Regression: The neural-network guarantee is dominated by the overhead of learning the complex feature map, while the final-layer poly(M(K)) dependence may be improvable.The stated limitation concerns the dependence on final-layer weights and the feature-map complexity.
Multitask Deep Neural Network Regression: When nt ≫ m, the transfer bound can be much smaller than the naive algorithm’s sample complexity because the latter ignores training-task samples.This is the few-shot regime for which ImageNet pretraining is often used.
Multitask Index Models: Index-model transfer can generalize beyond the training-task class, but this broader scope incurs a bias term ˜ε.The difficulty of the transfer problem also increases with t because the target class grows with t.

5 Conclusion

The paper presents a general framework for transfer learning with shared representations and identifies task diversity as central to fast convergence. It also points to relaxing common-design and realizability assumptions and supporting fine-tuning as future directions.

The framework analyzes generalization for generic models that share a common underlying representation.
Task diversity supports guarantees whose transfer-learning convergence rate decays with all available samples.
Future work includes relaxing common-design and realizability assumptions and extending the theory to fine-tuning learned representations.

Appendices

The appendices define the composition notation, data matrices, empirical covariance, and Gaussian and Rademacher complexities used throughout the analysis.

The paper uses F for scalar-valued task functions, H for vector-valued features, and F(H) for their composed function class.
The t-task class F⊗t is the Cartesian product of t task-function classes, with vector-valued compositions defined analogously.
The concatenated training-task design matrix is X, while Xj and yj denote the design matrix and responses for task j.
For a mean-zero data matrix X̄, the empirical covariance is ΣX̄ = 1/N X̄ᵀX̄.
The appendices define empirical and population Gaussian complexities, together with analogous empirical and population Rademacher complexities, for vector-valued classes.

A Proofs in Section 3

The appendix contains proofs of the paper’s central generalization guarantees and the Gaussian-process chain rule used in those proofs.

The proofs establish the central generalization guarantees and the Gaussian-process chain rule underlying their analysis.

A.1 Training Phase/Test Phase Proofs

The proofs establish training-phase control of representation error and test-phase generalization with a fixed learned representation, then combine these results into transfer guarantees.

Training Phase: Training ERM controls the task-average distance between the underlying and learned feature representations.The proof begins by relating the training objective to representation error.
Test Phase: The corollaries provide guarantees with probability at least 1 −2δ and, for the combined variants, at least 1 −4δ.These probability levels accompany the stated population and empirical complexity bounds.
Training Phase: The Gaussian-complexity analysis decomposes the composed class into representation and task-class complexity terms.The proof uses centering, contraction, Rademacher-to-Gaussian comparison, and the Gaussian-complexity chain rule.
Test Phase: Test-phase analysis decomposes excess risk into estimation terms and bias from using an imperfect representation.The learned representation is treated as fixed and independent of test-data randomness.
Test Phase: Theorem 3 combines the training and test guarantees with task diversity and a union bound to obtain an end-to-end result.The same framework also provides population and empirical-complexity variants, including a Bernstein-based test-phase guarantee.

A.2 A User-Friendly Chain Rule for Gaussian Complexity

The paper introduces a Gaussian-complexity chain rule that separates the complexity of learning task functions from that of learning a shared representation. The proof uses covering-number decompositions to handle the joint supremum over both function classes.

Theorem 7: Theorem 7 bounds the Gaussian complexity of F⊗t(H) using L(F)·ˆG_X(H) plus the worst-case task-class complexity over representation outputs.The decomposition is expressed through C(F⊗t(H)) = L(F)·ˆG_X(H) + max_Z∈Z ˆG_Z(F).
Motivation: The chain rule generalizes contraction because both the task functions and shared representation are optimized rather than one being fixed.The composed Gaussian complexity therefore contains a supremum over F⊗t and H.
Comparison: The proof differs from Maurer’s related result by using entropy integrals and Gaussian-process lower bounds rather than a purely Gaussian-like worst-case average.The related result can be tighter in logarithmic factors but is not expressed purely in Gaussian complexities.
Proof Strategy: Covering H at scale ϵ1 and F⊗t at scale ϵ2 yields a composed cover at scale ϵ1·L(F) + ϵ2.The Lipschitz property transfers representation approximation error into the composed-function metric.
Proof Strategy: The composed metric entropy is bounded by the representation entropy plus t times the maximum task-class entropy over representation outputs.This is the key decoupling step in the covering-number argument.

B Proofs in Section 4

The section instantiates the general transfer-learning framework in several concrete models by verifying task diversity and bounding the complexity terms required by the end-to-end guarantee.

Proofs in Section 4: Each example proceeds by establishing a task-diversity lower bound and then bounding the complexity terms in the transfer guarantee.The instantiated results use Theorem 3 or its variants.

B.1 Logistic Regression

The logistic-regression analysis verifies the assumptions and task-diversity condition needed to instantiate the general transfer-learning theory. It derives training and testing complexity bounds and combines them into a transfer-risk guarantee.

Transfer Guarantee: Theorem 4 applies the diversity result and the complexity corollaries to obtain the logistic-regression transfer guarantee with probability at least 1−2δ.The proof combines the training and testing bounds and uses data-dependent empirical covariance quantities for sharper guarantees.
Model and Assumptions: O(1)-Lipschitz logistic loss and O(D) loss boundedness supply the regularity parameters used in the general bounds.The Lipschitz claim follows from the derivative of the logistic loss under the stated assumptions.
Data-Dependent Guarantee: Corollary 3 gives a sharp data-dependent logistic-regression guarantee when the covariance has bounded maximum and positive minimum eigenvalues.The result assumes diverse training tasks and a shared representation in the specified function class.
Task Diversity: Ω(˜ν) task diversity is obtained when the task-vector conditioning parameter satisfies ˜ν = σr(A⊤A/t) under the stated coefficient condition.The proof uses smoothness and local strong convexity to relate logistic-loss quantities to representation distance.
Model and Assumptions: The logistic model uses h(x) = Bx with orthonormal representation columns and task maps parameterized by α.The analysis assumes bounded task parameters and a covariate distribution satisfying a sub-Gaussian condition.

B.2 Deep Neural Network Regression

This section develops complexity bounds for deep neural network transfer learning and a general task-diversity characterization for shared representations. It then specializes the framework to squared-loss regression with neural-network features.

Deep neural network complexity: The analysis assembles Gaussian-complexity bounds for feature learning and task-specific maps in the training and testing phases.The bounds use existing neural-network complexity results, norm constraints, and a chain-rule-based transfer guarantee.
Deep neural network complexity: Under norm constraints, neural-network feature outputs and task-map predictions are bounded by powers of the network norm parameter M(K).The feature-map norm is controlled by the network layers, while linear task maps contribute a complexity scaling with M(K)^2.
Deep neural network complexity: The resulting transfer-learning risk bound combines feature-learning, task-learning, Lipschitz, and uniform-loss-boundedness terms.For squared ℓ2 loss, the analysis uses L = O(M(K)^3) and B = O(M(K)^6).
Task diversity: The general task-diversity result applies to ℓ2 regression with generic feature maps and shared representations, not only neural-network features.The regression model uses task-specific parameters, a shared representation, and independent noise with bounded second moments.
Task diversity: For squared-loss regression, task diversity is characterized through the smallest singular value and trace of a generalized Schur complement.The worst-case representation difference is c2 · σ1(Λsc(ĥ, h⋆)), while the task-averaged difference is tr(Λsc(ĥ, h⋆)C).
Task diversity: Well-conditioned learned representations preserve the same diversity conclusions for sufficiently large task-parameter bounds.The conditions require a positive lower bound on the learned representation covariance and a bounded cross-covariance with the true representation.

B.3 Index Models

This section gives a generic task-diversity bound for index-model regression with convex task classes and broad norm-based losses. It also explains when the worst-case diversity guarantee can be conservative.

Index-model guarantees: The end-to-end guarantee is applied using the sample covariance of the training-phase data and separate complexity bounds for feature and task learning.The task-class complexity is bounded using the Dudley entropy integral and a metric-entropy estimate.
Index-model guarantees: Lemma 7 establishes (ν̃, ε̃)-diversity for regression when the task class is convex and F0 is its ε̃-enlargement.The result applies to regression models with losses of the form L(a − b), where L is positive, increasing, and obeys a triangle inequality.
Index-model guarantees: The generic diversity result encompasses multi-index models and supports a wide class of regression losses, including all p-norm losses.Only mild moment-boundedness conditions on the noise are required to ensure finiteness of the objective.
Proof strategy: The proof obtains the generic bound by combining convexity, enlargement in sup-norm, partial minimization, and the relationship between ℓ∞ and ℓ1 norms.The supremum over a convex hull is reduced to its generating task functions.
Interpretation: The stronger guarantee ν̃ ≥ ∥v∥1/∥v∥∞ can exceed the baseline bound when task-specific representation differences are well spread across tasks.In that case, the paper states that transfer may be more sample-efficient in practice than the worst-case bound suggests.
Limitations: Formalizing the well-spread-task notion cleanly for the nonparametric function classes considered remains difficult.The text also characterizes the simpler bound as potentially pessimistic.