PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptation

Haoxiang Wang, Han Zhao, Bo Li

arXiv:2106.09017v1cs.LGcs.CVstat.ML

TL;DR

MTL trains related tasks jointly, whereas meta-learning supports fast adaptation to unseen tasks, and the relationship between these approaches remains insufficiently explored. This paper analyzes their connection for a class of GBML methods and evaluates MTL theoretically and empirically. It finds similar predictions under stated network conditions and competitive few-shot performance with substantially faster training.

  • Problem

    The connection between MTL and meta-learning is insufficiently explored, despite their shared use of task relationships; MTL trains efficiently but does not straightforwardly adapt to unseen tasks, while meta-learning adaptation can be computationally costly.

    Source: §1 · p. 1

  • Method

    The paper compares MTL with a class of GBML algorithms through shared optimization analysis, predictor-closeness theory, and empirical evaluation.

    Source: §1 · pp. 1–2

  • Results

    MTL is competitive against state-of-the-art GBML algorithms on few-shot benchmarks while being significantly more efficient, and sufficiently wide networks yield similar predictions on unseen tasks.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The results indicate that fast adaptation to unseen tasks and efficient training can be achieved simultaneously with an MTL approach.

    Source: §6 · p. 9

    The theoretical result uses fully connected networks and does not directly apply to modern ConvNets with residual connections, max pooling, BatchNorm, and Dropout.

    Source: §5.1 · p. 7

Abstract

from arXiv · show

Multi-task learning (MTL) aims to improve the generalization of several related tasks by learning them jointly. As a comparison, in addition to the joint training scheme, modern meta-learning allows unseen tasks with limited labels during the test phase, in the hope of fast adaptation over them. Despite the subtle difference between MTL and meta-learning in the problem formulation, both learning paradigms share the same insight that the shared structure between existing training tasks could lead to better generalization and adaptation. In this paper, we take one important step further to understand the close connection between these two learning paradigms, through both theoretical analysis and empirical investigation. Theoretically, we first demonstrate that MTL shares the same optimization formulation with a class of gradient-based meta-learning (GBML) algorithms. We then prove that for over-parameterized neural networks with sufficient depth, the learned predictive functions of MTL and GBML are close. In particular, this result implies that the predictions given by these two models are similar over the same unseen task. Empirically, we corroborate our theoretical findings by showing that, with proper implementation, MTL is competitive against state-of-the-art GBML algorithms on a set of few-shot image classification benchmarks. Since existing GBML algorithms often involve costly second-order bi-level optimization, our first-order MTL method is an order of magnitude faster on large-scale datasets such as mini-ImageNet. We believe this work could help bridge the gap between these two learning paradigms, and provide a computationally efficient alternative to GBML that also supports fast task adaptation.

1. Introduction

MTL and meta-learning both exploit structure shared across tasks, but they differ in formulation: MTL trains efficiently while straightforwardly lacking adaptation to unseen tasks, and meta-learning supports fast adaptation at greater computational cost. The paper connects MTL with gradient-based meta-learning theoretically and empirically, finding that a properly implemented MTL approach can be competitive while training faster.

1. Introduction

  • The paper asks whether MTL can combine efficient training with fast adaptation to unseen tasks, addressing the gap between MTL's training efficiency and meta-learning's adaptation ability and cost.The authors note that rigorous exploration of the connection between the paradigms is lacking.Source: §1 · p. 1
  • The paper investigates the connection between MTL and meta-learning by focusing on gradient-based meta-learning methods inspired by MAML.Its contributions include theoretical analysis and empirical evaluation on few-shot image-classification benchmarks.Source: §1 · pp. 1–2
  • MTL and a class of GBML algorithms share an optimization formulation, and sufficiently wide neural networks yield similar predictions on unseen tasks.The prediction gap is inversely proportional to network depth; the result suggests MTL could support fast adaptation without losing training efficiency.Source: §1 · p. 2
  • Experiments find that properly implemented MTL achieves similar or better results than state-of-the-art GBML algorithms while incurring significantly lower computational costs.The paper reports an order-of-magnitude training-speed advantage over MetaOptNet on few-shot image-classification benchmarks, attributed to first-order optimization.Source: §1 · p. 2

2. Related Work

Related work describes MTL as joint learning of shared representations and reviews theory and optimization methods for it. It also situates the paper among meta-learning methods and contrasts its neural-network theory with prior work on a restricted toy setting.

2. Related Work

  • MTL jointly learns shared representations from multiple training tasks, with prior theoretical work studying its sample efficiency under assumptions about task relatedness, features, and model classes.Other cited MTL work explores optimization methods for task and feature relationships.Source: §2 · p. 2
  • Meta-learning targets fast adaptation to unseen tasks, and GBML methods following MAML are used across downstream domains.Prior work connects MAML's empirical success to representation learning.Source: §2 · p. 2
  • Prior comparison of MTL and Reptile studied sample-complexity bounds in 1D subspace learning with two-layer linear models, whereas this paper's theory covers nonlinear neural networks of any depth without restricting input dimension.The paper also notes similarities between its MTL implementation and metric-based few-shot methods because both retain trained hidden layers at test time.Source: §2 · p. 2

3. Preliminaries

The preliminaries set out the architectures, training schemes, and evaluation protocol used to compare MTL with GBML. They define shared hidden representations and task-specific heads, describe MTL and ANIL training, and specify how both models adapt to test tasks.

3. Preliminaries

  • The preliminaries introduce the network architectures, training algorithms, and evaluation protocols used for MTL and GBML.Source: §3 · p. 2

3.1. Neural Networks Architectures

  • The MTL architecture uses shared hidden layers as representations and a separate output head for each training task.The network parameters comprise the shared hidden-layer parameters and the task-specific output heads.Source: §3.1 · pp. 2–3

3.2. Multi-Task Learning

  • MTL trains a multi-head neural network jointly over N training tasks, each with n samples, by minimizing a loss-based objective.Source: §3.2 · p. 3

3.3. Gradient-Based Meta-Learning and ANIL

  • ANIL is presented as a simplification of MAML; its inner loop updates the head for τ gradient-descent steps, and this work omits the query-support split.Prior work cited in the passage reports no performance drop when the head is frozen across training, motivating omission of its outer-loop optimization.Source: §3.3 · p. 3

3.4. Fine-Tuning for Test Task Adaptation

  • For test-task adaptation, both trained models receive a randomly initialized head that is fine-tuned on labeled support samples before predictions are made on query samples.The protocol applies to an arbitrary test task and permits comparison of MTL and GBML on the same footing.Source: §3.4 · p. 3

4. Theoretical Analysis

The analysis organizes GBML variants by their inner-loop design, shows that MTL and a class of GBML methods share an optimization formulation, and establishes conditions under which MTL and ANIL predictors are close. The predictor result is derived through a kernel analysis and extends to specified residual architectures.

4.1. A Taxonomy of Gradient-Based Meta-Learning

  • GBML variants differ in inner-loop regularization and which network layers they optimize, dimensions used to organize the taxonomy.The paper presents typical instantiations of the last-layer objective and notes that loss and regularization choices vary.Source: §4.1 · pp. 3–4, Table 1 · p. 4, Table 2 · p. 4

4.2. Equivalence Between GBML and MTL from an Optimization Perspective

  • MTL and GBML share the same optimization structure up to regularization, but GBML uses bi-level optimization while MTL jointly optimizes the shared parameters and task heads.GBML optimizes the shared network parameters outside and transient task heads inside; MTL jointly optimizes both.Source: §4.2 · p. 4

4.3. Closeness Between MTL and GBML from a Functional Perspective

  • For sufficiently wide networks trained by gradient flow, MTL and ANIL test predictions correspond to special kernel regressions, and their predictors are close on test tasks.The proof sketch approximates network outputs with first-order Taylor expansions using NTK and NNGP kernels, then establishes convergence and computes the kernels.Source: §4.3 · pp. 4–6
  • For any ε > 0, a constant h* = O(ε^-2) suffices: when width h exceeds h*, ReLU networks with He initialization have a bounded average prediction difference between ANIL and MTL on query samples X.Source: §4.3 · p. 6

5. Experiments

Experiments show MTL predictions approach ANIL’s as depth grows or λτ decreases, and MTL is competitive with state-of-the-art GBML on few-shot benchmarks while training faster. The theorem’s fully connected network setting does not directly cover the modern ConvNets used in practice.

5.1. Closeness between MTL and GBML predictions

  • As L increases or λτ decreases, the norm of the MTL–ANIL prediction difference decreases, agreeing with Theorem 1’s bound.The synthetic-data study evaluates 20 test tasks over five runs.Source: §5.1 · p. 7, Figure 1 · p. 7
  • Theorem 1 is built on fully connected networks and does not directly apply to modern ConvNets with residual connections, max pooling, BatchNorm, and Dropout.The paper therefore separately evaluates modern ConvNets in the few-shot benchmark experiments.Source: §5.1 · p. 7

5.2. Few-Shot Learning Benchmarks

  • Across four 5-way few-shot benchmarks, MTL outperforms MetaOptNet in 4 of 8 settings, matches it in 2, and trails it in 2.The comparison covers mini-ImageNet, tiered-ImageNet, CIFAR-FS, and FC100.Source: §5.2 · p. 8, Table 3 · p. 8
  • MTL trains 23× faster than MetaOptNet on mini-ImageNet with optimal hyperparameters.The comparison reports GPU hours for training.Source: §5.2 · p. 9, Table 4 · p. 9
  • On tiered-ImageNet, MTL trains 11× faster while matching MetaOptNet’s test accuracy: 81.55% versus 81.56%.The study also reports that smaller batches or fewer epochs reduce training time with a performance drop of at most 2.2%.Source: §5.2 · p. 9, Figure 2 · p. 9

6. Conclusion

The conclusion connects shared optimization and similar predictions on unseen tasks with MTL’s competitive, efficient few-shot performance. The appendix outlines additional setup, framework details, and proofs.

6. Conclusion

  • The paper concludes that MTL and GBML share an optimization formulation and yield similar predictions on unseen tasks with sufficiently wide networks; its MTL variant is competitive and more efficient on few-shot benchmarks.The authors present this as a step toward models that combine efficient training with fast adaptation.Source: §6 · p. 9

Overview of the Appendix

  • The appendix collects additional meta-learning and neural tangent kernel setup, missing proofs, and detailed experimental discussion.It is organized into setup, proof, and experiment sections.Source: Overview of the Appendix · p. 13

A. More on Meta-Learning and Neural Net Setup

  • Appendix A covers the query-support split, a unified GBML framework that optimizes all inner-loop layers, and NTK parameterization.The appendix overview lists these as subsections A.1–A.3.Source: A. More on Meta-Learning and Neural… · p. 13

A.1. Query-Support Split

  • In the query-support setting, each training task’s labeled samples are divided into support and query sets; ANIL adapts on support data and evaluates loss on query data.The passage identifies these sets as Xs and Xq, respectively.Source: §A.1 · p. 13

A.2. Unified Framework for Gradient-Based Meta-Learning that Optimizes All Layers in the Inner Loop

  • The all-layers GBML framework is bi-level: outer optimization updates θ, while inner optimization finds transient task-specific parameters on a regularized loss.The task-specific parameters are not explicitly saved during training.Source: §A.2 · p. 13

A.3. NTK Parameterization

  • NTK parameterization supports theoretical analysis of neural-network optimization; its training dynamics and predictions match standard networks up to a width-dependent learning-rate factor.The appendix explains the parameterization using a fully connected feed-forward network and notes that multi-head networks share it.Source: §A.3 · p. 14

B. Proof

This block introduces missing proofs and defines shorthand for test-phase parameter sets.

B. Proof

  • The appendix section presents proofs omitted from the main paper.It states that the proofs are summarized in the following material.Source: B. Proof · p. 14
  • θtest denotes the set containing the first L − 1 layers’ parameters of θ and the test head wtest.The test head is randomly initialized for fine-tuning in the test phase.Source: B. Proof · p. 14
  • θ̂test denotes the set containing the first L − 1 layers’ parameters of θ̂ and the test head wtest.The appendix introduces this shorthand alongside θtest.Source: B. Proof · p. 14

B.1. Global Convergence of ANIL and MTL with Over-parameterized Deep Neural Nets

For sufficiently wide, over-parameterized networks, ANIL and MTL converge under stated optimization conditions and remain near their initial parameters. This neighborhood supports linearizing their outputs during training.

B.1. Global Convergence of ANIL and MTL with Over-parameterized Deep Neural Nets

  • The analysis characterizes both algorithms using deterministic positive-definite kernels independent of their initializations.The proof uses global convergence and parameter-neighborhood results to obtain the linearized output expressions.Source: §B.1 · pp. 15–16
  • With sufficient width and the theorem’s stated learning-rate conditions, ANIL and MTL training losses satisfy bounds with probability at least 1 − δ over random initialization.The theorem also bounds parameter displacement during training.Source: §B.1 · p. 15
  • With probability at least 1 − δ, sufficiently wide ANIL and MTL networks stay near initialization, allowing their outputs to be effectively linearized.The corollary gives a parameter-neighborhood radius of O(1/h^2).Source: §B.1 · p. 16

B.2. Training Dynamics of MTL and ANIL

The appendix characterizes ANIL and MTL training dynamics in the linearization regime and derives their corresponding kernels and test predictions. These expressions provide the analytical basis for comparing the two algorithms.

B.2. Training Dynamics of MTL and ANIL

  • In the linearization regime, ANIL and MTL dynamics under gradient descent on squared losses reduce to analytically solvable ODEs, yielding solutions for trained parameters and test predictions.The test protocol uses the trained hidden layers with a randomly initialized head fine-tuned on the test task.Source: §B.2 · p. 17

B.3. Derivation of Kernels and Outputs for ANIL and MTL.

  • As network width tends to infinity, the ANIL and MTL kernels converge to corresponding analytical expressions used to derive their test predictions.The kernel definitions build on the NTK and NNGP.Source: §B.3 · pp. 19–20

B.3.1. PROOF OF LEMMA 1

The proof derives the test-prediction expressions in the infinite-width limit and accounts for finite-width correction. It relates the ANIL–MTL prediction discrepancy to kernel differences, particularly when the adaptation-step product is small.

B.3.1. PROOF OF LEMMA 1

  • The infinite-width kernel expressions yield the lemma’s test-prediction formulas, with finite-width correction absorbed into O(1/√h) error terms.This correction bound holds with arbitrarily large probability for sufficiently large width.Source: §B.3.1 · p. 22

B.3.2. DISCREPANCY BETWEEN PREDICTIONS OF ANIL AND MTL

  • For small λτ, the discrepancy between infinite-time ANIL and MTL test predictions is expressed through their kernel terms.The proof assumes the initial network outputs have smaller magnitude than the training labels, an assumption achievable by reducing initialization scale or scaling down the network output.Source: §B.3.2 · p. 22, §B.3.1 · p. 22
  • If the relevant kernel difference vanishes in a limit, ANIL and MTL produce almost the same predictions on any test task.The discrepancy is analyzed through the kernel-inverse difference.Source: §B.3.2 · p. 22

B.4. Kernel Structures for Deep ReLU Nets

The appendix analyzes deep ReLU kernel structure to establish the prediction-closeness result, extends it to residual ReLU networks with and without LayerNorm, and provides experimental details. Its Figure 1 discussion notes variability across random-seed averages.

B.4. Kernel Structures for Deep ReLU Nets

  • For sufficiently large depth, the NTK and NNGP have characterized spectra, which the analysis uses to bound the discrepancy between ANIL and MTL kernel inverses.The inverse-discrepancy lemma considers ReLU networks with He initialization.Source: §B.4 · p. 23

B.5. Proof of Theorem 1

  • The proof of Theorem 1 follows from the kernel-inverse discrepancy bound together with the prediction-discrepancy expression.The proof also uses a bound on the relevant kernel operator norm.Source: §B.5 · pp. 25–26

B.6. Extension to Residual ReLU Networks

  • Theorem 1 also applies to residual ReLU networks and residual ReLU networks with LayerNorm.For residual ReLU networks, the kernel and inverse-kernel factors cancel in the predictors; with LayerNorm, the kernel structures match those of ReLU networks without residual connections.Source: §B.6 · p. 26

C. Details of Experiments

  • Appendix C provides additional experimental details for the empirical validation of Theorem 1 and the few-shot image-classification study.It separates the details into Appendices C.1 and C.2, respectively.Source: C. Details of Experiments · p. 26

C.1. Empirical Validation of Theorem 1

  • The kernel experiments construct ANIL and MTL kernels from NTKs and NNGPs using Neural Tangents.For Figure 1, the two panels use different random seeds; five-seed averages still show non-negligible variance.Source: §C.1 · p. 26

C.2. Experiments on Few-Shot Image Classification Benchmarks

For meta-validation and meta-testing, the method adapts a linear classifier on last-hidden-layer features using cross-entropy and ℓ2 regularization.

  • Validation and test adaptation uses logistic regression with ℓ2 regularization strength 0.33.The strength was selected based on an ablation study.Source: §C.2 · p. 26
  • The classifier is fine-tuned on last-hidden-layer features with cross-entropy loss.Source: §C.2 · p. 26
  • Table 5 ablates the ℓ2 penalty on the fine-tuned linear layer in mini-ImageNet 5-way 5-shot classification.Source: Table 5 · p. 26
Loading 2106.09017v1…