PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding and Improving Information Transfer in Multi-Task Learning

Sen Wu, Hongyang R. Zhang, Christopher Ré

arXiv:2005.00944v1cs.LGcs.CL

TL;DR

Multi-task learning can suffer from negative transfer when task data are poorly aligned, motivating a closer study of how shared representations transfer information. The paper analyzes shared-module models theoretically and develops alignment and reweighting methods, obtaining gains on GLUE, sentiment transfer, and ChestX-ray14.

  • Problem

    The paper asks which factors determine whether multi-task learning outperforms single-task learning, focusing on interference that can arise despite shared representations.

  • Method

    The paper studies shared-module architectures with separate task outputs in linear and ReLU settings, analyzing capacity, task covariance, and optimization weights.

  • Results

    The methods improve performance across language, transfer-learning, and multi-label image settings, including a 2.35% average GLUE score improvement over BERTLARGE on 5 tasks.

  • Takeaways & Limitations

    Task-data alignment and task weighting provide practical ways to improve the effectiveness and robustness of multi-task training and transfer learning.

  • Takeaways & Limitations

    The SVD-based optimization scheduler applies only when tasks have the same data, and the linear optimization guarantee does not extend to nonlinear ReLU settings.

Abstract

from arXiv · show

We investigate multi-task learning approaches that use a shared feature representation for all tasks. To better understand the transfer of task information, we study an architecture with a shared module for all tasks and a separate output module for each task. We study the theory of this setting on linear and ReLU-activated models. Our key observation is that whether or not tasks' data are well-aligned can significantly affect the performance of multi-task learning. We show that misalignment between task data can cause negative transfer (or hurt performance) and provide sufficient conditions for positive transfer. Inspired by the theoretical insights, we show that aligning tasks' embedding layers leads to performance gains for multi-task training and transfer learning on the GLUE benchmark and sentiment analysis tasks; for example, we obtain a 2.35% GLUE score average improvement on 5 GLUE tasks over BERT-LARGE using our alignment method. We also design an SVD-based task reweighting scheme and show that it improves the robustness of multi-task training on a multi-label image dataset.

1 INTRODUCTION

The paper examines when multi-task learning transfers information constructively or destructively, emphasizing that task data alignment matters alongside model similarity. It develops theoretical criteria and practical methods that improve multi-task and transfer-learning performance.

  • Multi-task learning can reduce per-task data requirements and produce shared representations, but task interference can also reduce individual-task performance.The paper frames predicting when interference occurs as a challenge because analytic tools are limited.
  • The analyzed architecture uses a shared lower module B and separate task-specific output modules.This shared-module design is the basis for studying how task information transfers across tasks.
  • Tasks with the same model can still produce positive or negative transfer when their data distributions differ.Tasks 2 and 3 share a model but differ in data distributions, while task 1’s outcome depends on contributing data near the decision boundary.
  • The theory identifies shared-module capacity, task covariance, and per-task training weights as key determinants of interference.Varying task covariance produces both positive and negative transfer, while sufficient contributor-task data can guarantee positive transfer.
  • Aligning task embedding-layer covariances improves performance by 2.35% average GLUE score across 5 tasks over BERTLARGE.The method also improves transfer-learning accuracy by up to 2.5% across six sentiment-analysis tasks.
  • An SVD-based task-reweighting scheme improves average ChestX-ray14 performance by 0.4% AUC score across all tasks.The scheme targets settings where tasks share features but have different labels.

2 THREE COMPONENTS OF MULTI-TASK LEARNING

The paper analyzes how shared-module capacity, task covariance, and optimization determine whether multi-task learning transfers information constructively or destructively. It develops theory for linear and ReLU settings and derives alignment and reweighting methods from these findings.

  • 2.1 MODELING SETUP: Multi-task learning uses a shared module B and separate task-specific modules Ai to learn a common representation across tasks.The shared module encodes inputs, while each Ai produces the output for task i.
  • 2.1 MODELING SETUP: Positive transfer improves a target task over single-task learning, whereas negative transfer hurts target validation performance.
  • 2.2 MODEL CAPACITY: Shared-module capacity controls interference: excessive capacity can eliminate transfer, while insufficient capacity can cause destructive interference.For linear tasks, r ≥ k yields no transfer; limiting capacity is therefore necessary to enforce information sharing.
  • 2.3 TASK COVARIANCE: Task covariance captures data geometry, so tasks with similar models can still transfer positively or negatively depending on covariance and sample allocation.In the example, varying source-task covariance and sample count produces both transfer types; sufficient source data can eventually guarantee positive transfer under covariance-dependent conditions.
  • 2.3 TASK COVARIANCE: The theory motivates covariance alignment, which inserts an alignment matrix Ri before each task’s inputs enter the shared module.
  • 2.4 OPTIMIZATION SCHEME: The linear analysis establishes global optimality of all local minima, but this result does not extend to the nonlinear ReLU setting.
  • 2.4 OPTIMIZATION SCHEME: The paper also derives SVD-based task reweighting by weighting tasks according to their projections onto principal directions of the task representations.The reweighting scheme is intended to reduce the influence of noisy labels.

3 EXPERIMENTS

Experiments evaluate covariance alignment and task reweighting across language, sentiment, transfer-learning, and image-classification settings. The methods improve benchmark performance, while ablations show that shared capacity and covariance similarity affect transfer.

  • Improving multi-task training: 2.35% average GLUE score improvement over BERTLARGE was achieved across five tasks using task-embedding alignment.
  • Improving transfer learning: Up to 2.5% accuracy improvement was obtained on four sentiment-analysis transfer-learning tasks.
  • Reweighting training for the same task covariates: 0.4% AUC score improvement over unweighted training was achieved across all ChestX-ray14 tasks with SVD-based reweighting.
  • Model capacity: MTL performance peaked at shared capacity 100, substantially below the combined capacities of the single-task models.
  • Task covariance: Accuracy increased for 13 of 15 sentiment-task pairs by up to 4.1%, while covariance similarity increased for all 15 pairs after alignment.
  • Optimization scheme: On ten randomly selected task pairs, reweighting improved over the unweighted scheme by an average 1.0% AUC score.

4 RELATED WORK

Related work positions this study within theoretical, architectural, domain-adaptation, and optimization research on multi-task learning. It distinguishes the paper’s hard-sharing setup and SVD-based weighting from nearby approaches.

  • Prior theory links multi-task effectiveness to task relatedness, while newer work studies generalization, subspace selection, and meta-learning.
  • The paper’s capacity restriction differs from recent over-parameterized-model studies and motivates future analysis involving double descent.
  • The studied architecture is hard parameter sharing, contrasting with soft sharing, where task-specific parameters are regularized toward similarity.
  • Unlike domain adaptation, this work minimizes the multi-task objective rather than typically minimizing the target-task objective.
  • The SVD-based weighting method is described as conceptually simpler and less computationally demanding than bandit-based weighting.

5 CONCLUSIONS AND FUTURE WORK

The paper studies multi-task learning theory in linear and ReLU-activated settings and verifies its practical implications through synthetic and real-world experiments. It also identifies capacity, task covariance, and task weighting as central factors shaping transfer.

  • The paper studies multi-task learning in linear and ReLU-activated settings and verifies its implications through synthetic and real-world experiments.
  • The paper leaves extending optimization guarantees to nonlinear settings and adapting SVD-based scheduling to heterogeneous task data as future questions.
  • The optimal shared subspace determines multi-task performance because task-specific output layers select solutions within that subspace.
  • Shared-module capacity can eliminate interference when too large but cause destructive interference when too small.
  • In the ReLU setting, a shared module larger than the total single-task capacities can store all single-task parameters, yielding no transfer between tasks.

A.2.1 THE EFFECT OF COSINE SIMILARITY

This section analyzes how task similarity and shared capacity affect multi-task learning when task covariances are the same or different. The optimal shared representation is tied to a best rank-r approximation of task-related structure.

  • When task features share covariance, the optimal multi-task solution depends on the singular structure of the aggregated task data.
  • With orthogonal task optima and identity inputs, single-task error is zero, but limited shared dimension forces approximation across tasks.
  • For hidden dimension r, the optimal shared module uses the best rank-r approximation subspace of the task-related matrix.
  • For different data covariances, the gap between multi-task and single-task solutions is bounded using the singular values of an orthogonalized task-data matrix.

A.2.2 PROOF OF THEOREM 2

Theorem 2 formalizes when source-task information transfers positively to a target task in linear multi-task learning. Positive transfer is guaranteed when task models are close and the source has sufficient data, with an accuracy–data trade-off.

  • Positive transfer is guaranteed when source and target models are close and the source task has enough samples.
  • Reducing the desired error margin c improves the error bound but requires more source data, creating an accuracy–data trade-off.
  • With noisy task-two labels, the analysis assumes c ≤ 1/3; without task-two noise, this assumption is unnecessary.
  • The target error depends on task-model angle, the condition number of the target design matrix, and task-two noise.
  • The proof bounds the target estimation error after optimizing a shared module from task one and a task-specific output layer for task two.

A.2.3 EXTENSION TO THE RELU MODEL

Theorem 2 is extended to a ReLU model by reducing training to a shared-module objective and analyzing concentration under Gaussian task-one inputs. The result retains a positive-transfer guarantee under this distributional assumption.

  • The ReLU multi-task problem reduces to optimizing a shared module using normalized correlations with both tasks’ ReLU-transformed inputs.
  • The ReLU extension assumes that task one’s inputs are drawn from a standard Gaussian distribution.
  • The Gaussian assumption is necessary for this analysis because worst-case inputs make even single-task ReLU optimization under squared loss NP-hard.
  • Concentration arguments bound the angle between the learned shared module and task one’s parameter vector.
  • The estimation analysis then bounds the task-two output coefficient and the resulting target error.

A.3 PROOF OF PROPOSITION 3

The proof reduces the reweighted linear MTL objective to low-rank matrix approximation, showing that its local minima are global minima. It then characterizes the optimal shared representation through an SVD-based solution, while noting the result does not extend to ReLU models.

  • All local minima of the reweighted linear objective are global minima.The proof establishes this refined result for the reweighted loss in the linear case.
  • The global-optimality characterization follows by applying a no-spurious-local-minima result to the low-rank approximation problem.The proof identifies the only local minima with solutions whose product equals the best rank-r approximation.
  • The transformed optimization problem reduces to low-rank matrix approximation of a weighted label matrix.The reduction uses stacked, square-root-weighted task parameters and labels.
  • The optimal shared matrix is B⋆ = (X⊤X)^−1XQ_r, where Q_r is the best rank-r approximation derived from the weighted task labels.This follows from the SVD of X and the best rank-r approximation of the transformed label matrix.
  • The result applies only to linear models; optimization landscapes for nonlinear ReLU models remain unresolved.The authors leave characterization of ReLU optimization landscapes for future work.

B.1 DATASETS

The experiments use synthetic data and three real-world domains: sentiment analysis, GLUE, and ChestX-ray14. These datasets support evaluations across language understanding, sentiment classification, and multi-label medical image classification.

  • The study evaluates synthetic settings, sentiment analysis, GLUE, and ChestX-ray14 datasets.The real-world datasets cover language understanding, sentiment classification, and medical imaging.
  • Synthetic settings: Synthetic experiments use linear regression, logistic regression, and one-layer ReLU neural networks.These settings provide controlled evaluations of the theory across linear and nonlinear models.
  • Sentiment analysis: The sentiment experiments use six benchmarks: MR, SUBJ, CR, TREC, MPQA, and SST.These datasets cover positive-versus-negative sentiment, subjectivity, question type, opinion polarity, and related classification tasks.
  • GLUE benchmark: The GLUE evaluation selects five tasks: CoLA, MRPC, QNLI, RTE, and SST-2.The authors use these tasks to validate the proposed method and emphasize that the goal is insight rather than a state-of-the-art result.
  • ChestX-ray14: ChestX-ray14 contains 112,120 frontal-view X-ray images from 30,805 patients with up to 14 pathology labels per image.The dataset is formulated as a 14-task multi-label image classification problem.
  • Model configurations: Sentiment models include an MLP, a one-layer unidirectional LSTM, and a CNN with convolution, ReLU, max-pooling, and classification layers.The sentiment models share embedding and feature-extraction modules while retaining task-specific output modules.

B.3 TRAINING PROCEDURES

Training procedures vary by task structure: batches are interleaved for tasks with different features, while tasks sharing features are updated jointly. GLUE uses fixed BERT-LARGE with alignment modules, and evaluation metrics are dataset-specific.

  • Mini-batch SGD: Different-feature tasks are trained by mixing and shuffling batches, with each batch updating its corresponding task parameters.For a batch from task i, the update applies to A_i and, depending on the setup, R_i or B while other task parameters remain fixed.
  • Mini-batch SGD: Same-feature tasks are trained jointly, updating all task-specific A_i parameters and the shared B together.This procedure is used for ChestX-ray14.
  • Synthetic settings: Synthetic experiments use learning rate 1e−3, 30 epochs, and batch size 50, reporting Spearman correlation for regression and accuracy for classification.These settings are selected through grid search.
  • Sentiment analysis: Sentiment data are split into 80% training, 10% development, and 10% test sets, with accuracy reported on the test set.The models follow the protocol of Lei et al. (2018), and hidden dimensions are varied for capacity experiments.
  • GLUE: GLUE training fixes the BERT-LARGE module while optimizing alignment and output modules.The BERT-LARGE module is fixed to isolate the effect of adding alignment modules.
  • Evaluation: GLUE uses learning rate 2e−5, five epochs, batch size 16, and the GLUE development-set score; ChestX-ray14 reports test-set AUC after 20 epochs.The GLUE hyperparameters are selected from grid search, while ChestX-ray14 follows the suggested configuration.

B.4 EXTENDED SYNTHETIC EXPERIMENTS

Extended synthetic experiments test how task similarity, data size, covariance alignment, and model capacity affect transfer. They find that greater similarity improves MTL over STL, while Algorithm 1 corrects negative transfer when source data are limited.

  • Task similarity: Greater task similarity makes MTL outperform STL more strongly, whereas sufficiently low similarity can produce negative transfer.This pattern appears for regression and classification tasks, including ReLU-activated settings.
  • Transfer factors: Figure 8 shows that data size, STL-model cosine similarity, and covariance alignment affect the rate of transfer.The authors report that these effects persist across the extended settings.
  • Algorithm 1: Algorithm 1 corrects negative transfer when the source task has limited data and matches baseline MTL when source data are sufficiently plentiful.The comparison covers both linear and ReLU regression tasks.
  • Synthetic construction: The synthetic setup varies task similarity through the cosine similarity α between two model-parameter subspaces.Task 2 combines task 1’s rotation matrix with an orthogonal rotation matrix, and labels are generated with ReLU and Gaussian noise.
  • Model capacity: Capacity experiments use cross-validation to select the best-performing capacity for each model.The extended evaluation includes MLP, CNN, and LSTM models for sentiment analysis.

B.5 EXTENDED ABLATION STUDIES

The extended ablations examine model-capacity selection and Algorithm 2 under label noise. Multi-task learning uses less aggregate capacity than separately optimized single-task models, while Algorithm 2 improves AUC against both comparison schemes.

  • Model capacity: Cross-validation evaluates model-capacity choices on six sentiment-analysis tasks trained with an LSTM layer.Validation accuracies compare multi-task and single-task models while capacities vary.
  • Model capacity: The best-performing multi-task model uses less capacity than the total best-performing capacities of single-task models across CNN, MLP, and LSTM architectures.The comparison uses SST and MR sentiment-analysis datasets for CNN and MLP, alongside LSTM results.
  • Label-noise robustness: Under label noise, Algorithm 2 is evaluated by randomly flipping labels for one task after subsampling two ChestX-ray14 tasks.The experiment compares Algorithm 2 with unweighted loss and Kendall et al. (2018).
  • Label-noise robustness: 1.0% AUC score improvement over unweighted training and 0.4% over Kendall et al. (2018) are observed across 10 randomly chosen task pairs.Figure 12 shows five example task pairs and compares Algorithm 2 with both baselines.
Loading 2005.00944v1…