PaperlayerView exact revision on arXiv

Source-linked AI summary

Linear Mode Connectivity in Multitask and Continual Learning

Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, Hassan Ghasemzadeh

arXiv:2010.04495v1cs.LGcs.AIcs.CV

TL;DR

Continual learning often forgets earlier tasks because it trains sequentially, while the geometric relationship between continual and multitask solutions remains unclear. The paper studies this relationship and finds that shared initialization enables linear low-error connectivity, which motivates MC-SGD; the method performs better than several established approaches on classical continual-learning benchmarks.

  • Problem

    Continual learning suffers catastrophic forgetting, and the relationship between its solutions and multitask minima is not well understood.

  • Method

    The paper studies linear low-error connectivity between sequential and multitask solutions under shared initialization and uses it to design MC-SGD.

  • Results

    Shared initialization yields linear low-error connectivity across more than 20 tasks, and MC-SGD outperforms several established continual-learning methods on classical benchmarks.

  • Takeaways & Limitations

    Linear connectivity provides a geometric basis for constraining continual-learning minima toward multitask behavior.

  • Takeaways & Limitations

    The connectivity result depends on shared initialization and shared task structure; destroying task structure prevents a linearly connected multitask solution.

Abstract

from arXiv · show

Continual (sequential) training and multitask (simultaneous) training are often attempting to solve the same overall objective: to find a solution that performs well on all considered tasks. The main difference is in the training regimes, where continual learning can only have access to one task at a time, which for neural networks typically leads to catastrophic forgetting. That is, the solution found for a subsequent task does not perform well on the previous ones anymore. However, the relationship between the different minima that the two training regimes arrive at is not well understood. What sets them apart? Is there a local structure that could explain the difference in performance achieved by the two different schemes? Motivated by recent work showing that different minima of the same task are typically connected by very simple curves of low error, we investigate whether multitask and continual solutions are similarly connected. We empirically find that indeed such connectivity can be reliably achieved and, more interestingly, it can be done by a linear path, conditioned on having the same initialization for both. We thoroughly analyze this observation and discuss its significance for the continual learning process. Furthermore, we exploit this finding to propose an effective algorithm that constrains the sequentially learned minima to behave as the multitask solution. We show that our method outperforms several state of the art continual learning algorithms on various vision benchmarks.

1 INTRODUCTION

The paper asks how multitask and continual-learning minima differ geometrically and finds that shared initialization enables reliable linear low-error connectivity. It uses this finding to motivate MC-SGD, a continual-learning algorithm designed to constrain sequential minima toward multitask behavior.

  • Motivation: Catastrophic forgetting arises because continual learning sees tasks sequentially, unlike multitask learning, which accesses all tasks simultaneously.This limits continual learning’s ability to learn efficiently and motivates methods approximating the multitask loss.
  • Research question: The paper investigates whether multitask and continual-learning solutions are related by local geometric structure rather than conventional model-similarity distances.The authors specifically question the relationship between the two minima when both regimes seek good performance on all tasks.
  • Connectivity finding: Shared initialization reliably enables a linear low-error path connecting multitask and continual solutions, including sequences exceeding 20 tasks.The setup first learns task 1, then either learns task 2 sequentially or trains on both tasks simultaneously.
  • Connectivity finding: Euclidean distance and CKA fail to meaningfully identify that multitask solutions better avoid catastrophic forgetting.For example, a continual solution can be closer to the first-task solution while performing worse on that task.
  • Algorithmic contribution: The authors propose Mode Connectivity SGD, which constrains sequentially learned minima to behave like multitask solutions and outperforms several established continual-learning methods.The method exploits the observed connectivity to impose a low-loss constraint during continual learning.

2 THE RELATION BETWEEN MULTITASK AND CONTINUAL MINIMA

The paper finds that multitask and continual minima cannot be reliably related by standard distance or representation-similarity measures, but same-initialization solutions are connected by low-loss linear paths.

  • Linear connectivity: Same-initialization multitask and continual solutions are connected by low-error paths that can be linear across more than 20 sequential tasks.The paper contrasts this connectivity with prior single-task settings using different initializations.
  • Distance and representation measures: Euclidean distance does not reliably track catastrophic forgetting: a continual-learning minimum can be closer to another continual minimum than to the multitask solution.The authors attribute this partly to the direction-dependent tightness of the Hessian-based bound.
  • Distance and representation measures: CKA can remain nearly unchanged even when previous-task accuracy drops sharply, limiting its usefulness for comparing these minima.Pairwise CKA between the first-task minimum and later continual or multitask minima is roughly similar in the reported example.
  • Linear connectivity: Interpolation plots show low validation loss between multitask minima and continual solutions for both task-one and task-two reference solutions.The surfaces are computed in parameter-space hyperplanes through the relevant minima rather than as low-dimensional projections.

3 WHEN DOES LINEAR MODE CONNECTIVITY HOLD?

The paper asks what local geometric properties distinguish multitask from continual minima, focusing on the linear path between the first-task solution and the multitask solution.

  • Geometric interpretation: The analysis focuses on what the linear path between the first-task minimum and multitask solution implies about their local loss geometry.This frames linear connectivity as a basis for interpreting the difference between the two training regimes.

2. One potential justification is that while minimizing the multitask

The paper links linear connectivity to low-curvature directions around earlier-task minima, while showing that low initial curvature alone does not guarantee resistance to forgetting.

  • Hessian evidence: Hessian analysis finds the continual direction aligned with top eigenvectors, whereas the multitask direction is nearly orthogonal to them and instead associated with low eigenvalues.The reported cosine-angle measurements support the proposed direction-confinement explanation.
  • Hessian evidence: Moving in a low-curvature direction is not sufficient: increasing training epochs from 5 to 20 affects both directions toward the continual and multitask solutions.The passage explicitly distinguishes necessity from sufficiency for low-curvature motion.
  • Curvature and forgetting: The interpolation line to the continual solution begins in low-curvature directions but later incurs a substantial earlier-task loss increase, unlike the multitask path.The authors attribute this difference to higher-order terms changing curvature before the continual minimum is reached.
  • Curvature and forgetting: The multitask minima remain in a region where a second-order Taylor approximation for earlier tasks holds and higher-order derivatives are less important.Within this region, learning is described as confined to directions associated with small Hessian eigenvalues for previous tasks.
  • Scope of the regime: The proposed low-curvature regime is reported to support learning up to 20 complex tasks and 50 permuted-MNIST tasks without task interference.The authors connect this capacity to the expressiveness of the region where the second-order approximation holds.
  • Scope of the regime: Sharing the same initial condition is vital for reliable linear connectivity, while shared task structure also contributes to the observed behavior.These are presented as setting-dependent factors rather than universal guarantees.

4 CONTINUAL LEARNING WITH MODE CONNECTIVITY

MC-SGD uses mode connectivity to constrain sequentially learned minima to remain in low-loss valleys connected to previous continual-learning solutions. It combines this connectivity objective with replay because earlier-task losses are unavailable during later-task training.

  • MC-SGD constrains each continual-learning minimum to lie in a low-loss valley connected to previous minima.
  • The connectivity loss minimizes task losses along lines linking the current solution to the relevant continual-learning minima.For two tasks, the objective integrates the losses along interpolations parameterized by α between the candidate solution and each prior minimum.
  • Using five equally spaced α values between 0 and 1 was sufficient to obtain good results in the experiments.
  • The decomposed objective jointly minimizes both task losses while encouraging low-loss paths to both continual-learning minima.
  • Because earlier-task losses are unavailable, MC-SGD uses a small replay buffer to approximate them during later-task learning.The replay memory contains randomly sampled examples from previous tasks.
  • The method is extended to an online version designed to reduce the computational cost of the mode-connectivity loss.

5 EXPERIMENTS AND RESULTS

The experiments evaluate MC-SGD on standard continual-learning benchmarks using established metrics and constrained episodic memory. MC-SGD outperforms the compared methods, with its performance gap increasing as the number of tasks grows, while interpolation plots show nearly flat losses between MC-SGD and continual-learning minima.

  • Benchmarks: The study evaluates Permuted MNIST, Rotated MNIST, and Split CIFAR-100, using more challenging datasets alongside the literature-standard Permuted MNIST benchmark.
  • Evaluation: The experiments report average accuracy and average forgetting to compare continual-learning algorithms over many tasks.Average forgetting measures the difference between each task’s peak and ending accuracy after the continual-learning experience.
  • Experimental setup: The comparison uses 20 tasks, with MC-SGD, A-GEM, and ER-Reservoir limited to one example per class per task in episodic memory.
  • Comparison with other methods: MC-SGD outperforms the other methods, with its performance gap increasing as the number of tasks increases.The paper reports that this trend also holds when Permuted MNIST grows from 20 to 50 tasks.
  • Mode connectivity: Interpolation losses between continual-learning and MC-SGD minima are nearly flat compared with interpolation losses among continual-learning minima.This pattern is shown for early, middle, and late learning stages on Rotated MNIST and for tasks 1 and 5 on Split CIFAR-100.

6 CONCLUSION

The study finds that, under matched starting conditions, multitask and continual-learning minima are connected asymmetrically by linear low-error paths. It uses this geometric observation to motivate MC-SGD for continual learning.

  • 6 CONCLUSION: Multitask minima connect to continual-learning minima by linear low-error paths when both regimes start from similar conditions.The pattern can persist when the process is repeated across more than 20 tasks.
  • 6 CONCLUSION: The analysis interprets multitask optimization as restricting learning to low-curvature directions where a second-order approximation applies.
  • 6 CONCLUSION: MC-SGD constrains sequentially learned solutions using a replay buffer and the assumed linear low-loss connectivity to previously learned-task solutions.The method uses less data and performs better than two other rehearsal-based methods in the reported comparison.
  • 6 CONCLUSION: The authors propose connectivity analysis as a way to clarify why multitask learning typically achieves better performance than continual learning.

B.2 SHARED STRUCTURE OR SEMANTICS OF DATA

The experiments probe when linear connectivity breaks by increasing task dissimilarity through input noise, label corruption, and class removal. Connectivity deteriorates as the shared structure or semantic overlap between tasks is reduced.

  • B.2 SHARED STRUCTURE OR SEMANTICS OF DATA: Increasing Gaussian noise in the second task’s images gradually separates the multitask and continual minima along their interpolation.The experiments use MNIST first and Fashion-MNIST second to increase distribution shift.
  • B.2 SHARED STRUCTURE OR SEMANTICS OF DATA: Increasing label corruption makes the continual and multitask minima progressively more disconnected.The interpolation loss is reported for different percentages of label corruption.
  • B.2 SHARED STRUCTURE OR SEMANTICS OF DATA: The class-removal experiment excludes first-task examples from multitask training when they belong to randomly selected classes.
  • B.2 SHARED STRUCTURE OR SEMANTICS OF DATA: Removing two classes breaks connectivity between the first-task and subsequent minima, and removing four classes continues the trend.
  • B.2 SHARED STRUCTURE OR SEMANTICS OF DATA: The authors emphasize that linear connectivity holds on common continual-learning benchmarks, while these experiments deliberately test its boundaries.

C DETAILS OF EXPERIMENTAL SETUP

The experimental setup is designed for comparison with continual-learning literature through established benchmarks, architectures, baselines, and reproducible procedures. Visualization procedures map neural-network parameters into low-dimensional planes and evaluate losses along interpolations.

  • C.1 BASELINES AND ARCHITECTURES: Baselines are selected for diversity, reproducibility, and performance, including naive SGD, A-GEM, ER-Reservoir, EWC, and stable-SGD.
  • C.2 EXPERIMENTAL SETUP: The main experiments use five tasks for detailed continual-learning trajectories and 20 tasks for a more challenging one-example-per-class online-like setting.The 20-task experiments train on each task for one epoch.
  • C.1 BASELINES AND ARCHITECTURES: The benchmark architectures are a two-layer 256-neuron MLP for MNIST and ResNet18 for CIFAR-100.These choices are intended to ease comparison with prior work.
  • C.5 VISUALIZATION: Surface-plane visualization flattens and concatenates network layers, constructs basis vectors from three parameter points, and evaluates empirical loss over a coordinate grid.
  • C.5 VISUALIZATION: Loss along a minimum-interpolation line is evaluated using w_α = ŵ_1 + α(ŵ_2 − ŵ_1) for 0 ≤ α ≤ 1.

C.5.2 MC-SGD IMPLEMENTATION

MC-SGD implementation accumulates gradients evaluated at multiple points along an interpolation line using replay-memory loss. The authors also provide executable runs to support independent verification.

  • C.5.2 MC-SGD IMPLEMENTATION: The implementation uses PyTorch because its dynamic graph capability supports the required gradient calculations.
  • C.5.2 MC-SGD IMPLEMENTATION: MC-SGD computes interpolation parameters between w_hat and w_bar, evaluates replay-memory loss, backpropagates, and collects gradients at each sampled point.
  • C.5.2 MC-SGD IMPLEMENTATION: The gradients from sampled interpolation points are accumulated to form the line-based gradient quantity used by MC-SGD.
  • C.5.2 MC-SGD IMPLEMENTATION: Five random-seed runs of MC-SGD on rotation MNIST with 20 tasks are provided for reproducing the Table 1 results.Runs include dependencies, source code, accuracy logs, and final-task checkpoints.

D.1 CONVENTIONAL DISTANCE MEASURES

Euclidean distance and pairwise CKA do not reliably characterize the relationship between continual-learning and multitask-learning minima. Detailed plots and comparisons instead examine distances, representations, and interpolation paths across rotated MNIST and split CIFAR-100.

  • Distance measures: Euclidean distance can mislead because multitask minima may be farther from continual minima while achieving higher accuracy.The paper attributes this to optimization directions and loss-landscape curvature, including movement through low-loss valleys.
  • Representation comparisons: Pairwise CKA scores are provided for hidden layers in rotated MNIST and ResNet blocks in split CIFAR-100.These representation comparisons accompany the Euclidean-distance analysis.
  • Experimental analyses: The experiments include a three-task continual-versus-multitask setup and detailed interpolation-loss plots for rotated MNIST and split CIFAR-100.Figure 15 extends the basic setup by adding a task, while Figures 16 and 17 detail interpolation paths.

D.3 PERMUTED MNIST WITH 50 TASKS

The 50-task permuted-MNIST experiment tracks average accuracy across different methods, using an expanded model and episodic memory relative to the stated setup.

  • Model setup: The MLP hidden layers increase from 256 to 512 as the number of tasks increases.This is described as an experiment-specific setup change.
  • Memory setup: Episodic memory increases from 200 to 500 examples, with one example per class per task.The passage specifies the memory allocation used in this experiment.
  • Evaluation: Average accuracy is evaluated for different methods on permuted MNIST with 50 tasks.The supplied figure caption identifies the plotted quantity and task count but does not state the comparative outcome.
Loading 2010.04495v1…