PaperlayerView exact revision on arXiv

Source-linked AI summary

Continual learning with hypernetworks

Johannes von Oswald, Christian Henning, Benjamin F. Grewe, João Sacramento

arXiv:1906.00695v4cs.LGcs.AIstat.ML

TL;DR

Continual learning must learn sequential tasks without catastrophically forgetting earlier ones. The paper uses task-conditioned hypernetworks to generate task-specific target weights and preserve them through output regularization. It reports state-of-the-art benchmark performance, long memory lifetimes in a compressive regime, and transfer to future tasks.

  • Problem

    Continual learning requires sequentially learning tasks while retaining prior performance and supporting backward and forward transfer.

    Source: §1 · p. 1

  • Method

    Task-conditioned hypernetworks map task embeddings to target-model weights, while output regularization preserves prior task-specific configurations without explicitly storing datasets.

    Source: §1 · pp. 1–2

  • Results

    The approach achieves state-of-the-art results on standard benchmarks and suggests long memory lifetimes and forward information transfer, including in a compressive regime.

    Source: §1 · p. 2, §3 · p. 6, §5 · p. 9

  • Takeaways & Limitations

    Task-conditioned hypernetworks provide a scalable, generic continual-learning framework usable alone or with generative replay.

    Source: §5 · p. 9

    The output regularizer protects previous solutions, allowing only weak backward transfer and leaving refinement of past memories for future work.

    Source: §4 · p. 8

Abstract

from arXiv · show

Artificial neural networks suffer from catastrophic forgetting when they are sequentially trained on multiple tasks. To overcome this problem, we present a novel approach based on task-conditioned hypernetworks, i.e., networks that generate the weights of a target model based on task identity. Continual learning (CL) is less difficult for this class of models thanks to a simple key feature: instead of recalling the input-output relations of all previously seen data, task-conditioned hypernetworks only require rehearsing task-specific weight realizations, which can be maintained in memory using a simple regularizer. Besides achieving state-of-the-art performance on standard CL benchmarks, additional experiments on long task sequences reveal that task-conditioned hypernetworks display a very large capacity to retain previous memories. Notably, such long memory lifetimes are achieved in a compressive regime, when the number of trainable hypernetwork weights is comparable or smaller than target network size. We provide insight into the structure of low-dimensional task embedding spaces (the input space of the hypernetwork) and show that task-conditioned hypernetworks demonstrate transfer learning. Finally, forward information transfer is further supported by empirical results on a challenging CL benchmark based on the CIFAR-10/100 image datasets.

1 INTRODUCTION

Continual learning presents tasks sequentially, requiring models to retain prior performance while learning new tasks and potentially transferring knowledge. The paper addresses this challenge by preserving task-specific parameter configurations instead of replaying all prior input-output data.

  • Continual learning presents tasks sequentially and seeks retention, positive backward transfer, and positive forward transfer across tasks.Source: §1 · p. 1
  • The proposed task-conditioned hypernetwork maps task embeddings to target-model weights, preserving prior solutions at the metamodel level.Source: §1 · p. 1
  • Replacing rehearsal of prior datasets with stored task-specific parameter configurations yields a low-dimensional memory representation and substantial memory savings.Source: §1 · p. 2
  • The approach reports no catastrophic forgetting on standard benchmarks and minimal performance loss over very long task sequences.Source: §1 · p. 2
  • Task-conditioned hypernetworks exploit task similarities for forward information transfer and also extend to generative replay methods.Source: §1 · p. 2

2 MODEL

The model uses task-conditioned hypernetworks to generate target weights from task and chunk embeddings, while output regularization protects previously learned configurations. Chunking supports compression, and the framework also accommodates unknown task identities and hypernetwork-protected replay.

Task-conditioned hypernetworks

  • Hypernetworks generate target-model weights from learned task embeddings rather than directly learning the target parameters.The metamodel output is the target network's parameter set.Source: §2.1 · p. 2, Figure 1 · p. 2

Memory preservation

  • Rehearsal-based output regularization requires storing and iterating over previous data and may fail to constrain function behavior in relevant regions.Source: §2.1 · p. 3
  • Output regularization preserves prior task solutions by constraining hypernetwork outputs, avoiding explicit storage of previous task data.A two-step optimization first proposes a task-loss update, then computes the actual change under a total loss with a memory-preserving term.Source: §2.1 · p. 3

Chunked hypernetworks

  • Chunked hypernetworks generate target weights iteratively, enabling reusable smaller hypernetworks and a compressive regime with fewer learned than target parameters.Source: §2.2 · p. 3
  • Chunk embeddings distinguish target-network partitions when a shared hypernetwork is reused across chunks.The full target parameter set is formed by iterating over chunk embeddings while keeping the task embedding fixed.Source: §2.2 · pp. 3–4

Unknown task identity and replay

  • The framework supports task inference from predictive uncertainty when task identity is unavailable and can protect replay generators with the same output-regularization principle.The replay extension combines synthetic replay with task-conditional metamodelling to reduce forgetting.Source: §2.3 · p. 4

3 RESULTS

Across standard and extended continual-learning benchmarks, task-conditioned hypernetworks retain previous tasks while supporting transfer and model compression. Results span toy regression, MNIST variants, and CIFAR-10/100 experiments.

Nonlinear regression

  • On nonlinear regression, output regularization protects earlier polynomial solutions, matching an offline learner, whereas fine-tuning forgets past tasks.The task sequence contains noisy samples from polynomials of increasing degree.Source: §3 · p. 5, Figure 2 · p. 5

Compression

  • Hypernetworks maintain good accuracy even when target-model parameters exceed their own, with performance remaining approximately constant across a range of compression ratios below one.The PermutedMNIST-10 compression experiment reports final and during-task task-averaged test accuracy across five random seeds.Source: §3 · p. 6

Permuted MNIST

  • For Permuted MNIST-10, hypernetwork-based models are the best performers across all three continual-learning scenarios.HNET+ENT infers task identity from predictive-distribution entropy, while HNET+TIR and HNET+R use hypernetwork-protected replay variants.Source: §3 · pp. 5–6, Table 1 · p. 7
  • At T = 100 Permuted MNIST tasks, task-conditioned hypernetworks show minimal memory decay and high-performance solutions in a compressive regime.SI and DGR+distill degrade gracefully, while online EWC remains rigid and cannot reach high accuracy; HNET+TIR strongly outperforms related work in CL2/3.Source: §3 · p. 6, Figure 3 · p. 6

Split MNIST

  • On Split MNIST, HNET+R improves DGR+distill on CL2 and CL3, while HNET+ENT outperforms EWC and SI without generative modeling.The benchmark uses five sequential binary classification tasks formed by pairing digits, creating task overlap.Source: §3 · p. 6, Table 1 · p. 7

Task embeddings

  • Two-dimensional task embeddings support high performance with virtually no forgetting, transfer across overlapping tasks, and moderate above-chance performance for earlier embeddings.The first task is solved across a volume containing later task embeddings, while the last task occupies a finite high-performance region.Source: Figure 4 · p. 7, §3 · p. 7

4 DISCUSSION

The discussion contrasts Bayesian and task-conditioned approaches to continual learning, highlighting hypernetworks’ retention and transfer properties alongside limitations and biological interpretation.

  • On split CIFAR-10/100, hypernetwork-protected ResNet-32 shows virtually no forgetting, with final averaged performance matching immediate performance.Source: Figure 5 · p. 8
  • Bayesian continual learning can produce stiff models because approximate inference seeks a compromise within the first task’s mode.Source: §4 · p. 8
  • Task-conditioning methods commonly use masks over modules, neurons, or weights, but work best with large networks and add storage overhead.Source: §4 · p. 8
  • The output regularizer protects previously learned solutions, so only weak backward information transfer can occur; improving backward transfer remains future work.Source: §4 · p. 8
  • A literal cortical implementation would require dynamically changing an entire network’s connectivity, although lower-dimensional modulatory signals offer a relaxed interpretation.Source: §4 · pp. 8–9

5 CONCLUSION

The paper concludes that task-conditioned hypernetworks provide a scalable, generic continual-learning model that retains long-lived memories and transfers information to future tasks.

  • Task-conditioned hypernetworks parameterize target functions with compressed task embeddings and preserve prior configurations through an output regularizer.Source: §5 · p. 9
  • The approach applies standalone or with generative replay, achieving state-of-the-art benchmark results, long memory lifetimes, and forward information transfer.Source: §5 · p. 9

A TASK-CONDITIONED HYPERNETWORKS: MODEL SUMMARY

The model uses a task-conditioned hypernetwork to generate target-network weights from task embeddings, while regularization preserves earlier task-specific outputs.

B ADDITIONAL DETAILS ON HYPERNETWORK-PROTECTED REPLAY MODELS

The replay extensions protect generative models with hypernetworks and use synthetic data for task inference and continual protection across VAE- and GAN-based setups.

Variational autoencoders

  • VAE replay generates past-task samples by varying task indicators and drawing latent samples, then mixes them with current-task data.Source: B ADDITIONAL DETAILS ON… · p. 13
  • In HNET+TIR and HNET+R, a task-conditioned hypernetwork parameterizes the VAE decoder and combines with output regularization to retain memories.Source: B ADDITIONAL DETAILS ON… · p. 13
  • The replay subsystem is optimized independently with a regularized VAE loss balancing reconstruction and prior-matching penalties.Source: B ADDITIONAL DETAILS ON… · p. 13

Task inference network

  • The task-inference classifier uses a growing softmax over tasks and replay data from the protected VAE to avoid catastrophic forgetting.Source: B ADDITIONAL DETAILS ON… · pp. 13–14
  • Synthetic data is necessary for task inference because its cross-entropy objective requires at least two ground-truth classes, and replay can be generated online.Source: B ADDITIONAL DETAILS ON… · p. 14

Replay model setups

  • Figure A1 depicts protected VAE, GAN, and task-inference-classifier configurations used across the replay experiments.Source: Figure A1 · p. 14

Hypernetwork-protected GANs

  • For GAN replay, the hypernetwork maps a condition to the generator’s full parameters, enabling one-condition-at-a-time conditional GAN training.Source: B ADDITIONAL DETAILS ON… · p. 14
  • The class-incremental MNIST GAN used a fully connected chunked hypernetwork with two hidden layers of size 25 and an output size of 75,000.Source: B ADDITIONAL DETAILS ON… · p. 14

C ADDITIONAL EXPERIMENTAL DETAILS

The appendix details experimental configurations for nonlinear regression, MNIST-based continual learning, replay models, and hyperparameter searches. It also records architectural choices, parameter counts, and limitations of the search procedure.

Training procedure

Nonlinear regression

  • The nonlinear regression experiment uses noisy samples from sequential polynomial tasks and a two-hidden-layer target network with 141 weights.The target and hypernetwork each use two fully connected hidden layers of 10 neurons; task embeddings have dimension 2.Source: C ADDITIONAL EXPERIMENTAL DETAILS · p. 15
  • Each nonlinear regression task is trained for 4000 iterations with Adam, a 0.01 learning rate, and batch size 32.Source: C ADDITIONAL EXPERIMENTAL DETAILS · p. 15

MNIST experiments

  • The MNIST experiments use fully connected target networks and match the reviewed benchmark setup in training iterations, learning rates, optimizer, and comparable parameter counts.The appendix states that hypernetworks use the same or fewer weights than the target network where applicable.Source: C ADDITIONAL EXPERIMENTAL DETAILS · p. 15
  • For the VAE in PermutedMNIST-10, the hypernetwork has 2,211,907 weights versus 2,227,024 in the corresponding target network.The configuration uses task embeddings of size 24, chunk embeddings of size 8, and βoutput = 0.05.Source: C ADDITIONAL EXPERIMENTAL DETAILS · pp. 15, 17

Hyperparameter selection

  • The appendix varies hypernetwork depth, output size, transfer functions, and learning rates, but explores only a random subset of possible hyperparameter combinations.Promising configurations were manually adjusted to approximately match target-network parameter counts; architecture strongly influences performance.Source: C ADDITIONAL EXPERIMENTAL DETAILS · pp. 17–18

D ADDITIONAL EXPERIMENTS AND NOTES

Additional experiments examine benchmark replication, replay and task inference, compression, regularization, and long task sequences. They report competitive accuracy, robust regularization behavior, efficient stochastic regularization, and unresolved task-boundary requirements.

Split CIFAR-10/100

Regularization

PermutedMNIST

Efficient regularization

  • 94.19 ± 0.16 final average accuracy is obtained on PermutedMNIST-250 while estimating the regularizer from at most 32 random previous tasks per iteration.This stochastic subset reduces the regularizer’s runtime dependence on the total number of tasks.Source: D ADDITIONAL EXPERIMENTS AND NOTES · p. 25

Open limitation

E UNIVERSAL FUNCTION APPROXIMATION WITH CHUNKED NEURAL

The appendix argues that chunked hypernetworks retain universal approximation capability under a discrete chunk-index construction. It proves approximation through continuity, universal approximation, and the triangle inequality, while noting that useful compression bounds remain unavailable.

Proposition

  • A continuous function on a compact domain can be approximated by a chunked neural network with discrete chunk set C and output dimension r.The proposition specifies a compact input subset K and a target dimension n = r · NC.Source: E UNIVERSAL FUNCTION APPROXIMATION… · p. 26

Proof strategy

Limitations

  • The result leaves the number of chunks, chunk dimension, and embedding dimension unspecified, and provides no constructive bound on performance or compression rate.These aspects are evaluated empirically rather than derived theoretically.Source: E UNIVERSAL FUNCTION APPROXIMATION… · p. 27

Related papers

Loading 1906.00695v4…