Source-linked AI summary
Superposition of many models into one
Brian Cheung, Alex Terekhov, Yubei Chen, Pulkit Agrawal, Bruno Olshausen
TL;DR
Online learning can cause a neural network to lose performance on earlier tasks, while methods such as replay may not suit memory-constrained settings. The paper stores multiple task models in parameter superposition and retrieves them with task-specific contexts. It reports overcoming forgetting across online image-classification settings and outperforming existing methods.
Problem
Method
Parameter superposition stores task weights together and uses task-specific contexts to retrieve the model for each task.
Source: §1 · p. 2Results
Takeaways & Limitations
The approach uses excess network capacity during training to learn multiple tasks, including in memory-constrained settings.
Source: §1 · pp. 1–2General rotational superposition requires M² additional parameters per task and is not memory efficient when M is approximately N.
Source: §2.1 · p. 3
Abstract
from arXiv · showhide
We present a method for storing multiple models within a single set of parameters. Models can coexist in superposition and still be retrieved individually. In experiments with neural networks, we show that a surprisingly large number of models can be effectively stored within a single parameter instance. Furthermore, each of these models can undergo thousands of training steps without significantly interfering with other models within the superposition. This approach may be viewed as the online complement of compression: rather than reducing the size of a network after training, we make use of the unrealized capacity of a network during training.
1 Introduction
The introduction frames parameter superposition as a way to use a network’s excess capacity during training, storing multiple task models in one parameter set. Task-specific contexts retrieve models while helping limit interference; the method is presented as an alternative to approaches that restrict weight reuse.
1 Introduction
- Over-parameterized networks can often be pruned after training, but the introduction identifies training a smaller network from scratch to comparable accuracy as unresolved.The paper instead proposes exploiting excess capacity while learning multiple tasks.Source: §1 · p. 1
- Task parameters may interfere as they change during training; the paper says suitable contexts can reduce this under a low-dimensional-input assumption.The stated assumption is that inputs occupy a lower-dimensional space than their ambient representation.Source: §1 · p. 1
- Earlier approaches to catastrophic forgetting use replay, separate task networks, or selectively protected weights, while weight-protection methods can limit future reuse and require L extra variables per task.Replay buffers are described as unsuitable when memory is constrained.Source: §1 · p. 2
- The method stores multiple task-specific models in superposition within one network, using task-specific contexts to retrieve their parameters.The network parameters act as a memory and contexts as keys that route inputs to the requested model.Source: §1 · pp. 1–2
- The approach is reported to overcome forgetting across permuting MNIST, rotating MNIST, fashion MNIST, and incremental CIFAR online-learning settings.The settings include changing input distributions and changing output-label distributions.Source: §1 · p. 2
- One variant requires only one additional variable per task, and the proposed method does not explicitly constrain how individual weights change within the superposition.The paper presents this as a substantially smaller per-task overhead than weight-protection approaches.Source: §1 · p. 2
2 Parameter Superposition
Parameter superposition stores task-specific weights after transforming them into nearly orthogonal subspaces, then uses task contexts to retrieve them. Context choices trade memory cost and flexibility, with binary contexts offering low-precision representations.
2 Parameter Superposition
- Task-specific transformations can place the rows of each model’s weights in mutually orthogonal subspaces, allowing the transformed parameters to be summed for storage.The method assumes each task’s weights use only a small subspace of the ambient space.Source: §2 · pp. 2–3
- Task contexts retrieve an individual model from the superposition, although the retrieved weights may be noisy and are expected to preserve performance when the output error stays small.The paper compares retrieval to representing a signal as a superposition of components.Source: §2 · p. 3
- The method rotates inputs into orthogonal subspaces and is motivated by over-parameterized inputs that occupy a low-dimensional manifold.Natural signals such as images and speech are given as examples of data with this property.Source: §2 · p. 3
2.1 Choosing the Context
- General rotational contexts sampled from O(M) require M² additional parameters per task and are described as not memory efficient when M is approximately N.In that setting, training K tasks requires MN +(K−1)M² parameters, approximately the cost of separate task networks.Source: §2.1 · p. 3
- Restricting contexts to subsets such as diagonal matrices reduces the added cost to M parameters per task in the diagonal case.Diagonal contexts apply element-wise multiplication between the context vector and input.Source: §2.1 · p. 3
- Binary superposition restricts phase values to 0 or π, yielding context entries of −1 or 1 with computational and memory advantages.The representation is compatible with real-valued and low-precision linear transformations.Source: §2.1 · p. 4
3 Neural Network Superposition
The paper extends superposition to neural networks by applying it to each layer’s linear transformation. For convolutional networks, it applies contexts to kernels rather than input images to reduce computation.
3 Neural Network Superposition
- Superposition extends to whole neural networks by applying the formulation to the linear transformation in every layer.The layer transformations are followed by a non-linearity such as ReLU.Source: §3 · p. 4
- For convolutional networks, applying the context tensor to the kernel instead of the image reduces computation because convolution parameters are usually smaller than the input image.The passage defines M and N as the input- and output-channel dimensions, respectively.Source: §3 · p. 4
4 Experiments
The experiments test parameter superposition (PSP) against catastrophic forgetting under changes in input distributions and output labels. Across sequential image-classification settings, PSP retains earlier-task performance, including with binary superposition and a single output layer.
4.1.1 Effect of network size on catastrophic forgetting
- With 2048-unit hidden layers, binary PSP leaves initial-task performance virtually unchanged after 49 later permuted-MNIST tasks.Across networks with 128 to 2048 hidden units, PSP is more robust to forgetting than the standard baseline, and its performance improves with network size.Source: §4.1.1 · p. 5, Figure 2 · p. 5
4.1.2 Effect of types of superposition on catastrophic forgetting
- On the last ten permuted-MNIST tasks after training on 50 tasks, PSP outperforms EWC and SI; among PSP variants, rotation slightly exceeds complex, which exceeds binary.Rotation uses M^2 additional parameters per task, versus M for complex and binary superposition.Source: §4.1.2 · pp. 5–6, Figure 3 · p. 6
4.1.3 Continuous Domain Shift
- On rotating image streams, PSP remains robust as inputs shift gradually, with performance trends similar to those on permuting MNIST.Images rotate through 360 degrees every 1000 steps; contexts change every 100 steps and are reused across cycles.Source: Figure 4 · p. 6, §4.1.3 · p. 7
- Without detailed task identity, pspFast stores 1000 models and performs worse than detailed-context PSP, while adjacent-phase mixing improves over pspFast.The authors identify less training data per model and increased interference as possible reasons for pspFast's weaker performance.Source: §4.1.3 · p. 7
4.2 Output Interference
- With ResNet-18 and one output layer, binary PSP is virtually unaffected by sequential CIFAR-100 label shifts that critically affect standard and multihead baselines.The evaluation measures accuracy on CIFAR-10 after training on CIFAR-10 and then four disjoint CIFAR-100 class sets.Source: Figure 6 · p. 8, §4.2 · p. 8
5 Discussion
The discussion presents parameter superposition as a way to diminish catastrophic forgetting, with reported robustness across input and output interference. It also identifies limits in understanding storage capacity and in selecting contexts automatically.
5 Discussion
- The method is reported to be robust to catastrophic forgetting from input and output interference and to outperform existing methods.It works with fully connected and convolutional networks and can scale to ResNet; coarse information about task-distribution changes can be incorporated without relying completely on task identity.Source: §5 · p. 8
- A thorough analysis of how many models can be stored remains needed, and capacity is likely to depend on architecture and task family.Source: §5 · p. 8
- Automatically and dynamically determining context, including making it differentiable rather than fixed, remains a future direction.Source: §5 · p. 9
Supplementary Material
The supplementary material analyzes retrieval noise and outlines supporting results on offline storage, online training, and compositional generation of complex vectors.
Supplementary Material
- The appendices address small-noise retrieval after offline storage, online training of superimposed models, and compositional generation of complex vectors.Source: Supplementary Material · p. 11
- Substituting Equation 1 into Equation 3 and unpacking the inner product yields the two-term form in Equation 9.Source: Supplementary Material · p. 11
- Retrieval decomposes into the recovered linear transformation W(k)x and a residual ϵ, which can sum destructively interfering terms for particular contexts.Source: Supplementary Material · p. 11
A Analysis of retrieval noise
The analysis assumes fixed vectors w and x and a random context vector c whose elements have unit amplitude and uniformly distributed phase.
A Analysis of retrieval noise
- The analysis fixes w and x, while each element of random context c has unit amplitude and uniformly distributed phase.Source: A Analysis of retrieval noise · p. 11
A.1 Proposition 1: superposition bias analysis
Under uniform context distributions, Proposition 1 finds no expected bias from other superimposed models in the recovered linear transformation. Proposition 2 describes how context-induced interference depends on input alignment and the number of superimposed models.
A.1 Proposition 1: superposition bias analysis
- Uniformly distributed binary, complex, or rotational contexts make other superimposed models introduce no expected bias to the recovered linear transformation.The proof considers fixed weights and input vectors across the three context types.Source: §A.1 · p. 11
A.2 Proposition 2: Variance induced by context vectors
- When η is large, context-bound inner products are relatively small compared with the unbound inner product.The passage applies this comparison to both element-wise context vectors and rotational contexts.Source: §A.2 · p. 12
- With equally large model norms, the residual error scales as K^-1.This scaling is stated under the assumption that the norms of the superimposed models are equally large.Source: §A.2 · p. 12
B Online learning with unitary transformations
The section establishes conditions for training a model in superposition with updates approximately matching independent training. It also describes the geometry of the context transformations.
B Online learning with unitary transformations
- Parameter updates for a model in superposition are approximately equal to updates from training that model independently, supporting online memory operations.The section states this for binary, complex, and rotational context cases.Source: B Online learning with unitary… · p. 14
- For binary, complex, and rotational contexts, the respective update conditions are satisfied by the chosen transformations.The binary transformation is orthogonal because each context component has square 1; the complex and rotational cases also satisfy their stated conditions.Source: B Online learning with unitary… · pp. 15–16
C Geometry of Rotations
- The context transformations define the topology of the embedding space for superimposed models.Supplementary Figure 1 illustrates the geometry of rotations applied to parameter vectors.Source: C Geometry of Rotations · p. 16
D From superposition to composition
Contexts can be composed to construct new contexts and retrieve parameters from their compositions. Complex contexts also support mixtures that create smoother transitions between neighboring contexts.
D From superposition to composition
- Complex contexts form a Lie group under multiplication, allowing parameters to be stored and recovered from composed contexts.The composition is written as c^(a+b) = c^a ⊙ c^b.Source: D From superposition to composition · p. 16
- Binary, complex, and rotational context operators act on parameter vectors with lattice, torus, and sphere topologies, respectively.These geometries are depicted in Supplementary Figure 1.Source: Figure 1 · p. 17
- Averaging neighboring complex contexts creates smoother transitions that reduce their orthogonality and can support information sharing when domain shift is smooth.The passage identifies transfer learning and continual learning as settings where such sharing is useful.Source: D From superposition to composition · p. 17
E Additional Results
Supplementary Figure 3 compares methods' accuracy over training steps on the first permutingMNIST task using networks with 2000 units.
E Additional Results
- Supplementary Figure 3 compares PSP methods' accuracy over training steps on the first permutingMNIST task for 2000-unit networks; pspRotation is omitted because its memory and computation footprint is comparable to independent networks.The caption describes the comparison but does not state its outcome.Source: Figure 3 · p. 18