PaperlayerView exact revision on arXiv

Source-linked AI summary

DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion

Arthur Douillard, Alexandre Ramé, Guillaume Couairon, Matthieu Cord

arXiv:2111.11326v3cs.CVcs.LG

TL;DR

Continual learning needs to acquire new classes without forgetting old ones, yet prior dynamic approaches can require task identifiers, tuning, or substantial overhead. DyTox shares transformer encoder and decoder components while expanding only task-specific tokens. It achieves state-of-the-art results on ImageNet100 and ImageNet1000 and strong results across CIFAR100, ImageNet100, and ImageNet1000 with limited parameter growth.

  • Problem

    Continual learning must adapt to new classes when previous data are unavailable, while avoiding catastrophic forgetting; existing approaches can require task identifiers, memory overhead, or tuning.

  • Method

    DyTox shares self-attention and decoder components across tasks, adding task-specific learned tokens that produce specialized embeddings and independent class predictions.

  • Results

    DyTox achieves state-of-the-art performance across the evaluated settings, including 71.29% “Avg” top-1 and 63.34% “Last” top-1 accuracy on ImageNet1000.

  • Takeaways & Limitations

    The token-expansion strategy scales to many tasks with limited parameter growth, small overhead, and no complex expansion-tuning hyperparameters.

  • Takeaways & Limitations

    The evaluation follows a setting where only a few samples from previous tasks remain available for rehearsal, rather than fully inaccessible prior data.

Abstract

from arXiv · show

Deep network architectures struggle to continually learn new tasks without forgetting the previous tasks. A recent trend indicates that dynamic architectures based on an expansion of the parameters can reduce catastrophic forgetting efficiently in continual learning. However, existing approaches often require a task identifier at test-time, need complex tuning to balance the growing number of parameters, and barely share any information across tasks. As a result, they struggle to scale to a large number of tasks without significant overhead. In this paper, we propose a transformer architecture based on a dedicated encoder/decoder framework. Critically, the encoder and decoder are shared among all tasks. Through a dynamic expansion of special tokens, we specialize each forward of our decoder network on a task distribution. Our strategy scales to a large number of tasks while having negligible memory and time overheads due to strict control of the parameters expansion. Moreover, this efficient strategy doesn't need any hyperparameter tuning to control the network's expansion. Our model reaches excellent results on CIFAR100 and state-of-the-art performances on the large-scale ImageNet100 and ImageNet1000 while having less parameters than concurrent dynamic frameworks.

1. Introduction

Continual learning must preserve earlier classes while adapting to evolving data, but existing dynamic approaches can require task identifiers, substantial memory, or tuning. DyTox addresses these constraints with shared transformer components and task-specific tokens, achieving strong scalable performance.

  • Continual learning balances preserving old distributions with adapting to new classes or domains when previous data become inaccessible.
  • Existing dynamic methods may require the test sample’s task identifier, while identifier-free alternatives incur dramatic memory overhead and complex pruning.
  • DyTox shares encoder layers and a decoder layer across tasks, specializing decoder forwards with one learned task token per task.
  • Each new task adds only a 384d vector, while the shared decoder remains computationally lightweight and requires no expansion-tuning hyperparameters.
  • DyTox reaches state-of-the-art results across CIFAR100, ImageNet100, and ImageNet1000 while incurring only a small overhead.

2. Related work

Related continual-learning methods use rehearsal, constrain model changes, or dynamically specialize network components. Their trade-offs include retained data, task-identifier requirements, pruning sensitivity, and increasing parameter costs, motivating DyTox’s shared-token design.

  • Rehearsal-based methods retain limited old-class data in raw, compressed, or trimmed form, while other approaches store generators for past classes.
  • Dynamic architectures create task-specialized subnetworks, but many require a task identifier at test time to select parameters.
  • DER avoids task identifiers by concatenating embeddings from task-specific feature extractors, but requires aggressive pruning to control parameter growth.
  • DER’s pruning is hyperparameter-sensitive, requiring different settings for different numbers of learning steps.
  • DyTox’s transformer design uses shared self-attention blocks, task-attention processing, and task-specific tokens for continual classification.

3. DyTox transformer model

DyTox uses a shared transformer encoder and task-specialized decoder forwards to produce embeddings and classifiers for continually introduced tasks. New tasks expand only the learned task-token set, while automatic loss weighting and linear-complexity task attention limit overhead.

  • Shared transformer backbone: DyTox processes images through shared Self-Attention Blocks, producing patch tokens that retain dimension (N+1)×D across the encoder.The encoder is followed by a single Task-Attention Block rather than separate full networks.
  • Task-Attention Block: Each task contributes a learned task token θi, which is concatenated with the encoder’s patch tokens and processed by Task-Attention to create a task-specific embedding.Unlike the class token in standard ViT, task tokens are added only at the last block.
  • Dynamic task token expansion: After t steps, DyTox retains t task tokens and runs the Task-Attention Block once per task, producing embeddings that feed independent task-specific classifiers.The classifiers use sigmoid outputs and binary cross-entropy, rather than concatenating all task embeddings into one classifier.
  • Dynamic task token expansion: The test-time forward pass evaluates every learned task token and concatenates the resulting predictions for all seen classes, without requiring the test image’s task identifier.The algorithm returns predictions for all classes in C1:t after applying each task-specific classifier.
  • Efficiency and training: Only a new 384-dimensional task token is added per task, and task attention has linear patch complexity versus quadratic self-attention, making the time overhead sub-linear.The loss weight α is automatically set from the ratio of old to new classes, avoiding fine-tuning of that hyperparameter.
  • Efficiency and training: DyTox trains with classification, probability-distillation, and divergence losses; distillation reduces forgetting, while the divergence term uses the current last task’s embedding.The coefficient λ is fixed at 0.1, whereas α is automatically determined.

4. Experiments

DyTox is evaluated across CIFAR100, ImageNet100, and ImageNet1000 under multiple continual-learning settings, achieving strong accuracy with controlled memory and computational overhead. Additional analyses examine its training procedure and architectural components.

  • Benchmarks & implementation: DyTox is evaluated on CIFAR100, ImageNet100, and ImageNet1000 using multiple task settings and Avg, Last, top-1, and top-5 accuracy metrics.CIFAR100 uses 10, 20, or 50 steps; ImageNet uses 10 steps.
  • Quantitative results: 71.29% Avg top-1 accuracy and 63.34% Last top-1 accuracy are reached by DyTox on ImageNet1000, outperforming DER w/o P despite fewer parameters.DyTox also exceeds pruned DER† by +4.56 p.p. in Avg top-1 and +1.51 p.p. in Avg top-5 accuracy.
  • Quantitative results: Up to 52x fewer parameters are used by DyTox than DER w/o P on CIFAR100, while Last accuracy is up to +25% above other baselines in the 50-step setting.Performance declines as the number of continual-learning steps increases, but DyTox remains close to DER w/o P with much fewer parameters.
  • Overhead: +0.004% memory overhead per task is incurred by adding one 384-dimensional vector, reaching only +0.2% in CIFAR100 with 50 tasks.The growing classifier is excluded because it is common to all continual-learning models.
  • Overhead: 2.24% more time is required for one forward pass per new task than for the base transformer.Shared self-attention blocks contain most computation, while the dynamic task-attention component has linear rather than quadratic token complexity.
  • Training procedure introspection: MixUp improves CIFAR100 continual-learning performance more strongly than joint-learning performance, with a +4.75 p.p. gain in the 50-step scenario versus +1.39 p.p. jointly.The analysis attributes the larger continual-learning gain to reduced forgetting rather than only improved raw performance.
  • Model ablations: DyTox combines task-token expansion, a divergence classifier, and independent classifiers; independent classifiers reduce forgetting by more than 24%.The divergence classifier increases the minimum Euclidean distance between task tokens by 8%.

5. Conclusion

DyTox is a transformer-based dynamic continual-learning strategy that shares self-attention and task-attention parameters while adding task-specific tokens. The paper reports scalable performance with low memory growth, but notes that continual learning still does not solve forgetting across an almost unlimited number of tasks.

  • Conclusion: DyTox shares self-attention layers across tasks and adds task-specific tokens processed by a task-attention layer to produce task-specialized embeddings.The architecture uses a shared transformer structure while dynamically processing new tasks.
  • Conclusion: State-of-the-art performance is reported on ImageNet1k, with reasonable parameter growth in the 50-step CIFAR100 setting.The conclusion contrasts this growth with previous dynamic strategies.
  • Limitations: Continual learning still lacks an approach that handles an almost unlimited number of tasks with low forgetting.The paper characterizes DyTox as a step toward that unresolved goal rather than a complete solution.

A.1. Experimental details

The experiments use three image-classification datasets and fixed training procedures, with hyperparameters tuned on CIFAR100 with 10 steps before transfer to other settings. Results are compared with established continual-learning baselines.

  • Implementation: Models are trained for 500 epochs per task with Adam, followed by 20 epochs of balanced-data fine-tuning after each task except the first.The base learning rate is 5e−4, with 5 warmup epochs, and fine-tuning uses 5e−5.
  • Hyperparameter tuning: Hyperparameters are tuned on a 10% CIFAR100 validation subset using 10 steps, then fixed for all datasets and numbers of steps.This procedure is intended to avoid optimizing separately for the number of tasks.
  • Baselines: Baseline results come from the respective papers for E2E and Simple-DER and from the DER paper for the other baselines.The comparison includes knowledge-distillation, fine-tuning, rehearsal, and dynamic-network approaches.

A.2. Parameter sharing of the TAB

DyTox shares the TAB across tasks while retaining task-specific tokens, trading some accuracy for substantially lower parameter overhead.

  • DyTox shares encoder and decoder parameters across tasks, retaining only different task tokens to specialize each task.This shares common information while limiting dynamic expansion.
  • 56% “Last” accuracy is achieved with a separate TAB per task, compared with 52% for DyTox’s shared-TAB strategy.The separate-TAB variant allows greater task-specific diversity.
  • 97M versus 10M parameters shows the memory cost of separate TABs compared with DyTox’s shared-TAB design.DyTox uses one shared TAB and distinct tokens per task.
  • DyTox strikes a balance between continual performance and memory overhead.The shared design sacrifices some accuracy relative to separate TABs while greatly reducing parameters.

A.3. Novel continual training procedure

DyTox+ combines MixUp with DyTox, while DyTox++ adds Sharpness-Aware Minimization to reduce task overspecialization. These procedures improve continual-learning performance and forgetting, especially in longer task sequences.

  • DyTox++ adds Sharpness-Aware Minimization to DyTox+ after MixUp, targeting wider minima and reduced task-specific overspecialization.Sharp task-specific minima can promote forgetting of other tasks.
  • ASAM doubles forward and backward passes, so it is evaluated separately rather than included in the main experiments.The paper notes that lower-overhead alternatives such as Look-SAM could be considered.
  • +4.7% “Avg” improvement over DER is reported on CIFAR100 50 steps, and +4.6% on ImageNet100 10 steps.These gains are attributed to applying MixUp and ASAM to DyTox.
  • +6.42 p.p. is the gain from MixUp and ASAM in the CIFAR100 50-step continual setting, compared with +1.79 p.p. in joint training.The larger continual-setting gain indicates the procedures are not explained by absolute model improvements alone.
  • Forgetting decreases from 33.15% with DyTox to 30.47% with DyTox++, a total reduction of 2.68 p.p.DyTox+ has 31.50% forgetting in between.

A.4. Patch size effect on forgetting

Smaller patches improve joint-training performance, but patch resolution has little effect on forgetting in continual learning. DyTox also remains effective with a convolutional backbone.

  • 33.15% forgetting with patch size 4 and 33.20% with patch size 16 show little sensitivity to patch resolution.The comparison is reported for CIFAR100 continual learning.
  • Patch size 4 performs best in joint training, although its forgetting is nearly identical to patch size 16.The paper therefore selects patch size 4 for the main experiments.
  • A modified ResNet18 can replace the SAB feature extractor while preserving the DyTox framework’s effectiveness.The convolution-based variant performs slightly below DyTox with SABs but remains above prior state-of-the-art such as WA in “Last” accuracy.

A.6. Alternative task decoders

DyTox conditions task-specific embeddings through a TAB, while residual adapters and FiLM provide alternative task-conditioning mechanisms. The alternatives are competitive on CIFAR100 but forget more on complex ImageNet100.

  • Residual adapters require a task or domain identifier at test time to select the appropriate residual branch.This requirement is described for the original residual-adapter approach.
  • DyTox applies a residual adapter or FiLM after global token pooling and sends each task-specific embedding to its task-specific classifier.These alternatives adapt feature conditioning to the transformer backbone.
  • Residual adapters and FiLM perform similarly to TAB on CIFAR100 50 steps.The comparison is reported for the smaller and easier dataset.
  • On ImageNet100 10 steps, FiLM and residual adapters forget significantly more than the complete DyTox framework.The performance gap emerges as task complexity increases.

B.1. Interest of the Distributed Memory

Distributed memory assigns each process or GPU its own memory, producing diverse rehearsal selections with low overlap and a direct CIFAR performance gain.

  • Each GPU or machine maintains its own memory, making distributed memory similar to federated learning.
  • Low overlap causes useful examples to be selected twice while different GPUs select examples from different modes, increasing diversity.
  • With 2 GPUs on CIFAR100, selected-example overlap is usually only 1%–10%.
  • Distributed rehearsal produces a direct performance gain on CIFAR, as highlighted in Table 15.

B.2. Updated results

The updated experiments compare global and distributed memory across CIFAR and ImageNet settings, while DyTox maintains strong performance relative to WA.

  • Global and distributed memory both use 20 samples per class overall, with distributed memory splitting them across GPUs.On CIFAR100 with 2 GPUs, each GPU samples at most 10 samples per class.
  • DyTox maintains strong performance, more often than not exceeding WA, while DyTox+ also reaches excellent performance.The passage attributes DyTox+ performance to reduced forgetting.
  • Gray denotes results presented in the original paper version.
  • CIFAR experiments use 2 GPUs with distributed memory and are reported in Table 16.
  • ImageNet experiments use global memory with 8 GPUs and distributed memory with 4 GPUs.The passage notes that distributed memory uses only 5 images per class per GPU and may therefore produce slightly lower results than global memory.
Loading 2111.11326v3…