PaperlayerView exact revision on arXiv

Source-linked AI summary

Incremental Learning Through Deep Adaptation

Amir Rosenfeld, John K. Tsotsos

arXiv:1705.04228v2cs.CVcs.LG

TL;DR

Existing incremental-learning methods face trade-offs among adaptation quality, parameter growth, joint-training requirements, and preserving prior performance. DAN learns new task-specific filters as combinations of frozen base filters and switches among learned representations. It matches or outperforms fine-tuning with typically 13% added parameters, reaching about 3% with compression and no accuracy loss in the reported setting.

  • Problem

    Incremental learning seeks new capabilities across related domains without catastrophic forgetting, joint training, or substantial parameter growth.

  • Method

    DAN attaches controller modules to a frozen network, constraining new filters to linear combinations of existing filters and enabling switching among task representations.

  • Results

    DAN matches or outperforms vanilla fine-tuning, typically adds 13% of the original parameters per task, and reaches 3% with compression without accuracy loss.

  • Takeaways & Limitations

    A single network can preserve learned representations while performing across multiple domains, with controller-based adaptation and task switching.

  • Takeaways & Limitations

    Because adapted filters remain linear combinations of base filters, tasks requiring information in an orthogonal feature space may not be expressible.

Abstract

from arXiv · show

Given an existing trained neural network, it is often desirable to learn new capabilities without hindering performance of those already learned. Existing approaches either learn sub-optimal solutions, require joint training, or incur a substantial increment in the number of parameters for each added domain, typically as many as the original network. We propose a method called \emph{Deep Adaptation Networks} (DAN) that constrains newly learned filters to be linear combinations of existing ones. DANs precisely preserve performance on the original domain, require a fraction (typically 13\%, dependent on network architecture) of the number of parameters compared to standard fine-tuning procedures and converge in less cycles of training to a comparable or better level of performance. When coupled with standard network quantization techniques, we further reduce the parameter cost to around 3\% of the original with negligible or no loss in accuracy. The learned architecture can be controlled to switch between various learned representations, enabling a single network to solve a task from multiple different domains. We conduct extensive experiments showing the effectiveness of our method on a range of image classification tasks and explore different aspects of its behavior.

1 INTRODUCTION

Deep Adaptation Networks incrementally add related-domain capabilities by reusing a frozen base network while preserving prior-task behavior. The method uses controller modules and switching to support multiple domains with substantially fewer added parameters than full fine-tuning.

  • DAN targets incremental learning across related domains while retaining one computational pipeline, modest parameter growth, prior performance, and no joint training requirement.
  • Controller modules augment a network learned for one task by reusing existing representations to learn another task without requiring the original task’s training data.
  • 13% added parameters per new task yielded average performance surpassing full fine-tuning, while preserving old-task performance and avoiding catastrophic forgetting.
  • DAN precisely preserves old-task performance, matches fine-tuning effectiveness, switches among any number of learned tasks, and applies across image-classification datasets.
  • A weaker, lower-parameter variant significantly outperforms shallow feature extraction for comparable parameter counts.
  • A switching variable can select original or re-parameterized convolutions and classifiers, manually or through a Dataset Decider.

2 RELATED WORK

Related work frames incremental learning as a trade-off among preserving old-task performance, adapting effectively, and limiting additional parameters. Existing strategies include feature extraction, output-preserving fine-tuning, added representations, and compression techniques.

  • Multi-task learning commonly trains one network jointly across tasks, using a shared representation and multiple losses as regularization.
  • Feature extraction freezes the network and preserves old performance but is substantially inferior to fine-tuning the entire architecture.
  • Output-preserving fine-tuning adds few parameters but does not guarantee retention of the model’s full old-task ability.
  • Coding, quantization, pruning, and tensor decomposition reduce network weights and can be combined with DAN.

3 APPROACH

The approach describes a deep convolutional classifier as a composition of layer transformations, nonlinearities, and a classifier component. Batch normalization is omitted from the compact notation, and fully convolutional networks may omit the classifier term.

  • A DCNN computes an input representation through layers φ_i interleaved with nonlinearities before applying its classifier.
  • The network output is defined as the composition of the classifier part with the preceding transformed representation.
  • Batch-normalization layers are omitted for brevity, and the classifier term Φ_CN may be dropped for fully convolutional networks.

3.1 Adapting Representations

DAN adapts a frozen base network by learning controller-generated filters and task-specific heads, then selects among representations with switching variables. Its parameterization supports multiple tasks, while experiments compare transferability and related-domain adaptation.

  • Adapting Representations: A controller module is attached to each convolutional layer, using a base network’s existing weights to create filters adapted to a new task.
  • Adapting Representations: Adapted filters are formed by flattening base filters, applying a learned linear transformation, and unflattening the result back to filter tensors.
  • Adapting Representations: The adapted network can switch between base and controller behavior using a binary parameter α.
  • Adapting Representations: Transferability compares top-1 accuracy after fine-tuning from one dataset to another under last-layer, full, or batch-normalization-frozen regimes.
  • Adapting Representations: For multiple tasks, α becomes an n-dimensional binary vector selecting the task-specific adapted filters and classifier head.
  • Adapting Representations: DAN variants are trained with frozen base filters, learned controller weights, task-specific fully connected layers, and fixed or newly learned batch-normalization parameters.

3.2 Additional Design Choices

The section examines controller-network variants and parameter costs, including multiple controllers, weaker diagonal parametrization, and amortized savings across tasks.

  • Weaker parametrization: The diagonal variant only scales original filters, reducing parameters but limiting expressive power and lowering classification accuracy relative to the full linear variant.
  • Multiple controllers: Multiple controller networks can attach to one base network and be switched using a one-hot vector determined by another sub-network.
  • Empirical comparison: Table 1 compares top-1 accuracy and parameter cost across independent baselines and controller variants, with lower amortized parameter cost preferred.
  • Parameter cost: 13% of the original parameters are required to adapt convolutional layers with a new fully-connected layer, though VGG-B requires roughly 21%.
  • Parameter cost: 2.17 · P parameters suffice for 10 classifiers using one base and nine controllers, compared with 10 · P for independently trained networks.

3.3 Limitations

DAN’s filter-combination constraint can limit expressive power or reduce efficiency when the base network’s representations are poorly suited to the new task. Toy experiments show that initialization and fixed early filters strongly affect average performance, although later layers or residual connections can sometimes recover missing information.

  • Expressibility: DAN constrains adapted filters to linear combinations of the base filters, so the base filters define the available subspace for new representations.This constraint can become a strong performance limitation when the required features lie outside that subspace.
  • Expressibility: A base network with first-layer filters lacking task-critical color information can fail because the missing information lies in an orthogonal feature space.If all first-layer filters have zero green and blue coefficients, a task requiring those channels cannot be represented through their combinations.
  • Expressibility: Residual connections can preserve information through an identity path, making expressive power less sensitive to the basis choice in a single layer.The authors nevertheless note that network efficiency may still be compromised.
  • Efficiency: Network efficiency is defined as the shallowest layer at which a single filter can strongly respond to a target image patch.A representation may preserve expressibility while requiring deeper computation to detect the same pattern.
  • Efficiency: A patch directly matched by a first-layer filter may instead require second-layer computation when the first layer uses basis filters that span the patch space less efficiently.The transposed toy case recovers the information through shifted responses but delegates computation to a later layer.
  • Toy Dataset: In the toy experiments, fixed orthogonal or channel-mismatched first-layer filters produced chance or lower average accuracy, while learned or cleanly initialized networks could reach near-perfect performance.The channel-switch case stayed at chance with the first filter fixed; allowing all layers to learn eventually reached good accuracy but averaged around 50%, versus 20% chance.

4 EXPERIMENTS ON REAL DATASETS

Experiments across eight image-classification datasets evaluate DAN using shared base networks, incremental controllers, and alternative initializations. DAN achieves strong average performance, while combining selected base networks further improves accuracy at moderate parameter cost.

  • Evaluation protocol: Eight datasets were evaluated using VGG-B networks, with standardized 64 × 64 preprocessing and dataset-specific train-validation splits.The datasets included Caltech256, CIFAR-10, Daimler, GTSR, Omniglot, Plankton, Human Sketch, and SVHN.
  • Single base networks: DANcaltech−256 achieved 79.9% and DANsketch achieved 83.7% mean performance across the evaluated datasets.The reported parameter cost was normalized to that of a single fully parametrized network.
  • Single base networks: 86.5% mean performance was obtained with DANimagenet, improving over DANsketch’s 83.7% but remaining below the 87.7% scratch baseline.DANimagenet performed better on Caltech-256, while Sketch and Omniglot favored DANsketch.
  • Multiple base networks: 87.76% mean performance was achieved by DANimagenet+sketch, which selected DANimagenet or DANsketch as the base according to the target dataset.Its parameter cost was 2.76 while covering nine tasks, including ImageNet.
  • Random initialization: A randomly initialized base network reached 76.3% mean top-1 precision with an effective parameter cost of 0.22.Only controller modules and fully connected layers were learned, while 10M randomly initialized parameters were excluded from the effective count.

4.2 Initialization

The study compares identity, random, and linear-approximation initialization schemes for controller weights by measuring loss after one epoch on DANsketch→caltech256.

  • Initialization schemes: Identity initialization makes the controller initially mimic the base network, while the alternatives use random noise or a linear approximation of independently trained new-task weights.The three schemes were evaluated by training for one epoch and observing loss.

4.3 Transferability

Transferability analysis examines which learned representations best support new domains and how controller strength shifts performance between base and learned tasks.

  • Transferability: Feature extraction was inferior to fine-tuning all layers, while training from scratch was usually most beneficial across the tested transfer scenarios.The analysis also compared full fine-tuning with batch-normalization parameters frozen.
  • Shifting representations: Increasing α shifts the network from the base representation toward learned tasks, lowering base-task performance while improving learned-task performance.The relatively slow changes for Sketch and Plankton indicate similarity between their learned representations.
  • Transferability: Natural-image features transferred less effectively to Sketch and Omniglot, whereas Sketch or Omniglot initialization worked well for most datasets.Plankton exhibited behavior different from the other mostly natural-image datasets.
  • Automatic switching: A dataset-decider achieved 99% accuracy and selected the controller for each input domain, allowing one network to classify images from multiple datasets.The highest predicted domain value was set to 1 and all others to 0.
  • Method comparison: The full linear DAN method outperformed vanilla fine-tuning on average, while 8-bit quantization caused no significant accuracy effect.The comparison also included feature extraction and a diagonal controller variant.

4.4 Visual Decathlon Challenge

On the Visual Decathlon Challenge, DAN is compared with parameter-efficient transfer methods across ten image-classification datasets. The experiments also test whether quantization preserves accuracy and whether DAN converges quickly.

  • Visual Decathlon results: DAN achieved a slightly higher decathlon score than Residual Adapters at the same architecture capacity, while Residual Adapters had slightly higher mean performance.DAN avoided joint training across all datasets and dataset-dependent weight decay.
  • Visual Decathlon results: Residual Adapters reached a 79.43% mean with parameter cost 12 using a stronger base architecture, whereas the comparison used architectures of similar capacity.A decathlon score of 2500 corresponds to independently fine-tuning the ImageNet network for each dataset.
  • Convergence: The linear DAN method converged to high accuracy faster than fine-tuning, while the weaker variant converged as fast as feature extraction but reached higher overall accuracy.The convergence comparison is shown in Figure 8.
  • Compression: 8-bit quantization caused only a marginal accuracy loss, reducing the added parameter requirement to 3.25% of the original amount.Several datasets maintained performance even at 6 bits.
  • Compression: DAN required significantly fewer parameters than fine-tuning to reach the same accuracy, and its diagonal variant outperformed feature extraction under parameter limits.The comparison evaluated feature extraction, fine-tuning, and diagonal and linear DAN variants across quantization levels.

4.6 Discussion

DAN can converge faster than vanilla fine-tuning and achieve slightly better accuracy, while its constrained representation may fail when related-task assumptions do not hold.

  • DAN converges faster than vanilla fine-tuning and eventually attains slightly better performance.The authors attribute this potential benefit to constraining each layer to a linear combination of the corresponding original layer.
  • The constrained construction limits expressive power, yet the authors observed increased classification accuracy in their experiments.
  • DAN may perform worse than vanilla fine-tuning or training from scratch when tasks are not of the same kind.The discussion gives counting horizontal lines versus counting vertical lines as a simple example where the method could fail.
  • For very few parameters, DAN outperforms feature extraction, and its full method reaches maximal accuracy with fewer parameters than linear fine-tuning.
  • The authors did not observe this failure on real image-classification datasets and leave balancing feature reuse against learning new features for future work.

5 CONCLUSIONS

The paper presents DAN as a parameter-efficient transfer-learning method that preserves existing representations while matching or outperforming vanilla fine-tuning. It also supports switching or smoothly transitioning among representations for different learned tasks.

  • DAN adapts an existing network to new tasks while fully preserving the existing representation.
  • DAN matches or outperforms vanilla fine-tuning while requiring only a fraction of the parameters.With network compression, the parameter cost reaches 3% of the original parameters with no loss of accuracy.
  • DAN enables a single network to switch its representation among multiple learned tasks and domains.The control parameter α can also be represented as a real-valued vector for smooth transitions between task representations.
Loading 1705.04228v2…