Source-linked AI summary
Efficient parametrization of multi-domain deep neural networks
Sylvestre-Alvise Rebuffi, Hakan Bilen, Andrea Vedaldi
TL;DR
Deep networks are highly specialized to individual tasks and domains, while fixed universal feature extractors underperform specialized models. The paper therefore develops compact universal parametric families using shared weights and small domain-specific adapters, finding that parallel, compressed adapters across shallow and deep layers outperform traditional fine-tuning for transfer learning.
Problem
Deep networks specialize strongly to individual tasks and visual domains, and fixed universal feature extractors remain inferior to specialized networks.
Method
The paper partitions networks into shared universal parameters w and small domain-specific adapter parameters α, evaluating residual-adapter designs, compression, regularization, and layer allocation.
Results
Parallel residual adapters distributed throughout the network and jointly rank-compressed produced the best network families, with strong multi-domain performance and transfer-learning gains over fine-tuning.
Takeaways & Limitations
Compact parametric families can preserve domain specialization while substantially reducing the storage, exchange, update, and transmission costs of multiple models.
Takeaways & Limitations
Adapters can be fused with convolutional layers without test-time cost, but fusion is difficult to undo and can prevent retargeting the network to another problem.
Abstract
from arXiv · showhide
A practical limitation of deep neural networks is their high degree of specialization to a single task and visual domain. Recently, inspired by the successes of transfer learning, several authors have proposed to learn instead universal, fixed feature extractors that, used as the first stage of any deep network, work well for several tasks and domains simultaneously. Nevertheless, such universal features are still somewhat inferior to specialized networks. To overcome this limitation, in this paper we propose to consider instead universal parametric families of neural networks, which still contain specialized problem-specific models, but differing only by a small number of parameters. We study different designs for such parametrizations, including series and parallel residual adapters, joint adapter compression, and parameter allocations, and empirically identify the ones that yield the highest compression. We show that, in order to maximize performance, it is necessary to adapt both shallow and deep layers of a deep network, but the required changes are very small. We also show that these universal parametrization are very effective for transfer learning, where they outperform traditional fine-tuning techniques.
1. Introduction
The paper addresses the challenge of building models that work across multiple visual domains while remaining extensible and efficient. It proposes compact parametric network families that preserve domain-specific specialization through small adapters rather than a single fixed feature extractor.
- Multi-domain models should handle diverse visual domains, while extensible models should reuse knowledge for new domains without forgetting prior abilities.
- Universal fixed feature extractors are less effective than problem-specific networks trained from scratch or through transfer learning.
- The proposed family partitions parameters into shared universal weights w and small domain-specific vectors α, enabling compact specialization and learning from few examples.
- Domain-specific adapters support efficient model storage, transfer, exchange, and updating while preserving most shared parameters.
- The study evaluates series and parallel residual adapters, adapter compression, regularization, and parameter allocation to identify effective parametrizations.
- The paper reports strong empirical performance, including a new state of the art on the Visual Decathlon benchmark and excellent transfer-learning capabilities.
2. Related Work
Related work spans multi-task, multi-domain, parameterized, domain-adaptation, and lifelong-learning approaches. These methods share computation, generate or specialize parameters, or preserve prior knowledge under changing tasks and domains.
- Multi-task learning shares weights across related tasks to exploit common regularities and improve representations, often specializing later layers.
- Multi-domain methods seek one compact network covering diverse domains with minimal task-specific parameters, including normalization-based and residual-adapter parameterizations.
- Parameterized multi-task learning dynamically generates task-conditioned weights, commonly using low-rank decompositions to make weight prediction feasible.
- Domain adaptation learns representations or classifiers suited to target domains, but typically prioritizes target performance without necessarily preventing forgetting.
- Lifelong-learning methods address sequential task learning and catastrophic forgetting by freezing old parameters, preserving responses, or retaining prior knowledge.
- The paper compares series and parallel residual-adapter designs as alternative modules within this broader parameter-sharing literature.
3. Method
The method constructs compact multi-domain neural-network families by sharing universal parameters while learning small domain-specific adapters. It compares adapter layouts, placement, regularization, and compression strategies to control adaptation and parameter cost.
- Residual adapters attach to standard deep networks and steer them toward different problems using a small number of adaptation parameters.The design space includes adapter construction, injection, regularization, and parameter allocation.
- 3.1.1 Series residual adapters: Series adapters combine an existing filter bank with a diagonal adapter, while setting α = 0 recovers the identity mapping and controls adaptation strength.The additive parameterization permits regularization toward zero, supporting control of generalization.
- 3.1.2 Parallel residual adapters: Parallel adapters connect alongside standard filters, producing an affine additive adjustment that can be fused for evaluation and later undone for retargeting.Parallel and series adapters have the same parameter count and both recover the original filter when α = 0.
- Adapter placement: Adapters can be placed in early, middle, late, or all network stages, with deeper adapters larger because channel counts increase with depth.The experiments also test adapting only the second convolutional layer in each residual block.
- 3.4. Cross-domain adapter compression: Low-rank adapter decomposition reduces parameters from C^2 to a fraction 2K/C, while joint cross-domain decomposition shares β across tasks and retains domain-specific γ_t factors.Joint compression is obtained by stacking domain adapters and retaining the top K singular values; K = C/2 gives a 2× reduction.
4. Experiments
Experiments evaluate residual-adapter designs for multi-domain learning and transfer learning, finding that parallel adapters, distributed across network blocks and jointly compressed, provide strong accuracy with compact parameterization.
- Experiments assess adapter topology, placement, regularization, compression, and transfer learning on Visual Decathlon and three target datasets.
- 76.9% mean accuracy and score 3096: fine-tuning is a strong Visual Decathlon baseline but requires ten times the base network’s parameter capacity.
- Adapter topology: Parallel adapters outperform series adapters by 1 point in average accuracy and 250 decathlon points while using compact domain-specific parameter sets.
- Adapter location: Adapters are crucial in all early, mid, and late macro blocks, with the last block providing the greatest benefit.
- Adapter compression: Joint SVD compression halves adapter dimensionality, preserves default parallel-module performance, and improves Aircraft, VGG-Flowers, and UCF101 accuracy by 1.8, 1.1, and 1.5 points.
- Transfer learning: Parallel and series adapters generally outperform fine-tuning with limited or moderate data, while fine-tuning reaches 51.13% versus 47.2% for adapters on full MIT Places.
- Pretraining influence: Increasing pretrained network size from 0.5× to 2.5× helps with limited data, whereas pretraining on fewer ImageNet classes steadily reduces target-task accuracy.
5. Conclusion
The paper concludes that compact universal parametric network families can share parameters efficiently across domains. Parallel residual adapters distributed throughout the network and jointly rank-compressed achieve strong benchmark and transfer-learning performance.
- Universal parametric families share parameters efficiently among multiple domains while retaining domain-specific parameters.
- The best design uses parallel residual adapters distributed throughout the network and jointly rank-compressed.
- These compact network families reduce model storage, exchange, update, and transmission costs.
- The approach significantly outperforms recent alternatives on benchmarks such as Visual Decathlon and generally surpasses traditional fine-tuning.