Source-linked AI summary
Learning multiple visual domains with residual adapters
Sylvestre-Alvise Rebuffi, Hakan Bilen, Andrea Vedaldi
TL;DR
The paper asks how to learn one visual representation that works across many different image domains rather than optimizing separate models for individual tasks. It uses residual adapters to steer a shared deep network across domains, and reports strong multi-domain performance while introducing the Visual Decathlon benchmark. The approach also supports parameter sharing and avoids forgetting in the described sequential-learning setting.
Problem
The paper addresses the need for feature extractors that work well across several different visual domains within a single model.
Method
The method uses domain-specific residual adapters within a shared parametric feature extractor, with domain-specific classifiers and on-the-fly domain conditioning.
Results
The architectures maintain or improve domain-specific performance with substantial parameter sharing, and the Visual Decathlon evaluates consistency across ten classification domains.
Takeaways & Limitations
Residual adapters provide a compact way to represent many visual domains, while the Visual Decathlon enables systematic comparison of multi-domain learning methods.
Takeaways & Limitations
Fine-tuning is often a poor strategy for learning shared representations because it can quickly forget the original tasks.
Abstract
from arXiv · showhide
There is a growing interest in learning data representations that work well for many different types of problems and data. In this paper, we look in particular at the task of learning a single visual representation that can be successfully utilized in the analysis of very different types of images, from dog breeds to stop signs and digits. Inspired by recent work on learning networks that predict the parameters of another, we develop a tunable deep network architecture that, by means of adapter residual modules, can be steered on the fly to diverse visual domains. Our method achieves a high degree of parameter sharing while maintaining or even improving the accuracy of domain-specific representations. We also introduce the Visual Decathlon Challenge, a benchmark that evaluates the ability of representations to capture simultaneously ten very different visual domains and measures their ability to recognize well uniformly.
1 Introduction
The paper addresses learning compact visual representations that work across diverse domains, using residual adapters to steer a shared deep network and introducing a ten-domain benchmark for evaluation.
- Problem: Multiple-domain learning seeks feature extractors that work well across several visually different image domains.The paper distinguishes this from generic multiple-task learning and focuses on domain variation in both style and content.
- Motivation: Compact multivalent representations can share low- and mid-level visual primitives across domains and may outperform individually trained models when sufficient synergies exist.The comparison is conditioned on using a given amount of training data.
- Method: The proposed architecture reconfigures a deep network on the fly, combining domain-agnostic shared layers with domain-specific parametric layers.Domain-specific parameters can be selected using the ground-truth domain or an auxiliary network's estimate, drawing on learning-to-learn formulations.
- Method: Residual adapters parameterize a standard residual network with less than 10% of the model parameters, enabling substantial sharing across domains.The adapters also modulate adaptation according to the size of the target dataset.
- Evaluation: The Visual Decathlon evaluates simultaneous classification across ten highly different visual domains and rewards models that exceed strong baselines consistently.The domains range from ImageNet and SVHN to action classification and describable texture recognition.
2 Related Work
Related work spans multi-task, lifelong, transfer, and domain-adaptation learning, but the paper emphasizes that its multiple-domain setting differs from these established formulations.
- Multi-task learning: Multi-task learning shares information and computation across different tasks, commonly by sharing earlier neural-network layers and training tasks jointly.Examples include language, speech, and computer-vision prediction tasks.
- Multiple-domain representations: The paper relates to work that absorbs multiple visual domains by tuning normalization parameters, while reporting superior results from residual adapters that include this approach as a special case.The cited comparison concerns the capacity needed to represent the union of visual domains.
- Lifelong learning: Lifelong-learning methods address catastrophic forgetting by freezing old-task parameters and adding new ones or by preserving the original network's responses or parameters.The paper characterizes its method as a hybrid that can retain previous tasks exactly while adding few parameters.
- Transfer learning: Transfer learning commonly initializes on ImageNet and fine-tunes on a target domain, but this typically forgets the original domain.The paper confirms this behavior in its experiments.
- Domain adaptation: Unlike standard domain adaptation, this setting changes both image style and object content, while domain adaptation focuses on target-domain performance without addressing potential forgetting.The paper also distinguishes its setup from transfer learning on these grounds.
3 Method
The method builds a compact family of domain-specific feature extractors by sharing most network parameters and adapting residual networks with small domain-specific modules. Residual adapters preserve the shared network while enabling domain modulation and reducing overfitting on small domains.
- Parametric feature extractors: The method targets shared representations across diverse visual domains using a compact parametric family of feature extractors and domain-specific classifiers.Each domain predictor combines a feature extractor φαd with a domain-specific linear classifier ψd.
- Learning to learn and filter prediction: Domain adaptation is represented by parameters selected for each domain, while the main goal is to maximize parameter sharing rather than learn independent models.The formulation can express domain parameters as columns selected by a domain indicator.
- Learning to learn and filter prediction: Filter prediction is implemented by separating domain-agnostic basis filters from domain-specific coefficients through an equivalent intermediate convolutional layer.This decomposition assigns shared parameters to F and domain-specific parameters to αd.
- Residual adapter modules: Residual adapters modify standard ResNet modules with additional small residual convolutional modules instead of directly changing the main filter coefficients.The adapters are introduced as blue blocks and use 1 × 1 filters to limit domain-specific parameters.
- Residual adapter modules: Zero adapter coefficients reduce the adapters to the identity function, which helps control overfitting when learning from small domains.The architecture also incorporates batch-normalization layers into the adapter modules.
- Residual adapter modules: Less than 10% of the model parameters are domain-specific, enabling high parameter sharing across domains.The parameter-count analysis gives approximately h^2 more domain-agnostic than domain-specific parameters, usually h^2 = 9.
- Sequential learning and avoiding forgetting: Fine-tuning only domain-specific adapter parameters can preserve the original task exactly, although fine-tuning is often poor for learning shared representations because it quickly forgets original tasks.The paper notes that sequential learning is not its main concern, though it can help bootstrap models with many domains.
4 Visual decathlon
The Visual Decathlon evaluates multiple-domain learning by requiring one model to perform well across ten heterogeneous visual classification datasets. Its single score rewards strong performance across all domains relative to domain-specific baselines.
- Benchmark goal: The Visual Decathlon is a benchmark for whether a method can learn strong performance simultaneously across ten different visual domains.It selects ten representative domains and uses an evaluation metric designed to reward performance on all tasks.
- Datasets: The challenge combines ten datasets spanning aircraft, objects, pedestrians, traffic signs, flowers, ImageNet categories, handwritten characters, and other visual domains.The datasets differ substantially in image content, class structure, resolution, and dataset size.
- Challenge and evaluation: Models must use only the provided training and validation data and are evaluated on test data with the ground-truth domain label supplied for each image.Each dataset has specified training, validation, and test subsets.
- Challenge and evaluation: The benchmark reports a single scalar score S that rewards models performing well on every task relative to a minimum baseline performance for each domain.The score is based on the average test error Ed for each domain.
- Challenge and evaluation: The score uses γd = 2 for all domains and scales a perfect result to 1,000 points per domain, or 10,000 points in total.No points are scored above each domain’s baseline error threshold Emax d.
5 Experiments
Experiments compare parameter-sharing strategies across ten visual domains, measuring accuracy, model size, forgetting, and the overall Decathlon score. Residual adapters provide strong multi-domain performance with compact models, while preserving the original domain and improving convergence relative to tested alternatives.
- Experimental setup: The experiments evaluate multiple-domain models on ten tasks using top-1 accuracy, model size relative to a ResNet, and the final Decathlon score.ImageNet primes the network in most settings, and the fully fine-tuned model defines the Decathlon-score baseline.
- Baselines: Full fine-tuning performs accurately on large and small datasets but substantially forgets ImageNet, requiring ten complete ResNet models for strong overall performance.Learning independently also requires 10× parameters, while freezing the feature extractor preserves ImageNet but generally performs poorly on target domains.
- Residual adapters: Residual adapters increase parameters by 11% per domain, yielding a 2× model without forgetting and performance that is close to or better than full fine-tuning.Higher weight decay helps smaller datasets, whereas lower decay is better for larger datasets, so decay selection affects performance across domains.
- Residual adapters: 64.7% mean accuracy from tuning only the last two convolutional layers is significantly below the adapters’ 73.9% across ten datasets.The authors attribute the gap likely to overfitting, which their technique is designed to control.
- Ablations: 70.3% accuracy without residual connections is worse than the adapters’ 73.9%, and the non-residual configuration requires notably more iterations to converge.The experiments associate residual adapter connections with better performance, overfitting control, and faster convergence.
- Overall evaluation: 2643 Decathlon points exceed all baselines using only 2× a single ResNet’s capacity, while the higher-capacity 12× model exceeds the fine-tuning baseline by 600 points.The 12× model also exceeds the reported state of the art on CIFAR100 by 3.5 points, although some domains remain affected by reduced resolution or representation choices.
6 Conclusions
The paper introduces residual adapter modules to compress many visual domains in relatively small residual networks while sharing substantial parameters. It also reports benefits for forgetting and adaptation to target domains.
- Residual adapter modules compress many visual domains in relatively small residual networks.
- The architecture shares substantial parameters between visual domains.
- The modules address forgetting and support adaptation to target domains.
A Decathlon scores
Table 1 compares multiple-domain network models using decathlon scores across multiple tasks and reports their size relative to a baseline ResNet. The fully fine-tuned model provides the baseline for computing the decathlon score.
- Decathlon scores compare different models across the multiple tasks.
- Model size is reported as the number of parameters relative to the baseline ResNet.
- The fully fine-tuned model is used as the baseline for computing the decathlon score.