Source-linked AI summary
Depthwise Convolution is All You Need for Learning Multiple Visual Domains
Yunhui Guo, Yandong Li, Rogerio Feris, Liqiang Wang, Tajana Rosing
TL;DR
The paper addresses how one model can handle images from multiple visual domains while limiting the cost of new domains. It uses depthwise separable convolution, assuming shared cross-channel and domain-specific spatial correlations, with softmax gating for knowledge transfer. On Visual Decathlon Challenge, it achieves the highest score with 50% of the parameters.
Problem
Multi-domain learning needs a single model that captures reusable structure across visually different domains while reducing the overhead of adding new tasks.
Method
The approach uses depthwise separable convolution to share cross-channel correlations while modeling domain-specific spatial correlations, supplemented by a softmax gating mechanism.
Results
50% of the parameters yield the highest score among current approaches on Visual Decathlon Challenge.
Takeaways & Limitations
Depthwise separable convolution provides a compact, extensible architecture for multi-domain learning with soft sharing between domains.
Takeaways & Limitations
The approach relies on the assumption that domains share cross-channel correlations but have domain-specific spatial correlations.
Abstract
from arXiv · showhide
There is a growing interest in designing models that can deal with images from different visual domains. If there exists a universal structure in different visual domains that can be captured via a common parameterization, then we can use a single model for all domains rather than one model per domain. A model aware of the relationships between different domains can also be trained to work on new domains with less resources. However, to identify the reusable structure in a model is not easy. In this paper, we propose a multi-domain learning architecture based on depthwise separable convolution. The proposed approach is based on the assumption that images from different domains share cross-channel correlations but have domain-specific spatial correlations. The proposed model is compact and has minimal overhead when being applied to new domains. Additionally, we introduce a gating mechanism to promote soft sharing between different domains. We evaluate our approach on Visual Decathlon Challenge, a benchmark for testing the ability of multi-domain models. The experiments show that our approach can achieve the highest score while only requiring 50% of the parameters compared with the state-of-the-art approaches.
Introduction
Multi-domain learning seeks a single feature extractor that captures reusable structure across visually different domains while minimizing the cost of adding tasks. The paper proposes depthwise separable convolution, assuming shared cross-channel but domain-specific spatial correlations.
- Current CNNs typically target one visual domain, requiring retraining when new tasks arrive.
- Multi-domain learning aims to build one feature extractor that captures universal structure across domains with minimal domain-specific parameters.
- The main challenges are identifying common structure among visually different domains and adding tasks without additional parameters.
- The proposed architecture uses depthwise separable convolution for multi-domain learning.
- 50% of the parameters are sufficient for the approach to outperform state-of-the-art results on Visual Decathlon Challenge.
- The method assumes different domains share cross-channel correlations but have domain-specific spatial correlations.
Related Work
Related work distinguishes multi-domain learning from multi-task and transfer learning. Prior multi-domain methods use shared networks with domain-specific adaptations, while transfer learning targets a related target domain.
- Multi-domain learning uses a single neural network to perform image classification across multiple visual domains.
- Prior approaches include instance normalization and universal parametric families with specialized models differing by few parameters.
- Multi-task learning extracts different features from one input to perform several tasks simultaneously, such as classification and edge detection.
- Transfer learning improves a target-domain model by leveraging information from a related source domain.
- Unlike transfer learning, multi-domain learning maximizes performance across multiple domains.
Preliminary
The paper formulates multi-domain learning as balancing average risk, shared parameters, and domain-specific parameters, then uses depthwise separable convolution to separate spatial and channel-wise computation. This structure supports compact sharing across domains.
- Problem Definition and Notations: The model considers multiple image domains, each with its own input space, labels, and joint data distribution.
- Problem Definition and Notations: Multi-domain architectures combine domain-specific parameters E(Di) with a sharable network portion C.
- Problem Definition and Notations: The design goals are to minimize average risk, maximize shared parameters, and minimize domain-specific parameters.
- Depthwise Separable Convolution: Depthwise separable convolution factorizes a standard 3 × 3 convolution into 3 × 3 depthwise and 1 × 1 pointwise convolutions.
- Depthwise Separable Convolution: Depthwise convolution applies one filter per input channel, while pointwise convolution linearly combines the resulting channels.
- Depthwise Separable Convolution: Depthwise convolution captures spatial correlations, whereas pointwise convolution captures channel-wise correlations.
- Depthwise Separable Convolution: The method assumes cross-channel correlations are shared across domains while spatial correlations remain domain-specific.
Proposed Approach
The proposed architecture replaces standard convolutions with depthwise separable convolutions, sharing pointwise filters while modeling domain-specific spatial filters. It adds soft sharing through a learned softmax gate and reduces parameters while retaining model capacity.
- Architecture: Replacing standard convolutions with depthwise separable convolutions yields a modified ResNet-26 with half the original parameter count.The original ResNet-26 has over 6M parameters; the modified architecture increases channel size while remaining compact.
- Cross-Domain Sharing: Pointwise filters are shared across domains, while depthwise filters are fine-tuned for each arriving domain.Domain-specific batch-normalization parameters and a new output layer are also added for each domain.
- Cross-Domain Sharing: The method assumes that domains share cross-channel correlations but have domain-specific spatial correlations.Depthwise convolution captures spatial correlations, whereas pointwise convolution captures channel-wise correlations.
- Model Efficiency: Each new domain introduces 0.3M additional parameters, equal to 10% of the modified ResNet-26.
- Soft Sharing: Soft sharing combines trained depthwise filters from multiple domains using a softmax gate while keeping convolutional filters fixed.The gate is trained separately for each target domain and controls the learned contribution of each domain's depthwise filters.
- Soft Sharing: The softmax gate can be placed in early, middle, or late network regions to compare different sharing strategies.These regions correspond to progressively higher-level feature processing in the convolutional network.
Experiment
The evaluation uses the Visual Domain Decathlon, measuring multi-domain recognition with a scalar score across 10 datasets and comparing the proposed model with several sharing and adaptation baselines.
- The Visual Domain Decathlon evaluates recognition across 10 visual domains, including ImageNet, CIFAR-100, Aircraft, SVHN, and VGG-Flowers.
- Performance is summarized by a scalar score S based on domain test errors relative to reasonable baseline errors, with γ_i set to 2.
- The baselines include independently fine-tuned networks, classifier-only adaptation, depthwise sharing, residual adapters, Deep Adaptation Networks, and PiggyBack.
- The Individual Network baseline fine-tunes a modified ResNet-26 separately for each domain, producing 10 models without cross-domain sharing.
- The experiments use PyTorch and two NVIDIA V100 GPUs, with SGD, momentum, learning-rate decay, and weight decay for training.
- Tables 2 and 3 report top-1 classification accuracy, Visual Decathlon score S, parameter counts, and soft-sharing strategy results.
Results and Analysis
The proposed depthwise-separable model achieves the strongest Visual Decathlon results with substantially fewer parameters, while analyses associate its effectiveness with shared pointwise convolutions and domain-specific depthwise processing.
- 100 points and 50% of the parameters separate the proposed approach from current state-of-the-art methods on the Visual Decathlon Challenge.
- 63.99 versus 60.32 is the proposed model’s ImageNet accuracy versus the original ResNet-26, while performance on smaller datasets is better or comparable.
- 3507 versus 2756 is the proposed score versus Individual Network, using only 20% of its parameters.
- 3507 versus 3234 is the proposed score versus Depthwise Sharing, supporting pointwise rather than depthwise convolution sharing in this setting.
- Network dissection finds that depthwise units capture parts or objects, whereas pointwise units mainly detect textures and fewer attributes.
- Without sharing, the base approach scores highest overall, although early-layer sharing slightly improves DTD and SVHN and later-layer sharing performs better than other alternatives.
Conclusion
The paper presents depthwise separable convolution for multi-domain learning under a channel-sharing and spatial-specificity assumption, and reports the highest Visual Decathlon score among current approaches.
- The approach assumes different domains share channel-wise correlations but have domain-specific spatial-wise correlations.
- The proposed method achieves the highest score among current approaches on the Visual Decathlon Challenge.
- Visualization shows that depthwise convolution captures more attributes and higher-level concepts than pointwise convolution.