PaperlayerView exact revision on arXiv

Source-linked AI summary

Xception: Deep Learning with Depthwise Separable Convolutions

François Chollet

arXiv:1610.02357v3cs.CV

TL;DR

Inception modules leave open how channel and spatial correlations should be separated and whether stronger alternatives can improve architecture design. Xception replaces them with depthwise separable convolutions and outperforms Inception V3 marginally on ImageNet and substantially on JFT.

  • Problem

    The paper asks how Inception modules differ from regular convolutions and whether unexplored intermediate formulations can guide improved architectures.

  • Method

    Xception replaces Inception modules with depthwise separable convolutions, decoupling cross-channel and spatial correlations with a parameter count similar to Inception V3.

  • Results

    On JFT, Xception achieved a 4.3% relative improvement on the FastEval14k MAP@100 metric and marginally better ImageNet results than Inception V3, with similar parameter counts.

  • Takeaways & Limitations

    Depthwise separable convolutions provided classification gains over Inception V3 without added capacity, indicating more efficient use of model parameters.

  • Takeaways & Limitations

    The paper does not establish that depthwise separable convolutions are optimal, leaving potentially advantageous intermediate formulations for future investigation.

Abstract

from arXiv · show

We present an interpretation of Inception modules in convolutional neural networks as being an intermediate step in-between regular convolution and the depthwise separable convolution operation (a depthwise convolution followed by a pointwise convolution). In this light, a depthwise separable convolution can be understood as an Inception module with a maximally large number of towers. This observation leads us to propose a novel deep convolutional neural network architecture inspired by Inception, where Inception modules have been replaced with depthwise separable convolutions. We show that this architecture, dubbed Xception, slightly outperforms Inception V3 on the ImageNet dataset (which Inception V3 was designed for), and significantly outperforms Inception V3 on a larger image classification dataset comprising 350 million images and 17,000 classes. Since the Xception architecture has the same number of parameters as Inception V3, the performance gains are not due to increased capacity but rather to a more efficient use of model parameters.

1. Introduction

The section frames Inception modules as factorizations of cross-channel and spatial correlations, forming a spectrum that culminates in depthwise separable convolutions. It motivates replacing Inception modules with depthwise separable convolutions to build a comparably parameterized architecture.

  • Inception architecture: Inception modules depart from VGG-style stacks by serving as the fundamental building blocks of Inception-style models.They are stacks of modules rather than simple convolution layers.
  • Inception architecture: Inception modules factor cross-channel and spatial correlations into separate operations to make convolution more efficient.Typical modules first use 1x1 convolutions to map inputs into smaller spaces, then process correlations in those spaces.
  • Inception architecture: A simplified Inception module can be reformulated as a large 1x1 convolution followed by spatial convolutions over nonoverlapping output-channel segments.This reformulation motivates examining how the number and size of channel segments affect the operation.
  • Depthwise separable convolutions: Depthwise separable convolution is the extreme case: spatial correlations are mapped independently for every output channel after cross-channel mapping.It consists of a depthwise spatial convolution followed by a pointwise 1x1 convolution.
  • Depthwise separable convolutions: The spectrum from regular convolution to depthwise separable convolution is parameterized by the number of independent channel-space segments used for spatial convolutions.Intermediate formulations correspond to different segment counts between the two extremes.
  • Proposed architecture: The authors propose improving Inception architectures by replacing Inception modules with stacks of depthwise separable convolutions.The proposed architecture is designed with a similar number of parameters, enabled by efficient depthwise convolution implementations in TensorFlow.

2. Prior work

Xception builds on prior work in convolutional neural networks, Inception’s branched convolutions, depthwise separable convolutions, and residual connections. Its closest foundations are depthwise separable convolutions and architectural ideas from VGG-16, Inception, and residual networks.

  • Convolutional neural networks, particularly VGG-16, provide prior architectural foundations and share some schematic similarities with the proposed architecture.
  • The Inception family demonstrated the advantages of factoring convolutions into multiple branches operating successively across channels and space.
  • Depthwise separable convolutions form the basis of the proposed architecture and extend a longer history of spatially separable convolutions.
  • 2013 work by Laurent Sifre used depthwise separable convolutions in AlexNet, yielding small accuracy gains, large convergence-speed gains, and reduced model size.
  • Residual connections, introduced by He et al., are used extensively in the proposed architecture.

3. The Xception architecture

Xception is a convolutional neural network built entirely from depthwise separable convolution layers, based on the hypothesis that cross-channel and spatial correlations can be fully decoupled. Its architecture is a linear stack with residual connections, comprising 36 convolutional layers in the feature-extraction base and followed by logistic regression for image classification.

  • Core design: Xception uses depthwise separable convolution layers throughout, hypothesizing that cross-channel and spatial correlations in feature maps can be entirely decoupled.The name Xception stands for “Extreme Inception” because this hypothesis is a stronger version of the one underlying Inception.
  • Network specification: 36 convolutional layers form Xception’s feature-extraction base, followed by a logistic regression layer for image classification.Fully connected layers may optionally be inserted before logistic regression.
  • Architecture: The Xception architecture is a linear stack of depthwise separable convolution layers with residual connections.This structure makes the architecture easy to define and modify.
  • Implementation: Xception can be defined in 30 to 40 lines of code using a high-level library, unlike the more complex Inception V2 or V3 architectures.An open-source Keras and TensorFlow implementation is provided under the MIT license.

4. Experimental evaluation · 4.1. The JFT dataset · 4.2. Optimization configuration

The evaluation compares Xception with similarly sized Inception V3 on ImageNet and the large-scale JFT classification setting, using FastEval14k to assess JFT performance. Both models use identical, Inception V3-tuned optimization settings, with separate configurations for ImageNet and JFT and Polyak averaging at inference.

  • 4. Experimental evaluation: Xception is compared with Inception V3 because the architectures have nearly the same number of parameters, limiting capacity differences as an explanation for performance gaps.The comparison covers ImageNet’s 1000-class single-label task and JFT’s 17,000-class multi-label task.
  • 4.1. The JFT dataset: JFT contains over 350 million high-resolution images annotated across 17,000 classes.FastEval14k is used as an auxiliary dataset for evaluating models trained on JFT.
  • 4.1. The JFT dataset: FastEval14k contains 14,000 densely annotated images spanning about 6,000 classes, averaging 36.5 labels per image.The dataset supports evaluation of performance after JFT training.
  • 4.1. The JFT dataset: JFT performance is measured with Mean Average Precision for top 100 predictions (MAP@100), weighted by estimated class frequency and importance in social media images.The procedure targets frequently occurring social-media labels.
  • 4.2. Optimization configuration: ImageNet and JFT use different optimization configurations.The supplied configuration details include an initial learning rate of 0.045, learning-rate decay of rate 0.94 every 2, and RMSprop.
  • 4.2. Optimization configuration: The second supplied configuration includes an initial learning rate of 0.001 and learning-rate decay of rate 0.9 every.The passage does not provide the remainder of the decay schedule.
  • 4.2. Optimization configuration: Both Xception and Inception V3 use exactly the same optimization configuration on each dataset.The configuration was tuned for Inception V3, not Xception, and may therefore be suboptimal for Xception because the networks have different training profiles.
  • 4.2. Optimization configuration: All models use Polyak averaging at inference time.This evaluation procedure is applied across both datasets.

4.3. Regularization configuration

The regularization setup differs by model and dataset: Xception uses a lower weight decay than Inception V3, dropout is used only for ImageNet, and no model includes an auxiliary loss tower.

  • Regularization configuration: 1e −5 was selected for Xception weight decay instead of Inception V3’s tuned 4e −5 rate, with both rates shared across ImageNet and JFT experiments.The Xception rate was considered suboptimal and was not extensively optimized.
  • Regularization configuration: 0.5 dropout was applied before the logistic regression layer in both models for ImageNet experiments.No dropout was used for JFT because its large dataset made overfitting unlikely within a reasonable time.
  • Regularization configuration: Neither model included Inception V3’s optional auxiliary loss tower, which provides earlier classification-loss backpropagation and additional regularization.The tower was omitted for simplicity.

4.4. Training infrastructure

The networks were implemented in TensorFlow and trained on 60 NVIDIA K80 GPUs, using different gradient-descent strategies for ImageNet and JFT. The Xception architecture comprises entry, repeated middle, and exit flows with batch normalization after convolutional layers.

  • Training infrastructure: 60 NVIDIA K80 GPUs trained all networks implemented with TensorFlow.The networks used the TensorFlow framework and were trained on 60 GPUs each.
  • Training infrastructure: Synchronous gradient descent supported ImageNet classification, whereas asynchronous gradient descent accelerated JFT training.ImageNet used data parallelism with synchronous updates; JFT used asynchronous updates to speed training.
  • Training infrastructure: 3 days each was the approximate ImageNet training time, compared with over one month each for JFT experiments.The JFT models were not trained to full convergence.
  • Xception architecture: The Xception architecture consists of entry, middle, and exit flows, with the middle flow repeated eight times.All convolutional and separable-convolutional layers are followed by batch normalization, and separable convolutions use depth multiplier 1.

4.5. Comparison with Inception V3

Xception marginally outperforms Inception V3 on ImageNet and achieves a larger improvement on JFT. The architectures are similarly sized, while Xception is marginally slower under the reported training setup.

  • Performance comparison: 4.3% relative improvement on FastEval14k MAP@100 is reported for Xception over Inception V3 on JFT.ImageNet gains are described as marginal, whereas the JFT improvement is much larger.
  • Performance comparison: Xception’s performance improvement is much larger on JFT than on ImageNet.The authors suggest Inception V3 may be over-fit to ImageNet, while neither architecture was tuned for JFT.
  • Limitations and opportunities: Better Xception hyperparameters on ImageNet, particularly optimization and regularization parameters, could yield significant additional improvement.Neither architecture was tuned for JFT, and the authors specifically identify hyperparameter search as a likely source of gains.
  • Efficiency comparison: Both architectures have approximately the same size, within 3.5%, while Xception is marginally slower.Parameter count is reported on ImageNet without fully-connected layers, and training speed uses 60 K80 GPUs with synchronous gradient descent.
  • Efficiency comparison: The authors expect engineering optimizations to depthwise convolution operations could make Xception faster than Inception V3.This expectation concerns future optimization rather than a reported measured result.

4.6. Effect of the residual connections

Residual connections are essential for the specific Xception architecture’s convergence speed and final classification performance, although this finding does not imply they are required for all depthwise-separable-convolution models.

  • Effect of the residual connections: Residual connections clearly improve convergence speed and final classification performance in the evaluated Xception architecture.The comparison used an ImageNet-modified Xception model without residual connections.
  • Effect of the residual connections: The non-residual model may have been disadvantaged by using the residual model’s optimization configuration.The authors note that better optimization configurations might produce more competitive non-residual results.
  • Effect of the residual connections: Residual connections are not required for stacks of depthwise separable convolutions, as non-residual VGG-style models achieved excellent results.These models replaced all convolution layers with depthwise separable convolutions using a depth multiplier of 1.
  • Effect of the residual connections: Non-residual VGG-style models with depthwise separable convolutions were superior to Inception V3 on JFT at equal parameter count.The models used a depth multiplier of 1 and replaced all convolution layers with depthwise separable convolutions.

4.7. Effect of an intermediate activation after pointwise convolutions

The study evaluates whether separable convolutions benefit from an intermediate non-linearity between depthwise and pointwise operations. On ImageNet, it compares no activation with ReLU and ELU, while suggesting that the effect may depend on intermediate feature-space depth.

  • Experimental comparison: The experiments compare no intermediate non-linearity against ReLU and ELU between depthwise and pointwise operations in separable convolution layers.Results are reported on ImageNet in Figure 10.
  • Interpretation: The authors note that this result contrasts with the opposite effect reported for Inception modules.They attribute the discrepancy possibly to differences in the depth of the intermediate feature spaces.
  • Interpretation: Non-linearity may help in deep intermediate feature spaces but harm shallow one-channel spaces by causing information loss.The authors present this as a possible explanation for the differing effects in Inception modules and depthwise separable convolutions.

5. Future directions

Future work centers on the discrete spectrum between regular and depthwise separable convolutions, with Inception modules occupying an intermediate point. The empirical evaluation suggests that the spectrum’s extreme formulation may offer advantages over regular Inception modules.

  • 5. Future directions: The convolutional design space includes a discrete spectrum between regular and depthwise separable convolutions, parameterized by the number of independent channel-space segments used for spatial convolutions.Inception modules occupy one point on this spectrum.
  • 5. Future directions: Inception modules represent an intermediate point within the spectrum of convolutional formulations.
  • 5. Future directions: The empirical evaluation found that the spectrum’s extreme formulation, depthwise separable convolution, may outperform a regular Inception module.

6. Conclusions

The paper frames regular convolutions and depthwise separable convolutions as extremes of a spectrum, with Inception modules between them. Based on this view, it proposes Xception, which replaces Inception modules with depthwise separable convolutions and has a similar parameter count to Inception V3.

  • 6. Conclusions: Convolutions and depthwise separable convolutions occupy opposite extremes of a discrete spectrum, while Inception modules form an intermediate point.This spectrum motivates interpreting Inception modules in relation to depthwise separable convolutions.
  • 6. Conclusions: The proposed Xception architecture replaces Inception modules with depthwise separable convolutions in neural computer vision architectures.The architecture is explicitly named Xception and is based on this replacement.
  • 6. Conclusions: Xception has a similar parameter count to Inception V3 and shows small gains compared to Inception V3.The supplied passage reports the comparison but does not specify the metric or dataset for these gains.
Loading 1610.02357v3…