Source-linked AI summary
Depthwise Separable Convolutions for Neural Machine Translation
Lukasz Kaiser, Aidan N. Gomez, Francois Chollet
TL;DR
Convolutional sequence-to-sequence models are effective for machine translation but face computational and parameter-cost concerns. The paper introduces SliceNet, applies depthwise separable and super-separable convolutions, and reports state-of-the-art results while reducing resources and eliminating the need for filter dilation.
Problem
Convolutional sequence-to-sequence models achieve strong results but have high computational complexity and large parameter counts.
Method
SliceNet uses depthwise separable convolution layers with residual connections, removes filter dilation through larger windows, and introduces grouped super-separable convolutions.
Results
SliceNet beats ByteNet and previous state of the art while using over two times fewer non-embedding parameters and floating point operations than ByteNet.
Takeaways & Limitations
Depthwise separability supports larger convolution windows without dilation, while super-separable convolutions provide incremental performance improvements.
Takeaways & Limitations
The separable-convolution simplification assumes input channels are fairly independent while spatial locations are highly correlated.
Abstract
from arXiv · showhide
Depthwise separable convolutions reduce the number of parameters and computation used in convolutional operations while increasing representational efficiency. They have been shown to be successful in image classification models, both in obtaining better models than previously possible for a given parameter count (the Xception architecture) and considerably reducing the number of parameters required to perform at a given level (the MobileNets family of architectures). Recently, convolutional sequence-to-sequence networks have been applied to machine translation tasks with good results. In this work, we study how depthwise separable convolutions can be applied to neural machine translation. We introduce a new architecture inspired by Xception and ByteNet, called SliceNet, which enables a significant reduction of the parameter count and amount of computation needed to obtain results like ByteNet, and, with a similar parameter count, achieves new state-of-the-art results. In addition to showing that depthwise separable convolutions perform well for machine translation, we investigate the architectural changes that they enable: we observe that thanks to depthwise separability, we can increase the length of convolution windows, removing the need for filter dilation. We also introduce a new "super-separable" convolution operation that further reduces the number of parameters and computational cost for obtaining state-of-the-art results.
1 Introduction
Convolutional sequence-to-sequence models offer efficient non-local temporal referencing but raise concerns about computational complexity and parameter count. This work applies depthwise separable convolutions from vision to language tasks.
- Convolutional sequence-to-sequence models can efficiently reference non-local information across time without fully sequential RNN processing.
- Their principal concerns are computational complexity and large parameter counts.
- The paper generalizes depthwise separable convolution techniques from Xception and MobileNets to neural machine translation.
2 Our contribution
The paper introduces SliceNet, a convolutional sequence-to-sequence architecture using depthwise separable convolutions, residual connections, and super-separable convolutions. Experiments show that separability supports larger convolution windows without filter dilation while achieving state-of-the-art machine translation results.
- SliceNet is a convolutional sequence-to-sequence architecture for machine translation that achieves state-of-the-art results.
- The architecture stacks depthwise separable convolution layers with residual connections and also explores grouped and super-separable convolutions.
- A depthwise separable convolution applies a depthwise spatial convolution independently per input channel, followed by a pointwise 1x1 convolution.
- Depthwise separable convolution replaces joint spatial-and-channel feature learning with separate spatial feature learning and channel combination steps.This simplification assumes channels are fairly independent while spatial locations are highly correlated.
- Super-separable convolution splits channels into groups, applies a separable convolution to each group, and concatenates the resulting feature maps.The operation further factors the final 1x1 convolution to reduce parameter count and computational cost.
- The best models use larger filters without dilation, because depthwise separability provides computational savings and experiments found no benefit from dilation.Larger windows increase receptive fields while avoiding dilation’s unequal coverage of the input space.
3 SliceNet architecture
SliceNet uses stacked convolutional modules, separable convolution steps, positional timing signals, and attention within an autoregressive encoder–decoder architecture.
- Architecture: SliceNet follows ByteNet, WaveNet, and PixelCNN with embedded inputs and outputs, separate encoders, concatenation, and autoregressive decoding.The decoder generates each output element from encoded inputs and previously generated outputs.
- Convolutional modules: Each convolution step applies ReLU, a depthwise separable convolution, and layer normalization.The complete step is ConvStepd,s(W, x) = LN(SepConvd,s(W, ReLU(x))).
- Convolutional modules: Stacks contain four convolutional steps with residual skip-connections from the stack input to the second and fourth outputs.ConvModules are stacked sequentially, with the output of the last module feeding the next.
- Attention modules: Attention compares source and target feature vectors using scaled inner products and softmax-weighted source representations.The scaling factor is 1/√depth.
- Attention modules: Timing signals add positional information through concatenated sine and cosine functions, which are added to targets before convolution and source attention.The full mechanism adds timing to targets, performs convolutional steps, and then attends to the source.
- Autoregressive structure: The decoder combines an attention convolution module with the encoded inputs, while autoregressive generation accesses previously generated outputs.Large receptive fields over inputs and past outputs support long-term dependencies.
4 Related Work
Neural machine translation progressed from recurrent sequence-to-sequence systems toward fully convolutional architectures that avoid fixed-vector bottlenecks and improve efficiency.
- Recurrent sequence-to-sequence models: Early neural translation systems used recurrent encoder–decoder sequence-to-sequence models with LSTM cells and token-by-token processing.The encoder transforms the source sentence into a fixed-sized state vector before decoding.
- Early convolutional models: Early convolutional translation models retained an RNN for output generation and could impose fixed-size representation bottlenecks that hurt longer-sentence performance.These limitations motivated architectures without such bottlenecks.
- Fully convolutional models: Fully convolutional translation removed the bottleneck; Extended Neural GPU used recurrent gated convolutions, whereas ByteNet used nonrecursive left-padded decoder convolutions.The left-padded approach was introduced in WaveNet and improves efficiency.
- Depthwise separable convolutions: Depthwise separable convolutions previously produced strong image-classification results in Xception and compact parameter-efficient models in MobileNets.These results motivated applying the operation to one-dimensional sequence-to-sequence models.
5 Experiments
The experiments evaluate separability, dilation, window size, and super-separable convolutions in ByteNet-like translation models. Larger SliceNet models achieve state-of-the-art results while using substantially fewer parameters and operations than ByteNet.
- Convolutional separability: Depthwise separable convolutions outperform regular convolutions in ByteNet-like models while using fewer parameters and reducing training and inference cost.The experiments report higher accuracy alongside lower parameter requirements and computational expense.
- Convolutional separability: Groups of size 16 produce a performance dip relative to full depthwise separability, favoring greater channel separation in this setup.The result supports using the smallest possible groups, tending toward full depthwise separable convolutions.
- Dilation and window size: Larger convolution windows remove the need for filter dilation without sacrificing the best observed results.Depthwise separability makes the increased window size computationally tractable, and the best models use larger filters with no dilation.
- Super-separable convolutions: Super-separable convolutions provide an incremental performance improvement over depthwise separable convolutions.A dedicated comparison tests the newly introduced operation against ordinary depthwise separable convolutions.
- Large-model results: The larger SliceNet models achieve state-of-the-art results while using over two times fewer non-embedding parameters and floating point operations than ByteNet.The comparison includes previously reported results and evaluates the larger models against published systems.