PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling

Shaojie Bai, J. Zico Kolter, Vladlen Koltun

arXiv:1803.01271v2cs.LGcs.AIcs.CL

TL;DR

Sequence modeling is commonly associated with recurrent networks, while the breadth of convolutional approaches’ effectiveness remains uncertain. This paper systematically compares generic convolutional and recurrent architectures across sequence-modeling benchmarks, finding that TCNs outperform canonical recurrent models across diverse tasks and exhibit longer effective memory.

  • Problem

    Although recurrent networks are commonly treated as the default for sequence modeling, it remains unclear whether convolutional successes extend beyond specific application domains.

  • Method

    The paper systematically evaluates a simple temporal convolutional network combining causal convolutions, dilations, and residual connections against recurrent architectures across sequence-modeling tasks.

  • Results

    TCNs substantially outperform generic recurrent architectures across diverse sequence-modeling tasks and exhibit longer memory than recurrent architectures with the same capacity.

  • Takeaways & Limitations

    Convolutional networks should be regarded as a natural starting point and powerful toolkit for sequence modeling.

  • Takeaways & Limitations

    TCNs have not yet benefited from the concerted community investment in LSTM regularization and optimization schemes.

Abstract

from arXiv · show

For most deep learning practitioners, sequence modeling is synonymous with recurrent networks. Yet recent results indicate that convolutional architectures can outperform recurrent networks on tasks such as audio synthesis and machine translation. Given a new sequence modeling task or dataset, which architecture should one use? We conduct a systematic evaluation of generic convolutional and recurrent architectures for sequence modeling. The models are evaluated across a broad range of standard tasks that are commonly used to benchmark recurrent networks. Our results indicate that a simple convolutional architecture outperforms canonical recurrent networks such as LSTMs across a diverse range of tasks and datasets, while demonstrating longer effective memory. We conclude that the common association between sequence modeling and recurrent networks should be reconsidered, and convolutional networks should be regarded as a natural starting point for sequence modeling tasks. To assist related work, we have made code available at http://github.com/locuslab/TCN .

1. Introduction

The paper systematically compares a simple generic temporal convolutional network (TCN) with canonical recurrent architectures across diverse sequence-modeling tasks. TCNs outperform baseline recurrent networks, exhibit substantially longer memory, and motivate reconsidering recurrent networks as the default starting point.

  • Motivation: Sequence modeling is commonly treated as the domain of recurrent architectures, despite recent convolutional successes in audio synthesis, language modeling, and machine translation.
  • Evaluation: The study systematically evaluates convolutional and recurrent architectures across polyphonic music modeling, word- and character-level language modeling, and synthetic stress tests.
  • Architecture: A simple generic temporal convolutional network is applied across all tasks and compared with canonical recurrent architectures such as LSTMs and GRUs.
  • Results: TCNs convincingly outperform baseline recurrent architectures across a broad range of sequence-modeling tasks, including benchmarks commonly used to evaluate recurrent designs.
  • Results: Despite recurrent networks’ theoretical ability to capture infinitely long history, TCNs exhibit substantially longer memory and are more suitable when long history is required.
  • Conclusion: The results suggest reconsidering recurrent networks as the default: TCNs appear more accurate than canonical LSTMs and GRUs while also being simpler and clearer.

2. Background

Sequence modeling has longstanding convolutional and recurrent traditions, with recurrent networks especially prominent in language modeling and machine translation. Although recurrent architectures and hybrid RNN-CNN models have been extensively studied, the passages identify a lack of similarly thorough comparisons between convolutional and recurrent approaches for whole-sequence synthesis.

  • Convolutional sequence models: Convolutional networks have been applied to sequences for decades, including speech recognition and NLP tasks such as part-of-speech tagging and semantic role labelling.The cited applications span the 1980s and 1990s in speech recognition and later NLP tasks.
  • Recurrent sequence models: Recurrent networks maintain hidden activations propagated through time and became prominent in language modeling and machine translation.Their hidden state provides the central mechanism for recurrent sequence modeling described in the passage.
  • Recurrent architecture evaluations: Prior empirical work evaluated recurrent units, deep RNN constructions, and multiple LSTM variants across tasks including music, speech, handwriting, and language modeling.These studies were motivated partly by the many design choices available in recurrent architectures.
  • Hybrid architectures: Hybrid models combine recurrent and convolutional components through convolutional LSTM layers, interleaved convolutional and recurrent layers, or dilations in recurrent architectures.The examples are Convolutional LSTM, Quasi-RNN, and dilated RNN models.
  • Research gap: The authors identify no similarly thorough comparison of convolutional and recurrent approaches for sequence modeling, distinguishing whole-sequence synthesis from prior classification comparisons.Sequence modeling is described as synthesizing sequences element by element.

3. Temporal Convolutional Networks

The TCN is a simple convolutional sequence-modeling architecture built from fully convolutional and causal convolutions, with modern techniques added to support very long effective histories. It offers parallel processing and lower training-memory requirements than RNNs, but may require more evaluation-time storage.

  • Core architecture: TCN combines a 1D fully convolutional network with causal convolutions to preserve sequence length and prevent future-to-past leakage.Hidden layers remain the same length through zero padding, while each output uses only current and earlier elements from the previous layer.
  • Long memory: Dilated convolutions expand the receptive field, and exponentially increasing dilation with depth enables extremely large effective histories.For one layer, the effective history is (k −1)d; the network increases dilation as d = O(2^i) at level i.
  • Residual connections: Residual blocks let deep TCN layers learn modifications to identity mappings rather than entire transformations, benefiting very deep networks.A residual block adds the output of a transformation branch to the block input.
  • Advantages: Unlike RNNs, TCN convolutions process long input sequences in parallel during training and evaluation.The same filter is used at each position in a layer, so later timesteps do not need to wait for predecessor predictions.
  • Limitations: TCNs require more memory during evaluation because they take the raw sequence up to the effective history length, whereas RNNs retain only a hidden-state summary.The passage also identifies data storage during evaluation as a notable TCN disadvantage.

4. Sequence Modeling Tasks

This section evaluates TCNs and RNNs on commonly used sequence-modeling benchmarks, combining synthetic stress tests with real-world datasets across multiple domains. The task suite spans long-memory classification and copying problems, polyphonic music modeling, and character- and word-level language modeling.

  • Evaluation scope: The evaluation covers a comprehensive suite of synthetic stress tests and real-world datasets commonly used to benchmark recurrent sequence-modeling architectures.Table 1 evaluates TCNs and recurrent architectures on stress tests, polyphonic music, and character- and word-level language modeling.
  • Synthetic stress tests: In the adding problem, models must sum two randomly selected values in a length-n, depth-2 input sequence.The two selected positions are marked by ones in the second dimension; predicting 1 everywhere yields an MSE of about 0.1767.
  • Synthetic stress tests: Sequential MNIST presents 784×1 image sequences for digit classification, while P-MNIST randomly permutes the sequence order.These tasks test a recurrent network’s ability to retain information from the distant past, with P-MNIST providing the more challenging setting.
  • Synthetic stress tests: Copy memory requires reproducing 10 randomly chosen initial digits after a delimiter and a delay of T positions.Each input has length T + 20, and the final 10 output values should repeat the initial 10 values.
  • Real-world datasets: The real-world suite includes polyphonic music datasets JSB Chorales and Nottingham, plus PTB, Wikitext-103, LAMBADA, and text8 for language modeling.JSB Chorales contains 382 four-part Bach chorales, Nottingham contains 1,200 folk tunes, and the language datasets provide character- and/or word-level modeling settings.

5. Experiments

Experiments compare a generic TCN with canonical recurrent networks across standard sequence-modeling tasks using a consistent architecture and minimal tuning. TCNs generally outperform recurrent baselines and show longer effective memory, though optimized or specialized recurrent models can remain stronger on some datasets.

  • Experimental setup: The experiments compare the same generic TCN architecture with LSTM, GRU, and vanilla RNN baselines, varying depth and occasionally kernel size to cover sufficient context.TCNs use exponential dilation d = 2^i and Adam with learning rate 0.002 unless otherwise noted; gradient clipping improved convergence.
  • Overall results: Across the benchmark suite, generic TCNs with minimal tuning generally outperform canonical recurrent architectures.The authors note that several recurrent task results are not state of the art.
  • Synthetic tasks: On the adding problem, TCNs quickly reach virtually perfect performance with MSE near 0, while GRUs converge more slowly and LSTMs and vanilla RNNs perform significantly worse.Models had roughly 70K parameters and were tested at sequence lengths T = 200 and 600.
  • Synthetic tasks: On Sequential MNIST and P-MNIST, TCNs substantially outperform recurrent architectures in convergence and final accuracy, exceeding the 95.9% recurrent-network state-of-the-art on P-MNIST.All models had roughly 70K parameters and were trained for 10 epochs.
  • Real-world sequence modeling: On polyphonic music, TCNs outperform recurrent models with virtually no tuning and exceed some enhanced recurrent architectures, although Deep Belief Net LSTM performs better on these small datasets.On PTB, an optimized LSTM outperforms TCN, while TCN outperforms GRU and vanilla RNN.
  • Effective memory: TCNs maintain 100% copy-memory accuracy across sequence lengths, whereas same-size LSTMs and GRUs degenerate toward random guessing as T grows.LSTM accuracy falls below 20% for T < 50, and GRU accuracy falls below 20% for T < 200, indicating longer effective history for TCNs.

6. Conclusion

The evaluation finds that a simple TCN, combining dilated residual causal convolutions, outperforms generic recurrent architectures across diverse sequence-modeling tasks and exhibits longer effective memory. These results challenge the presumed dominance of recurrent networks while noting that TCNs have received less optimization and regularization investment.

  • Architecture: A simple TCN combines dilations, residual connections, and causal convolutions for autoregressive prediction.These design elements are presented as best practices underlying the evaluated architecture.
  • Empirical findings: TCN models substantially outperform generic recurrent architectures such as LSTMs and GRUs across the evaluated sequence-modeling tasks.The evaluation spans a comprehensive suite of sequence-modeling tasks.
  • Long-range memory: 100% accuracy: TCN maintains perfect copy-memory accuracy for all sequence lengths, while LSTM and GRU degrade to random guessing as length grows.This result illustrates the TCN’s longer effective memory on the copy memory task.
  • Long-range memory: The recurrent networks’ presumed “infinite memory” advantage is largely absent in practice, whereas TCNs exhibit longer memory than recurrent architectures.The paper therefore argues that recurrent preeminence in sequence modeling may be largely historical.
  • Limitations and outlook: LSTM architectures have benefited from extensive regularization and optimization advances, while TCNs have not yet received comparable community-wide investment.The authors view further architectural and algorithmic investment in TCNs as desirable and expect it to improve performance.

for Sequence Modeling Supplementary Material … A.2. Hyperparameters for LSTM/GRU

The supplementary material describes TCN hyperparameter selection, initialization, and regularization, emphasizing sufficient receptive-field coverage. It also explains how LSTM and GRU settings were selected and size-matched for comparison.

  • A.1. Hyperparameters for TCN: TCN parameters should provide a receptive field large enough to cover the task’s required context.The key choices are kernel size k and dilation d.
  • A.1. Hyperparameters for TCN: TCN hidden units were selected so the model size approximately matched the recurrent models used for comparison.For larger tasks, gradient clipping with a randomly selected threshold from [0.3, 1] helped regularize TCN and accelerate convergence.
  • A.1. Hyperparameters for TCN: A gradient clip of N/A indicates that no gradient clipping was applied.
  • A.1. Hyperparameters for TCN: TCN weights were initialized from a Gaussian distribution N(0, 0.01).The model was generally insensitive to hyperparameter changes when its effective history, or receptive field, was sufficient.
  • A.2. Hyperparameters for LSTM/GRU: LSTM settings came from hyperparameter searches over models with up to 3 layers and optimizers selected from {SGD, Adam, RMSprop, Adagrad}.For certain larger datasets, settings from prior work were adopted, including Grave et al. (2017) on Wikitext-103.
  • A.2. Hyperparameters for LSTM/GRU: GRU hyperparameters were chosen similarly to LSTM settings but typically used more hidden units to keep total network size approximately equal.

B. State-of-the-Art Results

The generic TCN and LSTM/GRU models can be outperformed by specialized architectures on some tasks, while Table 4 summarizes state-of-the-art results using the same TCN architecture across tasks.

  • Specialized architectures outperform the generic TCN and LSTM/GRU models on some tasks.
  • Table 4 summarizes the state-of-the-art results.
  • The same TCN architecture is used across all tasks.The state-of-the-art model may differ in size from the TCN.

C. Effect of Filter Size and Residual Block

Controlled ablations on copy memory, permuted MNIST, and Penn Treebank show that both filter size and residual connections affect TCN sequence modeling. Larger filters help copy memory and P-MNIST, whereas k = 3 is best for word-level language modeling, and residual blocks consistently stabilize and improve training.

  • C. Effect of Filter Size and Residual Block: Controlled experiments on copy memory, permuted MNIST, and Penn Treebank held model size, depth, and dilation constant while testing filter size and residual connections.The ablation results confirm contributions from both factors to sequence modeling.
  • Filter size k: Filter sizes ≤3 converge only to random-guessing performance on copy memory and P-MNIST, while larger filters converge faster and achieve better accuracy.The benefit of larger filters applies to both tasks.
  • Filter size k: A filter size of k = 3 performs best on word-level language modeling, likely because smaller kernels emphasize local context under fixed dilation.The passage attributes this task-specific advantage to the importance of local context in language modeling.
  • Residual block: Residual functions stabilize training, accelerate convergence, and produce better final results across copy memory, P-MNIST, and language modeling.Residual connections contribute substantially to language-modeling performance.

D. Gating Mechanisms

The section compares ReLU-TCNs with gated TCNs using GLU activations, which require approximately twice as many convolutional layers. Gating improves accuracy on some language-modeling datasets but offers no comparable benefit on polyphonic music or long-retention synthetic tasks.

  • Gated versus ReLU activations: GLU gating represents an elementwise product of two convolutional layers, with one passed through sigmoid σ(x), and uses approximately twice as many convolutional layers as ReLU-TCN.The comparison follows the GLU choice used by Dauphin et al. (2017).
  • Gated versus ReLU activations: GLU further improves TCN accuracy on certain language-modeling datasets such as PTB.The results kept the number of model parameters at about the same size.
  • Gated versus ReLU activations: Gating provides no comparable benefit on polyphonic music modeling or synthetic stress tests requiring longer information retention.The experiments compared models with approximately equal parameter counts.
Loading 1803.01271v2…