PaperlayerView exact revision on arXiv

Source-linked AI summary

Dilated Recurrent Neural Networks

Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark Hasegawa-Johnson, Thomas S. Huang

arXiv:1710.02224v3cs.AIcs.LG

TL;DR

Long-sequence RNNs struggle with complex dependencies, unstable gradients, and sequential computation. The paper proposes DilatedRNN, which uses multi-resolution dilated recurrent skip connections and exponentially increasing dilation. Experiments and theory show improved long-term learning with fewer parameters and greater efficiency.

  • Problem

    Long-sequence RNN learning must capture complex dependencies, avoid vanishing and exploding gradients, and reduce the cost of sequential computation.

  • Method

    DilatedRNN stacks cell-independent recurrent layers with multi-resolution dilated skip connections and exponentially increasing dilation.

  • Results

    DilatedRNN improves performance across sequential learning tasks while using fewer parameters, training faster, and achieving state-of-the-art results.

  • Takeaways & Limitations

    The experiments and theoretical analysis support DilatedRNN as an efficient architecture for learning long-term dependencies across domains.

Abstract

from arXiv · show

Learning with recurrent neural networks (RNNs) on long sequences is a notoriously difficult task. There are three major challenges: 1) complex dependencies, 2) vanishing and exploding gradients, and 3) efficient parallelization. In this paper, we introduce a simple yet effective RNN connection structure, the DilatedRNN, which simultaneously tackles all of these challenges. The proposed architecture is characterized by multi-resolution dilated recurrent skip connections and can be combined flexibly with diverse RNN cells. Moreover, the DilatedRNN reduces the number of parameters needed and enhances training efficiency significantly, while matching state-of-the-art performance (even with standard RNN cells) in tasks involving very long-term dependencies. To provide a theory-based quantification of the architecture's advantages, we introduce a memory capacity measure, the mean recurrent length, which is more suitable for RNNs with long skip connections than existing measures. We rigorously prove the advantages of the DilatedRNN over other recurrent neural architectures. The code for our method is publicly available at https://github.com/code-terminator/DilatedRNN

1 Introduction

Long-sequence RNN learning is difficult because models must capture dependencies across multiple timescales, avoid gradient instability, and overcome sequential computation. The paper introduces DilatedRNN as a connection architecture intended to address these challenges efficiently.

  • Long-sequence RNN learning must handle complex dependencies while preserving short-, mid-, and long-term memory.
  • Back-propagation through time is impeded by vanishing and exploding gradients.
  • Sequential forward- and back-propagation makes RNN training time-consuming.
  • Prior approaches include specialized cells, multi-timescale designs, gradient clipping, and orthogonal or unitary weight optimization.
  • The paper introduces DilatedRNN, a cell-independent architecture using multi-resolution dilated recurrent skip connections.
  • DilatedRNN is empirically evaluated across long-term memorization, pixel-by-pixel classification, language modeling, and speaker identification tasks.

2 Dilated Recurrent Neural Networks

DilatedRNN combines dilated recurrent skip connections with exponentially increasing dilation across stacked layers. The design reduces recurrent path length and supports parallel computation, while generalized starting dilations trade some dependencies for greater efficiency.

  • 2.1 Dilated Recurrent Skip Connection: A dilated recurrent skip connection links recurrent states separated by a layer-specific skip length or dilation.The recurrent cell can be Vanilla RNN, LSTM, GRU, or another RNN cell.
  • 2.1 Dilated Recurrent Skip Connection: Unlike regular skip connections, dilated skip connections remove the direct dependency on the immediately preceding recurrent state.For dilation s(l) = 4, this removes a recurrent edge and reduces the number of parameters.
  • 2.1 Dilated Recurrent Skip Connection: Dilated skip connections enable parallel processing by splitting inputs into downsampled subsequences and processing their recurrent chains independently.For skip length 4, four subsequences can be computed in parallel, increasing parallelization by s(l) times.
  • 2.2 Exponentially Increasing Dilation: Stacked DilatedRNN layers use exponentially increasing dilations to learn temporal dependencies at different resolutions.The standard schedule is s(l) = M^(l−1) for layers l = 1, ..., L.
  • 2.2 Exponentially Increasing Dilation: Exponential dilation shortens average paths between timestamps, improving long-term dependency extraction and helping prevent vanishing and exploding gradients.
  • 2.2 Exponentially Increasing Dilation: Generalized DilatedRNNs can start with dilation M^l0, using a final 1-by-M^l0 convolution to compensate for missing shorter dependencies.This configuration can increase computational efficiency by M^l0 through downsampled subsequences and shared weights.

3 The Memory Capacity of DILATEDRNN

The paper evaluates memory capacity through mean recurrent length, which captures average shortest-path cost across time spans within a cycle. DILATEDRNN achieves logarithmic scaling and optimal memory capacity within a parameter-efficient class, while extending memory beyond dilated CNN receptive fields.

  • 3.1 Memory Capacity: Mean recurrent length evaluates average shortest-path length across time spans within a cycle, addressing cases where a single-period measurement misses difficult intermediate dependencies.The measure also takes the maximum over starting times, penalizing architectures whose short paths occur only at selected positions.
  • 3.1 Memory Capacity: Regular skip RNN mean recurrent length grows linearly with cycle size, whereas DILATEDRNN’s grows logarithmically.For DILATEDRNN, the reported expression is ¯d = (3m −1)/2m log2 m + 1/m + 1, compared with the regular skip RNN expression ¯d = (m −1)/2 + log2 m + 1/m + 1.
  • 3.2 Parameter Efficiency: DILATEDRNN has one recurrent edge per node versus two for regular skip RNNs, halving recurrent complexity through removal of the direct recurrent edge.The paper links this reduction to parameter efficiency while maintaining the stated memory-capacity advantage.
  • 3.1 Memory Capacity: Within d-layer architectures using only dilated skip connections and period m = M d−1, dilation rates {M 0, · · · , M d−1} achieve the smallest mean recurrent length.This is the theorem’s optimality result for the subset with one recurrent edge per node.
  • 3.3 Comparing with Dilated CNN: Compared with a same-depth, same-dilation dilated CNN, DILATEDRNN has the same recurrent-edge count, slightly smaller mean recurrent length, and memory beyond two cycles.The dilated CNN’s receptive field is finite; with the cited rates it covers two cycles, whereas DILATEDRNN memory can extend farther, especially with GRU and LSTM units.
  • 3.3 Comparing with Clockwork RNN: Clockwork RNN controls hidden-node update rates, while DILATEDRNN updates all nodes each step and instead controls data dependency.The paper states that Clockwork RNN memory capacity is time-dependent because shortest paths vary across output times.

4 Experiments

Experiments evaluate DILATEDRNN across long-term memorization, pixel-by-pixel classification, language modeling, and speaker identification, while also examining dilation's effects on performance and efficiency. Across these settings, the architecture improves over regular recurrent baselines, and deeper or strategically dilated models improve long-sequence learning and training efficiency.

  • Experiments cover long-term memorization, pixel-by-pixel MNIST classification, Penn Treebank character-level language modeling, and VCTK speaker identification.
  • 4.1 Copy memory problem: After about 1,000 training iterations, dilated Vanilla converges to a good optimum for copy-memory sequences with T = 500 and T = 1,000, while other methods remain near random guessing.Dilated LSTM and GRU converge more slowly, and Skip Vanilla does not improve beyond random guessing.
  • 4.2 Pixel-by-pixel MNIST: 99.2 evaluation accuracy is achieved by dilated GRU on unpermuted MNIST, while dilated Vanilla gains significantly over regular skip Vanilla.In the permuted setting, dilated Vanilla reaches 96.1 test accuracy with 44k parameters and outperforms Zoneout with a comparable parameter count.
  • 4.3 Language modeling: Dilated models outperform regular Vanilla, LSTM, and GRU on Penn Treebank without increasing model complexity, while dilated GRU reaches 1.27 BPC among similarly sized models without layer normalization.
  • 4.5 Discussion: Increasing the number of dilated layers improves noisy-MNIST accuracy and convergence rate because maximum skip and mean recurrent length grow with depth.
  • 4.5 Discussion: Dropping layers reduces training time by roughly 50% per layer, while a model starting dilation at 64 trains in 17 minutes and maintains 93.5% test accuracy.Testing performance decreases when dilation does not start at one, but the effect is marginal at s(0) = 2 and small for 4 ≤ s(0) ≤ 16.

5 Conclusion

DILATEDRNN improves long-term dependency learning across domains while training faster, using fewer parameters, and requiring less hyperparameter tuning. The paper also provides a theoretical analysis and proves optimality under a meaningful RNN architectural measure.

  • DILATEDRNN reliably improves recurrent models' ability to learn long-term dependencies across problems from different domains.
  • DILATEDRNN trains faster, requires less hyperparameter tuning, and needs fewer parameters to achieve state-of-the-art performance.
  • The paper theoretically analyzes DILATEDRNN's advantages and proves its optimality under a meaningful architectural measure of RNNs.

A Mean Recurrent Length

This analysis compares path lengths in regular skip RNNs and DILATEDRNNs. Exponentially structured dilations yield logarithmic path-length behavior, unlike the linear growth associated with regular skip connections.

  • For regular skip RNNs, the minimum path length grows linearly within a cycle as the number of layers or skip scale increases.
  • DILATEDRNN path construction uses recurrent edges across time and fixed layer-traversal edges, reducing path analysis to a minimum-edge change-making problem.
  • DILATEDRNN's exponentially increasing dilations allow shortest paths to be found greedily, analogous to converting the time span into binary representation.
  • 2m log2 m + 1 is the stated average-length expression associated with the DILATEDRNN path analysis.

B Optimality of the Proposed Skip Distribution

The optimality proof reduces skip-distribution design to minimizing average recurrent edge usage over admissible dilation denominations. Within the restricted candidate family, equal within-layer dilations and multiplicative skip lengths support the resulting optimization.

  • The proof reformulates the architecture problem as an optimal denomination problem that minimizes the average number of recurrent edges over time spans from 1 to m.
  • The unrestricted optimal denomination problem remains open mathematically, but solutions are available when candidate denominations satisfy the paper's specified constraint.
  • An architecture minimizing mean recurrent length must use the same dilation rate within each layer, because shortest paths can combine recurrent edges across layers.
  • With multiplicative dilations, minimizing mean recurrent length reduces to minimizing average recurrent edge usage, and a greedy procedure finds shortest paths.
  • For the maximum time span m, only the edge with dilation sd = m is used once.
Loading 1710.02224v3…