PaperlayerView exact revision on arXiv

Source-linked AI summary

Trellis Networks for Sequence Modeling

Shaojie Bai, J. Zico Kolter, Vladlen Koltun

arXiv:1810.06682v2cs.LGcs.AIcs.CLstat.ML

TL;DR

Sequence modeling needs architectures that handle diverse benchmarks while connecting recurrent and convolutional design principles. TrellisNet addresses this with shared-depth temporal convolutions and input injection, while generalizing truncated recurrent networks. It outperforms prior state-of-the-art methods across language-modeling and long-term-memory evaluations, although its performance and computational efficiency were not thoroughly optimized.

  • Problem

    Sequence modeling spans recurrent, temporal-convolutional, and self-attention architectures, leaving a need to improve benchmark performance while clarifying relationships between recurrent and convolutional models.

  • Method

    TrellisNet is a temporal convolutional architecture with weights tied across depth and direct input injection, and it generalizes truncated recurrent networks through broader weight matrices.

  • Results

    TrellisNet outperforms prior state-of-the-art models across the evaluated language-modeling and long-term-memory tasks, including 7.6% and 11.5% perplexity improvements on WikiText-103 comparisons.

  • Takeaways & Limitations

    TrellisNet provides a structural bridge between convolutional and recurrent models and supports incorporating techniques from either family into a high-performing sequence architecture.

  • Takeaways & Limitations

    The authors did not thoroughly optimize TrellisNet performance, and further activation search, hyperparameter tuning, and acceleration work may improve accuracy or efficiency.

Abstract

from arXiv · show

We present trellis networks, a new architecture for sequence modeling. On the one hand, a trellis network is a temporal convolutional network with special structure, characterized by weight tying across depth and direct injection of the input into deep layers. On the other hand, we show that truncated recurrent networks are equivalent to trellis networks with special sparsity structure in their weight matrices. Thus trellis networks with general weight matrices generalize truncated recurrent networks. We leverage these connections to design high-performing trellis networks that absorb structural and algorithmic elements from both recurrent and convolutional models. Experiments demonstrate that trellis networks outperform the current state of the art methods on a variety of challenging benchmarks, including word-level language modeling and character-level language modeling tasks, and stress tests designed to evaluate long-term memory retention. The code is available at https://github.com/locuslab/trellisnet .

1 INTRODUCTION

The paper introduces TrellisNet to improve sequence-modeling performance while clarifying the relationship between recurrent and convolutional architectures. It evaluates the architecture across language-modeling benchmarks and long-term-memory stress tests.

  • TrellisNet is introduced as a new sequence-modeling architecture targeting both benchmark performance and understanding of recurrent–convolutional relationships.
  • Its temporal-convolutional structure ties weights across layers and injects the input directly into every layer.
  • Trellis networks generalize truncated recurrent networks, forming a bridge that supports techniques from both recurrent and convolutional models.
  • The evaluation covers word-level and character-level language modeling, including PTB and WT103, plus stress tests for long-term memory retention.
  • 7.6% lower perplexity than Relational Memory Core and 11.5% lower perplexity than Merity et al. are reported on word-level WikiText-103.

2 BACKGROUND

Sequence modeling has been dominated by recurrent architectures, while convolutional approaches and hybrid designs provide complementary ways to process temporal data. Prior work combines these families through convolutional-recurrent units, interleaved layers, and transferred techniques such as dilation.

  • Recurrent networks, especially LSTMs and GRUs, are widely used for temporal sequence modeling and have produced major results in language and other domains.
  • Convolutional networks are also established tools for sequence processing, motivating comparisons between the two architectural families.
  • Existing hybrids include convolutional LSTMs, quasi-recurrent networks, and recurrent models using convolutional techniques such as dilation.

3 SEQUENCE MODELING AND TRELLIS NETWORKS

The paper defines causal sequence modeling and introduces TrellisNet as a repeatedly applied transformation across time and depth. The architecture uses shared operations and direct access to the input while retaining convolutional sequence-processing structure.

  • A sequence model maps an input sequence x1:T to an output sequence y1:T.
  • Causality requires each output yt to depend only on x1:t, preventing future-information leakage in autoregressive modeling.
  • TrellisNet applies the same production procedure across all time steps and layers, using shared weights in both dimensions.
  • The network represents inputs xt in R^p, hidden units z^(i)_t in R^q, and temporal transformations through one-dimensional convolution.
  • At each higher layer and time step, the basic trellis transformation combines hidden representations with injected input vectors before applying its learned transformation.
  • Figure 1 presents this interlayer transformation both for adjacent time steps and layers and for a longer sequence spanning steps 1 through 8.

2. A pre-activation output ˆz(i+1)

The pre-activation output is produced by learned linear transformations, then passed through a nonlinear activation to generate the next hidden representation.

  • W1 and W2 are learned matrices mapping the combined representation into the pre-activation output ẑ^(i+1)_(t+1).
  • The linear transformations may include additive biases, which are omitted for clarity.
  • A nonlinear activation f maps the pre-activation output and hidden state to produce z^(i+1)_(t+1).

3. The output z(i+1)

Each TrellisNet layer applies a shared one-dimensional convolution over hidden units, combines transformed inputs with prior-layer hidden states, and then applies a nonlinearity. Tiling this procedure across time and depth yields a feed-forward network with tied weights and progressively larger receptive fields.

  • A TrellisNet layer combines transformed inputs at adjacent time steps with transformed hidden units from the previous layer.The input transformation can be precomputed once and reused across all layers.
  • The same production procedure and weights are applied across every time step and network layer.This produces a regular trellis pattern through shared transformations in both temporal and depth dimensions.
  • The hidden-state operation is a one-dimensional convolution over the previous layer's sequence, followed by activation function f.The convolution uses a kernel weight matrix W ∈ R^r×q.
  • The resulting network is feed-forward, with deeper elements having progressively larger receptive fields.Unlike typical temporal convolutional networks, its filter matrix is shared across depth and transformed inputs are injected into every hidden layer.
  • The activation function f may be any nonlinearity that processes the pre-activation output and previous-layer output.The paper later introduces an activation based on the LSTM cell.

4 TRELLISNET, TCN, AND RNN

TrellisNet connects temporal convolutional and recurrent architectures: it is a structured TCN, while any finite-horizon truncated RNN can be represented by a TrellisNet with sparse kernels. This equivalence supports importing techniques from both model families and motivates denser TrellisNets as generalizations of truncated RNNs.

  • TrellisNet and TCNs: TrellisNet is a temporal convolutional network with causal convolutions and deep stacking that increases effective history length.It can incorporate larger kernels, dilations, and other TCN design elements.
  • TrellisNet and TCNs: Unlike general TCNs, TrellisNet ties weights across layers and injects the linearly transformed input sequence into every layer.The paper describes weight tying as a regularizer that can stabilize training, support generalization, and reduce model size.
  • TrellisNet and RNNs: An M-truncated RNN processes only the subsequence x_{t−M+1:t}, and any such finite-horizon RNN can be represented by a TrellisNet with sparse kernel structure.The construction uses depth M + L − 1 and layer width Ld for an L-layer RNN with hidden dimensionality d.
  • TrellisNet and RNNs: The constructed TrellisNet exactly reproduces the truncated RNN output at its final layer.The proof identifies the last d channels of the final TrellisNet representation with the RNN output.
  • TrellisNet and RNNs: The RNN-equivalent kernels are sparse mixed group convolutions, where grouped channels encode recurrent layers and connect across adjacent time positions.The construction can generalize to RNN layers with different widths.
  • TrellisNet and RNNs: Dense TrellisNet kernels may express a broader class of transformations than the sparse kernels required to represent truncated RNNs.This motivates viewing general TrellisNets as generalizations of truncated recurrent networks.
  • Shared techniques: The recurrent-convolutional connections let TrellisNets incorporate recurrent techniques such as structured nonlinearities and variational dropout alongside convolutional techniques.The paper presents this integration as a way to combine architectural and algorithmic motifs from both families.

5 EXPERIMENTS

The experiments evaluate TrellisNet across language-modeling and long-range-dependency benchmarks, using an LSTM-based gated activation and architectural techniques adapted from recurrent and convolutional models. TrellisNet achieves state-of-the-art results on PTB, WT103, and character-level PTB, while also targeting long-range memory tasks.

  • 5.1 A TRELLISNET WITH GATED ACTIVATION: TrellisNet uses a gated activation based on the LSTM cell, motivated by incorporating an effective recurrent cell into the architecture.The activation computes three information-controlling gates and updates a cell state through gated transformations.
  • 5.2 RESULTS: The experiments cover word-level PTB and WT103, character-level PTB, and sequential MNIST, permuted MNIST, and CIFAR-10 long-range modeling tasks.These benchmarks differ substantially in scale and challenge sequence models through overfitting risk, vocabulary size, and long-term dependencies.
  • Word-level language modeling: TrellisNet sets a new state of the art on word-level PTB, outperforming previously published results by more than one unit of perplexity.This holds both with and without mixture of softmaxes.
  • Word-level language modeling: TrellisNet achieves perplexity 29.19 on WT103, about 7.6% better than the contemporaneous Relational Memory Core.It reaches this accuracy in 25 epochs versus 90 for the Relational Memory Core.
  • Character-level language modeling: TrellisNet sets a new state of the art on character-level PTB with 1.158 bpc, outperforming Merity et al. (2018a).The character-level setting uses a deeper network, weight normalization, and deep supervision.
  • Long-range modeling: For long-range modeling, TrellisNet processes images as long pixel sequences and uses dilated convolutions in intermediate layers to cover larger context.The evaluated tasks are sequential MNIST, permuted MNIST, and sequential CIFAR-10, using a model with 8M parameters.

6 DISCUSSION

Trellis networks bridge convolutional and recurrent sequence models, enabling techniques from both families while achieving strong benchmark performance. The paper identifies optimization, tuning, acceleration, and broader architectural connections as future work.

  • Trellis networks form a structural bridge between convolutional and recurrent models, enabling direct assimilation of techniques from either family.The authors present this connection as supporting both empirical performance and a more unified understanding of sequence modeling.
  • The authors report new state-of-the-art results on highly competitive language-modeling benchmarks.
  • Thorough performance optimization remains incomplete, including architecture search for the gated activation and principled hyperparameter tuning.The authors also identify acceleration of training and inference as future work.
  • Future work includes connecting trellis networks with self-attention architectures and applying them to industrial-scale challenges such as machine translation.

A EXPRESSING AN LSTM AS A TRELLISNET

This appendix traces how an LSTM is represented as a TrellisNet with an LSTM nonlinearity. The construction maintains gated outputs and cell-state channels within the trellis representation.

  • The appendix applies Theorem 1 to transform an LSTM into a TrellisNet and revisits the construction through atomic and sequence views.
  • An LSTM cell computes its recurrent update at each time step before the corresponding TrellisNet construction is specified.
  • Recovering an LSTM requires mixed group convolution to produce 3q gated-output channels and maintain a separate group of cell-state channels.The gated outputs include forget, input, and candidate-style channels, while cell states are retained across layers.
  • The TrellisNet hidden units split into channels updated by gated activations, analogous to LSTM cell states, and channels processed by parameterized convolutions, analogous to hidden states.
  • Figure 5 illustrates a 2-layer LSTM expressed as a trellis network using mixed group convolutions over four feature-channel groups.

B OPTIMIZING AND REGULARIZING TRELLISNET WITH RNN AND TCN METHODS

The paper transfers recurrent and convolutional techniques into TrellisNets through their structural equivalence. These techniques include history repackaging, gated activations, dropout, dense kernels, parallelism, deep supervision, dilation, and weight normalization.

  • History repackaging: History repackaging transfers to TrellisNets as padding with channels from the previous sequence’s final-layer last step.This allows compressed history to pass between subsequences instead of storing the entire effective history.
  • Gated activations: Structured recurrent gates can be translated into TrellisNet gated activations, including LSTM, GRU, and architecture-searched activations.
  • RNN variational dropout: The proposed TrellisNet dropout applies one mask to post-activation outputs across every time step and depth dimension.The authors report that this scheme performed significantly better empirically than alternatives such as dropping channels entirely.
  • Dense convolutional kernel: A dense convolutional kernel removes recurrent sparse connections and computes directly over hidden units like a temporal ConvNet.
  • Deep supervision: Deep supervision applies losses at intermediate layers, where predictions use shorter histories, with λ controlling auxiliary-loss weight.The method transfers an approach from convolutional vision models and is not directly applicable to RNNs.
  • TCN methods: Larger kernels, dilations, and weight normalization transfer directly or immediately from temporal convolutional networks to TrellisNets.Changing kernel size or dilation may require modifying the activation function, while weight normalization can regularize filters and improve convergence.
  • Parallelism: TrellisNet’s convolutional structure enables parallel processing, while history padding or discarding early loss can address insufficient initial context.

C BENCHMARK TASKS

The benchmark suite spans word- and character-level language modeling and long-range image-sequence classification. The tasks vary substantially in scale, vocabulary, sequence length, and input complexity.

  • Word-level language modeling on PTB: Word-level PTB contains 888K training words, 70K validation words, and 79K test words, with sentence boundaries marked by <eos>.
  • Word-level language modeling on WikiText-103: WT103 is 110 times larger than PTB, with 103M training words and a vocabulary of about 268K.Unlike PTB, WT103 retains case, punctuation, and numbers from the raw text.
  • Character-level language modeling on PTB: Character-level PTB contains 5M training characters, 396K validation characters, and 446K test characters over an alphabet of 50.Character-level modeling uses much longer sequential tokenizations than word-level modeling.
  • Sequential and permuted MNIST classification: Sequential MNIST presents each 28 × 28 image as a flattened 784 × 1 sequence, requiring long-term memory of the pixel sequence.Permuted MNIST changes the pixel order while retaining the long-sequence classification setting.
  • Sequential CIFAR-10 classification: Sequential CIFAR-10 presents flattened 32 × 32 images one step at a time and is more challenging than sequential MNIST because of richer image structure, variation, and three input channels.

D HYPERPARAMETERS AND ABLATION STUDY

The experiments use TrellisNet configurations informed by recurrent and convolutional methods, with explicit learning-rate, auxiliary-loss, and dropout settings. An ablation study on word-level PTB evaluates the influence of individual ingredients while controlling model size and other hyperparameters.

  • Hyperparameters: TrellisNet hyperparameters were largely based on prior recurrent and convolutional work, with minor adjustments.The paper cites prior settings for embedding, dropout, optimizer, and regularization choices.
  • Training settings: Learning rates decay after validation error plateaus or according to a fixed schedule, such as after 100 epochs.
  • Training settings: Auxiliary loss is inserted after every fixed number of layers, with this frequency recorded as an experiment hyperparameter.
  • Training settings: Variational dropout translated from RNNs is applied at all hidden TrellisNet layers.
  • Ablation study: The ablation study uses a 24M-parameter word-level PTB TrellisNet and changes one factor at a time while holding other settings fixed.For the Dense Kernel condition, hidden units are adjusted to preserve model size.
Loading 1810.06682v2…