PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal FiLM: Capturing Long-Range Sequence Dependencies with Feature-Wise Modulations

Sawyer Birnbaum, Volodymyr Kuleshov, Zayd Enam, Pang Wei Koh, Stefano Ermon

arXiv:1909.06628v3cs.LGstat.ML

TL;DR

Sequence models must capture long-range dependencies, but recurrent networks can be difficult to train and convolutions have finite receptive fields. TFiLM uses an RNN to modulate convolutional activations with long-range information. The paper reports improved performance across discriminative and generative tasks, including results above domain-specialized approaches.

  • Problem

    Long-range dependencies matter in sequential inputs, but RNNs can be difficult to train and convolutional models capture interactions only within finite receptive fields.

    Source: §1 · p. 1

  • Method

    TFiLM uses recurrently captured sequence information to modulate the activations of a convolutional model.

    Source: §1 · p. 1

  • Results

    TFiLM significantly improves performance across discriminative classification and complex, high-dimensional time-series super-resolution, and outperforms specialized approaches using domain-specific features.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper demonstrates TFiLM across three diverse domains, with potential applications including text-to-speech generation, sentiment analysis, and reducing the cost of genomics experiments.

    Source: §7 · p. 8

    On the audio task, performance drops when a model trained on speech is evaluated on piano music, or vice versa, highlighting the need for diverse training data.

    Source: §5.4 · p. 8

Abstract

from arXiv · show

Learning representations that accurately capture long-range dependencies in sequential inputs -- including text, audio, and genomic data -- is a key problem in deep learning. Feed-forward convolutional models capture only feature interactions within finite receptive fields while recurrent architectures can be slow and difficult to train due to vanishing gradients. Here, we propose Temporal Feature-Wise Linear Modulation (TFiLM) -- a novel architectural component inspired by adaptive batch normalization and its extensions -- that uses a recurrent neural network to alter the activations of a convolutional model. This approach expands the receptive field of convolutional sequence models with minimal computational overhead. Empirically, we find that TFiLM significantly improves the learning speed and accuracy of feed-forward neural networks on a range of generative and discriminative learning tasks, including text classification and audio super-resolution

1 Introduction

Long-range dependencies are difficult for sequence models: recurrent networks can be hard to train, while convolutions have finite receptive fields. TFiLM combines convolutional processing with recurrently captured context and improves performance across discriminative and generative sequence tasks.

1 Introduction

  • Long-range dependencies link sequence elements that may be far apart, such as early speech signals influencing words enunciated later.Capturing these dependencies is a challenge in sequential-data processing.Source: §1 · p. 1
  • RNNs are often difficult to train because of vanishing gradients, whereas convolutional models are easier to train but capture interactions only within finite receptive fields..Source: §1 · p. 1
  • TFiLM combines convolutional and recurrent approaches by using long-range information from an RNN to modulate convolutional-model activations.Its rescaling parameters are based on batch normalization, as in earlier work on image stylization [11] [21] and visual question answering [43].Source: §1 · pp. 1–2
  • The paper studies TFiLM on discriminative text classification and generative time-series super-resolution, including reconstruction of audio from 15–50% of its original time-domain samples and genomic measurements from fewer, lower-quality measurements.Time-series super-resolution reconstructs a high-resolution signal from low-resolution measurements.Source: §1 · p. 2
  • TFiLM significantly improves performance across discriminative classification and complex, high-dimensional time-series super-resolution, and outperforms specialized approaches using domain-specific features.The introduction also reports performance above strong baselines across multiple domains.Source: §1 · p. 2

2 Background

The background contrasts recurrent sequence models with convolutions and introduces batch normalization and feature-wise modulation. TFiLM adapts normalization parameters over pooled blocks using an RNN.

2 Background

  • Batch normalization rescales convolutional activations using channel statistics and applies a trainable affine transformation; FiLM makes the affine parameters functions of an auxiliary input.FiLM can let the same feed-forward network apply different styles to a target image by changing those parameters for each style input [11].Source: §2 · p. 2
  • TFiLM takes 1D convolutional activations and a block length as input, then outputs an adaptively normalized activation tensor.It reshapes activations into blocks, pools each block, uses an RNN to compute block-wise normalizers, and normalizes the blocks.Source: §2 · p. 3
  • RNNs can be difficult to train and computationally impractical on long time series, while 1D convolutions are faster and easier to train but have limited receptive fields.A convolutional output subsequence depends on only a finite input subsequence.Source: §2 · p. 3

3 Temporal Feature-Wise Linear Modulation

TFiLM combines convolutional activations with recurrently captured long-range information by generating block-wise normalization parameters from pooled activations. This design uses context from the current and past blocks while keeping the recurrent computation small.

3 Temporal Feature-Wise Linear Modulation

  • TFiLM splits convolutional activations into time blocks, pools each block, and uses an RNN to compute affine parameters for that block.In the reported experiments, the RNN is an LSTM and pooling is max pooling.Source: §3 · pp. 3–4
  • Each block's activations are normalized using its RNN-generated parameters, which depend on the current and all past blocks.This lets activations use long-range signals; for audio, super-resolution of a phoneme could depend on previous phonemes beyond the convolution's receptive field.Source: §3 · p. 4
  • TFiLM remains computationally tractable because its RNN is small, with output dimensionality O(C), and is invoked only a small number of times.For a 5-second speech recording divided into 0.1-second phoneme blocks, the example requires about 50 RNN invocations for 80,000 samples at 16 kHz.Source: §3 · p. 4

4 Time Series Super-Resolution

Time series super-resolution reconstructs high-resolution signals from low-resolution measurements, assuming the two series are naturally aligned. The paper applies this task to audio and genomic measurements and describes a convolutional architecture with TFiLM.

4 Time Series Super-Resolution

  • Time series super-resolution reconstructs a high-resolution signal from low-resolution measurements, assuming natural alignment between the source and target series.The alignment may arise from applying a low-pass filter to the high-resolution signal.Source: §4 · p. 4

4.1 A Deep Neural Network Architecture for Time Series-Super Resolution

  • The architecture uses K downsampling blocks, a bottleneck, and K upsampling blocks, with symmetric skip connections reusing features.Downsampling halves the spatial dimension and doubles feature size; upsampling reverses this, and max pooling reduces LSTM input size.Source: Figure 2 · p. 4, §4.1 · p. 5

4 Time Series Super-Resolution

  • Audio super-resolution reconstructs higher-rate audio from 15-50% of its time-domain samples, equivalently predicting high frequencies from low frequencies.The upsampling ratio is r = R2/R1, with R2 > R1.Source: §4 · p. 4
  • Genomic super-resolution reconstructs high-quality experimental measurements from noisy measurements gathered using fewer probes; the paper focuses on ChIP-seq.Source: §4 · pp. 4–5

4.1 A Deep Neural Network Architecture for Time Series-Super Resolution

  • Subpixel shuffling increases the time dimension during upscaling to avoid artifacts, and training uses a mean squared error objective on sampled patches.Source: §4.1 · p. 5

5 Experiments

Experiments evaluate TFiLM on sentiment classification and time-series super-resolution in audio and ChIP-seq, then examine ablations, cross-dataset generalization, and missing-value imputation.

5.1 Text Classification

  • On Yelp-2 and Yelp-5, TFiLM improves SmallCNN performance to at or near state of the art while training several times faster.Models outperforming TFiLM use several times as many parameters or unsupervised pre-training on external data; Table 2 notes that starred methods use such pre-training.Source: §5.1 · p. 5, Table 2 · p. 6

5.2 Audio Super-Resolution

  • TFiLM adds 1.0 dB average SNR improvement and 0.2 average LSD improvement in audio super-resolution.Human raters also ranked the method best among the upscaling techniques.Source: §5.2 · pp. 6–7, Table 3 · p. 7

5.3 Chromatin Immunoprecipitation Sequencing

  • With 1M sequencing reads, TFiLM outputs match the quality of signals derived from 10M to 20M reads and beat the specialized CNN baseline on all but one histone mark.Quality is evaluated using Pearson correlation with the high-quality ChIP-seq signal.Source: Table 4 · p. 7, §5.3 · p. 7

5.4 Additional Analyses

  • In audio super-resolution and sentiment analysis, adaptive normalizer parameters tend to cluster by gender in the audio task, suggesting the layer learns useful features.The visualizations are in the Appendix.Source: §5.4 · p. 7
  • Skip connections significantly improve reconstruction accuracy, and temporal adaptive normalization significantly improves model performance.The ablations are reported in Figure 5 and the Appendix, respectively.Source: §5.4 · pp. 7–8
  • Audio performance drops when a model trained on speech is evaluated on piano music, or vice versa, highlighting the need for diverse training data.Source: §5.4 · p. 8
  • TFiLM layers consistently improve missing-value imputation on daily grocery retail sales across the tested zero-out rates.Source: §5.4 · p. 8

6 Previous Work and Discussion

The discussion situates TFiLM among feature-wise modulation and sequence-modeling approaches, emphasizing its domain-agnostic applications and computational efficiency.

Previous Work

  • Unlike Kim et al. [30], TFiLM uses an RNN conditioned on the entire sequence to adjust batch-normalization parameters in a feed-forward CNN, significantly improving CNN performance.Source: §6 · p. 8

Time Series Modeling

  • TFiLM generalizes conditional modeling ideas used in computer vision for image super-resolution and colorization to time-series modeling.Source: §6 · p. 8

Applications to Audio and Genomics

  • The approach is domain-agnostic, scales better with data size, and outperforms recent audio super-resolution methods; existing techniques may also rely on hand-crafted features.The discussion identifies potential applications in making genomics experiments more affordable.Source: §6 · p. 8

Computational Performance

  • The model can run in real time and begin generating output without receiving the complete input sequence, unlike sequence-to-sequence architectures.Source: §6 · p. 8

7 Conclusion

The conclusion presents TFiLM as an efficient way to combine convolutional and recurrent processing of long-term sequence information, demonstrated across three domains.

Conclusion

  • TFiLM integrates convolutional and recurrent layers to incorporate long-term information efficiently, with demonstrated applications across three diverse domains.The conclusion names text-to-speech generation, sentiment analysis, and potentially lower-cost genomics experiments as applications.Source: §7 · p. 8

A Additional Sentiment Analysis Experiments

In additional sentiment experiments, TFiLM outperformed SmallCNN and trained faster than both SmallCNN and LSTM. The super-resolution model combines down/up convolutional blocks with pooling, skip connections, and subpixel shuffling; bidirectional recurrence added little, while TFiLM activations reflected speaker gender and review sentiment.

A Additional Sentiment Analysis Experiments

  • TFiLM significantly outperformed SmallCNN in each experiment, slightly exceeded LSTM, and trained over 50% faster than SmallCNN and almost twice as fast as LSTM.On the 1-million-parameter Yelp-2 experiment, it trained over twice as fast as SmallCNN and over three times as fast as LSTM.Source: A Additional Sentiment Analysis… · p. 13

B Time Series Super-Resolution Model Details

C Bidirectional RNN

  • A unidirectional RNN preserves applicability when future samples are unavailable; BiLSTM usually offered only minor gains and sometimes reduced performance, perhaps through overfitting.Source: C Bidirectional RNN · p. 14, Table 6 · p. 14

D TSNE Embeddings

  • t-SNE plots show final-layer activations separating speakers by gender and Yelp reviews by sentiment.Source: D TSNE Embeddings · p. 14

E MUSHRA Test

In a human listening study of audio super-resolution, the authors’ method ranked best among the three upscaling techniques. The test used ratings of reconstructions from four VCTK speakers.

  • The authors’ method ranked best among the three upscaling techniques in the human-rated audio super-resolution study.Source: E MUSHRA Test · p. 15
  • Ten subjects rated six versions of four VCTK recordings on a 0–100 scale, including the original, downsampled input, spline, DNN, and proposed reconstructions.Source: E MUSHRA Test · p. 15

F Additional Ablation Analysis.

Ablations identify symmetric skip connections as crucial for performance, with additive residual connections providing a smaller improvement. Generalization tests show noisy outputs under filtering shifts and speech–music domain transfer.

F Additional Ablation Analysis.

  • Symmetric skip connections were crucial for good performance, while additive residual connections provided a smaller but perceptible improvement.The ablation compared the original model with variants removing the residual connection and then the additive skip connection while preserving filter count.Source: F Additional Ablation Analysis · p. 15

G Understanding the Generalization of the Super-Resolution Model

H Missing Data Imputation

For retail-sales imputation, the convolutional architecture outperformed both baselines, and TFiLM layers added further benefit. The experiments randomly zeroed values in grocery sales time series to simulate missing data.

  • The study used grocery sales for 1,452 items over 1,024 days, randomly zeroing 10%, 20%, or 30% of values and comparing with cubic B-spline and DNN baselines.Data were split 80%/20% for training and testing; missing values arise naturally from bookkeeping errors or censoring, and robustness can improve downstream reliability.Source: H Missing Data Imputation · p. 16
  • The convolutional architecture consistently outperformed both baselines, and adding TFiLM layers consistently provided further benefit.Source: H Missing Data Imputation · p. 16, Table 10 · p. 16

Related papers

Loading 1909.06628v3…