PaperlayerView exact revision on arXiv

Source-linked AI summary

Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks

Victor Campos, Brendan Jou, Xavier Giro-i-Nieto, Jordi Torres, Shih-Fu Chang

arXiv:1708.06834v3cs.AIcs.CV

TL;DR

Long-sequence RNNs suffer from sequential-computation and optimization challenges. Skip RNN learns when to update or copy the state, and experiments show reduced updates while matching or sometimes exceeding baseline performance. A budget loss further controls the computation–performance trade-off.

  • Problem

    Long sequences make RNN training and deployment difficult because their sequential computation produces large temporal graphs, slow inference, and long-term dependency challenges.

  • Method

    Skip RNN augments existing RNNs with a learned binary gate that updates or copies the recurrent state, using a budget loss to encourage fewer updates.

  • Results

    Skip RNNs reduced state updates while preserving, and sometimes improving, baseline performance across sequence-learning evaluations.

  • Takeaways & Limitations

    Skip RNN can reduce sequential computation and provide faster, more stable training while retaining baseline-level task performance.

Abstract

from arXiv · show

Recurrent Neural Networks (RNNs) continue to show outstanding performance in sequence modeling tasks. However, training RNNs on long sequences often face challenges like slow inference, vanishing gradients and difficulty in capturing long term dependencies. In backpropagation through time settings, these issues are tightly coupled with the large, sequential computational graph resulting from unfolding the RNN in time. We introduce the Skip RNN model which extends existing RNN models by learning to skip state updates and shortens the effective size of the computational graph. This model can also be encouraged to perform fewer state updates through a budget constraint. We evaluate the proposed model on various tasks and show how it can reduce the number of required RNN updates while preserving, and sometimes even improving, the performance of the baseline RNN models. Source code is publicly available at https://imatge-upc.github.io/skiprnn-2017-telecombcn/ .

1 INTRODUCTION

RNNs face efficiency and optimization challenges on long sequences because their inherently sequential computation creates large temporal graphs. Skip RNN addresses this by learning which state updates to omit, reducing sequential operations without extra supervision.

  • Long sequences cause RNN challenges including throughput degradation, slower training convergence, and memory leakage, even in gated architectures.
  • Heuristic sequence-shortening methods can be suboptimal because they do not learn which input samples are needed for the task.
  • Skip RNN adaptively decides whether to update or copy the recurrent state, decreasing sequential operations without additional supervision.
  • A budget penalty can encourage fewer state updates, enabling models to operate under different computation budgets.
  • The modification integrates with existing RNNs, including LSTM and GRU, and is evaluated on six sequence-learning problems.

2 RELATED WORK

Prior work uses conditional computation, attention, subsampling, or jump mechanisms to reduce sequence processing. Skip RNN is positioned as a differentiable alternative that learns unrestricted temporal sampling patterns without reinforcement learning.

  • Conditional computation methods allocate different computation paths across inputs, including temporal mechanisms that vary processing depth or pondering time.
  • Skip RNN resembles hard temporal attention by generating an input-dependent mask that selects which sequence elements receive processing.
  • Existing sequence-shortening approaches include subsampling input sequences and architectures that relax temporal dependencies to accelerate inference.
  • LSTM-Jump uses a classification layer to choose jumps but requires REINFORCE and predefined jump-related hyperparameters.
  • Unlike LSTM-Jump, Skip RNN is differentiable and is not restricted to predefined sample-selection patterns.

3 MODEL DESCRIPTION

Skip RNN augments recurrent models with a learned binary gate that either updates or copies the state, while differentiable training handles the gate during backpropagation. A budget loss encourages fewer updates, and implementation shortcuts avoid redundant computation during skipped steps.

  • MODEL DESCRIPTION: An RNN normally generates states by applying a parametric transition model sequentially to each input element.
  • MODEL DESCRIPTION: Skip RNN adds a binary gate that updates the state when u_t = 1 and copies the previous state when u_t = 0.
  • ERROR GRADIENTS: The gate uses a sigmoid-produced probability and deterministic rounding, while the straight-through estimator enables standard backpropagation through binarization.
  • MODEL DESCRIPTION: The gate's update probability accumulates increments across skipped steps and is flushed when a state update occurs.
  • MODEL DESCRIPTION: The number of skipped steps can be computed in advance, avoiding all recurrent computation while the state-update gate remains zero.
  • MODEL DESCRIPTION: Fewer updates reduce sequential operations, accelerate inference, lower energy consumption, and can improve long-term dependency modeling through shorter gradient paths.
  • LIMITING COMPUTATION: A per-sample budget penalty encourages convergence toward solutions requiring fewer state updates, with alternative loss terms possible for application-specific targets.

4 EXPERIMENTS

Experiments evaluate Skip RNN variants on sequence-learning tasks, measuring task performance alongside state updates and computational load. Across adding, MNIST, and Charades evaluations, learned skipping reduces updates while preserving or improving task behavior, with task-dependent patterns.

  • Experimental setup: Experiments report each task’s evaluation metric together with RNN state updates, used samples, and FLOPs to measure computational load.Skipping an RNN update ignores its corresponding input, so state updates and used samples are equivalent measures here.
  • 4.1 Adding task: Adding-task Skip RNN models solve the task with roughly half the updates of corresponding baselines while learning to retain marker-relevant information.They skip most samples in distractor regions and after the second marker, but have difficulty skipping many updates consecutively.
  • 4.2 MNIST classification from a sequence of pixels: MNIST Skip RNNs solve pixel-sequence classification with fewer updates, lower variation among runs, and faster training than their counterparts.The reported explanation is that shorter subsequences may simplify optimization and help capture long-term dependencies; randomly skipping updates causes a performance drop.
  • 4.2 MNIST classification from a sequence of pixels: On MNIST, Skip LSTM learns to skip nondiscriminative pixels such as padding regions, with attended samples varying by input.Figure 3 visualizes used pixels in red and skipped pixels in blue for λ = 10^-4.
  • 4.3 Temporal action localization on Charades: On Charades, Skip RNN models outperform random skipping when fewer updates are allowed and learn which frames to attend from RGB data without explicit motion information.Random and learned skipping are similar for GRU at low skipping probabilities, likely because supervision and consecutive-frame inputs and outputs are strongly correlated.
  • 4.3 Temporal action localization on Charades: Skip GRU uses fewer state updates than Skip LSTM when per-sample cost is low or absent, but both converge to comparable usage under large λ values.GRU models consistently outperform LSTM models on this task, while the update-count mismatch disappears at large penalties.

5 CONCLUSION

Skip RNN extends recurrent architectures by skipping state updates, reducing sequential operations while preserving or sometimes improving baseline performance. The approach also supports faster and more stable training for long sequences and complex models.

  • Skip RNN reduces sequential operations by learning when to update or copy the recurrent state.All parameters are trained with backpropagation.
  • Experiments with LSTMs and GRUs show that Skip RNNs can match or sometimes outperform baseline models with fewer computational requirements.
  • Skip RNNs provide faster and more stable training for long sequences and complex models.Backpropagation proceeds through fewer time steps, producing a simpler optimization task.
  • The computational savings are better suited to modern hardware than methods reducing computation within every time step.

A.1 FREQUENCY DISCRIMINATION TASK

The frequency discrimination task tests whether Skip RNNs can classify sinusoidal signals across sampling conditions while maintaining computational efficiency. With a positive update cost, models converge to similar sample usage despite doubled sampling frequency.

  • The task classifies sinusoids with periods sampled from T ∼U(5, 6) milliseconds versus T ∼{(1, 5) ∪(6, 100)} milliseconds.Each input is a scalar amplitude at each time step, and sampling periods of 0.5 and 1 milliseconds are studied.
  • RNNs use 110 units to process 100-millisecond signals, with the final state fed to a 2-way classifier.The batches are balanced between classes, giving 50% chance accuracy.
  • Models are considered successful when held-out classification accuracy exceeds 99%.
  • When λ > 0, models with the same per-sample cost use roughly the same number of input samples even when sampling frequency doubles.This indicates robustness to oversampled input signals.
  • When λ = 0, update counts differ across sampling conditions, which the authors attribute to multiple successful subsampling patterns and local minima.

A.2 SENTIMENT ANALYSIS ON IMDB

On IMDB sentiment classification, Skip RNN models achieve accuracy similar to baseline models while reducing updates. Selecting samples from longer reviews can improve accuracy at comparable token usage, though larger per-sample costs reduce accuracy.

  • The IMDB dataset contains 25,000 training and 25,000 testing reviews labeled positive or negative, averaging approximately 240 words.Words are represented with 300-dimensional embeddings before entering an RNN with 128 units.
  • Skip RNN models achieve similar accuracy rates to baseline models while reducing the number of required updates.The task makes it difficult to predict which input tokens will be discriminative.
  • A larger cost per sample results in lower accuracies, demonstrating a trade-off between accuracy and computational budget.
  • Skip LSTM with length 400 and λ = 10^-4 achieves higher accuracy than baseline LSTM with length 200 using roughly the same number of words per review.
  • Allowing Skip RNN models to select words from longer reviews can boost classification accuracy while using a comparable number of tokens per sequence.

A.3 ACTION CLASSIFICATION ON UCF-101

The UCF-101 experiment uses CNN frame features and stacked recurrent layers for action classification. Skip RNN models improve classification accuracy with few updates, and models performing fewer updates converge faster.

  • UCF-101 provides 13,320 trimmed videos across 101 action categories, represented by ResNet-50 frame-level features.The features are fed into two stacked RNN layers with 512 units each.
  • Skip RNN models improve classification accuracy over the baseline while requiring very few updates.The authors associate this with low motion between consecutive frames and high temporal redundancy in frame features.
  • Models performing fewer updates converge faster during training on UCF-101.The authors attribute this to gradients being preserved across longer spans during backpropagation through time.
  • Non-recurrent alternatives for UCF-101 include spatiotemporal-kernel CNNs and two-stream CNNs.Reported accuracies for RGB-only and RGB-plus-optical-flow systems are 0.845 and 0.934, respectively.
  • Table 6 reports accuracy, used samples, and average inference FLOPs per sequence on UCF-101 split 1.

B.1 ADDING TASK

Figure 5 shows Skip GRU sample usage on the adding task, distinguishing used samples from skipped samples.

  • Skip GRU sample usage is illustrated on the adding task with λ = 10−5.
  • Red dots indicate samples used by the network.
  • Blue dots indicate samples skipped by the network.

B.2 FREQUENCY DISCRIMINATION TASK

Figure 6 shows Skip LSTM selectively using samples for frequency discrimination, where the first samples suffice to classify sine-wave frequency without uniform-downsampling aliasing.

  • Skip LSTM sample usage is illustrated on the frequency discrimination task with λ = 10−4 and Ts = 0.5ms.
  • The network learns that using the first samples is enough to classify the frequency of the sine waves.
  • Uniform downsampling may result in aliasing in this task.
Loading 1708.06834v3…