PaperlayerView exact revision on arXiv

Source-linked AI summary

Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks

Rahul Dey, Fathi M. Salem

arXiv:1701.05923v1cs.NEstat.ML

TL;DR

Gated recurrent networks perform well on sequential data, but gate networks increase parameterization and computational expense. The paper evaluates three GRU variants that reduce gate parameterization across MNIST and IMDB sequence tasks. The variants generally perform comparably to the original GRU while using fewer parameters and reducing computational load.

  • Problem

    Gating networks support successful RNN learning but increase parameterization and computational expense.

  • Method

    The paper varies the structure of both GRU gates uniformly and evaluates three variants against the original GRU on MNIST and IMDB datasets.

  • Results

    The three GRU variants perform comparably to the original GRU across MNIST and IMDB experiments while using fewer parameters.

  • Takeaways & Limitations

    Reduced-gate GRU variants can retain comparable performance while offering computational savings in targeted applications or resource settings.

Abstract

from arXiv · show

The paper evaluates three variants of the Gated Recurrent Unit (GRU) in recurrent neural networks (RNN) by reducing parameters in the update and reset gates. We evaluate the three variant GRU models on MNIST and IMDB datasets and show that these GRU-RNN variant models perform as well as the original GRU RNN model while reducing the computational expense.

I. INTRODUCTION

Gated RNNs perform well on sequential and temporal tasks because gates regulate how current inputs and previous memory update the current state. This gating improves learning but increases parameterization and computational expense, motivating reduced-gate GRU variants evaluated across MNIST and IMDB sequences.

  • Gated RNNs have been applied successfully to speech recognition, natural language processing, machine translation, and other sequential or temporal tasks.
  • Gating signals control how present inputs and previous memory update the current activation and produce the current state.
  • Gate networks add parameters and computational expense compared with a simple RNN, with LSTM using three gates and GRU using two.
  • The paper explores three GRU variants with reduced gate parameterization and compares them with the original GRU on MNIST and IMDB sequences.

II. BACKGROUND: RNN, LSTM AND GRU

Simple RNNs use recurrent hidden states to model sequential relationships but struggle with long-term dependencies because gradients can vanish or explode. LSTM and GRU architectures were proposed to address these gradient problems.

  • RNNs are suited to capturing relationships among sequential data through a recurrent hidden state.
  • Simple RNNs have difficulty capturing long-term dependencies because stochastic gradients tend to vanish or explode over long sequences.
  • LSTM and GRU units were proposed to address the vanishing- and exploding-gradient problems in simple RNNs.

A. Long Short-Term Memory (LSTM) RNN

LSTM combines a candidate memory state with the previous memory through gated, element-wise interpolation. Its input, forget, and output gates use logistic nonlinearities and substantially increase parameter count.

  • LSTM forms the current memory state by combining the previous memory state and a candidate state through element-wise weighted summation.
  • The LSTM weighted sum uses Hadamard multiplication with input, forget, and output gating signals.
  • LSTM gate signals are replicas of the basic recurrent equation with logistic activation, constraining gate values between 0 and 1.
  • 4×(n^2 + nm + n) parameters are used by the LSTM RNN when the cell state is n-dimensional and the input is m-dimensional.

B. Gated Recurrent Unit (GRU) RNN

GRU reduces LSTM’s three gates to update and reset gates, lowering parameterization while retaining gated recurrent computation. The paper evaluates three further gate-structure variants applied uniformly to both gates.

  • GRU uses two gates—an update gate and a reset gate—instead of the three gating signals used by LSTM.
  • The GRU architecture has 3×(n^2 + nm + n) parameters, representing a threefold increase over a simple RNN.
  • Prior studies reported that GRU performance is comparable to or can outperform LSTM in most cases, while the one-gate MGU was also reported comparable.
  • The paper retains the GRU architecture but varies the structure of both gating signals identically across three new variants.

III. THE VARIANT GRU ARCHITECTURES

The paper introduces three GRU gate variants that reduce gate parameterization by restricting each gate’s inputs, then evaluates them against the base GRU model on MNIST and IMDB.

  • GRU1 computes each gate from only the previous hidden state and bias, reducing parameters by 2× nm versus GRU.
  • GRU2 computes each gate from only the previous hidden state, reducing parameters by 2× (nm+n) versus GRU.
  • GRU3 computes each gate from only the bias, reducing parameters by 2× (nm+n2) versus GRU.
  • The study compares the base GRU, labeled GRU0, with GRU1, GRU2, and GRU3 on sequences derived from MNIST and on IMDB movie reviews.
  • The variants use a single recurrent layer with ReLU activation and dataset-specific output layers for MNIST and IMDB classification.MNIST uses a softmax output, while IMDB uses a traditional logistic activation layer.

A. Application to MNIST Dataset – pixel-wise sequences

The MNIST pixel-wise experiments compare the original GRU0 with GRU1–GRU3 across learning rates on length-784 sequences. GRU1 and GRU2 nearly match GRU0, while GRU3 trades slower convergence for substantially fewer parameters.

  • Experimental setup: MNIST pixel-wise inputs are length-784 sequences formed by scanning each 28×28 image one pixel at a time.The evaluation uses 60000 training images and 10000 test images.
  • Results: GRU1 and GRU2 perform almost as well as GRU0 on MNIST pixel-wise sequence inputs.The comparison is reported from Table II and Figures 1 and 2.
  • Results: GRU3 performs worse at the initial constant learning rate, but reducing the rate to 0.0001 and below raises its test accuracy to 59.6% after 100 epochs.Its accuracy profile still has a positive slope, indicating further improvement after more epochs.
  • Trade-off: GRU3 uses about 33% of GRU0’s adaptively computed parameters, creating a trade-off between parameter reduction and accuracy performance.Within 100 epochs, GRU3 never reaches saturation, so the comparison remains incomplete.

B. Application to MNIST Dataset – row-wise sequences

The MNIST row-wise experiments test all four GRU architectures on short vector-valued sequences across several learning rates. All variants show comparable accuracy overall, although GRU3 lags at 1e-4 while retaining a much smaller parameter count.

  • Experimental setup: Row-wise MNIST inputs are sequences of length 28 whose elements are 28-dimensional image rows.The experiments use several constant base learning rates and summarize results over 50 epochs.
  • Results: GRU0, GRU1, GRU2, and GRU3 exhibit comparable accuracy performance across three constant base learning rates.The comparison uses the training and testing accuracy profiles in Figures 4–6 and Table III.
  • Results: GRU3 shows lower performance at a base learning rate of 1e-4 after 50 epochs, but its accuracy profile has not yet levelled off.The authors state that more epochs are likely to bring it to comparable levels.
  • Trade-off: GRU3 achieves comparable performance with roughly one third of the adaptively computed parameters.The reduced parameter count may favor it when computational expense or available resources matters.

C. Application to the IMDB Dataset– natural sequence

The IMDB experiments evaluate all four GRU architectures on binary movie-review sentiment classification at two learning rates. All three reduced variants perform comparably to the original GRU while using fewer parameters, with greater computational savings for vector inputs.

  • Experimental setup: The IMDB dataset contains 25000 training and 25000 test reviews, represented by up to 80 words from a 20000-word vocabulary.The variants are trained for 100 epochs at learning rates 1e-3 and 1e-4.
  • Results: All three GRU variants perform comparably to the original GRU while using fewer parameters.This is the paper’s principal reported outcome on the IMDB experiments.
  • Results: GRU3 has a learning pace similar to the other variants at the constant base learning rate of 1e-4.The result is reported alongside the cross-variant performance comparison.
  • Computational trade-off: All variant GRUs achieve greater computational-load savings when the input is represented as a large 128-dimensional vector.The experiments use 128-dimensional GRU variants with batch size 32.

V. CONCLUSION

The three reduced-parameter GRU variants achieved comparable accuracy across MNIST and IMDB sequence evaluations, while reducing gate redundancy and computational expense. Broader empirical evaluation remains necessary, especially across diverse datasets and training settings.

  • GRU1 and GRU2 had indistinguishable performance from GRU0, while GRU3 frequently lagged, especially on relatively long sequences.GRU3 may also require more execution time to achieve comparable performance.
  • The variants reduce redundancy because the recurrent state appears to provide the main driving signal for the gates.The authors suggest that bias-only gate signals can succeed because adaptive updates carry information about the network state.
  • The study comparatively evaluates GRU1, GRU2, and GRU3 as lower-parameter, lower-computational-expense alternatives to the original GRU RNN.
  • The three variants require further comparative evaluation on diverse datasets to establish broader empirical performance evidence.
Loading 1701.05923v1…