PaperlayerView exact revision on arXiv

Source-linked AI summary

On Multiplicative Integration with Recurrent Neural Networks

Yuhuai Wu, Saizheng Zhang, Ying Zhang, Yoshua Bengio, Ruslan Salakhutdinov

arXiv:1606.06630v2cs.LG

TL;DR

RNNs commonly rely on additive information integration, and the paper asks whether a simple structural alternative can improve learning without substantially increasing parameters. It introduces Multiplicative Integration, evaluates it across recurrent models and tasks, and reports state-of-the-art performance across four tasks or 11 datasets.

  • Problem

    RNN building blocks commonly integrate information sources additively, motivating a structural alternative that can improve recurrent models without substantially increasing parameter counts.

  • Method

    Multiplicative Integration replaces additive fusion with a Hadamard product, adds negligible bias parameters, and can be directly embedded in vanilla RNNs, LSTMs, and GRUs.

  • Results

    MI achieves state-of-the-art performance on four different tasks or 11 datasets of varying sizes and scales.

  • Takeaways & Limitations

    MI is presented as a simple, broadly applicable building block that can become a default component for training various RNN models.

  • Takeaways & Limitations

    One reported comparison is not directly comparable because another result used much larger models of approximately 16M parameters.

Abstract

from arXiv · show

We introduce a general and simple structural design called Multiplicative Integration (MI) to improve recurrent neural networks (RNNs). MI changes the way in which information from difference sources flows and is integrated in the computational building block of an RNN, while introducing almost no extra parameters. The new structure can be easily embedded into many popular RNN models, including LSTMs and GRUs. We empirically analyze its learning behaviour and conduct evaluations on several tasks using different RNN models. Our experimental results demonstrate that Multiplicative Integration can provide a substantial performance boost over many of the existing RNN models.

1 Introduction

The paper proposes Multiplicative Integration (MI), which replaces additive fusion of RNN information flows with a Hadamard product while adding almost no parameters. MI changes gradient behavior and can be embedded across popular RNN architectures and sequence-learning tasks.

  • RNN building blocks commonly integrate state vectors from different sources by summing transformed inputs before applying a nonlinearity.This additive structure appears in vanilla-RNN hidden states and LSTM and GRU gate or cell computations.
  • MI replaces the sum of Wx and Uz with their Hadamard product to change how information flows are integrated.The proposed modification preserves the surrounding computational structure while changing the fusion operation.
  • MI changes RNNs from first-order to second-order computations without introducing extra parameters.The product creates a gating structure in which Wx dynamically rescales Uz, unlike additive integration.
  • MI is evaluated across character-level language modeling, speech recognition, Skip-Thought representation learning, and reading comprehension.The paper compares MI-equipped vanilla RNNs, LSTMs, and GRUs with additive models and existing state-of-the-art systems.
  • The paper reports that MI improves gradient properties, generalization, and optimization while requiring almost no extra parameters.The conference metadata identifies the work as appearing at NIPS 2016.

2 Structure Description and Analysis

MI generalizes multiplicative fusion with bias terms and can replace additive blocks in recurrent architectures. Its gating effect lets current inputs directly influence gradient propagation, while the formulation preserves a near-equal parameter count and can recover the additive block as a special case.

  • MI combines transformed information flows with a Hadamard product and can add bias vectors to each flow for a more general formulation.The formulation includes first-order terms corresponding to additive integration.
  • The general MI formulation uses an additional bias vector to gate the Wx ⊙ Uz interaction, increasing flexibility.When α = 0, MI degenerates to the vanilla additive building block.
  • MI has about the same number of parameters as additive integration because its additional bias vectors are negligible relative to the total model.The block can directly replace vanilla building blocks in LSTMs and GRUs and can be extended pairwise to k information flows.
  • In MI-RNNs, the current input Wx directly gates the hidden-to-hidden matrix U in the gradient computation.This differs from vanilla RNNs, where gradient flow depends heavily on U and W and the current input have a more limited role.
  • The gating effect is reported to alleviate vanishing and exploding gradients by dynamically reconciling U, with benefits persisting in MI-enhanced LSTMs and GRUs.For tanh, keeping activation derivatives near 1 can help gradients propagate, whereas saturated activations impede propagation.

3 Experiments

Experiments analyze MI-RNNs through gradient flow, activation saturation, scale robustness, and language-modeling performance. Across these tests, MI improves gradient propagation and performance while remaining robust to initialization scale.

  • Gradient properties: MI-RNN gradients increase rapidly, whereas lin-RNN gradients remain near 10^-4 with negligible changes over time.The experiment attributes the difference to the diag(Wx_k) gating term alleviating gradient vanishing.
  • Gradient properties: Non-saturated MI-RNN activations support gradient propagation, while saturated vanilla-RNN activations produce near-zero nonlinear derivatives.For tanh, non-saturated activations imply diag(φ′) ≈1, whereas saturation implies diag(φ′) ≈0.
  • Scaling problem: Multiplication preserves dependence on both Wx_k and Uh_k−1 when their magnitudes differ, unlike addition, where the smaller term can become negligible.The reported experiments found Wx_k much smaller than Uh_k−1, with one-hot inputs contributing to this scale difference.
  • Scaling problem: MI-RNN is more robust than vanilla-RNN to different W initialization scales, while vanilla-RNN performance improves as W’s scale increases.The tested W scales were r_W ∈ {0.02, 0.1, 0.3, 0.6}, with U fixed to uniform[−0.02, 0.02].
  • Language modeling: MI-RNN-simple and MI-RNN-general outperform vanilla-RNN on validation curves, and MI-RNN-general converges faster.MI-RNN-general achieves a test BPC of 1.39 in the reported comparison.

3.2 Character Level Language Modeling

Multiplicative Integration improves character-level language modeling across Penn-Treebank, text8, and Hutter Challenge Wikipedia, while also improving WSJ speech recognition and Skip-Thought evaluations. These gains include faster training, stronger performance, and state-of-the-art results under comparable or smaller model sizes.

  • Character-level language modeling: On text8, MI-RNN performs best among RNNs without complex gating, while MI-LSTM outperforms all other models by a large margin with approximately 4M parameters.The reported initializations are {2, 0.5, 0.5, 0} for MI-RNN and {1, 0.5, 0.5, 0} for MI-LSTM.
  • Character-level language modeling: On Hutter Challenge Wikipedia, a single-layer MI-LSTM with approximately 17M parameters outperforms larger stacked, GF-LSTM, and grid-LSTM models and achieves state-of-the-art performance.The compared models use approximately 27M, 20M, and 17M parameters, respectively.
  • Speech recognition: MI-LSTM+CTC+WFST achieves strong CER and WER on WSJ and clearly improves over a same-sized vanilla-LSTM baseline.The authors note that they did not conduct careful hyper-parameter search, leaving potential gains from decoding and regularization.
  • Skip-Thought evaluation: MI-uni-skip shortens Skip-Thought training by a factor of two and significantly improves final performance across semantic relatedness, paraphrase detection, and four classification tasks.It outperforms the baseline across all reported tasks and often exceeds combine-skip despite using half its vector dimensionality.

3.5 Teaching Machines to Read and Comprehend

Multiplicative Integration improves attentive reader performance with or without Batch Normalization. Combining MI with Batch Normalization gives the best performance and fastest convergence among the evaluated models.

  • Model comparisons: MI improves attentive reader performance regardless of whether Batch Normalization is used.The comparison includes vanilla LSTM, MI-LSTM, MI-LSTM+BN, and MI-LSTM with Batch-Norm applied everywhere.
  • Model comparisons: The combination of MI and Batch Normalization achieves the best performance and fastest convergence.The validation curves show a clear margin between models with and without MI.

4 Relationship to Previous Models

MI-RNN is related to HMM forward algorithms, second-order RNNs, and multiplicative RNNs, while using simpler parameterizations. The paper reports that MI-RNN outperforms HF-MRNN on Penn-Treebank and text8.

  • Relationship to Hidden Markov Models: Under probability-matrix, one-hot-input, linear-activation, and zero-bias constraints, MI-RNN implements the HMM forward algorithm.The product W x_t ⊙ U h_t propagates observation and transition probabilities.
  • Relations to Second Order RNNs and Multiplicative RNNs: MI-RNN and second-order RNNs both contain multiplicative second-order terms, but MI uses shared rank-1 factors formed from W and U.The MI term can be expressed using an outer product of corresponding rows from W and U.
  • Relations to Second Order RNNs and Multiplicative RNNs: MI-RNN represents its second-order term through a diagonal factorization, unlike the tensor decomposition used by Multiplicative RNNs.The diagonal factorization is built from α, W x_t, and U.
  • Relations to Second Order RNNs and Multiplicative RNNs: MI uses fewer parameters than the compared second-order models by sharing parameters across first- and second-order terms.The paper presents this simpler parameterization as more practical for large-scale problems and as helping avoid overfitting.
  • Empirical comparison: MI-RNN outperforms HF-MRNN on both Penn-Treebank and text8.The comparison uses results reported for an HF-trained Multiplicative RNN.

5 Conclusion

The paper proposes Multiplicative Integration as a simple Hadamard-product building block for combining information flows in RNNs. MI integrates readily into LSTMs and GRUs with almost no extra parameters, and the paper reports state-of-the-art performance across four tasks or 11 datasets.

  • Multiplicative Integration combines information flows in recurrent neural networks using a simple Hadamard product.The design replaces additive information integration with multiplicative integration.
  • MI integrates into popular RNN models, including LSTMs and GRUs, while introducing almost no extra parameters.The paper also states that implementation requires almost no additional work beyond implementing the RNN models.
  • MI achieves state-of-the-art performance on four different tasks or 11 datasets of varying sizes and scales.

A.1 MI-LSTM

MI-LSTM replaces the additive computations used for the block input and three gates with multiplicative integrations of input and recurrent transformations. Each computation includes multiplicative, first-order, bias, and nonlinearity terms.

  • Block input: MI-LSTM computes the block input by applying tanh to multiplicative and first-order combinations of W_z x_t and U_z h_{t−1}.
  • Input gate: The input gate applies a sigmoid to the MI combination of W_i x_t and U_i h_{t−1}.
  • Forget gate: The forget gate applies a sigmoid to the MI combination of W_f x_t and U_f h_{t−1}.
  • Output gate: The output gate applies a sigmoid to the MI combination of W_o x_t and U_o h_{t−1}.

A.2 MI-GRU

MI-GRU applies Multiplicative Integration to its update gate, reset gate, and candidate activation, then combines the candidate with the previous hidden state through the update gate.

  • Update gate: The MI-GRU update gate uses a sigmoid over multiplicative and first-order combinations of W_z x_t and U_z h_{t−1}.
  • Reset gate: The MI-GRU reset gate uses a sigmoid over multiplicative and first-order combinations of W_r x_t and U_r h_{t−1}.
  • Candidate activation: The candidate activation applies tanh to MI terms involving W_h x_t and U_h(r_t ⊙ h_{t−1}).
  • Hidden activation: The hidden activation combines the previous hidden state and candidate activation using the update gate.
Loading 1606.06630v2…