Source-linked AI summary
R-Transformer: Recurrent Neural Network Enhanced Transformer
Zhiwei Wang, Yao Ma, Zitao Liu, Jiliang Tang
TL;DR
RNNs have limited long-term dependency modeling and poor parallelizability, while Transformer attention lacks explicit locality modeling and depends on position embeddings. R-Transformer combines LocalRNNs with multi-head attention, and experiments across varied sequence tasks show large-margin advantages over competing recurrent and non-recurrent models in most tasks.
Problem
RNNs struggle with very long-term dependencies and parallel computation, while attention-based models inadequately capture local structure and rely on position embeddings.
Method
R-Transformer combines a LocalRNN for local structures with multi-head attention for global dependencies, without position embeddings and with parallel processing across positions.
Results
Across sequence modeling tasks from a wide range of domains, R-Transformer outperforms TCN, standard Transformer, and canonical recurrent architectures, with large margins in most tasks.
Takeaways & Limitations
The model provides a generic sequence architecture that jointly captures local structures and global long-term dependencies while mitigating recurrent and attention-based drawbacks.
Takeaways & Limitations
The paper does not consider sequence-to-sequence learning, leaving its extension to that setting for future work.
Abstract
from arXiv · showhide
Recurrent Neural Networks have long been the dominating choice for sequence modeling. However, it severely suffers from two issues: impotent in capturing very long-term dependencies and unable to parallelize the sequential computation procedure. Therefore, many non-recurrent sequence models that are built on convolution and attention operations have been proposed recently. Notably, models with multi-head attention such as Transformer have demonstrated extreme effectiveness in capturing long-term dependencies in a variety of sequence modeling tasks. Despite their success, however, these models lack necessary components to model local structures in sequences and heavily rely on position embeddings that have limited effects and require a considerable amount of design efforts. In this paper, we propose the R-Transformer which enjoys the advantages of both RNNs and the multi-head attention mechanism while avoids their respective drawbacks. The proposed model can effectively capture both local structures and global long-term dependencies in sequences without any use of position embeddings. We evaluate R-Transformer through extensive experiments with data from a wide range of domains and the empirical results show that R-Transformer outperforms the state-of-the-art methods by a large margin in most of the tasks. We have made the code publicly available at \url{https://github.com/DSE-MSU/R-transformer}.
1 INTRODUCTION
RNNs struggle with very long-term dependencies and sequential computation, while Transformer-style attention captures global dependencies but inadequately models locality and relies on position embeddings. R-Transformer combines LocalRNNs with multi-head attention to address these limitations.
- RNNs face gradient vanishing or exploding and sequential computation, limiting long-term dependency learning and parallelization.
- Its three hierarchical components are local recurrent networks, multi-head attention, and position-wise feedforward networks connected through residual and layer-normalization operations.
- Multi-head attention connects every position directly to every other, but treats positions identically and therefore loses sequential information.
- R-Transformer first uses LocalRNNs over sliding local windows to encode neighborhood structure before applying multi-head attention to model global dependencies.
2 SEQUENCE MODELING PROBLEM
The paper formulates sequence modeling as learning a function that maps an input sequence to a label. This framework covers tasks such as language modeling, recommendation, and DNA function prediction, but excludes sequence-to-sequence learning here.
- Sequence modeling learns a function mapping a length-N input sequence x1, x2, · · ·, xN to a label y in a label space Y.
- The label definition determines the task, including next-token prediction, future-item recommendation, or DNA function prediction.
- The paper does not consider sequence-to-sequence learning, although it states that the proposed model could be extended to it in future work.
3 THE R-TRANSFORMER MODEL
R-Transformer combines LocalRNNs for local sequential structure with multi-head attention for global long-term dependencies, followed by position-wise feedforward transformation. Its design aims to retain sequential information without position embeddings while enabling parallel local processing.
- Overall architecture: R-Transformer stacks LocalRNN, multi-head attention, and position-wise feedforward sub-layers to model local structure, global dependencies, and nonlinear feature transformations.Residual and layer-normalization connections are added between sub-layers.
- LocalRNN: Modeling local structures: LocalRNN processes each position's fixed-size preceding window with a shared RNN, producing representations that explicitly incorporate local sequential information.Unlike a full-sequence RNN, independent local windows avoid requiring long-term recurrence and support parallel processing.
- LocalRNN: Modeling local structures: LocalRNN preserves within-window order more fully than convolution and uses one-by-one sliding windows to incorporate global sequential information without position embeddings.The paper contrasts this with convolution, which ignores sequential information within each receptive field, and with position embeddings whose effectiveness may be limited.
- Capturing global long-term dependencies: Multi-head attention pools each position over all past positions, allowing long-term dependencies to be captured while different heads model dependencies from different aspects.The attention sub-layer is built on LocalRNN-refined representations and is followed by an independently applied position-wise feedforward network.
- Comparisons: Compared with TCN and standard Transformer, R-Transformer uses LocalRNN to encode locality and sequential order while retaining attention-based long-term memorization and avoiding position embeddings.The supplied material describes these architectural comparisons but does not provide the numerical contents of Table 1.
4 EXPERIMENT
R-Transformer was evaluated across sequence-modeling tasks spanning images, audio, and natural language against recurrent and non-recurrent baselines. It outperformed TCN and Transformer by a large margin in most tasks, although LSTM achieved the best word-level language-modeling results.
- Experimental setup: The experiments use sequence data from images, audios, and natural languages, comparing R-Transformer with Vanilla RNN, GRU, LSTM, TCN, and Transformer.For Transformer and R-Transformer, experiments were implemented in PyTorch; recurrent and TCN results were copied from Bai et al. (2018) under the same settings.
- Language modeling: Word-level language modeling follows similar trends, except that LSTM achieves the best results among all evaluated methods.Perplexity is the evaluation metric, with lower values indicating better performance.
- Language modeling: Character-level language modeling results show R-Transformer significantly outperforming competing models because LocalRNN and multi-head attention capture local structures and long-term dependencies.Transformer performs only slightly better than RNN and worse than other models, while TCN also benefits from modeling both dependency types.
- Overall results: R-Transformer outperforms both TCN and Transformer by a large margin consistently in most sequence-modeling tasks.The comparison spans canonical recurrent architectures, TCN, and Transformer across multiple domains.
5 RELATED WORK
Related sequence models fall broadly into convolution-based and multi-head-attention-based approaches. R-Transformer is motivated by convolutional models’ local-to-global processing while addressing limitations of recurrent and attention-only architectures.
- Organization of prior work: Non-recurrent sequence models are broadly divided into convolution-based approaches and approaches that do not rely on convolutions.The second category is represented primarily by multi-head-attention models.
- Convolution-based models: Convolution-based models include WaveNet, convolutional encoder-decoder models, gated convolutions, and TCN for capturing sequential information.TCN was systematically compared with canonical recurrent networks and achieved better performance in most cases (Bai et al., 2018).
- R-Transformer motivation: R-Transformer is motivated by convolutional work to model local information first and then focus on global dependencies.This ordering connects its design to the local-to-global processing strategy described for TCN-related models.
- Attention-based models: Multi-head attention, introduced in Transformer (Vaswani et al., 2017), became a widely used approach for learning long-term dependencies in sequence models.It was subsequently applied in other models including Devlin et al. (2018), Dehghani et al. (2018), and Dai et al. (2019).
6 CONCLUSION
R-Transformer combines LocalRNN with multi-head attention to model local structures and long-term dependencies while mitigating recurrent and attention-based limitations. Its broad empirical evaluation reports advantages over recurrent and non-recurrent alternatives.
- Conclusion: R-Transformer uses LocalRNN for local structures and multi-head attention for long-term dependencies without position embeddings.The model is also described as fully parallelizable over sequence positions.
- Conclusion: Empirical results across sequence-modeling tasks from a wide range of domains show advantages over TCN, standard Transformer, and canonical recurrent architectures.The conclusion characterizes these as remarkable advantages over state-of-the-art non-recurrent and recurrent models.