Source-linked AI summary
The Importance of Being Recurrent for Modeling Hierarchical Structure
Ke Tran, Arianna Bisazza, Christof Monz
TL;DR
The paper addresses whether non-recurrent models can capture hierarchical structure as recurrent networks do. It compares Transformer-based Fully Attentional Networks with LSTMs across controlled agreement and inference tasks, finding that LSTMs slightly but consistently outperform FANs and generalize better to longer sequences. The authors conclude that recurrency is important when hierarchical structure matters, while noting that non-recurrent models can match or exceed LSTMs on machine translation.
Problem
The paper addresses limited evidence about whether non-recurrent networks can model hierarchical structure as recurrent networks do.
Method
The study compares LSTMs with Transformer-based Fully Attentional Networks on subject-verb agreement and logical inference tasks.
Results
LSTMs slightly but consistently outperform FANs, show greater robustness in agreement, and generalize better to longer logical-inference sequences.
Takeaways & Limitations
The findings suggest that recurrency should not be sacrificed for efficiency when hierarchical structure matters.
Takeaways & Limitations
The conclusion does not support always preferring LSTMs, since FAN- and CNN-based networks can perform comparably or better on machine translation.
Abstract
from arXiv · showhide
Recent work has shown that recurrent neural networks (RNNs) can implicitly capture and exploit hierarchical information when trained to solve common natural language processing tasks such as language modeling (Linzen et al., 2016) and neural machine translation (Shi et al., 2016). In contrast, the ability to model structured data with non-recurrent neural networks has received little attention despite their success in many NLP tasks (Gehring et al., 2017; Vaswani et al., 2017). In this work, we compare the two architectures---recurrent versus non-recurrent---with respect to their ability to model hierarchical structure and find that recurrency is indeed important for this purpose.
1 Introduction
The paper asks whether non-recurrent models can capture hierarchical structure as recurrent models do, focusing on Transformer-based Fully Attentional Networks. It motivates this comparison by the strong performance and parallelizable computation of non-recurrent architectures.
- Transformers achieve strong sequence-modeling and translation performance without recurrent or convolutional connections.
- Non-recurrent models are appealing because their computations are highly parallelizable on modern GPUs.
- The study compares recurrent and non-recurrent architectures on their ability to capture hierarchical structure without explicit structural representations.
- The paper provides a first answer to whether non-recurrent models have the same ability to model hierarchical structure.
2 FAN versus LSTM
FANs use self-attention to access all previous inputs directly, whereas LSTMs recursively compress prior information into a single vector. The paper then measures how these architectural differences affect hierarchical-structure learning.
- At time t, FANs access all previous time steps directly with O(1) computational operations through self-attention.Self-attention computes a weighted average of previous input representations.
- LSTMs recursively compress all previous information into a single vector based on the current input and previous computation.
- Figure 1 contrasts the summarized prediction vector and previous-input information flow in LSTMs and FANs.
- The paper evaluates both architectures with controlled experiments measuring their ability to learn hierarchical structure.
3 Tasks
The study uses subject-verb agreement and logical inference to test whether models exploit hierarchical structure in sequential data. These tasks require structural features for accurate predictions or inferences.
- The first task is subject-verb agreement, which tests syntactic-dependency capture in natural language.
- The second task is logical inference, which compares sequence-based and tree-based models on exploiting hierarchical structure.
- Both tasks are selected because accurate performance requires models to exploit hierarchical structural features.
4 Subject-Verb Agreement
The subject-verb agreement experiments compare LSTMs and FANs under language-model and explicit verb-number objectives. LSTMs consistently outperform FANs and are more robust as subject-verb distance and intervening attractors increase, while some FAN attention heads identify subjects less reliably at longer distances.
- Language-model objective: Both models achieve high subject-verb agreement accuracy under the language-model objective, but LSTMs consistently outperform FANs.The validation perplexities were 67.06 for LSTMs and 69.14 for FANs.
- Language-model objective: LSTMs are more robust than FANs as subject-verb distance and the number of agreement attractors increase.Agreement attractors are intervening nouns with the opposite number from the subject.
- Explicit verb-number objective: LSTMs again consistently outperform FANs when models are explicitly trained to predict verb number.This result holds despite FAN self-attention connecting words with O(1) executed operations, whereas RNNs require recurrent operations.
- Training objectives: The authors note that the language-model objective yields higher accuracies than the number-prediction objective, possibly because of optimization and embedding-output weight sharing.They leave a thorough investigation of this difference to future work.
- Attention analysis: Some higher-layer FAN attention heads frequently select the subject on correct predictions, but this accuracy decreases linearly with subject-verb distance.The reported heads include ℓ2 h1 and ℓ3 h0, with ℓ3 as the highest layer.
5 Logical inference
The logical inference task tests whether models learn nested structures and operator scope by classifying seven logical relations. FANs and LSTMs perform similarly with full training, but LSTMs are more accurate and generalize better when trained on limited logical complexity.
- Task setup: The task classifies sentence pairs into seven logical relations using an artificial language with three operators and samples containing 1–12 logical operations.The relations include entailment, equivalence, contradiction, and semantic independence.
- Task setup: The task is nontrivial because correct classification requires learning nested structures and the scope of logical operators.Bag-of-words models achieve less than 59% accuracy, indicating that surface aggregation is insufficient.
- Models: Premise and hypothesis sentences are encoded as fixed-size vectors, concatenated, and passed to a 3-layer ReLU network for 7-way classification.The LSTM uses its final hidden state, while the FAN uses self-attention with two trainable queries to form a fixed-size representation.
- Results: Both FANs and LSTMs perform similarly when trained on the whole dataset.The comparison uses training on samples with at most n logical operators and evaluation across 13 operator-count bins.
- Results: When trained on samples with at most 6 logical operators, LSTMs outperform FANs on similar examples and generalize better to examples with 6 < n ≤12 operators.Figure 4 compares training on all data with training restricted to samples containing at most n logical operators.
6 Discussion and Conclusion
The paper compares recurrent LSTMs with non-recurrent FANs on hierarchical structure and finds that LSTMs consistently perform better on the tested agreement and logical inference tasks. It concludes that recurrency matters when hierarchical structure is important, without implying that recurrent models should always be preferred.
- Scope and findings: The experiments compare LSTMs and FANs on their ability to capture hierarchical structure in sequential data.The comparison covers subject-verb agreement and logical inference.
- Scope and findings: LSTMs slightly but consistently outperform FANs across the experiments.The advantage includes greater robustness to misleading agreement features and better generalization to longer logical-inference sequences.
- Implication: Recurrency is suggested as a key model property when hierarchical structure matters for the task.The conclusion cautions against sacrificing recurrency solely for efficiency in such settings.
- Scope and limitations: The findings do not imply that LSTMs should always be preferred over non-recurrent architectures.FAN- and CNN-based networks have performed comparably or better than LSTMs on complex machine translation tasks.