Source-linked AI summary
What Do Recurrent Neural Network Grammars Learn About Syntax?
Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig, Noah A. Smith
TL;DR
The paper asks what syntactic information high-performing recurrent neural network grammars learn and how their latent representations can be interpreted. It uses model ablations and a gated-attention RNNG to investigate composition, headedness, and nonterminal labels. Explicit composition is crucial, attention largely tracks heads, and removing nonterminal labels supports largely endocentric phrasal representations.
Problem
The paper examines what syntactic information RNNGs learn despite neural parameters being difficult to interpret.
Method
The study combines RNNG and data ablations with a gated-attention composition mechanism to inspect phrasal representations.
Results
Explicit composition is crucial for performance; attention reflects heads, and nonterminal labels provide little additional information.
Takeaways & Limitations
The findings support a largely endocentric view of phrasal representations and show that RNNG latent attention can reveal syntactic structure.
Takeaways & Limitations
The unlabeled-grammar findings may change when the model is trained with nonterminal information.
Abstract
from arXiv · showhide
Recurrent neural network grammars (RNNG) are a recently proposed probabilistic generative modeling family for natural language. They show state-of-the-art language modeling and parsing performance. We investigate what information they learn, from a linguistic perspective, through various ablations to the model and the data, and by augmenting the model with an attention mechanism (GA-RNNG) to enable closer inspection. We find that explicit modeling of composition is crucial for achieving the best performance. Through the attention mechanism, we find that headedness plays a central role in phrasal representation (with the model's latent attention largely agreeing with predictions made by hand-crafted head rules, albeit with some important differences). By training grammars without nonterminal labels, we find that phrasal representations depend minimally on nonterminals, providing support for the endocentricity hypothesis.
1 Introduction
The paper treats RNNGs as flexible but difficult-to-interpret models for studying syntax. It manipulates their inductive biases to test composition, headedness, and nonterminal-label hypotheses, while reporting strong parsing results.
- RNNGs achieve state-of-the-art parsing and language-modeling performance while retaining linguistically motivated structure and relatively few independence assumptions.
- The paper frames model discoveries as syntax propositions whose relevance is limited to the particular genre and dialect represented in the data.
- The study uses ablations and gated attention to inspect how RNNGs represent syntax, focusing on composition, lexical heads, and nonterminal labels.The GA-RNNG supports closer inspection of latent attention and phrasal representations.
- Explicit composition is central to RNNG performance, while lexical heads help represent most phrase types and nonterminal labels add little information.The authors also report that multiple salient heads are especially common in conjunctions.
- 93.6 F1 on English PTB phrase-structure parsing was achieved by a non-ensembled RNNG variant, with converted PTB dependencies reaching 95.8 UAS and 94.6 LAS.
2 Recurrent Neural Network Grammars
An RNNG is a generative model over strings and phrase-structure trees that constructs trees through state-dependent actions. Its explicit REDUCE composition function converts completed constituents into vector representations used by subsequent decisions.
- An RNNG defines a joint probability distribution over terminal strings and phrase-structure nonterminals, parameterized by sets of symbols and neural-network parameters.
- The model constructs a tree top-down using NT, GEN, and REDUCE actions while maintaining a stack, generated-word buffer, and past-action history.During supervised training, the tree determines an oracle action sequence.
- REDUCE pops a completed constituent’s elements, applies a composition function, and pushes one composed representation back onto the stack.The composition function uses a bidirectional LSTM to compute the new constituent vector.
- Each timestep encodes the stack, buffer, and action history with separate LSTMs to predict the next feasible action.
- The model assigns joint probability by multiplying local action probabilities conditioned on all previous actions, supporting parsing and language modeling.Parsing seeks the most probable tree given a sentence, while language modeling marginalizes over possible parses.
3 Composition is Key
Explicit composition is central to RNNG performance: removing the stack is most damaging, while a stack-only model outperforms the full architecture across parsing and language modeling results.
- 3 Composition is Key: RNNGs parsed more accurately than sequence-based models that represent trees as linearized symbols without explicitly capturing well-formed tree structure.The comparison applies under both discriminative and generative settings.
- 3.1 Ablated RNNGs: The stack-only RNNG was the strongest ablation and outperformed the full RNNG with stack, buffer, and action history.This supports explicit composition as more useful than retaining all three redundant data structures.
- 3.1 Ablated RNNGs: Ablating the stack caused the worst new result, while stack-ablated syntax modeling provided little language-modeling benefit over a sequential LSTM.The stack is the component that applies the composition function.
4 Gated Attention RNNG
The GA-RNNG augments composition with attention and gating to inspect how phrasal representations are formed. Its attention distributes information across constituents, while a gate balances constituent content with nonterminal information.
- 4 Gated Attention RNNG: GA-RNNG attention provides a way to examine whether phrasal representations rely on privileged lexical heads, as predicted by head-based syntactic theories.The model’s weighted attention is more expressive than traditional head rules because it can divide attention among multiple constituents.
- 4 Gated Attention RNNG: GA-RNNG combines child representations through attention with a gated nonterminal embedding, enabling more interpretable phrasal composition.The model was introduced to investigate the nature of learned phrasal representations, including the roles of heads and nonterminal labels.
- 4 Gated Attention RNNG: At each REDUCE operation, attention weights assign a total of one unit across the new phrase’s child constituents.The weighted sum can distribute attention among multiple constituents rather than selecting only one lexical head.
- 4 Gated Attention RNNG: The constituent source vector is a convex combination of child representations, then combined with a separate nonterminal embedding through sigmoid gating.The gate is bounded between 0 and 1 in each dimension, allowing context-dependent modulation of the two sources.
- 4 Gated Attention RNNG: The GA-RNNG outperforms the baseline RNNG with all three structures and performs competitively with the strongest stack-only variant.Its results are reported in Tables 2–4.
5 Headedness in Phrases
GA-RNNG attention is sharply concentrated on a few constituents, revealing learned headedness that broadly follows hand-crafted rules but diverges in important cases.
- 5.1 The Heads that GA-RNNG Learns: Learned attention has much less than 2 average choices across phrase categories, substantially below the uniform baseline but not usually concentrated on one constituent alone.Lower perplexity indicates peaked attention, while values above one show that representations can distribute weight across multiple constituents.
- 5.1 The Heads that GA-RNNG Learns: High-entropy compositions correspond to complex phrases with conjunctions or multiple plausible heads, where attention is more divided.
- 5.1 The Heads that GA-RNNG Learns: Noun phrases usually emphasize the rightmost noun, give adjectives nontrivial weight, and nearly ignore determiners, matching established head intuitions.
- 5.1 The Heads that GA-RNNG Learns: Conjunctions produce several strategies: single-noun coordination may select either noun, whereas coordination of noun phrases consistently selects the conjunction.
- 5.1 The Heads that GA-RNNG Learns: Verb phrases often emphasize the noun phrase over the verb, while prepositional phrases almost always emphasize the preposition, including connector-like cases.
- 5.2 Comparison to Existing Head Rules: Attention-based dependencies overlap more with Collins than Stanford head rules, reaching 49.8 UAS versus 40.4 UAS, but errors remain high for verbs.The authors attribute the gap to Stanford rules incorporating more semantic considerations than this purely syntactic model.
- 5.2 Comparison to Existing Head Rules: GA-RNNG learns head-like conventions through joint likelihood rather than a parsing proxy, while retaining a noun-centered verb-phrase hypothesis for future study.
6 The Role of Nonterminal Labels
Removing nonterminal labels reduces parsing performance only slightly, while phrase vectors still recover category-like clusters and depend strongly on lexical connectors.
- 6 The Role of Nonterminal Labels: These results support largely endocentric phrasal representations, although training with nonterminal information could change the unlabeled-model findings.
- 6 The Role of Nonterminal Labels: Unlabeled GA-RNNG reaches 93.5% unlabeled F1 versus 94.2% for GA-RNNG, suggesting nonterminal labels add relatively little information.
- 6 The Role of Nonterminal Labels: Without nonterminal categories, t-SNE phrase vectors tend to recover Penn Treebank-like categories on unseen test data.
- 6 The Role of Nonterminal Labels: PP and SBAR representations depend less on category labels than on connectors, clustering phrases beginning with words such as for, at, to, under, and by.
- 6 The Role of Nonterminal Labels: The model treats SBARs with optional that as similar to SBARs without that, while separating some because- and of-based phrases.
7 Related Work
The paper builds on probes of syntax in neural sequence models and prior latent-category work, while distinguishing its end-to-end attention-based approach to learning heads.
- 7 Related Work: Prior sequence-to-sequence studies found syntactic information such as voice and tense emerging as a by-product of translation objectives.
- 7 Related Work: Earlier parsing research used lexicalized or nonterminal-augmented probabilistic context-free grammars, motivating comparisons with learned phrase representations.
- 7 Related Work: Prior work showed that dimensionality reduction and latent variables can recover conventional or fine-grained nonterminal categories from weaker structural information.
- 7 Related Work: Unlike earlier unsupervised head extraction, GA-RNNG uses end-to-end neural attention trained to maximize the log probability of the correct action.
8 Conclusion
The paper uses ablations and gated attention to show that composition, headedness, and limited dependence on nonterminal labels organize what RNNGs learn about syntax.
- 8 Conclusion: Composition is crucial for RNNG performance, while gated attention exposes head-like but not exclusively single-component phrase representations.
- 8 Conclusion: Training without nonterminal information supports largely endocentric representations, with traditional categories emerging from composed vectors.
- 8 Conclusion: The findings support the view that bracketing annotation does most of the syntactic work, while nonterminal categories remain readily discoverable.