Source-linked AI summary
Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
Yikang Shen, Shawn Tan, Alessandro Sordoni, Aaron Courville
TL;DR
Standard recurrent models lack an explicit bias for the hierarchical constituent structure of language. The paper introduces ON-LSTM, which orders neurons and uses master gates to coordinate their updates, achieving good performance across four language tasks and aligning induced parses with expert annotations.
Problem
Standard LSTMs can track multiple timescales but lack an explicit bias for modeling language's hierarchy of nested constituents.
Method
ON-LSTM orders neurons by timescale and uses cumax-based master input and forget gates so updates follow the neuron ordering.
Results
ON-LSTM performs well on language modeling, unsupervised constituency parsing, targeted syntactic evaluation, and logical inference, with parsing aligned to human expert annotation.
Takeaways & Limitations
The ordered-neuron inductive bias brings recurrent networks closer to tree-like composition while supporting long- and short-term information in separate neurons.
Takeaways & Limitations
The targeted syntactic evaluation results are not strictly comparable with the original paper because the generated test set differs.
Abstract
from arXiv · showhide
Natural language is hierarchically structured: smaller units (e.g., phrases) are nested within larger units (e.g., clauses). When a larger constituent ends, all of the smaller constituents that are nested within it must also be closed. While the standard LSTM architecture allows different neurons to track information at different time scales, it does not have an explicit bias towards modeling a hierarchy of constituents. This paper proposes to add such an inductive bias by ordering the neurons; a vector of master input and forget gates ensures that when a given neuron is updated, all the neurons that follow it in the ordering are also updated. Our novel recurrent architecture, ordered neurons LSTM (ON-LSTM), achieves good performance on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference.
1 INTRODUCTION
Natural language has tree-like structure, but recurrent models impose a sequential chain and standard LSTMs lack an explicit hierarchy bias. The paper introduces ordered neurons and ON-LSTM to address this gap across four tasks.
- Natural language uses syntax to compose words into nested, meaningful constituents with a tree-like underlying structure.
- Supervised parsers are limited by scarce annotated data, domain-specific syntax violations, and evolving language rules.
- Grammar induction remains open because models may learn trivial branching structures, require difficult reinforcement-learning training, or be complex to implement.
- RNNs impose a chain structure that can complicate long-term dependencies, generalization, and negation handling despite LSTMs' potential for implicit syntax processing.
- Ordered neurons and ON-LSTM introduce a hierarchy-oriented recurrent bias and perform well on language modeling, unsupervised parsing, syntactic evaluation, and logical inference.
2 RELATED WORK
Prior work incorporates tree structure through supervised parsing, external parser guidance, stacks, explicit structured models, and varying recurrence timescales. These approaches differ in supervision, complexity, and assumptions about hierarchy regularity.
- Earlier neural approaches use expert-labeled treebanks or external parsers to incorporate tree structure into language tasks.
- RNNGs can outperform LSTMs on subject-verb agreement, motivating broader evaluation of structural information in recurrent models.
- Grammar induction learns syntactic structure from raw corpora without expert-labeled data.
- PRPN learns syntactic distances through self-attention for parsing, but its added complexity makes the model unwieldy in practice.
- Clockwork RNN and related models update hidden-state segments at predetermined timescales, typically assuming hierarchy regularity.
3 ORDERED NEURONS
The proposed representation maps constituency hierarchy onto recurrent hidden states by assigning different neurons different update timescales. Higher-ranking neurons retain broader constituent information, while lower-ranking neurons track local structure.
- The model aims for each hidden state to represent constituents along the path from the current token to the sentence root.
- Ordered neurons impose different information timescales, with high-ranking neurons representing long-term or global information and low-ranking neurons representing short-term or local information.
- Figure 2 links constituency trees to hidden-state update frequencies, showing less frequent updates for topmost neuron groups and more frequent updates for lower groups.
4 ON-LSTM
ON-LSTM extends standard LSTM with ordered, cumulative master gates that impose structured update behavior across neurons, linking cell-state operations to constituent boundaries.
- Structured gating mechanism: ON-LSTM replaces the standard cell-state update with master input and forget gates whose neuron-dependent updates enforce an ordering.The standard input and forget gates remain for fine-grained operations within neuron blocks.
- Activation function: cumax(): cumax() forms continuous gate values by cumulatively summing softmax outputs, representing the expected value of an underlying binary gate.The binary gate divides neurons into zero and one segments, enabling different update rules for long- and short-term information.
- Structured gating mechanism: The master forget gate is monotonically increasing, while the master input gate is monotonically decreasing across the neuron ordering.Their product identifies overlapping blocks corresponding to incomplete constituents that should be updated.
- Structured gating mechanism: A large forget-gate split point erases more lower-ranked neurons, analogous to closing higher-level constituents in a parse tree.A small split point instead preserves high-level information while ending a lower-level constituent.
- Structured gating mechanism: The master input split point controls whether current input is treated as long-term information or local information that can soon be erased.Large split points preserve input for several time steps, whereas small split points allow erasure in the next few steps.
- Implementation: Master gates use reduced-dimensional vectors repeated across chunks, so neurons within each chunk share gates and require fewer additional parameters.The hidden-state dimension is divided by a chunk-size factor C for coarse-grained control.
5 EXPERIMENTS
The experiments evaluate ON-LSTM across language modeling, unsupervised constituency parsing, targeted syntactic evaluation, and logical inference. Across these tasks, the results show stronger language modeling and parsing performance, advantages on long-term dependencies, and better generalization to longer logical sequences.
- Language Modeling: ON-LSTM performs better than the standard LSTM on the Penn TreeBank language-modeling task while using the same layers, embedding dimensions, and hidden-state units.The improvement does not require skip connections or a significant parameter increase.
- Unsupervised Constituency Parsing: The second ON-LSTM layer achieves state-of-the-art unsupervised constituency-parsing results on the WSJ test set.The model infers split-point sequences from master forget-gate distributions and applies a top-down greedy recursive parsing algorithm.
- Targeted Syntactic Evaluation: ON-LSTM performs better on long-term dependency cases, whereas the baseline LSTM performs better on short-term cases in targeted syntactic evaluation.The authors associate this pattern with the hidden state having relatively few units to represent both long- and short-term information.
- Targeted Syntactic Evaluation: The targeted syntactic evaluation results are not strictly comparable with Marvin and Linzen (2018) because the released-code test set differs from the original test set.The experiment uses the released codebase and the same settings proposed in the earlier work.
- Logical Inference: Both models achieve nearly 100% accuracy on logical sequences of length ≤3, but ON-LSTM performs better on longer sequences, with the gap increasing for lengths ≥7 unseen during training.Models train on sequences with 6 or fewer logical operations and are tested on sequences with at most 12 operations.
6 CONCLUSION
The paper introduces ordered neurons and the ON-LSTM recurrent unit to give recurrent networks a tree-like compositional bias. ON-LSTM induces latent structures coherent with human annotations and performs well across language modeling, parsing, long-term dependency, and logical inference tasks.
- ON-LSTM combines ordered neurons, a new gating mechanism, and cumax(·) to allocate hidden-state neurons to long- and short-term information.This design brings recurrent neural networks closer to tree-like composition operations.
- ON-LSTM induces latent natural-language structure in unsupervised constituency parsing coherently with human expert annotation.
- The inductive bias supports good performance on language modeling, long-term dependency, and logical inference tasks.
A SAMPLE PARSES FROM THE MODEL WITH THE BEST PERPLEXITY
Figure A.1 presents sample parses from the second ON-LSTM layer alongside parses converted from human expert annotations with punctuation removed.
- The examples include sentences about competent RTC management and a decision expected next year.
- The left side shows parses produced by the second layer of the ON-LSTM model.
- The right side shows parses converted from human expert annotations after removing all punctuation.