Source-linked AI summary
A Fast Unified Model for Parsing and Sentence Understanding
Samuel R. Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D. Manning, Christopher Potts
TL;DR
Tree-structured sentence models offer syntactic advantages but are hindered by external-parser dependence and poor support for batched computation. SPINN integrates parsing with tree-based interpretation in a linearized tree-sequence model, yielding faster computation and strong SNLI performance with little accuracy loss from removing external parses.
Problem
TreeRNNs exploit recursive syntactic structure but are difficult to batch efficiently and commonly rely on externally parsed sentences, limiting their practicality for large-scale NLP.
Method
SPINN linearizes tree-structured computation within a shift-reduce parser and integrates a neural parser and tracking LSTM for context-aware sentence interpretation.
Results
SPINN significantly improves SNLI entailment performance, runs an equivalent TreeLSTM an order of magnitude faster at test time, and operates without external parses with little accuracy loss.
Takeaways & Limitations
The integrated tree-sequence design preserves tree-structured strengths while enabling batched computation, on-the-fly parsing, and improved handling of negation and long sentences.
Takeaways & Limitations
The paper does not investigate combining SPINN with soft attention, despite soft attention's demonstrated effectiveness on SNLI.
Abstract
from arXiv · showhide
Tree-structured neural networks exploit valuable syntactic parse information as they interpret the meanings of sentences. However, they suffer from two key technical problems that make them slow and unwieldy for large-scale NLP tasks: they usually operate on parsed sentences and they do not directly support batched computation. We address these issues by introducing the Stack-augmented Parser-Interpreter Neural Network (SPINN), which combines parsing and interpretation within a single tree-sequence hybrid model by integrating tree-structured sentence interpretation into the linear sequential structure of a shift-reduce parser. Our model supports batched computation for a speedup of up to 25 times over other tree-structured models, and its integrated parser can operate on unparsed data with little loss in accuracy. We evaluate it on the Stanford NLI entailment task and show that it significantly outperforms other sentence-encoding models.
1 Introduction
Tree-structured encoders reflect recursive sentence meaning but are difficult to batch and typically depend on external parses. SPINN integrates parsing and interpretation in a linearized architecture, enabling batched computation and strong SNLI performance.
- TreeRNNs propagate information up parse trees, matching the recursive structure through which sentence meaning is constructed.
- Batched computation is crucial for efficient large-dataset training, but TreeRNNs use sentence-specific structures that make standard batching impossible.
- External parsing slows and complicates test-time use because standard TreeRNNs generally require syntactically parsed sentences.
- SPINN linearizes tree-structured computation and can parse and interpret unparsed sentences on the fly with nearly no additional computational cost.
- 25× speed increases over a standard TreeRNN implementation were observed, while SPINN significantly outperformed other sentence-encoding models on SNLI.
2 Related work
Related work established neural transition-based parsing and models using similar architectures for generation, while SPINN applies the architecture to sentence interpretation. Earlier unparsed-input TreeRNNs required expensive test-time search, motivating SPINN’s faster alternative.
- Neural parsers have long used transition-based parsing operations and data structures, including shift-reduce parsing.
- Recent generative language-modeling systems also used this architecture, but SPINN applied it to sentence interpretation rather than parsing or generation.
- Earlier TreeRNN methods could process unparsed inputs but required expensive test-time search, whereas SPINN offered a much faster alternative.
3 Our model: SPINN
SPINN integrates shift-reduce parsing with tree-structured sentence interpretation, using stack and buffer representations plus a tracking LSTM to support unparsed inputs, batching, and contextual composition.
- Shift-reduce foundation: SPINN linearizes tree-structured computation within a shift-reduce parser that processes tokens left to right.Each transition shifts a buffer token or reduces two stack elements into a combined representation.
- Vector composition: The model stores fixed-length vector representations in the stack and buffer, and its reduce operation applies a neural composition function to two stack vectors.The composition function produces the parent-node representation and pushes it back onto the stack.
- Tracking LSTM: The tracking LSTM summarizes processed sentence context and supplies features both to transition prediction and to the composition function.Its inputs include the top buffer element and the top two stack elements.
- Tree-sequence hybrid: The tree-sequence hybrid uses left context from the tracking LSTM to mitigate lexical ambiguity during tree-based composition.This addresses the limited context available to a tree model when initially combining a word with a nearby constituent.
- Parsing unparsed inputs: SPINN predicts transitions for unparsed inputs with a two-way softmax classifier based on the tracking LSTM state.The transition predictor is intentionally simpler than feature-rich state-of-the-art transition-based parsers.
- Efficient implementation: The thin stack represents feedforward computation with one T × D matrix and a pointer queue, enabling batched in-place GPU updates while storing each element once.This representation supports efficient backpropagation and contributes to speed gains over naïve SPINN and standard TreeRNN implementations.
- TreeRNN-equivalence: Without the tracking LSTM, SPINN is functionally equivalent to a conventional tree-structured neural network while additionally supporting integrated parsing and faster computation.The equivalence applies to the parsed-input no-tracking variant.
- Speed comparison: At batch size 512, SPINN is about 25× faster than the standard CPU TreeRNN implementation and about 4× slower than the RNN baseline.This comparison concerns sentence encoding speed on random input data.
4 NLI Experiments
The experiments evaluate SPINN on SNLI, a three-way sentence-pair inference task, using shared-parameter encoders and a classifier built from paired sentence representations. SPINN models outperform prior sentence-encoding models, while tree-structured and hybrid models show advantages on negation and longer sentences.
- Dataset and task: SNLI evaluates whether a premise and hypothesis are related by entailment, contradiction, or neutrality.The corpus contains about 570k human-labeled sentence pairs, with roughly 549k training, 9,842 development, and 9,824 test examples.
- Model and training: The classifier concatenates the premise and hypothesis vectors with their difference and elementwise product before predicting the three labels.Two shared-parameter SPINN copies produce the sentence vectors; an MLP and softmax then generate the label distribution.
- Overall results: Both SPINN-PI and full SPINN significantly outperform all previous sentence-encoding models on SNLI.The results favor full tree-structured composition over partial tree composition, which in turn outperforms the sequence-only RNN comparison.
- Parsing behavior: 92.4% transition accuracy measures how closely the full SPINN reproduces the Stanford Parser’s automatic parses at test time.The reported transition score is averaged across timesteps, and many sentences still receive partially invalid transition sequences.
- Error patterns: On negation cases, the RNN reaches 67% accuracy, while all three tree-structured models exceed 73%.The paper connects this pattern to parse trees making negation scope easier to identify and separate from the remaining sentence content.
- Error patterns: For premises of at least 20 words, SPINN-PI reaches 80.2% accuracy versus 76.7% for the RNN.The RNN’s test accuracy declines more quickly with increasing sentence length than the hybrid tree-sequence models.
5 Conclusions and future work
The paper presents SPINN as a fast TreeLSTM-equivalent architecture expanded with tree-sequence context and integrated parsing, while identifying soft attention, limited tracking features, and hard parsing decisions as future directions.
- SPINN-PI-NT is equivalent to a TreeLSTM but runs an order of magnitude faster at test time.
- SPINN-PI extends this architecture with a tree-sequence hybrid model that yields significant gains on the SNLI entailment task.
- The full SPINN integrates a fast parser, allowing operation without an external parser with little loss in accuracy.
- The paper does not pursue soft attention despite its demonstrated effectiveness on SNLI, though it expects combining attention with SPINN could improve performance.
- The tracking LSTM uses only simple features from the buffer and stack heads, and richer state representations are proposed as a route to better parsing and sentence encoding.
- Hard shift/reduce decisions prevent the semantic objective from influencing key parsing parameters, motivating differentiable stack operations as a future direction.