Source-linked AI summary
When Are Tree Structures Necessary for Deep Learning of Representations?
Jiwei Li, Minh-Thang Luong, Dan Jurafsky, Eudard Hovy
TL;DR
The paper asks when syntactic parse trees are necessary for deep representation learning, given limited rigorous evidence about the tasks where recursive models are appropriate. It benchmarks recursive and recurrent models across four tasks under apples-to-apples comparisons. Recursive models help mainly when long-distance semantic dependencies must be handled, while clause-based processing can bring recurrent models to similar performance in some cases.
Problem
Rigorous evaluations have not established exactly which tasks require syntax-based recursive models rather than sequential recurrent models.
Method
The paper compares recursive models with simple recurrent and LSTM models across sentiment classification, phrase matching, discourse parsing, and semantic relation classification.
Results
Recursive models help mainly on semantic relation extraction and other settings requiring long-distance headword associations, especially for very long sequences.
Takeaways & Limitations
Breaking long sentences into punctuation-delimited clause-like units and combining them afterward can make recurrent models perform similarly to tree models in some sentiment settings.
Takeaways & Limitations
The conclusions may be limited to the mostly one-layer algorithms evaluated and may not extend to other variants or the latest state-of-the-art.
Abstract
from arXiv · showhide
Recursive neural models, which use syntactic parse trees to recursively generate representations bottom-up, are a popular architecture. But there have not been rigorous evaluations showing for exactly which tasks this syntax-based method is appropriate. In this paper we benchmark {\bf recursive} neural models against sequential {\bf recurrent} neural models (simple recurrent and LSTM models), enforcing apples-to-apples comparison as much as possible. We investigate 4 tasks: (1) sentiment classification at the sentence level and phrase level; (2) matching questions to answer-phrases; (3) discourse parsing; (4) semantic relation extraction (e.g., {\em component-whole} between nouns). Our goal is to understand better when, and why, recursive models can outperform simpler models. We find that recursive models help mainly on tasks (like semantic relation extraction) that require associating headwords across a long distance, particularly on very long sequences. We then introduce a method for allowing recurrent models to achieve similar performance: breaking long sentences into clause-like units at punctuation and processing them separately before combining. Our results thus help understand the limitations of both classes of models, and suggest directions for improving recurrent models.
1 Introduction
The paper compares recursive models, which follow syntactic parse trees, with recurrent sequence models to determine when each architecture is useful across four NLP tasks. It focuses on whether parsing helps with composition, discourse, and long-distance semantic dependencies.
- Model comparison: Recursive models combine neighboring words according to parse-tree structure, while recurrent models process tokens sequentially.Recursive computation proceeds bottom-up from leaves to the root; recurrent computation combines each token with the previously built representation.
- Research motivation: Recursive models may capture long-distance dependencies when tokens far apart in sequence are structurally close in a parse tree.The paper notes that parsing can also be slow, domain-dependent, and errorful, making its value an empirical question.
- Evaluation tasks: The paper evaluates sentence- and phrase-level sentiment classification, phrase matching, discourse parsing, and semantic relation classification.These tasks probe semantic composition, similarity between questions and answer phrases, larger text-unit meaning, and long-term dependencies.
- Research motivation: The study enforces apples-to-apples comparisons to identify which problems require recursive models and which recurrent models may handle sufficiently.It applies existing models to existing tasks and emphasizes analysis over novel algorithms or tasks.
- Contribution: The paper aims to clarify the limitations of recursive and recurrent models and suggest directions for improving recurrent models.
2 Recursive and Recurrent Models
Recursive and recurrent models both map token sequences to fixed-dimensional representations, but they differ in the order and structure used for composition. The section describes standard, bidirectional, LSTM, and tree-based variants.
- Standard recurrent models: Recurrent networks process words sequentially, combining the current word embedding with the previous hidden vector.For a sequence of length N_S, h_N_S represents the whole sequence.
- Standard recursive models: Recursive networks compute each parent representation from its immediate parse-tree children in a bottom-up process until reaching the root.
- Bidirectional models: Bidirectional recurrent models calculate embeddings in both forward and backward directions and combine the resulting representations.Combination can use concatenation or another compositional operation that preserves vector dimensionality.
- LSTM models: LSTMs associate each timestep with input, memory, and output gates while combining the current input with the preceding hidden state.The timestep representation h_t is produced by the LSTM recurrence.
- Tree LSTMs: Tree LSTMs extend the LSTM idea to parse trees by associating memory and forget gates with tree nodes.
3 Experiments
The experiments compare tree, sequence, bidirectional sequence, and LSTM variants across sentiment, question-answer matching, and semantic relationship classification. Tree models help most when tasks require integrating distant structures, while clause-based hierarchical sequences can approximate their benefits.
- Experimental setup: The experiments compare standard and LSTM tree models with standard, bidirectional, and hierarchical sequence models across multiple NLP tasks.The study uses the Stanford Sentiment Treebank, Pang sentiment classification, UMD-QA matching, and semantic relationship classification settings.
- 3.1 Stanford Sentiment TreeBank: Tree models slightly improve Stanford Sentiment Treebank root-level identification but provide no significant phrase-level improvement.The comparison concerns standard tree models versus sequence models; bidirectional recurrent models largely bridge the root-level gap.
- 3.1 Stanford Sentiment TreeBank: Hierarchical sequence models split sentences at punctuation, compute sub-sentence representations separately, and then combine them with another sequence LSTM.This design is presented as a recurrent approximation to parse-tree processing for long sentences.
- 3.4 Semantic Relationship Classification: Recursive models outperform corresponding recurrent versions on semantic relationship classification because parse-based processing brings distant target words together earlier.The standard tree-versus-standard sequence comparison is significant at p = 0.004.
4 Discussion
Across the evaluated tasks, recursive models help most when long-distance semantic associations matter, while recurrent models often match them on shorter or insufficiently supervised inputs. Clausal segmentation offers a simple approximation, and recurrent components can encode informative evidence despite lacking linguistic meaning.
- Caveats: The comparison primarily analyzes basic recursive and recurrent models because fair comparison becomes harder for more sophisticated variants.Most models use one neural-composition layer, limiting whether conclusions extend to other variants or current state-of-the-art systems.
- Task-dependent advantages: Recursive models shine when semantic relation extraction requires associating single headwords across a long distance.The paper suggests this advantage may extend to tasks involving long-distance semantic dependencies.
- Sequence length and supervision: Tree models help more on long sequences: they slightly improve sentence-level sentiment identification but add little at the phrase level.Bidirectional recurrent models largely bridge this gap, producing equivalent or sometimes better results.
- Sequence length and supervision: On long sequences with insufficient supervision, no significant difference appears between tree-based and sequence-based models.The cited example has supervision only at the tops of long sequences.
- Approximation to tree models: Clausal segmentation breaks long sentences at punctuation, processes clause-like units separately, and then joins them together.This simple approximation sometimes reaches tree-model performance on sentiment and provides a slight performance boost.
- Intermediate representations: Recurrent components can embed informative evidence as well as linguistically meaningful parse-tree phrases in the UMD-QA phrase-matching task.The comparison concerns outputs from recurrent time steps and representations of intermediate parse-tree nodes.