PaperlayerView exact revision on arXiv

Source-linked AI summary

Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks

Kai Sheng Tai, Richard Socher, Christopher D. Manning

arXiv:1503.00075v3cs.CLcs.AIcs.LG

TL;DR

Sentence representations must capture syntactic structure, not just word order, but prior LSTMs used only linear chains. This paper introduces Tree-LSTMs for tree-structured topologies and shows that they outperform existing systems and sequential LSTM baselines on semantic relatedness and sentiment classification.

  • Problem

    Prior sequential models do not fully capture meaning differences arising from word order or syntactic structure, motivating comparison with tree-structured sentence models.

  • Method

    The paper generalizes LSTMs to tree-structured topologies, with child-dependent gating and memory updates that selectively incorporate information from each child.

  • Results

    Tree-LSTMs outperform existing systems and sequential LSTM baselines on semantic relatedness prediction and sentiment classification.

  • Takeaways & Limitations

    Controlling for model dimensionality, Tree-LSTMs outperform sequential counterparts, supporting further study of structure in distributed sentence representations.

  • Takeaways & Limitations

    For large branching factors, Tree-LSTM parameters become impractical and may need to be tied or fixed to zero.

Abstract

from arXiv · show

Because of their superior ability to preserve sequence information over time, Long Short-Term Memory (LSTM) networks, a type of recurrent neural network with a more complex computational unit, have obtained strong results on a variety of sequence modeling tasks. The only underlying LSTM structure that has been explored so far is a linear chain. However, natural language exhibits syntactic properties that would naturally combine words to phrases. We introduce the Tree-LSTM, a generalization of LSTMs to tree-structured network topologies. Tree-LSTMs outperform all existing systems and strong LSTM baselines on two tasks: predicting the semantic relatedness of two sentences (SemEval 2014, Task 1) and sentiment classification (Stanford Sentiment Treebank).

1 Introduction

The introduction distinguishes bag-of-words, sequence, and tree-structured models for distributed phrase and sentence representations, motivating tree structures as a linguistically attractive response to syntactic effects on meaning. It presents Tree-LSTMs as an empirically strong architecture that outperforms existing systems and sequential LSTM baselines on semantic relatedness and sentiment classification.

  • Model classes: Distributed representations of phrases and sentences fall into bag-of-words, sequence, and tree-structured model classes.Bag-of-words representations are independent of word order, whereas sequence and tree-structured models are order-sensitive.
  • Motivation: Order-insensitive models cannot fully capture natural-language semantics because word order and syntactic structure can change meaning.The introduction illustrates this with “cats climb trees” versus “trees climb cats.”
  • Motivation: Tree-structured models are linguistically attractive because they relate to syntactic interpretations of sentence structure.The paper therefore turns to order-sensitive sequential or tree-structured models.
  • Evaluation: Tree-LSTMs outperform existing systems and sequential LSTM baselines on semantic relatedness prediction and sentiment classification.The evaluations use sentence pairs for semantic relatedness and sentences drawn from movie reviews for sentiment classification.

2 Long Short-Term Memory Networks

RNNs represent sequences through recurrent hidden states but can struggle to learn long-distance correlations because gradients vanish or explode. LSTMs address this with a gated memory cell that preserves information over long periods and supports multiple time scales, with bidirectional and multilayer variants extending the architecture.

  • Long Short-Term Memory Networks: RNNs process arbitrary-length sequences by recursively updating hidden state h_t from input x_t and previous state h_{t−1}.The hidden state can represent the sequence of tokens observed up to time t.
  • Long Short-Term Memory Networks: Exploding or vanishing gradients make it difficult for standard RNNs to learn long-distance correlations.Gradient components can grow or decay exponentially over long sequences during training.
  • Long Short-Term Memory Networks: LSTMs address long-term dependency learning with a memory cell that preserves state over long periods.The described unit contains input, forget, and output gates, a memory cell, and a hidden state, all in R^d.
  • Long Short-Term Memory Networks: Gated control determines forgetting, updating, and exposure of internal memory, allowing LSTMs to represent information over multiple time scales.The gates operate elementwise, so gating variables can vary across vector dimensions.
  • LSTM Variants: Bidirectional LSTMs combine forward and reverse sequence processing, while multilayer LSTMs pass each layer’s hidden state to the next layer to capture longer-term dependencies.The two variants can also be combined into a multilayer bidirectional LSTM.

3 Tree-Structured LSTMs

Tree-LSTMs generalize LSTMs from sequential chains to tree topologies, allowing each unit to combine information from multiple children. The Child-Sum variant suits unordered, variable-branching dependency trees, while the N-ary variant supports ordered trees with bounded branching and finer child-specific conditioning.

  • Architecture variants: Tree-LSTMs extend standard LSTMs with Child-Sum and N-ary variants that propagate information from multiple child units.Both variants address the limitation of strictly sequential information propagation in standard LSTM architectures.
  • Tree-LSTM units: Each Tree-LSTM unit uses child-dependent gates and memory updates, with one forget gate fjk for each child k.The units retain input and output gates, a memory cell, and a hidden state while conditioning computations on possibly many child states.
  • Dependency Tree-LSTMs: Child-Sum Tree-LSTMs sum child hidden states, making them suitable for unordered trees with highly variable branching, including dependency trees.A Child-Sum Tree-LSTM applied to a dependency tree is called a Dependency Tree-LSTM.
  • N-ary Tree-LSTMs: N-ary Tree-LSTMs support ordered trees with branching factor at most N and use separate child parameters for finer-grained conditioning.Separate parameter matrices let the model distinguish child positions, such as noun-phrase and verb-phrase children in constituency trees.
  • Parameterization limitation: For large N, additional off-diagonal forget-gate parameters are impractical and may be tied or fixed to zero.These parameters otherwise provide more flexible control over information propagation between child states and forget gates.
  • Applied architectures: The paper focuses on Dependency Tree-LSTMs and Constituency Tree-LSTMs, whose main difference is whether parameters distinguish dependents versus left and right children.Constituency Tree-LSTMs apply Binary Tree-LSTM units to binarized constituency trees, with inputs only at leaf nodes.

4 Models

Section 4 presents two Tree-LSTM models: a node-label classifier and a sentence-pair similarity model. The first predicts discrete labels from subtree representations, while the second predicts real-valued ordinal similarity scores from paired sentence representations.

  • Node labeling: The node-labeling model predicts discrete classes for selected tree nodes, such as properties of phrases spanned by parse-tree nodes.Labels are predicted for some subset of nodes in a tree.
  • Node labeling: At each node, a softmax classifier uses the hidden state computed from inputs in the rooted subtree to predict its label.The classifier takes the node hidden state hj as input.
  • Node labeling: Training minimizes the negative log-likelihood of true labels at labeled nodes with L2 regularization.The cost is defined over m labeled training nodes, with λ as the regularization hyperparameter.
  • Sentence similarity: The similarity model encodes each sentence’s parse tree into hL and hR, then predicts a real-valued score on the ordinal range [1, K].Higher scores indicate greater similarity, and real-valued outputs represent averages of multiple human ratings.
  • Sentence similarity: The similarity predictor considers both distance and angle between sentence representations, using combined measures h× and h+ because their combination outperforms either alone.The multiplicative measure h× can be interpreted as an elementwise comparison of representation signs.
  • Sentence similarity: Its training objective uses a sparse target distribution matching the gold rating’s expected value and minimizes regularized KL-divergence from that target.The target distribution p satisfies y = rT p, while the predicted expectation is ˆy = rT ˆpθ.

5 Experiments

The experiments evaluate Tree-LSTM architectures on sentiment classification and semantic relatedness prediction. They use standard benchmark datasets, tree structures, and controlled comparisons with sequential LSTM baselines.

  • 5 Experiments: Tree-LSTMs are evaluated on movie-review sentiment classification and prediction of human ratings for sentence-pair semantic relatedness.The semantic relatedness task measures how similar two sentences are in meaning.
  • Experimental Comparisons: Comparisons with sequential LSTMs control for the number of LSTM parameters by varying hidden-state dimensionality.Model-variant memory dimensions and composition-function parameter counts are summarized in Table 1.
  • Sentiment Classification: Sentiment classification uses the Stanford Sentiment Treebank for binary and fine-grained five-class prediction with standard train/dev/test splits.The splits are 6920/872/1821 for binary classification and 8544/1101/2210 for fine-grained classification.
  • Sentiment Classification: The sentiment models include Dependency Tree-LSTMs and Constituency Tree-LSTMs structured from dependency or constituency parses.Each relevant tree node is associated with sentiment supervision when its span matches a labeled training span.
  • Semantic Relatedness: Semantic relatedness uses the SICK dataset, containing 9927 sentence pairs split 4500/500/4927 across train, development, and test sets.Each pair receives a relatedness score y ∈[1, 5], averaged from 10 human ratings.

6 Results

The Tree-LSTM models achieve strong results across sentiment classification and semantic relatedness, with the Constituency Tree-LSTM excelling on fine-grained sentiment and the Dependency Tree-LSTM achieving the best relatedness results. Fine-tuning word representations improves sentiment performance, especially for fine-grained classification.

  • Sentiment classification: The Constituency Tree-LSTM outperforms existing systems on fine-grained sentiment classification and matches state-of-the-art accuracy on the binary subtask.It also outperforms the Dependency Tree-LSTM, which was trained on about 150K labeled nodes versus 319K for the Constituency Tree-LSTM.
  • Sentiment classification: Fine-tuning word representations significantly boosts fine-grained classification and yields a minor gain on binary classification.The initialized GloVe vectors were not originally trained to capture sentiment.
  • Semantic relatedness: The semantic-relatedness evaluation uses Pearson’s r, Spearman’s ρ, and mean squared error (MSE), with the first two measuring correlation against human evaluations.The models are compared with non-LSTM baselines and leading SemEval 2014 systems.
  • Semantic relatedness: The LSTM models outperform all compared systems without additional feature engineering, with the Dependency Tree-LSTM achieving the best results.Unlike sentiment classification, both Tree-LSTM models receive supervision only at the root in this task.
  • Semantic relatedness: The authors conjecture that the Dependency Tree-LSTM benefits from its more compact structure because it provides shorter paths from input words.This explanation is offered for its advantage in the semantic-relatedness setting.

7 Discussion and Qualitative Analysis

The qualitative analysis finds that Tree-LSTMs preserve and emphasize information from distant nodes, remain robust across sentence lengths, and encode useful structural information beyond sequential LSTMs. Dependency Tree-LSTMs outperform sequential counterparts on longer relatedness examples while also performing strongly on shorter sentences.

  • Qualitative Analysis: Dependency Tree-LSTM retrievals remain semantically related to “ocean” despite that word being distant from the dependency-tree root.“Ocean” is the second-furthest word from the root, with depth 4.
  • Qualitative Analysis: The Tree-LSTM associates “playing guitar” with the longer related phrase “dancing and singing in front of a crowd” despite zero token overlap.This illustrates robustness to differences in sentence length.
  • Sentence-Length Analysis: The length analysis tests whether tree structures mitigate long-sequence state preservation, using mean scores over 5 runs for each data point.Performance is examined as a function of sentence length using task-specific metrics.
  • Sentence-Length Analysis: Dependency Tree-LSTM significantly outperforms sequential counterparts on relatedness for sentences of length 13 to 15 while maintaining consistently strong performance on shorter sentences.The authors interpret this pattern as evidence that Tree-LSTMs encode semantically useful structural information in composed sentence representations.

8 Related Work

Prior work established distributed representations across words, phrases, sentences, and longer texts. The approach builds on recursive neural networks, where each tree node’s vector is composed from its children’s vectors.

  • Distributed Representations: Distributed word representations have found wide applicability across NLP tasks, motivating broader work on phrase, sentence, paragraph, and document representations.The cited research includes methods for learning distributed representations at each of these textual scales.
  • Recursive Neural Networks: The approach builds on recursive neural networks, abbreviated Tree-RNNs, which compose each tree node’s vector from the vectors of its children.Different composition functions produce numerous variants within the Tree-RNN framework.
  • Recursive Neural Networks: Tree-RNN variants differ according to their composition functions and have been applied to tasks including image parsing.The supplied passage identifies recursive neural networks as the framework underlying these variants.

9 Conclusion

The paper introduces Tree-LSTM, a generalization of LSTMs to tree-structured topologies with arbitrary branching factor. Tree-LSTMs outperform existing systems and sequential counterparts on semantic relatedness and sentiment classification, motivating further study of structure in sentence representations.

  • 9 Conclusion: Tree-LSTM generalizes LSTMs to tree-structured network topologies with arbitrary branching factor.The architecture extends beyond the linear-chain structure explored previously.
  • 9 Conclusion: Tree-LSTM outperforms existing systems on semantic relatedness and sentiment classification.The architecture was evaluated on both tasks.
  • 9 Conclusion: Controlling for model dimensionality, Tree-LSTM models outperform their sequential counterparts.This comparison isolates the effect of tree structure from model dimensionality.
  • 9 Conclusion: The results motivate further work on how structure produces distributed sentence representations.The paper identifies characterizing the role of structure as a future research direction.
Loading 1503.00075v3…