Source-linked AI summary
Learning to Compose Task-Specific Tree Structures
Jihun Choi, Kang Min Yoo, Sang-goo Lee
TL;DR
RvNNs benefit from hierarchical composition but require structured input, making data preparation and batching difficult. Gumbel Tree-LSTM learns task-specific trees from plain text using discrete parent selection with Straight-Through Gumbel-Softmax, and experiments show competitive or better performance with faster convergence.
Problem
RvNNs require structured input, while preparing parse-tree data is expensive and the best hierarchical composition may differ across tasks.
Method
Gumbel Tree-LSTM uses a composition query vector and Straight-Through Gumbel-Softmax to select candidate parents dynamically while training with standard backpropagation.
Results
The model outperforms all other RvNN models, is competitive with state-of-the-art models, and converges faster than other complex models.
Takeaways & Limitations
The results suggest that optimal input structure may differ by task, motivating investigation of task-specific latent tree structures.
Abstract
from arXiv · showhide
For years, recursive neural networks (RvNNs) have been shown to be suitable for representing text into fixed-length vectors and achieved good performance on several natural language processing tasks. However, the main drawback of RvNNs is that they require structured input, which makes data preparation and model implementation hard. In this paper, we propose Gumbel Tree-LSTM, a novel tree-structured long short-term memory architecture that learns how to compose task-specific tree structures only from plain text data efficiently. Our model uses Straight-Through Gumbel-Softmax estimator to decide the parent node among candidates dynamically and to calculate gradients of the discrete decision. We evaluate the proposed model on natural language inference and sentiment analysis, and show that our model outperforms or is at least comparable to previous models. We also find that our model converges significantly faster than other models.
Introduction
RvNNs represent sentences using hierarchical tree structure but typically require structured input, creating preparation and batching difficulties. The paper proposes Gumbel Tree-LSTM to learn task-specific trees from plain text and reports competitive performance with faster convergence.
- RvNNs encode sentences using structured inputs such as parse trees, extending recurrent models from linear chains to hierarchical structures.
- Parse-tree annotation is expensive to prepare, difficult to batch, and may not provide the optimal word-composition hierarchy for every task.
- Gumbel Tree-LSTM learns task-specific tree structures directly from plain text without explicit structural guidance.
- A composition query vector scores candidate compositions, which the model recursively selects until one sentence representation remains.
- Natural language inference and sentiment analysis experiments show performance that outperforms or matches previous sentence encoders while converging significantly faster.
Related Work
Prior work learns hierarchical sentence structures either softly, discretely, or through reinforcement learning. These approaches trade interpretability, memory efficiency, or convergence speed in different ways.
- Soft-structure models recursively compose words into latent sentence hierarchies and can be trained through backpropagation.Examples include grConv and Neural Tree Indexer, which use gating or Tree-LSTM composition.
- Soft predicted structures can be ambiguous, while CYK Tree-LSTM reduces ambiguity by representing nodes as weighted sums of candidate compositions.
- CYK Tree-LSTM is memory intensive because the number of candidate compositions increases linearly with depth.
- A discrete recursive model greedily merges adjacent nodes with the smallest reconstruction error rather than optimizing composition directly on classification loss.
- REINFORCE can learn discrete tree composition by estimating gradients for classification loss, but its reinforcement-learning setting has slow convergence.
Model Description
The proposed architecture extends Tree-LSTM with components that dynamically compose sentence trees bottom-up and encode the resulting sentence into a vector. The section presents these components in detail.
- The architecture is built on a tree-structured long short-term memory network.
- Additional components dynamically compose the tree structure in a bottom-up manner.
- The model uses these components to encode a sentence into a vector.
Tree-LSTM
Tree-LSTM controls information flow from children to parent through cell states, while Gumbel Tree-LSTM learns task-specific compositions directly from unstructured sentences. It selects compositions recursively and uses ST Gumbel-Softmax to preserve discrete choices while enabling backpropagation.
- Tree-LSTM controls information flow from children to parent and uses cell states to capture distant vertical dependencies.
- Gumbel-Softmax enables gradients for discrete random variables by replacing arg max with a differentiable softmax approximation.
- ST Gumbel-Softmax discretizes the forward computation while using continuous values in the backward pass, allowing error signals to backpropagate.
- Gumbel Tree-LSTM initializes word representations, recursively composes adjacent nodes into parents, and copies unselected node representations to the next layer.
- A trainable composition query scores candidate parents, which are sampled during training and selected by maximum validity during validation or testing.
- Applying an LSTM to leaf nodes incorporates information about previous words and produces a substantial gain over the basic leaf transformation.
Experiments
The model is evaluated on natural language inference and sentiment analysis using plain text, without parse-tree information. Across both tasks, it is competitive with or better than prior models and converges faster.
- Evaluation setup: The proposed model is evaluated on natural language inference and sentiment analysis.SNLI experiments use premise and hypothesis representations from Gumbel Tree-LSTM, while SST experiments evaluate sentence classification.
- Natural Language Inference: The LSTM-based leaf transformation substantially improves SNLI performance and accelerates convergence relative to the affine transformation.The affine transformation bases parent selection only on local information, whereas the LSTM-based transformation addresses this limitation.
- Natural Language Inference: 86.0% accuracy is achieved by the 600D SNLI model, comparable to the state-of-the-art while using far fewer parameters.The 100D and 300D models also outperform models with similar parameter counts.
- Training efficiency: All models converge within a few hours, and convergence is faster than competing models in both total training time and iteration count.The comparison includes models trained with smaller batch sizes, where the proposed models still converge faster.
- Sentiment Analysis: The SST-2 model substantially outperforms other models, while the SST-5 model matches the state of the art without external resources beyond GloVe vectors.The state-of-the-art comparison uses a model pretrained on large parallel datasets and equipped with character n-gram embeddings.
- Representation analysis: Nearest-neighbor analysis shows that the model maps sentences with similar semantics—not only word overlap—to nearby vectors.The analysis uses cosine distance on SNLI test-set sentence representations and reports five neighbors per query.
Conclusion
Gumbel Tree-LSTM learns task-specific tree structures while preserving discrete computation paths during forward propagation. Experiments support its effectiveness against RvNNs and motivate extending the approach beyond sentence encoding.
- Conclusion: Gumbel Tree-LSTM computes candidate-parent validity scores and learns task-specific tree structures.
- Conclusion: ST Gumbel-Softmax enables standard backpropagation while keeping computation-path decisions discrete during forward propagation.
- Conclusion: The model outperforms all other RvNN models, remains competitive with state-of-the-art models, and converges faster than other complex models.
- Future work: The authors propose applying the core idea beyond sentence encoding and exploring recursive architectures for sentence generation.They also identify intra-sentence or inter-sentence attention as a possible improvement.
Supplementary Material for “Learning to Compose Task-Specific Tree Structures”
The supplementary material identifies the authors and their institutional affiliation at Seoul National University in Seoul, Korea.
- Supplementary Material: The paper lists Jihun Choi, Kang Min Yoo, and Sang-goo Lee as its authors.
- Supplementary Material: The authors are affiliated with Seoul National University, Seoul 08826, Korea.
Implementation Details
The implementation uses mask matrices to express recursive composition in a single equation. During training, discrete forward selections are paired with continuous backward representations from Gumbel-Softmax.
- Implementation Details: Multiple mask matrices rewrite the recursive composition represented by Eq. 11 as a single equation.
- Implementation Details: The mask definitions use cumulative sums of the continuous selection vector to construct left, right, and parent masks.
- Implementation Details: In the forward pass, the model uses a one-hot vector sampled from candidate validity scores with the Gumbel-Max trick.
- Implementation Details: The Gumbel noise is sampled using inverse-transform expressions with ϵ = 10^-20 added for numerical stability.
- Implementation Details: In the backward pass, a continuous Gumbel-Softmax vector replaces the one-hot representation, while forward-pass Gumbel noise is reused.
- Implementation Details: The implementation can represent the forward–backward discrepancy in automatic-differentiation libraries using detach(·), which blocks error propagation through its input.
Detailed Experimental Settings
The experiments used a public codebase and varied model dimensions, embeddings, optimization settings, and batch configurations across tasks and model sizes.
- Experimental setup: All experiments were conducted using the publicized codebase.
- Optimization: The general setup used Adam with default hyperparameters and halved the learning rate after one epoch without accuracy improvement.
- Model dimensions and embeddings: In 100D experiments, GloVe 6B 100D embeddings were fine-tuned during training.
- Model dimensions and embeddings: In 300D and 600D experiments, GloVe 840B 300D embeddings were fixed, with batch normalization and dropout applied around the MLP.
- SST settings: For SST, Adadelta was used, with learning-rate halving after two epochs without accuracy improvement.
- SST settings: SST-2 used Dc = 300 and batch size 32, whereas SST-5 used Dc = 1024 and mini-batches of 64 sentences.