Source-linked AI summary
A Decomposable Attention Model for Natural Language Inference
Ankur P. Parikh, Oscar Täckström, Dipanjan Das, Jakob Uszkoreit
TL;DR
Natural language inference requires determining whether a hypothesis is entailed by, contradicts, or is neutral toward a premise, while prior neural approaches can be expensive and parameter-heavy. The paper uses attention to align local substructures, decompose inference into separate comparisons, and merge them for classification. On SNLI, it reports state-of-the-art results with nearly an order of magnitude fewer parameters, with further improvement from minimal intra-sentence order information.
Problem
Prior neural NLI approaches can be computationally expensive and use models with millions of parameters, motivating a simpler alternative.
Method
The model uses neural attention to soft-align sentence elements, decompose inference into aligned subphrase comparisons, and optionally encode within-sentence composition.
Results
State-of-the-art SNLI results are achieved with almost an order of magnitude fewer parameters than complex LSTM-based approaches.
Takeaways & Limitations
The results suggest that pairwise comparisons can be relatively more important than global sentence-level representations for NLI.
Takeaways & Limitations
The reported error analysis identifies missed critical words, number inference, and sequence information as cases all methods get wrong.
Abstract
from arXiv · showhide
We propose a simple neural architecture for natural language inference. Our approach uses attention to decompose the problem into subproblems that can be solved separately, thus making it trivially parallelizable. On the Stanford Natural Language Inference (SNLI) dataset, we obtain state-of-the-art results with almost an order of magnitude fewer parameters than previous work and without relying on any word-order information. Adding intra-sentence attention that takes a minimum amount of order into account yields further improvements.
1 Introduction
The paper introduces a lightweight attention-based NLI model that aligns local text substructures, decomposes inference into separately solved subproblems, and achieves strong SNLI performance with fewer parameters. It targets the computational cost and complexity of prior neural approaches while largely avoiding word-order information.
- 570K sentence pairs in the SNLI corpus provide the comparison setting for the proposed natural language inference model.NLI determines entailment and contradiction relationships between a premise and a hypothesis.
- Prior neural text-similarity models often use computationally expensive architectures with millions of parameters.
- The model aligns local text substructures and aggregates their comparisons instead of constructing a compact representation of an entire complex sentence.The paper illustrates this by aligning corresponding words or phrases and recognizing synonymy or incompatibility.
- Attention creates a soft alignment, decomposes inference into separately solved subproblems, and merges their results into the final classification.Optional intra-sentence attention enriches substructure representations before alignment.
- The approach achieves state-of-the-art SNLI results with almost an order of magnitude fewer parameters than complex LSTM-based approaches.It performs the same total work asymptotically as a vanilla LSTM encoder while remaining trivially parallelizable across sentence length.
2 Related Work
The paper situates its method within prior alignment-based work in translation, sentence similarity, NLI, and semantic parsing. It uses attention as a neural alignment mechanism without relying on recurrent encoders.
- The method is motivated by alignment's central role in machine translation, sentence similarity modeling, natural language inference, and semantic parsing.
- Unlike attention methods commonly paired with LSTMs or sometimes CNNs, this approach uses attention based purely on word embeddings.Its feed-forward networks operate largely independently of word order.
3 Approach
The model processes two sentences through jointly trained Attend, Compare, and Aggregate components. Attention creates aligned subphrases for separate comparison, whose results are aggregated and classified; optional intra-sentence attention enriches the input representation with limited order information.
- The input consists of two sentences represented as word-embedding sequences, with each sentence prepended by a NULL token.
- Input representation: In the vanilla model, the input representation is the original word embeddings, so the model does not use word order.
- Attend: Attend soft-aligns elements across the two sentences and decomposes inference into comparisons of aligned subphrases.
- Compare: Compare applies a feed-forward function separately to each aligned pair, combining each element with its softly aligned subphrase.
- Aggregate: Aggregate sums the comparison vectors for both sentences and feeds the results to a final classifier that selects the highest-scoring class.
- Intra-Sentence Attention: Optional intra-sentence attention creates self-aligned phrases before cross-sentence alignment, adding limited sequence information while remaining parallelizable.Distance-sensitive biases are bucketed so distances greater than 10 words share one bias.
4 Computational Complexity
The model has the same asymptotic complexity as an LSTM with attention and, under the assumption that sentence length is smaller than embedding dimension, matches a regular LSTM while allowing parallelization over sentence length.
- Attend step: Each attention weight eij adds O(ℓ2d) computation through one dot product.The Attend step evaluates F O(ℓ) times, contributing O(ℓd2), and computes attention weights for sentence pairs.
- Complexity: O(ℓd2 + ℓ2d) is the model’s total complexity, equal to that of an LSTM with attention.The Compare and Aggregate steps contribute O(ℓd2) and O(d2), while Attend contributes O(ℓd2 + ℓ2d).
- Complexity: When ℓ < d, the model’s complexity becomes O(ℓd2), matching a regular LSTM.The analysis assumes sentence length ℓ is smaller than embedding dimension d; this holds for SNLI, where ℓ < 80 and recent LSTM approaches used d ≥300.
- Parallelism: Unlike an LSTM, the model is parallelizable over ℓ, which can be useful at test time.This parallelism follows from the model’s non-recurrent computation across sentence length.
5 Experiments
The model is evaluated on SNLI using standard preprocessing and training choices, achieving strong accuracy with substantially fewer parameters. Error analyses show strengths on lexical alignment and neutral examples, but weaknesses when inference requires sequence, numerical, or critical-word information.
- Data preprocessing: 549,367 pairs remain for training, with 9,842 for development and 9,824 for testing after removing unlabeled examples.The data use tokenized non-binary parses, a prepended NULL token, masked batch padding, and length-based semi-sorting.
- Implementation details: 300-dimensional GloVe embeddings are normalized, projected to 200 dimensions, and kept fixed during training.Out-of-vocabulary words are assigned to one of 100 random embeddings, while the projection matrix is trained.
- Results: The vanilla model achieves state-of-the-art 3-class accuracy with almost an order of magnitude fewer parameters than LSTMN.Adding intra-sentence attention improves performance by 0.5 percentage points over the existing state of the art; most gains come from neutral examples, while most losses involve contradiction pairs.
- Error analysis: Both model variants correctly handle several cases that SPINN-PI and mLSTM classify incorrectly, including examples favoring neutral despite lexical overlap.The analysis also suggests word order may confuse competing models in an example involving different grammatical roles for “fountain.”
- Error analysis: Intra-sentence attention helps on some cases that do not require word order, possibly because contextual composition resolves a “play/watch” contradiction.The authors attribute this to embeddings being insufficiently fine-grained and intra-attention adding composition and nonlinearity.
- Error analysis: All methods fail on examples requiring a critical-word distinction, numerical inference, or sequence information.The examples include “close” versus “open,” numerical reasoning, and sequential information.
6 Conclusion
The paper presents a simple attention-based approach to natural language inference that is trivially parallelizable. It outperforms considerably more complex neural methods, suggesting that pairwise comparisons can matter more than global sentence representations for this task.
- Conclusion: The proposed attention-based approach to natural language inference is trivially parallelizable.The conclusion characterizes the model as simple and attention-based.
- Conclusion: The approach outperforms considerably more complex neural methods aimed at text understanding.This conclusion is consistent with the reported SNLI results.
- Conclusion: The results suggest that pairwise comparisons are relatively more important than global sentence-level representations for natural language inference.This is presented as a task-specific implication of the reported findings.