Source-linked AI summary
Can recursive neural tensor networks learn logical reasoning?
Samuel R. Bowman
TL;DR
The paper asks whether learned word representations can support linguistic interpretation and reasoning, an ability that had received little study. It trains a recursive model on constructed inference examples and finds perfect accuracy on familiar patterns, with generalization to many unseen patterns but not all.
Problem
Although learned word representations had succeeded on increasingly sophisticated NLP tasks, their ability to capture meaning needed for interpretation and reasoning remained little studied.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The results suggest that the model’s structure is basically capable of learning meaning representations that support inference.
Source: §6 · p. 8The target pattern (some x) bark ∧ (no x) bark performs consistently poorly across training settings.
Source: §5 · pp. 7–8
Abstract
from arXiv · showhide
Recursive neural network models and their accompanying vector representations for words have seen success in an array of increasingly semantically sophisticated tasks, but almost nothing is known about their ability to accurately capture the aspects of linguistic meaning that are necessary for interpretation or reasoning. To evaluate this, I train a recursive model on a new corpus of constructed examples of logical reasoning in short sentences, like the inference of "some animal walks" from "some dog walks" or "some cat walks," given that dogs and cats are animals. This model learns representations that generalize well to new types of reasoning pattern in all but a few cases, a result which is promising for the ability of learned representation models to capture logical reasoning.
1 Introduction
The paper asks whether learned word representations can support interpretation and reasoning, beyond lexical similarity. It isolates inference with constructed, unambiguous examples and reports perfect learning of familiar patterns and generalization to many unseen ones.
1 Introduction
- Learned word vectors have succeeded on increasingly sophisticated NLP tasks, but their ability to represent meaning for interpretation and reasoning remains little studied.The introduction contrasts lexical similarity, such as Paris and France, with reasoning about asymmetric relations such as whether going to Paris entails going to France.Source: §1 · p. 1
- The study uses hand-constructed, unambiguous inference problems to isolate inference from failures in other aspects of sentence understanding.The introduction notes that coreference, word-sense disambiguation, or parsing errors could otherwise supply inaccurate input.Source: §1 · p. 2
- The model learns familiar inference patterns with perfect accuracy and generalizes successfully to unseen patterns in many cases.The introduction describes the results as largely positive, while noting that generalization is not universal.Source: §1 · p. 2
2 Reasoning with monotonicity
The paper uses monotonicity to define inference patterns involving quantifiers, then evaluates them with strict unambiguous NLI because unrestricted datasets can confound inference with other language-understanding problems. It situates this approach alongside natural logic and prior inference systems.
2 Reasoning with monotonicity
- Monotonicity specifies which substitutions in a semantic position preserve truth; quantifiers differ in their licensed directions, and most is not monotone in its first argument.For example, all is downward monotone in its first argument and upward in its second, while some and two are upward in both.Source: §2 · p. 2
- Natural logic treats natural-language strings as logical symbols, and its theories define the inference types included in the experiments rather than being implemented directly.The study focuses on quantifiers such as some and all because they license many monotonicity patterns and are difficult to interpret through ordinary vector-similarity behavior.Source: §2 · p. 2
2.1 The task: natural language inference
- Existing unrestricted NLI datasets can confound inference evaluation with lexical or syntactic ambiguity, coreference, and pragmatic enrichment.The paper argues that a purpose-built dataset is needed to test specific inference behaviors precisely.Source: §2.1 · p. 3
- Strict unambiguous NLI counts only entailments licensed by literal sentence interpretations and constrains language to minimize ambiguity.The task also excludes generic noun readings, context-dependent elements, pronouns, tense, copulas, and agreement marking.Source: §2.1 · p. 3
- The paper presents learned-vector inference as distinct from prior systems, including MacCartney’s engine, which uses supplied projectivity signatures and linguistic knowledge.Earlier vector-based work either used distributional vectors not learned for the task or proposed augmenting a deterministic engine without published evaluations.Source: §2.1 · pp. 3–4
3 Methods
The method recursively composes each phrase into a vector, compares the two vectors, and classifies their relation. The paper also tests model variations and reports no consistent improvement from added layers or alternative composition functions.
3 Methods
- Each phrase is recursively composed into a single vector, and a separate comparison layer produces relation features for a seven-way softmax classifier.The two phrase trees use the same composition function before their resulting vectors enter the comparison layer.Source: §3 · p. 4
- Using a leaky rectified linear function for the comparison function substantially improved performance over a sigmoid.The function is applied elementwise as max(x, 0) + 0.01 min(x, 0).Source: §3 · p. 4
- The model mirrors each phrase’s provided tree structure, passing word vectors upward through composition and comparison functions into the classifier.Training uses backpropagation through structure and the negative log probability assigned to the correct label as its cost.Source: §3 · pp. 4–5
- The model uses stochastic gradient descent with minibatches of 32 and AdaGrad from a starting learning rate of 0.2.L2 regularization uses λ = 0.0002; L-BFGS and L1 regularization showed no improvement.Source: §3 · p. 5
- Additional neural-network layers yielded no consistent measurable improvement, and a variant with different phrase-specific composition functions did not improve performance.The added layers were placed between the comparison layer and the softmax classifier.Source: §3 · p. 5
4 Data
The dataset comprises short, labeled sentence pairs organized into constrained reasoning-pattern datasets. It varies lexical predicates, quantifiers, and negation to cover distinct forms of monotonicity inference.
4 Data
- Table 2 presents sample datasets from four different classes.Source: Table 2 · p. 6
- The dataset contains 12,000 labeled short-sentence pairs divided into about 200 datasets, each varying at most one lexical item within a specific valid reasoning pattern.Pairs within each dataset share the same relation label.Source: §4 · p. 6
- Basic monotonicity datasets vary predicates in corresponding positions, with examples covering first-argument, second-argument, and combined substitutions.Some combined substitutions make argument entailments proceed in the same direction, while others proceed in opposite directions.Source: §4 · p. 6
- Quantifier-substitution datasets pair different quantifiers while keeping predicates the same, and combined datasets vary both quantifiers and arguments.The combined datasets sample combinations because exhaustive coverage would be too large to verify manually; they include every applicable relation except equivalence.Source: §4 · p. 6
- Negation datasets systematically negate one of four argument positions in variants of the other dataset patterns.Source: §4 · p. 6
5 Experiments and results
The model learned the reasoning patterns in the ALL-SPLIT test with perfect accuracy, but transfer to held-out patterns varied, with one target remaining consistently difficult. Experiments compared increasingly broad training-data exclusions and evaluated accuracy across combined, target, and held-out test sets.
Experiments and results
- 100% test accuracy in ALL-SPLIT shows the model learned the reasoning patterns represented in those datasets.The setting randomly assigned 85% of each dataset to training and 15% to testing.Source: §5 · p. 6
- SET-OUT, SUBCLASS-OUT, and PAIR-OUT progressively withheld the target dataset, its reasoning-pattern class, or all examples involving its quantifier pair.Table 3 illustrates training inclusion for a target dataset, with a check marking datasets for which 85% of examples were seen in training.Source: §5 · p. 7, Table 3 · p. 7
- Table 4 reports accuracy on combined test data, the target dataset, and held-out datasets; for SET-OUT, the target and held-out results coincide.The combined test data includes test portions of split datasets as well as target and other held-out datasets.Source: §5 · p. 7, Table 4 · p. 7
- Some held-out datasets had perfect accuracy in SET-OUT and SUBCLASS-OUT and near-perfect accuracy in PAIR-OUT, while the target (some x) bark ∧ (no x) bark remained poor.The target's poor performance was consistent across random initializations and was identified for further investigation.Source: §5 · pp. 7–8
6 Discussion
The discussion argues that the model learns representations supporting inference and often generalizes beyond training patterns, though performance is inconsistent. It relates a difficult quantifier-pair case to what can be inferred from the training data and leaves the minimum information needed for generalization unresolved.
Discussion
- Generalization to novel data succeeds in most, but not all, training configurations; the mixed results suggest optimization could improve, while supporting the model's basic capacity to learn inference-supporting representations.The author argues that success in some configurations suggests the model structure is basically capable of learning meaning representations that support inference.Source: §6 · p. 8
- ALL-SPLIT results show the model can learn sentence-pair distinctions and lexical relations from examples, while other settings show quantifier representations can support inference when a quantifier substitution was unseen.The author presents this as evidence of learned underlying logic rather than memorized fixed patterns.Source: §6 · p. 8
- For some–no quantifier pairs, the author suggests poor performance may reflect that training data supports only a weaker relation, not the strict negation relation required by the target.The model predicted the weaker relation for the strict-negation target in all SET- and SUBCLASS-OUT cases and 84% of PAIR-OUT cases.Source: §6 · p. 8
- The minimum information needed for general-purpose quantifier representations remains unknown; proposed tests include longer constructions, smaller training sets, and more structured train/test configurations.The discussion also proposes formal study of projectivity and its underlying logic to clarify theoretical bounds.Source: §6 · p. 8
Appendix A: The syntactically untied model
The syntactically untied variant uses separate composition parameter sets for different phrase-composition roles instead of one universal function. Its RNTN layers retain the same basic structure, including a sigmoid nonlinearity.
Appendix A: The syntactically untied model
- Following Socher et al., the variant uses three separately parameterized composition functions instead of a single universal function.The paper describes this as a potentially more powerful model variant.Source: Appendix A · p. 10
- Each composition layer keeps the same basic RNTN structure, including the sigmoid nonlinearity, while selecting among three parameter sets.The sets are selected according to the composition role.Source: Appendix A · p. 10
- The three roles are composing negation with a predicate, a quantifier with its first argument, and a quantifier phrase with its second argument.Examples include not + dog, all + dog, and all dog + bark.Source: Appendix A · p. 10
Appendix B: Wordlist
The wordlist supplies vocabulary for constructed inference examples, while limiting explicit lexical relations so the model must learn them indirectly from monotonicity examples.
Appendix B: Wordlist
- The wordlist includes category terms and contrasts such as animal, dog, and cat; able and unable; and dead and live, alongside predicates, quantifiers, and modifiers.Other entries include bark, European, sofa, and indestructible.Source: Appendix B · p. 10
- Seven relation types hold between some word pairs, but most are absent from the experiment data, so lexical knowledge enters indirectly through monotonicity examples.For example, seeing (some dog) bark ⊏ (some animal) bark could allow the model to infer dog ⊏ animal.Source: Appendix B · p. 10
- The author avoids specifying most possible word relations to limit background knowledge that could let the model judge sentence truth and complicate learning.For instance, training on all dog bark ⊏ all hungry bark could teach dog ⊏ hungry and affect later examples.Source: Appendix B · p. 10