Source-linked AI summary
Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge
Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, Jonathan Berant
TL;DR
Pre-trained language models encode implicit knowledge, but their reasoning is difficult to control. This paper trains models to combine that knowledge with explicit statements, and finds they can compose separately taught skills beyond the training distribution, pointing toward interactive improvement through user-provided statements.
Problem
Pre-trained language models encode substantial implicit knowledge, but their reasoning abilities are not systematic and are difficult to control.
Source: §1 · p. 1Method
The authors automatically generate training examples pairing true-or-false hypotheses with explicit textual knowledge to teach defined inference types.
Source: §3 · p. 3Results
Takeaways & Limitations
A user-provided natural-language statement could correct an error immediately and, if retrievable later, also address future errors without retraining.
Source: §1 · p. 2User-supplied false facts require monitoring, and the authors advise against mission-critical use such as medical advice at current accuracy.
Source: Broader Impact · p. 9
Abstract
from arXiv · showhide
To what extent can a neural network systematically reason over symbolic facts? Evidence suggests that large pre-trained language models (LMs) acquire some reasoning capacity, but this ability is difficult to control. Recently, it has been shown that Transformer-based models succeed in consistent reasoning over explicit symbolic facts, under a "closed-world" assumption. However, in an open-domain setup, it is desirable to tap into the vast reservoir of implicit knowledge already encoded in the parameters of pre-trained LMs. In this work, we provide a first demonstration that LMs can be trained to reliably perform systematic reasoning combining both implicit, pre-trained knowledge and explicit natural language statements. To do this, we describe a procedure for automatically generating datasets that teach a model new reasoning skills, and demonstrate that models learn to effectively perform inference which involves implicit taxonomic and world knowledge, chaining and counting. Finally, we show that "teaching" models to reason generalizes beyond the training distribution: they successfully compose the usage of multiple reasoning skills in single examples. Our work paves a path towards open-domain systems that constantly improve by interacting with users who can instantly correct a model by adding simple natural language statements.
1 Introduction
Pre-trained language models encode substantial implicit knowledge, but their reasoning over it is not systematic or easy to control. This work trains models to combine that knowledge with explicit natural-language statements, including inferences that generalize beyond training examples.
1 Introduction
- Models can be trained to combine implicit taxonomic and world knowledge with explicit rules, perform chaining and counting, and succeed when the required prior knowledge is present.Inference fails when that knowledge is missing.Source: §1 · pp. 1–2
- For example, adding “A mammal has a belly button” lets the model combine that rule with its implicit knowledge that a whale is a mammal.The model initially answers the direct whale question incorrectly, but reaches the right conclusion after the rule is supplied.Source: §1 · p. 1, Figure 1 · p. 2
- Training inference skills independently generalizes to examples that require composing multiple skills absent from training.The authors propose gradually adding capabilities as a way to improve inference while retaining this generalization.Source: §1 · p. 2
- A user can correct an error with one natural-language statement, allowing the model to apply acquired inference skills without retraining.If the system retrieves the statement later, it may also correct future errors.Source: §1 · p. 2
2 A Motivating Example
A motivating experiment tests whether a model trained separately on implicit fact retrieval and explicit-statement reasoning can combine them in a new task. The model performs well on animal hypotheses requiring both kinds of knowledge, though the authors describe the training procedure as not yet systematic or controlled.
2 A Motivating Example
- Fine-tuning uses 50K TWENTY QUESTIONS examples for implicit real-world fact retrieval and 100K RULETAKER examples for reasoning over explicit natural-language assertions.The model is fine-tuned with standard multi-task training.Source: §2 · p. 2
- The figure tests substitutions of animal arguments in a hypothesis, with explicit knowledge specifying the desired size and excluding fish.The table shows predictions, while color and checkmarks indicate correct predictions.Source: Figure 2 · p. 2, §2 · p. 2
- 91.6% accuracy over 24 animal pairs shows the model combined implicit animal-size and taxonomy knowledge with explicit size and not-fish statements.The task asks whether Chen would like to buy an animal; the required combination was not observed during fine-tuning.Source: §2 · pp. 2–3
- The experiment suggests separate training can yield combined reasoning, but the authors call the procedure “alchemy” and ask how to make it more systematic and controlled.The remainder of the paper explores those questions.Source: §2 · p. 3
3 Method
The method defines inference types as logical patterns for combining natural-language statements, then builds training examples in which explicit statements alone do not determine a hypothesis. It generates examples from knowledge sources and fine-tunes a pre-trained language model for truth-value prediction.
3 Method
- The procedure automatically generates data from existing sources and then applies standard supervised training.The paper presents two instantiations of the procedure in Sections 4.1 and 4.3.Source: §3 · p. 3
- Each example pairs a true-or-false hypothesis with explicit facts or rules, constructed so that deciding its truth also requires knowledge already encoded in the language model.The explicit statements alone are insufficient; for example, the model may need implicit knowledge that a whale is a mammal.Source: §3 · p. 3
- Data generation samples true and false triples from ConceptNet, WordNet, and Wikidata, then uses predicate-specific templates to create statements.The generated examples randomize statement order and balance true and false hypotheses.Source: §3 · p. 3
- RoBERTa-Large receives explicit knowledge followed by a hypothesis and predicts its truth value using two logits and a softmax.Training minimizes binary cross-entropy, and evaluation uses accuracy.Source: §3 · p. 3
- The experiments also use ESIM over non-contextualized GloVe representations as a baseline for text-fragment inputs.The paper additionally tests whether the RULET.+20Q model can use implicit knowledge without direct training for that goal.Source: §3 · p. 3
4 Experiments
The experiments test whether models can combine implicit taxonomic and world knowledge with explicit statements, explain success through the model’s prior beliefs, and compose separately trained skills. Across taxonomy, counting, and mixed-skill evaluations, the results show both strong performance and boundaries tied to missing or unreliable knowledge.
4.1 Implicit Knowledge of Taxonomic Relations
- Generated taxonomy examples combine a hypernym and a property into a hypothesis, with distractors; removing relevant and distractor hypernyms in half the training examples teaches implicit-knowledge use.Subjects and objects in development and test examples are disjoint from training, and meronymy is tested zero-shot without a training set.Source: §4.1 · p. 4
- ROBERTA reaches 88.8% accuracy in IMPLICIT REASONING, versus 65.2% on HYPOTHESIS-ONLY, applying implicit hypernym knowledge to select a property rule.EXPLICIT REASONING is near perfect when the relevant hypernym and property are supplied.Source: Table 1 · p. 5, §4.1 · p. 5
4.2 Analyzing systematicity
- When required hypernym beliefs are correct, accuracy is 99.7%; adding rules for incorrect beliefs raises scores from 88.8 to 98.3, while IMAGINARY intervention raises overall accuracy from 76.9 to 94.1.The hypernym intervention supplied correct relevant and distractor rules for 17% of cases.Source: §4.2 · pp. 5–6, Table 2 · p. 6
4.3 Counting over Implicit Facts
- Counting sets pair one quantity fact with K member facts and vary how many member facts accompany positive and negative hypotheses, requiring explicit-plus-implicit facts except when k = K.The sets draw on relations such as band membership, capital, and parenthood.Source: §4.3 · p. 6
- ROBERTA reaches 99.7% accuracy when k = K; for partial counts, explicit knowledge improves accuracy from 64.1 to 73, but removing the quantity fact returns it to 64.4.The partial-count accuracy gain is 9 points; the quantity-fact ablation is close to the 64.1 hypothesis-only result.Source: Table 3 · p. 7, §4.3 · p. 7
- Adding lower-confidence member facts first improves performance by 5.5 points at c = 0.25 and 2.7 at c = 0.75; counting can nevertheless treat false member statements as facts.The false-statement test found the model predicted False in all such cases.Source: §4.3 · pp. 7–8
4.4 Generalizing to New Skill Combinations
- Models combine separately trained capabilities on mixed-skill examples; hypothesis-only performance averages 40.2, and omitting COUNTING training yields low accuracy on the counting subset.The MULTI-SKILL-SET examples were selected from hypotheses ROBERTA answered incorrectly without explicit knowledge.Source: §4.4 · p. 8
- On MULTI-SKILL-SET, training on all four datasets scores 85.4 accuracy; composed hypernymy+counting scores 88, and hypernymy+size and hypernymy+age/year score 81.8 and 73.9.The latter combinations were not present in training.Source: Table 4 · p. 8, §4.4 · p. 8
5 Related Work
This work extends systematic reasoning beyond self-contained tasks by harnessing implicit knowledge in an inference process. It contrasts this approach with opaque behavior in textual entailment and multi-hop QA, and proposes one-shot teaching from general statements.
5 Related Work
- Unlike textual entailment and multi-hop QA work, this approach aims for predictable reasoning rather than opaque hypothesis scoring or behavior.The cited examples are textual entailment [21] [22] [23] and multi-hop QA [26] [28] [29] [30].Source: §5 · p. 9
- The work extends systematic neural behavior from largely self-contained tasks to systematic inference that harnesses implicit knowledge.Earlier demonstrations included semantic parsing, symbolic integration, mathematics, knowledge prediction, and rule-based reasoning.Source: §5 · p. 9
- Pre-trained networks can be taught on-the-fly with a few general statements, without retraining the model.The paper characterizes this as one-shot teaching that exploits pre-trained knowledge.Source: §5 · p. 9
6 Discussion
The discussion argues that pre-trained LMs can systematically combine internal knowledge with explicit statements and compose independently added inference skills. It presents user interaction as a route to one-shot correction without data collection or retraining.
6 Discussion
- Pre-trained LMs can consistently combine implicit knowledge with explicit rules and facts across multi-hop reasoning, counting, number comparisons, and taxonomic knowledge.Inference capabilities can be injected independently, with generalization to examples combining multiple skills.Source: §6 · p. 9
- Users can provide natural-language facts and rules that the model uses immediately alongside its internal knowledge to correct current and future errors.The proposed one-shot interaction requires neither data collection nor retraining.Source: §6 · p. 9
Broader Impact
The proposed interactive approach could let users teach models and improve them without retraining, but it raises risks from false user-provided facts and model errors. The paper advises against relying on current state-of-the-art accuracy for mission-critical tasks such as medical advice.
Broader Impact
- False user-provided rules or facts may spread ungrounded information, so they must be monitored and curated.Source: Broader Impact · p. 9
- At current state-of-the-art accuracy, the paper advises against relying on these systems for mission-critical tasks such as medical advice.Source: Broader Impact · p. 9
- The proposed user-interaction approach could support one-shot model improvement without retraining, if successful.Source: Broader Impact · p. 9
7 Supplementary Material
The supplementary material details template-based pseudo-language and counting data, additional taxonomic controls, an analysis separating prior beliefs from reasoning, and tests of generalization to unseen skill combinations.
7.1 Method
- The supplementary method uses manually templated statements for ConceptNet relations and combines WordNet with ConceptNet for hypernymy and meronymy; counting quantities support up to five member facts.Source: §7.1 · p. 12
7.2 Implicit Knowledge of Taxonomic Relations
- In the taxonomic experiments, removing context slightly improves average results, while language-selectivity accuracy is 55%; the table covers hypernymy and meronymy reasoning.The 55% result suggests explicit knowledge reveals the answer for a small number of examples, while the full hypothesis is needed for the rest.Source: §7.2 · p. 12, Table 5 · p. 12
7.3 Analyzing systematicity
- The IMAGINARY analysis controls for prior beliefs about hypothesis truth; with disjoint hypernym trees, accuracy is 52.3% for HYPOTHESIS-ONLY, 99.0% for EXPLICIT REASONING, and 79.2% for IMPLICIT REASONING.The dataset uses neutral conclusions and mixes hypernym, meronym, and size-comparison rules with distractors.Source: §7.3 · pp. 12–13
7.4 Generalizing to New Skill Combinations
- On template-generated examples, adding explicit knowledge to the ALL COMBINED model raises accuracy by 56% on average, to 93.5% on average.The examples require combinations of skills not present during training, including implicit relations and explicit multi-hop reasoning.Source: Table 6 · p. 13, §7.4 · p. 13