Source-linked AI summary
A Simple Method for Commonsense Reasoning
Trieu H. Trinh, Quoc V. Le
TL;DR
Commonsense reasoning remains difficult for neural networks when labeled data is scarce. The paper uses language models trained on unlabeled text to score candidate substitutions, outperforming prior systems on both evaluated challenges while discovering decisive contextual features.
Problem
Commonsense reasoning tasks such as Winograd Schema provide no training data or very little labeled data, challenging supervised neural methods.
Method
The method substitutes candidate references into pronouns and uses language models trained on unlabeled data to score the resulting sentences.
Results
The method outperforms previous state-of-the-art systems on both tasks, achieving 70.0% accuracy on PDP-60 and 63.7% on WSC-273.
Takeaways & Limitations
The system discovers contextual features that decide many answers, and diverse training corpora improve ensemble performance.
Abstract
from arXiv · showhide
Commonsense reasoning is a long-standing challenge for deep learning. For example, it is difficult to use neural networks to tackle the Winograd Schema dataset (Levesque et al., 2011). In this paper, we present a simple method for commonsense reasoning with neural networks, using unsupervised learning. Key to our method is the use of language models, trained on a massive amount of unlabled data, to score multiple choice questions posed by commonsense reasoning tests. On both Pronoun Disambiguation and Winograd Schema challenges, our models outperform previous state-of-the-art methods by a large margin, without using expensive annotated knowledge bases or hand-engineered features. We train an array of large RNN language models that operate at word or character level on LM-1-Billion, CommonCrawl, SQuAD, Gutenberg Books, and a customized corpus for this task and show that diversity of training data plays an important role in test performance. Further analysis also shows that our system successfully discovers important features of the context that decide the correct answer, indicating a good grasp of commonsense knowledge.
1 Introduction
The paper addresses commonsense reasoning when labeled data is scarce by scoring candidate answers with language models trained on unlabeled data. It reports large gains on Pronoun Disambiguation and Winograd Schema challenges and analyzes how the system makes decisions.
- Commonsense reasoning is difficult for neural networks because these tasks typically provide no training data or very little labeled data.
- The method substitutes each candidate reference into the pronoun position and uses language models to score the resulting sentences.
- 70.0% accuracy on PDP-60 exceeds the previous state-of-the-art result of 66.7%.
- 63.7% accuracy on WSC-273 is 11% above the current state-of-the-art result of 52.8%.
- The system often discovers special words that determine the correct reference, suggesting that its decisions use important contextual features.
2 Related Work
Prior commonsense-reasoning systems commonly combine supervision, annotated knowledge bases, search, semantic parsing, or engineered features. This paper instead builds on the evidence that language models trained on massive unlabeled text can capture contextual and commonsense information.
- The paper applies the intuition that language modeling can capture commonsense knowledge while using more contextual information than word vectors.
- Language models have improved downstream NLP applications as pretrained sentence or paragraph representations.
- Earlier Winograd Schema systems used annotated knowledge bases, rule-based reasoning, hand-crafted features, or additional supervised data.
- Some prior systems queried Google Search and used semantic parsers or graphs to reason about the questions.
3 Methods
The method reframes pronoun resolution as comparing language-model probabilities for sentences formed by substituting candidate references. It includes full and partial scoring strategies and analyzes contextual evidence word by word.
- Pronoun resolution becomes identifying which candidate substitution produces the more probable sentence under a language model.
- The method’s analysis examines probability ratios across word positions to identify contextual contributors and tests whether changing a decisive word switches the answer.
- The model substitutes each candidate into the pronoun position before scoring the resulting sentence.
- Full scoring evaluates the probability of the entire substituted sentence, whereas partial scoring evaluates the continuation conditioned on the substituted context.
- Experiments find that partial scoring is generally better than the naive full-scoring strategy.
4 Experimental settings
Experiments evaluate the method on Pronoun Disambiguation and Winograd Schema tasks using recurrent language models with word- or character-level inputs. Training data spans several large and specialized text corpora.
- The experiments evaluate Pronoun Disambiguation Problems and the Winograd Schema Challenge as commonsense-reasoning tasks.
- PDP-60 contains 60 original questions, while WSC-273 contains 273 questions and is described as qualitatively more difficult.
- The recurrent language models process either word inputs or character inputs while producing word-output distributions.
- Word-level and character-level models use distinct embedding and processing architectures.
- Training corpora include LM-1-Billion, CommonCrawl, SQuAD, and Gutenberg Books.
5 Main results
The experiments show that unsupervised language-model resolvers perform strongly on PDP-60 and WSC-273, with customized training data further improving Winograd performance.
- PDP-60: Unsupervised single-model language-model resolvers outperform previous PDP-60 results by a large margin.The models use character-level and word-level language models trained on Gutenberg Books; full scoring performs better for this task.
- PDP-60: 70.0% accuracy on PDP-60 exceeds the previous state-of-the-art result of 66.7%.
- WSC-273: 61.5% accuracy on WSC-273 is nearly 10% above the previous best result.The result comes from adding four language models trained on LM-1-Billion, CommonCrawl, SQuAD, and Gutenberg Books to an earlier ensemble.
- WSC-273: WSC-273 is more difficult than PDP-60, and adding supervised learning and annotated knowledge bases provides only a +3% gain on WSC-273.The corresponding gain on PDP-60 is reported as +19%.
- Customized training data: A STORIES corpus built from the top 0.1% of CommonCrawl documents improves the final WSC-273 system by 2% accuracy.The selected documents have high n-gram overlap with task questions and commonly contain long chains of coherent events.
- Customized training data: 62.6% accuracy is achieved by a single word-level language model trained on STORIES, exceeding the earlier 10-model ensemble's 61.5%.
6 Analysis
The analysis shows that partial scoring helps language-model resolvers focus on decisive later words, while diverse training corpora improve ensemble accuracy.
- 6.1 Discovery of special words in Winograd Schema: The ratio q_t identifies words most responsible for choosing between the correct and incorrect candidates.Selecting positions with the largest q_t values provides a way to retrieve likely decision keywords.
- 6.1 Discovery of special words in Winograd Schema: Large q_t values often coincide with Winograd Schema special keywords that decide the correct reference.The analysis interprets these values as low probability assigned to the keyword after the wrong substitution.
- 6.1 Discovery of special words in Winograd Schema: 115 out of 178 correctly answered questions had their special keywords discovered using forward and backward scoring.Backward-scoring models address cases where the keyword appears before the reference and achieve 58.2% accuracy on WSC-273.
- 6.2 Partial scoring is better than full scoring.: Full scoring often fails because the candidate at the pronoun position is rare in the language model’s training corpus.In the trophy example, the rare candidate receives low probability early enough to overpower later evidence.
- 6.2 Partial scoring is better than full scoring.: More than 62.7% of wrong predictions from full scoring were corrected by partial scoring on WSC-273.Score normalization fixed full scoring in 9 out of 10 tested language models on PDP-122, while partial scoring strongly outperformed both alternatives on WSC-273.
- 6.3 Importance of training corpus: The diverse-corpus ensemble outperformed ensembles trained on every single corpus, although STORIES alone produced the highest single-corpus accuracy.The tested corpora were LM-1-Billion, CommonCrawl, SQuAD, Gutenberg Books, and STORIES.
7 Conclusion
The paper presents an unsupervised commonsense-reasoning method based on large language models trained on diverse corpora. It reports higher accuracy than previous best systems and identifies decisive question features.
- The paper introduces a simple unsupervised method for commonsense reasoning using large language models trained on massive, diverse text corpora.
- The resulting systems outperform previous best systems on both Pronoun Disambiguation Problems and the Winograd Schema Challenge.
- The system discovers key features deciding the correct answer, and diverse training corpora benefit model ensembles most.
- The paper anticipates that the technique will serve as a building block for future commonsense-reasoning systems.
A Recurrent language models
The recurrent language-model systems use large LSTM architectures with word- or character-level inputs, extensive vocabularies, and multiple training variants.
- The base recurrent model uses two LSTM layers with 8192 hidden units, peephole output gates, a 1024-dimensional projection, and 0.25 dropout.
- Word inputs use 800,000-word embeddings of dimension 1024, while character inputs use 256-character embeddings of dimension 16.
- Character inputs are padded to shape (50, 16) and processed by eight one-dimensional convolutional layers.
- The output layer applies Softmax over an 800K-word vocabulary, with importance sampling using 8192 negative samples during training.
- The experiments include recurrent language-model variants and record each model’s training corpus in the reported LM tables.
B Data contamination in CommonCrawl
The CommonCrawl-based training data contains substantial low-quality text, creating a scope concern for using the full corpus in commonsense reasoning.
- A large amount of low-quality CommonCrawl text consists of documents that are unintelligible or unrecognized by the vocabulary.
- Training commonsense-reasoning language models on full CommonCrawl might therefore not be ideal.
- The data-cleaning process also removed some PDP-122 questions that appeared as extremely highly ranked documents.