Source-linked AI summary
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
Alon Talmor, Jonathan Herzig, Nicholas Lourie, Jonathan Berant
TL;DR
CommonsenseQA addresses the limited availability of scalable QA benchmarks requiring commonsense and background knowledge beyond an explicit context. It constructs questions from ConceptNet relations using crowd-authored discriminations among related concepts and evaluates models on the resulting dataset. The dataset contains 12,247 questions, with BERT-LARGE reaching 55.9% accuracy versus 88.9% for humans.
Problem
Existing QA benchmarks mostly provide contexts requiring little commonsense knowledge, while scalable commonsense benchmarks remain limited and difficult to construct without annotation artifacts.
Method
CommonsenseQA groups multiple ConceptNet target concepts sharing a relation to one source concept and asks crowd-workers to author discriminating multiple-choice questions with additional distractors.
Results
12,247 final examples were generated, and fine-tuned BERT-LARGE achieved 55.9% accuracy compared with 88.9% human accuracy.
Takeaways & Limitations
CommonsenseQA provides a large commonsense QA benchmark whose questions often require background knowledge that is trivial to humans but seldom explicitly reported on the web.
Takeaways & Limitations
Commonsense benchmarks remain difficult to design because models can exploit distributional biases or annotation artifacts instead of understanding the intended reasoning.
Abstract
from arXiv · showhide
When answering a question, people often draw upon their rich world knowledge in addition to the particular context. Recent work has focused primarily on answering questions given some relevant document or context, and required very little general background. To investigate question answering with prior knowledge, we present CommonsenseQA: a challenging new dataset for commonsense question answering. To capture common sense beyond associations, we extract from ConceptNet (Speer et al., 2017) multiple target concepts that have the same semantic relation to a single source concept. Crowd-workers are asked to author multiple-choice questions that mention the source concept and discriminate in turn between each of the target concepts. This encourages workers to create questions with complex semantics that often require prior knowledge. We create 12,247 questions through this procedure and demonstrate the difficulty of our task with a large number of strong baselines. Our best baseline is based on BERT-large (Devlin et al., 2018) and obtains 56% accuracy, well below human performance, which is 89%.
1 Introduction
CommonsenseQA introduces a large-scale commonsense question-answering dataset built by pairing ConceptNet relations with crowd-authored questions. Its questions require background knowledge beyond an explicit context, and current models remain well below human performance.
- Motivation: CommonsenseQA targets question answering that requires common sense and background knowledge beyond information explicitly provided in a document.The paper contrasts this setting with factoid QA, where answers can usually be found in a supplied context.
- Dataset construction: Crowd-workers author questions from a source concept and three same-relation target concepts, then add two further distractors, yielding five candidate answers per question.The construction is designed so that only one target concept answers each question while the other related concepts remain plausible distractors.
- Dataset and evaluation: Freely authored questions often require background knowledge that is trivial to humans but seldom explicitly reported on the web.The paper attributes this scarcity of explicit reporting to reporting bias.
- Dataset and evaluation: 12,247 commonsense questions were collected using the proposed generation method.The authors also analyze the questions’ commonsense skills and evaluate pretrained, fine-tuned, and reading-comprehension models.
- Dataset and evaluation: 55.9% accuracy is achieved by fine-tuned BERT-LARGE, compared with 88.9% human accuracy.The reported model result is substantially below human performance.
2 Related Work
Prior commonsense benchmarks include small, difficult datasets and newer large-scale event or visual reasoning tasks. The paper presents benchmark construction as challenging because models may exploit dataset biases or annotation artifacts instead of understanding common sense.
- Existing commonsense benchmarks: The Winograd Schema Challenge and COPA directly target commonsense reasoning but remain small, limiting scalability for modern modeling approaches.The cited collections contain 150 examples for Winograd and 500 development plus 500 test questions for COPA.
- Existing commonsense benchmarks: SWAG and related crowdsourced datasets address commonsense relations between situations or events, including predicting what happens next.The Story Cloze Test contrasts plausible and implausible story endings, while SWAG asks models to choose a subsequent event description.
- Benchmark challenges: Creating benchmarks that measure program understanding rather than distributional biases or annotation-process modeling remains difficult.The paper cites annotation artifacts in the Story Cloze Test that permit high performance without using the stories.
- Scope: Science-question datasets require specialized scientific knowledge, whereas this work studies common sense without requiring additional information.The related work also contrasts CommonsenseQA with the smaller hand-curated SQUABU benchmark.
3 Dataset Generation
COMMONSENSEQA is generated from CONCEPTNET by pairing a question concept with semantically related answer concepts, then using crowdsourcing, distractors, verification, and web snippets to create multiple-choice questions.
- The generation process starts by extracting CONCEPTNET subgraphs containing one source concept and three target concepts.
- 236,208 triplets remain after filtering general or well-explored relations, long or non-English concepts, and pairs with low edit distance.
- Semantically similar target concepts are grouped to discourage surface-based elimination and encourage questions requiring background knowledge.
- AMT workers formulate three questions per set, add two plausible distractors, and avoid words strongly associated with the answer concept.
- Each question is verified by two workers, and questions with at least one correct verification are retained.The verification process filters out 15% of questions.
- 12,247 final examples are generated from 16,242 formulated questions, with textual context retrieved from search-result snippets at a total cost of $0.33 per question.
4 Dataset Analysis
The dataset analysis examines the concepts, relations, language variation, and commonsense skills represented in COMMONSENSEQA, including manually annotated skill categories and examples.
- COMMONSENSEQA includes concepts such as dog, house, and row boat, with Person as its most frequent question concept at 3.1%.
- Questions generated from CONCEPTNET relations can probe different relationships, such as social conventions rather than the source relation itself.“What do audiences clap for?” was generated from AtLocation but focuses on social conventions.
- 122 formulators contributed questions, although 10 workers produced more than 85% of them.
- The sampled questions average 1.75 annotated commonsense skills, represented as labeled edges connecting answer, explicit, or latent concepts.
5 Baseline Models
The paper evaluates several baseline families for COMMONSENSEQA, varying whether they are trained on the dataset and whether they use additional context such as web snippets.
- Baseline models differ in whether they train on COMMONSENSEQA and whether they use extra context.The context distinction concerns input such as retrieved web snippets.
- VECSIM selects the answer whose average pre-trained embedding has the highest cosine similarity to the question.
- LM1B: LM1B uses a One Billion Words language model either by concatenating each answer with the question or rephrasing frequent prefixes as declarative sentences.The rephrasing variant clusters questions by their first two words and covers 35% of the development set with five high-frequency prefixes.
- QABILINEAR: QABILINEAR scores each candidate with a learned bilinear interaction between averaged question and answer embeddings.A softmax over candidate answers is trained with cross-entropy loss.
- QACOMPARE: QACOMPARE represents each question-answer pair using concatenation, element-wise product, and difference before predicting an answer score.
- ESIM and BIDAF++ provide stronger comparison baselines, with BIDAF++ additionally using retrieved Google web snippets as context.BIDAF++ augments BIDAF with self-attention and ELMo representations.
g GENERATIVE PRE-TRAINED TRANS-
The generative pre-trained language-model baselines encode each question and candidate answer as a delimiter-separated sequence and fine-tune pre-trained weights for multiple-choice prediction.
- GPT: GPT converts each question-answer pair into a delimiter-separated sequence and predicts candidate probabilities from the hidden representation of the end token.The resulting logits are passed through a linear transformation and softmax.
- BERT: BERT similarly linearizes each question-answer pair with special delimiters and fine-tunes uncased BERT-LARGE weights.BERT’s pre-training uses masked language modeling on unlabeled text.
6 Experiments
Experiments compare trained and untrained baselines, split strategies, human performance, model behavior across categories, and performance as training data increases.
- Experimental setup: The data use an 80/10/10 training/development/test split with either random or disjoint question-concept partitions.The evaluation uses accuracy, with hyperparameters tuned on development data.
- Human evaluation: 88.9% human accuracy was obtained from majority votes over five workers answering each of 100 sampled questions.
- Main results: 55.9% BERT-LARGE accuracy and 45.5% GPT accuracy were achieved on the random split, compared with 63.6% and 55.5% on the question-concept split.Random performance was five points lower on average across trained models than question-concept-split performance.
- Main results: 20% random accuracy was exceeded by the best baselines, but their performance remained well below human accuracy.
- Context analysis: Using web snippets with BIDAF++ did not lead to high performance, suggesting that the snippets contained little useful information.
- Sanity analysis: 92% BERT-LARGE accuracy in SANITY mode shows that replacing hard distractors with random ConceptNet distractors makes the task much easier.The result indicates that difficult distractor selection is crucial to benchmark difficulty.
- Baseline analysis: 77.7% accuracy occurred when surface clues hinted at the answer, while accuracy was 42.8% for negation or antonyms and 38.4% for factoid knowledge.Accuracy fell to 35.4% for finer-granularity cases and 23.8% when the correct answer required a conjunction of conditions.
- Learning curves: Roughly 75% BERT-LARGE accuracy is expected with 100k examples, still substantially below human performance.Each learning-curve point was selected as the best of three runs to address training instability.
7 Conclusion
The paper introduces COMMONSENSEQA, a 12,247-example commonsense QA dataset generated from ConceptNet, and finds that its best model remains far below human accuracy.
- COMMONSENSEQA contains 12,247 examples and is designed to test commonsense knowledge.
- The dataset is generated at scale using ConceptNet, analyzed for its distinctive properties, and evaluated with a broad suite of baselines.
- The best pre-trained language model tuned for the task obtains 55.9% accuracy, dozens of points below human accuracy.
- The authors hope the dataset facilitates future work on incorporating commonsense knowledge into NLU systems.