Source-linked AI summary
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
Todor Mihaylov, Peter Clark, Tushar Khot, Ashish Sabharwal
TL;DR
Existing QA benchmarks often provide self-contained context, while OpenBookQA targets science questions requiring retrieval and multi-hop combination of an open-book fact with common knowledge. The paper constructs and evaluates this dataset, finding a large gap between humans and existing systems, while oracle experiments show that both evidence sources improve performance without eliminating the reasoning challenge.
Problem
OpenBookQA addresses the limited evaluation of QA systems on questions requiring partial provided context, external common knowledge, and multi-hop reasoning.
Method
The paper constructs about 6000 elementary science questions from an open book of core facts using crowdsourcing and automatic, crowd, and expert filtering, then evaluates existing and neural QA baselines.
Results
92% human performance contrasts with existing systems near the 25% random baseline, while oracle access to both the open-book and additional facts reaches 76% accuracy.
Takeaways & Limitations
OpenBookQA provides a challenge for retrieval and reasoning systems because open-book facts help but remain insufficient without additional knowledge and non-trivial fact combination.
Takeaways & Limitations
The paper leaves formal analysis of dependence across questions, caused by workers answering multiple questions, for future work.
Abstract
from arXiv · showhide
We present a new kind of question answering dataset, OpenBookQA, modeled after open book exams for assessing human understanding of a subject. The open book that comes with our questions is a set of 1329 elementary level science facts. Roughly 6000 questions probe an understanding of these facts and their application to novel situations. This requires combining an open book fact (e.g., metals conduct electricity) with broad common knowledge (e.g., a suit of armor is made of metal) obtained from other sources. While existing QA datasets over documents or knowledge bases, being generally self-contained, focus on linguistic understanding, OpenBookQA probes a deeper understanding of both the topic---in the context of common knowledge---and the language it is expressed in. Human performance on OpenBookQA is close to 92%, but many state-of-the-art pre-trained QA methods perform surprisingly poorly, worse than several simple neural baselines we develop. Our oracle experiments designed to circumvent the knowledge retrieval bottleneck demonstrate the value of both the open book and additional facts. We leave it as a challenge to solve the retrieval problem in this multi-hop setting and to close the large gap to human performance.
1 Introduction
OpenBookQA is designed as an open-book science QA task requiring systems to retrieve core facts and combine them with common knowledge for multi-hop reasoning. Existing QA systems perform far below humans, while oracle evidence shows that both open-book facts and additional knowledge help but do not close the gap.
- Dataset and task: OpenBookQA contains 5957 multiple-choice questions and 1326 elementary science facts, with the facts generally insufficient on their own.Systems must retrieve a relevant fact from F and apply it using their own common knowledge.
- Dataset and task: Questions require combining a core science fact with simple common knowledge, such as connecting thermal conductivity, steel, and heat transfer.The difficulty lies in identifying and meaningfully combining the supporting knowledge with the fact from F.
- Dataset and task: OpenBookQA differs from self-contained reading-comprehension and knowledge-base QA because its open book provides partial rather than complete context.Successful systems must go beyond paraphrase matching and coreference resolution while lacking the canonicalized completeness of knowledge bases.
- Dataset construction: The question-generation pipeline combines crowdsourcing with automatic, crowd, and expert filters to create novel, answerable, and higher-quality questions.The automatic filter targets hardness for retrieval- and association-based systems, while expert filtering is applied to Dev and Test sets.
- Results: 92% human performance contrasts with existing science QA systems performing near the 25% random-guessing baseline.The evaluated pre-trained systems were used without retraining.
- Results: 76% accuracy is reached when oracle experiments provide both the open-book fact and needed additional knowledge, but this remains below human performance.The open-book facts alone provide a 5% improvement but are not sufficient, indicating that combining the facts still requires non-trivial reasoning.
2 Related Work
OpenBookQA is positioned among QA datasets by combining structured elementary science facts with external common knowledge rather than relying on self-contained documents or unconstrained science questions. Its distinguishing focus is a well-defined multi-hop setting centered on one core fact and a small number of supporting facts.
- Task distinctions: OpenBookQA requires base science facts, broader common knowledge, and reasoning to combine them.This setup differs from tasks that can be solved using only a document or structured knowledge source.
- Reading comprehension: Many reading-comprehension datasets are self-contained and can often be solved through context token matching, even when they use multiple-choice, cloze, or span formats.Multi-step RC datasets address reasoning across sentences, but their input passages remain self-contained in the cited examples.
- Tasks with external knowledge: Story Cloze, MCScript, and ProPara require additional domain knowledge, but they model domain-specific events, scripts, or processes rather than OpenBookQA’s science-fact setting.The cited comparison distinguishes OpenBookQA from these tasks by the type of external knowledge required.
- Multi-hop question generation: Other multi-hop question-generation work uses structured knowledge such as a Biology ontology, whereas OpenBookQA starts from an open book of science facts.The contrast concerns the source and structure of the knowledge used to generate questions.
- Science QA: Science QA datasets may require broad external knowledge, but OpenBookQA additionally associates questions with a core fact set and narrows the expected supporting knowledge.The paper describes this as one core fact plus one or a few relatively simple commonly known supporting facts.
3 OpenBookQA Dataset
OpenBookQA is a 4-way science question dataset built from a small core book of facts, with questions generated by combining those facts with common knowledge and filtered for answerability. Its analysis estimates human performance and shows that questions typically require simple but non-trivial multi-step reasoning.
- Dataset composition: The core book F contains 1326 general elementary-science facts selected from WorldTree facts judged central to scientific explanations.The facts were further filtered for applicability across multiple situations.
- Human performance: The human-accuracy estimate averages independent worker judgments, with Hoeffding-based bounds used to quantify uncertainty.The analysis assumes answer samples are independent, although workers may answer multiple questions.
- Human performance: The estimated human performance is reported conservatively using a bound within 3% of the empirical estimate with probability over 98.8%.A tighter 2.5% bound holds with probability 95.6%, but the former is reported as the conservative estimate.
- Dataset composition: OpenBookQA contains 5957 questions, each with four answer choices and one associated core science fact, divided into Train/Dev/Test splits of 4957/500/500.The dataset also includes an auxiliary set of roughly 6000 additional facts.
- Question set analysis: Most questions require simple additional facts, averaging 1.16 facts per question, but baseline systems still score relatively low even when the core fact is provided.The authors attribute this difficulty to chaining facts and identifying the relations needed for the answer.
4 Baseline Models
The paper evaluates retrieval-based, pretrained, and trained neural baselines for multiple-choice OpenBookQA, including models that exploit answer-choice or question biases without external knowledge.
- No Training, External Knowledge Only: Existing science QA solvers rely on background knowledge and retrieval mechanisms that do not incorporate the core fact set F.These design choices limit their effectiveness on OpenBookQA despite their suitability for multiple-choice science questions.
- No Training, External Knowledge Only: TableILP connects questions to answers through support graphs over semi-structured knowledge tables, but its tables lack knowledge for 24% of OpenBookQA questions.TupleInference instead retrieves text, extracts subject-verb-object tuples, and combines evidence from multiple tuples with integer linear programming.
- No Training, External Knowledge Only: DGEM converts each question-choice pair into a hypothesis and retrieved facts into premises, then selects the choice with the highest entailment score.The IR and TupleInference solvers are also evaluated with the provided core facts F.
- Trained Models, No Knowledge: The trained baselines encode each question and answer choice independently with a bidirectional LSTM and model multiple-choice QA as four-class classification.Token embeddings, contextual representations, and max aggregation form the shared representation pipeline.
- Trained Models, No Knowledge: The plausible answer detector ignores the question and scores each choice for plausibility, while the odd-one-out solver selects the choice least similar to the other options.These models target answer-choice biases such as implausible distractors or homogeneous incorrect options.
- Trained Models, No Knowledge: The question-match solver selects the answer choice with the highest attention score against the question without relying on external knowledge.The paper also tests token-wise interaction representations for this question-choice matching model.
- Trained Model with External Knowledge: The knowledge-enhanced reader independently encodes questions, choices, and retrieved facts, then attends to the external knowledge before scoring answers.It extends the question-match baseline with representations constructed from question-choice and fact-context combinations.
5 Baseline Performance
Baseline results show a large gap between human performance and most automated systems, while simple neural biases and oracle knowledge substantially improve accuracy. These findings highlight both dataset artifacts and the importance of retrieving relevant external facts.
- 92% human performance indicates that OpenBookQA is largely solvable by laypeople.Additional annotations yielded 88.6%, 90.2%, and 91.6% on Train, Dev, and Test samples, respectively.
- Pre-trained state-of-the-art solvers for multiple-choice science questions perform poorly on OpenBookQA.Their background knowledge and retrieval mechanisms are closely related to an information-retrieval solver designed to fail on this task.
- Adding the open-book facts improves TupleInference by 8.7% but does not change DGEM.TupleInference relies on brittle word-overlap similarity, whereas DGEM does not show the same benefit.
- 49.6% accuracy is achieved by a plausible-answer detector without question text, while odd-one-out and question-match solvers reach 50.2%.These results expose exploitable answer-choice and dataset biases in knowledge-free neural baselines.
- The information-retrieval baseline is guaranteed to obtain 0% by design on OpenBookQA.This design choice makes comparisons involving added facts appear artificially magnified.
- Knowledge-enhanced models reach about 76% when given the oracle core fact and the question author’s interpretation of an additional fact.This remains below the 92% human level, while retrieved WordNet knowledge gives about a 0.5% gain and ConceptNet can reduce performance.
6 Conclusion
OpenBookQA is a roughly 6000-question benchmark requiring systems to combine provided science facts with external common knowledge through multi-hop reasoning. Simple neural methods reach about 50%, far below the 92% human performance, while oracle experiments indicate potential for improved retrieval and reasoning.
- OpenBookQA contains about 6000 questions testing the combination of provided science facts with broad external common knowledge.The task requires multi-hop reasoning that combines the two sources.
- About 50% accuracy is achieved by simple neural methods, compared with 92% human performance.The paper leaves closing this performance gap for future research.
- Oracle-style experiments demonstrate the potential value of better knowledge retrieval and reasoning.
A Knowledge Retrieval Module
The knowledge retrieval module selects potentially relevant external facts for each question and answer-choice set, after which a neural network predicts the answer. Retrieval combines the open-book facts with common-knowledge sources using similarity-based selection.
- The two-stage module first retrieves a fixed-size subset of potentially relevant facts, then predicts the answer with a neural network.The retrieval input includes the question, answer choices, and an external knowledge source.
- TF-IDF vector representations are built for questions, answer choices, and external facts.Similarity scores are calculated between each fact and both the question and answer choices.
- Cosine-distance similarity selects the top N_k facts associated with the question and each answer choice.N_k is chosen from {5, 10, 20} using Dev-set performance.
- The retrieval experiments combine the open-book facts F with ConceptNet’s Open Mind Common Sense and its WordNet subset.
B Implementation and Training
The models are implemented with AllenNLP and PyTorch, trained with cross-entropy and Adam, and selected using Dev-set performance. Results are averaged over five random seeds with corresponding test-score variability reported.
- The neural models use AllenNLP and PyTorch with cross-entropy loss and the Adam optimizer.The initial learning rate is 0.001.
- Training uses early stopping and learning-rate halving based on Dev-set improvement.Models without external knowledge typically train for up to 30 epochs, while knowledge-augmented models typically train for 60 epochs with greater patience.
- TF-IDF vectorization is based on scikit-learn’s term-frequency, inverse-document-frequency representation.
- Most neural models use an LSTM hidden size of 128, with embedding dropout selected from 0.1, 0.2, and 0.5 using Dev performance.
- Five random-seed experiments are run per configuration, reporting average best-Dev accuracy and corresponding Test accuracy with standard deviation.
- The model code and configuration files are publicly available for reproducing the results.
C Additional Experiments
Additional experiments evaluate models on ARC and SciTail, extending assessment beyond OpenBookQA to related science reasoning and entailment tasks.
- ARC Challenge: 33.87% accuracy was achieved on the ARC Challenge Test set by the selected Question Match system.The system used the best-performing configuration selected from multiple LSTM hidden sizes and embedding-dropout settings.
- SciTail: The Question Match system was adapted to SciTail by replacing question–choice inputs with premise–hypothesis inputs for binary entailment classification.The adapted model used BiLSTM encoders with shared parameters between the premise and hypothesis.
- Qualitative analysis: The paper also examines examples of questions answered correctly or incorrectly by different model groups.These examples are organized around the first three questions in each case.
D.1 Neural Baseline Successes
Neural models without external knowledge succeed on some questions by exploiting semantic relationships between question words and answer choices rather than deeper reasoning.
- Interpretation: These examples show that neural models can answer some questions through lexical or semantic similarity without using external knowledge.The success pattern is described for Question Match, Plausible Answer, One-Odd-Out, and ESIM.
- Shared successes: 172/500 questions were predicted correctly by all trained neural models without external knowledge.Table 5 summarizes examples from this jointly successful subset.
- Semantic cues: The correct answer often contains a word semantically closer to an important question word than the words in competing choices.Examples include “pores” with “body,” “non-renewable” with “gone,” and “iron” with “magma.”
D.2 Neural Baseline Failures, Oracle Success
Oracle access to the relevant facts helps on questions that defeat neural models without knowledge, especially when solving requires chaining multiple facts.
- Oracle successes: 405/500 questions were predicted correctly by the f + k Oracle model, including 69 missed by all four neural models without knowledge.These questions require chaining multiple facts, for which simple semantic similarity is insufficient.
D.3 Neural Baseline and Oracle Failures
Both knowledge-free models and the f + k Oracle model fail on subsets of OpenBookQA for distinct reasons, including domain shifts, aggregate answer formats, noisy facts, and difficult reasoning.
- Shared failures: 42/500 questions were answered incorrectly by every model, including the f + k Oracle model.The Oracle model nevertheless assigned confidence above 0.9 to each of these incorrect predictions.
- Oracle limitations: The Oracle model can fail when the core fact is unhelpful, the second fact is noisy or incomplete, or combining the facts exceeds its reasoning ability.These are described as broad reasons for failure despite access to oracle information.
- Domain shift: Questions outside the predominant science domain indicate weak performance on distant domains even when core facts are provided.One example transforms a science fact about measuring data into a law-related question.
- Aggregate answers: An “all of these” option requires checking whether every answer is correct rather than selecting the most likely individual answer.The model did not apply this aggregate-answer interpretation in the example.
- Aggregate answers: An “all but xyz” option requires selecting a special aggregate answer even when related Oracle facts point to a specific answer.This creates a mismatch between the answer format and the retrieved factual support.