Source-linked AI summary
WorldTree: A Corpus of Explanation Graphs for Elementary Science Questions supporting Multi-Hop Inference
Peter A. Jansen, Elizabeth Wainwright, Steven Marmorstein, Clayton T. Morrison
TL;DR
Question-answering models often lack gold explanations for training, limiting work on human-readable justifications that matter in domains such as science and medicine. The paper constructs explanation graphs and a supporting tablestore for elementary science questions. It reports predictable corpus relationships, including a strong correlation between tablestore growth and an exponential proportional decrease.
Problem
Question-answering models generally lack human-readable explanations, and training explainable inference is hindered by the lack of gold explanations and the challenge of aggregating separate knowledge pieces.
Source: §1 · p. 1Method
The paper constructs explanations for elementary science questions as lexically connected sentence graphs, alongside semi-structured tables containing knowledge used to build them.
Source: §1 · p. 2, §4.2 · p. 4, §4.3 · p. 5Results
The number of unique tablestore rows needed as the corpus grows has a strong correlation (R=0.99) with an exponential proportional decrease.
Source: §5.3 · p. 7, §6 · p. 7Takeaways & Limitations
The corpus provides resources for developing explainable inference methods for elementary science questions.
Source: §6 · p. 7Knowledge estimates reflect exams as they currently exist and may not cover the many possible example variations that use the same core knowledge.
Source: §5.3 · p. 7
Abstract
from arXiv · showhide
Developing methods of automated inference that are able to provide users with compelling human-readable justifications for why the answer to a question is correct is critical for domains such as science and medicine, where user trust and detecting costly errors are limiting factors to adoption. One of the central barriers to training question answering models on explainable inference tasks is the lack of gold explanations to serve as training data. In this paper we present a corpus of explanations for standardized science exams, a recent challenge task for question answering. We manually construct a corpus of detailed explanations for nearly all publicly available standardized elementary science question (approximately 1,680 3rd through 5th grade questions) and represent these as "explanation graphs" -- sets of lexically overlapping sentences that describe how to arrive at the correct answer to a question through a combination of domain and world knowledge. We also provide an explanation-centered tablestore, a collection of semi-structured tables that contain the knowledge to construct these elementary science explanations. Together, these two knowledge resources map out a substantial portion of the knowledge required for answering and explaining elementary science exams, and provide both structured and free-text training data for the explainable inference task.
1. Introduction
Question-answering systems often lack human-readable justifications, despite their importance for trust and error detection in science and medicine. This paper addresses the shortage of gold explanations with a corpus of elementary science explanations and an explanation-centered tablestore.
1. Introduction
- Human-readable explanations matter in science and medicine for user trust and detecting potentially costly errors, and may support human-like generalization and inference.Source: §1 · p. 1
- Explaining answers requires aggregating separate knowledge pieces, with the authors estimating about six pieces per question while inference methods often struggle beyond two.Source: §1 · p. 1
- The corpus covers 1,680 standardized elementary science questions with free-text explanations and lexically connected explanation graphs that make links among sentences explicit.The graphs are intended as training data for inference models.Source: §1 · p. 2
- The explanation-centered tablestore contains 62 semi-structured tables and 4,950 rows covering substantial knowledge for non-spatial, non-mathematical elementary science questions.Source: §1 · p. 2
- The authors analyze corpus knowledge growth and explanation overlap to identify inference requirements and estimate the difficulty of constructing explanation corpora in other domains.Source: §1 · p. 2
2. Related Work
Prior work explores explanation generation, multi-piece knowledge aggregation, and semi-structured table stores for question answering. Elementary science presents a challenging, relatively low-resource setting for these approaches.
2. Related Work
- Jansen et al. (2017) generated high-quality explanations for 60% of elementary science questions, 15% more than a single-passage baseline, but many successful graphs were not selected.Their system ranked candidate explanation graphs using answer correctness and graph-connectivity features.Source: §2 · p. 2
- Table-store QA commonly chains rows to connect question terms with answers, using tables collected from the web, extracted from text, or manually constructed.Source: §2 · p. 2
- Prior table-store work spans large-scale extraction and relational reasoning, including Sun et al. (2016) and Yin et al. (2015).Sun et al. extracted 104 million tables; Yin et al. created a 120M-relation table using OpenIE.Source: §2 · pp. 2–3
- Elementary science QA has limited grade-appropriate resources, and Khashabi et al. (2016) reported 61% performance on 129 questions using a table-store system.Dalvi et al. (2016) introduced an interactive tool that improved knowledge-generation speed by up to four times over manual methods.Source: §2 · p. 3
- Information aggregation remains difficult: elementary science explanations average four knowledge pieces, while systems can suffer semantic drift when combining more than two.Fried et al. (2015) and Jansen et al. (2016) report these aggregation challenges.Source: §2 · p. 3
3. Design Goals
The design goals are explanations that are computable, sufficiently deep, and reusable. The authors target explanations between the young-child and first-principles levels, while noting the practical burden of going further.
3. Design Goals
- The design goals call for explicit lexical links among explanation sentences, enabling graph representations, and reuse of knowledge across explanations.Source: §3 · p. 3
- Explanations should include enough domain and world knowledge for an answer to be reached with little extra knowledge, ideally targeting a five-year-old's level or lower.Source: §3 · p. 3
3.1. Explanation Depth
- Explanatory depth increases as more implicit knowledge is made explicit; levels are additive, and the corpus targets between young-child and first-principles depth.For example, a novice may need the added link that thicker things provide more protection to understand why thick bark helps a tree survive fire.Source: §3.1 · pp. 3–4, Table 1 · p. 4
- Explanations near first principles may be useful for training but are laborious to author, so protocols for producing them are left to future work.Source: §3.1 · p. 4
4. Explanation Authoring
The section describes how elementary science explanations are represented with table-backed sentences and lexical links, then authored and reviewed through a collaborative tool and annotation process.
4.1. Questions
- The corpus began with 2,201 elementary science questions and retained 1,680 after excluding questions whose specialized knowledge did not fit the explanation formalism.The questions were multiple-choice items for grades 3–5, and diagram-interpretation questions were excluded.Source: §4.1 · p. 4
4.2. Tables and Table Rows
- Each explanation sentence is a single-topic relation statement represented by a row in one of 62 semi-structured tables, with explanations averaging 6.3 rows.Tables cover relations such as taxonomy, causality, and changes, as well as topic-specific properties.Source: §4.2 · p. 4
- Tables use fine-grained columns for relation roles and filler columns that render each row as a stand-alone natural-language sentence.The tables have 2–16 content columns, compared with 2–5 in the Ariso tablestore (Khashabi et al., 2016).Source: §4.2 · p. 4
4.3. Explanation Graphs and Sentence Roles
- An explanation graph links every sentence lexically to the question, answer, or another explanation sentence.Sentence roles are annotated as central, grounding, background, or lexical glue; lexical glue links concepts to satisfy the explicit-link requirement.Source: §4.3 · p. 5
4.4. Annotation Tool
- The authoring tool lets annotators search the tablestore and prior explanations, assemble a graph, and inspect lexical overlap to spot content gaps.It also displays the question and answer candidates and provides flags for annotation and quality review.Source: §4.4 · p. 5, Figure 3 · p. 6
4.5. Procedure and Explanation Review
- Annotators identify a question’s central concept and required inference, then build its graph by querying the tablestore and reusing previously annotated explanations.When needed knowledge is absent, annotators add it to an appropriate table before adding it to the explanation.Source: §4.5 · pp. 5–6
- Because explanations reference unique table-row identifiers, tables and rows can be reorganized without rewriting existing explanations.The authors found this flexibility critical for consistency and organization during corpus construction.Source: §4.5 · p. 6
- Four review passes address consistency and completeness, with a second annotator checking each explanation before the original annotator applies revisions.Review takes about one third of annotation time; most explanations take 5–10 minutes to construct and about 5 more minutes to review, after roughly 60 hours of initial training.Source: §4.5 · p. 6
5. Explanation Corpus Properties
This section characterizes how table knowledge is reused, how explanations overlap across questions, and how the explanation tablestore grows with corpus size.
5.1. Knowledge Use and Row Frequency
- Table rows recur in 2.9 explanations on average; 1,535 rows are reused, 531 appear at least five times, and “an animal is a kind of organism” appears in 89 explanations.Each StatesOfMatter row is used in 15.7 explanations on average.Source: §5.1 · p. 6, Table 3 · p. 7
5.2. Explanation Overlap
- The tablestore contains 4,950 rows, of which 3,686 are actively used in at least one explanation; per question, averages are 17, 9, and 5 other questions with 1+, 2+, and 3+ overlapping explanation rows, respectively.Source: §5.2 · pp. 6–7
- Overlap proportions rise logarithmically; for 80% coverage, one shared row suggests 500 explanations, two require approximately 2,500, and three or more likely require at least 20,000.If the relationship transfers beyond elementary science, the authors suggest estimating corpus-size requirements after authoring a few hundred explanations.Source: §5.2 · p. 7
5.3. Explanation Tablestore Growth
- For this elementary science corpus, the tablestore asymptotes at approximately 6,000 unique rows by 10,000 questions, estimating the domain’s upper-bound knowledge and question counts.The simulated row-growth relationship is strongly correlated (R=0.99) with an exponential proportional decrease.Source: §5.3 · p. 7
6. Conclusion
The paper provides an explanation-graph corpus for elementary science questions and reports predictable patterns in knowledge frequency, explanation overlap, and tablestore growth.
6. Conclusion
- The explanation-graph corpus is intended to support research on explainable inference methods for elementary science questions.The authors report that knowledge frequency, explanation overlap, and tablestore growth follow predictable relationships.Source: §6 · p. 7
- The corpus and its generation tools are open source.Source: §6 · p. 7