Source-linked AI summary
Crowdsourcing Multiple Choice Science Questions
Johannes Welbl, Nelson F. Liu, Matt Gardner
TL;DR
Science exam QA lacks large in-domain training data, while generating diverse, relevant multiple-choice questions and plausible distractors is difficult. The paper uses domain-specific passages and model suggestions to guide crowd workers, producing SciQ and showing that its additional training data improves performance on real science exams.
Problem
Science exam QA requires integrated information extraction, reading comprehension, and common sense reasoning, but targeted-domain progress has lacked large in-domain training sets.
Method
The method presents workers with candidate passages for question writing and uses a classifier to suggest plausible distractors for converting questions into multiple-choice items.
Results
SciQ contains 13,679 multiple-choice science questions, and using it as additional training data improves performance on real 4th- and 8th-grade exam questions.
Takeaways & Limitations
SciQ is a useful research resource for studying neural models with medium-sized data and augmenting training for real science exam questions.
Takeaways & Limitations
Predicted distractors can be semantically misaligned, mismatched in specificity, unrelated, or ungrammatical.
Abstract
from arXiv · showhide
We present a novel method for obtaining high-quality, domain-targeted multiple choice questions from crowd workers. Generating these questions can be difficult without trading away originality, relevance or diversity in the answer options. Our method addresses these problems by leveraging a large corpus of domain-specific text and a small set of existing questions. It produces model suggestions for document selection and answer distractor choice which aid the human question generation process. With this method we have assembled SciQ, a dataset of 13.7K multiple choice science exam questions (Dataset available at http://allenai.org/data.html). We demonstrate that the method produces in-domain questions by providing an analysis of this new dataset and by showing that humans cannot distinguish the crowdsourced questions from original questions. When using SciQ as additional training data to existing questions, we observe accuracy improvements on real science exams.
1 Introduction
The paper addresses the lack of large, in-domain science QA training sets by using crowdsourced question generation supported by domain-specific passages and model suggestions. This process produces SciQ, a 13,679-question dataset whose additional training data improves performance on real science exam questions.
- Broad-domain dataset progress has not translated into comparable improvements on targeted science exam QA.
- Science exam QA requires integrating information extraction, reading comprehension, and common sense reasoning.The highest-scoring systems in an 8th-grade science competition achieved only 60% on a multiple-choice test.
- Large in-domain training sets are difficult to create because crowd workers lack domain expertise and distractors can make questions trivial.Questions may also lack relevance and diversity in structure and content.
- The method uses candidate passages for question writing, then model-predicted distractors to support workers in producing multiple-choice questions.It shifts the human task toward selection, modification, and validation rather than purely generative writing.
- $10,415 produced 13,679 multiple-choice science questions in the SciQ dataset.Most instances include the document used to formulate the question, and each instance has four answer options.
- SciQ improves performance on real 4th- and 8th-grade exam questions when used as additional training data.On SciQ's multiple-choice version, neural models do not beat simple information-retrieval baselines.
2 Related Work
Prior work constructed datasets from broad textual sources and developed multiple approaches to science exam QA and automatic question generation. The paper positions its dataset as a response to limited training data and shortcomings in earlier question-generation methods.
- Dataset Construction: Earlier QA datasets drew on sources including Freebase, Wikipedia, web search queries, news, children’s books, novels, and image captions.
- Dataset Construction: The paper extends this dataset-construction line to science exam QA with natural-language questions.
- Science Exam Question Answering: Science exam QA research has used coherence evaluation, entailment, probabilistic reasoning, proof chains, ensemble methods, lexical co-occurrence, and information retrieval.
- Science Exam Question Answering: Limited training data has constrained neural-network applications to science exam QA, motivating construction of a larger dataset.
- Automatic Question Generation: Earlier automatic question-generation methods used syntactic templates or cloze-style questions, but produced low-quality science questions.Problems included selecting relevant text, generating reasonable distractors, and formulating coherent questions.
- Automatic Question Generation: Answer-distractor selection has used similarity measures from WordNet, thesauri, distributional context, ontologies, phonetic or morphological similarity, probability scores, and lexical inference.
3 Creating a science exam QA dataset
The paper constructs science exam questions through a two-step crowdsourcing process that uses filtered textbook passages and model-suggested distractors. Human workers select relevant source material, formulate questions, and validate or revise candidate answer options to produce SciQ questions.
- Method overview: The method first presents workers with candidate textbook passages, then has another worker generate three distractors for each question-answer pair with model assistance.The resulting multiple-choice item contains a question, passage, correct answer, and incorrect answer options.
- Base corpus: Science study textbooks were chosen as the base corpus because they target school-level topics and student-oriented language.The corpus comprised 28 openly licensed books from online learning resources.
- Document filtering: A rule-based filter removes unsuitable sentences using lexical, grammatical, pragmatic, and complexity criteria before workers see candidate passages.The filter accepts paragraphs when a minimum number of their sentences passes the rules.
- Question formulation: Workers receive three textbook paragraphs and may choose one or reject all, reducing the risk of irrelevant questions.All three documents were rejected in 12.1% of cases, fewer than expected when only one document is presented.
- Distractor criteria: Distractors are designed to remain grammatically, categorically, and semantically consistent with the question and correct answer.Nonsensical distractors make questions trivial, whereas plausible false answers preserve the intended reasoning challenge.
- Distractor generation: The distractor model ranks candidate expressions by estimating whether they are plausible false answers from question, correct-answer, and candidate features.It is trained on observed false answers from in-domain questions as positive examples and random expressions as negative examples.
- Distractor evaluation: 94.2% validation accuracy was achieved by the distractor model, but its predictions still required additional filtering by crowd workers.Errors included semantic-type mismatches, incorrect specificity, and unrelated or ungrammatical multiword expressions.
- Crowd validation: 36.1% of final-dataset distractors were generated by the model, while workers retained responsibility for validating and supplementing the suggestions.Workers could select at most two predicted distractors and had to write at least one themselves; acceptance was higher for short answers.
4 SciQ Experiments
SciQ is evaluated in multiple-choice and direct-answer settings, including transfer to real science exams. The evaluations show useful performance and improved results when SciQ augments real exam training data.
- Multiple Choice Setting: The evaluation compares Aristo components and neural reading-comprehension models on SciQ’s multiple-choice task.The systems include Lucene, TableILP, the Aristo ensemble, the Attention Sum Reader, and the Gated Attention Reader.
- Multiple Choice Setting: Neural models do not outperform a simple information-retrieval baseline when given the same background information.The authors interpret this result as evidence that SciQ is useful for studying reading comprehension in medium-data settings.
- Direct Answer Setting: 66.7% exact match and 75.7 F1 are achieved by BiDAF on SciQ’s direct-answer setting.These scores are 1.3% and 1.6% below BiDAF’s performance on SQuAD, respectively.
- Using SciQ to answer exam questions: Adding SciQ improves AS Reader and GA Reader performance on both 4th- and 8th-grade science exam questions.The authors report that some improvements are substantial and contrast this with earlier purely synthetic data that reduced overall performance.
5 Conclusion
The paper concludes that its crowdsourcing methodology yields a 13.7K-question science dataset for research and transfer to real science exams. It identifies multitask learning, improved automation, and broader distractor applications as future directions.
- Conclusion: 13.7K science questions constitute SciQ, a dataset released for future research.The dataset is presented as a resource for studying neural models in medium-sized data settings and augmenting training for real science exam questions.
- Conclusion: Baseline evaluations show that SciQ is useful for investigating neural model performance and augmenting training data for real science exams.The conclusion identifies both medium-data model analysis and exam-question training as supported uses.
- Future Work: Future work includes systematic multitask exploration, adapting distractor generation to negative sampling, and further bootstrapping toward fully automatic question generation.The proposed automation targets document selection, question generation, and distractor prediction.
A List of Study Books
The study-book corpus draws from open educational resources covering biology, chemistry, earth science, physics, anatomy, and related introductory material. The listed sources span elementary through college introductory levels.
- Study-book sources: The data sources include OpenStax, CK-12, Saylor Foundation, and other educational textbooks.The list contains named works in biology, chemistry, earth science, physics, anatomy, and life or physical science.
- Coverage: The listed materials span elementary, middle-school, high-school, and college introductory science levels.Examples include grade 4 science testing material, middle-school concepts, high-school earth science, and college physics or chemistry.
- Exclusions: Documents from two listed resources were not included in the dataset.The exclusions are stated in the notes associated with the source list.
- Licensing: CK-12 books were obtained under the Creative Commons Attribution-Non-Commercial 3.0 Unported license.The license is stated alongside the CK-12 source list.
B Training and Implementation Details
The implementation uses early stopping and prior-work hyperparameters for the reading-comprehension baselines. AS Reader, GA Reader, and BiDAF are implemented with the specified embedding, gating, and training configurations.
- Training procedure: Training is stopped when validation error increases, with patience of one and a hard limit of ten epochs.Most models reached peak validation accuracy after the first or second epoch, and test evaluation used the peak-validation parameters.
- AS Reader: The AS Reader uses 256-dimensional embeddings and a 384-dimensional GRU hidden layer.The remaining AS Reader hyperparameters were taken from prior work.
- GA Reader: The GA Reader uses three gated-attention layers with multiplicative gating and fixed pretrained 100-dimensional GloVe embeddings.Character-level embeddings and question-evidence common-word features are not used.
- Direct Answer Reading Comprehension: The direct-answer baseline implements BiDAF as described in prior work and adopts its reported hyperparameters.The implementation targets the direct-answer version of SciQ.