Source-linked AI summary
QuaRel: A Dataset and Models for Answering Questions about Qualitative Relationships
Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, Ashish Sabharwal
TL;DR
Qualitative story questions are challenging because corpus-based methods perform poorly on novel scenarios, while mapping natural language into qualitative models is difficult. The paper introduces a framework, a dataset with logical forms, and two adapted semantic-parsing models; QuaSP+ outperforms off-the-shelf tools, and QuaSP+Zero handles new relations without additional training data.
Problem
Corpus-based methods perform poorly on qualitative questions about novel scenarios, and mapping natural-language questions into qualitative models is a formidable parsing challenge.
Source: §1 · p. 1, §7.1 · p. 7Method
The paper represents questions as comparisons between two worlds, builds a dataset with logical forms, and extends type-constrained semantic parsing with two models.
Source: §1 · pp. 1–2, Predicates · p. 3Results
Takeaways & Limitations
The work makes inroads into answering complex qualitative questions that require reasoning and scaling to new relationships at low cost.
Source: §1 · p. 2The approach does not handle context-dependent mapping of quantitative property values well, and some questions fall outside its two templates or require extra commonsense inference.
Source: §8 · p. 8
Abstract
from arXiv · showhide
Many natural language questions require recognizing and reasoning with qualitative relationships (e.g., in science, economics, and medicine), but are challenging to answer with corpus-based methods. Qualitative modeling provides tools that support such reasoning, but the semantic parsing task of mapping questions into those models has formidable challenges. We present QuaRel, a dataset of diverse story questions involving qualitative relationships that characterize these challenges, and techniques that begin to address them. The dataset has 2771 questions relating 19 different types of quantities. For example, "Jenny observes that the robot vacuum cleaner moves slower on the living room carpet than on the bedroom carpet. Which carpet has more friction?" We contribute (1) a simple and flexible conceptual framework for representing these kinds of questions; (2) the QuaRel dataset, including logical forms, exemplifying the parsing challenges; and (3) two novel models for this task, built as extensions of type-constrained semantic parsing. The first of these models (called QuaSP+) significantly outperforms off-the-shelf tools on QuaRel. The second (QuaSP+Zero) demonstrates zero-shot capability, i.e., the ability to handle new qualitative relationships without requiring additional training data, something not possible with previous models. This work thus makes inroads into answering complex, qualitative questions that require reasoning, and scaling to new relationships at low cost. The dataset and models are available at http://data.allenai.org/quarel.
1 Introduction
The paper targets story questions that require reasoning about qualitative relationships, a setting where corpus-based methods perform poorly and small wording changes can reverse answers. It introduces a representation, dataset, and models for parsing these questions and reasoning over them.
1 Introduction
- Corpus-based methods perform poorly on novel scenarios, and changing a word such as “more” to “less” can flip an answer.Answering requires knowledge of underlying qualitative relations, such as how friction relates to speed.Source: §1 · p. 1
- Qualitative models express relationships without numeric equations, but mapping natural-language questions into them remains a formidable semantic-parsing challenge.The paper cites prior qualitative-modeling work (Forbus 1984; Weld and De Kleer 2013; Kuipers 1994).Source: §1 · p. 1
- The paper contributes a framework for comparing two scenarios, the QUAREL dataset with logical forms, and two models extending type-constrained semantic parsing.The framework represents questions as comparisons between two worlds.Source: §1 · pp. 1–2
- QuaSP+ significantly outperforms off-the-shelf tools, while QuaSP+Zero handles new qualitative relationships without additional training data.The zero-shot capability was not possible with previous models.Source: §1 · p. 2
2 Related Work
Prior QA and semantic-parsing work spans many tasks, but qualitative relationships and the language and reasoning challenges they pose have received less focused attention. QuaRel situates its contribution among qualitative reasoning, language interpretation, story problems, and crowdsourced logical forms.
2 Related Work
- Although QA has advanced across many phenomena, QuaRel specifically targets qualitative relationships so their challenges can be studied.The paper describes it as the first dataset specifically targeted at qualitative relationships.Source: §2 · p. 2
- QuaRel adds diverse open-domain language and reasoning over entities without explicit formal representations to semantic-parsing datasets.The paper adapts existing semantic parsers to address these phenomena.Source: §2 · p. 2
- Prior work connecting language with qualitative reasoning mainly extracts qualitative models from text, whereas QuaRel interprets story questions involving qualitative comparisons.The paper draws on qualitative-reasoning work to design a constrained logical-form language (Forbus 1984; Weld and De Kleer 2013; Kuipers 1994).Source: §2 · p. 2
- Unlike arithmetic story problems, QuaRel questions lack readily identifiable entities such as numbers, adding a challenge to mapping them into formal representations.The paper compares its task with arithmetic story-problem work (Ling et al. 2017; Kushman et al. 2014; Wang, Liu, and Shi 2017; Shi et al. 2015).Source: §2 · p. 2
- QuaRel shares two-way choice and coreference with Winograd, but tests qualitative-relation reasoning; its logical forms are elicited indirectly without teaching workers the formalism.The paper notes that crowdsourcing target logical forms is difficult and relates its approach to Yih et al. (2016).Source: §2 · p. 2
3 Knowledge Representation
The representation maps questions into compact logical forms that describe two compared worlds, then applies qualitative background knowledge and inference rules to identify an answer. It uses qualitative properties and values rather than requiring numerical equations.
3 Knowledge Representation
- The framework represents questions and the knowledge needed to answer them in a language whose logical forms are included in the dataset.It defines properties and qualitative values as primitives for representing domain knowledge.Source: §3 · p. 2, §3.1 · p. 2
3.1 Qualitative Background Knowledge
- Qualitative theories encode proportional or inverse property relationships and ordered values; the paper simplifies values to high and low and notes that mapping varied language remains the main challenge.The formalism can be extended with other qualitative-modeling primitives when the theory can compute an answer from the supplied values or relations.Source: §3.1 · pp. 2–3
Predicates.
- Questions describe two compared situations—different entities or one entity at different times—and predicates encode relative or absolute property values in each world.The world tags abstract away irrelevant scenario details while requiring the parser to associate them with the phrases describing each situation.Source: Predicates · p. 3
3.3 Logical Forms for Questions
- A question’s logical form contains a setup and two answer predicates; templates cover questions using relative comparisons or absolute property values.An answer is selected when the reasoner infers that its predicate follows from the setup.Source: §3.3 · p. 3
3.4 Inference
- A Prolog-style inference engine applies qualitative rules to a question’s logical form and domain model to determine which answer follows.For example, a qualitative relation can transfer a higher value in one property between worlds to a higher value in another.Source: §3.4 · p. 3
4 The QUAREL Dataset
QUAREL is a crowdsourced collection of multiple-choice qualitative story questions paired with logical forms. Its construction aims for varied language and recovers logical forms through follow-up questions rather than requiring workers to use the formalism.
4 The QUAREL Dataset
- QUAREL contains 2,771 multiple-choice story questions with logical forms, and its constrained logical-form space supports rich exploration.The dataset is similar in size to several other semantic-parsing datasets with annotated logical forms.Source: §4 · p. 4
- Workers created varied questions by comparing objects, people, or situations using a supplied qualitative relation as a seed.They were encouraged to be imaginative and use their own words.Source: §4 · p. 4
- Logical forms were reverse-engineered from follow-up questions, allowing workers to provide the needed information without learning the underlying formalism.The technique relies on the constrained space of logical forms.Source: §4 · p. 4
1. What is the correct answer (A or B)?
The annotation procedure derives a target logical form from answer options, their property and direction, and the question’s comparison structure. QUAREL includes varied property relationships and received a 96.4% human dev-set score.
1. What is the correct answer (A or B)?
- Three independent workers answer follow-up questions used to infer each target logical form.The questions identify the answer-option property, whether it is higher or lower in the correct answer, and how the options relate to the same or different situations.Source: §1 · p. 4
- The annotation process arbitrarily sets one world to world1, allowing the other variables to be inferred.The resulting logical forms are used with qualitative relationships to infer an answer.Source: §1 · p. 4
- 96.4%: a human answered the dev questions correctly, with occasional failures attributed to annotation errors or question ambiguities.The authors interpret this as evidence of high content fidelity.Source: §1 · p. 4
- About half of QUAREL concerns friction and five related properties; the rest covers 14 additional properties and their relations.The friction questions form the QUARELF subset.Source: §1 · p. 4
- The dataset has 5,226 unique words, and its questions use varied phrases for smoothness.Figure 4 provides examples of that phrasing variety.Source: §1 · p. 4
5 Baseline Systems
The baseline suite ranges from corpus-based guesses and a friction-specific hand-built parser to a type-constrained neural semantic parser. QUASP predicts logical forms by choosing grammar production rules step by step.
5 Baseline Systems
- Four systems assess dataset difficulty: two corpus-based baselines, a friction-specific rule-based parser, and a neural semantic parser.The corpus-based methods do not parse questions; the rule-based parser predates dataset construction.Source: §5 · pp. 4–5
Information Retrieval (IR) System
- The information-retrieval system scores each answer option by searching a 280GB corpus for a sentence overlapping both the question and option.It selects the option whose top retrieved sentence receives the highest search-engine score.Source: Information Retrieval (IR) System · p. 5
Pointwise Mutual Information (PMI)
- The PMI baseline selects the option with the largest average association score across question and answer-option n-grams.It uses pointwise mutual information to measure associations between parts of the question and answer option.Source: Pointwise Mutual Information (PMI) · p. 5
Rule-based Semantic Parser
- The friction-specific parser uses hand-written world-tagging rules and 142 CCG-like grammar rules to map selected phrases into logical forms.It skips most story words, attending to phrases tagged or referenced by its grammar.Source: Rule-based Semantic Parser · p. 5
Type-constrained Neural Semantic Parser (QUASP)
- QUASP decodes valid logical forms by selecting grammar production rules sequentially, training on annotated forms and using beam search at test time.Its type-constrained encoder-decoder builds an abstract syntax tree rather than directly emitting logical-form tokens.Source: Type-constrained Neural Semantic… · p. 6
6 Baseline Experiments
On the baseline evaluation, QUASP outperforms approaches that are only slightly above random, while parse accuracy exceeds a near-zero random baseline. Its errors often concern world assignments, and the rule-based parser misses many questions because of vocabulary variation.
6 Baseline Experiments
- The rule-based parser cannot parse 66% of questions, largely because varied vocabulary is not covered by its hand-written grammar.Unparsed questions receive a score of 0.5, reflecting a random guess.Source: §6 · p. 6
- QUASP scores 56.1% on the full QUAREL dataset and 61.7% on the friction subset, performing better than baseline approaches that are only slightly above random, indicating the dataset’s challenges.Source: §6 · p. 6
- QUASP parse accuracy is relatively high compared with a near-zero random baseline, though the supplied passage gives no exact score.The parse-accuracy table reports semantic-parser results.Source: Table 3 · p. 6, §6 · p. 6
- On the dev set, world prediction accounts for about 70% of QUASP’s errors, while property prediction accounts for about 25%.Incorrect world assignment can flip the answer, turning a potentially partial score into zero.Source: §6 · p. 6
7 New Models
The section presents two QUASP extensions: QUASP+ canonicalizes references to compared worlds, while QUASP+ZERO targets properties unseen during training. QUASP+ significantly outperforms baselines, and QUASP+ZERO achieves above-random zero-shot performance.
7.1 QUASP+: A Model Incorporating World Tracking
- World tracking is difficult because world descriptions vary across questions, multiple phrases can denote one world, and parses must assign world identifiers consistently.The phrases are often question-specific, with no obvious learnable mapping from phrases to worlds.Source: §7.1 · p. 7
- QUASP+ trains a BIO-tagging span detector, groups mentions into two worlds, canonicalizes them as WORLD1 and WORLD2, then trains QUASP on the transformed questions.The two longest-span groups define the worlds; other spans are grouped by substring or closest edit distance, and labels follow question order.Source: §7.1 · p. 7
- Over 12% absolute, QUASP+ significantly outperforms baselines; parse accuracy improves from 32.2% to 43.8%.The authors attribute the progress to delexicalizing world references.Source: §7.1 · p. 7
7.2 QUASP+ZERO: A Model for the Zero-Shot Task
- The zero-shot setting withholds questions involving one property, trains on the others, and tests on the withheld property without annotated examples for it.The goal is to add a qualitative relation using only minimal extra information about the new property.Source: §7.2 · pp. 7–8
- During decoding, QUASP+ZERO links question tokens to properties using similarity with a small list of words associated with each property.For example, “longer” can cue the property length even when it was unseen during training.Source: §7.2 · p. 8
- On full QUAREL, QUASP+ZERO achieves 25.7% parse accuracy and 59.5% answer accuracy on questions with unseen properties.These are weighted averages across zero-shot test sets; the authors report both as significantly better than random.Source: §7.2 · p. 8
8 Summary and Conclusion
The conclusion recaps a formalism, the QUAREL dataset, and two models for qualitative story-question parsing. It also identifies current task boundaries and frames the work as a contribution intended to encourage further progress.
8 Summary and Conclusion
- 25.7% parse and 59.5% answer are the weighted average scores on full QUAREL for questions with unseen properties using QUASP+ZERO.Table 4 reports these zero-shot scores for the full dataset.Source: Table 4 · p. 8
- The paper presents a formalism for qualitative questions, QUAREL as a dataset of qualitative story questions, and two models adapting parsing techniques to the task.QUASP+ uses delexicalization for world tracking; QUASP+ZERO applies entity linking to properties for zero-shot learning.Source: §8 · p. 8
- The current task does not handle context-dependent numeric-to-low/high mappings well, some questions outside its two templates, or questions needing an extra inference step.One example requires inferring that “best for a race” implies higher speed.Source: §8 · p. 8
- The dataset and models are publicly available, and the authors offer them as resources to encourage further progress on qualitative comparisons.The conclusion describes the work as making inroads into recognizing and reasoning with qualitative relationships.Source: §8 · p. 8