Source-linked AI summary
Interpretation of Natural Language Rules in Conversational Machine Reading
Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocktäschel, Mike Sheldon, Guillaume Bouchard, Sebastian Riedel
TL;DR
Conversational machine reading addresses questions whose answers must be derived from natural-language rules, background knowledge, and sometimes clarification dialogs. The paper formalizes this task, introduces a controlled crowdsourcing protocol and a corpus exceeding 32k utterances, and evaluates initial baselines. Results show substantial challenges when models must integrate rule interpretation, question generation, and background-knowledge reasoning.
Problem
Existing machine-reading tasks mainly answer questions whose answers are stated directly, whereas many practical questions require interpreting rules with background knowledge and resolving underspecified information.
Method
The paper formalizes conversational machine reading, develops a crowdsourced annotation protocol, collects over 32k utterances, and evaluates rule-based and machine-learning baselines.
Results
Initial baselines show that the task remains challenging, requiring stronger integration of entailment-like reasoning and question generation.
Takeaways & Limitations
The corpus and annotation protocol establish a benchmark for systems that must generate clarifying questions and reason over rules, scenarios, and dialog history.
Takeaways & Limitations
The study evaluates classification without scenarios to focus on rule interpretation, and notes that the dataset may be insufficient for training end-to-end neural models.
Abstract
from arXiv · showhide
Most work in machine reading focuses on question answering problems where the answer is directly expressed in the text to read. However, many real-world question answering problems require the reading of text not because it contains the literal answer, but because it contains a recipe to derive an answer together with the reader's background knowledge. One example is the task of interpreting regulations to answer "Can I...?" or "Do I have to...?" questions such as "I am working in Canada. Do I have to carry on paying UK National Insurance?" after reading a UK government website about this topic. This task requires both the interpretation of rules and the application of background knowledge. It is further complicated due to the fact that, in practice, most questions are underspecified, and a human assistant will regularly have to ask clarification questions such as "How long have you been working abroad?" when the answer cannot be directly derived from the question and text. In this paper, we formalise this task and develop a crowd-sourcing strategy to collect 32k task instances based on real-world rules and crowd-generated questions and scenarios. We analyse the challenges of this task and assess its difficulty by evaluating the performance of rule-based and machine-learning baselines. We observe promising results when no background knowledge is necessary, and substantial room for improvement whenever background knowledge is needed.
1 Introduction
The paper introduces conversational machine reading for questions whose answers require interpreting natural-language rules, applying background knowledge, and sometimes asking clarifying questions. It formalizes the task, develops a controlled crowdsourcing protocol, and finds that existing models struggle with its combined reasoning and generation demands.
- Motivation: Conversational machine reading answers questions by combining rule interpretation with scenario information and background knowledge rather than extracting literal answers.The motivating example requires connecting a National Insurance rule with the fact that Canada is outside the EEA.
- Motivation: Missing information can make a follow-up question necessary before the system can determine the final answer.The example asks whether the user has worked abroad for 52 weeks or less because the rule limits payment to the first 52 weeks.
- Task: The task takes an input question, scenario, supporting rule text, and dialog history, then predicts Yes or No or generates a necessary follow-up question.The paper’s goal is to create a corpus, analyze the challenges, and develop initial models for this setting.
- Data collection: The annotation protocol incrementally constructs dialogs from rule snippets and underspecified questions, using virtual-user answers to control the balance of Yes and No outcomes.Annotators then produce more elaborate scenarios from selected portions of the dialog.
- Results: Despite 21,890 training utterances, successful models need stronger inductive bias because the task combines rule interpretation, question generation, and background-knowledge reasoning.Heuristics identify useful questions but still fail to interpret scenarios correctly, while oracle systems help humans answer faster and more accurately.
- Contributions: Over 32k utterances across domains including grants, traffic laws, and benefit programs form the corpus, alongside evaluation metrics and baseline models.The contributions include the task definition, a crowdsourcing protocol, corpus analysis, and comparisons of several baseline models.
2 Task Definition
The paper formalizes conversational machine reading per utterance using a question, rule text, dialog history, and scenario as input. The system predicts a final answer, irrelevance, or another follow-up question when the available information is insufficient.
- Interaction: A follow-up question is generated when its answer is necessary to determine the original question’s final response.For example, the system may ask how long someone has been working abroad before applying a 52-week rule.
- Inputs: Each input consists of a question, supporting rule text, previous follow-up question-answer pairs, and a scenario describing the question’s context.The supporting rule snippet is assumed to be pre-identified.
- Outputs: The output is YES, NO, IRRELEVANT, or another follow-up question in the vocabulary.IRRELEVANT applies when the rule text is unrelated to the input question.
3 Annotation Protocol
The annotation protocol decomposes dataset creation into controlled stages that generate underspecified questions, complete dialog trees, and scenarios consistent with selected dialog evidence. It also adds negative examples and quality controls to support scalable, varied data collection.
- Stages: The protocol has four stages: rule text extraction, question generation, dialog generation, and scenario annotation.The supplied figures focus on the latter three stages.
- Question generation: Annotators create topic-matching but underspecified questions that require follow-up questions rather than direct answers.This stage was performed by expert annotators, with future crowdsourcing planned.
- Dialog generation: During dialog generation, human annotators ask rule-guided follow-up questions while a virtual user supplies random YES or NO answers.Different annotators label additional decision-tree branches so the corpus reflects all possible dialogs.
- Scenario annotation: Scenario annotators receive selected question-answer portions and write scenarios consistent with them, sometimes including an unrelated question to add irrelevant information.The resulting dialog combines the scenario, original question and rule text with omitted dialog utterances restored.
- Negative examples: Negative questions are sampled from unrelated questions, and negative scenarios are sampled from scenarios paired with other question-rule combinations.These examples target IRRELEVANT predictions and reduce bias toward relevant scenarios.
- Quality and cost: Quality control includes branch re-annotation, dialog validation, annotator scoring, qualification tests, approval requirements, and geographic restrictions.The process took two weeks, with payments of $0.15 per initial question, $0.11 per dialog part, and $0.20 per scenario.
4 ShARC
ShARC is a dataset for conversational machine reading, built from rule snippets, dialog trees, scenarios, and quality-controlled annotations. Its analysis shows that rule interpretation frequently requires distinguishing logical conditions, while scenario interpretation involves multiple reasoning types.
- Dataset construction: ShARC contains 948 rule snippets, 6058 dialog-tree utterances, 6637 scenarios, and 32436 final utterances after scenario and reachability processing.Scenarios can skip dialog questions when their answers are inferable, while negative sampled scenarios contribute to the expanded utterance count.
- Dataset quality: Dialog-generation annotations reached 72.3% agreement on YES, NO, or FOLLOW-UP QUESTION decisions and an averaged unweighted Cohen’s Kappa of 0.82.These measures evaluate agreement among three workers on sampled rule texts and questions, with kappa averaged across repeated pair selections.
- Dataset quality: Annotators’ follow-up questions achieved average BLEU scores of 0.71, 0.63, 0.58, and 0.58 at maximum orders 1 through 4.These scores approximate whether independently generated follow-up questions are equivalent, complementing categorical agreement measures.
- Dataset quality: Human accuracy on classifying YES, NO, or FOLLOW-UP QUESTION responses was 93.9%.The estimate treats the second answer for each question as a human prediction and majority vote as ground truth.
- Dataset quality: The annotation protocol produced high scenario validity, with annotators agreeing on 91-92 of 100 sampled scenarios in the reported validation cases.The qualification test and validation process substantially improved generated-scenario quality.
- 4.2.1 Interpreting Rules: Rule interpretation commonly requires multiple conditions: approximately 41% of cases involve disjunctions and 27% involve conjunctions.In 13% of cases, linguistic cues do not distinguish conjunctions from disjunctions, requiring language understanding.
- Scenario interpretation: Scenario interpretation is framed as entailment classification, with YES, NO, or IRRELEVANT corresponding to entailment, contradiction, or an answer not inferable from the scenario.The analyzed reasoning types include numerical, temporal, and implication reasoning, and a scenario may require more than one type.
5 Experiments
The experiments evaluate classification, follow-up question generation, scenario interpretation, and end-to-end conversational machine reading using rule-based, feature-based, and neural baselines. Results show that scenario interpretation and full-task performance remain difficult, while structured or span-based approaches often outperform end-to-end neural models.
- Experimental setup: The experiments assess baseline feasibility and difficulty across classification, follow-up generation, scenario interpretation, and end-to-end CMR.They use micro- and macro-accuracy for classification and BLEU scores for follow-up generation.
- Classification: Random Forest slightly outperforms the heuristic on classification, while all learned models outperform random and majority baselines.
- Follow-up question generation: Systems returning contiguous rule-text spans outperform other follow-up-generation systems on BLEU, although only rule-based and NMT-Copy models generate genuine questions.The authors speculate that logical forms are challenging for existing models to extract and manipulate.
- Scenario interpretation: Scenario interpretation baselines perform poorly, especially on macro accuracy, including simple and neural entailment models.This performance highlights the difficulty of interpreting scenarios in ShARC.
- Conversational machine reading: The combined model outperforms the neural end-to-end model on CMR, while the neural model still learns better-than-random classification and follow-up-question prediction.The combined model pipelines classification, follow-up generation, and scenario interpretation.
- User study: Users with the conversational agent reach conclusions more than twice as fast and with 93% accuracy, compared with 68% without the agent.
6 Related Work
The paper connects conversational machine reading to machine reading, dialog-based question answering, rule extraction, and question generation. Its key distinction is that regulatory questions often require clarification, rule interpretation, and background knowledge rather than direct text retrieval.
- Machine Reading: Unlike traditional machine reading, conversational machine reading often requires clarification questions before a regulatory question can be answered.The task also differs in its regulatory-text domain rather than Wikipedia, books, or newswire.
- Dialog: The task relates to dialog-based question answering because complex information needs are addressed through a series of questions.Prior work in that area primarily concerns dialogs about images and knowledge graphs.
- Rule Extraction From Text: Rule-extraction research interprets regulatory text as structured representations, whereas this paper frames the problem as end-to-end question answering.Structured approaches can be effective but require ontologies and linguistic preprocessing that are difficult for non-experts to annotate.
- Question Generation: Question-generation research typically creates questions answerable from the source text, whereas this task generates follow-up questions derived from background text but not answerable by it.
7 Conclusion
The paper introduces conversational machine reading with an annotation protocol, dataset, and baselines. Its initial results indicate that the task remains challenging and requires integrating entailment-like reasoning with question generation.
- Conclusion: The paper presents a new task, annotation protocol, dataset, and set of baselines for conversational machine reading.The task requires models to generate language, copy tokens, and make logical inferences.
- Conclusion: Initial baseline results suggest substantial improvement requires sophisticated integration of entailment-like reasoning and question generation.
A Annotation Interfaces
The annotation interfaces support dialog generation and scenario generation through Mechanical Turk. The dialog interface elicits rule-relevant evidence as YES/NO follow-up questions, while the scenario interface distinguishes relevant from irrelevant questions.
- Dialog interface: The Mechanical-Turk dialog-generation interface includes validation for utterances produced by different annotators.
- Dialog interface: The dialog-style web interface encourages workers to extract rule-relevant evidence through YES/NO follow-up questions.
- Scenario interface: The scenario-generation interface presents a relevant first question and an irrelevant second question.
B Quality Control
The dataset-quality protocol combines controlled annotation, worker screening, automated checks, expert review, and noise estimation to improve reliability.
- Automated validation: Binary decision-tree structure enables automatic detection of redundant or misannotated follow-up questions and guarantees at least two annotations per leaf.Identical YES or NO values at sibling leaves trigger subtree trimming and re-annotation.
- Validation: The protocol also uses back-validation, contradiction detection, validation sampling, and subtree re-annotation to identify invalid annotations.Workers can reject contradictory dialog parts and must provide an invalidation reason.
- Worker screening: Workers are screened through qualification tests, validation-based quality scores, approval rates above 97%, prior HIT completion, and location restrictions.The qualification-test threshold is 100%, and participation requires more than 1,000 completed HITs in the UK, US, or Canada.
- Noise estimation: 8% of negatively sampled questions and 5% of negatively sampled scenarios were erroneous in expert assessments of 100 examples each.These checks estimate noise from negative-question and negative-scenario generation.
- Noise estimation: 12% of assessed utterances had errors from applying scenarios to dialog trees, with an estimated total effect of 10%.The issue arises because scenarios may not be independent of other follow-up questions in the same dialog tree.
D Negative Data
The dataset augments rule-based dialogs with negative questions and scenarios, entailment examples, and follow-up-question-generation baselines to test relevance and reasoning.
- Negative questions: Negative questions are formed by pairing a rule with a randomly sampled question from another source document and labeling the utterance IRRELEVANT.The sampled question has an empty history sequence.
- Negative scenarios: Negative scenarios are selected to provide no information for answering a question, encouraging models to ignore irrelevant scenario details.Candidates are accepted when all cross-dialog-question TF-IDF cosine similarities fall below a validated threshold.
- Entailment corpus: The entailment corpus labels scenario-question pairs as ENTAILMENT, CONTRADICTION, or NEUTRAL according to whether the follow-up answer is derivable from the scenario.These instances are created for the train, development, and test sets.
- Dataset challenges: The dataset includes challenges involving complex rule relationships, negation, conjunctions, disjunctions, and cases requiring world knowledge to resolve ambiguity.The challenge breakdown is reported in Table 8.
- Baselines: Follow-up-question generation is evaluated with first-sentence, random-sentence, SurfaceLR, sequence-tagging, and neural encoder-decoder baselines.The NMT-Copy model uses attention, copying, restricted generation vocabulary, and greedy decoding at test time.
- Baselines: Span-extraction supervision is produced by matching the longest common token subsequence between each rule text and follow-up question.The resulting spans supervise BiDAF and sequence-tagger models.
I Evaluating Utility of CMR
A user study compares conversational machine reading with independently reading rules, questions, and scenarios. Conversational assistance makes users faster and more accurate.
- Study design: 100 randomly selected scenarios and questions were evaluated with annotations from 5 workers per instance.Gold answers and follow-up questions were available for all possible user responses.
- Results: Users with the conversational agent were almost twice as fast as users who read the rule text themselves.This advantage held despite the agent users answering as many follow-up questions as required.
- Results: Users with conversational agents were much more accurate than users without them.The study directly evaluates answer accuracy as the primary quality objective.