Source-linked AI summary
The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants
Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, Benno Stein
TL;DR
The paper addresses how to reconstruct implicit warrants that connect reasons to claims in natural-language arguments. It develops an eight-step crowdsourcing methodology and a two-option comprehension task, finding that current language-based models are insufficient while attention models outperform language models.
Problem
Implicit warrants are usually left unstated, making their reconstruction difficult because it depends on contextual world knowledge and reasoning patterns.
Method
The paper develops a systematic warrant-reconstruction methodology, operationalizes it through crowdsourcing, and evaluates a task requiring selection between opposing warrants.
Results
Human performance varies with formal argumentation training, the task is solvable without topic knowledge, and neural attention models outperform language models.
Takeaways & Limitations
The resulting task and dataset provide a basis for studying automatic reconstruction of implicit warrants in natural-language argumentation.
Takeaways & Limitations
The error analysis found no explicit pattern because the data contain diverse topics and reasoning patterns, leaving warrant categorization for future work.
Abstract
from arXiv · showhide
Reasoning is a crucial part of natural language argumentation. To comprehend an argument, one must analyze its warrant, which explains why its claim follows from its premises. As arguments are highly contextualized, warrants are usually presupposed and left implicit. Thus, the comprehension does not only require language understanding and logic skills, but also depends on common sense. In this paper we develop a methodology for reconstructing warrants systematically. We operationalize it in a scalable crowdsourcing process, resulting in a freely licensed dataset with warrants for 2k authentic arguments from news comments. On this basis, we present a new challenging task, the argument reasoning comprehension task. Given an argument with a claim and a premise, the goal is to choose the correct implicit warrant from two options. Both warrants are plausible and lexically close, but lead to contradicting claims. A solution to this task will define a substantial step towards automatic warrant reconstruction. However, experiments with several neural attention and language models reveal that current approaches do not suffice.
1 Introduction
The paper addresses the difficulty of reconstructing implicit warrants, the unstated knowledge connecting reasons to claims, and introduces a crowdsourced comprehension task for identifying them.
- Implicit warrants connect reasons to claims but are usually presupposed, requiring context-sensitive common-sense reasoning.
- Existing approaches struggle because reconstructing the missing reasoning depends on world knowledge and reasoning patterns, with substantial variation in manually written warrants.
- The proposed methodology twists an argument toward the opposite claim, then reverses the reasoning to reconstruct a warrant for the original claim.
- A crowdsourcing study produced 1,970 high-quality instances requiring selection of the correct warrant from two opposing options.
- Neural attention and language models based on argument and warrant words and phrases do not suffice to solve the task.
- The paper contributes a scalable warrant-reconstruction methodology, a new task, and datasets containing annotated arguments, reasons, and rephrased reason gists.
2 Related Work
Related work situates implicit-warrant reconstruction within argumentation theory, computational argument analysis, and semantic inference, while highlighting the difficulty of formalizing natural-language reasoning.
- Toulmin’s model represents an argument as Reason → Warrant → Claim, with the warrant functioning as an inference rule supporting the claim.
- Formal logic is unsuitable for much natural-language argumentation because such arguments are often inductive or defeasible rather than deductively valid.
- The distinction between premises and warrants remains contentious, and linking multiple premises can alternatively model their joint support for a claim.
- Claims and warrants are commonly implicit, so readers infer contextually relevant content using common-sense knowledge.
- Prior computational work identified automatic reconstruction as difficult, while annotation studies found written warrants varied substantially in depth and content.
- Unlike entailment, natural-language inference, and cloze-story tasks, this task targets reasoning types central to argumentation, including causal, logical, and analogical reasoning.
3 Argument Reasoning Comprehension
The argument reasoning comprehension task asks readers to choose which of two plausible warrants correctly links a reason to its claim, using an alternative warrant tied to the opposite claim.
- An argument instance contains a reason R, claim C, implicit warrant W, and alternative warrant AW supporting the opposite claim ¬C.
- The declawing example illustrates how a reason about cats’ claws supports illegality through an implicit warrant about nature.
- To construct AW, the method twists the claim, keeps the reason fixed, and then reverses the resulting reasoning to obtain W.
- Alternative warrants cannot be constructed for every reason-claim pair because some positions are deeply embedded or yield rebuttals rather than warrants.
- The task presents R and C with debate context and requires identifying the correct warrant W from W and AW.
4 Reconstruction of Implicit Warrants
The paper operationalizes warrant reconstruction as an eight-step, scalable crowdsourcing process applied to authentic, controversial online arguments and producing a dataset of task instances.
- The dataset contains authentic task instances formatted as (R,C,W,AW), rather than extending an existing resource.
- The source was the New York Times Room for Debate, selected for authentic contemporary argumentation across controversial polar questions.
- The process created opposing explicit claims, crawled debate comments, and sampled approximately 11,000 high-ranked root-level comments.
- The methodology consists of eight consecutive steps, each operationalizable through crowdsourcing, and was applied to 5,000 comments using Amazon Mechanical Turk.
- Crowdworkers classified stance, with neutral or non-stance comments treated separately; stance-taking comments were also annotated for sarcasm or irony.
- The methodology is summarized in an overview describing the reconstruction workflow for argument reasoning comprehension.
2. Reason Span Annotation
The annotation process identifies reason spans, removes clutter, and produces reason–claim pairs for downstream tasks. It yields a licensed collection of summarized reason gists.
- Workers selected spans that provide reasons for claims, using a single EDU as the minimal unit.
- 5,119 reason spans were identified, including 2,026 within arguments; about 40 comments lacked an explicit reason.
- Each reason was rewritten to preserve its gist while removing clutter and excluding wrongly annotated reasons, producing reason–claim pairs.
- 4,294 reason gists were summarized under a Creative Commons Zero license.
4. Reason Disambiguation
Reason disambiguation determines whether a reason supports the stated claim, its opposite, or both, while filtering for implicit presupposition and licensing the resulting warrants.
- The procedure assesses whether a reason itself implies C, ¬C, or both, based on how much presupposition it encodes.
- Only 1,955 instances where workers judged that R implied C were retained as evidence of implicit presupposition.
- The opposite-claim reasoning step requires a plausible explanation for why R supports ¬C and may be judged impossible.
- All 5,342 alternative warrants in the dataset were written under a CC-0 license.
6. Alternative Warrant Validation
Alternative warrants are validated by testing whether they relate to the original reason rather than a distractor. The process retains logically rated instances for the dataset.
- Each fabricated alternative warrant was validated with the opposite claim, the original reason, and a distracting reason.Only instances with correctly validated original reasons were kept.
- The distractor was sampled from the same debate topic as the most dissimilar reason according to skip-thought cosine similarity.Workers also rated the explanation’s logicality on a 0–2 scale.
7. Warrant For Original Claim
The final construction step converts validated alternative warrants into warrants supporting the original claim. A logic-score threshold filters out nonsensical alternatives before licensing the results.
- Workers minimally modified each alternative warrant so that R → W → C held for the original claim.
- 2,613 instances with a logic score of at least 0.68 were retained to filter out nonsense alternative warrants.
- The process produced 2,447 warrants, all written by workers under a CC0 license.
8. Warrant Validation
The study validates warrant instances through independent crowd estimates, agreement measures, and expert correction. It reports 1,970 final instances and illustrates how alternative and original warrants encode opposing reasoning paths.
- Each tuple was revalidated so that only one of the original or alternative warrant could logically explain the claim.
- 1,970 instances remained after disputed cases were corrected by an expert for quality assurance.
- 18 assignments per item were split into two time-based groups, whose MACE-derived labels enabled agreement measurement across independent crowd experts.
- With five annotators and a 0.95 confidence threshold, stance annotation achieved Cohen’s κ = 0.58, indicating moderate to substantial agreement.
- The methodology constructs alternative and original warrants by reversing the stance and recovering reasoning toward the original claim.
5 Experiments
Experiments assess human difficulty and computational performance on temporally separated argument-reasoning data. Humans perform substantially better with formal reasoning training, while current neural and language-model approaches remain weak on the test set.
- Experimental setup: The 1,970 instances were split chronologically into 1,210 training, 316 development, and 444 test instances to reduce lexical and topical overlap.
- Human upper bounds: Education had little effect on human performance, whereas more extensive formal reasoning training was associated with higher scores.
- Human upper bounds: Topic familiarity showed no significant correlation with mean performance in almost all education and training configurations.
- Computational models: Intra-warrant attention with rich context outperformed standard attention models but only slightly surpassed the language model on development data, while the language model was nearly random on test data.
- Error analysis: Manual analysis of 50 wrong predictions found no explicit error pattern, limiting conclusions about failures across diverse topics and reasoning patterns.
- Computational models: Adding alternative-warrant and opposite-claim instances to training left neural models no better than random guessing.
6 Conclusion and Outlook
The paper concludes with a new warrant-focused task, a scalable crowdsourced dataset, and initial evidence about human and model performance. It positions these contributions as a first empirical step toward understanding common-sense reasoning in argumentation and improving warrant reconstruction.
- Contributions: The authors introduced argument reasoning comprehension to address implicit warrants and proposed an eight-step methodology for reconstructing them systematically.
- Dataset: The crowdsourcing process produced 1,970 task instances and additional stance, argument, reason, and reason-summary annotations from authentic arguments.The byproducts include 5,000 stance-annotated comments, 2,026 arguments with 4,235 reasons, and 4,235 summarized reason gists.
- Evaluation: Human performance suggests the task is harder without formal argumentation training but remains solvable without knowing the topic.
- Evaluation: Neural attention models outperform language models on the task, while the language model is essentially random on the test set.Intra-warrant attention with rich context outperforms standard neural models with simple attention and only slightly beats the language model on development data.
- Outlook: The authors frame the work as a first empirical step toward determining how much common-sense reasoning is needed in argumentation.Future directions include a shared task, qualitative analysis of reasoning patterns, and eventually automatic warrant generation and validation.