PaperlayerView exact revision on arXiv

Source-linked AI summary

Parsing Argumentation Structures in Persuasive Essays

Christian Stab, Iryna Gurevych

arXiv:1604.07370v2cs.CL

TL;DR

Parsing argumentation structures requires covering multiple subtasks while maintaining globally consistent component types and argumentative relations. This paper introduces an end-to-end parser with joint integer-linear-programming optimization and a corpus of annotated persuasive essays, improving component classification and relation identification while achieving 97.9% of human performance for component identification.

  • Problem

    Existing approaches focus on particular subtasks or rely on manually created rules, limiting their use for parsing global argumentation structures in real application scenarios.

  • Method

    The paper presents an end-to-end parser that identifies argument components at the token level and globally optimizes component types and argumentative relations with integer linear programming.

  • Results

    97.9% of human performance was achieved for argument component identification, while joint modeling simultaneously improved component classification and relation identification.

  • Takeaways & Limitations

    The annotation scheme supports substantial human annotator agreement on 402 persuasive essays, providing an annotated corpus for computational argumentation research.

  • Takeaways & Limitations

    Applying argument diagramming to real texts leaves ambiguities about convergent versus linked structures, graph versus tree representations, and component types in serial structures.

Abstract

from arXiv · show

In this article, we present a novel approach for parsing argumentation structures. We identify argument components using sequence labeling at the token level and apply a new joint model for detecting argumentation structures. The proposed model globally optimizes argument component types and argumentative relations using integer linear programming. We show that our model considerably improves the performance of base classifiers and significantly outperforms challenging heuristic baselines. Moreover, we introduce a novel corpus of persuasive essays annotated with argumentation structures. We show that our annotation scheme and annotation guidelines successfully guide human annotators to substantial agreement. This corpus and the annotation guidelines are freely available for ensuring reproducibility and to encourage future research in computational argumentation.

1. Introduction

Computational argumentation seeks to analyze arguments in natural-language texts, but existing methods did not jointly cover all structure-parsing subtasks or globally optimize their predictions. This paper introduces an annotation scheme, corpus, and end-to-end parser for persuasive essays.

  • Motivation: Computational argumentation analyzes arguments in natural-language texts and supports applications including legal decision support, information retrieval, policy making, debating, and writing support.Writing-support systems can provide feedback about written arguments.
  • Research gap: Existing approaches separately address component identification, component classification, or argumentative-relation identification, while most operate locally rather than optimizing global structure.The authors identify the absence of an approach covering all subtasks and globally optimizing argumentation structure.
  • Contributions: The paper introduces an annotation scheme that models document argumentation structures as connected trees.The scheme is derived from argumentation theory.
  • Contributions: The authors create a corpus of 402 persuasive essays with discourse-level argumentation-structure annotations and report substantial human-annotator agreement.The corpus and annotation guidelines are intended to support reproducibility and future research.
  • Contributions: The end-to-end parser identifies argument components at the token level and globally optimizes component types and argumentative relations.The model considerably improves base classifiers and significantly outperforms challenging heuristic baselines.

2. Related Work

Related work spans component identification, component classification, and structure identification, alongside discourse analysis and several argumentation corpora. The paper positions its contribution against limited coverage, missing discourse-level resources, and unknown or constrained annotation reliability.

  • Argument mining tasks: Argument-mining research distinguishes component identification, component classification, and structure identification as separate tasks.These tasks respectively address boundaries, component functions such as claims and premises, and support or attack relations.
  • Structure identification: Structure-identification approaches divide into macro-level methods relating complete arguments and micro-level methods relating argument components.The paper focuses on micro-level structure identification.
  • Existing corpora: Discourse-level corpora with full argumentation structures remain rare despite resources covering argument strength, factual knowledge, argument properties, component types, and multi-document structures.The related-work discussion surveys existing corpora and their annotation scopes.
  • Existing corpora: Existing resources have important scope or reliability constraints, including heterogeneous text types, unknown annotation reliability, absent component types, single-argument microtexts, or limited realism.The cited corpora differ in document size, annotation types, text complexity, and inclusion of non-argumentative units.
  • Discourse analysis: Argumentation-structure parsing is related to discourse analysis, but the paper focuses on support and attack relations rather than the broader relation inventories used in discourse theories.The paper contrasts its relation scope with approaches using additional relations such as counter-attack or other discourse relations.

3. Argumentation: Theoretical Background

The paper represents persuasive-essay argumentation as a connected tree of claims, premises, and directed support or attack relations. It adopts a hybrid component-type scheme to handle serial arguments while acknowledging ambiguities in real argument diagrams.

  • Theoretical background: Argumentation theory studies how verbal activity increases or decreases the acceptability of controversial standpoints, with models spanning philosophical, dialogical, and rhetorical perspectives.This paper adopts a monological perspective suited to computational analysis of written arguments.
  • Argument diagramming: Argument diagrams represent argument components as nodes and directed argumentative relations as links from justifying or refuting sources to targets.Basic, convergent, serial, and divergent structures are described as elementary argument forms.
  • Argument diagramming: Applying argument diagramming to real texts involves ambiguity about convergent versus linked structures, graph versus tree representation, and component types in serial structures.These ambiguities motivate the paper’s modeling choices.
  • Argument component types: The paper uses a hybrid approach combining a level approach for the tree’s first level with a “one-claim” approach for each individual argument.Individual arguments are represented through directed support and attack relations.
  • Argumentation structures in persuasive essays: Persuasive essays are modeled as connected trees whose root is a major claim, whose body arguments contain claims and premises, and whose claims carry “for” or “against” stance attributes.Body arguments support or attack the author’s standpoint expressed in the major claim.
  • Argumentation structures in persuasive essays: Within each individual argument, the claim has incoming relations but no outgoing relation, while each premise has one outgoing relation and may have incoming relations.A premise’s outgoing relation indicates whether its stance is support or attack; inner premises model serial reasoning implicitly.
  • Example essay: The example essay illustrates a major claim, supporting claims, premises, and body-paragraph arguments that support or attack the author’s standpoint.It also shows that argumentative relations help separate multiple arguments within a paragraph.

4. Corpus Creation

The paper creates a corpus and annotation scheme for persuasive-essay argumentation structures, then evaluates annotator agreement and disagreement patterns.

  • Annotation procedure: The annotation guidelines use topic and stance identification, argument-component annotation, and premise linking through support or attack relations.Annotators read each essay, mark component boundaries and stance, then connect premises to claims or other premises.
  • Corpus: The corpus contains 402 persuasive essays annotated with discourse-level argumentation structures.The essays were randomly selected from essayforum.com and manually screened for sufficiently detailed writing prompts.
  • Annotation study: Three annotators independently annotated 80 essays, while the expert annotator labeled the remaining 322 essays.Agreement was evaluated using sentence-level measures and unitized alpha, which accounts for component-boundary differences.
  • Component agreement: 97.9% agreement and κ = .877 were achieved for major claims, while premises reached κ = .833 and αU = .824.Claims had the lowest component agreement at κ = .635; joint unitized agreement for all components was αU = .767, improving by .043 over the previous study.
  • Relations and stance: Annotators reliably differentiated supporting and attacking claims, achieving 88.5% agreement and κ = .623 for stance.For argumentative relations, both support and attack achieved κ-scores above .7, although agreement was lower than in the previous study.
  • Corpus statistics: The final corpus contains 6,089 argument components: 751 major claims, 1,506 claims, and 3,832 premises.Non-argumentative text comprises 47,474 tokens and 1,631 sentences, requiring token-level boundary recognition and argumentative-text separation.

5. Approach

The parser identifies argument components at the token level and jointly predicts component types and argumentative relations. Its CRF identification model improves substantially over heuristic baselines and approaches human performance.

  • Architecture: The approach uses five consecutive subtasks, beginning with token-level component identification and followed by joint structure and stance models.The joint model globally recognizes argumentation structure, while the stance model differentiates support and attack relations.
  • Identification model: Argument components are encoded with IOB tags over an entire essay, using “Arg-B”, “Arg-I”, and “O” labels in a CRF sequence-labeling model.“Arg-B” marks component starts, “Arg-I” marks covered tokens, and “O” marks non-argumentative tokens.
  • Features and evaluation: The models use structural, punctuation, syntactic, lexico-syntactic, and token-probability features, with probabilities conditioned on up to three preceding tokens.Model selection uses 5-fold cross-validation before assessment on test data.
  • Baseline: The heuristic baseline achieves a macro F1 score of .642, including F1 scores of .677 for “O” and .867 for “Arg-I”.Although it separates argumentative from non-argumentative text effectively, it performs poorly at identifying component beginnings.
  • Results: .867 macro F1 is achieved by the CRF model with all features, significantly outperforming the heuristic baseline.The model identifies 1,272 components versus 1,266 gold components and reaches 97.9% of human performance.
  • Error analysis: The model’s most frequent errors are false-positive “Arg-I” labels and misclassified beginnings of argument components.Errors include labeling non-argumentative conclusion or body-paragraph sentences as argumentative and missing beginnings after introductory phrases.

5.3 Recognizing Argumentation Structures

The parser combines component-type and relation classifiers in a globally optimized model, using features from lexical, structural, contextual, syntactic, probability, embedding, and discourse information. Integer linear programming enforces a coherent argumentation structure, including tree constraints and acyclicity.

  • Joint model: The model jointly optimizes argument component types and argumentative relations because their predictions share mutual information.Claims are less likely to have outgoing relations and more likely to have incoming relations, whereas premises show the opposite pattern.
  • Component classification: Argument components are classified as major claims, claims, or premises using multiple feature groups.The feature set includes lexical, structural, indicator, contextual, syntactic, probability, embedding, and discourse features.
  • Component classification: Discourse features encode the relation type, component position within the relation, and whether the discourse relation is implicit or explicit.These features are derived from the output of a PDTB-style discourse parser.
  • Relation identification: The relation model classifies ordered component pairs as linked or not-linked, treating both support and attack as linked.Its features include lexical, syntactic, structural, indicator, and PMI representations.
  • Joint model: 37.1% of premises lacked a link in model-selection experiments, motivating global optimization of the locally predicted structure.The joint task is formalized as an integer linear programming problem.
  • Joint model: The ILP constraints enforce a tree structure and prevent cycles by representing direct and transitive paths between argument components.The path constraints iteratively cover longer paths and exclude directed paths that start and end at the same component.

5.4 Classifying Support and Attack Relations

The stance recognition model classifies argumentative components according to whether they support or attack. It uses an SVM with lexical, sentiment, syntactic, structural, discourse, and embedding features.

  • Stance classification: The stance model classifies each claim and premise as support or attack using binary classification.A premise’s stance is encoded in its outgoing relation type, while a claim’s stance is encoded in its stance attribute.
  • Features: The model uses lexical, sentiment, syntactic, structural, discourse, and embedding features for stance recognition.Discourse features are intended to help identify attacking components through contrast and concession relations in the PDTB.
  • Features: The stance-recognition feature inventory is documented in Table 11.The table is titled “Features used for stance recognition.”

5.5 Evaluation

The ILP joint model improves component classification and relation identification over heuristic baselines, while approaching human performance most closely for component classification.

  • The heuristic baselines achieve an average macro F1 score of .674 across classification, relation identification, and stance recognition.
  • The relation identification model achieves a macro F1 score of .717, significantly improving on the heuristic baseline.
  • The ILP joint model significantly outperforms heuristic baselines for component classification and relation identification.
  • Compared to base classifiers, the ILP joint model improves claims by .071 F1 and linked component pairs by .077 F1.
  • The ILP joint model reaches 95.2% of human performance for component identification, compared with 87.9% for relation identification and 80.5% for stance recognition.
  • On average, the authors’ base classifiers outperform Peldszus and Stede’s base classifiers on the English microtext corpus by .025 macro F1.

5.6 Error Analysis

Error analysis finds that the model produces shallower structures than the gold standard, while controlled improvements show that joint modeling benefits both tasks but depends strongly on relation quality.

  • The highest component-classification confusion is between claims and premises, with 74 premises labeled claims and 82 claims labeled premises.
  • The model correctly identifies only 34.7% of 98 serial arguments, producing structures that are shallower than the gold standard.
  • The objective function’s claim-centered weight calculation is identified as a reason for errors in serial arguments.
  • Less accurate argumentative relations harm component-type outcomes more than inaccurate component types harm relation-identification outcomes.
  • When both base classifiers improve, the ILP joint model improves both tasks and improves component types more effectively than argumentative relations.

6. Discussion

The parser’s pipeline and tree assumption constrain generalization, especially beyond persuasive essays, and the usefulness of its extracted structures for feedback remains unvalidated.

  • Errors in upstream identification or classification can negatively influence downstream parser results.
  • Identification errors are considered unlikely to substantially affect persuasive-essay results because the identification model achieves αU of .958 for argument components.
  • Applying the annotation scheme to genres with less explicit argumentation may be more challenging than applying it to persuasive essays.
  • It is unknown whether the tree assumption generalizes to genres containing divergent or cyclic argumentation structures.
  • It remains unknown whether the identified structures can provide adequate feedback about argumentation.

7. Conclusion

The paper presents an end-to-end parser, joint modeling approach, annotation scheme, and persuasive-essay corpus for global argumentation-structure analysis.

  • The end-to-end approach covers all required subtasks for identifying documents’ global argumentation structures.
  • Jointly modeling argumentation structures simultaneously improves component classification and relation identification.
  • The paper introduces a novel annotation scheme and the largest resource of its kind: a corpus of persuasive essays annotated with argumentation structures.
  • The corpus and annotation guidelines are freely available to support reproducibility and future computational-argumentation research.

Appendix A: Class Distributions

Appendix A presents the persuasive essay corpus's training and test class distributions for each analysis step.

  • The corpus's training and test data are divided by analysis step and class.

Appendix B: Detailed Results of Model Selections

Appendix B reports feature-selection and joint-model results across argument component identification, classification, relation identification, joint modeling, and stance recognition.

  • Argument component identification: All features produce the best argument component identification results, while omitting genre-dependent features yields macro F1 .847, only .002 below the full model.Lexico-syntactic features perform best individually, especially for “Arg-B”; structural features are second-best.
  • Argument component classification: Structural features are the only individual group significantly outperforming the heuristic baseline for component classification and are most effective for major claims.Discourse features are the second-best features for identifying claims.
  • Argumentative relation identification: Structural features are the most effective feature group for identifying argumentative relations, followed by indicator and PMI features.Syntactic and discourse features provide slight improvements when combined with other features.
  • Joint modeling: The heuristic baseline identifies 3.3% more trees than base classifiers, but this difference is not statistically significant.The baseline is triggered in 31 paragraphs.
  • Joint modeling: ILP-balanced is selected as the best joint model, and all three ILP variants significantly outperform the base classifier for component classification.Combining both base classifiers produces a more balanced proportion of component-type conversions than single-source variants.
  • Stance recognition: Lexical, syntactic, and discourse features significantly improve stance recognition over the heuristic baseline, while the full feature model is selected as best.Sentiment, structural, and embedding features do not improve over the majority baseline individually.

Appendix C: Indicators

Appendix C lists lexical indicators extracted from 30 persuasive essays for several argumentative functions.

  • The indicator lists contain 22 forward indicators, 33 backward indicators, 48 thesis indicators, and 10 rebuttal indicators.
Loading 1604.07370v2…