PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Argument Generation Augmented with Externally Retrieved Evidence

Xinyu Hua, Lu Wang

arXiv:1805.10254v1cs.CL

TL;DR

Generating coherent arguments that aggregate evidence is challenging, especially given the domain limits of prior NLG systems. This paper uses retrieved Wikipedia evidence and a two-stage neural decoder to generate arguments of an alternative stance, with evaluations finding more informative and topic-relevant arguments than popular seq2seq models.

  • Problem

    Argument generation must aggregate supporting evidence and produce coherent logical text, while prior NLG systems have been limited to a small set of domains.

    Source: §1 · p. 1

  • Method

    The model retrieves Wikipedia evidence for a controversial statement, generates talking-point keyphrases, and then decodes an alternative-stance argument from the input and keyphrases.

    Source: §1 · p. 2

  • Results

    Automatic and human evaluations found more informative arguments than popular seq2seq models, and generated arguments were more likely to be predicted on-topic than other seq2seq generations.

    Source: §1 · p. 2, §10 · p. 9

  • Takeaways & Limitations

    The paper demonstrates an evidence-augmented neural approach to generating arguments of a different stance for a given statement.

    Source: §10 · p. 9

    System arguments still show a noticeable gap from human-constructed text, with linguistic style and coherence identified as future-work needs.

    Source: §8 · p. 9

Abstract

from arXiv · show

High quality arguments are essential elements for human reasoning and decision-making processes. However, effective argument construction is a challenging task for both human and machines. In this work, we study a novel task on automatically generating arguments of a different stance for a given statement. We propose an encoder-decoder style neural network-based argument generation model enriched with externally retrieved evidence from Wikipedia. Our model first generates a set of talking point phrases as intermediate representation, followed by a separate decoder producing the final argument based on both input and the keyphrases. Experiments on a large-scale dataset collected from Reddit show that our model constructs arguments with more topic-relevant content than a popular sequence-to-sequence generation model according to both automatic evaluation and human assessments.

1 Introduction

The paper frames generating an argument of an alternative stance to a controversial statement as a difficult but needed task. It proposes a neural system using Wikipedia evidence and talking-point keyphrases, with evaluations reporting stronger arguments than comparison systems.

1 Introduction

  • Automating argument construction is presented as valuable because arguments support reasoning and decision-making, while constructing them is difficult for both people and computers.The introduction names debate coaching and essay-writing support as possible applications.Source: §1 · p. 1
  • Existing argument-generation work relied on retrieval and selection, lacking synthesis across resources and risking redundancy and incoherence.The introduction also describes NLG systems as limited to a small set of domains and template-dependent.Source: §1 · pp. 1–2
  • The task is to generate an argument of an alternative stance for a controversial statement, using externally retrieved evidence and separate talking-point and argument decoders.The encoder takes the statement and retrieved Wikipedia evidence; the decoders first generate keyphrases, then the argument.Source: §1 · p. 2
  • Arguments scored better by BLEU than retrieved sentences and evidence-trained seq2seq models, and were more likely to be predicted on-topic than other seq2seq generations.Source: §1 · p. 2

2 Framework

The framework retrieves Wikipedia evidence for a statement, then generates an argument through two decoder stages. The first produces talking-point keyphrases; the second uses them with the input to construct the argument.

2 Framework

  • Queries built from a statement’s topic-signature words retrieve Wikipedia articles, and reranking selects evidence sentences as additional model input.Source: §2 · p. 2
  • A shared encoder processes the statement and evidence in sequence, while the keyphrase decoder generates talking points as an intermediate representation.Source: §2 · p. 2
  • A separate argument decoder produces the final argument after the keyphrase decoder, using both the input and the generated keyphrases.Source: §2 · p. 2

3 Data Collection and Processing

The study builds its argument data from Reddit’s ChangeMyView discussions, where original posts state controversial viewpoints and replies can offer counter-arguments. It filters replies and focuses its domain-specific dataset on politics.

3 Data Collection and Processing

  • The dataset comes from 26,761 ChangeMyView threads collected from January 2013 through June 2017; threads pair original viewpoints with replies, sometimes marked by awarded deltas.Source: §3 · p. 2
  • After filtering root replies for length, offensiveness, delta or vote balance, and moderator authorship, the dataset contains 26,525 original posts and 305,475 replies.Source: §3 · p. 3
  • The researchers focus on politics because that domain has many discussions and broad argument coverage, using a bootstrapped classifier to label posts as political or non-political.The classifier uses unigram features and politics/non-politics labels derived from manually selected keywords in Wikipedia abstracts.Source: §3 · p. 3
  • The politics subset contains 12,549 original posts, each paired with an average of 9.4 target arguments.Source: §3 · p. 3

4 Model

The model uses a shared encoder and separate keyphrase and argument decoders; its argument decoder attends to the input and generated keyphrases, while hybrid beam search promotes diversity and content coverage.

4 Model

  • The model generates keyphrases as talking points before a separate decoder produces the argument, with both decoders sharing an encoder.The training objective combines keyphrase and argument sequence losses.Source: §4 · p. 3, §4.1 · p. 4

4.1 Model Formulation

  • The training objective sums the keyphrase and argument losses, weighted by α, which is set to 0.5 in the experiments.Source: §4.1 · p. 4
  • The argument decoder attends to both encoder states and keyphrase-decoder states when updating its state.This lets it use the input and the generated talking points during argument generation.Source: §4.1 · p. 4

4.2 Hybrid Beam Search Decoding

  • Hybrid beam search expands hypotheses with some top-probability words and randomly sampled alternatives, then reranks beams by input-content-word coverage every 10 steps.The approach is designed to increase beam diversity and promote more informative arguments.Source: §4.2 · pp. 4–5

5 Relevant Evidence Retrieval

The retrieval method finds Wikipedia evidence through statement-based queries and reranks retrieved text; training keyphrases are extracted from evidence phrases reused in human arguments.

5.1 Retrieval Methodology

  • For each statement sentence, the method builds a query from topic-signature-bearing noun phrases and verbs, retrieves Wikipedia articles, and reranks paragraphs and sentences by TF-IDF.It retains up to five articles per query, 100 positive-scoring paragraphs, and 10 positive-scoring evidence sentences.Source: §5.1 · p. 5

5.2 Gold-Standard Keyphrase Construction

  • Training keyphrases come from 2–10-word noun or verb phrases in evidence sentences that overlap argument content words, with overlapping spans resolved by content-word coverage.The retained phrases are concatenated with a delimiter as keyphrase-generation targets.Source: §5.2 · p. 5

6 Experimental Setup

The experiments use sampled Wikipedia evidence to construct a large Reddit dataset, train with staged sequence-length increases and pretraining, and compare evidence-aware generation variants under different retrieval settings.

6.1 Final Dataset Statistics

  • 224,553 training examples, 13,911 validation examples, and 30,417 test examples remain after evidence sampling and duplicate removal.For each statement sentence, up to three retrieved evidence sentences are sampled; sampling is repeated three times per statement.Source: §6.1 · p. 6

6.2 Training Setup

  • Curriculum training gradually increases truncated input and output lengths, with the first stage excluding evidence and keyphrases.The staged truncation sizes are listed in Table 2.Source: §6.2 · p. 6, Table 2 · p. 6
  • Pretraining initializes the first layer of comparison models and most proposed models, and boosts all methods by roughly 2 METEOR points.The keyphrase decoder is excluded from first-layer initialization.Source: §6.2 · p. 6

6.3 Baseline and Comparisons

  • Comparisons include concatenated retrieved sentences and seq2seq models trained with the post, evidence, or gold-standard keyphrases.The gold-keyphrase comparison assumes some talking points are known.Source: §6.3 · p. 6
  • The study tests separate versus shared decoders, keyphrase attention, and system versus oracle evidence retrieval.Oracle retrieval uses queries based on user arguments, while system retrieval uses the original post.Source: §6.3 · p. 6

7 Results

Separate-decoder models outperform seq2seq baselines on automatic argument-generation measures, and evidence-trained models score higher in topic-relevance evaluation. Human judges rate the proposed model more informative and relevant than seq2seq, but prefer retrieved arguments.

7.1 Automatic Evaluation

  • Separate-decoder models significantly outperform all seq2seq models in BLEU and METEOR (p < 0.0001).The retrieval baseline achieves better METEOR scores, mainly because its arguments are significantly longer.Source: Table 3 · p. 7, §7.1 · p. 7
  • Attention over input and generated keyphrases further improves performance, while system retrieval yields better BLEU than oracle retrieval.The authors suggest system retrieval may yield more generic argumentative phrasing and thus higher precision.Source: §7.1 · p. 7
  • Smaller reranking steps, such as p = 5, generally improve METEOR; smaller k values produce more diverse beams without significant score differences.Figure 3 compares reranking intervals of 5, 10, and 20 steps and different values of k.Source: §7.1 · p. 7, Figure 3 · p. 7

7.2 Topic-Relevance Evaluation

  • The relevance ranker scores evidence-trained models significantly higher than seq2seq trained without evidence on MRR and P@1 (p < 0.0001).The evaluation ranks each system argument against negative examples drawn from arguments for other posts.Source: Table 4 · p. 8, §7.2 · p. 8
  • More than 75% of seq2seq outputs contain a generic response, compared with 16.2% of outputs from the separate decoder with keyphrase attention.The authors interpret this frequency difference as further evidence of more topic-relevant content.Source: §7.2 · p. 8

7.3 Human Evaluation

  • Human judges rate the separate decoder with keyphrase attention significantly higher than seq2seq in informativeness and relevance, but prefer retrieved arguments.The ratings difference is significant at p < 0.005.Source: Table 5 · p. 8, §7.3 · p. 8

8 Further Discussion

The discussion finds that generated keyphrases can guide arguments even when they are not reused verbatim, while sample outputs still fall short of human arguments in style and coherence.

Keyphrase Generation Analysis

  • About 10% of generated keyphrases also appear in gold-standard human arguments, while 36% are reused in system arguments.The authors report that some unused keyphrases still represent high-level talking points that suggest what the argument should cover.Source: §8 · p. 8

Sample Arguments and Future Directions

  • The system generally captures more relevant concepts in sample arguments, but a noticeable gap from human arguments remains in argumentative style.The authors identify better linguistic-style control and discourse-aware generation for text planning as future directions.Source: §8 · pp. 8–9
  • The three aspects have inter-rater agreement scores of 0.50, 0.60, and 0.48 by Krippendorff’s α.Figure 4 presents only the main thesis for the input post and compares human, system, and evidence-trained seq2seq samples.Source: §8 · p. 8, Figure 4 · p. 9

9 Related Work

Prior work has explored argument construction and retrieval, but the cited approaches do not provide a fully automated system that synthesizes retrieved content into fluent arguments.

9 Related Work

  • Earlier argument-construction work focused on strategies, and Rhetorica produced a text plan using heuristic rules rather than a fully automated argument.The cited discussion notes that text generation difficulty left previous work short of a fully automated system.Source: §9 · p. 9
  • Argument retrieval and extraction deliver relevant arguments for user queries, while Wachsmuth et al. (2017) built a search engine over arguments from online debate portals.Sentence ordering has also been used after retrieval, but merging arguments from different resources can introduce redundancy.Source: §9 · p. 9
  • The authors describe their system as the first automatic argument-generation system to synthesize retrieved content from different articles into fluent arguments.They distinguish this synthesis goal from selecting and ordering retrieved arguments.Source: §9 · p. 9

10 Conclusion

The paper presents a Wikipedia-evidence-enhanced neural framework for generating arguments that take a different stance from a given statement. It reports that automatic evaluation and human assessment found its arguments more informative than popular seq2seq-based generation models.

10 Conclusion

  • Automatic evaluation and human assessment showed that the framework produced more informative arguments than popular sequence-to-sequence-based generation models.The framework uses Wikipedia evidence and separate decoders to generate keyphrases as talking points before generating the final argument.Source: §10 · p. 9
Loading 1805.10254v1…