PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing Things from a Different Angle: Discovering Diverse Perspectives about Claims

Sihao Chen, Daniel Khashabi, Wenpeng Yin, Chris Callison-Burch, Dan Roth

arXiv:1906.03538v1cs.CL

TL;DR

Search and recommendation systems retrieve information by claim relevance and popularity, rather than perspective diversity or evidence support. The paper defines substantiated perspective discovery and constructs PERSPECTRUM, a dataset of claims, perspectives, and evidence. Human baselines achieve high scores across the formulated tasks, and the paper develops competitive baselines using state-of-the-art techniques.

  • Problem

    Search and recommendation systems prioritize claim relevance and document popularity, rather than ensuring diverse perspectives or evidence support.

    Source: §1 · p. 1

  • Method

    The paper formulates substantiated perspective discovery and constructs PERSPECTRUM by combining online debate content, search data, paraphrases, and crowdsourcing.

    Source: §1 · p. 2

  • Results

    Human baselines achieve high scores on the identified tasks, and the paper develops competitive baselines using state-of-the-art techniques.

    Source: §1 · p. 2

  • Takeaways & Limitations

    PERSPECTRUM and the formulated tasks are intended to support research on perspective discovery and its underlying language-understanding challenges.

    Source: §1 · p. 2, §7 · p. 9

    The dataset is not exhaustive or representative of the world's claim distribution, omits expertise and trustworthiness labels, and assumes valid claims while leaving claim validation and argument extraction outside scope.

    Source: §6 · p. 9

Abstract

from arXiv · show

One key consequence of the information revolution is a significant increase and a contamination of our information supply. The practice of fact checking won't suffice to eliminate the biases in text data we observe, as the degree of factuality alone does not determine whether biases exist in the spectrum of opinions visible to us. To better understand controversial issues, one needs to view them from a diverse yet comprehensive set of perspectives. For example, there are many ways to respond to a claim such as "animals should have lawful rights", and these responses form a spectrum of perspectives, each with a stance relative to this claim and, ideally, with evidence supporting it. Inherently, this is a natural language understanding task, and we propose to address it as such. Specifically, we propose the task of substantiated perspective discovery where, given a claim, a system is expected to discover a diverse set of well-corroborated perspectives that take a stance with respect to the claim. Each perspective should be substantiated by evidence paragraphs which summarize pertinent results and facts. We construct PERSPECTRUM, a dataset of claims, perspectives and evidence, making use of online debate websites to create the initial data collection, and augmenting it using search engines in order to expand and diversify our dataset. We use crowd-sourcing to filter out noise and ensure high-quality data. Our dataset contains 1k claims, accompanied with pools of 10k and 8k perspective sentences and evidence paragraphs, respectively. We provide a thorough analysis of the dataset to highlight key underlying language understanding challenges, and show that human baselines across multiple subtasks far outperform ma-chine baselines built upon state-of-the-art NLP techniques. This poses a challenge and opportunity for the NLP community to address.

1 Introduction

Search and recommendation systems retrieve claim-relevant, popular information rather than ensuring perspective diversity or evidential support. The paper frames substantiated perspective discovery as an NLP challenge and introduces PERSPECTRUM to study it.

1 Introduction

  • Search and recommendation systems prioritize query relevance and document popularity, not the diversity or evidential support of perspectives.The authors argue that understanding nontrivial claims requires insights from various perspectives.Source: §1 · p. 1
  • Substantiated perspective discovery seeks a diverse set of nonredundant perspectives, each taking a stance on a claim and supported by evidence.The authors characterize a perspective as an opinion, possibly conditional, for or against a claim.Source: §1 · p. 2
  • The dataset is intended to support study of core NLP challenges, not to provide an exhaustive set of perspectives or evidence or assess source credibility.The paper treats credibility and trustworthiness as important but orthogonal issues.Source: §1 · p. 2
  • PERSPECTRUM provides pools of perspectives and evidence paragraphs for a given claim, which a system is expected to select and connect.The dataset was seeded with online debate content and augmented with search data and paraphrases, then crowdsourced to improve quality and reduce annotation noise.Source: §1 · p. 2
  • The paper formulates multiple NLP tasks, reports high human scores on them, and develops competitive baselines using state-of-the-art techniques.These contributions accompany the dataset construction.Source: §1 · p. 2

2 Design Principles and Challenges

The paper decomposes substantiated perspective discovery into interdependent tasks, while assuming the input claims are already discussion-worthy. The tasks include finding relevant perspectives, resolving claim-dependent equivalence, assigning stance, and finding supporting evidence.

2 Design Principles and Challenges

  • The task decomposition assumes argue-worthy claims as input, leaving systems to distinguish disputed claims from those with straightforward responses outside this work.The authors note that practical systems need to understand argumentative structures to make that distinction.Source: §2 · p. 3
  • Perspective discovery requires finding argumentative sentences that directly address a claim, not merely content that is topically related.The paper illustrates this distinction with perspectives related to animal use that do not address a claim about animal use in entertainment.Source: §2 · p. 3
  • Perspective equivalence is claim-dependent, so identifying a minimal diverse set requires more than generic paraphrase detection.Two perspectives may be equivalent for one claim but not another.Source: §2 · p. 3
  • Systems must classify each perspective's stance relative to the claim, including supporting, opposing, and other stance categories.The paper describes stance classification as an assessment of perspectives with respect to a given claim.Source: §2 · p. 3
  • Systems must find evidence supporting each perspective, with entailment decisions dependent on the associated claim.Source: §2 · p. 3

3 Related Work

The paper situates perspective discovery among work on fact verification, stance classification, and argumentation. It distinguishes its task by combining claim-relative perspectives and evidence in a dataset constructed from online content and crowdsourcing.

3 Related Work

  • Perspective verification from evidence is encompassed by the paper's problem structure, linking perspective discovery to fact verification.The related-work discussion connects fact verification with textual entailment and evidence identification.Source: §3 · p. 3
  • Stance classification is a component of this work, but its pool of relevant reasons is not restricted as in Hasan and Ng (2014).Source: §3 · p. 3
  • Argumentation research has addressed formal argument structures and extracting argumentative elements and their internal relations from free text.The discussion cites Teufel et al. (1999), Toulmin (2003), Freeman (2011), Palau and Moens (2009), Khatib et al. (2016), and Ajjour et al. (2017).Source: §3 · p. 3
  • Earlier claim-perspective-evidence datasets are characterized as small or noisy; this work combines online content and crowdsourcing to construct a sizable, high-quality dataset.The cited examples include Aharoni et al. (2014) and Wachsmuth et al. (2017); Hua and Wang (2017).Source: §3 · p. 4

4 The PERSPECTRUM Dataset

PERSPECTRUM is built through staged crowd verification and targeted augmentation, then characterized by its clustered perspectives, topic breadth, and language-understanding demands.

4.1 Dataset construction

  • Initial debate-site data contained ∼1k claims, ∼8k perspectives, and ∼8k evidence paragraphs, and was significantly noisy.Source: §4.1 · p. 4
  • Workers verified perspective sentences for completeness and clear claim-relative stance, using five labels that were collapsed to three for agreement measurement.Low-agreement perspectives were filtered; the stated retention threshold was ρ ≥0.5.Source: §4.1 · pp. 4–5
  • 94% accuracy on 328 perspectives and 87.7% accuracy for adjudicated evidence–perspective labels were reported against AMT annotations.Perspective accuracy was assessed on 328 perspectives for 10 claims; evidence-label accuracy was assessed on four evidence–perspective pairs.Source: §4.1 · p. 5
  • Paraphrasing yielded ∼4.5 paraphrased perspectives; web search added similar claim-related sentences, and expert review refined equivalent clusters and added shared perspectives across similar claims.Source: §4.1 · p. 5
  • Evidence verification retrieved each paragraph’s top eight candidate perspectives for annotators to judge whether the paragraph sufficiently supported a perspective.Each HIT was annotated by at least four independent annotators.Source: §4.1 · p. 5

4.2 Statistics on the dataset

  • ∼12k perspectives form equivalence clusters averaging 2.3 perspectives, alongside about 1k claims.Source: §4.2 · p. 5
  • Politics, World, and Society were the largest claim-topic groups, while the dataset covered 10+ topics.Source: Figure 3 · p. 6, §4.2 · p. 6

4.3 Required skills

  • Stance classification requires commonsense understanding and coreference resolution, including event and entity coreference.Source: §4.3 · p. 6

5 Empirical Analysis

The empirical analysis evaluates perspective discovery through separate tasks for extracting perspectives, labeling stance, grouping equivalent perspectives, and retrieving evidence. Machine systems are compared with human performance, with reported gaps between them across the tasks.

5 Empirical Analysis

  • Claims sharing a topic stay in one split across 60/15/25 train/dev/test, limiting topic-keyword leakage between splits.Source: §5 · p. 6

5.1 Systems

  • IR retrieves ranked matches, BERT provides a contextual baseline, and two experts estimate human performance on 10 test claims.Source: §5.1 · p. 6

5.2 Evaluation metrics.

  • Systems use shared perspective and evidence pools; overall performance multiplies T1, T2, and T4 measures, without modeling error propagation through the pipeline.Source: §5.2 · pp. 6–7

5.3.1 Minimal perspective extraction (T1)

  • IR top-15 candidates exceed 90% recall; two annotators then select a minimal set with no equivalent perspectives.Source: §5.3.1 · p. 8

5.3.2 Perspective stance classification (T2)

  • BERT is about 20% below human on stance classification; the authors suggest commonsense reasoning may explain the gap and omit IR because retrieval is unlikely to separate stances.Source: §5.3.2 · p. 8

5.3.3 Perspective equivalence (T3)

  • BERT gains ~36% over the IR baseline on perspective equivalence, yet remains ~20% behind human performance.Source: §5.3.3 · p. 8

5.3.4 Extraction of supporting evidence (T4)

  • Top-60 evidence candidates exceed 85% recall; BERT trains on three-sentence gold windows, but evidence-task performance remains lower than human.At runtime, a development-tuned sentence-positive threshold determines whether the whole paragraph is treated as evidence.Source: §5.3.4 · p. 8

6 Discussion

The paper aims to improve information organization and access, hoping that access to more diverse information can address over-personalization. It also identifies limits in the dataset and leaves several components of a complete perspective-discovery pipeline unaddressed.

6 Discussion

  • The paper hopes that better organization and access to diverse information can address over-personalization.Information pollution and over-personalization are described as having detrimental effects.Source: §6 · pp. 8–9
  • The dataset is neither exhaustive nor representative of the world's important claims and perspectives, and it does not label expertise or trustworthiness.Crowd workers' prior beliefs may influence their judgments of validity; creators' intentions, such as offensive content, are not used as a filtering criterion.Source: §6 · p. 9
  • The work leaves claim validation, argument extraction, factuality, and trustworthiness outside its main focus.The complete pipeline would first verify that claims are well-phrased and arguable and construct perspective pools by extracting relevant arguments; factuality and trustworthiness are treated as separate issues.Source: §6 · p. 9
  • The authors hope future work will address some of these challenges and limitations.Source: §6 · p. 9

7 Conclusion

The paper defines substantiated perspective discovery, formulates the language-understanding tasks involved, and builds a dataset and supervised baselines. It assumes valid claim sentences and defers claim-structure recognition and credibility issues.

7 Conclusion

  • The paper defines substantiated perspective discovery and characterizes the language-understanding tasks needed to address it.Source: §7 · p. 9
  • It combines online resources, web data, and crowdsourcing to create a dataset, and builds and evaluates supervised baseline systems.Source: §7 · p. 9
  • The systems assume valid claim sentences, leaving argumentative-structure recognition and trustworthiness and credibility issues for future work.Source: §7 · p. 9

A.1 Statistics

The section identifies the sources and stages whose data sizes are reported in Table 4, without stating their values in the supplied text.

A.1 Statistics

  • Table 4 reports dataset statistics for data collected from online debate websites, data filtered out, paraphrase additions, and web-added perspective candidates.Source: §A.1 · p. 13, Table 4 · p. 13
  • The reported statistics include the size of the initial collection from online debate websites.Source: §A.1 · p. 13
  • The statistics also cover filtered data and perspective candidates added through paraphrases and web search.Source: §A.1 · p. 13

A.2 Measure of agreement

The measure of agreement summarizes how raters' assignments are distributed across categories. Its interpretation ranges from concentrated assignments to uniform assignments, and it can also be related to pairwise agreement and a chi-square statistic.

A.2 Measure of agreement

  • For a fixed subject, n_j counts raters assigning it to category j, and ρ can be interpreted as the fraction of rater pairs in agreement.Source: §A.2 · p. 13
  • ρ is 1.0 when all raters assign a subject to one category and 0.0 when assignments are uniform across categories.The measure captures concentration in the category-count vector.Source: §A.2 · p. 13
  • The measure is used in calculating more extensive agreement measures, including Fleiss’ kappa, and can be expressed in terms of a conventional chi-square statistic.Source: §A.2 · p. 13
  • Equivalently, ρ gives the probability that two randomly selected elements from the category sets belong to the same set.Source: §A.2 · p. 13

A.3 crowdsourcing interfaces

The figures identify interfaces for annotating claim topics and for verifying perspectives, annotating evidence, and generating perspective paraphrases.

A.3 crowdsourcing interfaces

  • The annotator interfaces cover perspective verification, evidence annotation, and perspective-paraphrase generation.These are labeled steps 2a, 3a, and 2b, respectively.Source: Figure 9 · p. 15
  • The perspective workflow separates verification and paraphrase generation from evidence annotation.Verification and paraphrase generation are labeled step 2a and step 2b; evidence annotation is step 3a.Source: Figure 9 · p. 15
  • A separate interface is used to annotate claim topics.Source: Figure 10 · p. 16

Related papers

Loading 1906.03538v1…