PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantifying Controversy in Social Media

Kiran Garimella, Gianmarco De Francisci Morales, Aristides Gionis, Michael Mathioudakis

arXiv:1507.05224v5cs.SI

TL;DR

The paper asks how to identify and quantify controversial social-media topics beyond domain-specific political case studies. It introduces a graph-based pipeline that builds conversation graphs, partitions them into potential sides, and measures controversy. The random-walk-based measure best separates controversial from non-controversial topics, while simple content features perform poorly and sentiment variance is promising.

  • Problem

    Existing controversy studies rely heavily on political, carefully curated case studies, motivating a general framework that works across domains without prior domain-specific knowledge.

  • Method

    The paper builds topic conversation graphs, partitions users into two potential sides, and computes controversy measures using graph, network, content, endorsement, and sentiment features.

  • Results

    The random-walk-based RWC most neatly separates controversial from non-controversial topics and generalizes to datasets from other domains and previous studies.

  • Takeaways & Limitations

    Domain-agnostic graph measures can quantify controversy, while simple content representations do not work in general and sentiment analysis offers promising results.

  • Takeaways & Limitations

    The main experiment uses only 20 topics, creating a real risk of overfitting despite development on 40% of topics and agreement with external results.

Abstract

from arXiv · show

Which topics spark the most heated debates on social media? Identifying those topics is not only interesting from a societal point of view, but also allows the filtering and aggregation of social media content for disseminating news stories. In this paper, we perform a systematic methodological study of controversy detection by using the content and the network structure of social media. Unlike previous work, rather than study controversy in a single hand-picked topic and use domain specific knowledge, we take a general approach to study topics in any domain. Our approach to quantifying controversy is based on a graph-based three-stage pipeline, which involves (i) building a conversation graph about a topic; (ii) partitioning the conversation graph to identify potential sides of the controversy; and (iii) measuring the amount of controversy from characteristics of the graph. We perform an extensive comparison of controversy measures, different graph-building approaches, and data sources. We use both controversial and non-controversial topics on Twitter, as well as other external datasets. We find that our new random-walk-based measure outperforms existing ones in capturing the intuitive notion of controversy, and show that content features are vastly less helpful in this task.

1 INTRODUCTION

The paper develops a domain-agnostic framework for quantifying controversy across social-media topics, addressing the limitations of political case studies built with domain-specific knowledge. Its graph-based approach compares conversation structure, content, sentiment, and endorsement features to identify controversial topics.

  • Previous studies largely examine political controversies through case studies using carefully curated datasets and extensive domain knowledge.
  • The proposed framework identifies and compares controversy for topics in any domain without prior domain-specific knowledge.
  • Twitter topics are operationalized as text queries, with related activity collected from posts containing the query or closely related hashtags.
  • Endorsement, social-network, content, and sentiment features are evaluated; endorsement features are most useful on Twitter, while content-based features are not.
  • The random-walk-based measure discriminates controversial topics accurately, generalizes to external datasets, and sentiment variance reliably indicates controversy.
  • The pipeline builds a conversation graph, identifies potential controversy sides, and quantifies controversy from graph characteristics.

2 RELATED WORK

Prior controversy research emphasizes political case studies and specialized data, whereas this paper targets domain-independent quantification across topics. It builds on graph, content, and sentiment approaches while evaluating their broader applicability.

  • Earlier work studies controversy across blogs, opinion fora, and especially Twitter, with much of it centered on political debates.
  • The paper extends controversy analysis to arbitrary, including short-lived and ad-hoc, social-media topics rather than domain-specific political cases.
  • Existing graph-based studies use modularity, graph partitioning, or boundary analysis to identify or measure polarized communities.
  • Other approaches quantify polarity through influential-user opinion propagation or signed bipartite opinion graphs, often in specialized settings.
  • The paper assumes controversial topics induce clustered graphs representing distinct opinions, consistent with an echo-chamber view of social interaction.
  • Text and sentiment methods have limitations for short, noisy Twitter content, but the paper evaluates content features and reports that its method outperforms existing measures.

3 PIPELINE

The controversy-scoring pipeline converts topic-related social-media activity into a conversation graph, partitions users into two potential sides, and computes a score reflecting their separation.

  • The pipeline consists of graph building, graph partitioning, and controversy measurement, producing a value whose higher values indicate greater controversy.
  • Graph building represents topic-related activity with users as vertices and agreement, endorsement, or shared viewpoints as edges.
  • Topics may be defined by a keyword or related hashtags, with the resulting query selecting matching social-media activity.
  • Graph partitioning extracts two disjoint user groups that potentially correspond to different sides of the discussion.
  • The final stage computes controversy measures intended to capture how well separated the two partitions are, including random-walk, betweenness, and embedding-based measures.

4 GRAPH BUILDING

The study builds Twitter conversation graphs from domain-agnostic hashtag topics, expands seeds using normalized hashtag similarity, and compares retweet, follow, and content-based graph constructions. These choices aim to capture opposing activity while reducing noise and overly narrow topic definitions.

  • The datasets use Twitter posts and a balanced selection of controversial and non-controversial topics, with statistics reported for follow and retweet graphs.
  • From hashtags to topics: A topic is expanded from a seed hashtag into related hashtags so opposing sides using different labels are less likely to be missed.
  • From hashtags to topics: Related hashtags are ranked by a similarity function combining co-occurring words and hashtags, normalized by hashtag popularity through inverse document frequency.
  • From hashtags to topics: Using α = 0.3 and k = 20, the method retrieves the top-20 related hashtags and filters tweets containing at least one topic hashtag.
  • From hashtags to topics: The related-hashtag sets for #baltimoreriots and #netanyahuspeech illustrate that opposing sides can appear within one expanded topic.
  • Data aspects: Retweet edges require at least two retweets between users, while topic-level graphs union separately filtered retweet graphs for each hashtag.
  • Data aspects: Follow graphs connect users through follower relations, relying on homophily to reflect agreement but requiring expensive crawling of Twitter’s social network.
  • Data aspects: Content graphs connect users sharing non-topic hashtags, URLs, or URL domains, enabling comparison of content-based activity with network interactions.

5 GRAPH PARTITIONING

The paper partitions conversation graphs into two candidate sides and evaluates how well those partitions reflect controversy. METIS produces the cleanest partitions, while content-based and hybrid graph approaches yield noisy, overlapping structures.

  • Graph partitioning: METIS partitions conversation graphs into two sides, visualized in blue and red using retweet and follow graph structures.The visualization layout is based solely on graph structure, not METIS’s partitioning.
  • Graph partitioning: Partitions on retweet and follow graphs match intuitive controversy judgments, with controversial topics producing well-separated groups.The authors also checked alternative layouts and manually sampled tweets from partitions.
  • Graph-building comparison: Content-based graph variants produce sparse graphs and highly overlapping partitions, even for highly controversial issues.The hybrid approach shows the same noisy pattern, including a variant based on URL-domain frequencies.
  • Partitioning algorithms: Other partitioning methods do not differ significantly, but METIS generates the cleanest partitions by visual inspection.The comparison includes spectral clustering, label propagation, and affiliation-graph-based models.

6 CONTROVERSY MEASURES

The paper compares graph-based controversy measures, emphasizing a random-walk formulation that models exposure across two inferred sides. Its restart-based implementation preserves the original score while making computation substantially faster.

  • Random-walk controversy: The random-walk measure models how likely users on either side are to encounter authoritative content from the opposing side.High-degree vertices serve as proxies for authoritative users.
  • Random-walk controversy: RWC compares same-side and cross-side random-walk probabilities, approaching one when crossing between sides is unlikely.The probabilities are conditioned on ending partitions to avoid skew from partition sizes and total degree.
  • Efficient RWC variant: The restart-based RWC variant handles dangling vertices and directed, star-like retweet graphs by restarting walks at high-degree vertices.This preserves directionality and accommodates authoritative users that generate information spreading through the graph.
  • Efficient RWC variant: Pearson’s r = 0.96 between Monte Carlo and RWR scores, while the RWR algorithm is up to 200 times faster on the evaluated datasets.The new variant is reported as almost identical to the original score while reducing computational cost.
  • Other controversy measures: Betweenness Centrality Controversy compares edge-betweenness distributions on the METIS cut and the rest of the graph using KL divergence.Kernel density estimation and 10 000 samples are used to estimate and compare the distributions.
  • Other controversy measures: Embedding Controversy uses average within-partition and across-partition distances from a ForceAtlas2 embedding to characterize separation.The measure is inspired by the Davies-Bouldin index.

7 CONTROVERSY SCORES FOR USERS

The paper extends controversy quantification from conversation graphs to individual users, assigning scores that indicate neutrality or bias toward one side. It proposes random-walk-based user measures and compares them with BiasWatch and human annotations.

  • User-level controversy scores: User controversy scores range from −1 to 1, with 0 neutral and ±1 representing the extremes for each side.Users interacting with both sides score near zero, while consistent activists score near an extreme.
  • RWCuser: RWCuser estimates how often a random walk starting and restarting at user u reaches high-degree vertices on either controversy side.The graph is partitioned into disjoint sides X and Y, with high-degree vertices treated as teleportation points.
  • Expected hitting time: Expected hitting time compares each user’s expected random-walk steps to reach high-degree vertices on sides X and Y.Users closer to X and farther from Y receive scores approaching 1, while the reverse pattern approaches −1.
  • Comparison with BiasWatch: BiasWatch uses manually selected seed hashtags, hashtag expansion, partisan identification, and label propagation to score users.The paper compares its user measures with BiasWatch on hashtag-based datasets and additional datasets from Table 2.
  • Human validation: κ = 0.35 indicates fair agreement between the paper’s discretized user scores and human annotations across five categories.The annotations covered 500 randomly selected users and used a controversy-score range of [−2,2].
  • Comparison with BiasWatch: The proposed approach performs at par with BiasWatch without using content information, while RWCuser provides an intuitive user-level extension of RWC.The unified formulation is presented as useful for defining objective functions in efforts to reduce controversy.

8 EXPERIMENTS

The experiments evaluate graph-based controversy measures across Twitter topics, external datasets, evolving conversations, and synthetic graphs. RWC consistently separates controversial from non-controversial topics and behaves plausibly as graph structure and controversy evolve.

  • Experimental design: The evaluation tests controversy measures on retweet and follow graphs, Twitter topics, external datasets, and an evolving Venezuelan Twitter dataset.Content and hybrid graph-building approaches are omitted because they do not perform well.
  • Experimental design: Eight graphs—four controversial and four non-controversial—were used as a development testbed to reduce potential overfitting.The procedure resembles a 40/60% train/test split in traditional machine-learning applications.
  • Twitter hashtag evaluation: RWC is the best measure on the paper’s datasets, while GMCK and MBLB fail to separate the controversial and non-controversial groups reliably.BCC and EC show varying separation and overlap, with EC slightly better because its distributions are more concentrated.
  • Twitter hashtag evaluation: All measures have higher median scores for controversial than non-controversial topics, supporting both retweet and follow graphs as useful graph-building methods.The result is consistent across topics with varied provenance and characteristics.
  • External datasets: Across external datasets, the methods generally distinguish controversial graphs, except for Karate club, where RWC and MBLB report low controversy scores.The authors suggest the graph may be too small for random-walk-based measures, while BCC captures the desired behavior.
  • External datasets: The pipeline identifies two opposing communities in the Gun control dataset without the external help needed to manually distinguish three partitions.All measures agree that the topic is highly controversial.
  • Evolving controversy: RWC and EC correlate closely with the evolving-event reference measure, with Pearson r values of 0.944 and 0.949, respectively.All proposed measures show the event-related dip and later increase, whereas GMCK remains nearly constant with r = 0.542.
  • Robustness analysis: RWC increases with within-community density and decreases as inter-community noise increases in synthetic planted-community graphs.For fixed p1, smaller p2 yields a larger RWC score because the communities are more separated.

9 CONTENT

The paper tests whether textual content alone can distinguish controversial from non-controversial topics. Bag-of-words features provide insufficient signal, while sentiment variance separates the groups but has practical limitations.

  • Content-based measures: Content-only evaluation uses bag-of-words and sentiment features extracted from social-media discussions.The analysis asks whether textual content helps measure topic controversy without user-interaction structure.
  • Bag-of-words: At p = 0.05, the Wilcoxon rank-sum test cannot reject equal bag-of-words measures between controversial and non-controversial topics.The representation therefore lacks enough signal to distinguish the topic classes confidently.
  • Sentiment analysis: Controversial topics have sentiment variance of at least 2, whereas non-controversial topics have variance of at most 1.5.The result indicates stronger variation in the tones used to debate controversial topics.
  • Sentiment analysis: Sentiment analysis detects a clear variance signal, but incorporating it into graph-based measures is not straightforward.The feature also depends on technologies that do not work reliably for languages other than English, excluding topics such as #russia_march.

10 DISCUSSION

The discussion reports important scope limitations and negative results, while concluding that random-walk-based controversy measurement is reliable across several data settings. It also identifies applications and future directions for measuring polarization and improving users’ exposure to diverse content.

  • Limitations: Twitter is the main data source, although the measures generalize well to datasets from other social media and the Web.
  • Limitations: Manual selection of controversial topics may introduce bias, while ground truths for controversial, especially ephemeral, issues are difficult to obtain.
  • Limitations: Using only 20 topics creates a real risk of overfitting, which the authors reduce by developing measures on 40% of the topics and comparing with external results.
  • Limitations: The pipeline depends on graph partitioning, whose quality is fundamental, and the authors rely on off-the-shelf techniques because the problem is hard.
  • Limitations: Two-sided partitioning does not directly cover multifaceted controversies; extending the principles requires automatically selecting the optimal number of partitions.
  • Negative results and conclusions: Simple cuts, conductance, modularity, assortativity, and some partitioning-free approaches produced poor, mixed, or unreliable results, whereas RWC most neatly separates controversial from non-controversial topics.
Loading 1507.05224v5…