Source-linked AI summary
The effect of wording on message propagation: Topic- and author-controlled natural experiments on Twitter
Chenhao Tan, Lillian Lee, Bo Pang
TL;DR
The paper asks whether wording affects message propagation independently of author and topic, a question rarely isolated in prior work. It uses same-user, same-URL tweet pairs as a natural experiment and evaluates human and computational predictions. The computational methods outperform both average human performance and a strong non-controlled-data baseline, while humans remain above chance.
Problem
The paper addresses the limited evidence on wording effects because author popularity and topic commonly confound social-media propagation.
Method
The study compares alternative wordings in same-user tweets containing the same URL, then tests wording features with human judgments and prediction experiments.
Results
66.5% cross-validation accuracy and 65.6% held-out accuracy were achieved, exceeding average human accuracy and comparison algorithms including a strong non-TAC classifier.
Takeaways & Limitations
Helpful wording heuristics include adding information, aligning language with community norms and prior messages, and mimicking news headlines.
Takeaways & Limitations
The paper leaves generalization to longer, more extensive arguments and the psychological and cultural mechanisms underlying feature effectiveness for future work.
Abstract
from arXiv · showhide
Consider a person trying to spread an important message on a social network. He/she can spend hours trying to craft the message. Does it actually matter? While there has been extensive prior work looking into predicting popularity of social-media content, the effect of wording per se has rarely been studied since it is often confounded with the popularity of the author and the topic. To control for these confounding factors, we take advantage of the surprising fact that there are many pairs of tweets containing the same url and written by the same user but employing different wording. Given such pairs, we ask: which version attracts more retweets? This turns out to be a more difficult task than predicting popular topics. Still, humans can answer this question better than chance (but far from perfectly), and the computational methods we develop can do better than both an average human and a strong competing method trained on non-controlled data.
1 Introduction
The paper isolates whether wording affects message propagation by comparing tweets from the same author about the same URL. Humans can predict the better-worded version above chance, and the proposed classifier outperforms baselines.
- Research question: Wording—how a message is said rather than what it concerns—is one of the few propagation factors directly under an author’s control.The paper contrasts wording with author identity, social network, timing, and topic.
- Research question: The study uses same-user tweets containing the same URL as a natural experiment controlling for author and fine-grained topic.These pairs can differ substantially in wording while referring to the same linked content.
- Human prediction: 61.3% average human accuracy shows that people can identify which wording attracts more retweets better than chance, but far from perfectly.The controlled prediction task is more difficult than predicting popularity when topic and author vary freely.
- Computational approach: The experiments test wording features such as informativeness, headline resemblance, and conformity to community norms through hypothesis testing and fresh held-out prediction.The held-out data were used only once after standard cross-validation experiments.
- Computational approach: 66.5% cross-validation accuracy and 65.6% held-out accuracy came from combining custom features with bag-of-words.The classifier significantly outperformed baselines, including a strong model trained with author and timing metadata on non-controlled data.
2 Related work
Prior work links message propagation to authors, networks, timing, content, and topic, while controlled studies examine communication strategies and changes in descriptions. This paper focuses specifically on wording under topic and author control.
- Controlled communication studies: Earlier controlled communication studies examined effective strategies, article-sharing rates, description changes, and textual meme variation.The cited work provides precedents for controlled experiments but does not establish this paper’s specific wording comparison.
- This paper’s focus: The paper’s contribution is to study alternative wording while holding topic and author constant, rather than predicting popularity with those factors freely varying.This framing distinguishes the work from non-controlled propagation prediction.
3 Data
The dataset consists of large-scale same-user, same-URL tweet pairs, with controls designed to reduce timing and follower-count confounding. Identical pairs validate the selected comparison setting.
- Dataset construction: The main dataset contains 1.77M topic- and author-controlled tweet pairs differing in more than spacing.Crawling 236K user timelines also produced 632K spacing-only pairs and 558M unpaired tweets for auxiliary resources.
- Dataset construction: Within each pair, t1 is the earlier tweet and t2 the later tweet, while n1 and n2 denote their respective retweet counts.The tweet with higher ni is called better and the one with lower ni worse.
- Confound checking: The study uses 632K identical pairs to estimate temporal and follower-count confounding, since identical wording should otherwise yield equal retweet rates.The estimate conditions on n1 because pairs with larger n1 become exponentially rarer.
- Confound checking: The chosen setting is t2 within 12 hours of t1 and the author having more than 5,000 followers, where identical-pair retweet counts are on average close.This setting minimizes confounding while retaining substantial data.
- Pair selection: Filtering removes pairs with insufficient textual change and retains only pairs with sufficiently large retweet-count differences.The similarity filter uses tf-based cosine similarity, and the remaining pairs are selected from the extremes of n2 − n1.
4 Human accuracy on TAC pairs
Human judgments on topic- and author-controlled tweet pairs show that wording alone provides a detectable but imperfect signal of retweet success. Agreement strength substantially improves accuracy for pairs with dominant majority responses.
- Experimental setup: The AMT pilot asked subjects to choose which of two randomly reordered controlled tweets other people would be more likely to retweet.Each task used disjoint samples of 20 pairs, with subjects also flagging close calls and justifying their choices.
- Results: 62.4% accuracy across 3,900 binary judgments rose to 63.8% after excluding 587 judgments marked as close calls.These figures treat individual judgments as independent responses.
- Results: 73% accuracy was achieved by the majority response across the 39 judgments per pair.Majority accuracy exceeded 90% when at least 80% of judgments agreed, although fewer than one-third of pairs met that criterion.
- Results: 61.3% was the average accuracy across 106 subjects, better than chance but far from 100%.Subject performance varied substantially: one reached 85%, while eight scored below 50%.
- Interpretation: The controlled task is harder than the 73.8%-81.2% accuracy reported for a similar experiment using non-topic- or author-controlled pairs.The comparison indicates that controlling these factors makes propagation prediction more challenging.
5 Experiments
The experiments evaluate wording features for retweet efficacy and author preference, then test whether they predict the better tweet in topic- and author-controlled pairs. Custom features combined with n-gram features outperform baselines, including a strong classifier trained on non-controlled data.
- 5.1 Features: efficacy and author preference: Explicit requests to share help attract more retweets, using verb occurrences such as “retweet,” “please,” and “spread” as the feature.
- 5.1 Features: efficacy and author preference: More informative messages, including longer text and content-related parts of speech, perform better, although simply adding meaningless text is not expected to help.
- 5.1 Features: efficacy and author preference: Authors’ wording changes only partly align with feature efficacy: they increase ineffective @-mentions and second-person pronouns while failing to increase some effective features.
- 5.1 Features: efficacy and author preference: 39 custom features span lexicon, informativeness, language-model, retweet-score, and readability groups for evaluating wording efficacy and author preference.The feature groups cover explicit sharing requests, informativeness, conformity, headline resemblance, and words associated with retweeted tweets.
- 5.2 Predicting the “better” wording: 66.5% cross-validation accuracy and 65.6% held-out accuracy are achieved by combining custom features with bag-of-words, outperforming a strong non-TAC classifier and other baselines.
- 5.2 Predicting the “better” wording: A non-TAC classifier reaches around 55% on alternate-version selection despite 98.8% accuracy on unpaired most- and least-retweeted tweets, showing the controlled task differs from popularity prediction.
6 Conclusion
The paper presents a large-scale topic- and author-controlled study of wording effects on information propagation. Its wording features outperform comparison algorithms and estimated average human performance, while their generalization to longer arguments remains future work.
- The study provides a large-scale topic- and author-controlled experiment on how wording affects information propagation.
- The developed features outperform all comparison algorithms, including one using author and timing features trained on non-TAC data, and exceed estimated average human performance.
- Helpful wording heuristics include adding information, aligning language with community norms and prior messages, and mimicking news headlines.
- The generalization of these features to longer and more extensive arguments remains an open direction for future work.