Source-linked AI summary
TweetCred: Real-Time Credibility Assessment of Content on Twitter
Aditi Gupta, Ponnurangam Kumaraguru, Carlos Castillo, Patrick Meier
TL;DR
During crisis events, rumors and fake content spread quickly on Twitter, while judging tweet credibility is difficult in rapidly evolving situations. TweetCred addresses this with a real-time semi-supervised ranking system; 80% of scores appeared within 6 seconds, and 63% of users agreed with or nearly agreed with the scores.
Problem
Rumors and fake news spread quickly on Twitter during major events, making credible information difficult to identify during rapidly evolving crises.
Method
TweetCred uses a semi-supervised ranking algorithm trained on crowdsourced human labels to assign each timeline tweet a 1–7 credibility score using more than 45 real-time computable features.
Results
80% of credibility scores were computed and displayed within 6 seconds, while 63% of users agreed with or disagreed by only 1 or 2 points with the automatically generated scores.
Takeaways & Limitations
TweetCred’s deployment suggests that users expect technologies supporting credibility evaluation and that automatic credibility ratings can reach users at large scale.
Takeaways & Limitations
User feedback and observations identify unresolved challenges in personalization and context, including tweets that do not convey factual information.
Abstract
from arXiv · showhide
During sudden onset crisis events, the presence of spam, rumors and fake content on Twitter reduces the value of information contained on its messages (or "tweets"). A possible solution to this problem is to use machine learning to automatically evaluate the credibility of a tweet, i.e. whether a person would deem the tweet believable or trustworthy. This has been often framed and studied as a supervised classification problem in an off-line (post-hoc) setting. In this paper, we present a semi-supervised ranking model for scoring tweets according to their credibility. This model is used in TweetCred, a real-time system that assigns a credibility score to tweets in a user's timeline. TweetCred, available as a browser plug-in, was installed and used by 1,127 Twitter users within a span of three months. During this period, the credibility score for about 5.4 million tweets was computed, allowing us to evaluate TweetCred in terms of response time, effectiveness and usability. To the best of our knowledge, this is the first research work to develop a real-time system for credibility on Twitter, and to evaluate it on a user base of this size.
1 Introduction
Twitter provides rapid crisis information but also spreads rumors and fake content, making real-time credibility assessment difficult. TweetCred addresses this need with a semi-supervised ranking system deployed as a browser extension and evaluated at scale.
- Twitter can disseminate crisis information rapidly, assisting emergency responders, the public, and relief coordination.
- Rumors and fake content spread quickly on Twitter and can adversely affect thousands of people during crisis events.
- TweetCred assigns real-time credibility scores to tweets using ranking techniques and defines credibility as whether users would trust or believe the information is true.
- Unlike prior post-hoc classification approaches, TweetCred uses data available from each message without assuming extensive historical or complete user or event data.
- TweetCred uses a semi-supervised SVM-rank model trained on data from six 2013 crisis events and 45 single-tweet features.
- 1,127 Twitter users used TweetCred for three months, producing credibility scores for about 5.4 million tweets.
- 80% of credibility scores were computed and displayed within 6 seconds, while 63% of users agreed with scores or disagreed by only 1 or 2 points on a 1-to-7 scale.
2 Survey
Prior work studied credibility, expertise, and information diffusion on Twitter through automated analysis. TweetCred is positioned as the first practical Twitter credibility system deployed and evaluated with real users.
- Credibility Assessment: Automated classification has been used to distinguish news from conversational topics and assess credibility from Twitter features, achieving 70–80% precision and recall.
- Credibility of users: Research has ranked Twitter information sources by topic using trust and domain expertise, and identified topic-based experts from user-created lists.
- System: Truthy computed trustworthiness scores for public Twitter streams related to events, focusing on political smears, astroturfing, and politically motivated disinformation.
- System: The paper describes the creation and deployment of a practical Twitter credibility system evaluated with real users, which the authors identify as a first of its kind.
3 Training Data Collection
TweetCred’s training data combines crisis-event tweets with crowdsourced labels for event relevance and credibility. The resulting labels distinguish informative, related-but-non-informative, and unrelated content before credibility scoring.
- TweetCred requires a training set of tweets with known credibility labels because it uses semi-supervised learning.
- Training data came from Twitter’s streaming API, filtered by keywords for six prominent 2013 crisis events.
- Around 500 tweets per event were labeled through CrowdFlower by three United States annotators, using the majority label.
- The first annotation step classified tweets as informative, event-related but non-informative, unrelated, or skipped.
- 45% of tweets were informative, 40% were related but non-informative, and 15% were unrelated to their event.
- The informative tweets were then labeled for credibility using categories ranging from definitely credible to definitely incredible.
4 Credibility Modeling
TweetCred ranks tweets by credibility using real-time features and compares several learning-to-rank algorithms with NDCG and execution time. SVM-rank was implemented because its ranking time was under one second and its learning time was shorter than the alternatives.
- Feature Extraction: The model uses semi-supervised learning-to-rank after extracting features from individual tweets in real time.The feature set is restricted to information derivable from single tweets, avoiding extensive historical or complete user or event data.
- Feature Extraction: 45 features combine tweet fields, author metadata, and Web of Trust reputation scores for tweets containing URLs.Examples include posting date and the author's follower count.
- Learning Scheme: The evaluated ranking methods are Coordinate Ascent, AdaRank, RankBoost, and SVM-rank.These methods represent standard learning-to-rank approaches, including pairwise ranking and boosting-based techniques.
- Evaluation: NDCG measures ranking quality for multiple relevance grades, while running time measures computational cost.NDCG normalizes discounted cumulative gain against the ideal ranking, and test-set NDCG averages query-level values.
- Evaluation: AdaRank and Coordinate Ascent achieved the best NDCG@n, with SVM-rank a close second and smaller gaps deeper in the result list.All methods ranked in under one second; SVM-rank had the shortest learning time, motivating its implementation for future user-feedback retraining.
- Feature Importance: The SVM-rank model's top features were predominantly tweet-based rather than user attributes.The listed features include character and word counts, retweets, tweet age, URLs, and author-status ratios.
5 Implementation and Performance Evaluation
TweetCred is delivered primarily through a browser extension, with server-side APIs retrieving tweet data, computing features and SVM-rank scores, and returning ratings to the timeline. In deployment, response times were generally under 6–10 seconds, while an initial pilot informed the rating scale and interface feedback mechanism.
- 5.1 Implementation: The system was provided as a browser extension, web application, and REST API, with the browser extension used most commonly.Heavy computation was offloaded to the server to minimize browser memory and CPU load.
- 5.1 Implementation: The browser extension sends displayed tweet IDs to a server, which retrieves tweet data through Twitter’s API before scoring each tweet.The system passes IDs rather than scraping page HTML because scoring requires more information than Twitter’s interface displays.
- 5.1 Implementation: TweetCred generates feature vectors, predicts scores with SVM-rank, rescales them from 1 to 7, and returns them to the browser.Scores are cached for 15 minutes after computation.
- 5.1 Implementation: An initial pilot with 10 avid Twitter users found the 1–5 scale difficult to differentiate, motivating a revised rating presentation.Eight of the ten participants were reported as favoring the change, although the supplied passage truncates the specific recommendation.
- 5.1 Implementation: The interface lets users agree or disagree with displayed credibility scores and provide their own rating when disagreeing.This feedback mechanism is exposed when users interact with the displayed score.
- 5.2 Response Time: 82% of users received scores in under 6 seconds, while 99% received them in under 10 seconds across 5.4 million API requests.Response time was dominated by requests to Twitter’s API for tweet details.
6 User Testing
User testing examined TweetCred through deployment activity, direct tweet feedback, and a usability survey. Users often rated the system as easy to use and potentially useful, while feedback also revealed systematic disagreement and a tendency toward lower-than-expected scores.
- 6 User Testing: 1,127 unique users generated 5,451,961 credibility-score API requests during the three-month deployment.The users varied substantially in follower counts and usage levels.
- 6.1 User Feedback: Users provided feedback on individual scores and through a usability survey, with direct feedback collected for 1,273 of approximately 5.4 million requests.Direct feedback allowed users to agree or disagree and, when disagreeing, specify the score they considered correct.
- 6.1 User Feedback: 40% of feedback responses agreed with TweetCred’s score, while 60% disagreed.The authors note that self-selection and cognitive dissonance may partly explain why users reacted to scores.
- 6.1 User Feedback: Among disagreement responses, 49% said TweetCred’s score should have been higher and approximately 11% said it should have been lower.The authors interpret this pattern as evidence that TweetCred tends to produce scores lower than users expect.
- 6.1 User Feedback: Crisis-related tweets received higher credibility scores than general tweets across three sampled crisis events.The comparison used tweets from Ukraine, Oklahoma/Arkansas tornadoes, and an earthquake in Mexico against a background sample.
- 6.2 Usability Survey: Among 67 survey participants, 74% found TweetCred easy to use and 81% said they might use it daily.23% also reported perceived inconsistencies in the system.
7 Conclusions and Future Work
The paper concludes that real-time automated credibility ratings can be deployed at scale, while identifying personalization and context as important unresolved challenges. TweetCred’s deployment also prompted debate and reflected users’ expectation for tools that help evaluate online content.
- 7 Conclusions and Future Work: TweetCred assigns each timeline tweet a 1–7 credibility rating using a semi-supervised ranking algorithm and more than 45 single-tweet features.Features include tweet content, author characteristics, and external-URL information, with training labels obtained through crowdsourcing.
- 7 Conclusions and Future Work: The evaluation indicates that automatic credibility ratings can be delivered with performance, accuracy, and usability at large scale.The conclusion frames this as a demonstrated possibility rather than a completed solution to all credibility-assessment challenges.
- 7 Conclusions and Future Work: Personalization remains unresolved because users want ratings to reflect greater trust in some contacts than others.The deployed model was built across the broader Twitter space rather than individual friendship relationships.
- 7 Conclusions and Future Work: Context remains unresolved because some tweets, including opinions or jokes, may not be suitable for a credibility rating.The authors specifically identify tweets that do not attempt to convey factual information as cases where ratings may not be valid.
- 7 Conclusions and Future Work: TweetCred’s deployment generated broad discussion and indicated user interest in technologies for evaluating the credibility of social-media content.The authors describe the system as a first step toward fulfilling that expectation.