Source-linked AI summary
When is it Biased? Assessing the Representativeness of Twitter's Streaming API
Fred Morstatter, Jürgen Pfeffer, Huan Liu
TL;DR
Researchers have evidence of sampling bias in Twitter’s Streaming API, but checking it against Firehose is prohibitively costly for many. This paper tests the freely available Sample API as an alternative signal for detecting hashtag-trend bias over time. It reports that the Sample API is representative overall and that Streaming API results overlap substantially across query locations and start times, while the approach is less useful for sparse queries.
Problem
Method
The paper tests the freely available Sample API as a proxy for true hashtag activity and uses it to identify time periods when Streaming API trends may be biased.
Source: §3.1 · p. 2, §3 · p. 2Results
The authors report that Sample API tweets are representative overall and that Streaming API results substantially overlap across queries from the United States and Austria and across different start times.
Source: §5 · p. 6Takeaways & Limitations
The Sample API can indicate likely Streaming API bias without Firehose data, particularly when a researcher’s query receives substantial attention.
Source: §3 · p. 2, §5 · p. 6The method is less useful for queries receiving less data from the sparse Sample API.
Source: §5 · p. 6
Abstract
from arXiv · showhide
Twitter has captured the interest of the scientific community not only for its massive user base and content, but also for its openness in sharing its data. Twitter shares a free 1% sample of its tweets through the "Streaming API", a service that returns a sample of tweets according to a set of parameters set by the researcher. Recently, research has pointed to evidence of bias in the data returned through the Streaming API, raising concern in the integrity of this data service for use in research scenarios. While these results are important, the methodologies proposed in previous work rely on the restrictive and expensive Firehose to find the bias in the Streaming API data. In this work we tackle the problem of finding sample bias without the need for "gold standard" Firehose data. Namely, we focus on finding time periods in the Streaming API data where the trend of a hashtag is significantly different from its trend in the true activity on Twitter. We propose a solution that focuses on using an open data source to find bias in the Streaming API. Finally, we assess the utility of the data source in sparse data situations and for users issuing the same query from different regions.
1. INTRODUCTION
The paper addresses whether Twitter’s parameterized Streaming API provides representative samples despite its unpublished sampling mechanism and 1% limit. It seeks an openly available alternative to Firehose-based bias checks and defines bias in terms of hashtag trends over time.
1. INTRODUCTION
- The Streaming API returns at most 1% of tweets for a query when its volume exceeds that threshold, and Twitter does not publish its sampling method.Prior work found evidence of bias under certain conditions, prompting questions about when the data is representative.Source: §1 · p. 1
- Firehose comparisons can verify Streaming API results, but Firehose access is restrictively expensive and limited to users with a Twitter business agreement.The paper therefore seeks a bias-detection alternative using openly available data.Source: §1 · p. 1
- The study defines a hashtag as biased when its relative trend is statistically significantly over- or underrepresented against its true Twitter trend during particular time periods.The focus is sample bias in the Streaming API, not every possible form of bias on Twitter.Source: §1 · p. 1
- The paper reports an openly available method for detecting likely Streaming API bias and finds substantial overlap across queries issued from different countries and at different times.It also reports that identical queries from different regions receive nearly the same results.Source: §1 · p. 2
2. RELATED WORK
The related work spans prior uses of the Streaming API, assessment of bias in black-box systems, and evidence of bias in Twitter data. Prior studies compared Streaming API data with Firehose across several facets and found differing results.
2.1 Work with the Streaming API
- Researchers have used the Streaming API for topic modeling, network analysis, and statistical analysis of content.The authors describe this reliance as significant.Source: §2.1 · p. 2
2.2 Bias in Black Box Systems
- Black-box assessment work includes studies of Mechanical Turk user representativeness and response bias, as well as sampling effects on perceived link propagation in Twitter.Source: §2.2 · p. 2
2.3 Bias in Twitter Data
- Twitter user demographics also differ from U.S. Census estimates, with adults aged 18–29, African Americans, and urban residents reported as over-represented.Source: §2.3 · p. 2
- Prior Twitter bias research found differences across facets: low-ranked top hashtags and LDA topics showed bias, central users overlapped with Firehose results on average 50% of the time, while geotagged tweets showed no bias and exceeded 90% of Firehose volume.The compared facets included hashtag occurrence, topics, retweet-network centrality, and geotagged-tweet distribution.Source: §2.3 · p. 2
3. DISCOVERING BIAS IN THE STREAMING API WITHOUT THE FIREHOSE
The paper uses Twitter’s open Sample API to identify periods when a hashtag’s Streaming API trend differs from true activity, without relying on Firehose data. It evaluates the Sample API as a reference, develops a bootstrap comparison, and examines when sparse data limits that comparison.
3.1 Validation of the Streaming API
- The Sample API returns a parameter-free 1% sample of all Twitter tweets; its trend can indicate Streaming API bias but may not recover the true trend.The authors use the Sample API as a proxy because its sparsity makes recovering the true trend likely impossible.Source: §3.1 · p. 2
3.2 Vetting the Randomness of the Sample API
- Across every tested k, the experiment rejects H0 at a 95% confidence level using τβ significance levels for top-k hashtags comparing the Sample API with Firehose, supporting representativeness.Source: §3.2 · p. 3, Table 2 · p. 3
- For a query matching 10% of Firehose, filtering the Sample API yields 0.1% of Firehose, while the Streaming API sample is 10× larger.The authors therefore use the Sample API to identify bias periods rather than replace the Streaming API with its filtered data.Source: §3.2 · p. 3
3.3 Finding Bias in the Trend of a Hashtag
- The method normalizes both time series, bootstraps the Sample API 100 times, and flags Streaming API values outside the bootstrap distribution’s ±3σ threshold.The threshold is applied at each time point after comparing the normalized Streaming API series with the bootstrap distribution.Source: §3.3 · pp. 3–4
- For #believemovie, the method flags an underrepresented hours 10–11 spike and overrepresented post-hour-16 spikes at 99.7% confidence.The example compares the hashtag’s Streaming API and Sample API time series using bootstrap-derived confidence intervals.Source: §3.3 · p. 4, Figure 2 · p. 4
3.4 Signal Usability Under Sparsity
- Only 4 known zeros occur among the top 10 hashtags, but missing Sample API observations increase down the popularity ranking.Known zeros are Streaming API observations absent from the Sample API; many hours for many hashtags therefore remain unverifiable.Source: §3.4 · p. 5, Figure 3 · p. 5
4. GEOGRAPHIC AND TEMPORAL STABILITY OF QUERIES
The study tests whether identical Streaming API queries return consistent tweet sets across countries and staggered start times. Overlapping queries yielded highly similar results, with some wider variation in Austrian time comparisons.
4.1 Between-Country Results
- Cross-country Jaccard scores were very high on average with low standard deviation, indicating nearly identical USA and Austria query results and no preference for US queries.The authors are hopeful that researchers outside the US will obtain similar results.Source: Table 3 · p. 5, §4.1 · p. 5
4.2 Between-Time Results
- Overlapping USA query windows had extremely high average Jaccard scores and very low standard deviation; Austrian windows varied more but retained an extremely high mean.The authors interpret the Austrian results as providing confidence in coverage across the tested windows.Source: Table 3 · p. 5, §4.2 · p. 5
4. GEOGRAPHIC AND TEMPORAL STABILITY OF QUERIES
- Queries started every 20 minutes and ran for 30 minutes in the USA and Austria, enabling comparisons of 10-minute overlaps and simultaneous cross-country runs.The collection spanned 2013-10-20 06:20 to 2013-10-22 22:20 UTC, with 194 datasets from each country.Source: §4 · p. 5
5. DISCUSSION AND FUTURE WORK
The Sample API provides a way to identify potentially biased time periods in Streaming API data without costly Firehose data. The approach works best for popular queries, and the paper discusses ways researchers can respond to detected bias.
5. DISCUSSION AND FUTURE WORK
- The Sample API is representative overall and can help identify potentially biased Streaming API periods; results show significant overlap across queries from the United States and Austria and across different start times.The overlap applies to Streaming API results for a given time period.Source: §5 · p. 6
- Researchers can focus on periods with little or no detected bias, buy reseller data only for biased periods, or incorporate other social media for added depth.One proposed approach to incorporating other platforms is cross-referencing Twitter users with other social media users, as proposed in.Source: §5 · p. 6
- The method is useful for high-attention queries but less useful for sparse data; neighboring-value bootstrapping such as is proposed as a possible remedy.Finding bias in sparse scenarios remains future work.Source: §5 · p. 6