Source-linked AI summary
Where in the World are You? Geolocation and Language Identification in Twitter
Mark Graham, Scott A. Hale, Devin Gaffney
TL;DR
The paper examines the reliability of methods for identifying Twitter language and location, comparing automated language detection with human coding and assessing profile locations against device locations. It finds that automated methods do not match human coding consistently and that profile locations are not useful proxies for users’ physical tweeting locations.
Problem
Research using Twitter’s linguistic and spatial metadata has limited understanding of how to handle the contexts in which content was produced, and standard practices have not emerged.
Method
The paper compares three automated language-identification packages with Twitter’s interface language setting and human coding, and evaluates profile locations against device locations across four research sites.
Results
Automated language-identification methods did not match the accuracy of human coding, while profile-location geolocation results were not useful proxies for device locations.
Takeaways & Limitations
Language should not be used as a proxy for location, and researchers should account for the differing reliability of automated language and location measures.
Takeaways & Limitations
Language use varied within each research site, so language-based location inference could miss users in the location and include users outside it.
Abstract
from arXiv · showhide
The movements of ideas and content between locations and languages are unquestionably crucial concerns to researchers of the information age, and Twitter has emerged as a central, global platform on which hundreds of millions of people share knowledge and information. A variety of research has attempted to harvest locational and linguistic metadata from tweets in order to understand important questions related to the 300 million tweets that flow through the platform each day. However, much of this work is carried out with only limited understandings of how best to work with the spatial and linguistic contexts in which the information was produced. Furthermore, standard, well-accepted practices have yet to emerge. As such, this paper studies the reliability of key methods used to determine language and location of content in Twitter. It compares three automated language identification packages to Twitter's user interface language setting and to a human coding of languages in order to identify common sources of disagreement. The paper also demonstrates that in many cases user-entered profile locations differ from the physical locations users are actually tweeting from. As such, these open-ended, user-generated, profile locations cannot be used as useful proxies for the physical locations from which information is published to Twitter.
Related Work
Prior Twitter research used automated language detection and several kinds of location metadata, but these approaches face important reliability and validity challenges. Existing geolocation methods were also developed mainly for English-language or U.S. geocoded tweets.
- Language identification: Short texts, informal phrasing, and abbreviations make automated language identification difficult.These are identified as major challenges in prior work on language detection for Twitter.
- Location metadata: Profile locations are problematic because they are unstructured, yet researchers have used them to geographically bound online events.Their free-form nature complicates interpretation compared with structured device coordinates.
- Location metadata: Profile-location data include blank and non-geographic entries, making them difficult for geocoding APIs to interpret.Non-geographic entries include popular-culture references.
- Geolocation methods: Newer user-location methods use tweet text, tweeting time, or social connections, but were evaluated mainly on English or U.S. geocoded tweets.This paper instead evaluates two off-the-shelf geocoding services.
- Research gap: Reliable geographic analysis requires attention to the overlapping ways Twitter information is assigned language and location.The paper examines accuracy and error sources across methods to clarify how Twitter data can support geographic research.
Methods
The study collected globally geocoded tweets, sampled four metropolitan areas, and compared existing language and geolocation services against manually coded language data. Its design emphasizes cross-method evaluation rather than reliance on a single algorithm or toolkit.
- Data collection: 111,143,814 tweets were collected from November 10 to December 16, 2011 using Twitter’s streaming API and a planet-wide bounding box.Only tweets with explicit GeoIP or GPS device locations were included; the search API was not used.
- Data collection: The dataset contains device locations from both IP addresses and GPS-enabled devices.These structured locations were captured through the streaming-API method.
- Sampling: 1,000 tweets were randomly selected from each of Cairo, Montreal, San Diego, and Tokyo, with at most one tweet per user.The sites were chosen for geographic, linguistic, and cultural differences, and tweet locations were determined from device coordinates.
- Service selection: Three language detection services and two geolocation services were selected after reviewing available solutions for ease of use, throughput, and thoroughness.The study compared existing solutions instead of assuming one algorithm provides superior accuracy or validity.
- Implementation: Custom scripts ran Alchemy, Compact Language Detection, Xerox, Google, and Yahoo services, storing their outputs in a database.The services were selected as readily implementable off-the-shelf options, not as an exhaustive or authoritative set.
- Human coding: 1,000 tweets from each study region were manually coded for primary language, with disagreements resolved through discussion.The authors used their collective experience across Arabic, English, German, Japanese, Korean, Mandarin, Persian, Spanish, and Thai.
Findings
The findings show that language and location signals in Twitter vary substantially in reliability across methods and research sites. Automated language tools can approach, but do not match, human coding, while profile and timezone metadata provide imperfect proxies for where users tweet.
- Language identification: Human coders achieved Fleiss’ kappa of 0.888, while the best algorithm agreed with human classifications in 76.4 percent of cases at a Fleiss’ kappa of 0.670.CLD achieved the best overall score; Alchemy reached 91 percent agreement in San Diego, corresponding to a Fleiss’ kappa of 0.644.
- Language identification: Language-identification performance varied by writing system and site, with CLD failing all 89 cases of Arabic written in Latin characters.CLD performed particularly well on several Asian scripts but struggled with Arabic chat alphabet text in Cairo.
- Language identification: The user-interface language matched human coding for more than 75 percent of tweets in Montreal, San Diego, and Tokyo, but only 45 percent in Cairo.The Cairo discrepancy likely reflects the absence of an Arabic interface-language option when data were collected.
- Location identification: 35.6 percent of problematic profile locations were legitimate places outside the study bounding boxes, while 24.1 percent were non-geographic or generic text.Another 21.2 percent were broader geographic locations that included the relevant research site.
- Location identification: Preprocessing abbreviations, multiple locations, and coordinates embedded with extra text could likely improve geolocation accuracy, especially for single-area studies.Both geocoders recognized latitude and longitude without additional text but failed when extra text was present.
Discussion and Conclusions
The paper finds substantial limitations in automated language identification and in using profile information as a proxy for where Twitter content was published. It nevertheless identifies practical conditions under which language and location methods can be refined.
- Language identification: None of the tested automated language-identification methods matches the accuracy of human coding.Short, informal, multilingual tweets containing URLs and emoticons complicate classification.
- Language identification: The usefulness of Twitter’s user-interface language setting varies across regions and languages.The setting omits many languages and does not capture multilingual users who write in multiple languages.
- Language identification: CLD and Alchemy show useful promise, but the best algorithm depends on the study’s languages, scripts, and regional context.CLD performed better than Alchemy in Japan, while Alchemy performed better at the other research sites; neither recognized the Arabic chat alphabet.
- Practical recommendations: The paper recommends running multiple language detectors and reviewing subsets with human coders to expose biases and sample-specific reliability issues.Temporarily removing emoticons, URLs, and third-party generated text may improve detection while preserving that text for other analyses.
- Location identification: Profile-location geocoding results are not a useful proxy for device locations, which represent where information was disseminated.The paper compares open-ended profile locations across four research sites and identifies discord between the two location types.
- Location identification: Three factors explain the lack of correlation between profile and device location: invalid or broad text, outdated locations, and failed geocoding.Recommended preprocessing includes handling fictitious names, abbreviations, multiple locations, and embedded latitude-longitude coordinates.
- Location identification: Country-level location estimates are more likely to succeed than city- or metropolitan-level estimates, while UTC-offsets indicate longitude but not latitude.UTC-offsets are easier to process than free-form profile locations, though their accuracy differs across study sites.
- Location identification: Using language as a location proxy can miss language users within a place and include users outside it.All four research sites contained mixed languages, motivating further study of how language use clusters geographically.
Authors
The authors are researchers and practitioners whose work spans Internet geography, language separation online, information transmission, and quantitative social-media methodology.
- Mark Graham’s research focuses on Internet and information geographies and their overlaps with information and economic development.
- Scott A. Hale studies language separation online and how platform design affects information transmission between speakers of different languages.
- Devin Gaffney’s research focuses on quantitative analyses and methodologies of social-media data.