Demographic Inference and Representative Population Estimates from Multilingual Social Media Data

Zijian Wang; Scott A. Hale; David Adelani; Przemyslaw A. Grabowicz; Timo Hartmann; Fabian Flöck; David Jurgens

Source-linked AI summary

Demographic Inference and Representative Population Estimates from Multilingual Social Media Data

Zijian Wang, Scott A. Hale, David Adelani, Przemyslaw A. Grabowicz, Timo Hartmann, Fabian Flöck, David Jurgens

arXiv:1905.05961v1May 15, 2019cs.CYcs.CLcs.CVcs.LG

TL;DR

The paper addresses the difficulty of estimating broader population characteristics from demographically biased, multilingual social media data. It combines a multilingual multimodal demographic model with interpretable post-stratification, improving population estimation while reducing some bias. The authors identify non-binary gender support and regional homogeneity assumptions as scope boundaries.

Problem
Social media are large and detailed but non-representative, while demographic attributes and platform inclusion probabilities are often unavailable or unknown.
Method
The paper combines M3, a multilingual multimodal model jointly inferring age, gender, and organization status, with interpretable multilevel regression for inclusion probabilities and post-stratification.
Results
The combined demographic inference and debiasing approach produces more accurate population estimates, with leave-one-region-out MAPE decreasing from 88% without demographics to 33% using joint demographics, census joint distributions, and log-space modeling.
Takeaways & Limitations
The approach supports more representative social sensing and yields interpretable inclusion-probability estimates that can be shared for future reuse.
Takeaways & Limitations
The current approach does not support non-binary gender identities, and its regional assumptions may not fully capture inclusion-probability variation.

Abstract

from arXiv · show

Social media provide access to behavioural data at an unprecedented scale and granularity. However, using these data to understand phenomena in a broader population is difficult due to their non-representativeness and the bias of statistical inference tools towards dominant languages and groups. While demographic attribute inference could be used to mitigate such bias, current techniques are almost entirely monolingual and fail to work in a global environment. We address these challenges by combining multilingual demographic inference with post-stratification to create a more representative population sample. To learn demographic attributes, we create a new multimodal deep neural architecture for joint classification of age, gender, and organization-status of social media users that operates in 32 languages. This method substantially outperforms current state of the art while also reducing algorithmic bias. To correct for sampling biases, we propose fully interpretable multilevel regression methods that estimate inclusion probabilities from inferred joint population counts and ground-truth population counts. In a large experiment over multilingual heterogeneous European regions, we show that our demographic inference and bias correction together allow for more accurate estimates of populations and make a significant step towards representative social sensing in downstream applications with multilingual social media.

1 INTRODUCTION

Social media offer large-scale, timely behavioral data, but their demographic biases limit population-level inference. The paper combines demographic inference with post-stratification to improve representativeness.

Representative surveys support policy, research, and commercial prediction but are expensive and infrequent.
Social media data are generally non-representative of broader offline populations because platform participation and usage vary across demographic groups.Examples include overrepresentation of young and male users in some settings and geographic or racial imbalances.
Exact demographic-group inclusion probabilities are typically unknown, making representative population estimates difficult.
Post-stratification can correct non-representative samples using demographic attributes, but social media platforms usually do not provide those attributes.
The paper introduces multilingual demographic inference, interpretable debiasing models, and evaluations showing lower estimation error than models without post-stratification.

2 DEMOGRAPHIC INFERENCE

The paper models gender, age, and organization status to stratify social media users for demographic analysis. Organization status distinguishes organizational accounts from individual accounts, a known confounder in people-based studies.

The proposed demographic inference model jointly predicts gender, age, and binary organization status.
Gender and age are included because census data widely report them and they are central to measuring demographic bias.
Organization status separates accounts belonging to organizations from individual accounts.The paper selects this attribute because organizational accounts can confound people-based studies of online platforms.

2.1 Classification Task

The classification tasks infer gender, age, and organization status from account-level information rather than tweets. Age uses four census-aligned categories, while the gender formulation does not yet support non-binary identities.

The model uses username, screen name, biography, and profile image instead of tweet content to scale when users contribute little text.Avoiding tweet-based stratification also prevents demographic biases in downstream tasks analyzing the same language.
Gender and organization status are binary classification tasks, while age is divided into four levels: ≤18, (18, 30), [30, 40), and [40, 99).
The age ranges align with census reports and surveys while keeping the task manageable for downstream applications.Finer-grained age classes are difficult even for humans.
The gender formulation treats gender as performed through style, names, and pictures and reports identity on a probability continuum.Training data use self-declared binary gender identities.
The current approach does not support non-binary gender identities, which the authors identify as an important future task.

2.2 Method and Training Procedure

M3 combines image and text pipelines with multilingual, multimodal, multi-attribute learning. Its training uses co-training, multilingual augmentation, and modality dropout to exploit complementary information across languages and modalities.

The multimodal model combines image and text because demographic attributes are expressed in both sources.The paper reports that leveraging both modalities significantly augments predictive accuracy.
M3 processes profile images and three text inputs through separate pipelines, then uses multi-task learning for the three attribute outputs.The text inputs are username, screen name, and biography.
The training procedure combines co-training, multilingual data augmentation, and modality dropout rather than initially training end-to-end.
Co-training adds high-confidence predictions from unlabeled data as additional labeled examples for separate image and text views.
The image model is trained first to identify high-confidence labels in multilingual unlabeled data, which then augment training of the text model and full M3 system.Image data are treated as a more universal signal across sociolinguistic contexts.
Word-based translation with MUSE dictionaries is used because online translation services are prohibitive at the scale of the training data.Translated instances are retained only when at least 80% of their words can be translated.

2.3 Data

The study combines multiple labeled, unlabeled, and crowdsourced Twitter datasets spanning 32 languages to train and evaluate M3. It balances training instances across demographic attributes and uses modality-specific confidence thresholds and fixed data partitions.

Datasets: M3 uses five datasets covering heuristically labeled Twitter profiles, organizational accounts, face images, unlabeled users, and crowdsourced labels across 32 languages.The datasets support training, augmentation, and multilingual evaluation.
Datasets: 36.97M unlabeled profiles in 32 European languages provide users whose high-confidence modality labels can augment training.Users enter training only when a text or image classifier produces a high-confidence label.
Crowdsourcing Data: The crowdsourced dataset samples up to 200 Twitter users per language, with annotators reviewing usernames, biographies, and profile images for demographic attributes.The annotation design includes categories for gender, organization status, and age, plus “don’t know” responses.
Crowdsourcing Data: Annotation reliability is measured separately for each crowdsourcing job using Krippendorff’s α, with age brackets over 40 collapsed for comparison with M3.The reliability calculation excludes specified English test questions and profiles with only one annotation.
Data Splits: Labeled data is split into 80% training, 10% development, and 10% test sets, while crowdsourced data is used only for testing.Development data for self-reporting users supports model selection.
Training Data: Training balances minibatches across gender, age, and organization status, using oversampling and undersampling to address attribute and age-class imbalance.Separate image and text models train for 10 epochs, followed by an 8-epoch co-training stage.

2.4 Evaluations

The evaluation compares M3 with existing systems across demographic attributes, modalities, datasets, and languages. M3 generally achieves strong or improved performance, including multilingual recognition and reduced algorithmic bias, while age remains more difficult.

Gender Recognition: M3 achieves state-of-the-art gender recognition across the IMDB, WIKI, and Twitter image datasets.It significantly improves F1 at each comparison system’s recall level, except for similar Microsoft performance on WIKI.
Text-Based Recognition: M3 outperforms all but one text-based gender system using only usernames, and performance improves substantially when all text information is available.The full-text setup uses information included by default in the Twitter API, with profile-image downloading as the additional practical step.
Age Recognition: 0.16 and 0.11 absolute F1 improvements over Face++ and Microsoft, respectively, occur for age recognition on real-world Twitter profiles.On IMDB and WIKI, which mainly contain well-lit face-forward headshots, M3 performs similarly to commercial models.
Organization Recognition: 16.3% higher accuracy than the next closest system is achieved for recognizing organizational accounts, without reducing human-account recognition.On a separate organization test set, M3 obtains an overall F1 of 0.898.
Multilingual Evaluation: In 32 languages, gender and organization performance are similar for most languages, while age F1 ranges from 0.28 for Bosnian to 0.73 for Slovenian and Welsh.Overall multilingual performance is on par with primarily English heuristically labeled data, although age remains the most difficult task.
Ablation Study: Both modalities benefit performance, with removing textual information causing the largest drop; co-training and translation particularly improve multilingual evaluation.These augmentation methods hurt performance on primarily English heuristically labeled data but substantially help when bootstrapping from mostly monolingual data.
Algorithmic Bias: M3 shows substantially less algorithmic bias than the two tested commercial systems on Gender Shades.It improves performance on dark-skinned women but is least accurate on darker-skinned men, leaving performance parity unresolved.

3 LEARNING INCLUSION PROBABILITIES

The paper estimates demographic inclusion probabilities to correct social-media sampling bias, using regression models that relate biased platform counts to ground-truth population counts under explicit homogeneity assumptions.

Social-media measurements are biased because demographic groups can have different probabilities of joining a platform.This probability is the inclusion probability and is typically unknown.
The method estimates inclusion probabilities from demographic counts observed on the platform and corresponding ground-truth population counts.The approach draws on statistical survey analysis with missing data.
Under homogeneity, inclusion probabilities are assumed equal within demographic strata across subsets of a population partition.The partition may divide a population into countries, regions, or cities.
The homogeneous model regresses population counts against platform counts, with coefficients equal to the inverse inclusion probabilities when the assumption holds.This provides global and group-specific debiasing coefficients.
When homogeneity fails, the paper models partition subsets separately or specifies an inhomogeneous functional form using an exponent and demographic functions.The inhomogeneous formulation is estimated by regressing log ground-truth counts on log platform counts and demographic indicators.
Cross-validation and model selection compare these debiasing models by their population-size prediction errors.The evaluation uses populations of regions in EU countries to identify the most accurate assumption for each dataset.

4 EUROPEAN POPULATION INFERENCE FROM TWITTER DATA

The study evaluates interpretable debiasing models that combine inferred demographic distributions with census population counts to estimate European regional populations from Twitter. Joint age–gender information and log-space, country-specific modeling substantially improve leave-one-region-out accuracy, although temporal mismatch limits interpretation.

4.1 Models: The models regress census population counts against demographic Twitter counts to estimate country-specific inclusion probabilities.Multilevel models separate global Twitter bias from country-specific bias, while homogeneity assumptions govern whether inclusion probabilities vary across regions.
4.2 Data: 3,202,964 users from Europe’s NUTS3 regions were analyzed alongside 2011 age–gender census distributions from 30 European countries.Twitter users came from the random 10% stream, were observed from September 2015 to January 2016, and were assigned to regions using inferred locations.
4.3 Results: 88% MAPE without demographics fell to 59% using gender, 61% using age, 54% using inferred joint age–gender counts, and 33% with joint census counts in log space.These results come from leave-one-region-out evaluation; removing organization accounts produced only small, statistically nonsignificant error reductions in most models.
4.3 Results: The joint Twitter-and-census model produced predictions noticeably closer to the y = x line in scatter plots of true versus predicted regional populations.The paper attributes this pattern partly to training in log space and also reports geographical variation in MAPE across EU regions.
4.4 Discussion of Debiasing Results and Potential Sources of Error: Joint inferred age–gender distributions improve prediction even without joint census training, while the log-space inhomogeneous model is more accurate than the homogeneous alternative.The authors suggest the homogeneous assumption does not hold across regions and identify the roughly five-year Twitter–census time mismatch as a possible source of imperfect predictions.
4.4 Discussion of Debiasing Results and Potential Sources of Error: MAPE was not significantly correlated with region area, population density, or income in the models, with only Czech Republic and Norway showing statistically different-from-zero correlations across models.This analysis tested whether performance was biased toward particular regional characteristics.

5 CONCLUSION

The paper addresses demographic and sampling bias in social-media population estimates by combining multilingual demographic classification with interpretable regression-based debiasing. It introduces a 32-language system and shows across Europe that inferred demographic strata improve population estimation and support more representative social sensing.

The paper develops a holistic solution for demographic skew in social-media usage by assigning users to demographic strata and using regression to estimate platform inclusion probabilities.
The multilingual neural system performs multi-attribute classification in 32 languages and includes a released system and a new dataset of gender, age, and is-organization annotations.
The paper derives interpretable debiasing models with explicit formal interpretations for correcting social-media measurements.
A Europe-wide study shows that the combined demographic inference and debiasing approach supports more representative population-level estimates from social media.