PaperlayerView exact revision on arXiv

Source-linked AI summary

PANDORA Talks: Personality and Demographics on Reddit

Matej Gjurković, Mladen Karan, Iva Vukojević, Mihaela Bošnjak, Jan Šnajder

arXiv:2004.04460v3cs.CLcs.CYcs.SI

TL;DR

Personality and demographic information can help NLP research, but NLP systems may be complex and less interpretable, and can be susceptible to societal biases. PANDORA combines Reddit comments with personality and demographic labels, and its experiments examine trait prediction, gender-classification bias, and psychological variables related to philosophy.

  • Problem

    NLP systems can be complex and less interpretable, and may be susceptible to societal biases present in training data.

    Source: §1 · p. 1

  • Method

    PANDORA provides 17M Reddit comments with personality and demographic labels, which the authors use in three experiments and to train benchmark prediction models.

    Source: §1 · pp. 1–2, §6 · p. 9

  • Results

    PANDORA supports experiments estimating Big 5 traits from MBTI and Enneagram labels, identifying personality-linked gender-classification bias, and analyzing psycho-demographic predictors of philosophy propensity.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Certain psycho-demographic variables are good predictors of Reddit users’ propensity for philosophy.

    Source: §6 · p. 9

    Selection biases mean PANDORA may not represent Reddit users and is not representative of internet users or the general population.

    Source: §4 · p. 5

Abstract

from arXiv · show

Personality and demographics are important variables in social sciences, while in NLP they can aid in interpretability and removal of societal biases. However, datasets with both personality and demographic labels are scarce. To address this, we present PANDORA, the first large-scale dataset of Reddit comments labeled with three personality models (including the well-established Big 5 model) and demographics (age, gender, and location) for more than 10k users. We showcase the usefulness of this dataset on three experiments, where we leverage the more readily available data from other personality models to predict the Big 5 traits, analyze gender classification biases arising from psycho-demographic variables, and carry out a confirmatory and exploratory analysis based on psychological theories. Finally, we present benchmark prediction models for all personality and demographic variables.

1 Introduction

The paper addresses the scarcity and limitations of text-based personality datasets that also include demographic labels. It introduces PANDORA and demonstrates its use in personality prediction, gender-bias analysis, and psycho-demographic research.

1 Introduction

  • Personality and demographic variables matter in social-sciences research as target or control variables and can help NLP researchers interpret model decisions, address societal bias, and control for confounders.The paper identifies computational sociolinguistics as one field that can benefit from textual datasets containing these variables.Source: §1 · p. 1
  • Scarce personality-labeled text datasets and the lack of demographics in existing resources limit research using established personality models and demographic variables together.Existing datasets may also have few authors or comments, short comments, non-anonymity, or topic bias; MBTI9k uses MBTI rather than Big 5 and lacks demographic labels.Source: §1 · p. 1
  • PANDORA is a Reddit dataset with more than 17 million comments from over 10,000 users, labeled with personality and demographic variables.It includes Big 5 labels for more than 1,600 users and their more than 3 million comments.Source: §1 · pp. 1–2
  • PANDORA supports predicting Big 5 traits from more readily available MBTI and Enneagram labels by exploiting correlations between models and their expression in text.The paper also presents benchmark prediction models for personality and demographic variables.Source: §1 · p. 2
  • The gender-bias experiment finds that a Reddit-trained gender classifier more often fails for users with certain personality-trait combinations.The paper also examines the relationship between propensity for philosophy and psycho-demographic variables using confirmatory and exploratory analyses.Source: §1 · p. 2

2 Background and Related Work

Prior work links personality to language across several platforms, but large-scale personality datasets are scarce, and existing resources leave gaps in model coverage and joint labeling. PANDORA is situated as a resource for prediction and demographic-aware NLP research.

2 Background and Related Work

  • MBTI and the Five Factor Model are commonly used personality models, but MBTI is criticized for low validity while the Five Factor Model represents people on five trait continua.The five traits are Extraversion, Agreeableness, Conscientiousness, Neuroticism, and Openness to Experience.Source: §2 · p. 2
  • Personality research using language spans essays, emails, devices, blogs, social networks, Facebook, Twitter, and Reddit.The literature review notes increasing difficulty obtaining personality datasets and the scarcity of large-scale resources.Source: §2 · p. 2
  • Personality datasets are difficult to obtain because labeling is costly and raises privacy concerns, and large-scale datasets are virtually nonexistent.After MyPersonality became unavailable, research relied on smaller resources from essays, forums, Twitter, and workshop datasets.Source: §2 · pp. 2–3
  • Earlier work compared MBTI and Big 5 prediction models on Twitter but could not use MBTI labels to predict Big 5 because no users had both labels.The paper presents PANDORA as a multilabeled dataset that can support new architectures and help minimize overfitting to spurious correlations.Source: §2 · p. 3
  • Demographic data has been used in debiasing research, while personality-based debiasing had been explored using predicted rather than self-reported traits.The review also points to Reddit’s use in studies of mental health.Source: §2 · p. 3

3 PANDORA Dataset

PANDORA combines Reddit comments with personality and demographic labels, extending MBTI9k with Enneagram, Big 5, and demographic information. The paper describes how these labels were collected and normalized, then reports dataset composition and patterns in the scores.

3 PANDORA Dataset

  • The authors state that archival research met psychological-research ethics guidance, removed content when users requested removal, and received academic IRB approval.They note that future removal requests would be handled similarly and escalated to Reddit.Source: §3 · p. 3

3.1 MBTI and Enneagram Labels

  • MBTI labels were taken from Reddit flairs, and Enneagram labels were manually added from flairs; 9,084 users reported MBTI and 793 also reported Enneagram.The source dataset was MBTI9k, built from Reddit comments dating back to 2015.Source: §3.1 · p. 3, §3 · p. 3

3.2 Big 5 Labels

  • Big 5 scores required normalization because tests used different inventories, trait names, score formats, and scales.The authors mapped descriptions to values from 0–100 in increments of 10 and converted some raw scores and percentages to percentiles using test-score distributions.Source: §3.2 · pp. 3–4
  • Big 5 scores were extracted semiautomatically from comments, with manual verification and manual extraction for reports that automated rules did not handle correctly.A test-identification classifier was used for reports without a known test; the procedure yielded scores for 600 additional users, for 1,608 total.Source: §3.2 · p. 4
  • The resulting PANDORA dataset contains 17,640,062 comments by 10,288 users, including 393 users labeled with both Big 5 and MBTI.Comments from 2015 onward were retrieved for Big 5-labeled users and added to the MBTI dataset.Source: §3.2 · p. 4

3.3 Demographic Labels

  • Age, gender, and location labels were manually inspected from user flairs, supplemented with comment-level age and gender self-reports, and location names were normalized.For users reporting age at multiple time points, the most recent age was used.Source: §3.3 · p. 4
  • The dataset’s Figure 1 presents the distribution of Big 5 percentile scores, and Table 1 reports means for Big 5 scores and age alongside comment-count and MBTI-trait distributions.Table 1 reports Big 5 means for 1,608 users and age means for 2,324 users.Source: Figure 1 · p. 5, Table 1 · p. 5

3.4 Analysis

  • The average user scored near average on neuroticism, higher on openness, and lower on extraversion, agreeableness, and conscientiousness.Males averaged younger and lower on agreeableness, but higher on neuroticism, than females; MBTI dimensions also correlated with Big 5 traits in patterns consistent with prior psychological research.Source: §3.4 · p. 5

4 Experiments

Across three experiments, the authors use PANDORA to test personality-label transfer to Big 5, examine gender-classification error patterns, and analyze theory-backed and exploratory links with philosophical language.

4 Experiments

  • The authors caution that selection biases may make PANDORA unrepresentative of Reddit users, and it is not representative of internet users or the general population.Source: §4 · p. 5

4.1 Predicting Big 5 with MBTI/Enneagram

  • Predicted Big 5 traits correlated more with gold traits than predicted MBTI dimensions did, except for conscientiousness, supporting transfer from the more abundant personality labels.For example, predicted openness was a better predictor of openness than the MBTI intuitive dimension.Source: Table 5 · p. 6, §4.1 · p. 6

4.2 Gender Classification Bias

  • Errors were 14.4% for females versus 8.1% for males; thinking and perceiving preferences related to misclassification in both groups, with age and extraversion differences also reported.Thinking and perceiving preferences related to misclassification in opposite ways by gender; younger females and more extraverted males were more often misclassified.Source: Table 6 · p. 7, §4.2 · p. 7

4.3 Propensity for Philosophy

  • Across 430 users, 11 predictors explained 41% of philosophy-feature variance; openness, insight, readability, and negative emotions were positive predictors.Positive emotion was significant until readability entered the model, while males had higher philosophy-word frequency; posemo and readability were moderately correlated (-0.40).Source: §4.3 · pp. 7–8, Table 7 · p. 8

5 Prediction Models

The section benchmarks models for predicting PANDORA’s personality and demographic variables from user comments, using several feature sets and model families. The results favor n-grams for most LR tasks, with personality-prediction features helping Big 5 prediction.

5 Prediction Models

  • Features included word and character n-grams, stylistic and dictionary measures, gender predictions, subreddit information, named entities, and part-of-speech counts.Comment-level features were aggregated per user; Big 5 models also used MBTI/Enneagram predictions.Source: §5 · p. 8
  • The authors trained six regression and eight classification models, including LR and separate neural networks that encoded each user’s comments with BERT vectors.The neural networks applied convolution layers, max pooling, and fully connected layers to 1,024-dimensional comment vectors.Source: §5 · p. 8
  • Models were evaluated with target-specific stratified five-fold cross-validation, with feature selection and hyperparameter optimization performed separately within each fold.Regression F-tests selected top-K features using held-out data.Source: §5 · p. 8
  • Up to 5 points: using 1,000 rather than the last 100 comments increased scores; LR performed best with n-grams except that Big 5 benefited from MBTI/Enneagram predictions.The table reports LR results across n-grams, personality predictions, and other features, alongside NN results.Source: §5 · p. 8, Table 8 · p. 9

6 Conclusion

The conclusion presents PANDORA as a large Reddit dataset with personality and demographic labels and summarizes three applications. It also reports benchmark models and identifies weak deep-learning performance as motivation for further model development.

6 Conclusion

  • PANDORA contains 17M comments from over 10k Reddit users, including 1.6k users with Big 5 labels, and covers Big 5, MBTI, Enneagram, and demographic variables.The authors describe it as the first Reddit dataset with Big 5 traits and the first to cover multiple personality models.Source: §6 · p. 9
  • The experiments used MBTI/Enneagram labels to estimate Big 5 traits, found gender-classification bias for users with certain personality traits, and analyzed propensity for philosophy.The philosophy analysis was confirmatory and exploratory and drew on existing psychological theories.Source: §6 · p. 9
  • The authors trained and evaluated benchmark prediction models for all psycho-demographic variables.Source: §6 · p. 9
  • The authors cite poor deep-learning baseline performance, rich labels, and many comments per user as reasons for further work on efficient user representations and more advanced architectures.Source: §6 · p. 9

A Demographics of PANDORA

This appendix reports PANDORA’s geographic and language distributions, including how users’ locations were grouped into regions.

A Demographics of PANDORA

  • Users were mostly from English-speaking countries and were regionally evenly distributed across the US and Canada.The authors used five US regions and three Canadian regions; one Canadian region had no users.Source: A Demographics of PANDORA · p. 14, Table 9 · p. 14
  • The appendix reports the language distribution of comments, including counts for the ten most common languages.Language was identified for each comment using fast text-based language identification.Source: A Demographics of PANDORA · p. 14, Table 10 · p. 14

B Additional Information on Personality Scores

This appendix summarizes reported personality-score distributions and inventories, and presents MBTI-type counts and Enneagram–Big 5 correlations.

B Additional Information on Personality Scores

C Predicting Big 5 with MBTI/Enneagram

The experiment tests whether abundant MBTI/Enneagram labels can be leveraged to predict Big 5 traits from text, using a domain-adaptation pipeline that transfers MBTI predictions to Big 5 regression models.

C Predicting Big 5 with MBTI/Enneagram

  • The domain-adaptation pipeline uses MBTI classifier scores as features for Big 5 regression, then applies both stages to users with Big 5 but no MBTI labels.The experiment partitions users into groups with both labels (382), MBTI only (8,691), and Big 5 only (1,588).Source: C Predicting Big 5 with MBTI/Enneagram · p. 15
  • Four text-based MBTI classifiers are trained on the MBTI-only subset, and their probabilities become features for five ridge-regression models trained on users with both label sets.The MBTI classifiers use 7,000 filter-selected TF-IDF-weighted 1–5 word and character n-grams; both stages use 5-fold cross-validation.Source: C Predicting Big 5 with MBTI/Enneagram · p. 15
  • 45% type-level accuracy (82.4% for one-off prediction) is observed for MBTI classification on the shared-label subset; the section also reports an Enneagram–Big 5 correlation analysis.The passage identifies Table 14 for the correlation-analysis results but does not state those results here.Source: C Predicting Big 5 with MBTI/Enneagram · p. 15

D Parameters of the DL Model

The deep-learning model combines convolution, max pooling, and fully connected layers to aggregate information across a user's comments; the supplied figures describe separate logistic-regression learning curves.

D Parameters of the DL Model

  • The model uses a convolutional layer, max pooling, and fully connected layers; convolution kernels span BERT's representation and aggregate information across comments.The pooled outputs form M vectors of length K per user, which feed fully connected layers with Leaky ReLU activations.Source: D Parameters of the DL Model · p. 15
  • Figures 2 and 3 present logistic-regression learning curves for MBTI trait prediction and Big 5 trait prediction, respectively.Source: Figure 2 · p. 15, Figure 3 · p. 15
  • Kernel sizes range from 2 to 6 and kernel counts M from 4 to 6; outputs are sliced into K segments before max pooling.L2-norm regularization and dropout are applied only to the fully connected layers.Source: D Parameters of the DL Model · p. 15

E Learning Curves for the Logistic Regression Models

For logistic regression with 1-gram features, performance levels off at about 1,000 comments, with little significant change from using more training comments.

E Learning Curves for the Logistic Regression Models

  • Around 1,000 comments, logistic-regression performance plateaus, with little significant change as training data increases further.The learning curves measure F1-macro against the number of comments for the 1-gram model.Source: E Learning Curves for the Logistic… · p. 15
  • Figures 2 and 3 show learning curves for logistic regression with 1-gram features, plotting number of comments against F1-macro score; performance plateaus around 1000 comments, with little significant change beyond that training amount.Source: E Learning Curves for the Logistic… · p. 15
  • Beyond roughly 1,000 comments, increasing the training-comment count yields little significant change in the plotted performance.This plateau is reported for the logistic-regression learning curves.Source: E Learning Curves for the Logistic… · p. 15

Related papers

Loading 2004.04460v3…