PaperlayerView exact revision on arXiv

Source-linked AI summary

Demographic Dialectal Variation in Social Media: A Case Study of African-American English

Su Lin Blodgett, Lisa Green, Brendan O'Connor

arXiv:1608.08868v1cs.CL

TL;DR

Dialectal language is increasingly common online, but NLP resources and tools may not handle it equitably. The paper uses census-linked geolocation and distant supervision to identify and characterize AAE-like Twitter language, then evaluates and adapts NLP tools. It finds poorer existing-tool performance on AAE-like text and reports an ensemble classifier that closes the language-identification disparity.

  • Problem

    Few resources exist for developing NLP tools that handle increasingly abundant dialectal language in social media.

  • Method

    The paper uses geo-located Twitter messages, neighborhood census demographics, and a distantly supervised probabilistic model to identify and characterize AAE-like language.

  • Results

    Existing language-identification and dependency-parsing tools perform poorly on AAE-like text compared with white-associated text, while an ensemble classifier eliminates the language-identification disparity.

  • Takeaways & Limitations

    Demographic correlations can uncover dialectal language on Twitter and help close performance gaps between dialectal and standard text.

  • Takeaways & Limitations

    The approach benefits from highly accurate census records and demographic associations that may not exist for other language communities.

Abstract

from arXiv · show

Though dialectal language is increasingly abundant on social media, few resources exist for developing NLP tools to handle such language. We conduct a case study of dialectal language in online conversational text by investigating African-American English (AAE) on Twitter. We propose a distantly supervised model to identify AAE-like language from demographics associated with geo-located messages, and we verify that this language follows well-known AAE linguistic phenomena. In addition, we analyze the quality of existing language identification and dependency parsing tools on AAE-like text, demonstrating that they perform poorly on such text compared to text associated with white speakers. We also provide an ensemble classifier for language identification which eliminates this disparity and release a new corpus of tweets containing AAE-like language.

1 Introduction

Dialectal language is increasingly present in social media, but mainstream NLP tools may inadequately handle it. This study develops and evaluates methods for identifying AAE-like Twitter language and adapting language identification to reduce disparities.

  • Motivation: Social media makes dialectal language increasingly available, while NLP tools trained on mainstream text may be insufficient for online conversational language.This insufficiency can affect applications such as social media sentiment analysis.
  • Motivation: AAE is studied as a widespread American English dialect with established syntactic-semantic, phonological, and lexical features.Its demographic associations and sociolinguistic history support its use as a case study.
  • Approach: The study develops a statistical method using geographic census demographics and distant supervision to identify demographically aligned text and language.The model assumes a soft correlation between demographics and language.
  • Validation: The approach is validated by testing whether African-American-demographics-aligned text exhibits known AAE phonological and syntactic properties and divergences from SAE.This connects demographic alignment to linguistic characterization.
  • Evaluation: Existing language identification and dependency parsing tools perform poorly on AAE-like text compared with text associated with white speakers.The paper frames this as a racial disparity in NLP tool efficacy.
  • Adaptation: The study provides an ensemble classifier intended to eliminate racial disparity in language-identification accuracy and releases a corpus of 830,000 tweets aligned with African-American demographics.The classifier adapts language identification for U.S. online conversational text.

2 Identifying AAE from Demographics

The paper identifies AAE-like Twitter language by linking geo-located messages to neighborhood demographics and modeling demographic-language associations. A mixed-membership model addresses limitations of seedlist methods and produces corpora for analysis.

  • 2.1 Twitter and Census data: Geo-located Twitter messages are linked to Census blockgroup demographics, averaged per user as a rough proxy for author and neighborhood demographics.The dataset focuses on U.S. users and filters messages toward casual conversational text.
  • 2.2 Direct Word-Demographic Analysis: The direct word-demographics analysis associates each word type with average demographic values and finds highly AA-associated terms are markedly non-standard.This analysis provides initial validation that demographic information can yield dialectal corpora.
  • 2.2 Direct Word-Demographic Analysis: The seedlist approach collected 2.3 million messages from 41,774 users using the 100 highest-πw,AA frequent terms and author-level usage thresholds.Authors qualified when at least 20% of their messages contained a seedlist term, illustrating the method’s reliance on ad-hoc thresholds.
  • 2.3 Mixed-Membership Demographic-Language Model: The mixed-membership model links demographic variables to topic-specific unigram language models, allowing authors’ language to deviate from demographic priors.Tokens receive latent topics through message-level mixtures, while topic distributions generate words; the model uses a 191,873-word vocabulary.
  • 2.3 Mixed-Membership Demographic-Language Model: Four demographic categories removed Spanish terms from the AA topic, while the seedlist method could not capture occasional aligned language or distinguish language use at message level.These observations motivate modeling demographic-language interactions with a mixed-membership approach.
  • 2.3 Mixed-Membership Demographic-Language Model: The resulting user-based corpora contain 830,000 AA-aligned tweets and 7.3 million white-aligned tweets after requiring at least 80% AA-associated posterior probability and less than 5% Hispanic or Asian language.Message IDs are released in accordance with Twitter’s Terms of Service.

3 Linguistic Validation

The model is validated against lexical, orthographic, phonological, and syntactic properties associated with AAE, which increasingly align with AA-associated language.

  • 3.1 Lexical-Level Variation: 79.1% of heavily AA-aligned words were absent from the dictionary, compared with 58.2% of heavily white-aligned words.Heavily aligned words satisfy rAA(w) ≥2 or rwhite(w) ≥2, respectively.
  • 3.2 Internet-Specific Orthographic Variation: AA-associated words include online-specific abbreviations, shortenings, and spelling variants that expose usages such as thoe as a discourse marker.The analysis identifies orthographic patterns not attested in prior literature and illustrates how variants can reveal linguistic phenomena.
  • 3.3 Phonological Variation: For 30 of 31 phonological variants, rAA(w) ≥1, and 13 have rAA(w) ≥100, while brotha is the sole exception.The selected variants represent features including derhotacization, deletion of initial g and d, and realization of voiced th as d.
  • 3.4 Syntactic Variation: The analysis tracks habitual be, future gone, and completive done as three well-attested AAE aspectual or preverbal markers.The constructions are detected using sequences of unigrams and POS tags, including patterns for habitual be.
  • 3.4 Syntactic Variation: As posterior probability of AA increases, messages become more likely to contain each of the three syntactic constructions, peaking around the [0.7, 0.8) decile.Messages are divided into posterior-probability deciles, with 200,000 sampled from each decile for calculating construction proportions.

4 Lang ID Tools on AAE

Existing language identifiers often misclassify AAE-like English tweets as non-English, with disparities relative to white-aligned tweets. A demographic-language model combined with langid.py improves English identification on such messages.

  • Motivation: Language identification is crucial for social-media processing, but short, non-standard, and multilingual messages make the task challenging.Misclassification can affect downstream analyses such as filtering tweets before sentiment analysis.
  • Motivation: AAE is a dialect of American English and should therefore be classified as English for major-world-language identification.The authors hypothesize that identifiers trained on standard English may perform differently on AA- and white-aligned tweets.
  • Existing classifiers: 13.2% of AA-aligned tweets versus 7.6% of white-aligned tweets were classified as non-English by langid.py.Similar disparities were observed for both Twitter identifiers.
  • Existing classifiers: Only 3 of 300 sampled tweets classified as non-English by the tested systems were unambiguously written in another language.The sampled non-English predictions were therefore mostly English tweets.
  • Ensemble classifier: The ensemble accepts langid.py’s English predictions and revisits non-English predictions using demographic-language posterior probabilities.It returns English when the combined AA, Hispanic, and white posterior probability is at least 0.9 and the message contains model-vocabulary tokens.
  • Evaluation: On 2.2 million 2014 U.S. tweets, the ensemble classified an additional 5.4% as English, reflecting at least a 5.4% recall gain because these classifications were all or nearly all correct.Manual samples yielded 100% precision across the evaluated groups.

5 Dependency Parser Evaluation

The evaluation tests whether a widely used dependency parser performs differently on AA- and white-aligned tweets. Parse quality is substantially lower on AAE-like text, with statistically significant group differences.

  • Motivation: The study hypothesizes differential syntactic-tool accuracy because AAE varies lexically and syntactically from Standard American English.This extends earlier evidence of disparate accuracy for SAE-trained part-of-speech taggers.
  • Evaluation setup: SyntaxNet was evaluated against manual annotations of 200 messages, sampled equally from AA- and white-aligned corpora.The annotations used Stanford Dependencies and a partial edge-annotation approach focused on the root of the first major sentence.
  • Evaluation setup: The evaluation reports labeled recall for annotated dependency edges, with bootstrapped standard errors from 10,000 message resamplings.The reported between-group differences were statistically significant at p < 10^-6.
  • Results: 80.4% white-aligned accuracy contrasts with 64.0% AAE-tweet parse quality, while Stanford CoreNLP shows an even larger disparity.White-aligned performance is broadly consistent with prior work, whereas AAE performance is substantially worse.

6 Discussion and Conclusion

The paper presents a distantly supervised probabilistic model for uncovering dialectal language on Twitter and adapting NLP tools to it. The authors frame the work as a case study whose approach relies especially on demographic information available for AAE speaker communities.

  • Contributions: The model uses demographic correlations between a dialect and its speaker communities to uncover dialectal language on Twitter.The approach is described as distantly supervised and probabilistic.
  • Contributions: The model can close the performance gap between NLP tools applied to dialectal and standard text.The paper presents this capability as part of its broader adaptation of language technology to AAE.
  • Scope and future work: The work is a case study in dialect identification, characterization, and language-technology adaptation for AAE.Its identification strategy is assisted by the strong association between AAE speakers and demographic information in U.S. Census records.
  • Scope and future work: The authors note that many language communities and groups are not officially recognized, motivating future combinations of distant supervision with unsupervised linguistic models.This is presented as a direction for uncovering underrecognized dialectal language.

A Census demographics (§2.1)

The analysis simplifies Census race and ethnicity information into four commonly used demographic variables. Because these concepts are distinct and complex, the simplification is explicitly qualified.

  • Demographic variables: The model uses four demographic categories: non-Hispanic whites, Hispanics, non-Hispanic African-Americans, and Asians.These categories are commonly used in sociological studies of the United States.
  • Demographic variables: The paper sometimes calls these variables races for simplicity, although Census race and ethnicity are separate concepts.The authors note that the relationship between race and ethnicity is complex.
  • Data handling: The demographic values are renormalized when they do not sum to one, and cases summing to less than 0.5 are discarded.Such cases occur only in a small minority of observations.

B Unicode ranges for emoji removal (§4.1)

The preprocessing removes emoji and other symbolic characters using selected Unicode ranges. The ranges cover supplementary planes, symbol blocks, private-use characters, zero-width controls, and emoji variation sequences.

  • Preprocessing: Emoji and other symbolic characters are removed by filtering characters in specified Unicode ranges.The preprocessing rules were developed using Unicode and emoji documentation plus message-sample inspection.
  • Unicode ranges: 10000–1FFFF covers the Supplemental Multilingual Plane, including emoji, other symbols, and Egyptian hieroglyphics.
  • Unicode ranges: 30000–10FFFF covers the fourth and higher Unicode planes.
  • Unicode ranges: 02500-02BFF covers symbol blocks including box drawing, dingbats, mathematical symbols, arrows, and braille patterns.
  • Unicode ranges: The rules also remove private-use characters, zero-width controls, and emoji variation sequences.These include 0E000–0EFFF, 0200B–0200D, and 0FE0E–0FE0F.

C Posterior inference via CVB0 for ensemble classifier (§4.2.1)

Posterior inference estimates topic proportions from message words using CVB0 variational Bayes, with topic-word distributions learned from Gibbs samples and smoothed with pseudocounts. The method uses a symmetric Dirichlet prior and converges rapidly in the reported test cases.

  • Posterior inference: The topic-proportion prior is θ ∼Dir(α) with fixed symmetric α_k = 1/16.
  • Topic-word distributions: The topic-word distributions are estimated by averaging Gibbs samples from the last 50 samples of the sampler.The averaged counts are then smoothed and normalized.
  • Topic-word distributions: A pseudocount of 1 is added before normalization to prevent rare words from receiving overly low posterior expected counts.The authors note this choice is motivated by observed performance rather than detailed-balance theory.
  • Posterior inference: CVB0 iteratively updates soft topic assignments for each token to infer the posterior mean topic proportions of a message.The final estimate averages the soft topic assignments across token positions.
  • Convergence: Within five iterations, ˆθ converges to within absolute 0.001 of a fixed point on test cases.

D Syntactic dependency annotations (§5)

The study evaluates SyntaxNet dependency parses against manual Stanford Dependencies annotations for AAE- and white-aligned tweets. A partial annotation strategy focuses on root-centered edges, while the formalism accommodates several AAE constructions but leaves important annotation limitations.

  • Parser and formalism: SyntaxNet outputs grammatical relations in Stanford Dependencies version 3.3.0.
  • Annotation procedure: Manual annotation used Brat and compared parser predictions with randomized messages whose demographic inferences were hidden from annotators.Annotators could inspect parser output while adding dependency edges to an unannotated message copy.
  • Annotation procedure: For each tweet, annotation covered the immediate dependent edges of the root word of the first major sentence, typically its main verb.This partial approach increased message coverage and statistical power relative to fully annotating every word.
  • Tokenization: Twitter tokenization and parser processing create challenges for apostrophe-less and multiword forms such as dont and af.The annotation convention assigns incoming constituent-word edges to the token, with the earliest token taking precedence in conflicts.
  • AAE syntax: The Stanford Dependencies formalism represented AAE constructions such as gone-V, done-V, habitual be, and null copulas.Several aspectual forms were analyzed as auxiliary verbs, while the copula treatment extended to null copulas.
  • Limitations: Punctuation edges were excluded, discourse edges were difficult and sometimes deferred to parser predictions, and partial labeling does not penalize missing edges.The authors identify these issues as requirements for a future full treebank.

E Annotation materials

The authors provide the dependency annotations and accompanying working notes as online materials in Brat plaintext format.

  • Materials: The annotations are supplied with the paper’s online materials.
  • Materials: The materials include working notes about difficult annotation cases.
  • Materials: Annotations are formatted in Brat’s plaintext format.
Loading 1608.08868v1…