PaperlayerView exact revision on arXiv

Source-linked AI summary

Predicting Race and Ethnicity From the Sequence of Characters in a Name

Rajashekar Chintalapati, Suriyan Laohaprapanon, Gaurav Sood

arXiv:1805.02109v2stat.APstat.ML

TL;DR

Existing Census-based name methods are limited, so this paper models race and ethnicity from name characters. Long Short-Term Memory performs best, reaching 85% out-of-sample accuracy for full names and supporting finer-grained inference.

  • Problem

    Census popular-last-name approaches suffer from a biased small set and aggregated information, limiting name-based race and ethnicity inference.

  • Method

    The paper predicts the modal race and ethnicity associated with names by modeling character sequences in last names and full names.

  • Results

    85% out-of-sample accuracy is achieved by the best-performing full-name model.

  • Takeaways & Limitations

    Full-name models provide finer-grained inference and may improve prediction when first names are available.

  • Takeaways & Limitations

    Census data used in the modeling is incomplete and biased.

Abstract

from arXiv · show

To answer questions about racial inequality and fairness, we often need a way to infer race and ethnicity from names. One way to infer race and ethnicity from names is by relying on the Census Bureau's list of popular last names. The list, however, suffers from at least three limitations: 1. it only contains last names, 2. it only includes popular last names, and 3. it is updated once every 10 years. To provide better generalization, and higher accuracy when first names are available, we model the relationship between characters in a name and race and ethnicity using various techniques. A model using Long Short-Term Memory works best with out-of-sample accuracy of .85. The best-performing last-name model achieves out-of-sample accuracy of .81. To illustrate the utility of the models, we apply them to campaign finance data to estimate the share of donations made by people of various racial groups, and to news data to estimate the coverage of various races and ethnicities in the news.

Inferring Race and Ethnicity from Names

The paper predicts the modal race and ethnicity associated with names, addressing limitations of Census popular-last-name data. The resulting models extend prediction to names absent from that list and to databases containing spelling errors.

  • Limitations of Existing Approaches: Census popular-last-name data are limited by a biased set of popular names, missing first names, and updates only once every 10 years.The data also disproportionately represent non-Hispanic Whites because names must be shared by at least 100 people.
  • Limitations of Existing Approaches: First names are especially important for identifying African Americans because their last names resemble those of non-Hispanic Whites, whereas their first names tend to be distinctive.
  • Modeling Approach: The models predict the modal race and ethnicity for people sharing a last name or full name.Last-name models group people by last name, while full-name models calculate the modal race and ethnicity for each full name.
  • Model Utility: The models predict race and ethnicity for names absent from the popular last-names list and for names in databases with spelling errors.For a person selected at random, the aggregated Census popular-last-name file provides the Bayes-optimal guess when the name is represented there.

Data

The study uses Florida’s 2022 voter-registration data, containing nearly 15 million voters with self-reported race, supplemented by Census last-name and North Carolina voter-registration data. Florida’s reporting instrument has debatable quality because it combines race and ethnicity and permits only crude categorization.

  • Florida voter-registration data: Florida’s 2022 voter-registration data cover nearly 15 million voters and include self-reported race.Race and ethnicity are self-reported, but the data’s quality is debatable.
  • Florida voter-registration data: The reporting instrument combines race and ethnicity, treats Hispanic as one category, and groups Indian Americans under Asians and Pacific Islanders.The instrument permits only crude categorization.
  • Data preparation: The final dataset has five categories: Asian/Pacific Islander, Hispanic, Non-Hispanic Blacks, Non-Hispanic Whites, and Other.Missing race or ethnicity cases are assumed missing at random and removed; multi-racial and Native American respondents are condensed into Other.
  • Data preparation: Cases with one-character last names are dropped, removing 75,000 of roughly 15 million observations.The study also uses Census Popular Last Name Data and North Carolina Voter Registration Data.

Models

The paper evaluates distance-based, character-feature, sequence, and synthetic-data models for predicting ethno-racial categories from names. Names are standardized and grouped into last-name or full-name datasets before classifiers are trained on fixed train, validation, and test splits.

  • Model families: The study compares edit-distance KNN, Bag-of-Characters Random Forest and Gradient Boosted Trees, LSTM, Transformer, and synthetic-data LSTM models.The synthetic-data LSTM is used for last names, where substantially less data are available.
  • Preprocessing: Names are title-cased, stripped of non-alphabetical characters and hyphens, and concatenated as last name plus first name for full-name models.Middle names are ignored.
  • Data construction: The final datasets group observations by last or full name, calculate conditional means for five ethno-racial categories, and assign the modal race.Classifiers use train, validation, and test proportions of .8, .1, and .1, respectively.
  • KNN: The KNN approach uses bi-character edit distance, cosine distance for last names, and Jaccard distance via LSH Minhash for 70% of full-name training data.The paper motivates this as an approximation when a complete name census is unavailable or names may contain spelling errors.
  • Neural models: The LSTM embeds each character in a 256-length real-valued vector, passes sequences through two LSTM layers and a fully connected layer, and applies log softmax.Optimization uses Negative Log Likelihood Loss and Adam.

Results

LSTM models outperform the alternatives for both last-name and full-name race and ethnicity prediction, with accuracy reaching 87% on census popular-name data and 85% for the best full-name model. Adding first names substantially improves performance, while synthetic-name augmentation yields no gain in Florida voter-registration training but raises census-data accuracy by one percentage point.

  • Last-name models: 78% accuracy is achieved by the best-performing KNN last-name model on the hold-out set, while LSTM is the only alternative that performs better and leads across subgroups.The LSTM model is also the top-performing model across subgroups.
  • Census popular-name data: 87% overall accuracy is achieved on census popular-name data, with accuracies of 98%, 58%, 42%, and 26% for Non-Hispanic Whites, Hispanics, Asians, and Non-Hispanic Blacks, respectively.The result is based on predicting race and ethnicity in the census popular names data.
  • Synthetic-name augmentation: No appreciable gains in accuracy result from augmenting Florida voter-registration training data with synthetic alternate spellings of popular last names.Synthetic names are generated by prompting ChatGPT for 10 common alternate spellings of each last name.
  • Census last-name model: 86% accuracy is achieved by an LSTM trained on census popular last names, increasing by a percentage point to 87% after synthetic-data augmentation.The Florida voter-registration-trained LSTM also performs well on census data, but the census data itself is used for this model.
  • Full-name models: 73% accuracy is achieved by the best-performing KNN full-name model, whereas LSTM dominates the other full-name models and never falls below 63% for any category except Other.LSTM is the only full-name model that outperforms KNN.

Applications

The models are applied to campaign-finance and news data to estimate racial and ethnic representation. Campaign contributions and news authorship and mentions are disproportionately associated with Non-Hispanic Whites, while African Americans and Hispanics are underrepresented in news.

  • Campaign contributions: 89.5% of campaign-contribution money was contributed by Non-Hispanic Whites, compared with 4.4% by Non-Hispanic Blacks, 2.6% by Hispanics, and 3.4% by Asians.Race and ethnicity were imputed for individual contributors in the 2014 campaign contribution database using the Florida full-name LSTM model.
  • News diversity: 78% of newsroom authors and 73.5% of people mentioned in the news were Non-Hispanic Whites.The analysis used the Top News dataset, parsing articles and extracting people’s names with named-entity recognition before inferring race and ethnicity.
  • News diversity: African Americans and Hispanics were underrepresented in both the newsroom and news mentions.This finding accompanies the reported overrepresentation of Non-Hispanic Whites among authors and people mentioned.

Discussion

The Census popular last-name dataset is a baseline when only a last name is known, but incomplete or biased data, spelling errors, and available first names complicate prediction. Under these conditions, last-name or full-name models may improve accuracy, with LSTM models performing best and KNN models second.

  • The Census popular last-name dataset provides the baseline race and ethnicity guess when only a last name is known.
  • Incomplete and biased Census data, spelling errors, and available first names complicate race and ethnicity prediction from names.
  • Under these conditions, using a last-name or full-name model may provide more accurate predictions.
  • LSTM models provide excellent accuracy, while KNN models surprisingly rank second.

Supporting Information · Performance of the KNN Models

The supporting information reports test-set performance for two KNN models using cosine distance and LSH Minhash Jaccard distance. The models achieve accuracy values of 0.78 and 0.73, respectively, with accompanying precision, recall, and f1-score averages.

  • Performance of the KNN Models: The KNN (K = 5) Cosine Distance model is evaluated on the test set using precision, recall, f1-score, and support.Its performance table reports accuracy separately from these metrics.
  • Performance of the KNN Models: The KNN (K = 10) LSH Minhash Jaccard Distance model is evaluated on the test set using precision, recall, f1-score, and support.Its performance table reports accuracy separately from these metrics.

Performance of the Random Forest Models · Performance of the Gradient Boosting Models

On the test set, Random Forest models achieved accuracy of 0.78 for last names and 0.77 for full names, while Gradient Boosted Trees achieved 0.75 and 0.68, respectively. The corresponding macro averages were higher for Random Forest than Gradient Boosted Trees across both name inputs.

  • Performance of the Random Forest Models: The Full Name Random Forest model reported macro-average precision of 0.77, recall of 0.77, and f1-score of 0.73.The test-set support was 901,862.
  • Performance of the Gradient Boosting Models: The Last Name Gradient Boosted Trees model reported macro-average precision of 0.74, recall of 0.75, and f1-score of 0.71.The test-set support was 105,664.
  • Performance of the Gradient Boosting Models: The Full Name Gradient Boosted Trees model reported macro-average precision of 0.70, recall of 0.68, and f1-score of 0.59.The test-set support was 901,862.

Performance of the LSTM Models · Performance of the Transformer Models

On the test set, LSTM models achieved accuracy of 0.81 for last names and 0.85 for full names, while Transformer models achieved 0.74 and 0.71, respectively.

  • Performance of the LSTM Models: 0.84 was the Full Name LSTM model’s average precision, while its average recall and f1-score were 0.85 and 0.84.These averages were reported over 901,760 test observations.
  • Performance of the Transformer Models: 0.67 was the Full Name Transformer model’s average precision, while its average recall and f1-score were 0.71 and 0.67.These averages were reported over 901,760 test observations.
Loading 1805.02109v2…