Source-linked AI summary
Inferring Gender from Names on the Web: A Comparative Evaluation of Gender Detection Methods
Fariba Karimi, Claudia Wagner, Florian Lemmerich, Mohsen Jadidi, Markus Strohmaier
TL;DR
The paper addresses limited evidence about the accuracy and subgroup bias of automated gender inference from names on the Web. It evaluates name-based and image-based methods on labeled scientists, finding that country-dependent weaknesses in name-based approaches can be reduced by combining both sources.
Problem
Automated gender-detection methods are widely used, but their accuracy and bias across demographic sub-populations are insufficiently documented.
Method
The study compares name-based methods on manually labeled scientists and tests mixed approaches that combine name-based detection with web-retrieved facial images.
Results
Mixed approaches achieve the best overall performance, while name-based accuracy varies by country and is particularly weak for several emerging nations.
Takeaways & Limitations
Combining name-based and image-based gender detection can reduce country-related performance disparities.
Takeaways & Limitations
Crowdsourcing does not scale well or cheaply, and content-based supervised methods require separate models for each language community.
Abstract
from arXiv · showhide
Computational social scientists often harness the Web as a "societal observatory" where data about human social behavior is collected. This data enables novel investigations of psychological, anthropological and sociological research questions. However, in the absence of demographic information, such as gender, many relevant research questions cannot be addressed. To tackle this problem, researchers often rely on automated methods to infer gender from name information provided on the web. However, little is known about the accuracy of existing gender-detection methods and how biased they are against certain sub-populations. In this paper, we address this question by systematically comparing several gender detection methods on a random sample of scientists for whom we know their full name, their gender and the country of their workplace. We further suggest a novel method that employs web-based image retrieval and gender recognition in facial images in order to augment name-based approaches. Our findings show that the performance of name-based gender detection approaches can be biased towards countries of origin and such biases can be reduced by combining name-based an image-based gender detection methods.
1. INTRODUCTION
The Web supports large-scale social research, but demographic information is often difficult to obtain. This motivates automated gender inference from names, whose accuracy and subgroup bias remain insufficiently documented.
- Web-based social research benefits from demographic information, but age, gender, and ethnicity are often difficult to obtain.
- Previous studies inferred gender from names using repositories such as the US Social Security Administration database.
- Existing work often does not report the accuracy or subgroup bias of gender-detection methods.
- Crowdsourcing does not scale well or cheaply, while content-based supervised models require separate models for each language community.
- This paper compares name-based methods and proposes augmenting them with face-recognition techniques to improve accuracy across heterogeneous sub-populations.
2. DATA AND METHOD
The study evaluates gender-detection methods on manually labeled scientists using names, databases, commercial services, and images. It also introduces mixed approaches that combine name-based and image-based evidence, with one variant retrieving images only for unresolved names.
- The evaluation uses a manually labeled random sample of academics containing full names, institutions, countries, and gender.The final list contains 693 male names and 723 female names after ambiguous and repetitive names were removed.
- Researchers evaluate name-based methods with full names and then combine name-based and image-based gender detection.
- Name-based and image-based methods: Unsupervised approaches use names or pictures as input without training, while common databases include SSA baby names and IPUMS Census data.
- Evaluation overview: The mixed approach outperforms other methods by at least 9% in Table 1, while Table 2 reports country-specific accuracy with mixed approaches best for most countries.
- Name-based and image-based methods: Sexmachine associates names with gender and country-specific popularity, enabling disambiguation by country of origin.
- Name-based and image-based methods: Genderize uses large datasets from online social-network profiles and returns a confidence value through its API.
- Name-based and image-based methods: The image method retrieves the first five Google thumbnails for a full-name query and applies image recognition to the resulting pictures.Using the full name can better indicate gender for ambiguous or unisex first names within a culture.
- Mixed approaches: Mixed1 applies Genderize first and Face++ only to unidentified names, whereas Mixed2 gives both methods equal confidence-weighted influence.Mixed1 is more efficient because it does not retrieve pictures for the entire population.
3. RESULTS AND DISCUSSION
Mixed approaches achieve the strongest overall performance and outperform other methods across most countries. Results also show that error rates vary substantially by country, with name-based methods weaker for several emerging nations and no systematic male–female bias detected.
- Mixed approaches achieve the best overall results, outperforming all other methods by at least 8% accuracy.Among individual methods, Face++ and Genderize perform relatively better than the others.
- No systematic gender bias can be asserted because all methods produce comparable results for male and female classes.
- Country of residence strongly affects error rates, with name-based methods performing well in western industrialized countries but deteriorating in China, South Korea, and Brazil.The paper attributes this pattern to insufficient coverage of popular names from these countries in existing databases.
- For South Korea, image-based approaches achieve accuracy at least 16% better than the best name-based method.
- The proposed mixed approaches outperform existing methods for most countries and can substantially improve performance for emerging nations.