Source-linked AI summary
Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchy
Kaiyu Yang, Klint Qinami, Li Fei-Fei, Jia Deng, Olga Russakovsky
TL;DR
ImageNet’s person subtree may encode offensive concepts and unequal representation, motivating dataset-level scrutiny. The paper analyzes and filters the subtree, assesses imageability and demographics, and recommends balancing procedures. It reports that at most 158 of 2,832 synsets should remain, while noting limitations in annotation scope and demographic measurement.
Problem
ImageNet’s person subtree may contain inappropriate labels and unequal demographic representation, while dataset bias can contribute to discriminatory computer vision behavior.
Method
The paper analyzes the person subtree, filters unsafe and nonvisual synsets, annotates demographic attributes, and recommends procedures for increasing image diversity.
Results
At most 158 of 2,832 existing person-subtree synsets should remain; 139 have at least 100 images for further demographic analysis.
Takeaways & Limitations
The authors provide initial dataset-filtering and balancing steps for improving representation in ImageNet’s person subtree.
Takeaways & Limitations
The work does not address subject privacy and ethics in depth, and demographic annotations may be imperfect or affected by detector bias.
Abstract
from arXiv · showhide
Computer vision technology is being used by many but remains representative of only a few. People have reported misbehavior of computer vision models, including offensive prediction results and lower performance for underrepresented groups. Current computer vision models are typically developed using datasets consisting of manually annotated images or videos; the data and label distributions in these datasets are critical to the models' behavior. In this paper, we examine ImageNet, a large-scale ontology of images that has spurred the development of many modern computer vision methods. We consider three key factors within the "person" subtree of ImageNet that may lead to problematic behavior in downstream computer vision technology: (1) the stagnant concept vocabulary of WordNet, (2) the attempt at exhaustive illustration of all categories with images, and (3) the inequality of representation in the images within concepts. We seek to illuminate the root causes of these concerns and take the first steps to mitigate them constructively.
1 INTRODUCTION
The paper examines how ImageNet’s person subtree can encode bias and misrepresentation, then proposes filtering unsafe and nonvisual categories and balancing demographic representation.
- Motivation: ImageNet’s influence makes its data and label distributions consequential for downstream computer vision systems.The dataset has supported models across object detection, activity understanding, image captioning, and depth-related tasks.
- Scope: The full ImageNet person subtree contains 2,832 categories, whereas the ILSVRC subset contains only three people categories.The person subtree accounts for roughly 8.3% of ImageNet images.
- Research focus: The analysis identifies stagnant WordNet vocabulary, exhaustive category illustration, and unequal demographic representation as potential sources of downstream problems.The authors seek to understand these root causes and take initial mitigation steps through annotation procedures.
- Contributions: The authors identify 1,593 potentially unsafe labels among 2,832 people categories and plan to remove them from ImageNet.The remaining 1,239 categories are then examined for visual relevance and demographic representation.
- Contributions: Only 158 of the remaining 1,239 categories are visual, motivating further filtering of the person subtree for visual-recognition training.The authors also release annotation interfaces intended to support similar cleanup of other benchmarks.
2 RELATED WORK ON FAIRNESS IN MACHINE LEARNING
Prior work links biased dataset distributions to discriminatory model behavior and motivates dataset-level interventions alongside algorithmic fairness methods.
- Dataset bias: Models exploit demographic correlations in data, which can produce discriminatory behavior against underrepresented groups.Underrepresentation also makes categories harder to learn, while majority-focused prediction can amplify dataset bias.
- Transparency: Dataset documentation efforts encourage creators to disclose collection procedures so downstream users can assess suitability.This paper similarly examines ImageNet’s data-collection pipelines and their implications.
- Ethical concepts: Prior research has identified ethically questionable concepts involving beauty, sexual orientation, and gender annotation in datasets.The paper considers underlying causes for such concepts and proposes initial solutions.
- Scope: Privacy and subject-ethics concerns are acknowledged but treated as outside this work’s scope.The paper directs readers to prior detailed discussion rather than addressing these issues directly.
- Algorithmic limits: Algorithmic fairness interventions can satisfy fairness constraints at the expense of overall model accuracy.The paper therefore argues that algorithmic interventions alone are unlikely to be sufficient.
3 BACKGROUND: THE IMAGENET DATA COLLECTION PIPELINE
ImageNet grounds WordNet synsets with Internet images and manual annotations, a pipeline whose stages shape the dataset’s visual concepts and labels.
- Dataset scope: The full ImageNet dataset contains 14,197,122 images illustrating 21,841 concepts, beyond the 1,000-concept ILSVRC subset.The paper focuses on this broader dataset rather than only the commonly used challenge subset.
- Concept vocabulary: WordNet supplies ImageNet’s concept vocabulary through synonym sets organized by the “is a” relation.ImageNet provides visual grounding for these semantic synsets.
- Candidate images: ImageNet retrieves candidate images from Internet search engines for each WordNet concept, using multiple engines, query expansion, and translated terms.The pipeline is designed to increase the variety of retrieved images.
- Manual cleanup: Because image search engines were only about 10% accurate, ImageNet requires manual cleanup of candidate images.Workers receive a target concept, its WordNet definition, a Wikipedia link, and candidate images through AMT tasks.
4 PROBLEM 1: STAGNANT CONCEPT VOCABULARY
The paper finds that WordNet’s aging vocabulary leaves inappropriate concepts in ImageNet’s person subtree and applies a binary unsafe-labeling procedure to filter them.
- Problem: WordNet contains outdated or abusive synsets, including racial and sexual slurs, creating particular concerns in ImageNet’s person subtree.Earlier construction-time removal of explicitly offensive synsets was imperfect.
- Annotation method: Annotators flag a synset as unsafe when it is offensive or sensitive because applying it to people may cause offense.The procedure excludes potentially inappropriate synsets rather than assigning graded offensiveness scores.
- Results: 1,593 of 2,832 person-subtree synsets were identified as unsafe, associated with 600,040 ImageNet images.Removing them would leave 577,244 images in the safe synsets.
- Interpretation: A safe synset is defined only by its label not being deemed offensive, not by whether visual cues can ethically or usefully support the label.Safety therefore does not establish visual validity or ethical appropriateness of recognition.
- Limitations and future work: Offensiveness judgments are subjective and evolve with cultural context, so the authors invite community reporting of additional unsafe synsets.The remaining concept set is relatively small, making this approach feasible.
5 PROBLEM 2: NON-VISUAL CONCEPTS
ImageNet contains many person concepts that are difficult to characterize accurately from images, partly because large-scale search and annotation preserve distinctive but biased visual examples. The paper measures imageability, recommends filtering low-imageability synsets, and finds higher recognition accuracy for imageable concepts, while noting important limitations.
- 5 PROBLEM 2: NON-VISUAL CONCEPTS: Search engines can surface distinctive images for concepts that are inherently non-imageable, producing biased visual representations.Stock photography can amplify and perpetuate stereotypes, especially for categories such as nationality.
- 5.1 Annotating imageability: Annotators rate 1,239 safe person synsets for imageability using a five-point crowdsourced scale with gold-standard quality control.The final synset score is the average of worker ratings, and workers with high gold-question error are removed.
- 5.2 Results and impact on ImageNet after removing the non-imageable synsets: A threshold below 4 flags 1,081 synsets and 443,547 images, leaving 158 synsets and 133,697 images for potential use.Examples retained include rock star (4.86), skier (4.50), and cashier (4.20).
- 5.3 Limitations of the imageability annotation: The annotation uses ease of forming a mental image as a surrogate for ease of characterizing a concept with visual cues.The paper notes that daughter and sister can score highly because they arouse imagery despite lacking distinctive visual cues.
- 5.3 Limitations of the imageability annotation: Unfamiliar terms receive lower imageability ratings, so filtering them may also improve annotation accuracy and remove less relevant concepts.Cotter scores 1.70 versus 3.36 for generic peasant.
- 5.3 Limitations of the imageability annotation: Even imageable synsets may retain culturally or search-engine-biased depictions, so downstream users should exercise caution.Examples include mother images featuring women holding children and beekeeper images featuring people with bees.
- 5.4 Relationship between imageability and visual recognition models: 63.8% accuracy on easily imageable synsets exceeds 53.0% on non-imageable synsets, with a positive correlation between imageability and recognition accuracy.The reported Pearson correlation is r = 0.23 with p = 0.0048.
- 5.4 Relationship between imageability and visual recognition models: Imageability can mismatch visual learnability: black belt is easy to classify, conversational partner is hard to classify, and bridegroom is imageable but culturally biased.The cited examples report accuracies of 92% for black belt and 20.5% for conversational partner.
6 PROBLEM 3: LACK OF IMAGE DIVERSITY
The paper finds persistent demographic imbalance in ImageNet’s person subtree and proposes balancing image distributions while acknowledging annotation, privacy, and evaluation trade-offs.
- Sources of insufficient representation: ImageNet’s demographic bias persists despite diversification efforts, partly because search and manual cleanup can reinforce stereotypes.The paper identifies underrepresentation as a remaining concern in the person subtree.
- Demographic annotation: The authors annotate imageable protected attributes—color, gender, and age—using worker judgments, quality control, and aggregation across independent workers.Workers with average IOU below 0.5 are excluded before annotations are aggregated.
- Gender distribution: Female representation is underrepresented overall, although individual synsets may be male- or female-dominated.Birth and scuba diver have unusually high Unsure rates because visual gender cues are obscured.
- Skin-color distribution: Only 6.2% of images are labeled Dark on average across synsets, with substantial Dark representation concentrated in rapper and basketball player.The reported proportions are 66.4% for rapper and 34.5% for basketball player; first lady is an exception at 51.9%.
- Balancing methodology: The proposed interface rebalances one attribute within each synset by removing overrepresented-category images, while retaining 90% of minority-category images to reduce inference of removed demographics.Balancing multiple attributes would leave too few images and create additional privacy concerns.
- Impact and trade-offs: Balancing could be offered for 131 synsets by gender, 117 by skin color, and 81 by age, but it prevents demographic-subset error-rate evaluation.The authors present balancing as preferable to retaining strong bias, releasing ethically problematic annotations, or collecting additional images.
7 DISCUSSION
The paper filters and analyzes the ImageNet person subtree, finding that most categories should be removed and recommending balance improvements for the retained categories. The authors also identify remaining work beyond this subtree.
- 7 DISCUSSION: At most 158 of 2,832 person-subtree synsets should remain because the others are inappropriate for visual recognition.Of the retained synsets, 139 have at least 100 images for further analysis.
- 7 DISCUSSION: The authors analyze gender, skin color, and age distributions in the 139 sufficiently populated synsets and recommend procedures for better balance.
- 7 DISCUSSION: The retained 139 categories remain large-scale compared with datasets such as PASCAL, CelebA, COCO, and CUB-200.
- 7 DISCUSSION: The filtering does not affect the three person-subtree synsets in the commonly used 1,000-category ILSVRC subset.
- 7 DISCUSSION: Further work is needed outside the person subtree because incidental people appear in synsets for pets, household objects, and sports.The authors release annotation interfaces to support continued work.
A APPENDIX
The appendix provides annotation interfaces and additional results, organized according to the main paper's sections.
- A APPENDIX: The appendix includes annotation interfaces and additional results organized by the corresponding sections of the main paper.
A.1 PROBLEM 1: STAGNANT CONCEPT VOCABULARY
The appendix documents the instructions and synset lists used to annotate offensive and safe categories in the person subtree.
- A.1 PROBLEM 1: STAGNANT CONCEPT VOCABULARY: The in-house instructions for annotating synset offensiveness are shown in Fig. A.
- A.1 PROBLEM 1: STAGNANT CONCEPT VOCABULARY: The offensive and safe synset lists are available as separate files at image-net.org/filtering-and-balancing/unsafe_synsets.txt and safe_synsets.txt.
A.2 PROBLEM 2: NON-VISUAL CONCEPTS
The appendix describes crowdsourced imageability annotation and quality control for safe person-subtree synsets. Ratings are filtered, aggregated, and collected dynamically when concepts are ambiguous.
- A.2 PROBLEM 2: NON-VISUAL CONCEPTS: Fig. C shows the crowdsourcing interface used to collect imageability scores.
- A.2 PROBLEM 2: NON-VISUAL CONCEPTS: Quality control uses gold-standard questions rated either 5 for obviously imageable concepts or 1 for obviously non-imageable concepts.
- A.2 PROBLEM 2: NON-VISUAL CONCEPTS: Workers with imageability-rating error at least 2.0 are excluded, and multiple workers' ratings are averaged for stable synset estimates.
- A.2 PROBLEM 2: NON-VISUAL CONCEPTS: The heuristic collects more ratings for ambiguous synsets by modeling existing ratings with a Gaussian distribution and stopping after three consecutive high-probability ratings.
- A.2 PROBLEM 2: NON-VISUAL CONCEPTS: The gold-standard table records synset IDs, synset names, and ground-truth imageability values for example occupations and people categories.
- A.2 PROBLEM 2: NON-VISUAL CONCEPTS: The average number of ratings is 8.8, and 72% of synsets need no more than 10 ratings.
A.3 PROBLEM 3: LACK OF IMAGE DIVERSITY
The paper presents a user interface for annotating protected attributes, with the interface shown in Figure D.
- The annotation task concerns protected attributes.
- The paper uses a user interface for this annotation task.
- Figure D displays the interface used to annotate protected attributes.
Instructions:
The instructions define three labels for synsets and emphasize cautious handling of potentially harmful or ambiguous concepts. They also specify the annotation format and distinguish this task from imageability judgments.
- Annotators label synsets as safe, sensitive, or offensive using labels 2, 1, and 0, respectively.
- Sensitive synsets are not inherently offensive but may cause offense when applied inappropriately.
- Sensitive examples include classifications by culture, race, politics, religion, sex, gender, sexual orientation, and negative connotations.
- Offensive synsets are inherently offensive, including racial slurs and profanity.
- Annotators should mark a synset sensitive or offensive whenever they can think of any reason it would be bad to include.
- The spreadsheet contains synset identifiers, words, definitions, and a labels column initialized to zeros.