Source-linked AI summary
REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets
Angelina Wang, Alexander Liu, Ryan Zhang, Anat Kleiman, Leslie Kim, Dora Zhao, Iroha Shirai, Arvind Narayanan, Olga Russakovsky
TL;DR
Visual datasets contain many forms of under-representation, while no algorithm identifies every bias and manual review cannot scale. REVISE automates preemptive analysis across object, person, and geography dimensions, surfacing potential biases and suggesting mitigation steps, with examples involving object size, inferred gender, and geographic portrayal.
Problem
Visual datasets under-represent objects, contexts, people, geographic locations, and cultures, while no algorithm identifies all biases and manual review is infeasible at modern scale.
Method
REVISE automates discovery of potential biases and annotation issues across object-based, person-based, and geography-based analyses, including duplicate-object identification.
Results
REVISE surfaces concrete dataset patterns: airplane images are frequently large, annotators infer male gender for 69% of ambiguous small people in OpenImages, and about 47% of YFCC100m countries are visitor-photographed.
Takeaways & Limitations
The tool brings potential visual-dataset biases to light and offers actionable next steps, while users must judge which patterns are problematic in cultural and historical context.
Takeaways & Limitations
Determining which surfaced biases are problematic cannot be automated, and their downstream impact depends on the specific models and tasks.
Abstract
from arXiv · showhide
Machine learning models are known to perpetuate and even amplify the biases present in the data. However, these data biases frequently do not become apparent until after the models are deployed. Our work tackles this issue and enables the preemptive analysis of large-scale datasets. REVISE (REvealing VIsual biaSEs) is a tool that assists in the investigation of a visual dataset, surfacing potential biases along three dimensions: (1) object-based, (2) person-based, and (3) geography-based. Object-based biases relate to the size, context, or diversity of the depicted objects. Person-based metrics focus on analyzing the portrayal of people within the dataset. Geography-based analyses consider the representation of different geographic locations. These three dimensions are deeply intertwined in how they interact to bias a dataset, and REVISE sheds light on this; the responsibility then lies with the user to consider the cultural and historical context, and to determine which of the revealed biases may be problematic. The tool further assists the user by suggesting actionable steps that may be taken to mitigate the revealed biases. Overall, the key aim of our work is to tackle the machine learning bias problem early in the pipeline. REVISE is available at https://github.com/princetonvisualai/revise-tool
1 Introduction
REVISE addresses the difficulty of detecting biases in large visual datasets before model deployment by automatically surfacing patterns across object, person, and geography dimensions. It illustrates these analyses with findings from several widely used datasets and leaves judgments about problematic bias to users.
- Manual review cannot feasibly identify all biases in modern large-scale datasets.
- REVISE automatically surfaces potential under- and different-representations through object-based, person-based, and geography-based metrics.
- Object-based analysis examines object frequency, scale, context, and diversity of representation.
- Person-based analysis examines demographic representation and how object portrayals differ across demographic groups.
- Geography-based analysis considers regional portrayal and its interaction with represented objects and people.
- REVISE supports dataset builders during collection and dataset users assessing biases their models may inherit from a dataset.
- Example Findings: 69% of inferred gender labels for people too small to assess were male, while around 47% of YFCC100m countries were predominantly photographed by visitors rather than locals.
- The tool surfaces unusual patterns and possible mitigation actions, but users must determine which findings are problematic.
2 Related Work
Related work addresses dataset bias, model and algorithmic fairness, and automated bias-detection tools. REVISE differs by examining visual datasets and annotations directly rather than relying primarily on trained-model interrogation.
- Visual datasets are collected through search queries or website scraping, then extensively cleaned and annotated by humans or automated tools.
- Dataset-bias research includes cross-dataset analysis, alternative data-collection practices, and concerns about consent and justice.
- Prior computer-vision tools diagnose object-detector weaknesses or investigate dataset bias in video activity recognition.
- Algorithmic-fairness research develops model-side interventions, whereas this work examines bias earlier from the dataset side.
- Unlike several automated detectors, REVISE directly analyzes datasets and annotations for model-agnostic patterns across visual data.
3 Tool Overview
REVISE is a general, interactive notebook tool that computes metrics from visual datasets and available annotations across object, person, and geography axes. It supports local, customizable exploration and produces summaries and actionable insights.
- REVISE requires an image dataset and available annotations, with computable analyses determined by the annotations supplied.
- REVISE uses external tools and pretrained models to derive some metrics, while acknowledging that those models may contain biases.
- Metrics identify anomalous patterns and provide data-driven suggestions for correcting distributions.
- The Jupyter notebook enables interactive exploration, local execution for privacy, metric customization through open-source code, and automatic summary-PDF generation.
- Object-based insights use instance labels and optionally bounding boxes and categories, while semantic labels such as scenes may be inferred automatically.
- Person-based analyses require sensitive-attribute labels and can support arbitrary demographic groupings and regression for ordinal attributes.
3. Geography-based insights flexibly allow for la-
Geography-based analysis accepts region names or GPS coordinates and can use global or user-supplied geographic boundaries. It combines geographic labels with object, demographic, and external annotations.
- Geography labels can be supplied as region strings such as “Portugal” or “Nigeria,” or as GPS latitude and longitude coordinates.
- REVISE uses a global map by default, while users can override it with a custom GeoJSON file.
- Geography labels are analyzed alongside object, demographic, and external annotations at image or regional level.
- All metrics run automatically, after which statistically significant findings are selected to demonstrate the insights provided by each metric.
4 Object-Based Analysis
REVISE’s object-based analysis quantifies object frequency, scale, context, and diversity, using annotations and derived visual features to expose unusual dataset patterns. It also supports analysis at both object and supercategory levels.
- Object counts: Oven and refrigerator counts fall below COCO’s median object-class count but are over-represented within the appliance category.
- Object counts: REVISE reports object counts within supercategories and across all objects to assess consistency with a dataset’s target distribution.Supercategories make large label inventories more interpretable; for example, COCO’s appliance category contains oven, refrigerator, and microwave.
- Duplicate annotations: Duplicate annotations can assign two names to the same object instance, such as cup and mug or trumpet and trombone.
- Object scale: Object scale is measured using each instance’s fraction of image area to support comparisons across object classes and image resolutions.Object size is linked to recognition accuracy and semantic importance in an image.
- Context and diversity: REVISE measures contextual diversity through pairwise object co-occurrence and scene diversity through entropy over scene categories.Scene categories are assigned to one of 16 scene groups using a ResNet-18 trained on Places; appearance diversity uses 64-dimensional bounding-box features from a ResNet-110 trained on CIFAR-10.
4.2 Object-based Actionable Insights
REVISE turns detected object-distribution issues into data-collection guidance, combining contextual search queries with a visualization of diversity-versus-commonness tradeoffs. The user makes the final mitigation decision.
- Mitigation guidance: REVISE provides actionable steps because identifying dataset biases or other issues does not always make remediation clear.
- Search queries: For web-query datasets, REVISE generates “XX and YY” searches combining a target object class with a contextual term.
- Search queries: COCO airplanes are predominantly large, so queries pairing airplanes with surfboards or mountain, desert, and sky scenes can guide scale diversification.
- Tradeoff visualization: Furniture in water, ice, snow offers greater appearance diversity but is rarer than furniture in outdoor sports fields and parks.
- Tradeoff visualization: The tradeoff visualization helps users balance a pair’s commonness against its contribution to diversity when selecting augmentation queries.
5 Person-Based Analysis
REVISE analyzes how people are represented by prominence, context, object co-occurrence, and interaction, revealing demographic patterns that may warrant dataset intervention.
- Person Prominence: Male people occupy more image area and sit closer to the center than female people in COCO.Male prominence is .268 ± .213 versus .138 ± .148 for females, while center distance is .363 ± .218 versus .510 ± .250.
- Person Prominence: As skin tone becomes darker, people are more likely to be smaller and farther from the image center.Jonckheere’s trend test reports p-values of 2.11e−7 for size and .014 for distance.
- Contextual representation: COCO associates females with indoor shopping, dining, furniture, accessories, and appliances, while males align more with outdoor sports, water, and vehicles.These trends reflect gender stereotypes and may propagate into models.
- Contextual representation: In COCO, lighter skin tones trend toward home or hotel scenes and furniture, whereas darker skin tones trend toward outdoor transportation and vehicles.The reported trends are statistically significant under Wald tests for non-zero regression slopes.
- Instance Counts and Distances: Object-instance analysis identifies motorcycles and buses with darker skin tones, and beds and couches with lighter skin tones.OpenImages similarly overrepresents cosmetics, dolls, and washing machines with females, and rugby balls, beer, and bicycles with males.
- Instance Counts and Distances: Distance between people and objects reveals interaction differences that frequency counts can miss, such as males playing organs while females are merely nearby.The scaled distance measure is used as a proxy for interaction and distinguishes co-occurrence from direct engagement.
- Actionable Insights: Interpreting and mitigating demographic representation requires considering historical context and whether equal representation is appropriate for the intended task.The downstream impact of unequal representation depends on the specific models and tasks.
6 Geography-Based Analysis
Geography-based analysis examines how geographic regions interact with the objects and people represented in images. REVISE analyzes these relationships because geographic bias can affect model generalization.
- Geographic analysis considers regional portrayal alongside object and person appearances.Geography influences both the types of objects represented and the people pictured.
- The study evaluates geography in COCO, YFCC100m, and the New York-centric BDD100K dataset.COCO geography labels are derived for a subset by querying Flickr, while YFCC100m and BDD100K provide additional geographic settings.
- The geography section presents metric findings and discusses possible responses to the revealed patterns.
6.1 Geography-based Metrics
REVISE measures geographic representation through distributions, object and language associations, demographic portrayals, income, and weather. The analyses reveal substantial regional imbalances and appearance differences across datasets.
- Geographic distribution: YFCC100m is drastically overrepresented by the USA, while Africa is very sparsely represented, creating potential generalization problems for underrepresented regions.The analysis compares overall geographic distribution and relates regional representation to model performance on unfamiliar locales.
- Geography by Object: YFCC100m tag frequencies expose regional object-label differences, while visual examples show that “dish” refers mainly to food in Eastern Asia and to other objects elsewhere.These differences indicate that the semantic meaning and visual appearance associated with a tag can vary across subregions.
- Geography by Language: Many poorly represented countries are also under-represented by local photographers, even after applying a lower-bound confidence interval.The analysis combines geographic labels with language annotations to distinguish locale from who portrays a country.
- Geography by Income: The two wealthiest BDD100K ZIP-code deciles contain far more images than the rest, and the difference persists when measured per capita rather than per square mile.Income-linked representation is also associated with systematic differences in bicycle and pedestrian labels.
- Geography by Weather: Snowy images comprise 0.3% of Northern California images and 10% of New York City images in BDD100K.The differing weather distributions matter when deploying models in settings unlike their training environment.
6.2 Geography-based Actionable Insights
REVISE’s geography-based findings support dataset augmentation and geographically diverse evaluation. The appropriate interventions depend on the dataset, model, and deployment context.
- Dataset builders can augment data to address geographic under- and over-representation and avoid one-sided views of regions.
- Dataset users should test models on geographically diverse data representative of deployment regions to check for regional overfitting.
- Data collection should explicitly seek images from underrepresented locales, particularly images made by people who live there.The paper connects geographic diversity with avoiding representation based solely on digital availability.
7 Discussion
REVISE can surface many visual dataset biases but cannot identify all of them or produce a universally unbiased dataset. Mitigation must account for practical constraints, conceptual choices, downstream harms, and the social categories used in analysis.
- REVISE makes no claim to identify all visual biases, and creating an “unbiased” dataset may be unrealistic.Practical barriers include many dataset categories and difficulty collecting images from places with few people online.
- Appropriate interventions vary by dataset and application, requiring attention to specific downstream harms and sometimes additional data sources.Removing people may suit an object-recognition benchmark but not a self-driving application; production interventions should consider harms such as poor neighborhood performance.
- The analysis reifies socially constructed categories by operationalizing gender and using skin tone as a proxy for race.The authors note that these choices can harm groups including non-binary individuals.
8 Conclusion
REVISE automates discovery of potential biases in visual datasets and annotations across object-based, person-based, and geography-based dimensions. It surfaces anomalous patterns and actionable next steps, while users must determine which biases are problematic.
- REVISE automates discovery of potential biases in visual datasets and their annotations.
- The tool investigates object-based, person-based, and geography-based bias dimensions.
- Users must determine which surfaced biases are problematic, because that judgment cannot be automated.
A Appendices
The appendices detail REVISE’s person-based and object-based validation procedures, limitations, and examples of actionable dataset-collection queries. They examine gender-label inference, distance as an interaction proxy, annotation inconsistencies, and object-scale biases.
- Gender label inference: Gender label inference flags gender assignments when a person is too small to identify or lacks a detected face.The analysis uses a 1,000-pixel threshold for small people and automated face detection, while noting that gender is not visually perceivable by annotators.
- Gender label inference: Clearer annotator guidance and removal of indeterminate gender labels are recommended, although the broader question of whether gender labels should be assigned remains outside scope.
- Distance as an interaction proxy: Distances between person and object centers are used as a proxy for interaction, with smaller distances more likely to indicate interaction.The measure is normalized by the square root of the product of person and object areas and assumes people and object instances have comparable sizes within classes.
- Distance as an interaction proxy: For six tested object categories, interaction distances are lower than non-interaction distances, but overlap is greater for ball and table.The overlap corresponds to lower mean per class accuracy and reflects that these objects may be approached without direct interaction.
- Query-based mitigation: Pairwise search queries can augment datasets with desired object scales or contexts, but some queries may return no results and search distributions may differ from dataset distributions.Examples include pairing airplane with scene terms or objects to alter apparent scale and context.