Source-linked AI summary
Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, Kai-Wei Chang
TL;DR
Web-sourced datasets for visual semantic role labeling and multilabel object classification contain gender bias, which structured predictors further amplify. The paper proposes corpus-level constraints with Lagrangian-relaxation calibration, reducing amplification across both tasks with almost no recognition-performance loss.
Problem
Web-sourced datasets and structured prediction models for visual recognition can encode and amplify social bias in correlations among labels.
Method
RBA calibrates existing structured predictors with corpus-level constraints and Lagrangian relaxation to reweight bias-creating factors.
Results
Bias amplification decreases by 47.5% for multilabel classification and 40.5% for vSRL, without affecting underlying visual-system performance.
Takeaways & Limitations
Structured predictors can amplify gender bias from data, while RBA reduces this effect across both visual recognition settings.
Takeaways & Limitations
The analysis considers only a binary gender representation perceived by dataset annotators, and finer-grained analysis is needed for production deployment.
Abstract
from arXiv · showhide
Language is increasingly being used to define rich visual recognition problems with supporting image collections sourced from the web. Structured prediction models are used in these tasks to take advantage of correlations between co-occurring labels and visual input but risk inadvertently encoding social biases found in web corpora. In this work, we study data and models associated with multilabel object classification and visual semantic role labeling. We find that (a) datasets for these tasks contain significant gender bias and (b) models trained on these datasets further amplify existing bias. For example, the activity cooking is over 33% more likely to involve females than males in a training set, and a trained model further amplifies the disparity to 68% at test time. We propose to inject corpus-level constraints for calibrating existing structured prediction models and design an algorithm based on Lagrangian relaxation for collective inference. Our method results in almost no performance loss for the underlying recognition task but decreases the magnitude of bias amplification by 47.5% and 40.5% for multilabel classification and visual semantic role labeling, respectively.
1 Introduction
Web-sourced visual recognition datasets contain gender bias, and structured predictors can amplify those associations. The paper proposes corpus-level calibration that reduces amplification while preserving recognition performance.
- Over 45% of verbs and 37% of objects exhibit gender bias greater than 2:1.
- Models amplify existing bias by 5.0% for vSRL and 3.6% for multilabel classification.
- The analysis simplifies gender to a binary perceived by dataset annotators, while noting that finer-grained analysis is needed for production deployment.
- RBA introduces corpus-level constraints and uses Lagrangian relaxation to reweight bias-creating factors in structured predictors.
- RBA reduces average bias amplification by 40.5% for vSRL and 47.5% for multilabel classification.
- Calibration methods do not affect underlying visual-system performance while reducing reliance on socially biased correlations.
2 Related Work
Prior work addresses bias in simpler classification settings and uses Lagrangian techniques for structured constraints. This paper applies corpus-level constraints specifically to reduce gender-bias amplification rather than improve accuracy.
- Bias in binary classification can be detected and corrected more easily than bias in structured prediction tasks.
- The paper presents the first proposed methods for reducing bias amplification in structured prediction tasks.
- Lagrangian relaxation and dual decomposition have been used for instance-level constraints in NLP.
- Related corpus-level constraint methods have addressed semi-supervised multilabel classification, whereas this work targets gender-bias reduction.
3 Visualizing and Quantifying Biases
The paper represents predictions as interdependent output variables and identifies demographic attributes whose correlations with other outputs may be amplified. Bias amplification is measured by comparing training-set and predicted evaluation-set bias scores.
- Structured prediction models jointly represent interdependent output variables as a structured output y.
- Demographic variables g are compared with correlated outputs o to identify correlations potentially amplified by a learned model.
- The bias score b(o, g) is computed from corpus co-occurrence counts of output o and demographic attribute g.
- An output is considered positively correlated with a demographic attribute when b(o, g) exceeds 1/|G|.
- Bias amplification is evaluated by comparing training bias scores with scores from predictions on an identically distributed unlabeled evaluation set.
- Mean bias amplification estimates the average amplification magnitude for output-demographic pairs that exhibited bias.
4 Calibration Algorithm
The paper introduces RBA, a corpus-level calibration framework that reduces bias amplification in structured predictions by enforcing training-distribution constraints through approximate Lagrangian-relaxation inference.
- Corpus-level Constraints: RBA injects corpus-level constraints so model predictions follow gender distributions observed in the training data.The constraints require predictions across the test corpus to match desired activity-specific gender ratios within a margin.
- Algorithm Properties: RBA functions as a meta-algorithm that can reuse original per-instance inference procedures and apply to structured models generally.The approach avoids requiring developers to implement a new inference algorithm and is described as applicable to any structured model.
- Algorithm Properties: The method’s optimality is guaranteed only if the relaxation algorithm converges and all constraints are satisfied.In practice, obtaining a solution satisfying every corpus-level constraint is difficult.
- Structured Output Prediction: The structured predictor scores feasible assignments by decomposing outputs into activity variables and activity-role variables.In vSRL, y_v indicates whether an activity is selected, while y_v,r indicates assignment of an activity and semantic role.
- Corpus-level Constraints: Corpus-level constraints require joint inference over all test instances rather than independent instance-level predictions.The constraints can be represented as AΣ_i y_i − b ≤ 0 over the full test corpus.
- Lagrangian Relaxation: Lagrangian relaxation introduces nonnegative multipliers for each corpus-level constraint and iteratively updates them during inference.With fixed multipliers, each iteration decomposes the joint problem into per-instance inference using the original inference algorithms.
5 Experimental Setup
The experiments evaluate bias calibration on visual semantic role labeling and multilabel object classification using gender annotations and structured models built from visual features.
- Tasks and Bias Definition: The study evaluates bias in visual semantic role labeling and multilabel object classification, focusing on gender indicators.It analyzes gender in the vSRL agent role and gender occurrences associated with images in MLC.
- vSRL Setup: The imSitu vSRL setup uses activity, role, and noun categories to represent visual situations and filters out non-human-oriented activities.The original dataset contains about 125,000 images, with training, development, and test splits; filtering leaves 212 verbs.
- vSRL Setup: The vSRL model is a structured CRF that captures correlations between activities and role nouns using visual features from a pretrained VGG network.The model builds on a released CRF baseline and learns common priors between activities and nouns in semantic roles.
- MLC Setup: The MLC setup uses MS-COCO captions to annotate gender and retains 66 object categories associated with gender mentions.Images mentioning both genders are removed, and objects occurring with either gender fewer than 100 times in training are filtered out.
- MLC Setup: The MLC model adapts a structured CRF to jointly represent object categories and person gender using ResNet-50 visual features.The output includes all object categories and the gender of a person associated with the image.
- Calibration Evaluation: Calibration matches test-set gender ratios to training-set ratios within a margin of .05 without using test-set ground truth.The procedure assumes test-set gender statistics should be similarly distributed to those in the training data.
6 Bias Analysis
Both imSitu and MS-COCO contain substantial gender bias, and training structured predictors on these datasets amplifies that bias on development data. Greater initial category bias is associated with stronger amplification.
- imSitu gender bias: 64.6% of imSitu verbs favor male agents, with average bias 0.707.46.95% of verbs favor a gender with bias of at least 0.7.
- imSitu gender bias: Shopping, microwaving, and washing favor female agents, whereas driving, shooting, and coaching strongly favor male agents.
- imSitu amplification: 0.050 mean bias amplification occurs for imSitu verbs, with 45.75% exhibiting amplification.Verbs with training bias above 0.7 in either direction have mean amplification of 0.072.
- MS-COCO gender bias: 86.6% of MS-COCO objects favor men, with average bias 0.65; 37.9% favor men with bias of at least 0.7.
- MS-COCO amplification: 0.036 mean bias amplification occurs across MS-COCO objects, with 65.67% of nouns exhibiting amplification.Objects with training bias above 0.7 have mean amplification of 0.081.
- Cross-dataset pattern: Across both datasets, models amplify existing gender bias, and categories with stronger initial bias exhibit more amplification.The analysis warns that uncalibrated systems may reinforce or worsen existing social bias.
7 Calibration Results
RBA calibrates predictions using corpus-derived constraints in imSitu vSRL and MS-COCO MLC. It reduces bias amplification and constraint violations across settings while causing negligible recognition-performance loss, though low initial bias remains challenging.
- Evaluation setup: RBA is evaluated on imSitu vSRL and MS-COCO MLC using training-derived corpus constraints applied to development or testing predictions.
- Visual Semantic Role Labeling: 52% reduction in vSRL development-set bias amplification accompanies a 30.5% decrease in verbs violating the 5% threshold.Top-1 semantic role accuracy decreases negligibly, while distance to the training distribution falls by over 39%.
- Visual Semantic Role Labeling: RBA reduces bias amplification across all initial training biases, but struggles more in regions of low initial bias.
- Visual Semantic Role Labeling: 40.5% reduction in vSRL test-set bias amplification supports the development-set result.
- Multilabel Classification: 47.5% reduction in MS-COCO test-set bias amplification follows a 40% decrease in objects violating the 5% threshold.Development-set amplification falls 31.3%, with negligible loss in top-1 mean average precision.
- Overall result: RBA makes substantial progress in reducing amplification with little or no loss in recognition performance across both problems.It reduces amplification at every initial training-bias value but does not remove all amplification.
8 Conclusion
The paper presents a framework for visualizing and quantifying bias in structured prediction and proposes RBA to reduce bias amplification. It also identifies unresolved questions about how predictors and bias measurements interact.
- RBA calibrates structured predictors under two settings and helps reduce gender bias amplified by conditional random fields.The approach targets social bias transferred from data into predictions.
- Future work should compare how different predictors amplify bias and examine interactions among predictors, bias measurements, and deamplification methods.The authors also propose applying bias-reduction methods to other structured domains.