Source-linked AI summary
Measuring Model Biases in the Absence of Ground Truth
Osman Aka, Ken Burke, Alex Bäuerle, Christina Greer, Margaret Mitchell
TL;DR
Existing fairness metrics often require extensive ground-truth annotations and do not directly measure associations a model has learned between labels and identity groups. The paper instead analyzes co-occurrences among model predictions, comparing association metrics to rank identity-linked biases. It concludes that normalized pointwise mutual information is particularly useful in large-label-space settings and provides a TensorBoard visualization tool.
Problem
Ground-truth-dependent fairness metrics are difficult to use for large computer-vision label spaces, where comprehensive annotations may be unavailable.
Method
The paper estimates label and joint-label probabilities from model predictions and applies association metrics to compare labels against sensitive identity labels.
Results
Normalized pointwise mutual information is preferable to other commonly used association metrics for detecting biases without ground-truth labels in large-label-space settings.
Takeaways & Limitations
nPMI can surface associations across rare and common labels and support per-label exploration of biases learned by a model.
Takeaways & Limitations
The analysis treats predicted labels as a flat set, so it does not capture richer within-image structure and relationships between objects.
Abstract
from arXiv · showhide
The measurement of bias in machine learning often focuses on model performance across identity subgroups (such as man and woman) with respect to groundtruth labels. However, these methods do not directly measure the associations that a model may have learned, for example between labels and identity subgroups. Further, measuring a model's bias requires a fully annotated evaluation dataset which may not be easily available in practice. We present an elegant mathematical solution that tackles both issues simultaneously, using image classification as a working example. By treating a classification model's predictions for a given image as a set of labels analogous to a bag of words, we rank the biases that a model has learned with respect to different identity labels. We use (man, woman) as a concrete example of an identity label set (although this set need not be binary), and present rankings for the labels that are most biased towards one identity or the other. We demonstrate how the statistical properties of different association metrics can lead to different rankings of the most "gender biased" labels, and conclude that normalized pointwise mutual information (nPMI) is most useful in practice. Finally, we announce an open-sourced nPMI visualization tool using TensorBoard.
1 INTRODUCTION
The paper addresses model bias measurement when ground-truth annotations are sparse by analyzing associations among model predictions and identity labels directly. It argues that nPMI is especially useful for revealing learned, stereotype-aligned associations without requiring ground truth.
- Motivation: Ground-truth-dependent fairness metrics become difficult to apply as vision label spaces expand from 20 categories to more than 20,000.Large label spaces increase annotation costs, while datasets assembled from online resources can contain systematically omitted labels.
- Contribution: The proposed approach measures problematic associations between model predictions directly, enabling bias analysis after ordinary inference without unbiased ground-truth labels.It is intended to identify biases learned by the model and analyze associations between predictions and identity attributes.
- Contribution: The authors identify normalized pointwise mutual information (nPMI) as particularly useful for detecting stereotype-aligned associations learned by models.They also release an open-sourced TensorBoard tool for exploring label-bias patterns with nPMI.
- Contribution: The framework ranks labels by their associations with identity labels and can compare biases for both low- and high-frequency labels.The paper treats predicted labels as an unordered set and studies association metrics grounded in information theory and related work in NLP.
2 RELATED WORK
The paper builds on pointwise mutual-information methods for measuring word associations and adapts their unordered co-occurrence view to model-predicted image labels. It positions this analysis as complementary to fairness metrics and notes that richer contextual structure is not captured by simple paired counts.
- Association metrics: Pointwise mutual information measures how closely labels are related by comparing their co-occurrence rates with chance in the evaluation dataset.The approach adapts Church and Hanks’ word-association method to labels predicted by computer-vision models.
- Association metrics: Without syntactic preprocessing, the bag-of-words perspective treats tokens as an unordered set, motivating a comparable treatment of image-label co-occurrences.The paper applies this simplified association view to identify problematic associations learned by machine-learning models.
- Limitations of prior approaches: Simple paired-word counts do not capture how labels are distributed with other words or contextual structure, leaving richer relationships for later analysis.The paper identifies this limitation when discussing sentence syntax and future incorporation of contextual information.
- Relation to fairness: The proposed association analysis complements fairness work by targeting specific learned associations and avoiding reliance on comprehensive ground-truth labeling.Unlike overall fairness criteria, it focuses on quantifying particular label–identity associations in large datasets.
3 PROBLEM DEFINITION
The paper defines bias as differences in associations between model-predicted labels and identity labels, enabling label ranking without comprehensive ground-truth annotations. It compares multiple association metrics and normalization strategies to support scalable analysis across large, unevenly frequent label spaces.
- 3 PROBLEM DEFINITION: The approach estimates label and label-pair probabilities from classifier predictions, requiring no ground-truth annotations for the analyzed labels.For each image, the classifier predicts whether each label is relevant; P(yi) and P(yi,yj) are inferred from the resulting dataset.
- 3 PROBLEM DEFINITION: The gender example uses man and woman for simplicity, but the framework extends to more than two identity labels through pairwise or one-versus-all comparisons.The authors caution that this paired example should not be interpreted as a one-dimensional account of gender representation.
- 3 PROBLEM DEFINITION: The method is designed for large label spaces, where ground-truth-dependent fairness metrics become costly and label frequencies vary widely.The formulation aims to compare labels such as common and rare concepts without manually bucketizing frequencies or setting separate thresholds.
- 3 PROBLEM DEFINITION: Association gaps compare how strongly each label y is associated with two identity labels, measuring its skew toward one subgroup.The gap is defined as G(y|x1,x2,A(·)) = A(x1,y) − A(x2,y), and labels can then be ranked by gap magnitude.
- 3.1 Association Metrics: The paper studies Demographic Parity, PMI and nPMI, overlap metrics, correlation, and statistical tests as alternative association measures.These metrics differ in how they quantify associations and in their sensitivity to marginal label frequency.
- 3.1 Association Metrics: PMI-based association gaps measure dependence between an identity-label distribution and another label distribution, while nPMI variants normalize frequency effects to enable cross-label comparisons.The paper considers normalization by P(y), by joint probabilities, and squared joint probabilities; it evaluates their usefulness for comparing differently frequent labels.
4 EXPERIMENTS
Experiments on Open Images show that association-metric choice substantially changes which labels rank as most biased, partly because metrics respond differently to label frequency. nPMI variants surface both rare and common labels, whereas PMI and tau_b favor different frequency ranges.
- 4.1 Label Ranks: PMI, PMI2, and LLR shared all top 100 labels, with over 99% of label pairs preserving relative rankings; DP, JI, and SDC shared about 70%.PMI was selected as representative of the first cluster, and DP as representative of the second because of mathematical simplicity and prominence in fairness literature.
- 4.1 Label Ranks: Increasing a label’s count while holding its identity-association ratio constant moved its rank roughly as predicted by metric-specific partial derivatives.Metrics with larger partial derivatives for x1, x2, or y generally produced larger rank changes; nPMI_y was more affected than PMI in the illustrated co-occurrence setting.
- 4.2 Top 100 Labels by Metric Gaps: The top-100 count ranges differed by orders of magnitude across metrics, with nPMI_y and nPMI_xy spanning broader marginal and joint-frequency ranges than the other clusters.The broader normalized-PMI ranges included the ranges of other metrics, especially for joint co-occurrences C(x1,y) and C(x2,y).
- 4.2 Top 100 Labels by Metric Gaps: PMI gaps ranked mostly rare labels, tau_b gaps mostly common labels, and nPMI_xy gaps both rare and common labels among the top 100.The corresponding distributions were shown for PMI, nPMI_xy, and tau_b in Figure 2.
- 4.2 Top 100 Labels by Metric Gaps: In Table 3, DP highlighted labels predicted for over 100,000 images, PMI labels predicted for fewer than 1,000 images, and nPMI_xy included both frequency ranges.Examples included Happiness and Fashion for DP, Treggings and Boho-chic for PMI, and Long Hair alongside Boho-chic for nPMI_xy.
5 DISCUSSION
The discussion argues that normalization is essential for bias rankings across labels with different frequencies, and presents nPMI as a practical choice alongside an open-source visualization tool.
- 5 DISCUSSION: nPMI is preferable because it surfaces both common and rare labels, whereas unnormalized PMI can over-rank rare labels through unstable ratios.Labels with small counts can obtain very large joint-probability ratios from a single example; nPMI reduces this noise while retaining common labels.
- 5 DISCUSSION: Common labels remain important audit targets, so normalization should balance joint-probability ratios against marginal label rarity.This assumes the evaluation set matches the classification model’s real-world use-case distribution.
- 5 DISCUSSION: Frequency bucketing is less elegant than metric normalization because it requires manually tuned thresholds for each bucket.The authors describe bucketing as discontinuous and burdensome when fairness constraints depend on thresholding.
- 5 DISCUSSION: The open-source TensorBoard tool filters annotations, displays association distributions and counts, and compares selected annotations across models or datasets.Its parallel-coordinates view helps investigate where association differences arise.
6 CONCLUSION AND FUTURE WORK
The paper concludes that association metrics can measure label-specific biases without ground-truth annotations in large computer-vision label spaces. It identifies nPMI as particularly useful while leaving smaller label spaces, exhaustive-ground-truth comparisons, and richer image structure for future work.
- 6 CONCLUSION AND FUTURE WORK: nPMI is particularly useful for measuring specific biases in real-world datasets with large label spaces such as Open Images.The conclusion positions nPMI among association metrics that measure skews toward specific labels without ground truth.
- 6 CONCLUSION AND FUTURE WORK: Association metrics measure biases toward specific labels without requiring ground-truth annotations, enabling use where standard fairness metrics are difficult to apply.The approach is framed for current computer-vision classifiers with large label spaces.
- 6 CONCLUSION AND FUTURE WORK: Future work will test nPMI in small label spaces, compare it with ground-truth-dependent fairness metrics, and model within-image label relationships beyond flat co-occurrence.The current treatment represents labels predicted for an image as a flat set.
APPENDIX
The appendix defines a family of association and fairness metrics for measuring label-association gaps between sensitive labels, including probability, entropy, overlap, correlation, and likelihood-based measures.
- APPENDIX: The appendix defines gap calculations for sensitive labels x1 and x2 and derives metric orientations from marginal and joint label probabilities.The orientations describe theoretical sensitivity to different probability terms.
- APPENDIX: The compared metric families include Demographic Parity, PMI variants, Sørensen-Dice, Jaccard, Kendall rank correlation, Log-Likelihood Ratio, and t-test.These metrics span fairness, entropy, intersection-over-union, correlation, and statistical-test approaches.
- APPENDIX: The appendix includes normalized PMI variants using p(y) and p(x,y) normalization, along with squared PMI.These variants modify how association gaps respond to label frequencies and joint probabilities.
- APPENDIX: Kendall’s τ_b gap is defined as the difference between its association with x1 and with x2 for target label y.The appendix also introduces concordant, discordant, tied-pair, condition-count, and dataset-size notation.
C METRIC ORIENTATIONS
This section derives metric orientations, interpreting them as sensitivities to label probabilities, and summarizes the resulting formulas for overlap, PMI, and rank-correlation metrics.
- C METRIC ORIENTATIONS: Metric orientations represent each metric’s sensitivity to the relevant marginal or joint label probabilities.Table 4 summarizes these orientations before the longer derivations.
- C METRIC ORIENTATIONS: The Sørensen-Dice orientation is expressed through the joint probabilities and denominators involving p(x1), p(x2), and p(y).The derivative with respect to p(y) is given explicitly.
- C METRIC ORIENTATIONS: The Jaccard Index orientation is derived with respect to p(y), p(x1,y), and p(x2,y).The displayed derivatives specify how the gap changes with each probability term.
- C METRIC ORIENTATIONS: The squared PMI gap combines conditional-probability and joint-probability log differences for the two sensitive labels.Its expression is included among the metric-orientation derivations.
- C METRIC ORIENTATIONS: The section gives orientation expressions for nPMI normalized by p(y) and by p(x,y), as well as the Kendall rank-correlation gap.These formulas expose how the normalized association metrics and τ_b respond to probability structure.
D COMPARISON TABLES
This section presents comparison tables for association gaps, including mean and standard deviation summaries and minimum–maximum ranges.
- Mean and standard deviation are reported for the top 100 male association gaps.
- Mean and standard deviation are also reported for the top 100 male-female association gaps.
- Minimum and maximum values are reported for the top 100 male-female association gaps.
E.1 Overall Rank Changes
This section compares demographic parity and several association measures, including PMI, normalized PMI, Kendall's tau-b, squared PMI, and log-likelihood ratio.
- E.1 Overall Rank Changes: Additional comparisons examine PMI, squared PMI, and the log-likelihood ratio.
- E.1 Overall Rank Changes: The tables compare demographic parity, PMI, normalized PMI, and Kendall's tau-b.
E.2 Movement plots
This section provides comparison tables and example top results for alternative association metrics, including overlap-based measures, PMI variants, and normalized PMI.
- E.2 Movement plots: One comparison examines demographic parity, the Jaccard index, and the Sørensen-Dice coefficient.
- E.2 Movement plots: Another comparison examines PMI, squared PMI, and the log-likelihood ratio.
- E.2 Movement plots: A further table compares demographic parity, PMI, normalized PMI, and Kendall's tau-b.
- E.2 Movement plots: Example top results are presented in a separate table.