Source-linked AI summary
Characterising Bias in Compressed Models
Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, Emily Denton
TL;DR
Compressed models are widely used under strict resource constraints, but top-line accuracy can mask concentrated errors and amplified bias on underrepresented groups. The paper compares compressed and non-compressed model populations, defines Compression Identified Exemplars (CIE) from prediction divergences, and proposes them for human-in-the-loop auditing. It finds that CIE provides a tractable, attribute-agnostic subset for inspecting challenging examples when protected-attribute labels are limited.
Problem
Minimal changes in top-line accuracy can obscure disproportionately high errors and leave unclear whether compression amplifies algorithmic bias.
Method
The paper identifies CIEs where compressed and non-compressed model populations produce divergent labels and uses them to surface examples for auditing.
Results
Compression consistently amplifies algorithmic bias while overall accuracy changes minimally, with newly introduced errors unevenly distributed across subgroups.
Takeaways & Limitations
CIE offers a tractable, attribute-agnostic human-in-the-loop auditing tool when labels for protected attributes are limited.
Takeaways & Limitations
CIE cannot be computed on training data in an overfit regime with 0% training error because there is no predictive difference.
Abstract
from arXiv · showhide
The popularity and widespread use of pruning and quantization is driven by the severe resource constraints of deploying deep neural networks to environments with strict latency, memory and energy requirements. These techniques achieve high levels of compression with negligible impact on top-line metrics (top-1 and top-5 accuracy). However, overall accuracy hides disproportionately high errors on a small subset of examples; we call this subset Compression Identified Exemplars (CIE). We further establish that for CIE examples, compression amplifies existing algorithmic bias. Pruning disproportionately impacts performance on underrepresented features, which often coincides with considerations of fairness. Given that CIE is a relatively small subset but a great contributor of error in the model, we propose its use as a human-in-the-loop auditing tool to surface a tractable subset of the dataset for further inspection or annotation by a domain expert. We provide qualitative and quantitative support that CIE surfaces the most challenging examples in the data distribution for human-in-the-loop auditing.
1 Introduction
Compression methods preserve top-line accuracy but can obscure concentrated errors and amplify bias on underrepresented or protected subgroups. The paper introduces Compression Identified Exemplars (CIE) as a tractable, label-agnostic subset for human auditing.
- Research question: Compression disproportionately affects a small subset of examples, motivating analysis of its effects on datasets with sensitive attributes.The paper asks whether compression amplifies existing algorithmic bias.
- Motivation: This issue matters in sensitive, resource-constrained domains because compression trade-offs may affect human welfare in hiring, health care, driving, and facial recognition.The paper frames fairness-related compression effects as especially urgent in these applications.
- Contribution: Compression amplifies algorithmic bias while leaving overall accuracy minimally changed, with disproportionately high errors concentrated in a small subset called Compression Identified Exemplars (CIE).The result is established across widely used pruning and quantization techniques.
- CIE: An example is a CIE when predictions from compressed and non-compressed model populations diverge.This operationalizes CIE by comparing labels produced by the two populations.
- Auditing: CIE surfaces challenging examples without protected-attribute labels, enabling a tractable human-in-the-loop audit that domain experts can inspect or annotate.The method is attribute agnostic and can support auditing multiple attributes simultaneously.
2 Characterising Compression Induced Bias in Data with Sensitive Attributes
The paper evaluates whether compression amplifies bias in CelebA models, focusing on subgroup error disparities and challenging exemplars. It compares pruning and quantization with a non-compressed baseline and finds that aggregate accuracy changes little while errors become unevenly distributed across subgroups.
- Dataset and evaluation: CelebA provides binary blonde-hair classification with protected attributes used to analyze demographic and intersectional subgroup performance.The analysis focuses on Male, Young, and their intersection, Young Male.
- Compression methods: The study evaluates magnitude pruning, fixed-point 8-bit quantization, and hybrid 8-bit dynamic-range quantization on ResNet-18 models.Quantization is post-training, whereas pruning is applied progressively during training.
- Aggregate performance: 94.73% mean top-1 test-set accuracy is achieved by the non-compressed baseline, compared with 94.07% after 90% pruning and 93.39% after 95% pruning.These results illustrate the narrow overall accuracy range across the reported pruning levels.
- Bias results: Compression consistently amplifies disparate treatment of underrepresented protected subgroups, while aggregate metrics change minimally and FNR is amplified more than FPR.The newly introduced errors are unevenly distributed across subgroups.
- Bias results: At 95% pruning, Male FPR increases 49.54% relative to baseline, versus 6.32% for not Male and 12.72% overall.The reported subgroup difference coincides with Blond not Male comprising 14% of training data versus 0.85% for Blond Male.
3 Auditing Compressed Models in Limited Annotation Regimes
The paper introduces Compression Identified Exemplars (CIEs) to surface difficult examples without protected-attribute labels, enabling human-in-the-loop auditing of compressed models. Modal CIE detects changed modal predictions, while Taxicab CIE ranks examples by prediction-distribution divergence; these subsets expose amplified errors and underrepresented attributes.
- CIE Definition: CIEs are examples where predictions diverge between independently trained compressed and non-compressed model populations.
- Divergence Measures: Modal CIE identifies examples with different modal labels, whereas Taxicab CIE scores the dataset to produce a thresholdable ranking.
- Surfacing Challenging Examples: 49.82% and 55.35% are the baseline top-1 accuracies on Modal CIE and Taxicab CIE subsets, compared with 94.76% on the entire test set.
- Amplified Sensitivity: 52.51% is the accuracy loss for a 99% pruned model on Modal CIE data, compared with a 45.86% baseline loss.
- Underrepresented Attributes: Underrepresented attributes over-index on CIE, linking the surfaced difficult examples to rare and atypical instances.
- Human-in-the-Loop Auditing: Taxicab CIE lets practitioners threshold a ranked subset for auditing, surfacing examples regardless of attribute label and supporting intersectional inspection.
4 Related Work
Prior work has largely evaluated compression through overall accuracy, while related studies have also examined robustness to adversarial attacks. This paper builds on work showing that pruning can affect a small subset of examples disproportionately.
- Compression research has primarily centered on the trade-off between compression level and overall accuracy.
- Related work has studied the sensitivity of pruned models to Lp norm adversarial attacks as a different notion of robustness.
- This paper builds on work measuring disproportionate effects of pruning on a small subset of examples.
5 Conclusion
The paper finds that compression can leave overall error largely unchanged while concentrating error in a subset of data, with fairness implications. It proposes identifying this subset through prediction divergences and using it for attribute-agnostic human-in-the-loop auditing.
- Overall error remains largely unchanged after compression, but a subset of data bears a disproportionately high portion of the error.
- Compression can create fairness issues, illustrated by its impact on CelebA.
- The high-error subset can be isolated by identifying examples where dense and compressed model populations produce divergent labels.
- The paper proposes CIE as an attribute-agnostic human-in-the-loop auditing tool.
A.1 Equivalence of Taxicab CIE and Jaccard CIE
The paper establishes that Jaccard CIE and Taxicab CIE rank examples identically. The equivalence follows from comparing label distributions through absolute differences and applying the stated algebraic identities.
- Jaccard CIE and Taxicab CIE select examples in the same order.
- For each example, baseline and variant model populations are represented as distributions of class-label counts.
- Taxicab and weighted Jaccard distances compare these two label distributions, with Jaccard distance accounting for label multiplicity.
- The proof assumes both model families contain N models and uses label pairing, absolute-difference identities, and a positive-real-number comparison.
A.2 Absolute Performance Metrics Disaggregated
Table 5 reports absolute performance for every subgroup and subgroup intersection considered in the analysis.
- Table 5 includes absolute performance measurements for the subgroups considered.
- The table covers every subgroup included in the analysis.
- The table also includes each considered intersection of subgroups.