Source-linked AI summary
Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, Lucy Vasserman
TL;DR
The paper addresses limited and potentially misleading views of unintended identity-related bias in machine-learning classification. It proposes threshold-agnostic, multi-metric evaluation alongside a large real-world comment dataset, finding that these tools reveal bias and further challenges for mitigation in public toxicity models.
Problem
Unintended identity-related bias can produce differing model performance across groups, while threshold choices and single metrics may obscure this behavior.
Method
The paper introduces five metrics based on ROC-AUC, Equality Gap, and Mann-Whitney U, and evaluates them with synthetic data and nearly 2 million human-labeled comments.
Results
Evaluation of two public toxicity models shows that the metrics and real-data dataset illuminate unintended bias and provide new insight into bias mitigation effectiveness.
Takeaways & Limitations
Real-data evaluation with a suite of metrics can reveal new challenges for bias mitigation, including bias remaining in models that underwent mitigation.
Takeaways & Limitations
Metric interpretation remains case dependent: some highlighted variations may not constitute problematic bias, and subtle bias may escape detection.
Abstract
from arXiv · showhide
Unintended bias in Machine Learning can manifest as systemic differences in performance for different demographic groups, potentially compounding existing challenges to fairness in society at large. In this paper, we introduce a suite of threshold-agnostic metrics that provide a nuanced view of this unintended bias, by considering the various ways that a classifier's score distribution can vary across designated groups. We also introduce a large new test set of online comments with crowd-sourced annotations for identity references. We use this to show how our metrics can be used to find new and potentially subtle unintended bias in existing public models.
1 INTRODUCTION
The paper addresses unintended identity-related bias in toxicity classification by proposing threshold-agnostic metrics, a large real-data test set, and evaluations of public toxicity models.
- Toxicity classification is subjective, and models can learn incorrect associations that reproduce societal biases involving identity terms.Potential sources include training-data composition, annotator biases, and item-selection procedures.
- The paper measures unfairness as performance variation across designated groups, focusing on score and output-label skew caused by identity-related text.The definition assumes a test set with reliable labels across groups.
- Threshold-agnostic metrics avoid dependence on a single decision threshold and instead capture behavior in models that return scores.Threshold-dependent metrics can change substantially with threshold selection and may obscure unintended bias.
- The proposed suite contains five metrics derived from ROC-AUC, Equality Gap, and Mann-Whitney U measures, each targeting different bias patterns.The authors argue that examining multiple metrics across groups reveals more nuance than relying on one metric.
- The evaluation uses a synthetic test set and a new human-labeled dataset of nearly 2 million comments, including 450,000 identity annotations.The datasets are applied to two publicly accessible Perspective API toxicity models, one reportedly trained with bias mitigation.
- The metrics and datasets expose unintended bias in the original model and provide new insight into the effectiveness of bias mitigation between the evaluated models.
2 RELATED WORK
Prior approaches often evaluate bias at a selected threshold or with a single metric. The paper’s metrics are designed to remain threshold agnostic, handle class imbalance, and distinguish different bias patterns.
- Prior fairness work introduced metrics and mitigation techniques, while transparency efforts emphasized choosing appropriate data sources, methods, and evaluation criteria.
- Equality Gap measures are limited to binary-label classifiers and evaluate performance at one threshold, despite many models producing reusable probability scores.
- Counterfactual-fairness approaches likewise measure Equality Gap at a threshold selected to maximize accuracy on a designated test set.
- A prior threshold-agnostic metric is not robust to class-distribution differences across identity groups, and its single score can conceal offsetting bias types.
- The proposed metrics are threshold agnostic, robust to class imbalances, and intended to provide more nuanced insight into bias types.
3 NEW METRICS
The paper introduces threshold-agnostic metrics that compare identity subgroups with background data to diagnose distinct score-distribution biases. AUC-based metrics detect mis-orderings affecting common thresholds, while AEGs additionally capture distributional shifts that may not cause mis-orderings.
- AUC-Based Metrics: AUC measures the probability that a randomly chosen negative receives a lower score than a randomly chosen positive, without fixing a threshold.An AUC of 1.0 indicates perfect ordering and separability of negative and positive examples.
- AUC-Based Metrics: The metrics divide data into background and identity-subgroup positives and negatives, then compare these four subsets rather than evaluating each subgroup in isolation.The four subsets are background negatives, background positives, subgroup negatives, and subgroup positives.
- AUC-Based Metrics: Subgroup AUC measures separability within the subgroup, whereas BPSN and BNSP AUC measure cross-group mis-orderings associated with false positives and false negatives.BPSN uses background positives and subgroup negatives; BNSP uses background negatives and subgroup positives.
- Average Equality Gap: The AEG metrics measure the area between subgroup and background true-rate curves across thresholds, with analogous positive and negative versions.Positive AEG uses true positive rates, while Negative AEG uses true negative rates; together they form the Average Equality Gap metrics.
- Comparison of Metrics: The metrics remain informative under subgroup class imbalance, detecting the same score shift when subgroup data contains many more positive than negative examples.The cited scenario reports that BPSN, BNSP, and AEG metrics detect the shift despite the imbalance.
- Comparison of Metrics: The metric suite distinguishes bias patterns: AUCs identify threshold-relevant mis-orderings, while AEGs can reveal score-distribution variation that is not severe enough to cause mis-orderings.BPSN and BNSP AUCs and both AEGs also provide insight into score-shift directionality.
4 EXPERIMENTAL RESULTS
The paper demonstrates its metric suite on synthetic and real-data test sets using two publicly accessible Perspective API toxicity classifiers.
- 4 EXPERIMENTAL RESULTS: The evaluation uses a synthetic bias-focused test set and a large human-labeled dataset of online comments.The real dataset contains labels for identity and toxicity.
- 4 EXPERIMENTAL RESULTS: The experiments use the metrics and datasets to illuminate unintended bias and assess the effectiveness of mitigation between the two models.The evaluation focuses on publicly accessible toxicity models.
- 4 EXPERIMENTAL RESULTS: The two evaluated models are Perspective API’s initial TOXICITY@1 and latest TOXICITY@6 classifiers.TOXICITY@6 was built using a previously proposed bias-mitigation technique.
- 4 EXPERIMENTAL RESULTS: The synthetic dataset contains 77k template-generated examples spanning 50 identity terms with balanced toxic and non-toxic labels.Its simple examples are designed to isolate identity-term bias.
4.3 Synthetic Test Set Results
On the synthetic test set, TOXICITY@1 shows identity-related false-positive bias, while TOXICITY@6 reduces but does not eliminate it; AEG exposes additional score shifts.
- 4.3 Synthetic Test Set Results: Low BPSN AUC for homosexual, gay, lesbian, transgender, and heterosexual indicates elevated toxicity scores for non-toxic examples containing these identities.The strongest pattern appears for homosexual, gay, and lesbian.
- 4.3 Synthetic Test Set Results: TOXICITY@6 reduces the BPSN AUC bias observed in TOXICITY@1, but the unintended bias is not completely eliminated.The comparison concerns identity terms associated with toxicity misclassification.
- 4.3 Synthetic Test Set Results: High Subgroup AUC values show that both models generally distinguish toxic from non-toxic examples within each identity group.This remains true even for groups with incorrect toxicity tendencies in BPSN AUC.
- 4.3 Synthetic Test Set Results: High BNSP AUC values indicate that mis-orderings causing false negatives for identity-referencing toxic comments are rare in both models.The metric captures ordering errors between subgroup toxic examples and background non-toxic examples.
- 4.3 Synthetic Test Set Results: AEG reveals score shifts that AUCs do not fully capture, including residual upward shifts for bisexual, trans, queer, and black identities.TOXICITY@6 improves BPSN AUC more than Negative AEG, indicating reduced but persistent shifts.
Score distributions vary widely across groups.
The paper combines synthetic and human-labeled resources for bias evaluation, while noting that synthetic coverage and the identity-label set have important boundaries.
- Score distributions vary widely across groups.: Synthetic test sets can expose issues absent from real data but may not accurately represent scenarios with different data distributions.They are also limited to manually curated identity terms.
- Score distributions vary widely across groups.: Human raters label identity references by selecting applicable identities from provided lists for questions about gender, race, ethnicity, and other categories.This approach captures nuanced references that term-based analysis may miss.
- Score distributions vary widely across groups.: The identity labels are not comprehensive or universally covering, reflecting a balance among identity coverage, rater accuracy, and sufficient examples.The design prioritizes usable representation rather than exhaustive identity coverage.
- Score distributions vary widely across groups.: The human-labeled dataset contains 1.8 million online comments with toxicity labels, including 450,000 comments additionally labeled for identity.Some identity-labeled comments were preselected to increase the frequency of identity content shown to raters.
- Score distributions vary widely across groups.: Toxicity percentages vary across identities in the human-labeled dataset, demonstrating the value of metrics robust to data skews.The paper presents this variation for a selection of identities.
4.5 Human Labeled Dataset Results
On real comments, model performance is lower and more variable across identity subgroups than on synthetic data, although the weakest groups broadly align across datasets.
- 4.5 Human Labeled Dataset Results: The real-data evaluation compares TOXICITY@1 and TOXICITY@6 across all metrics for short comments and for comments of all lengths.Short comments are separated because mitigation focused on them and results vary by comment length.
- 4.5 Human Labeled Dataset Results: Identity terms with the lowest performance on synthetic data align with identity groups showing the lowest performance on real data.This alignment occurs despite the broader and potentially noisier real-data setting.
- 4.5 Human Labeled Dataset Results: Synthetic data reveals large discrepancies tied narrowly to specific identity terms, whereas real data provides broader and more nuanced results.The paper characterizes real-data evaluation as potentially noisier because of its broader setting.
Real data reveals more unintended bias than synthetic data.
Across both toxicity models and both short and long comments, the metrics indicate that identity-related content can skew non-toxic comments toward toxicity.
- Both models show lower Subgroup AUC and BPSN AUC, but higher BNSP AUC, across short and long comments.
- Positive Negative AEG values and negative Positive AEG values accompany this pattern across both models and comment lengths.
Bias tends to skew towards toxicity.
The evaluated models tend to skew identity-related, non-toxic comments toward toxicity, with bias levels varying across groups.
- Bias levels are high for some groups, but all metrics remain important because bias can appear in unexpected places.
- TOXICITY@6 outperforms TOXICITY@1 on AUCs and AEGs for most identities and metrics on short comments.
Short comments show expected bias mitigation.
Short comments show stronger unintended bias and greater improvement from the mitigation between TOXICITY@1 and TOXICITY@6 than all comments overall.
- For all comments, TOXICITY@1 and TOXICITY@6 change much less, raising the possibility that mitigation did not generalize to longer comments.
- Short comments in TOXICITY@1 have lower AUC values, especially Subgroup AUC and BPSN AUC, and greater metric variation across groups.
- For homosexual_gay_or_lesbian, TOXICITY@6 shifts the non-toxic score distribution left on short comments, indicating improvement.
- The metrics and distributions show that mitigation affected short-comment bias as designed, although room for improvement remains.
5 CONCLUSION AND FURTHER WORK
The paper introduces ROC-AUC- and Mann-Whitney-U-based metrics and a large human-labeled dataset to study unintended bias in real text-classification data. Its evaluation reveals persistent bias-mitigation challenges and identifies several directions for further work.
- Conclusion: The new metric suite is based on ROC-AUC and Mann-Whitney U scores and is intended to provide a detailed view of bias types.
- Conclusion: The human-labeled dataset contains nearly 2 million comment annotations and extends unintended-bias evaluation beyond synthetic test sets.
- Conclusion: Evaluation on real data shows that bias remains in models that have undergone some mitigation.
- Further work: Future work includes strategies for choosing thresholds that minimize unintended bias.
- Further work: Future work also includes comparing the new dataset with substring matching and systematically defining synthetic distributions for metric evaluation.
- Further work: A full taxonomy of possible biases and a systematic diagnostic approach for the metrics remain future work.