Source-linked AI summary
Hurtful Words: Quantifying Biases in Clinical Contextual Word Embeddings
Haoran Zhang, Amy X. Lu, Mohamed Abdalla, Matthew McDermott, Marzyeh Ghassemi
TL;DR
The paper asks how clinical contextual embeddings encode protected-group differences and whether those differences appear in clinical prediction. It pretrains BERT on MIMIC-III notes, measures fill-in-the-blank and downstream fairness disparities, and examines adversarial debiasing. The authors find significant group differences across embedding behavior and clinical-task performance, while debiasing leaves important gaps unresolved.
Problem
The paper investigates whether clinical contextual embeddings encode marginalized populations differently and whether this is associated with disparities in clinical prediction.
Method
The authors pretrain BERT on MIMIC-III medical notes, use fill-in-the-blank log probability analyses, evaluate 57 binary clinical tasks, and test adversarial debiasing.
Results
The baseline clinical BERT shows significant performance differences across protected groups, and adversarial debiasing does not greatly reduce the number of significant gaps.
Takeaways & Limitations
Clinical contextual embeddings should be examined for unwanted latent relationships and group performance disparities before use in high-stakes healthcare settings.
Takeaways & Limitations
The study treats post-stay ICD-9 billing labels as gold-standard outcomes, does not analyze intersectional discrimination, and does not disentangle bias sources.
Abstract
from arXiv · showhide
In this work, we examine the extent to which embeddings may encode marginalized populations differently, and how this may lead to a perpetuation of biases and worsened performance on clinical tasks. We pretrain deep embedding models (BERT) on medical notes from the MIMIC-III hospital dataset, and quantify potential disparities using two approaches. First, we identify dangerous latent relationships that are captured by the contextual word embeddings using a fill-in-the-blank method with text from real clinical notes and a log probability bias score quantification. Second, we evaluate performance gaps across different definitions of fairness on over 50 downstream clinical prediction tasks that include detection of acute and chronic conditions. We find that classifiers trained from BERT representations exhibit statistically significant differences in performance, often favoring the majority group with regards to gender, language, ethnicity, and insurance status. Finally, we explore shortcomings of using adversarial debiasing to obfuscate subgroup information in contextual word embeddings, and recommend best practices for such deep embedding models in clinical settings.
1 INTRODUCTION
The paper investigates how clinical contextual embeddings encode protected-group differences and whether those differences produce disparities in clinical prediction. It combines fill-in-the-blank bias analysis with evaluation across 57 downstream tasks and examines adversarial debiasing.
- Motivation: SciBERT can generate different clinical courses of action when the patient race is changed in an otherwise similar prompt.The templates are adapted from real MIMIC-III clinical notes.
- Motivation: Clinical embedding models may capture relationships reflecting gender and ethnic biases present in clinical text.The paper frames these relationships as a concern because healthcare algorithms should not exacerbate existing subgroup treatment disparities.
- Approach: The study evaluates fairness across gender, language spoken, ethnicity, and insurance type using log probability bias scores and downstream predictive performance.The downstream evaluation covers 57 binary clinical prediction tasks.
- Findings: Statistically significant gaps in parity, recall, and specificity occur across a majority of downstream tasks and all protected attributes.The reported gaps concern the baseline clinical BERT model.
- Findings: Adversarial pretraining debiasing does not greatly reduce the number of statistically significant performance gaps.The authors report this result across the evaluated protected-group settings.
2 BACKGROUND AND RELATED WORK
The background distinguishes contextual from non-contextual embeddings and describes how pretraining can encode bias from source text. The paper addresses a gap in debiasing research by studying clinical contextual embeddings and releasing a tailored model.
- Embedding Models: Non-contextual embeddings keep a word representation fixed across contexts, whereas contextual embeddings change it according to surrounding text.Word2Vec and GloVe are examples of non-contextual models; ELMo and BERT are contextual models.
- Embedding Models: Pretraining on domain-specific clinical text can hard-code biases present in that text into contextual embedding models.These models are commonly pretrained through self-supervised tasks before downstream fine-tuning.
- Related Work: Prior work has documented societal biases in embeddings, including gender, race, and ethnicity, but detecting them reliably is not straightforward.Simple tests may conceal bias when they are not robustly designed.
- Related Work: Debiasing research has largely focused on non-contextual embeddings, despite studies showing that contextual embeddings also capture biases.The paper positions its clinical contextual-embedding analysis within this gap.
- Paper Positioning: The authors train a clinical BERT model from SciBERT, improve PHI handling and masking, and publicly release the model and code.They cite SciBERT initialization, whole-word masking, and improved PHI encoding as motivations for their model design.
3 DATA AND PRE-TRAINING
The study uses MIMIC-III clinical notes to pretrain a SciBERT-initialized BERT model and evaluates it on a broad set of patient-level clinical classification tasks. It also constructs note templates for fill-in-the-blank bias analysis.
- Data: MIMIC-III contains electronic health records for 38,597 adult patients and 7,870 neonates, including approximately 2 million clinical notes.The records come from intensive-care admissions at Beth Israel Deaconess Medical Center between 2001 and 2012.
- Downstream Tasks: The evaluation includes in-hospital mortality and 28-task phenotyping settings based on ICD-9-linked HCUP CCS groups.The phenotyping settings use either all notes or the first note from the initial 48-hour ICU period.
- Pre-training: The model is initialized from SciBERT and trained on MIMIC-III notes after PHI replacement, outpatient-note removal, sentence aggregation, and SciBERT tokenization.Short sentences are aggregated into sequences of at least 20 tokens for pretraining.
- Pre-training: Pretraining uses one epoch on length-128 sequences followed by one epoch on length-512 sequences.The two stages use batch sizes of 32 and 16, respectively.
- Downstream Tasks: Long notes are split into up to 10 subsequences of 512 tokens, and subsequence probabilities are merged into note-level predictions using maximum and mean outputs with a task-specific scaling factor.The scaling factor is tuned separately on the validation set.
- Bias Evaluation: Note templates compare the likelihood of predicting gender pronouns in fill-in-the-blank tasks across chronic illness, mental health and addiction, sexually transmitted disease, and end-of-life topics.The templates are based on clinical notes and target contexts where biased treatment differences have been studied.
4 METHODS
The study evaluates clinical BERT representations using fairness gaps, prior-adjusted gender likelihoods, and adversarial pretraining debiasing. It preserves pretrained representations during downstream classification to assess their encoded bias.
- Clinical BERT and downstream tasks: The authors train a baseline clinical BERT on MIMIC-III notes and evaluate 57 downstream clinical prediction tasks.The model is initialized from SciBERT and trained using MIMIC NoteEvents data.
- Clinical BERT and downstream tasks: The downstream analysis freezes BERT weights and concatenates the last four [CLS] hidden-layer vectors before classifier training.This design aims to evaluate bias in the pretrained representations rather than alter them during fine-tuning.
- Fairness evaluation: Downstream classifiers are assessed with demographic parity, equality of opportunity for positive predictions, and equality of opportunity for negative predictions.For multiple protected groups, the reported gap is the maximum difference between a group and all other groups.
- Fairness evaluation: Recall gaps are prioritized because diagnostic applications favor fewer false negatives, whereas specificity gaps can be masked by negative-class imbalance.A classifier that always predicts the majority negative class can achieve 100% true negative rate and zero specificity gap.
- Log probability bias scores: Prior-adjusted log probability bias scores compare gendered-word likelihoods in fill-in-the-blank clinical templates across medical topics.Wilcoxon signed-rank tests assess whether male and female categories differ significantly, while disease prevalence is examined as context for interpretation.
- Adversarial debiasing: Adversarial pretraining uses a discriminator to make embeddings minimally predictive of protected subgroup information without using downstream labels.The adversarial network’s gradients are reversed during training and combined with loss functions to form BERT proxy tasks.
5 RESULTS
Baseline clinical BERT encodes gender-associated likelihood differences and produces statistically significant performance gaps across protected groups. Further clinical-text pretraining increases gender disparity in likelihood scores, while recall gaps generally favor majority groups.
- Overall findings: The baseline clinical BERT model is biased according to both log probability bias scores and downstream performance discrepancies.The study evaluates these two forms of bias across contextual clinical representations and prediction tasks.
- Further pretraining on clinical text: 7/8 medical context categories show significant gender differences for baseline clinical BERT, compared with 2/8 for the original SciBERT model.Further pretraining shifts predictions toward male-gendered pronouns, aligning with male-majority representation in the clinical notes.
- Performance gaps: 13/57 gender tasks and 7/57 language tasks have significant recall gaps at varying levels of disparity.The figures visualize parity, recall, and specificity gaps, while highlighting tasks with significant recall differences.
- Performance gaps: Recall gaps favor the majority group for every protected-class analysis except language.Gender gaps favor Male in eight of 13 significant tasks; ethnicity gaps often favor White and Asian patients, while insurance gaps favor Medicare over Medicaid and private insurance.
- Performance gaps: Many ethnicity- and insurance-related gaps remain after Benjamini-Hochberg false-discovery-rate correction.This indicates that those disparities persist under the reported multiple-testing adjustment.
Is Insufficient
Debiased clinical BERT retains strong downstream performance, but adversarial debiasing is insufficient to eliminate clinically relevant disparities or prevent post-hoc recovery of sensitive information.
- AUPRC correlation remains 0.996 between baseline and debiased clinical BERT across downstream classifier performance.The debiased model retains strong overall downstream performance relative to the baseline.
- Slightly fewer tasks show significant recall gaps after debiasing, but the reduction is insufficient for high-stakes medical deployment.Debiasing also does not appear to reduce the specificity gap.
- Post-hoc classifiers can still extract sensitive-attribute information after adversarial training, despite the adversary fooling its discriminator.
6 DISCUSSION
Clinical BERT captures gender-associated patterns from notes and exhibits subgroup performance gaps across clinical tasks. These disparities may reflect both unwanted bias and genuine medical confounding, complicating interpretation and mitigation.
- 6.1 Further Pretraining Captures Note Biases: Clinical BERT is significantly more likely to predict male pronouns, even for hypertension where real-world prevalence is roughly equal across men and women.Hypertension-related words occur more often in male discharge notes, linking note frequency with the model’s gendered predictions.
- 6.2 Fairness Gaps Along Known Biases: Over half of the Medicare-versus-Other task comparisons show a significant performance difference.The authors caution that not every performance gap directly corresponds to real-world treatment or outcome disparities.
- 6.2 Fairness Gaps Along Known Biases: 13/57 gender-task comparisons and 7/57 spoken-language-task comparisons have significant recall gaps.The gender recall gap generally favors men over women, while the language recall gap generally favors non-English speakers over English speakers.
- 6.2 Fairness Gaps Along Known Biases: Baseline clinical BERT classifiers show statistically significant parity, recall, and specificity gaps across gender, language, ethnicity, and insurance status.Table 4 counts significant differences across 57 downstream classification tasks and reports which subgroup each gap favors.
- 6.3 Challenges in Using Adversarial Debiasing for Clinical Contextual Embedding Models: Adversarial debiasing may be underpowered for high-capacity BERT and may encode demographic parity, which is problematic in healthcare.Applying debiasing only to [CLS] would theoretically address classification but not sequence-based tasks such as NER or question answering.
7 LIMITATIONS AND FUTURE WORK
The study’s evaluation relies on post-stay ICD-9 billing labels and does not address intersectional discrimination or disentangle the sources of observed bias.
- Downstream tasks use post-stay ICD-9 labels as the gold standard without accounting for possible label errors or biases.
- The study does not evaluate intersectional discrimination, such as combined gender and race effects.
- The study does not disentangle bias arising from data imbalance, data quality, or social biases in healthcare.
- The authors do not directly debias the data and note that debiasing methods may hide rather than remove bias.
8 CONCLUSION
The paper shows that clinical BERT embeddings can encode unwanted subgroup relationships and produce statistically significant performance gaps across protected groups. It argues for specialized methods to detect and reduce these risks in healthcare machine learning.
- Conclusion: Clinical BERT embeddings propagate unwanted latent relationships involving gender, language speakers, ethnicity, and insurance groups.The analysis combines quantitative and qualitative methods, including gender-pronoun completion probabilities and race-conditioned medical text completion.
- Conclusion: Adversarial pretraining debiasing does not guarantee removal of group disparities during post-hoc classification.The paper reports that disparities can remain in embeddings labeled “debiased.”
- Conclusion: The authors recommend specialized algorithms for fair contextual word embeddings tailored to healthcare settings.They also publicly release a pretrained BERT model and code.
- Conclusion: Changing patient race in a medical word-completion example produces different predicted courses of action, illustrating a potential route to discriminatory clinical recommendations.The example keeps the medical context fixed while replacing the race token.
- Conclusion: 57 downstream clinical prediction tasks show statistically significant gaps in parity, recall, and specificity across all protected attributes.The reported gaps occur across gender, language, ethnicity, and insurance status, with subgroup advantages varying by comparison.
D.1 Method
The adversarial method jointly trains BERT for language modeling and next-sentence prediction while discouraging recovery of protected-group variables from its representations. A gradient-reversal component balances representation utility against fairness.
- D.1 Method: Encoded sequence pairs are represented by BERT and passed to two discriminators that predict the protected-group variables for each sequence.The discriminators receive the last-layer [CLS] representation and predict z1 and z2.
- D.1 Method: The adversarial objective combines two protected-group losses with masked-word prediction and next-sentence prediction losses.The loss is Ladv(a1(J(h)),z1) + Ladv(a2(J(h)),z2) + LLM + LN S.
- D.1 Method: J is an identity function with a negative gradient, reversing the gradient passed to BERT during adversarial training.This preserves the representation value while directing optimization against protected-group predictability.
- D.1 Method: λ balances the utility-to-fairness trade-off, and the authors use λ = 1.The hyperparameter controls the relative influence of adversarial debiasing.
D.2 Gender Debiasing
The gender-debiasing analysis compares how well baseline and debiased BERT representations support gender prediction. The supplied passage identifies the comparison but reports no numerical outcome.
- D.2 Gender Debiasing: Baseline and debiased BERT representations are compared by training fully connected neural networks to predict gender.The comparison concerns the representations generated by each BERT model.
D.3 Other Protected Groups
The paper compares fairness gaps for baseline and debiased embeddings across language, ethnicity, and insurance groups. Each comparison uses three fairness definitions and reports statistically significant task gaps with subgroup-favoring percentages.
- D.3 Other Protected Groups: Language fairness comparisons stratify downstream performance between English and Non-English speakers.Other language groups are collapsed because of small individual group sizes.
- D.3 Other Protected Groups: Ethnicity fairness comparisons report the number of tasks with statistically significant gaps and the percentage favoring each subgroup.The comparisons use baseline and debiased embeddings across three fairness definitions.
- D.3 Other Protected Groups: Insurance fairness comparisons likewise report significant-gap task counts and the percentage of tasks favoring each subgroup.All comparisons apply Benjamini-Hochberg correction for false discovery rate and assess three fairness definitions.