PaperlayerView exact revision on arXiv

Source-linked AI summary

Hurtful Words: Quantifying Biases in Clinical Contextual Word Embeddings

Haoran Zhang, Amy X. Lu, Mohamed Abdalla, Matthew McDermott, Marzyeh Ghassemi

arXiv:2003.11515v1cs.CLcs.CYcs.LGstat.ML

TL;DR

Clinical contextual embeddings may encode protected groups differently, raising fairness questions as machine learning systems enter society. The paper evaluates clinical BERT using latent-association tests and downstream prediction tasks, finding performance gaps across most tasks and significant fairness-metric gaps across protected attributes.

  • Problem

    It remains important to assess whether clinical contextual embeddings encode protected populations differently, given documented fairness concerns in real-world machine learning systems.

    Source: §1 · pp. 1–2

  • Method

    The authors pretrain BERT on MIMIC-III notes, use clinical-note fill-in-the-blank and log-probability tests plus fairness metrics across 57 prediction tasks, and test adversarial pretraining debiasing.

    Source: §1 · p. 2, §3.4 · p. 4

  • Results

    Across a majority of 57 downstream clinical prediction tasks, baseline clinical BERT showed performance gaps, with statistically significant parity, recall, and specificity gaps across all protected attributes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Before healthcare deployment, model biases should be carefully examined across protected groups and their intersections.

    Source: §8 · p. 9

    The authors treat post-stay ICD-9 billing labels as gold standards without accounting for possible errors or biases in those labels.

    Source: §7 · p. 9

Abstract

from arXiv · show

In this work, we examine the extent to which embeddings may encode marginalized populations differently, and how this may lead to a perpetuation of biases and worsened performance on clinical tasks. We pretrain deep embedding models (BERT) on medical notes from the MIMIC-III hospital dataset, and quantify potential disparities using two approaches. First, we identify dangerous latent relationships that are captured by the contextual word embeddings using a fill-in-the-blank method with text from real clinical notes and a log probability bias score quantification. Second, we evaluate performance gaps across different definitions of fairness on over 50 downstream clinical prediction tasks that include detection of acute and chronic conditions. We find that classifiers trained from BERT representations exhibit statistically significant differences in performance, often favoring the majority group with regards to gender, language, ethnicity, and insurance status. Finally, we explore shortcomings of using adversarial debiasing to obfuscate subgroup information in contextual word embeddings, and recommend best practices for such deep embedding models in clinical settings.

1 INTRODUCTION

The introduction reports gender differences in clinical-text log-probability bias scores and statistically significant performance gaps across most downstream tasks and all protected attributes. Adversarial debiasing did not greatly reduce the number of significant gaps.

Introduction

  • Contextual embeddings can reflect biases and encode historical corpus bias, dataset class imbalance, and data-quality differences, raising concern as their use grows in healthcare.The introduction notes that machine-learning systems may exacerbate existing treatment disparities.Source: §1 · pp. 1–2
  • The study fine-tuned SciBERT on MIMIC-III clinical notes to create a baseline clinical BERT and assessed demographic parity and equality of opportunity for positive and negative classes.The evaluation considered gender, language spoken, ethnicity, and insurance type.Source: §1 · p. 2
  • In a clinical word-completion example, changing the patient race produced a worse suggested course of action for African American patients.The prompt example used SciBERT and templates adapted from real MIMIC-III clinical notes.Source: Figure 1 · p. 1, §1 · p. 2
  • Statistically significant gaps in parity, recall, and specificity appeared across a majority of 57 downstream clinical prediction tasks for all protected attributes.The attributes examined were gender, language spoken, ethnicity, and insurance type.Source: §1 · p. 2
  • Clinical-text log-probability bias scores differed significantly by gender, measuring the probability of filling in gender demographics given medical context.Source: §1 · p. 2
  • Adversarial debiasing during pretraining did not greatly reduce the number of statistically significant performance gaps.The approach used a discriminator to make embeddings minimally predictive of protected subgroup information; disparities could remain after post-hoc classification [19].Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

Contextual embeddings vary with surrounding text, and their pretraining can encode biases present in training data. Prior debiasing methods have focused on non-contextual embeddings, while this work trains a clinical BERT to address limitations in existing models.

2.1 Contextual Embeddings

  • Word embeddings represent text as dense vectors; unlike Word2Vec [45] and GloVe, contextual ELMo [50] and BERT [16] vary a word’s representation with surrounding context.Both approaches use context in forming representations, consistent with the distributional hypothesis.Source: §2.1 · p. 2
  • Contextual embeddings are often pretrained through self-supervised tasks, sometimes on clinical text [3] [29] [60], where training-text bias may become encoded.This makes pretraining another potential source of bias in the resulting model.Source: §2.1 · p. 2

2.2 Fairness of Word Embeddings

  • Prior work examines bias in contextual embeddings [5] [36], but debiasing methods have focused on non-contextual GloVe and Word2Vec [45].Embedding models have captured gender [9] and race and ethnicity [21] biases, and simple tests can miss bias without robust evaluation [24].Source: §2.2 · p. 2

2.3 Pretrained Clinical Embeddings

  • The authors train their own clinical BERT to address existing models’ initialization, personal health identifier (PHI) handling, and lack of whole-word masking.Existing models are initialized from BioBERT [39] or BERTBASE [16]; SciBERT [6] outperforms BioBERT on several downstream tasks. The new model addresses all three limitations.Source: §2.3 · p. 3

3 DATA AND PRE-TRAINING

The study uses MIMIC-III notes and demographic data to pretrain clinical BERT and construct downstream prediction tasks. It also specifies note preprocessing, training, and templates for probing gender-pronoun predictions.

3.1 Data Source

  • MIMIC-III provides about 2 million clinical notes and demographic information for 38,597 adults and 7,870 neonates admitted to one ICU from 2001 to 2012.The records include ethnicity, spoken language, and insurance status.Source: §3.1 · p. 3

3.2 Clinical Prediction Tasks

  • Downstream tasks comprise in-hospital mortality and 28 binary phenotyping problems using all-note or first-note cohorts [27].The mortality cohort uses selected notes from the first 48 ICU hours; the all-notes phenotype cohort adds discharge summaries, while the first-note cohort uses a single early nursing or physician note, with different prevalences.Source: §3.2 · p. 3

3.3 Baseline Clinical BERT Pretraining

  • The baseline model is initialized from SciBERT and uses whole-word masking to avoid easy prediction of partially masked wordpiece tokens [6] [14].SciBERT is described as having performed better on a variety of benchmarking tasks [6].Source: Tokenization and PHI Identifier Removal · p. 3

Tokenization and PHI Identifier Removal.

Training.

  • Pretraining uses one epoch of length-128 sequences followed by one epoch of length-512 sequences, trained across four GPUs.The stages use approximately 8 million samples with batch size 32, then 4 million samples with batch size 16.Source: Training · p. 4

Whole-note Embedding.

  • For notes longer than 512 tokens, the model processes up to 10 overlapping subsequences and merges their probability outputs into a note-level prediction [29].The merge uses the maximum and mean subsequence probabilities with a task-specific scaling factor tuned on the validation set.Source: Whole-note Embedding · p. 4

3.4 Note Template Generation

  • Gender-pronoun templates cover chronic illness, mental health and addiction, HIV, and end-of-life treatment to compare prediction likelihoods.The motivating concerns include treatment differences in chronic illness, mental health, addiction, and analgesic access, HIV, and DNR assignment.Source: §3.4 · p. 4

4 METHODS

The study evaluates bias in a clinical BERT model trained on MIMIC-III notes using downstream prediction tasks, fairness gaps, fill-in-the-blank log-probability scores, and pretraining-time adversarial debiasing.

4 METHODS

  • The baseline clinical BERT model uses all MIMIC-III NoteEvents notes; the study evaluates 57 fine-tuning tasks and bootstraps 1,000 test-set samples for 95% confidence intervals on fairness gaps.It reports the number of statistically significant gaps for each protected group.Source: §4 · p. 4

4.1 Downstream Training

  • For downstream training, the authors freeze BERT and concatenate the last four [CLS] layers’ representations, aiming to assess representation bias without altering those representations during fine-tuning.They also concatenate age and admission acuity scores, then select a neural network using validation AUPRC and choose binary-prediction thresholds using validation F1.Source: §4.1 · pp. 4–5

Train-Test Splits.

  • The 57 downstream tasks use a held-out test set from prior work [27] and a 20% validation split, with no patient appearing across splits.The tasks cover phenotyping and other clinical predictions; phenotype prevalence is reported in Appendix Table A1.Source: Train-Test Splits · p. 5

4.2 Evaluation of Classifier Fairness

  • Classifier fairness is measured with demographic parity and equality of opportunity for positive and negative classes; recall gaps are prioritized as clinically relevant.Because tasks are negative-class imbalanced, an always-negative classifier can have 100% true negative rate and zero specificity gap; demographic parity is included for comparison but may worsen other healthcare biases [26] [51] [35]. For more than two groups, the reported gap is the maximum difference from any other group.Source: §4.2 · p. 5, Multi-class Fairness Expansions · p. 5

Multi-class Fairness Expansions.

  • For multiple protected groups, the study extends demographic parity and similarly derives recall and specificity gaps following prior work.For each group, the reported fairness gap is its maximum performance difference from all other groups.Source: Multi-class Fairness Expansions · p. 5

4.3 Evaluation of Log Probability Score

  • The log-probability bias score measures prior-adjusted likelihoods of predicting words in fill-in-the-blank templates [36]; male-word and female-pronoun scores are compared using a Wilcoxon signed-rank test.Because significant scores need not indicate unintended bias when condition base rates differ, the authors also examine published prevalences where possible and calculate gender ratios in MIMIC-III.Source: §4.3 · p. 5

4.4 Algorithmic Debiasing

  • The adversarial debiasing method uses discriminator networks during pretraining to reduce subgroup predictability without downstream labels, with gender results in Section 5.3 and other attributes in Appendix D.3 [8] [18] [19] [42] [71].Two three-layer, fully connected ReLU discriminators are used, one for each BERT subsequence; training reverses the adversarial gradients (Figure 3).Source: §4.4 · p. 5, Figure 3 · p. 6

5 RESULTS

Baseline clinical BERT encodes gender-related associations from clinical notes and shows downstream performance gaps; adversarial debiasing retains predictive performance but does not sufficiently reduce disparities.

5.2 Performance Gaps Favor the Majority Group

  • 13/57 gender tasks and 7/57 language tasks show significant recall gaps, the study’s most clinically relevant fairness measure.The visualized tasks are those with significant recall differences; the gaps vary in magnitude.Source: §5.2 · p. 6, §5 · p. 6

5.1 Further Pretraining on Clinical Text Captures Gender Prevalence in Notes

  • 7/8 medical context categories show significant gender likelihood differences after clinical-note pretraining, versus 2/8 for SciBERT, with predictions shifting toward male pronouns.The passage attributes the shift to gender-related associations in the notes, where most considered conditions occur more often in male discharge summaries.Source: §5.1 · p. 6, Table 3 · p. 7

5.2 Performance Gaps Favor the Majority Group

  • Recall gaps favor majority groups except for language; gender gaps often favor men, ethnicity gaps often favor White and Asian patients, and insurance gaps favor Medicare patients.The model performs poorly for the heterogeneous “Other” ethnicity category; many ethnicity- and insurance-related gaps remain after Benjamini-Hochberg FDR correction.Source: §5.2 · p. 7

5.3 Adversarial Debiasing During Pretraining Is Insufficient

  • 0.996 AUPRC correlation with baseline clinical BERT shows that debiased clinical BERT retains strong downstream performance overall.The comparison is between classifiers using embeddings from baseline and debiased clinical BERT.Source: §5.3 · p. 7
  • Adversarial debiasing slightly reduces tasks with significant recall gaps but does not appear to reduce specificity gaps, leaving disparity reduction insufficient for high-stakes deployment.Similar results are reported when language, ethnicity, and insurance are protected groups.Source: §5.3 · p. 7
  • Although the adversary fools the discriminator during training, a post-hoc classifier can still extract sensitive-attribute information from the embeddings.This finding reaffirms past work [19].Source: §5.3 · p. 7

6 DISCUSSION

The discussion argues that clinical-note pretraining can encode note-level gender skew, while baseline clinical BERT shows demographic performance gaps that warrant auditing before deployment. It also identifies practical and conceptual limits of adversarial debiasing.

6.1 Further Pretraining Captures Note Biases

  • Clinical-note pretraining makes the model significantly more likely to predict male gendered pronouns, reflecting note patterns that may encode unintended bias.Hypertension prevalence is roughly equal for men and women, but hypertension-related words appear more often in male discharge notes, and baseline clinical BERT is more likely to predict male gender after pretraining.Source: §6.1 · p. 7

6.2 Fairness Gaps Along Known Biases

  • Males have higher recall on eight of 13 tasks, while Black and Hispanic/Latino patients have lower recall; co-morbidities and healthcare under-utilization are proposed explanations.The discussion connects higher male heart-disease prevalence in MIMIC-III to previous studies of heart-disease under-diagnosis in women, and notes that male co-morbidities may complicate prediction.Source: §6.2 · p. 7
  • Baseline clinical BERT shows significant performance gaps across gender, language, ethnicity, and insurance status, supporting further disparity research before deployment.The discussion cautions that performance gaps do not necessarily correspond directly to real-world treatment or outcome disparities, and may be difficult to disentangle from true medical confounders.Source: §6.2 · pp. 7–8, Table 4 · p. 8

6.3 Challenges in Using Adversarial Debiasing for Clinical Contextual Embedding Models

  • Adversarial debiasing during pretraining does not greatly reduce fairness gaps, and a discriminator comparable in capacity to BERT is infeasible to train.The authors suggest that the adversarial decoder may be underpowered for a high-capacity encoder.Source: §6.3 · p. 8, Table 5 · p. 9
  • Debiasing only the [CLS] token would not address sequence tasks such as NER or question answering, while pretraining debiasing targets demographic parity, considered problematic in healthcare [8] [71].The discussion identifies adversarial debiasing during finetuning as a possible route to other fairness definitions [71].Source: §6.3 · p. 9

7 LIMITATIONS AND FUTURE WORK

The authors identify limits in label reliability, intersectional and causal bias analysis, direct debiasing, and the ethical tradeoffs of fairness interventions in healthcare.

7 LIMITATIONS AND FUTURE WORK

  • The study treats post-stay ICD-9 billing labels as gold standard, omits intersectional discrimination, and does not disentangle bias sources such as imbalance, data quality, or healthcare-system biases.It also does not directly debias pretraining or finetuning data.Source: §7 · p. 9
  • Debiasing methods may hide rather than remove bias [19] [24], while automated augmentation may help in some cases and building unbiased training data can be time consuming [10].The authors do not attempt direct data debiasing.Source: §7 · p. 9
  • Whether to use a fairer model that trades off performance and patient outcomes across groups remains an ethical, case-by-case decision for healthcare providers.Source: §7 · p. 9

8 CONCLUSION … D ADVERSARIAL DEBIASING DURING PRETRAINING

Medical-note BERT embeddings encode unwanted relationships across protected groups, and downstream classifiers show significant group performance differences. The authors call for healthcare-specific methods to detect and reduce these biases before clinical deployment.

8 CONCLUSION

  • The authors recommend examining bias across protected groups and their intersections before deploying models, especially in high-stakes healthcare settings.They encourage development of contextual-embedding methods tailored to healthcare to help minimize bias in clinical machine learning.Source: §8 · p. 9
  • BERT pretrained on medical notes propagates unwanted latent relationships involving gender, language, ethnicity, and insurance groups, while classifiers show significant performance differences across groups.The authors also report that medical-note pretraining makes their baseline clinical BERT more confident when predicting a note’s gender, potentially capturing gender–condition relationships beyond biological associations.Source: §8 · p. 9
  • Medical-note pretraining increased the baseline clinical BERT’s confidence in the note’s gender and may have captured gender–condition relationships beyond biological associations.The study also qualitatively examined SciBERT’s predicted course of action when race was changed while medical context was kept the same.Source: §8 · p. 9

A APPENDIX A - DESCRIPTIVE STATISTICS AND MODEL PERFORMANCE

  • Appendix A covers sample counts, gender-majority percentages and prevalence by gender, plus AUROC and AUPRC for baseline and debiased clinical BERT models.Its figure compares baseline and debiased-model AUPRC across downstream tasks, with 95% bootstrap confidence intervals.Source: Table A1 · p. 12, Table A2 · p. 13, Figure A1 · p. 14

B PROCEDURE FOR GENERATING SENTENCE COMPLETION FIGURE

  • The sentence-completion procedure changes the race token in MIMIC-III-adapted templates and asks SciBERT to predict the next two tokens, representing course of action.The tested race terms are “caucasian,” “white,” “african,” “african american,” and “black.”Source: B PROCEDURE FOR GENERATING SENTENCE… · p. 15

C MULTIPLE HYPOTHESIS CORRECTION OF TABLE 4

  • After controlling the false discovery rate, most multi-group fairness performance gaps remain across the 57 downstream classification tasks.The table counts statistically significant tasks and the percentage favoring a subgroup; it specifies Benjamini–Hochberg correction.Source: Table C3 · p. 15

D.1 Method

The method adds adversarial protected-group prediction to BERT training while retaining masked-word and next-sentence objectives, with a hyperparameter balancing utility and fairness.

D.1 Method

  • BERT representations of two encoded sequences feed separate discriminators that try to recover each sequence’s protected-group variable.Both discriminators receive h[CLS], alongside its use for the next-sentence task.Source: §D.1 · p. 15
  • The final loss combines adversarial losses for both protected-group variables with masked-word prediction and next-sentence prediction losses.The objective sums Ladv(a1(J(h)), z1), Ladv(a2(J(h)), z2), LLM, and LNS.Source: §D.1 · p. 15
  • The identity function J applies a negative gradient, and λ balances the utility–fairness trade-off; this work uses λ = 1.Source: §D.1 · p. 15

D.2 Gender Debiasing

This section compares baseline and debiased BERT representations on their ability to predict gender.

D.2 Gender Debiasing

  • Table D4 compares gender prediction from baseline versus debiased BERT representations using a fully connected neural network.The supplied passage identifies the comparison but does not report its results.Source: Table D4 · p. 15

D.3 Other Protected Groups

This section compares downstream performance gaps across language, ethnicity, and insurance groups using baseline and debiased embeddings. The tables assess three fairness definitions and report statistically significant gaps with subgroup-favoring percentages.

D.3 Other Protected Groups

  • Table D5 compares English and Non-English speakers, reporting the percentage of gaps favoring English after collapsing other language groups because of small group sizes.It assesses three fairness definitions and applies Benjamin-Hochberg false discovery rate correction [7].Source: Table D5 · p. 16
  • Table D6 reports, by ethnicity, the number of tasks with statistically significant gaps and the percentage favoring each subgroup.It assesses three fairness definitions and applies Benjamin-Hochberg false discovery rate correction [7].Source: Table D6 · p. 16
  • Table D7 reports, by insurance group, the number of tasks with statistically significant gaps and the percentage favoring each subgroup.It assesses three fairness definitions and applies Benjamin-Hochberg false discovery rate correction [7].Source: Table D7 · p. 16

Related papers

Loading 2003.11515v1…