A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learning

1 Introduction

• Vision-language models trained on large web-scraped datasets can perpetuate damaging biases, while their scale makes manual quality auditing difficult.The paper also notes that training data represent a snapshot in space and time, raising concerns about cultural and temporal specificity.Source: §1 · p. 1
• Representational harms can reinforce stereotypes in image search, while allocational harms can arise when search rankings affect perceptions of job suitability.The paper contrasts these risks with more direct dangers from applying face models to criminality or homosexuality.Source: §1 · p. 1
• The paper focuses on representational harm when sensitive text queries are mapped onto face datasets, where vision-language bias measures and feasible debiasing approaches are comparatively limited.Retraining is constrained by lack of access to original training data and the compute it would require.Source: §1 · p. 1
• The paper evaluates bias measures and gender and racial bias in two face datasets, then proposes learned text tokens trained for unbiasedness and image-text contrastive loss.The framework uses sensitive image-attribute labels as supervision.Source: §1 · p. 2

2.1 Problem Statement

• The target is a pretrained vision-language model that retains semantically meaningful image-text similarity scores while producing similar score distributions across attributes for neutral queries.The formal setup uses image-attribute pairs and sensitive text queries with associated concepts.Source: §2.1 · p. 2

2.2 Sensitive Attributes and Relevancy

• The framework assumes some queries, such as “a good/bad person,” should be independent of demographic categories, but which associations are legitimate remains normative and context-dependent.Prompts and sensitive attributes should be adapted to the model’s deployment context and culture.Source: §2.2 · pp. 2–3

2.3 Bias Metrics

• The paper examines WEAT and applies information-retrieval ranking measures, including MaxSkew and NDKL, to assess attribute representation in text-image retrieval.The retrieval framing reflects concern that images associated with some attributes may rank higher in top-k results.Source: §2.3 · p. 3

3.1 Fairness Objective with Adversarial Debiasing

• The adversarial objective trains a classifier to predict image attribute labels from similarity logits for sensitive text queries, while the vision-language model is optimized to maximize the adversary’s loss.The representation is treated as unbiased when it is blind to sensitive attributes over the selected neutral queries.Source: §3.1 · p. 3

3.2 Adaptation Methods

• Unregularized adversarial optimization can collapse image-text scores to trivial values, destroying semantic information needed for downstream tasks.The paper therefore investigates restrictions on trainable parameters and joint training with image-text similarity objectives.Source: §3.2 · p. 3
• Finetuning depth offers one adaptation constraint; the paper also uses learnable text tokens prepended to embedded queries to regularize adversarial debiasing.The cited prior work reports that text-only finetuning can improve generalization and reduce catastrophic forgetting relative to full finetuning.Source: §3.2 · pp. 3–4
• An auxiliary image-text contrastive loss is added to preserve image-text similarity performance during debiasing.The paper notes that another image-text matching loss could substitute for ITC.Source: §3.2 · p. 4

4.1 Datasets

• FairFace (108,501 images) is balanced by age, gender, and ethnicity; UTKFace (20,000) groups several populations into “Others” and varies in lighting, color quality, and portrait angle.Source: §4.1 · p. 4

4.2 Experimental Protocol

• The test pairs adjectives unrelated to expressions or sensitive attributes, adds unseen templates and concepts, and averages queries; WEAT was sensitive to architecture, dataset, and syntax, so MaxSkew@1000 and NDKL were used.Source: §4.2 · p. 4

4.3 Results

• At λ*=0.05, joint training outperformed CLIP-clip in downstream performance for every m; CLIP-clip necessarily removes information beyond gender when clipping features.Figure 2 compares NDKL with IN1Kacc across ITC loss weights and numbers of removed dimensions.Source: Figure 2 · p. 5, §4.3 · pp. 5–6
• Larger pretraining datasets reduced MaxSkew and NDKL, while larger image encoders generally improved feature quality; FiT broke the encoder-size pattern.SLIP models trained with self-supervised learning had lower MaxSkew than their non-SSL counterparts.Source: §4.3 · p. 5, Table 2 · p. 6
• -52% to -65% bias metrics with +3% to -1% feature quality: joint adversarial and ITC training significantly reduced bias while feature quality improved or stayed relatively unchanged.The comparison is against the baseline in the ablation of debiasing approaches.Source: §4.3 · p. 5, Table 3 · p. 6
• -73% to -37% bias change followed FairFace training versus -35% to -3% after UTKFace training; on UTKFace, FairFace training increased NDKL but decreased MaxSkew.The authors suggest FairFace's training subset, about four times larger than UTKFace, may explain the larger reductions and say further work is needed to confirm generalization.Source: §4.3 · p. 6, Table 4 · p. 7
• -45% to -40% racial-bias reduction followed gender debiasing, compared with -79% to -69% for gender, indicating reductions can extend across attribute classes.Source: §4.3 · p. 6, Table 4 · p. 7
• For the “smart person” query, CLIP’s top-ranked images skewed male before debiasing, while the post-debiasing images were more balanced by gender and age.The qualitative example ranks images from the FairFace validation set.Source: Figure 3 · p. 7, §4.3 · p. 7

Related work overview

• Despite recent open-source model releases, only a few studies had addressed vision-language bias measurement and mitigation.The authors drew on dataset- and model-level bias research in NLP and computer vision.Source: §5 · p. 7

Bias in NLP

• NLP data augmentation, perturbation, and supervised finetuning were considered infeasible for the studied large-scale models because the original data and required compute were unavailable.Source: §5 · p. 7

Bias in computer vision

• Computer-vision work offers model-level strategies such as confusion optimization and adversarial training; the authors adopted adversarial finetuning as computationally cheap and independent of original training data.Source: §5 · p. 7

Bias in vision-language

• Vision-language debiasing has included feature removal, dataset balancing, and model-level adjustments, but feature removal loses information and balancing alone may be insufficient.The reviewed literature also notes that dataset-level correction may miss demographic proxies.Source: §5 · p. 8

Domain adaptation of pretrained models

• Prompt learning adapts pretrained models by learning input tokens, and joint image-text contrastive training is used to counter feature degradation from bias reduction.The section also notes that freezing the image encoder while optimizing the text encoder can improve generalization.Source: §5 · p. 8

Ethical considerations

• The authors identify how bias and fairness are operationalized as an ethical consideration.Source: §6 · p. 8

Assumptions on computational restrictions

• The method assumes the model is too large to pretrain within budget and the original training data are inaccessible; otherwise, the authors encourage ethical data curation and fairness-aware initial training.Source: §6 · p. 8

Context-dependency of the debiasing goal

• The debiasing goal depends on context: choosing relevant identity attributes and text prompts, and defining their relation to bias, is normative and subjective.Demographic distributions and lived experiences vary across regions.Source: §6 · p. 8

Lack of intersectional analysis

• Experiments cover gender and ethnicity but not intersectional bias, which may be larger when attributes such as age and gender combine.The authors expect the techniques to work with similar efficacy and efficiency in intersectional experiments.Source: §6 · p. 8

Focus on representational harms

• The paper focuses primarily on representational harms and does not explicitly test allocational harms that may accompany them in downstream uses.Future work is encouraged to examine general and case-specific settings with stakeholders and affected communities.Source: §6 · pp. 8–9

Sole focus of bias in face images

• Experiments use face datasets, a narrow view that does not capture biases tied to cultural artifacts, practices, and geographic localities.Source: §6 · p. 9

7 Conclusion

• MaxSkew and NDKL are identified as effective ranking-based bias measures for image-text representations.The paper reports these metrics for pretrained models on gender and racial bias in face photos.Source: §7 · p. 9
• More pretraining data correlates with lower model bias, and training with self-supervised learning can reduce bias.Source: §7 · p. 9
• Joint training with an image-text contrastive loss wholly mitigates feature degradation while learned debiasing tokens substantially reduce gender and race bias metrics.The experiments use publicly available face-image datasets with attribute labels.Source: §7 · p. 9
• Future work includes debiasing during pretraining, broadening the attributes studied, and examining intersectional bias; the authors encourage transparent documentation and relatively cheap debiasing protocols.Code, models, and debiasing tokens are publicly available.Source: §7 · p. 9

A Word Embedding Association Test (WEAT)

• WEAT compares differential associations between two target concepts and two attributes; in this study, text queries supply concepts and image embeddings supply attributes.The effect size is described as a normalized measure of separation between target-attribute associations.Source: A Word Embedding Association Test (WEAT) · p. 14

• Skew@k compares the desired and observed proportions of an image attribute among the top-k results, with negative values indicating under-representation.The chosen k affects the result, and measuring one attribute at a time requires aggregation for a holistic view.Source: B Ranking metrics · p. 14
• MaxSkew@k selects the largest Skew@k across image attribute labels for a text query, with zero as the desired value when actual and desired distributions match.The maximum represents the largest unfair advantage among the attribute labels.Source: B Ranking metrics · p. 14
• NDKL uses a rank-weighted average of KL divergence between top-k attribute distributions and the desired distribution, addressing Skew's aggregation and choice-of-k drawbacks.Because NDKL is non-negative, it cannot distinguish between opposite-biased search procedures.Source: B Ranking metrics · pp. 14–15

C Measuring bias across different model architectures, datasets, and syntactic changes.

• MaxSkew@1000 and NDKL perform consistently across tested model architectures, datasets, and minor query-syntax changes, unlike WEAT, which varies substantially.WEAT produced opposing signs across FairFace and UTKFace and both positive and negative results after small template changes; the authors attribute this possibly to using long prompts with a metric designed primarily for single-word embeddings.Source: C Measuring bias across different… · p. 15

D Performance effects of learnable text token initialization

• Zero-pad initialized text tokens substantially reduce zero-shot retrieval and classification performance before debiasing training, with larger drops as token count increases in both prepending and appending settings.The authors suggest the change is due to learnable tokens shifting the text-query distribution, rather than adversarial learning.Source: D Performance effects of learnable… · p. 15, Table 5 · p. 15

E Debiasing

• Even zero-pad initialized prepended tokens slightly modify text-encoder outputs before training, degrading model performance.The tokens are initialized as zero-pad embeddings while the remaining model weights are frozen.Source: E Debiasing · p. 15

• The adversarial classifier is a two-hidden-layer MLP, with input size set by the training prompts and output size by the sensitive attributes being debiased.The reported configuration uses hidden layers of size 32 and batch size 256.Source: F Experimental protocol · p. 15
• Figure 4 measures gender bias across minor syntactic changes, three model architectures, and the FairFace validation and UTKFace datasets.The reported standard deviation is taken over model-architecture and dataset combinations.Source: Figure 4 · p. 16
• After two initial epochs training only the adversary, CLIP and the adversary are alternately trained for 10 batches each, with early stopping if CIFAR100 or Flickr-1k performance falls below 50% of original accuracy.The learning rates are 2 · 10−5 for CLIP and 2 · 10−4 for the adversarial classifier.Source: F Experimental protocol · p. 16
• Finetuning considers combinations of the last three layers of each encoder and both projection layers, rather than the full model, to limit expected feature-quality loss and computational demands.This entails training up to 25% of the model's parameters.Source: F Experimental protocol · p. 16
• The experiments use two learned tokens as a feature-quality trade-off, while other tested prompt-learning variants showed no significant final-result changes.For joint image-text contrastive training, λ = 0.05 and Flickr30K training batches are used unless otherwise specified.Source: F Experimental protocol · p. 16

• For ethnicities with baseline misclassification rates above 1%, gender debiasing reduces harmful misclassification rates by 43% to 96% relative to pretrained CLIP.The comparison is between a pretrained CLIP ViTB/16 baseline and a model trained on FairFace gender attributes using learned prompt-token embeddings.Source: Table 6 · p. 17, G Harmful Zero-Shot Misclassification · p. 17
• The evaluation classifies FairFace validation images into 14 ethnicity-gender pairs, non-human categories, and crime-related categories using the template “a photo of a {}”.Without a query template, classification accuracy was significantly reduced for all images.Source: G Harmful Zero-Shot Misclassification · p. 17
• The gender-debiased model has a lower crime-related misclassification rate than pretrained CLIP, while its non-human rate is marginally higher but remains below 1%.The comparison covers crime-related and non-human categories across FairFace ethnicity groups.Source: G Harmful Zero-Shot Misclassification · p. 17, Table 6 · p. 17

H Additional Results

• Few learned embeddings provided strong regularization that kept feature quality at an acceptable level.The passage contrasts this with fine-tuning larger parts of the model, which quickly lowered performance to an unacceptable level.Source: H Additional Results · p. 17
• Fine-tuning larger parts of the model quickly lowered performance to an unacceptable level.Source: H Additional Results · p. 17
• The comparison covered different fine-tuning depths and pure prompt learning, all with pure adversarial training.Source: H Additional Results · p. 17