Source-linked AI summary
Debiasing Vision-Language Models via Biased Prompts
Ching-Yao Chuang, Varun Jampani, Yuanzhen Li, Antonio Torralba, Stefanie Jegelka
TL;DR
Vision-language models can inherit and propagate social bias and spurious correlations from their training data. The paper projects biased directions out of text embeddings and calibrates that projection with a closed-form, training-free objective. Experiments report improved robustness and reduced retrieval bias without additional data or labels, while projection quality remains limited by inaccurate or insufficient descriptions of irrelevant attributes.
Problem
Vision-language models can inherit social bias and spurious correlations from training data, which can propagate to zero-shot classifiers and generative applications.
Method
The method projects biased directions out of text embeddings and calibrates the projection using positive prompt pairs with a closed-form solution.
Results
The approach improves group robustness and significantly reduces retrieval MaxSkew across attributes and backbones without downstream data or labels.
Takeaways & Limitations
Text-only, training-free debiasing can be applied across discriminative and generative vision-language pipelines.
Takeaways & Limitations
Projection quality is limited by approximation error and by difficulty describing irrelevant attributes with a small number of prompts.
Abstract
from arXiv · showhide
Machine learning models have been shown to inherit biases from their training datasets. This can be particularly problematic for vision-language foundation models trained on uncurated datasets scraped from the internet. The biases can be amplified and propagated to downstream applications like zero-shot classifiers and text-to-image generative models. In this study, we propose a general approach for debiasing vision-language foundation models by projecting out biased directions in the text embedding. In particular, we show that debiasing only the text embedding with a calibrated projection matrix suffices to yield robust classifiers and fair generative models. The proposed closed-form solution enables easy integration into large-scale pipelines, and empirical results demonstrate that our approach effectively reduces social bias and spurious correlation in both discriminative and generative vision-language models without the need for additional data or training.
1 Introduction
Vision-language foundation models can propagate dataset biases into discriminative and generative applications. The paper proposes a training-free, calibrated projection of text embeddings to reduce these biases across model types.
- Vision-language models can inherit stereotypes, racist content, ethnic slurs, spurious correlations, and demographic bias from their training data.
- The method defines biased directions from prompts and projects them out of the text embedding.
- Calibration makes prompt embeddings for the same class but different biased attributes similar, using a closed-form objective.
- The approach extends to generative models by applying a universal debiasing matrix before text embeddings enter the generator.
- Experiments cover discriminative zero-shot and retrieval models as well as generative diffusion models without requiring training, data, or labels.
2 Related Works
Prior work addresses bias in language, vision, and multimodal models through projections, training, adversarial methods, and data augmentation. The paper distinguishes its approach as fully zero-shot and data-free.
- Language-model debiasing includes orthogonal projections, sentence-embedding extensions, training constraints, and dataset modification.
- Vision-model methods address social and spurious bias through adversarial representation learning and biased-dataset augmentation.
- Multimodal approaches have removed gender-correlated embedding dimensions, used prompt learning, adversarial methods, residual image representations, and contrastive learning.
- Unlike these data-oriented approaches, the proposed method is fully zero-shot and requires no downstream dataset or model training.
3 Biases and Spurious Correlations
The paper frames bias as spurious correlation between target labels and irrelevant attributes that can transfer into discriminative and generative outputs. It introduces the relevant model settings and fairness measures.
- A spurious attribute is an irrelevant feature correlated with the target label, such as gender correlated with the class “doctor.”
- Zero-shot classifiers use class-prompt embeddings as classifier weights and can inherit biases without data or training during construction.
- Text-image retrieval fairness is evaluated with MaxSkew, based on attribute proportions among the top-k retrieved images.
- A fair generative model minimizes distributional discrepancy by making each spurious attribute equally probable.
4 Debiasing Discriminative Models
The discriminative debiasing pipeline identifies irrelevant prompt directions, projects them out, and calibrates the projection with positive prompt pairs. The calibration improves robustness while remaining vulnerable to inaccurate direction estimates.
- Debiasing Discriminative Models: Zero-shot robustness requires invariance to irrelevant image features such as backgrounds, race, and gender, but conventional dataset-based debiasing is unsuitable for zero-shot settings.
- Measuring Biases with Prompts: Prompt embeddings describe spurious features, allowing classifier bias to be measured through cosine similarity with irrelevant directions.
- Debiasing via Orthogonal Projection: Orthogonal projection aligns classifier weights with the complement of spurious prompt embeddings and applies the transformation P0z to text embeddings.
- Calibrating the Projection Matrix: Projection-direction estimation can incur approximation error, and limited prompts may inadequately describe irrelevant attributes; orthogonal projection consequently sometimes fails to improve performance.
- Calibrating the Projection Matrix: Positive prompt pairs describe the same class with different spurious attributes, encouraging the projection to remove variation between those attributes.
- Calibrating the Projection Matrix: The calibration objective has a closed-form minimizer whose SVD interpretation down-weights embedding variation associated with spurious directions.
- Relation to an Equalization Loss: The calibrated solution is equivalent to an equalization objective that preserves proximity to the original embedding while equalizing similarity to positive-pair embeddings.
- Relation to an Equalization Loss: The equalization interpretation connects the method to centering gender-related word embeddings so that other words become equidistant from the centered set.
5 Experiments: Discriminative Models
The experiments evaluate debiasing on spurious correlations and retrieval fairness using CLIP backbones and FairFace. Calibration improves group robustness and reduces bias without data or labels.
- Group robustness: Waterbird and CelebA evaluate group robustness against background and gender spurious correlations, respectively.
- Baselines: The comparison includes zero-shot classification, ERM probes, adapters, WiSE-FT, DFR, and Contrastive Adapter baselines.
- Group robustness: Orth-Cali significantly improves robustness across datasets and backbones, achieving comparable or smaller average–worst-group accuracy gaps than Contrastive Adapter without data or labels.
- Ablations: Increasing λ improves worst-group accuracy and stabilizes the solution, while calibration without orthogonal initialization drops on Waterbird.
- Retrieval fairness: FairFace retrieval evaluates MaxSkew@1000 across gender, race, and age, and the approach significantly reduces MaxSkew across attributes and backbones.
6 Debiasing Generative Models
The generative-model extension debiases Stable Diffusion through a universal matrix learned from paired attribute–profession prompts. The resulting calibration balances generated gender and race distributions while applying to unseen professions.
- Objective: The method targets gender and race discrepancies in Stable Diffusion outputs generated from profession-related prompts.
- Design: A universal debiasing matrix preprocesses any prompt embedding before it enters the generative model.
- Training pairs: The calibration matrix is learned with equalization loss over paired prompts combining gender or race attributes with sampled professions.
- Design: The generative extension omits the initial orthogonal projection because it aims to balance, rather than completely eliminate, biased information.
- Results: Applying the calibration matrix improves male–female balance in Stable Diffusion images generated with fixed noise and doctor or firefighter prompts.
7 Experiments: Generative Models
The generative-model experiments evaluate a calibrated projection matrix on Stable Diffusion using automatic and human evaluation. Debiasing reduces gender and racial discrepancy, generalizes to unseen professions, and mitigates a non-social background correlation.
- Evaluation setup: The Stable Diffusion evaluation uses 100 professions split into 80 training and 20 testing professions, with gender and racial attributes assessed.The full experiment generates 100 images for each train/test profession and uses the CLIP classifier to estimate sensitive attributes.
- 7.1 Measuring the Generalization of Calibration: The calibration matrix minimizes projected-embedding differences even for unseen testing professions.Table 6 reports the average L2 difference between projected embeddings for testing prompts.
- 7.2 Quantitative and Qualitative Results: The calibration matrix significantly reduces both gender and race discrepancy for training and testing professions.The reported improvements indicate that the debiasing matrix generalizes beyond the prompts used for calibration.
- 7.2 Quantitative and Qualitative Results: Racial debiasing is more challenging because ethnicity is ambiguous, but calibrated prompt embeddings still improve racial diversity in generated images.Figure 3 provides qualitative evidence for improved racial diversity after calibration.
- 7.2 Quantitative and Qualitative Results: Human evaluation also reports a nontrivial improvement in Stable Diffusion image diversity.Annotators of different genders, races, and nationalities labeled sensitive attributes for generated test-profession images.
- 7.4 Beyond Social Biases: The method generates waterbird images with both land and water backgrounds, whereas the original models generated only water-background images.This experiment tests mitigation of a non-social spurious correlation.
8 Conclusion
The paper concludes that calibrated projection debiases vision-language models through prompt-based text-embedding transformations. It reports effectiveness for generative settings, while acknowledging that some generative classes or biases remain unsupported.
- 8 Conclusion: Calibrated projection mitigates bias in generative vision-language models without additional training or data.The conclusion presents the approach as prompt-based and applicable without retraining.
- 8 Conclusion: The approach demonstrates generation of waterbirds in both land and water backgrounds.Figure 4 contrasts this result with original models that generated only water-background images.
- 8 Conclusion: The generative debiasing technique does not work for certain classes or biases.The paper identifies this as a limitation despite the broader potential impact of debiasing techniques.
B Proof
The proof rewrites the calibration objective and derives its solution using first-order optimality. The formulation uses positive prompt pairs describing the same class with different backgrounds or spurious attributes.
- B Proof: The proof derives the calibration solution by applying first-order optimality criteria.The derivation sets the derivative with respect to the projection matrix to zero.
- B Proof: The loss is rewritten into equivalent objective forms, including a matrix formulation.The text states that the two optima are equivalent and that the second is the matrix form of the first.
- B Proof: Positive prompt pairs describe the same bird type while differing in appended background descriptions.The WaterBird prompts pair landbird or waterbird descriptions with distinct background phrases.
C.2 Human Evaluation
Human evaluation uses a fixed-image comparison and an interface for annotators to label sensitive attributes. The evaluation also tests the importance of class-conditioned prompts.
- C.2 Human Evaluation: Human evaluation generates 100 images per profession and uses the same random seed for original and debiased models.Automatic and human evaluation use the same images for fair comparison.
- C.2 Human Evaluation: The prompt tables use concepts selected to evaluate bias independently of gender, race, or age.The FairFace retrieval prompts use ten training concepts described as irrelevant to those attributes.
- C.2 Human Evaluation: The interface lets annotators select sensitive-attribute labels, confirm choices, and advance to the next image.The interface also supports marking images that are corrupted or do not contain identifiable humans for removal from discrepancy calculations.
- C.2 Human Evaluation: Removing class names from spurious-attribute prompts significantly reduces performance, showing that class-conditioned prompts matter.The comparison replaces class-conditioned prompts with prompts containing only the spurious attribute.
D.2 More Samples from Biased and Debiased Generative Models
Qualitative samples show that debiasing improves gender and racial diversity across training and testing professions, while some strong associations remain difficult to correct. The method also retains failure cases for engineer-related professions such as carpenter.
- The generative-model evaluation uses prompts describing gender and race attributes and a set of 100 GPT-4-generated professions divided into training and testing sets.
- Debiasing improves gender diversity for Stable Diffusion on both training and testing professions.The testing-set distributions become more balanced after applying the calibration matrix.
- Both the biased and debiased models fail to generate females for several engineer-related professions, including carpenter and builder.
- Debiasing improves racial diversity across training and testing professions, including unseen classes.The qualitative results show clearer diversity differences after debiasing, but gains are limited for classes strongly associated with historical figures such as mathematicians.