PaperlayerView exact revision on arXiv

Source-linked AI summary

SEM: Sparse Embedding Modulation for Post-Hoc Debiasing of Vision-Language Models

Quentin Guimard, Federico Bartsch, Simone Caldarella, Rahaf Aljundi, Elisa Ricci, Massimiliano Mancini

arXiv:2603.19028v1cs.CVcs.AIcs.LG

TL;DR

Vision-language models inherit social and spurious biases, while existing post-hoc methods often manipulate entangled dense embeddings and may compromise useful information. SEM uses a Sparse Autoencoder to intervene selectively on sparse CLIP text-embedding features, and across four benchmarks and two backbones it reports substantial fairness gains, including improved worst-group accuracy and complementary gains with BENDVLM.

  • Problem

    Existing post-hoc debiasing methods operate on dense CLIP embeddings where bias and task-relevant information are entangled, limiting precise bias removal.

  • Method

    SEM is a post-hoc, zero-shot framework that decomposes CLIP text embeddings with a Sparse Autoencoder and performs precise, non-linear neuron-level interventions.

  • Results

    Across four benchmarks and two CLIP backbones, SEM consistently improves fairness and worst-group accuracy in retrieval and zero-shot classification.

  • Takeaways & Limitations

    Sparse, feature-level interventions provide a flexible foundation for VLM debiasing and can complement other methods such as BENDVLM.

  • Takeaways & Limitations

    Prior zero-shot projection methods assume a single linear bias direction, while BENDVLM requires a labeled reference image set at test time.

Abstract

from arXiv · show

Models that bridge vision and language, such as CLIP, are key components of multimodal AI, yet their large-scale, uncurated training data introduce severe social and spurious biases. Existing post-hoc debiasing methods often operate directly in the dense CLIP embedding space, where bias and task-relevant information are highly entangled. This entanglement limits their ability to remove bias without degrading semantic fidelity. In this work, we propose Sparse Embedding Modulation (SEM), a post-hoc, zero-shot debiasing framework that operates in a Sparse Autoencoder (SAE) latent space. By decomposing CLIP text embeddings into disentangled features, SEM identifies and modulates bias-relevant neurons while preserving query-relevant ones. This enables more precise, non-linear interventions. Across four benchmark datasets and two CLIP backbones, SEM achieves substantial fairness gains in retrieval and zero-shot classification. Our results demonstrate that sparse latent representations provide an effective foundation for post-hoc debiasing of vision-language models.

1. Introduction

Vision-language models inherit social and spurious biases from uncurated training data, while existing debiasing methods struggle to remove bias without harming useful content. SEM addresses this by intervening on sparse SAE features of CLIP text embeddings, improving fairness across retrieval and zero-shot classification.

  • Motivation: Uncurated training data cause vision-language models to inherit stereotypes and spurious correlations that undermine fairness and reliability.Reported failures include gendered occupational associations, ethnic associations with criminality, and context-driven recognition errors.
  • Limitations of Existing Methods: Retraining-based mitigation is computationally prohibitive, while other post-hoc approaches require complex additional modules, training overhead, or task-specific retraining.The paper therefore focuses on efficient debiasing of frozen-model text embeddings.
  • Limitations of Existing Methods: Zero-shot projection methods model bias with a single linear subspace, leaving complex bias structure entangled with task-relevant content.The paper links this coarse intervention to inconsistent fairness gains and weak worst-group accuracy improvements.
  • SEM: SEM uses a Sparse Autoencoder to decompose CLIP text embeddings into sparse, disentangled features for precise, non-linear neuron-level intervention.The decomposition is intended to isolate concepts more effectively than manipulation in the original dense embedding space.
  • SEM: SEM provides bias-agnostic, bias-aware, and combined variants that adapt to available information without task-specific fine-tuning.The variants are SEMi, SEMb, and SEMbi, using paraphrases, bias prompts, or both.
  • Results: Across two CLIP backbones and four challenging datasets, SEM substantially improves fairness in retrieval and zero-shot classification, especially worst-group accuracy.The evaluation covers social biases involving ethnicity and gender alongside spurious background bias.

2. Related Work

Prior work addresses VLM bias through detection, retraining, projection, and reference-based post-hoc methods, but these approaches face supervision, computational, or representation limitations. SEM instead applies sparse feature-level intervention to final CLIP text embeddings using an SAE.

  • Bias Discovery and Debiasing: Bias-discovery research identifies structured societal biases in models, while debiasing methods intervene during training or after model pretraining.The related work distinguishes detection from intervention and groups mitigation by intervention point.
  • Training-Time Debiasing: Training-time approaches commonly require group-labeled data, model-specific retraining, or other changes to the original training process.These requirements limit their applicability to already pretrained models.
  • Post-Hoc Debiasing: Reference-based post-hoc methods such as BendVLM require a labeled reference image set at test time, imposing an additional practical constraint.This requirement distinguishes them from methods that operate without labeled test-time images.
  • Sparse Feature Decomposition: SEM uses a Sparse Autoencoder to decompose dense CLIP embeddings into sparse features that can be selectively modulated rather than globally projected.The SAE reconstructs dense embeddings from a high-dimensional sparse latent representation, enabling targeted intervention.
  • Sparse Autoencoders for VLMs: Unlike SAE work focused on internal language-model activations, SEM operates on final CLIP text embeddings and uses a Matryoshka SAE for multi-granularity representations.The paper presents this as a post-hoc intervention framework for CLIP embeddings.

3. Sparse Embedding Modulation

SEM addresses entangled bias and semantic content in CLIP embeddings by decomposing text embeddings into sparse SAE features, scoring neurons, and modulating activations before reconstruction. The resulting representation improves disentanglement while targeting bias-sensitive features and preserving content-relevant ones.

  • Motivation: Quantifying Disentanglement: SAEs provide a more disentangled latent space for separating profession features from bias features.Disentanglement improves by 1.7–2.6× for Gender and 5.6–5.7× for Race relative to original CLIP embeddings.
  • Neuron Scoring: SEM scores SAE neurons for content relevance and bias sensitivity using activations from neutral, query, and bias-class prompt sets.Content scoring identifies query-relevant neurons, while structured bias prompts distinguish class-specific bias neurons from general-concept neurons.
  • Sparse Autoencoder Representation: SEM projects CLIP text embeddings into a high-dimensional sparse SAE representation before debiasing.The SAE maps the original embedding into sparse latent activations with dimensionality s ≫ d, then reconstructs the modified embedding.
  • Steering via Activation Modulation: The SEMi variant uses augmented content scores, and removing its attenuation causes a severe drop in worst-group accuracy.The full SEMb and SEMbi variants further combine content and bias scores to preserve essential entangled content features.
  • Steering via Activation Modulation: SEM combines neuron scores into modulation coefficients that attenuate bias-sensitive features and boost content-relevant features.The modulated latent vector is decoded into a new debiased text embedding while the pretrained encoders remain frozen.

4. Experiments

Experiments evaluate SEM across two CLIP backbones, four benchmarks, retrieval, and zero-shot classification. SEM improves fairness and worst-group robustness, preserves semantic content, and complements existing debiasing methods.

  • Experimental Setup: Evaluation covers two pretrained CLIP backbones, two tasks, and four standard benchmarks, with retrieval and classification fairness metrics.Retrieval uses KL Divergence@500, MaxSkew@500, and Precision@500; classification uses Accuracy, Worst-Group Accuracy, and Gap.
  • Entanglement Study: 0.878 content similarity and 0.974 gender-pair similarity show SEMb preserves profession semantics while neutralizing gender differences.ORTH-PROJ reaches 0.415 similarity to the neutral concept and 0.916 for gendered pairs, compared with 0.956 for the original baseline.
  • Retrieval: 0.170 FairFace Race KL divergence is achieved by SEMi on ViT-B/16, improving over BASE CLIP at 0.237 while ROBOSHOT worsens it to 0.327.SEMb and SEMbi also outperform or match other zero-shot methods on 12 of 16 social-bias metrics.
  • Modularity: 24 out of 28 metrics favor BENDSEMbi over BENDVLM alone, including Waterbirds worst-group accuracy rising from 0.416 to 0.745.On UTKFace Race, KL divergence falls from 0.087 to 0.043, a 50.6% reduction.
  • Ablation Study: Removing the SAE or content-boosting term reduces performance, while the full SEM formulations provide the most robust and balanced ablation results.Without content boosting, Waterbirds worst-group accuracy drops from 0.624 to 0.445; dense “median CLIP” also produces a larger CelebA decline.

5. Conclusion

The paper introduces SEM as a flexible, post-hoc, zero-shot framework for mitigating VLM bias through sparse, feature-level interventions. Across four benchmarks, SEM improves fairness and worst-group accuracy and can further enhance BENDVLM.

  • Conclusion: SEM is a post-hoc, zero-shot framework that uses a Sparse Autoencoder to enable precise, non-linear interventions in a disentangled latent space.Three variants adapt to bias-agnostic and bias-aware settings.
  • Conclusion: Across four benchmarks, SEM consistently improves fairness and worst-group accuracy while addressing a key failure of prior methods.The conclusion presents these improvements as evidence for sparse, feature-level debiasing.
  • Conclusion: Combining SEM with BENDVLM further improves results, demonstrating that SEM can function as a modular complementary framework.The paper uses this combination to highlight the benefits of sparse, feature-level debiasing.

A. SAE Training Details

The appendix details MSAE training, probing data, and evaluation procedures for the SAE representations. It specifies the architecture, optimization setup, controlled profession prompts, and sequential probes used to study entanglement.

  • SAE Architecture: Each CLIP backbone receives a separate Matryoshka Sparse Autoencoder with total latent dimensionality 16384.The MSAE minimizes reconstruction error at nested granularities g ∈ {256, 512} and applies Reverse Weighting.
  • Probing Dataset: The probing dataset combines gender, seven race classes, 100 professions, and 20 diverse prompt templates.The profession list and templates are provided in Tables 5 and 6.
  • Probing Methodology: Sequential probing measures profession prediction, directly extractable bias information, and bias prediction from profession logits.Logistic Regression probes use five-fold stratified cross-validation with standardized features.
  • Probing Assumption: The probing setup assumes the classifier should predict without relying on gender-related information.This assumption defines the intended separation between task-relevant semantics and bias information.

B.3. Full Results

The full-results appendix reports a controlled profession-prompt study using female, male, and neutral variants. The 300-prompt construction isolates the effect of gender on profession semantics.

  • Full Results: Both CLIP and SAE representations achieve near-perfect profession prediction, with accp > 0.99.The same setup also measures directly extractable bias information through accb.
  • Full Results: SAE latents yield significantly lower sequential bias-prediction accuracy accb←p than dense CLIP embeddings.This gap corresponds to the higher Disentanglement Score reported in the main paper.
  • Dataset Construction: The probing dataset contains 300 prompts formed from 100 professions, each paired with female, male, and neutral variants.The controlled design isolates the gender attribute’s effect on profession semantics.

C.2. Methodology

The methodology evaluates embedding preservation and bias neutralization using prompt sets, PCA visualizations, and cosine-similarity metrics across multiple debiasing variants.

  • Embedding evaluation: The evaluation compares BASE CLIP, ORTH-PROJ, and SEMb embeddings using 300 prompts with ViT-L/14@336px.PCA is applied independently to each embedding set to visualize male, female, and neutral profession clusters.
  • Metrics: Content Preservation measures cosine similarity between debiased gendered embeddings and the original neutral profession embedding.Values close to BASE CLIP indicate preserved profession semantics, while substantial drops indicate concept corruption.
  • Metrics: Bias Neutralization measures cosine similarity between male and female embeddings for the same profession, with higher similarity indicating reduced gender separation.The intended trade-off is high Bias Neutralization alongside high Content Preservation.
  • Prompt construction: Bias prompts contain 20 generated captions per bias class, while diverse prompts contain 328 neutral captions spanning varied semantic concepts.The diverse prompts establish a broad baseline activation distribution for identifying bias-related neurons.
  • Prompt construction: Augmented query prompts use 10 LLM-generated paraphrases per retrieval query or zero-shot class label, with median activations producing stable query representations.This augmentation is intended to improve semantic diversity and reduce sensitivity to wording.

E. Extended Retrieval Results

Extended CelebA retrieval results show that SEM variants preserve retrieval precision while reducing bias, and that combining SEM with a baseline improves semantic consistency.

  • Fairness–precision trade-off: SEMi surpasses ROBOSHOT in Hair Color precision on ViT-B/16, achieving 0.679 versus 0.632.The comparison illustrates SEM’s precision advantage in the bias-agnostic setting.
  • Fairness–precision trade-off: SEMb outperforms ORTH-PROJ in Hair Color precision on ViT-B/16, achieving 0.729 versus 0.660.The result supports stronger preservation of query semantics than projection-based debiasing in this setting.
  • Modularity improves semantic consistency: BENDSEMbi boosts retrieval precision by 9.5%, from 0.656 to 0.751, while achieving fairness scores comparable to BENDVLM alone.This result occurs in the Bias prompts + input-specific prompts + labeled images setting on ViT-B/16.

F. Extended Ablation Study

The extended ablations support SEM’s full design: relevance-based attenuation and content boosting are necessary for robust fairness and semantic performance across tasks and backbones.

  • Overall ablations: Full SEMi and SEMb provide the most robust performance across retrieval and zero-shot classification benchmarks.The ablations cover FairFace, UTKFace, CelebA, and Waterbirds across multiple backbones.
  • SEMi ablation: Removing relevance-based attenuation collapses Waterbirds ViT-B/16 worst-group accuracy from 0.498 to 0.210.The result highlights the importance of modulating spurious features in SEMi.
  • SEMb ablation: Removing content boosting reduces Waterbirds ViT-B/16 worst-group accuracy to 0.081 and lowers CelebA ViT-B/16 precision from 0.729 to 0.641.On ViT-L/14, CelebA precision also drops from 0.699 to 0.545.
  • SEMb ablation: Using only general or specific bias scores produces inconsistent fairness or semantic-consistency results across datasets and settings.The full combination is therefore more reliable than either score alone.

G. Extended Results on ResNet Backbones

On ResNet-50 and ResNet-101 CLIP backbones, SEM retains strong fairness improvements, improves zero-shot robustness, and remains complementary to existing debiasing methods.

  • Retrieval: SEMi lowers FairFace Race KL divergence to 0.126 on ResNet-50, compared with 0.215 for BASE CLIP and ROBOSHOT.SEMb and SEMbi achieve the best fairness metrics in bias-aware settings across nearly all evaluated demographics and datasets.
  • Zero-shot classification: SEMb raises Waterbirds ResNet-50 worst-group accuracy from 0.394 with BASE CLIP to 0.577, a gain of 18.3 points.It also outperforms ROBOSHOT at 0.458 and ORTH-PROJ at 0.457.
  • Fairness–precision trade-off: SEMi reduces Stereotype KL to 0.050 on ResNet-50 but records Hair Color precision of 0.508.Bias-aware SEMb and SEMbi better navigate this fairness–precision trade-off.
  • Modularity: BENDSEMbi raises Waterbirds worst-group accuracy to 0.638 on ResNet-101, compared with 0.194 for BENDVLM alone.The combined method also provides the lowest social-bias metrics across nearly all retrieval benchmarks.
Loading 2603.19028v1…