Source-linked AI summary
Mitigating Bias in Large Vision-Language Models via Counterfactual Ensemble Decoding
Yisong Xiao, Aishan Liu, Yongxin Huang, Zonghao Ying, Shiji Zhao, Tianlin Li, Yong Han, Jian Yang, Xianglong Liu
TL;DR
LVLMs can inherit social biases from training data, while existing decoding-based methods rely on a single stereotyped perspective. CED constructs multi-group counterfactual visual representations and ensembles their token distributions during decoding, reducing bias across three benchmarks while preserving core capabilities with minimal degradation.
Problem
Existing LVLM debiasing methods are limited by costly training requirements or reliance on a single stereotyped perspective that omits diverse social viewpoints.
Method
CED learns social-group directions, steers visual representations to create counterfactual perspectives, and ensembles their token distributions at the most divergent decoder layer.
Results
47.97% average reduction in race-related stereotypical bias on ModSCAN, with consistent bias-mitigation improvements across three benchmarks and minimal degradation of core LVLM capabilities.
Takeaways & Limitations
CED provides an inference-stage approach that combines diverse social-group perspectives to promote fairer LVLM outputs while retaining general model capabilities.
Takeaways & Limitations
CED requires white-box access to the full LVLM, and current evaluation focuses on gender and race because resources for other protected attributes remain limited.
Abstract
from arXiv · showhide
Large Vision-Language Models (LVLMs) have achieved remarkable performance across a wide range of tasks; however, they often inherit social biases from their training data, resulting in biased behavior when processing portraits from different social groups. Existing debiasing approaches typically compare token probabilities between the original and biased generations during decoding, but they are fundamentally limited by their reliance on a single, stereotyped viewpoint and fail to account for the diversity of social perspectives. Inspired by the social science principle that diversity fosters fairness, we propose Counterfactual Ensemble Decoding (CED), a novel framework that constructs multi-group counterfactual perspectives within the visual representation space and integrates them during decoding to promote equitable model behavior. CED first performs counterfactual steering in the visual space by identifying semantic directions associated with each social group and generating counterfactual representations along these directions, thereby offering diverse perspectives that disrupt stereotypical narratives. During decoding, CED locates the decoder layer exhibiting the greatest divergence among these perspectives and ensembles their token distributions using uncertainty-aware weights, prioritizing high-confidence tokens from different groups to yield a more balanced probability distribution that guides fairer generation. Extensive experiments on three social bias evaluation benchmarks demonstrate that \tool achieves substantial improvements over leading baselines, reducing bias by up to 47.97% across scenarios involving occupations, descriptors, and persona traits. Moreover, CED also preserves the core capabilities of the original model with minimal degradation.
I. INTRODUCTION
LVLMs can inherit harmful social stereotypes, while existing mitigation methods are limited by resource demands or reliance on a single stereotyped perspective. CED addresses this gap by combining multi-group counterfactual perspectives during inference and improves bias mitigation while preserving core capabilities.
- Motivation: LVLMs may inherit gender and race biases from training data, producing harmful stereotype-driven outcomes for specific social groups.Reported examples include associations between females and particular emotions, occupations, and sexualized content.
- Prior limitations: Training-stage debiasing can require costly data collection and substantial computation, limiting practical viability.Inference-stage methods improve efficiency but often compare token probabilities from a single stereotypical perspective.
- CED: CED constructs multi-group counterfactual perspectives in visual representation space and integrates them during decoding to promote fairer token distributions.It uses counterfactual steering to generate diverse perspectives and ensembles distributions at the most conflicting decoder layer.
- Results: 61.21% average reduction in occupation-related gender bias is achieved on GenderBias-VL, alongside 47.97% average reduction in race-related stereotypical bias on ModSCAN.On VisBias, CED reduces stereotypical word frequency difference by 38.22% and sentiment difference by 65.75%.
- Results: CED significantly outperforms leading baselines in bias mitigation while preserving LVLM core capabilities with minimal performance degradation.The reported evaluation covers three bias benchmarks and widely used LVLMs.
III. PRELIMINARIES
An LVLM combines visual representations with a textual query and autoregressively predicts tokens through a multi-layer language model. Token probabilities are ultimately derived from the final decoder-layer hidden state.
- LVLM decoding: An LVLM encodes an input image into visual representations, concatenates them with a tokenized query, and feeds them into an LLM for autoregressive generation.The framework includes a vision encoder, a vision-language alignment interface, and an L_text-layer LLM.
- LVLM decoding: At generation step t, the model predicts the next token y_t conditioned on the visual representation, query, and previously generated tokens.The token probability is represented as p(y_t | v, x, y<t).
- LVLM decoding: Layer-wise hidden states are produced across the text decoder, while next-token logits are obtained by applying the vocabulary projection head to the final-layer hidden state.The final layer is indexed by L_text and the projection head is denoted ϕ(·).
B. Problem Definition
CED treats biased LVLM behavior as skewed token probabilities triggered by protected attributes and constructs counterfactual group perspectives in visual representation space. It learns group directions, steers representations without changing images, and uses these perspectives to support fairer decoding.
- Problem Definition: Protected attributes depicted in images can trigger stereotype-driven inferences that assign disproportionately high probabilities to stereotype-reinforcing tokens.The goal is to mitigate these stereotypes and improve fairness of LVLM outputs.
- Motivation: CED is motivated by integrating diverse social perspectives to disrupt harmful stereotypes and promote more equitable token distributions.The framework constructs multi-group counterfactual perspectives for integration during decoding.
- Counterfactual Steering: CED learns semantic directions associated with social groups using layer-wise linear classifiers, including one-vs-one directions for attributes with multiple groups.Classifier parameters serve as directions for subsequent visual-representation editing.
- Counterfactual Steering: During inference, CED steers the original visual representation toward counterfactual social-group representations rather than generating real counterfactual images.This preserves other contextual elements such as the image background while introducing alternative group perspectives.
- Counterfactual Steering: The resulting counterfactual representations provide diverse perspectives for the subsequent decoding process.These representations function as fairness resources while retaining the original image’s contextual elements.
C. Ensemble Decoding
CED ensembles token distributions from original and counterfactual representations at the decoder layer where their perspectives diverge most. This produces a more balanced distribution intended to foster fairer generation.
- Ensemble Decoding: CED identifies the decoder layer with the greatest divergence among original and counterfactual perspectives before ensembling their token distributions.The selected layer is intended to capture the strongest conflict among perspectives.
- Ensemble Decoding: The layer-specific ensemble integrates distributions from multiple perspectives to yield a more balanced probability distribution.The balanced distribution guides the model toward fairer generation.
1) Bias Layer Identification:
CED identifies the decoder layer with the greatest divergence among original and counterfactual perspectives by comparing their token distributions, then uses that layer for current-token integration.
- Bias Layer Identification:: CED projects decoder representations into vocabulary distributions and compares original and counterfactual perspectives across candidate middle and deeper layers.The candidate set C consists of pre-defined middle and deeper LLM decoder layers.
- Bias Layer Identification:: Jensen-Shannon Divergence measures the symmetric, bounded difference between original and counterfactual token probability distributions.The divergence is computed at each time step to quantify perspective conflict.
- Bias Layer Identification:: The layer with the highest divergence is selected as the most biased layer for the current token-generation step.For more than two social groups, CED averages the divergence between the original representation and each counterfactual representation.
2) Prediction Ensemble:
At the selected layer, CED combines next-token distributions from original and counterfactual perspectives using uncertainty-aware weights, while restricting outputs to plausible high-confidence tokens.
- Prediction Ensemble:: CED computes entropy-based uncertainty for each perspective’s next-token distribution at the selected layer.Lower uncertainty corresponds to more perspective-rich tokens.
- Prediction Ensemble:: Higher weights are assigned to lower-uncertainty tokens through exponential normalization controlled by the sensitivity hyperparameter γ.The normalized weights preserve representative high-probability tokens from different social-group perspectives.
- Prediction Ensemble:: The weighted ensemble of projected perspective distributions directly guides next-token generation through early exiting at the selected layer.For multiple social groups, uncertainties are normalized into group-specific adaptive weights before ensembling.
- Prediction Ensemble:: An adaptive plausibility constraint assigns zero probability to tokens outside the original distribution’s high-confidence candidate set.The truncation strength is controlled by β in the range [0, 1].
1) Datasets and Evaluation Metrics:
CED is evaluated across three social-bias benchmarks and multiple LVLMs using bias, accuracy, and fairness-oriented metrics, with results showing strong debiasing and limited task-performance interference.
- Datasets and Evaluation Metrics:: CED is evaluated on GenderBias-VL, ModSCAN, and VisBias, covering multiple-choice and open-ended visual question answering tasks.The benchmarks assess occupation-related gender bias, racial stereotypes, and gender bias in image descriptions.
- Datasets and Evaluation Metrics:: The evaluation uses accuracy and fairness-oriented metrics, including Bpair, Bovl, Ipss, Sbias, stereotypical word-frequency difference, and sentiment difference.Higher Acc and Ipss indicate better performance, whereas higher Bpair, Bovl, and Sbias indicate greater bias.
- Results on GenderBias-VL: CED reduces GenderBias-VL overall bias Bovl by 61.21% on average relative to the original models and by 25.58% compared with the second-best baseline.On InternVL2, CED lowers Bovl from 16.21% to 9.12%, while explanation lowers it to 14.20%.
- Results on GenderBias-VL: CED preserves accuracy well, improving it by 0.36% on LLaVA-1.5 and 2.72% on InternVL2 while dropping from 77.65% to 75.99% on Qwen3-VL.The explanation baseline incurs a 7.78% accuracy drop.
- Results on GenderBias-VL: CED improves Ipss by 7.46% on average and achieves the lowest Bpair on all 10 occupation pairs for InternVL2.It achieves the lowest Bpair on 9 of 10 pairs for both LLaVA-1.5 and Qwen3-VL.
C. Results on ModSCAN
On ModSCAN, CED reduces racial stereotypical bias across occupation, descriptor, and persona trait scenarios, with especially balanced associations in the descriptor example.
- Overall results: 47.97% average reduction in race-related stereotypical bias across ModSCAN occupation, descriptor, and persona trait scenarios.This was the largest reduction among the evaluated methods, while explanation and SelfDebias achieved 16.98% and 9.95%.
- Comparison with baselines: CED outperforms explanation, SelfDebias, and reprompt across the reported ModSCAN comparisons.The explanation method reduces bias by 4.27% in LLaVA-1.5’s occupation scenario, 23.04% in the descriptor scenario, and 31.61% in persona traits; reprompt averages 1.50%.
- Overall results: CED consistently mitigates racial bias across occupation, descriptor, and persona trait scenarios on ModSCAN.Figure 4 visualizes the distribution of LLaVA-1.5’s Sbias across these three scenario types.
- Descriptor scenario: In the descriptor scenario, CED reduces the terrorist-association Sbias from 9.62% to 2.07%.The association probabilities become Black 28.70%, White 25.43%, Asian 21.04%, and Indian 24.83%, compared with the original Black 39.20% and White 12.19%.
D. Results on VisBias
On VisBias, CED achieves the lowest reported stereotype and sentiment bias scores while preserving general LVLM capabilities with limited degradation.
- Bias results: 38.22% reduction in stereotype word bias and 65.75% reduction in sentiment bias compared with the original models on VisBias.CED also outperforms SelfDebias by 25.93% on stereotype word frequency difference and achieves an average sentiment score of 0.67% versus 1.93% originally.
- Bias results: CED reduces stereotypical word frequency difference and sentiment difference on the open-ended VisBias benchmark.The explanation method is not applicable because VisBias has no predefined answer options.
- Qualitative example: For a female CEO image, CED replaces stereotypical terms such as attractive with the more neutral term professional.It also reduces the positive sentiment score from 12.4% to 9.4%.
- Capability preservation: 0.93% average MMBench accuracy drop across LVLMs indicates limited impact on common reasoning abilities.On MMMU, CED achieves the highest accuracy among the evaluated debiasing methods and improves LLaVA-1.5 and Qwen3-VL accuracy by 3.33% and 1.33%.
VI. DISCUSSION
Additional analyses show that CED’s debiasing depends on appropriately calibrated counterfactual manipulation and conflict-layer selection, while remaining more efficient than several effective baselines. The method also consistently outperforms baselines but requires full LVLM access and is currently evaluated mainly on gender and race.
- Counterfactual Manipulation Magnitude: Increasing α from 1 to 3 reduces Bovl by 2.63%, but increasing it to 9 raises bias by 2.75%.Small perturbations may be insufficient for strong counterfactual perspectives, whereas excessive perturbations can make the model skew toward the introduced social group.
- Ensemble Sensitivity Factor: Bovl remains stable across γ values from 0.25 to 4.0, ranging from 7.86% to 7.99% with a standard deviation of 0.05%.The authors attribute this stability to dynamic adjustment of token uncertainty.
- Distribution of Biased Layers: Layers 20, 26, and 32 are selected most frequently as biased layers, accounting for 26.89%, 21.64%, and 19.12%, respectively.Bias is concentrated in a few layers rather than distributed uniformly.
- Distribution of Biased Layers: Ensembling at the final layer yields a Bovl of 11.09%, compared with 7.90% for CED with bias-layer identification.The comparison emphasizes the importance of locating the most conflicting layer.
- Efficiency of CED: CED requires 7.80 seconds to generate 128 tokens, faster than SelfDebias at 12.09 seconds and reprompt at 10.20 seconds.CED also achieves lower average stereotype and sentiment bias scores than faster GenProj, at 6.19% and 0.67% versus 10.02% and 1.51%.
- Conclusion and Limitations: CED consistently outperforms baseline methods in bias mitigation while preserving the original model’s general capabilities with minimal degradation.Its current evaluation focuses on gender and race, and the method requires full access to the LVLM.