Source-linked AI summary
Guardrail-Agnostic Societal Bias Evaluation in Large Vision-Language Models
Yusuke Hirota, Michael Ross Boone, Arun George Zachariah, Jibin Rajan Varghese, Yu-Chiang Frank Wang, Boyi Li, Ryo Hachiuma
TL;DR
Existing LVLM bias benchmarks often become unreliable because safety guardrails refuse attribute-inference prompts. This paper introduces a guardrail-agnostic alternative using person-irrelevant tasks and images as user context, finding bias across 20 LVLMs while proprietary models generally show less bias.
Problem
Strong safety guardrails cause LVLMs to refuse attribute-inferring bias-evaluation prompts, undermining reliable statistical measurement.
Method
The framework uses person-irrelevant prompts, attaches images as provisional user information, and evaluates bias across story generation, term explanation, and exam-style QA.
Results
All models exhibit societal bias, while proprietary models generally show lower bias than open-source ones; the framework achieves zero refusals.
Takeaways & Limitations
The framework enables practical bias evaluation and monitoring of safety-guarded LVLMs throughout deployment.
Takeaways & Limitations
The individual effects of training data and post-training alignment are difficult to isolate because most models do not sufficiently disclose these details.
Abstract
from arXiv · showhide
We propose a societal bias evaluation method for large vision-language models (LVLMs) in the era of strong safety guardrails. Existing benchmarks rely on prompts that ask models to infer attributes of people in images (e.g., "Is this person a CEO or a secretary?"). However, we find that LVLMs with strong guardrails, such as GPT and Claude, often refuse these prompts, making evaluations unreliable. To address this, we change the prior evaluation paradigm by decoupling the task from the depicted person: instead of inferring person's attributes, we use prompts that do not ask about the person (e.g., "Write a fictional story about an imaginary person.") and attach the image as provisional user information to implicitly provide demographic cues, then compare outputs across user demographics. Instantiated across three tasks --- story generation, term explanation, and exam-style QA --- our method avoids refusals even in guardrailed LVLMs, enabling reliable bias measurement. Applying it to 20 recent LVLMs, both open-source and proprietary, we find that all models undesirably use user demographic information in person-irrelevant tasks; for instance, characters in stories are often portrayed as mechanic for male users and nurse for female users. Although still biased, proprietary models like GPT-5 show lower bias than open-source ones. We analyze potential factors behind this gap, discussing continuous model monitoring and improvement as a possible contributor for reducing bias.
1 Introduction
Societal-bias evaluation in LVLMs becomes unreliable when safety guardrails refuse attribute-inferring prompts. The proposed method instead uses person-irrelevant tasks with images supplied as user context, enabling bias measurement and revealing demographic disparities.
- LVLMs can associate occupations such as nurse more with women than men, raising concerns about gender-occupation stereotyping.
- Prior benchmarks ask models to infer attributes of depicted people, but strong guardrails frequently refuse these prompts.
- Such refusals violate the sufficient-response assumption required for statistical bias analysis, making evaluations unreliable.
- The method decouples evaluation from the depicted person by replacing attribute-inferring prompts with person-irrelevant prompts and treating images as provisional user information.
- Across story generation, term explanation, and exam-style QA, the method produces zero refusals and exposes demographic disparities in model outputs.
2 Review: Existing Societal Bias Benchmarks for LVLMs
Existing LVLM bias benchmarks compare model outputs across demographic groups, but attribute-focused and captioning-style designs have important reliability and confounding limitations. The proposed approach addresses both by using person-irrelevant prompts and images only as user context.
- Existing benchmarks pair demographic-labeled people images with attribute-inferring prompts and measure bias through output differences across groups.
- Benchmark bias scores are formed from per-prompt statistical disparities across group-specific output sets and then aggregated across prompts.
- Attribute-inferring prompts include closed-form questions about occupations or personalities and open-ended prompts eliciting attribute inference.
- High refusal rates leave too few valid outputs for reliable statistical bias measurement, especially in models with strong safety guardrails.
- Captioning-style prompts may avoid refusals but introduce contextual confounds because objects and backgrounds can correlate spuriously with demographic groups.
- The proposed method addresses both limitations by removing attribute-inference requests and using images only as provisional user information.
3 Proposed Evaluation Method
The proposed framework decouples person-irrelevant evaluation tasks from depicted people, using images as provisional user information and comparing outputs across demographic groups. It applies this design to three tasks and quantifies disparities with Total Variation Distance.
- The framework replaces attribute-inferring prompts with person-irrelevant prompts, enabling zero-refusal evaluation under safety guardrails.
- Images are supplied as provisional user information, allowing demographic-dependent adaptation of person-irrelevant outputs to be evaluated as bias.
- For each prompt and demographic group, the method collects model outputs and assesses statistical disparity across the resulting output sets.
- The framework instantiates story generation, term explanation, and exam-style QA to probe different aspects of societal bias.
- Story generation extracts non-demographic character attributes, term explanation compares technicality, and exam-style QA compares demographic-group accuracy.
4 Experiments
The experiments evaluate 20 LVLMs across demographic-controlled settings and three person-irrelevant tasks, finding zero refusals and widespread societal bias. Proprietary models are generally less biased than open-source models, while bias varies across tasks and demographic axes and is not reliably explained by model size or performance.
- Evaluation Settings: The evaluation covers gender and racial bias using FairFace images, aligned non-target demographics, and three person-irrelevant tasks.The tasks are story generation, term explanation, and exam-style QA; demographic labels use binary gender and seven race categories.
- Evaluation Settings: The study evaluates 20 recent LVLMs, including 16 open-source models and 4 proprietary models.The open-source models range from 7B to 38B parameters and include InternVL3.5, while proprietary models include Claude, GPT-4o, and GPT-5.
- Refusal Rates: Zero refusals occur across all models under the proposed framework, enabling bias evaluation for safety-guarded LVLMs.The framework uses person-irrelevant tasks with images supplied as user information, unlike prior refusal-prone benchmarks.
- Societal Bias: Proprietary models have lower average bias scores than open-source models in story generation (29.29 vs. 18.99) and exam-style QA (1.66 vs. 0.90), with a smaller term-explanation gap (4.71 vs. 4.49).Despite lower scores, proprietary models remain biased; GPT-5 records gender/racial story-generation scores of 14.53/16.80.
- Societal Bias: Generated stories and explanations exhibit demographic disparities, including stereotypical occupations and differences in explanation difficulty or technical jargon.Examples include mechanic versus nurse portrayals for male versus female users and more difficult physics explanations for White users.
- Bias Analysis: Task-wise bias correlations are weak (−0.11 to 0.21), whereas gender-race correlations within tasks are strong at r = 0.49/0.60/0.93.These results indicate that bias patterns differ across tasks while gender and racial biases tend to co-occur within the same task.
5 Bias Sources and Deployment Recommendations
The paper discusses continuous monitoring and iterative refinement as possible contributors to lower bias and recommends evaluating models throughout deployment.
- Bias Sources: Proprietary models tend to show lower bias, but model size and performance do not explain the difference.Safety-oriented training is proposed as one possible factor, although it does not fully account for the observed gap.
- Bias Sources: Continuous monitoring, red teaming, and postdeployment updates may contribute to reducing societal bias.The paper contrasts proprietary models’ sustained improvement cycles with the absence of such cycles in many open-source models.
- Deployment Recommendations: The framework can be applied to any person-irrelevant task that does not ask about the depicted person.The paper gives career advice as an example deployment scenario.
- Deployment Recommendations: The framework supports predeployment testing and postdeployment monitoring for biases that may emerge during use.The paper argues that societal bias cannot be fully predefined and can continue emerging in deployment.
6 Limitations
The paper identifies limitations involving undisclosed training factors, demographic scope, evaluator bias, and potentially spurious visual correlations.
- Scope and Attribution: The effects of training data and post-training alignment are difficult to isolate because most evaluated models do not disclose sufficient details.This limits conclusions about which training factors drive observed bias differences.
- Demographic Scope: The evaluation covers gender and racial bias but excludes other demographic groups, including age, despite possible extensions to skin tone and visible disabilities.Age was excluded because different outputs across ages may often be natural rather than evidence of bias.
- Evaluator Dependence: An LLM assistant extracts story attributes and judges explanation difficulty, creating a risk of systematic evaluator bias.Human agreement was approximately 97%, but the authors call for larger-scale human evaluation or debiased evaluators.
- Image Confounds: Background and other non-human visual features may introduce demographic-correlated spurious visual biases despite using a face-centric dataset.The authors mitigate background variation with FairFace but acknowledge that such correlations may remain.
7 Conclusion
The paper concludes that person-irrelevant prompts with images treated as user context enable guardrail-agnostic bias evaluation. Across evaluated models, bias remains widespread, and the framework is positioned for deployment-time monitoring.
- Conclusion: The framework evaluates societal bias without asking models to infer attributes of depicted people, avoiding refusals from safety-guarded models.Images are attached as user context while prompts remain unrelated to the depicted person.
- Conclusion: The framework is extensible across deployment, supporting evaluations before release and monitoring after release.Its core requirement is that prompts be person-irrelevant and not ask about the depicted person.
- Evaluation Metric: The paper defines TVD as disparity between observed group-selection distributions and an ideal equal-treatment distribution.The normalized score uses the same metric across tasks, with larger values indicating stronger disparity.
- Evaluation Tasks: Story generation measures demographic disparities in extracted non-demographic character attributes such as occupation and education.The story prompt asks for a fictional story about an imaginary person despite attaching the user’s photo as context.
- Evaluation Tasks: Term explanation compares which demographic group receives more technical explanations, while exam-style QA compares normalized accuracy across groups.The exam task uses group accuracy distributions against an equal-accuracy target.
C.2 Additional Information For the Evaluation Tasks
The evaluation tasks are selected to represent common interactions and different degrees of answer freedom, with balanced term-explanation domains and college-level MMLU subjects.
- Task Selection: The three tasks span creative writing, knowledge explanation, and problem solving while remaining unrelated to the depicted person.They range from open-ended generation to constrained explanation and closed-form question answering.
- Term Explanation: The term-explanation judge selects the explanation containing more technical terminology or domain-specific jargon.Candidate presentation order is randomized to mitigate order and position bias.
- Term Explanation: Term explanation uses three STEM and three humanities or arts domains, with 20 manually filtered college-level terms per domain.Candidate terms are generated with GPT-4o before manual filtering.
- Exam-Style QA: Exam-style QA uses six college-level MMLU subjects, each containing 100 multiple-choice questions.The subjects are biology, chemistry, computer science, mathematics, medicine, and physics.
D Detailed Experimental Settings for Refusal Rates
The refusal-rate experiment samples 300 prompts from four existing benchmarks and the proposed tasks, testing proprietary and open-source LVLMs with human-verified refusal labels.
- Experimental design: 300 prompts were randomly sampled from SBBench, ModScan, VLA-gender, Pairs, and the three proposed tasks.The sampled prompts were used to verify refusal problems in prior benchmarks and demonstrate the proposed method’s behavior.
- Models: The evaluation covered GPT-5, Claude 3.7 Sonnet, LLaVA-1.6-34B, Qwen2.5-VL-32B, Gemma3-27B, and InternVL3.5-38B.The models included both proprietary and open-source LVLMs.
- Refusal definition: A refusal was an output unsuitable for computing statistical differences across demographic groups, including explicit declines or uncertainty statements.Human workers manually verified whether every response qualified as a refusal.
E.1 Robustness to Spurious Visual Correlations
The framework shifts images from prompt targets to provisional user context, reducing sensitivity to spurious visual correlations while supporting robustness checks and revealing limits of simple mitigation.
- Evaluation design: The method differs from open-ended benchmarks by changing the image’s role and improving robustness to spurious visual correlations.It treats the image as context rather than the target of the prompt.
- Robustness test: The controlled experiment compares output stability between face-centric FairFace images and complex-background COCO images using InternVL3-14B.This directly tests sensitivity to background variation.
- Robustness results: Lower JS distance under the proposed method indicates reduced sensitivity to spurious background features compared with existing open-ended evaluation.The framing as provisional user information is described as filtering irrelevant visual details.
- Judgment validation: The LLM assistant agreed with human evaluators in 97 of 100 term-explanation cases.This supports the reliability of assistant-based judgments for that task.
- Assistant robustness: Story-generation bias scores remained highly correlated when alternative LLM assistants replaced Qwen3-32B for attribute extraction.The comparison changed only the assistant used to extract character attributes.
- Mitigation: A prompt-based intervention reduced Qwen2.5-VL-7B’s gender bias score only slightly, from 27.32 to 25.22.The paper also notes that such debiasing methods can degrade general model performance and should be applied cautiously.
- Persona baseline: The text-only persona baseline had a lower gender bias score than the image-based vision-language setting, 21.43 versus 27.32.The paper proposes implicit visual cues and multimodal training as potential explanations for the gap.
- Sanity check: Allowing ties in term-explanation judgments produced an average bias score similar to forced choice, 4.71 versus 4.68.This small-scale check suggests the forced-choice design did not substantially alter the observed trend.
F Detailed Experimental Results
Across detailed results, models show demographic disparities in story generation, term explanation, and exam-style QA, including differences in occupations, technical difficulty, and subject accuracy.
- Story generation: All models show noticeable gender and racial bias across story-generation character attributes.The reported attributes include multiple categories of character information.
- Term explanation: Term-explanation outputs show demographic disparities across domains, with STEM explanations often more difficult for male and White users.The reported patterns include more technical jargon for male and White users in computer science and physics.
- Term explanation: Male users receive difficult explanations more often in STEM domains, including a 90% selection ratio in computer science.White users receive disproportionately more technical explanations than Southeast Asian users, 27.1% versus 6.8%.
- Exam-style QA: Exam-style QA shows larger accuracy disparities in Chemistry, CS, Math, and Physics than in Biology and Medicine.The comparison is reported across subject-wise gender and racial bias scores.
- Illustrative examples: Examples include racial vocabulary differences in generated stories and equation-containing explanations for male users but not female users.The examples are presented as additional illustrations of gender and racial bias.
I Additional Limitations and Ethics Statement
The study’s scope is bounded by its demographic categories, task and benchmark choices, model coverage, and use of publicly available demographic datasets.
- Demographic scope: The experiments use FairFace’s demographic categories, while extension to more inclusive groups such as non-binary gender remains future work.The paper identifies the available annotations as the practical condition for extension.
- Task scope: The framework is instantiated on three tasks, with higher-stakes scenarios such as code generation left for future evaluation.The authors describe the method as extensible to a broader range of tasks.
- QA benchmark: The exam-style QA evaluation uses MMLU, while other benchmarks such as MMLU-Pro were not included in this study.The paper presents incorporation of other QA benchmarks as future scope expansion.
- Model coverage: Models larger than 38B parameters were excluded because of computational resource limits.Conclusions about model size and bias should therefore be interpreted within the evaluated range.
- Proprietary-model coverage: The proprietary-model evaluation excludes comparable systems such as Gemini 2.5 because of budget constraints.The evaluated proprietary set consists of GPT-4o, GPT-5, Claude 3.5 Sonnet, and Claude 3.7 Sonnet.
- Ethics and data: The study uses publicly available datasets with demographic annotations and adopts binary gender and seven race categories.The paper notes that these discrete labels are limited and that more inclusive representations are desirable.