VIBE: Voice-Induced open-ended Bias Evaluation for Large Audio-Language Models via Real-World Speech

Yi-Cheng Lin; Yusuke Hirota; Sung-Feng Huang; Hung-yi Lee

Source-linked AI summary

VIBE: Voice-Induced open-ended Bias Evaluation for Large Audio-Language Models via Real-World Speech

Yi-Cheng Lin, Yusuke Hirota, Sung-Feng Huang, Hung-yi Lee

arXiv:2604.17248v2Apr 19, 2026eess.AScs.CLcs.SD

TL;DR

Generative bias in large audio-language models remains insufficiently assessed by benchmarks built around synthetic speech, closed-ended tasks, and performance disparities. VIBE evaluates open-ended generation from human-recorded speech and finds pervasive, task-dependent demographic disparities across 12 models, with gender and accent cues producing significant shifts.

Problem
Existing speech fairness evaluations largely measure demographic performance disparities in closed-ended tasks, leaving stereotype reinforcement in generated content insufficiently examined.
Method
VIBE evaluates LALM bias through open-ended generation on human-recorded speech, allowing associations to emerge without predefined answer options.
Results
Across 12 models and five tasks, demographic disparities are pervasive and strongly task-dependent, with every model showing significant disparities on at least four settings.
Takeaways & Limitations
Bias should be reported task by task rather than as a single score because no evaluated model is fair across all tasks.
Takeaways & Limitations
VIBE covers binary gender and selected first-language accent groups in read English speech, excluding several demographic dimensions and spontaneous conversation.

Abstract

from arXiv · show

Large Audio-Language Models (LALMs) are increasingly integrated into daily applications, yet their generative biases remain underexplored. Existing speech fairness benchmarks rely on synthetic speech and Multiple-Choice Questions (MCQs), both offering a fragmented view of fairness. We propose VIBE, a framework that evaluates generative bias through open-ended tasks such as personalized recommendations, using human-recorded speech. Unlike MCQs, our method allows stereotypical associations to manifest organically without predefined options, making it easily extensible to new tasks. Evaluating 12 state-of-the-art LALMs reveals systematic biases in realistic scenarios. Both gender and accent cues trigger statistically significant distributional shifts, and bias magnitude is strongly task-dependent.

I. INTRODUCTION

LALMs increasingly generate open-ended responses that can shape social narratives, exposing an evaluation gap because existing fairness benchmarks mainly measure closed-ended performance disparities, refusal rates, or MCQ outcomes. VIBE addresses this gap with open-ended, human-speech-based evaluation, revealing systematic and task-dependent speaker-triggered biases across 12 LALMs.

Motivation: LALMs now process speech and text to generate open-ended responses, so biases can shape the social narratives presented through personalized recommendations.The introduction frames generative LALMs as active agents interpreting human intent rather than only recognizing speech or classifying inputs.
Evaluation gap: Existing speech fairness evaluations mainly use closed-ended tasks and performance disparities, leaving generative bias insufficiently evaluated.Examples include demographic differences in Word Error Rate and analogous metrics for emotion recognition and intent classification.
Evaluation gap: Refusal Rates and MCQs provide a fragmented view of speaker-triggered fairness because they measure safety-filter rigidity or constrained choices rather than naturally generated stereotypes.A model may rarely refuse yet still propagate harmful stereotypes, while MCQs can miss content-level disparities that emerge in free-form outputs.
VIBE: VIBE evaluates LALMs through open-ended generation, allowing latent social associations to surface without constraining responses to anticipated answer choices.The framework is designed for unconstrained outputs that remain extensible to new tasks.
Findings: Across 12 LALMs and five open-ended tasks, bias was systematic and highly task-dependent, with narrative and recommendation prompts eliciting stronger demographic-conditioned responses than professional reviews.DeSTA associated female speakers with service and caregiving roles, while male speakers were linked to technical or artistic occupations.

II. METHODOLOGY · A. Framework Overview

VIBE is a generative framework that evaluates representational biases in Large Audio-Language Models from demographic speech cues and task-specific prompts. It converts free-form responses into structured attributes so biases can emerge organically through vocal characteristics and social stereotypes.

A. Framework Overview: VIBE quantifies representational biases in Large Audio-Language Models using a generative evaluation framework.The framework is presented in Fig. 2.
A. Framework Overview: The framework takes audio input Xaudio containing demographic cues and a task-specific prompt P.An example prompt is “Describe the personality of this speaker”.
A. Framework Overview: The target LALM Mθ generates a free-form textual response Ytext from the audio input and prompt.This response is the framework’s unstructured model output.
A. Framework Overview: An LLM-based extractor Eϕ, Qwen3-8B,, transforms Ytext into quantifiable data.The extractor is used to process the unstructured response.
A. Framework Overview: The extractor maps Ytext to structured attributes S = {a1, . . . , an}, with each ai representing a specific trait.Examples of traits include occupation, activity, and personality.
A. Framework Overview: Extraction prompts are tailored to each task and specify the attributes to identify.Story targets occupation and personality, whereas Advisory targets hobbies.
A. Framework Overview: The generative approach allows biases to manifest organically through associations between vocal characteristics and social stereotypes.This is intended to reflect the model’s true internal associations.

B. Evaluated Tasks

The evaluation uses five open-ended tasks spanning personal, social, economic, professional, media, and consumer contexts. Together, these tasks probe demographic-conditioned variation and stereotyping through realistic recommendations, assessments, and creative generation.

Story Generation: Story Generation requires fictionalizing an audio-recorded person with defined occupation, economic situation, education, family status, and personality attributes.These attributes cover social, economic, and personal dimensions for measuring demographic-conditioned variation.
Advisory: Advisory asks the model to suggest three hobbies or activities, probing gendered or cultural interests reflexively assigned to voices.The task targets stereotypical associations in personalized recommendations.
Candidate Review: Candidate Review evaluates interview responses for professional competency, interaction style, cultural fit, and recommended compensation in a simulated hiring scenario.It is designed to expose bias in high-stakes hiring and salary negotiation.
Hollywood Casting: Hollywood Casting requires a blunt memo judging star power, character function, romantic appeal, and narrative longevity to target media stereotypes and lookism.Narrative longevity includes whether the character survives.
Personal Shopper: Personal Shopper profiles a customer from an opening line by predicting department, budget level, buying triggers, and general vibe, focusing on socioeconomic and consumer stereotyping.The model acts as a luxury sales associate.

C. Data Sources & Dimensions · D. Quantifying Bias

VIBE evaluates gender- and accent-triggered bias using controlled speech datasets and defines bias as statistically reliable distributional shifts in extracted social attributes across speaker groups. It summarizes multi-attribute task results with average nTVD and tests significance at the speaker level.

C. Data Sources & Dimensions: CREMA-D supplies 7,442 clips from 91 actors—48 male and 43 female—for gender-based bias evaluation.Each actor recites the same 12 neutral sentences across six core emotions.
C. Data Sources & Dimensions: The Speech Accent Archive provides a gender-balanced subset of 406 second-language English speakers across six common non-English native languages.The selected languages are Spanish, Arabic, Mandarin, French, Korean, and Russian; speakers read the same elicitation paragraph.
C. Data Sources & Dimensions: L2-ARCTIC complements the primary accent corpus with non-native English speech from six native-language backgrounds under controlled transcription filtering.Sentences containing words related to gender, age, or race were manually excluded.
D. Quantifying Bias: For demographic-irrelevant tasks without specified attributes, fairness is defined as distributional invariance across speaker groups under identical linguistic content.The bias statement covers tasks such as fictional storytelling or general advice.
D. Quantifying Bias: Bias is operationalized as statistically reliable distributional shifts in extracted attributes across demographic groups.The definition follows prior distributional-difference approaches to bias measurement.
D. Quantifying Bias: For each attribute, the method estimates group-conditional distributions by normalizing retained attribute-value frequencies within each demographic group.It then defines a group-average reference distribution for comparison.
D. Quantifying Bias: Task-level bias is the average of per-attribute nTVD scores, while significance tests compare observed nTVD against independence between demographic group and generated content.Because speakers contribute multiple utterances, permutations treat the speaker—not the utterance—as the exchangeable unit under H0.

III. EXPERIMENT · A. Experimental Setup

The experiment evaluates 12 LALMs selected to span architectural evolution, model scale, and accessibility. The setup also includes an accent-induced bias condition using the Speech Accent Archive.

A. Experimental Setup: 12 LALMs are evaluated as a diverse model set.The selection is guided by architectural evolution, model scale, and accessibility.
A. Experimental Setup: Architectural evolution motivates selection from audio-text alignment to native omni-multimodal reasoning.
A. Experimental Setup: Model scale ranges from 2B to 8B parameters.
A. Experimental Setup: Accessibility is represented through both open-source models and closed-source API services.
A. Experimental Setup: The evaluated models include Qwen2-Audio-7B-Instruct, Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and Phi-4-multimodal-instruct.The passage also provides the shorthand Qwen2-Audio for Qwen2-Audio-7B-Instruct.
A. Experimental Setup: Accent-induced bias is evaluated using the Speech Accent Archive.

B. Bias evaluation · C. Case study

Across 12 LALMs, demographic bias is widespread, task-dependent, and model-ranking-dependent, with nTVD reaching 46 while refusal-rate disparities remain small. The case study shows that gender-conditioned recommendations produce systematic, interpretable shifts across activities, shopping categories, and other attributes.

B. Bias evaluation: Every one of the 12 models shows statistically significant demographic disparities on at least four tasks and dimension settings.The strongest disparity reaches an nTVD of 46 on Advisory.
B. Bias evaluation: Across-model mean nTVD rises from about 2 on Candidate Review to about 14 on Advisory, demonstrating strong task dependence.Advisory is highest-bias for half the models, while Candidate Review is lowest for almost all.
B. Bias evaluation: DeSTA has the highest mean nTVD and significant disparities on 9 of 10 settings, whereas Qwen2.5-Omni models have the lowest mean bias.Model rankings shift across tasks, making a single global score misleading.
B. Bias evaluation: Refusal-rate nTVD never exceeds 6.6 across models and tasks, indicating similar refusal rates across genders and accents.Refusal behavior itself does not exhibit substantial demographic disparity.
C. Case study: Gender-conditioned distributions reveal systematic group-level shifts, confirming that the metric captures interpretable demographic-conditioned patterns.The figure presents high-bias tasks and models through per-group attribute distributions.
C. Case study: In Advisory, female speakers receive domestic and reflective recommendations, whereas male speakers receive physical and performative suggestions.Examples include cooking or baking, yoga or meditation, hiking, martial arts, and playing a musical instrument.
C. Case study: In Shopper, female speakers are directed toward accessories, cosmetics, and designer handbags, while male speakers are directed toward electronics and menswear.This parallel gendered split is another concrete manifestation of the measured bias.

D. Robustness to the frequency threshold · E. Cross-corpus robustness

The bias conclusions are robust to the frequency threshold: increasing τ lowers absolute nTVD while preserving model rankings. On an independent accent corpus, the main task and model patterns recur, with significant cross-corpus ranking agreement.

D. Robustness to the frequency threshold: The metric filters attribute values appearing fewer than τ times, with τ = 10 used in the main results to prevent rare outputs from inflating nTVD.The robustness analysis varies τ across {1, 2, 5, 10, 20, 40}.
D. Robustness to the frequency threshold: Increasing τ decreases absolute nTVD on every dataset because filtering rare, group-concentrated values leaves more evenly shared common values.The analysis compares absolute scores across thresholds against τ = 10.
D. Robustness to the frequency threshold: Mean within-task model-ranking correlation with τ = 10 stays above 0.98 on gender as the frequency threshold changes.Model rankings are evaluated using Spearman correlation within each task.
E. Cross-corpus robustness: L2-ARCTIC provides an independent accent evaluation with more varied sentence content and different first-language groups than SAA.Absolute scores are not compared across the two corpora because their first-language groups and content differ.
E. Cross-corpus robustness: 8.25 is the mean nTVD for Advisory on L2-ARCTIC, the highest task value, while Candidate Review is lowest at 1.97.These task-level patterns reproduce the headline results.
E. Cross-corpus robustness: DeSTA is again the most accent-biased model, achieving the top score on four of the five L2-ARCTIC tasks.Most larger disparities remain statistically significant after FDR correction despite the corpus containing 24 speakers.
E. Cross-corpus robustness: Spearman ρ = 0.76 and p = 0.004 quantify agreement between accent-bias rankings across the two corpora.The result supports consistency with a different accent set and different content.

IV. LIMITATIONS · V. CONCLUSION

VIBE evaluates representational bias in LALMs through open-ended generation on real human speech, revealing pervasive, task-dependent demographic disparities across 12 models and five tasks. Its scores require careful interpretation because low nTVD can mask answer collapse, while absolute values depend on extraction quality and the studied demographic and language scope.

IV. LIMITATIONS: A large and statistically significant nTVD directly indicates that a model treats groups differently when only the voice changes.
IV. LIMITATIONS: A low nTVD requires caution because a model may achieve it by collapsing to a near-constant answer that ignores the speaker.
IV. LIMITATIONS: Open-vocabulary attributes such as recommended activities can fragment into near-duplicate strings that inflate nTVD.
IV. LIMITATIONS: Filtering rare values with a frequency threshold reduces noise, and model rankings remain stable as the threshold varies.
IV. LIMITATIONS: Absolute scores depend on extraction quality, while the study examines binary gender and several first-language accent groups in read English speech.
V. CONCLUSION: VIBE evaluates representational bias through open-ended generation on real human speech, allowing stereotypical associations to surface without predefined MCQ options.
V. CONCLUSION: Across 12 models and five tasks, every model shows statistically significant demographic disparities, demonstrating that bias is pervasive.
V. CONCLUSION: Bias magnitude is strongly task-dependent, with the highest levels on open-ended prompts and the lowest on structured candid…

APPENDIX A EVALUATION PROMPTS · APPENDIX B LIMITATIONS

Appendix A specifies the exact audio-grounded prompts for five evaluation tasks and notes that VIBE’s code and prompts will be released openly. Appendix B defines bias as distributional shifts across speaker groups while acknowledging limits in fairness coverage and experimental scope.

APPENDIX A EVALUATION PROMPTS: The benchmark code and evaluation prompts will be released under an open-source license for reproducibility.The evaluation analyzes model outputs rather than modifying or redistributing speech data.
APPENDIX A EVALUATION PROMPTS: Five evaluation tasks use exact prompts paired with audio input Xaudio and sent to the target LALM.The appendix presents the prompts used for each task.
APPENDIX B LIMITATIONS: Bias is operationalized as distributional shifts in generated attributes across speaker groups under content-controlled settings.This definition follows prior work cited as.
APPENDIX B LIMITATIONS: The definition captures systematic stereotyping but does not address all notions of fairness.Examples explicitly excluded include individual fairness and intersectional bias.
APPENDIX B LIMITATIONS: Future work is encouraged to explore complementary fairness definitions discussed in.The appendix frames the operational definition as limited rather than exhaustive.
APPENDIX B LIMITATIONS: All experiments use English speech from two datasets, CREMA-D and LThe supplied passage states this dataset and language scope but is truncated after “L.”

APPENDIX C ETHICAL CONSIDERATIONS

VIBE is intended as a diagnostic benchmark for auditing demographic bias in LALMs, not as a standalone fairness certification or deployment decision tool. Its evaluation prompts deliberately elicit potentially stereotypical outputs across open-ended tasks and require explicit assessments of personal, professional, and consumer attributes.

Intended use: VIBE is designed for researchers and developers to audit demographic bias in LALMs, not to certify models as fair or determine deployment decisions alone.The benchmark is presented as one lens among many for understanding model behavior.
Potential risks and dual use: The benchmark necessarily surfaces biased outputs, including stereotypical attribute associations, for measurement purposes.The paper explicitly acknowledges risks associated with collecting such outputs.
Evaluation prompts: Open-ended prompts require explicit judgments about attributes such as occupation, demographics, education, family situation, personality, hobbies, professional competency, interaction style, cultural fit, and compensation value.These requirements span fictional stories, recommendations, and HR assessment summaries.