PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring Fairness in Large Audio Language Models via Semantic-Aware Bias Estimation

Zhe Liu

arXiv:2608.13624v1cs.CLcs.AIcs.SD

TL;DR

Fairness evaluation for spoken-input LALMs can misattribute subgroup disparities when semantic variation and speaker characteristics are not controlled. This paper introduces semantic-aware mixed-effects regression and finds that controlling these factors reduces spurious fairness findings and produces more reliable subgroup-gap estimates.

  • Problem

    Fairness evaluation in spoken-input LALMs lacks adequate control for semantic variation and speaker-specific characteristics, risking overstated or misattributed subgroup disparities.

  • Method

    The framework uses reference-text sentence embeddings as covariates and speaker identity as a random effect, with embeddings extracted from the evaluated LALM.

  • Results

    Across simulated and benchmark evaluations, semantic-aware regression reduces spurious fairness findings and attenuates apparent gender gaps to statistical insignificance.

  • Takeaways & Limitations

    The framework provides more reliable and interpretable fairness estimates by distinguishing subgroup performance differences from semantic and speaker confounding.

Abstract

from arXiv · show

Large Audio Language Models (LALMs) have seen increasing use for audio understanding tasks such as speech recognition and audio question answering, raising concerns about fairness across demographic subgroups. Fairness evaluation in spoken-input settings is challenging due to confounding factors, including semantic variation in spoken content and speaker-specific characteristics. Ignoring these factors can result in misleading conclusions about model bias. We propose a semantic-aware mixed-effects regression framework for fairness evaluation in LALMs that explicitly accounts for these confounders. Our approach incorporates sentence-level semantic embeddings of reference text as covariates and models speaker identity as a random effect. Notably, semantic representations are extracted from the same LALM under evaluation, enabling semantic control over variation as perceived by the model itself. Experiments on simulated data and real-world benchmarks demonstrate that the proposed approach substantially reduces spurious fairness findings and yields more robust and interpretable estimates of subgroup performance differences.

I. INTRODUCTION

LALM fairness evaluation can confound demographic disparities with semantic variation in spoken content and speaker-specific characteristics. The paper proposes a semantic-aware mixed-effects regression framework that controls semantic variation using model-derived sentence embeddings and speaker identity as a random effect.

  • Motivation: LALMs extend language models to spoken inputs and outputs, enabling audio understanding and spoken-interaction applications.
  • Motivation: Spoken-input fairness evaluations can misattribute disparities because demographic groups may differ in paralinguistic characteristics or semantic complexity.
  • Proposed framework: The proposed method uses sentence-level semantic embeddings of reference text as covariates and models speaker identity as a random effect.
  • Relation to prior work: Unlike prior audio fairness work centered on traditional ASR and external semantic representations, this framework targets LALMs and broader audio understanding tasks.
  • Contributions: The framework is presented as the first LALM fairness evaluation approach to explicitly account for semantic variation in spoken content.

II. RELATED WORK … III. METHODS

The paper extends mixed-effects fairness evaluation from ASR to LALMs and introduces semantic-aware modeling that uses LALM-derived sentence representations to control semantic confounding. Its method tests factor effects across subgroup levels and quantifies resulting performance gaps.

  • A. Fairness Evaluation in Audio Understanding: Fairness research in speech and audio systems has examined ASR performance disparities across demographic groups, including gender, age, and accent.
  • A. Fairness Evaluation in Audio Understanding: A prior ASR framework uses mixed-effects regression to account for speaker-level dependencies in fairness evaluation.
  • A. Fairness Evaluation in Audio Understanding: This work builds on that framework by targeting fairness evaluation for Large Audio Language Models.
  • B. Sentence Embeddings: Sentence representation extraction is longstanding, but this approach obtains sentence representations directly from the LALM under evaluation.
  • B. Sentence Embeddings: Using LALM-derived embeddings makes semantic representations reflect the model’s own perception of content, supporting control of semantic confounding.
  • III. METHODS: The methods define a factor variable for each utterance and test whether its effect is statistically significant across factor levels.
  • III. METHODS: The proposed semantic-aware mixed-effects regression quantifies performance gaps between subgroups for disparity and fairness analyses.

A. Semantic-Aware Mixed-Effects Regression

The framework estimates subgroup fairness in LALMs while controlling for utterance-level semantic variation and speaker-specific characteristics. For ASR, it uses distributionally appropriate regressions and quantifies disparities through expected-WER ratios.

  • ASR error modeling: ASR insertion errors use Poisson regression, whereas deletion-plus-substitution errors use Binomial regression because their outcome structures differ.Insertions are unbounded counts, while deletions and substitutions are bounded by reference-word counts and modeled as word-level Bernoulli trials.
  • Semantic control: Reference-text semantic embeddings enter as fixed-effect covariates to account for semantic variation across evaluation utterances.Reference text concatenates the audio groundtruth transcription with accompanying textual content such as instructions or prompts.
  • Speaker effects: Speaker-level random effects capture speaker-specific characteristics while targeting population-level rather than individual-speaker effects.The model assumes speakers are randomly sampled from a larger population.
  • Estimation and disparity: The model is fitted by maximum likelihood with adaptive Gauss-Hermite quadrature approximating integration over random effects.Fairness disparities are quantified as ratios of expected WERs across factor levels using estimated fixed effects and variance-covariance matrices.
  • Fairness interpretation: Including semantic embeddings isolates the factor of interest’s contribution to audio understanding and yields a more precise, semantic-aware fairness measure.PCA may optionally reduce high-dimensional embeddings to a lower-dimensional subspace that preserves the relevant information.

B. Extract Sentence Representations with LALMs

The section compares two approaches for extracting semantic embeddings from reference text with the same LALMs being evaluated, then uses these representations as regression covariates for semantic-aware bias estimation.

  • B. Extract Sentence Representations with LALMs: Two approaches compute semantic embeddings of reference text using the same LALMs under evaluation.These representations are added as explanatory covariates to the regression models introduced previously.

1) Prompt LALMs with reference text:

The method prompts the evaluated LALM with reference text and derives sentence embeddings from its hidden states. It uses Embed-AGG, which averages token representations from the first, middle, and final Transformer layers.

  • Prompting and representation: Passing reference text x to the evaluated LALM produces hidden states across the embedding layer and L Transformer layers.The final hidden state is represented across n′ tokens after tokenization.
  • Prompting and representation: Sentence embeddings can use the final token’s hidden state or alternative aggregation and pooling strategies.This work considers averaging hidden states across all tokens from the first, middle, and final Transformer layers.
  • Prompting and representation: Embed-AGG combines representations from different Transformer depths to summarize lexical features and higher-level semantic abstractions.Earlier layers tend to encode lower-level lexical features, while deeper layers progressively capture higher-level semantic abstractions.

2) Prompt LALMs with explicit one word limitation:

The explicit one-word limitation (EOWL) prompts the LALM to summarize a sentence’s meaning in one word. Embed-EOWL uses next-token logits as compact semantic representations, extracted from the evaluated LALM to control model-perceived input difficulty.

  • 2) Prompt LALMs with explicit one word limitation:: EOWL instructs the LALM to summarize the meaning of reference text x using a single word.The prompt is formatted as: LALM(This sentence: “x” means in one word:).
  • 2) Prompt LALMs with explicit one word limitation:: Embed-EOWL represents each sentence with the evaluated LALM’s next-token prediction logits rather than hidden states.The one-word prompt encourages the model to condense the entire sentence’s semantic content into a single-word prediction.
  • 2) Prompt LALMs with explicit one word limitation:: Using embeddings from the evaluated LALM controls variation in input difficulty as perceived by that same model.The model’s own representations are intended to provide the most faithful characterization of this variation.

IV. SIMULATION EXPERIMENTS · A. Setups

The simulation tests whether semantic-aware regression prevents spurious fairness findings when audio semantic content differs across speaker-gender groups despite no true gender effect. It constructs gender-linked question difficulty and evaluates responses from Qwen2-Audio.

  • IV. SIMULATION EXPERIMENTS: The experiment demonstrates that semantic differences across factor levels can create spurious unfairness findings when unaccounted for.
  • A. Setups: The simulation targets speaker gender’s effect on response accuracy in LALM audio question answering.
  • A. Setups: Llama-3 70B generated 2,000 questions divided into simple and hard sets through differing prompt instructions.
  • A. Setups: The simple questions were short, clear, and mainly commonsense-based, whereas hard questions were longer, detailed, and required cross-domain reasoning.
  • A. Setups: Female speakers synthesized 200 simple and 800 hard questions, while male speakers synthesized 800 simple and 200 hard questions.
  • A. Setups: Qwen2-Audio generated textual responses to the synthesized audio questions.
  • A. Setups: Response accuracy followed N(5, 1) for hard questions and N(6, 1) for simple questions, making gender correlated with difficulty but causally irrelevant.

B. Methods and Results · V. REAL DATA EXPERIMENTS

The paper evaluates vanilla and semantic-aware methods for gender fairness, using model-derived sentence embeddings and PCA to control semantic variation. It then applies the proposed mixed-effects regression to real speech data and visualizes semantic separation in the embeddings.

  • B. Methods and Results: The vanilla approach compares average male and female response scores by their ratio, with 95% confidence intervals estimated via bootstrap.
  • B. Methods and Results: Sentence embeddings are extracted from the evaluated Qwen2-Audio model using Embed-AGG and Embed-EOWL, then reduced to 8 dimensions with PCA.The 8-dimensional representation retains around 83% of total variance and yields stable regression estimates in practice.
  • B. Methods and Results: Table I compares vanilla and semantic-aware methods for gender fairness evaluation on simulated data.
  • B. Methods and Results: 1.124 was the male-female response accuracy score ratio for both vanilla estimation and regression without semantic covariates, producing false positive unfairness conclusions.The confidence intervals excluded 1 even though the ground truth was gender-neutral.
  • B. Methods and Results: The proposed semantic-aware regression methods produced male-female response accuracy score ratio estimates close to 1.
  • B. Methods and Results: After Embed-AGG embeddings were projected from 8 to 2 dimensions with PCA, they clearly differentiated simple and hard questions.This separation indicates that the extracted sentence embeddings captured the questions’ underlying semantic difficulty.
  • V. REAL DATA EXPERIMENTS: The real-data experiments apply the proposed semantic-aware mixed-effects regression to speech data for fairness analysis.
  • V. REAL DATA EXPERIMENTS: Figure 1 presents a PCA visualization of sentence embeddings.

A. Setups

The study evaluates LALM fairness using LibriSpeech for gender-based ASR and AIR-Bench-Chat for gender-annotated audio question answering. Qwen2-Audio generates responses, while Llama-3 70B judges AIR-Bench-Chat answers.

  • A. Setups: The evaluation uses two datasets: LibriSpeech and AIR-Bench-Chat.These datasets support ASR fairness and audio question-answering evaluation, respectively.
  • A. Setups: LibriSpeech evaluation uses Test-Clean with 2,620 utterances from 40 speakers and Test-Other with 2,939 utterances from 33 speakers.Test-Clean includes 20 male speakers, while Test-Other includes 16 male speakers; gender is the ASR fairness dimension.
  • A. Setups: AIR-Bench-Chat includes 436 utterances from male speakers and 163 from female speakers with available gender annotations.The benchmark evaluates comprehension of complex audio inputs and instruction following.
  • A. Setups: Qwen2-Audio generates textual responses for both datasets, and Llama-3 70B scores AIR-Bench-Chat answers from 1 to 10.Qwen2-Audio transcribes audio inputs or answers audio-based questions; Llama-3 70B serves as the judge for generated answers.

B. Methods and Results

The evaluation compares progressively structured regression models for gender fairness, showing that controlling for speaker and semantic confounders reduces misleading male–female performance gaps. The proposed semantic-aware mixed-effects framework yields more stable, interpretable, and reliable fairness assessments across audio understanding tasks.

  • Baseline and Model Design: Aggregated male–female performance ratios are intuitive but fail to account for speaker-specific characteristics and semantic variation.These confounders can distort group-level fairness comparisons.
  • Baseline and Model Design: The models progress from uncontrolled fixed effects to speaker-level mixed effects and semantic-aware mixed effects with reference-text embeddings as covariates.Sentence embeddings are extracted using Qwen2-Audio, the LALM under evaluation.
  • LibriSpeech Results: On LibriSpeech Test-Clean and Test-Other, speaker random effects eliminate much of the apparent WER bias, while semantic control further attenuates the gaps to statistical insignificance.Uncontrolled and partially controlled models indicate significant male–female WER gaps.
  • AIR-Bench-Chat Results: On AIR-Bench-Chat, all methods show no statistically significant gender differences in response quality, while semantic-aware regression produces ratios closest to unity.Speaker-level random effects are unavailable because the benchmark provides no speaker identifiers.
  • Embedding Analysis: Embed-AGG and Embed-EOWL perform similarly, but Embed-AGG is more effective because its lower-dimensional PCA projection explains more variance in the original embedding space.The comparison concerns the semantic embedding methods used in practice.
  • Scope: The framework is model-agnostic and attribute-agnostic by design, although the experiments focus on Qwen2-Audio and gender.The authors identify extension to additional models and demographic attributes as natural.

VI. CONCLUSIONS

The paper presents a semantic-aware mixed-effects regression framework for fairness evaluation in LALMs, showing that semantic and speaker confounding can explain apparent subgroup disparities. It emphasizes semantic control as a general requirement for reliable fairness estimates across audio understanding tasks.

  • Conclusions: The proposed semantic-aware mixed-effects regression model controls sentence-level semantic variation to produce more reliable subgroup performance-gap estimates than conventional methods.The framework is designed for fairness measurement in large audio language models.
  • Conclusions: Many apparent subgroup disparities reflect semantic or speaker confounding rather than intrinsic model bias.Accounting for these confounders improves the interpretation of fairness results.
  • Conclusions: Semantic control is important for fairness evaluations and supports a general framework applicable across diverse audio understanding tasks.The conclusion extends the framework beyond a single task or benchmark.
Loading 2608.13624v1…