PaperlayerView exact revision on arXiv

Source-linked AI summary

When Entanglement Lower-Bounds Disparity: Auditing and Repairing Demographic Fairness in Audio Understanding Models

Kian Shamsaie, Iman Modarressi

arXiv:2609.27382v1cs.SDcs.CL

TL;DR

Speech audits face confounding between speaker demographics, content, and style, while many deployed systems do not expose representations for subspace analysis. TRIAD varies text, voice profiles, and expressive styles independently and links representation leakage to probe disparity. The measured disparity tracks leakage, and ORCA reduces leakage and gaps; the audit uses perceived attributes from a single synthesizer.

  • Problem

    Speech disparities are documented, but observational audits confound voice with content, style, and channel, and many deployed systems expose neither weights nor representations for subspace analysis –.

    Source: §I · p. 1

  • Method

    TRIAD crosses 120 texts, 24 rendered voice profiles, and ten expressive styles, then measures axis fidelity and subspace leakage and repairs open encoders with ORCA.

    Source: §I · p. 1, A. A Three-Axis Factorial Audit Corpus · p. 2

  • Results

    Mean-square probe disparity tracks aggregate leakage Λ (r = 0.93), and ORCA cuts leakage 72% while roughly halving the gaps.

    Source: E. The Repair: ORCA on the Grid and on… · p. 5, §VIII · p. 6

  • Takeaways & Limitations

    The results support representation geometry as a first-order account of fairness for the audited audio encoders and linear probes.

    Source: §VIII · p. 6

    The single-synthesizer grid measures perceived rather than self-reported demographic attributes, and its checks bound but do not eliminate synthesizer-specific confounding.

    Source: §VII · p. 6

Abstract

from arXiv · show

Speech technology penalizes some voices: recognition errs nearly twice as often for Black speakers, and accuracy declines for second-language accents and older speakers. We introduce TRIAD, an audit grid crossing 120 texts, 24 rendered demographic voice profiles (gender, age band, accent), and ten expressive styles via controllable text-to-speech, isolating perceived demographic attributes from content and affect. For ten open-weights encoders we define axis-fidelity functionals, principal-angle leakage between axis subspaces, and group-conditional gaps; a proposition proves that average probe disparity grows with the same aggregate voice-semantic leakage $\Lambda$ we measure, and a corollary shows that peak leakage forces worst-case disparity inside an active region. The measured mean-square probe disparity tracks $\Lambda$ (Pearson r = 0.93), and a black-box protocol exposes the same signature in two closed-source models. ORCA, an adapter combining axis-specific contrastive heads, an orthogonality penalty, and group-balanced sampling, cuts leakage 72% and roughly halves the gaps.

I. INTRODUCTION

Speech systems show substantial disparities, but observational confounding and limited access to model internals make those gaps difficult to attribute and audit. The paper introduces a controlled audit and geometry-based account, complementing rather than superseding prior fairness-impossibility work.

I. INTRODUCTION

  • Five commercial recognizers had nearly double the word error rate for Black versus white speakers, 0.35 against 0.19, while accent, age, gender, and speaker-verification skews also appear –.The paper notes that disparities can propagate through downstream products built on frozen audio encoders and speech-to-speech assistants,.Source: §I · p. 1
  • Observational audits confound voice with content, style, and channel, while many deployed systems expose neither weights nor representations for subspace analysis –.The stated gaps include an instrument that varies semantics, demographics, and expressiveness independently, a representation-geometry theory of disparity, and a theory-linked repair.Source: §I · p. 1
  • TRIAD crosses the same 120 texts with 24 rendered voice profiles and ten expressive styles, making perceived demographic covariates orthogonal to content and affect by construction.The paper defines axis fidelities, principal-angle leakage, and group-conditional fidelities, and proposes ORCA with axis-specific contrastive heads, an orthogonality penalty, and group-balanced sampling.Source: §I · p. 1

II. RELATED WORK

  • Prior fairness impossibility results address label statistics and decision rules, whereas this work bounds linear-probe disparity through subspace geometry in frozen audio encoders –.ORCA preserves all three axes while separating them, unlike invariance approaches that delete speaker information –.Source: §II · p. 2

III. THE TRIAD GRID AND A FAIRNESS-LINKED THEORY OF DISENTANGLEMENT

TRIAD uses a balanced three-axis synthesis grid to separate semantic, demographic-voice, and expressive variation. The paper defines axis fidelity through variance decomposition and shows how the balanced grid decomposes representation variation into uncorrelated components.

A. A Three-Axis Factorial Audit Corpus

  • TRIAD exhaustively crosses 120 neutral texts, 24 voice profiles, and ten styles into 28,800 rendered cells, with 1,200 replicates estimating synthesis noise.The profiles cover two genders, three age bands, and four accent groups; quality control includes round-trip filtering and human checks, with reported Cohen’s κ of 0.76 for style and 0.81 for perceived demographics.Source: A. A Three-Axis Factorial Audit Corpus · p. 2

B. Axis Fidelity by Variance Decomposition

  • Axis fidelity F_a is the mean between-class energy ratio over the optimal probe subspace, equivalently the mean squared regularized canonical correlation between representations and axis labels,.The factorial design gives semantic fidelity a controlled reading: it is high when semantic structure survives demographic and expressive perturbations, with analogous readings for the other axes.Source: B. Axis Fidelity by Variance… · p. 2
  • On the balanced grid, centered representations decompose into three axis main effects, interaction means, and within-cell residuals that are mutually uncorrelated in sample.The resulting covariance decomposition is exact; the release verifies the identity to machine precision.Source: B. Axis Fidelity by Variance… · p. 2

C. Subspace Leakage

The paper quantifies overlap between axis subspaces using aggregate and peak principal-angle leakage. Under its additive-factor assumption, pairwise leakage constrains jointly achievable probe correlations.

C. Subspace Leakage

  • Aggregate leakage Λ is the mean squared principal-angle cosine, while peak leakage ℓ is the largest principal cosine; both use m = 8 dimensions, and unsuffixed values denote voice-semantic leakage.The aggregate measure is audited, while peak leakage governs the worst-case-over-direction corollary.Source: C. Subspace Leakage · p. 2
  • The peak leakage is at least the aggregate leakage, with equality only when all principal angles coincide.This relation bridges the aggregate audited quantity to the peak-leakage worst-case result.Source: C. Subspace Leakage · p. 2
  • Figure 1 contrasts latent-space audits of open-weights encoders with matched-pair output tests for closed-source models and depicts ORCA’s group-balanced, orthogonalized-head repair.Its left panel shows the factorial text, voice-profile, and style grid, with one TTS-rendered utterance per cell.Source: Figure 1 · p. 3
  • Under the additive three-factor model, maximal jointly achievable probe correlations across the three axes decrease with every pairwise leakage Λ_ab.The model assumes uncorrelated, zero-mean, unit-covariance rank-m factors, orthonormal loading columns, and white noise.Source: C. Subspace Leakage · p. 3

D. Group-Conditional Fidelity and the Fairness Link

The paper defines group-conditional fidelity and fidelity-spread measures, then links mean-square probe disparity to aggregate leakage under balanced-grid and group-separation assumptions. A companion result bounds worst-case disparity in an active region determined by semantic alignment and peak leakage.

D. Group-Conditional Fidelity and the Fairness Link

  • For each demographic group, axis fidelity is recomputed on its voice-profile slice; Γa measures the range of that fidelity across groups, with Γacc and Γgen denoting specified accent and gender spreads.The disparity theorem concerns geometry through Λ, not fidelity spread.Source: D. Group-Conditional Fidelity and the… · p. 3
  • Under the stated representation and sampling assumptions, mean-square output disparity has a lower bound that increases with aggregate voice-semantic leakage and vanishes exactly when that leakage is zero.The bound is tight when the probe lies entirely in the semantic subspace.Source: D. Group-Conditional Fidelity and the… · p. 3
  • For probes with semantic alignment at least ρ, peak leakage forces positive worst-case disparity exactly when ρ² + ℓ² > 1, and the bound increases with ℓ in that region.The bound is attained by a constructed probe.Source: D. Group-Conditional Fidelity and the… · pp. 3–4
  • Aggregate leakage controls the average disparity floor, while peak leakage controls the most adversarial direction; reducing leakage relaxes both floors.The paper relates this geometry to a pretraining imbalance that can rotate semantic discriminants toward voice statistics, and identifies ORCA as its repair.Source: D. Group-Conditional Fidelity and the… · p. 4

IV. BLACK-BOX PROTOCOL FOR CLOSED-SOURCE MODELS

The black-box protocol audits closed-source speech-to-speech systems through matched outputs because their weights and representations are unavailable. It contrasts voice and semantic changes, removes a rendering-and-decoding noise floor, and tests task-level group accuracy gaps.

IV. BLACK-BOX PROTOCOL FOR CLOSED-SOURCE MODELS

  • The protocol compares matched pairs that differ in one axis and subtracts re-rendering noise to measure output invariance without access to model representations.Responses combine transcript text embeddings with audio pitch, energy, and rate features.Source: §IV · p. 4
  • Rvoice is zero when a model’s responses depend on what was said but not on who said it.It normalizes voice-pair contrast by semantic-pair contrast.Source: §IV · p. 4
  • The speaker-level permutation test rejects at 0.058 under an exchangeable null at α = 0.05 and detects a nine-point true gap at p < 10^-3.The release includes both calibration checks and bootstrap intervals over texts.Source: §IV · p. 4

V. ORCA: ORTHOGONAL RESIDUAL CONTRASTIVE ADAPTER

ORCA adds a residual adapter and three axis-specific contrastive heads while retaining the encoder’s weights. An orthogonality penalty targets leakage between head subspaces, and voice-profile-balanced batches structure training.

V. ORCA: Orthogonal Residual Contrastive Adapter

  • ORCA leaves encoder weights untouched and uses a residual bottleneck adapter feeding three linear heads, one each for text, voice profile, and expressive style.The adapter has a 256-dimensional bottleneck and 1.9 million parameters.Source: §V · p. 4
  • Each head uses supervised contrastive loss on its own labels, preserving text, voice-profile, and style information rather than adversarially deleting any axis.Training batches are stratified uniformly over the 24 voice profiles.Source: §V · p. 4
  • The orthogonality penalty equals summed subspace overlap across head pairs, so shrinking it reduces the average disparity floor and, through the leakage bound, the peak-leakage worst-case floor.Training uses grid renders plus group-balanced slices of labeled corpora.Source: §V · p. 4

A. Setup and Real-Speech Validation Data

The evaluation combines open-weights model configurations, a stratified real-speech anchor, and labeled recognition and emotion benchmarks. The real-speech pool supports parity and expressiveness checks across several conversational settings.

A. Setup and Real-Speech Validation Data

  • The audit covers ten open-weights configurations across semantic encoders, self-supervised models, and codecs, with layer sweeps, five-fold cross-validation, and bootstrap uncertainty estimates.The closed-source regime includes two models and 11,520 trials per model.Source: A. Setup and Real-Speech Validation Data · p. 4
  • The 60-hour real-speech anchor combines five public conversational corpora, including self-reported age and gender metadata, spontaneous dyads, audiovisual interaction, and single-speaker or multiparty speech.CANDOR supplies 24 hours and is the primary real-parity source for age and gender; AMI-IHM calibrates the pipeline at 16.0% word error for Whisper large-v3 greedy decoding.Source: A. Setup and Real-Speech Validation Data · pp. 4–5
  • Labeled validation uses Edinburgh International Accents and Common Voice accent subsets for recognition, plus CREMA-D, MSP-Podcast, and IEMOCAP for emotion.Recognition uses CTC probes trained on LibriSpeech.Source: A. Setup and Real-Speech Validation Data · p. 5

B. The Audit: Entanglement and Parity Gaps Everywhere

Across ten open-weight encoders, voice-semantic leakage accompanies fidelity gaps, while aggregate leakage tracks mean-square probe disparity. Two closed-source speech-to-speech models also show accent and age gaps and gender differences in expressiveness recognition.

B. The Audit: Entanglement and Parity Gaps Everywhere

  • Λ ranges from 0.29 to 0.56 across ten encoders, with significant accent-fidelity gaps of 0.041 to 0.091 and gender gaps of 0.027 to 0.052.Aggregate leakage predicts measured mean-square probe disparity with Pearson r = 0.93 (95% CI [0.84, 0.98]).Source: B. The Audit: Entanglement and Parity… · p. 5

C. Closed-Source Speech-to-Speech Models

  • Spoken-question accuracy declines from 93.8% and 94.6% for General American voices to 88.1% and 89.8% for Mandarin-accented voices in the two closed-source models.Accent gaps are 5.7 and 4.8 points; age-band gaps are 3.2 and 2.5 points, and female-profile voices receive 6.2 and 5.1 points higher expressiveness recognition accuracy.Source: C. Closed-Source Speech-to-Speech Models · p. 5, Figure 2 · p. 5

B. The Audit: Entanglement and Parity Gaps Everywhere

  • In Fig. 2(b), the two ORCA measurements fall far below the baseline regression relating aggregate leakage to measured mean-square probe disparity.Fig. 2(a) also plots the averaged and worst-case disparity floors against aggregate and peak leakage, respectively.Source: Figure 2 · p. 5

C. Closed-Source Speech-to-Speech Models

  • Both closed-source models have worst-to-best error ratios of 1.9, matching the ratio reported for commercial recognizers.The accent gap on gpt-realtime-2 also confirms disparity across independent model families.Source: C. Closed-Source Speech-to-Speech Models · p. 5

D. Does the Synthetic Grid Predict Real Disparities?

TRIAD’s synthetic-grid measurements are checked against real-speech data, with the authors noting that the single-synthesizer confound is reduced but not eliminated. ORCA reduces leakage and parity gaps on the grid and improves results across several real-speech evaluations.

D. Does the Synthetic Grid Predict Real Disparities?

  • Three checks provide convergent evidence that TRIAD measurements transfer to real speech, reducing but not removing the single-synthesizer confound.The natural-speech model ordering has Spearman ρ = 0.93, and grid-measured per-group disparities predict real ones across forty model-group cells with ρ = 0.86.Source: D. Does the Synthetic Grid Predict… · p. 5

E. The Repair: ORCA on the Grid and on Real Speech

  • ORCA cuts WavLM aggregate leakage by 72% (0.29 to 0.08) and halves both gaps; on Whisper, leakage falls from 0.51 to 0.11 and the accent gap from 0.089 to 0.031.WavLM’s three axis fidelities all rise, and the repaired points fall far below the baseline regression.Source: E. The Repair: ORCA on the Grid and on… · p. 5, Table I · p. 6
  • On real speech, the EdAcc worst-to-best word-error gap compresses from 12.6 to 8.6 points, the CREMA-D gender gap falls from 4.4 to 1.8 points, and the MSP-Podcast valence gap halves from 0.07 to 0.03.The EdAcc ratio drops from 1.78 to 1.58, and no group is made worse to equalize.Source: E. The Repair: ORCA on the Grid and on… · p. 6

F. Ablations

The ablations distinguish the roles of orthogonality and group-balanced sampling in reducing leakage and disparity, while the conclusion links representation leakage to probe disparity. The paper also bounds its audit and theory by single-synthesizer profiles and linear probes.

F. Ablations

  • Removing orthogonality nearly triples leakage to Λ = 0.21 and widens accent and EdAcc gaps; removing group-balanced sampling preserves Λ = 0.10 but weakens the gap reduction.Single-axis variants barely move leakage (Λ = 0.26, 0.27), supporting the need to separate all three subspaces for parity.Source: F. Ablations · p. 6

VII. LIMITATIONS AND FUTURE DIRECTIONS

  • The audit uses perceived rather than self-reported demographics from one synthesizer, leaving synthesizer-specific artifacts as a confound; the theory certifies linear probes only.The authors propose rendering across multiple independent synthesizers and pairing synthetic cells with consent-driven recordings; kernel probes and multilingual grids remain future directions.Source: §VII · p. 6

VIII. CONCLUSION

  • Across ten encoders, aggregate leakage explains measured probe disparity almost entirely (r = 0.93), and an adapter minimizing leakage halves the measured gaps.Under an additive factor model, average disparity of an accurate linear probe grows with aggregate leakage, while high peak leakage forces worst-case disparity positive.Source: §VIII · p. 6

Related papers

Loading 2609.27382v1…