Source-linked AI summary
Training-Free Instruction TTS Gender Bias Calibration Using Model-Adaptive Steering
Kuan-Yu Chen, Yi-Cheng Lin, Jeng-Lin Li, Jian-Jiun Ding
TL;DR
Instruction prompts can induce implicit acoustic gender skews, yet calibrating aggregate output distributions in generative TTS while preserving fidelity and explicit user intent remains an open challenge. The paper proposes model-adaptive, training-free steering of post-encoder representations, with explicit-gender prompts bypassed. Across four models, aggregate calibration error falls substantially, with bounded reported utility costs.
Problem
Calibrating aggregate gender skews from prompts without demographic constraints in generative ITTS, while preserving acoustic fidelity and explicit user intent, remains an open challenge.
Source: §1 · p. 1Method
Model-adaptive steering estimates a model-specific deviation direction and operating point, then shifts post-encoder conditioning representations without retraining and bypasses prompts with explicit gender keywords.
Source: §1 · p. 1Results
11.5–28.1 to 0.8–5.9 pp: aggregate calibration error across four models, with UTMOS ≤0.08 and WER ≤3.3 pp utility costs.
Source: §1 · p. 1Takeaways & Limitations
At fixed operating points, output rates stay within 0.9 pp down to 500 anchor pairs.
Source: §4 · p. 4The evaluation treats gender as binary, omits intersecting demographic dimensions, and is restricted to English benchmarks, lexicons, and models.
Source: Compliance with Ethical Standards · p. 5
Abstract
from arXiv · showhide
Instruction text-to-speech (ITTS) systems systematically encode acoustic gender skews from descriptive style prompts, such as occupations or personas, even when demographic attributes are left unspecified. Calibrating the implicit gender distribution of the synthesized voices, across heterogeneous architectures and without model retraining or overriding explicit user prompts, is an open problem. In this work, we propose \textit{model-adaptive steering}, a training-free bias calibration method that steers post-encoder conditioning representations using a group deviation vector paired with a coarse-to-fine strength search on a development set. A deterministic lexical gate bypasses intervention whenever explicit gender keywords are detected, preserving intended prompt semantics. Evaluated on a 12,800-prompt held-out benchmark across four ITTS models (three architectures), the proposed method reduces aggregate calibration error from 11.5--28.1 to 0.8--5.9 percentage points (e.g., shifting Parler-TTS Mini from 78.1\% and PromptTTS++ from 27.3\% female to 50.8--52.4\%). Output rates stay within 0.9 pp across anchor set sizes at fixed operating points, while UTMOS decreases by at most 0.08 and WER degrades by at most 3.3 pp.
1. INTRODUCTION
The introduction frames implicit gender skews in instruction prompts as an underexplored calibration problem and proposes model-adaptive steering with a gate for explicit gender prompts. Across four models, the method reduces aggregate calibration error with a small reported utility cost.
1. INTRODUCTION
- Descriptor prompts can skew acoustic gender profiles [6], while bias mitigation for generative instruction-guided speech remains comparatively underexplored despite related speech-task work.The introduction also identifies acoustic-quality degradation from large offsets and overriding explicit demographic attributes through indiscriminate steering as concerns.Source: §1 · p. 1
- A deterministic lexical gate bypasses steering on prompts with explicit gender keywords, preserving the specified prompt semantics.Steering is activated only for prompts without explicit gender keywords.Source: §1 · p. 1, Figure 1 · p. 2
- Model-adaptive steering estimates a separate deviation direction and operating point at each model’s post-encoder interface, without generator updates or adapters.The framework estimates the deviation vector from anchor pairs, searches for an operating point on development data, then shifts conditioning representations before acoustic decoding.Source: §1 · p. 1, Figure 1 · p. 2
- A joint objective over overall and stratum-wise disparities, with saturation filtering, prevents the single-gender collapse that aggregate parity alone admits.The method partitions prompts into descriptor-type strata and selects an operating point through coarse-to-fine grid search.Source: §1 · p. 1
- 11.5–28.1 to 0.8–5.9 pp: aggregate calibration error across four models, significant against every baseline, with UTMOS ≤0.08 and WER ≤3.3 pp.These are the reported calibration-error and utility bounds for the proposed method.Source: §1 · p. 1
2. METHOD
The method estimates model-specific gender steering directions from paired counterfactual prompts, then calibrates operating strengths on development data. At inference, it intervenes at each model’s conditioning interface only for prompts without explicit gender terms.
2.1. Group Deviation Estimation
- Paired prompts differing only in gender descriptors estimate a model-specific deviation direction from post-encoder representations, with positive steering associated with male acoustic profiles.The paired prompts share templates and transcripts; sequence representations are aggregated using masked mean pooling.Source: §2.1 · p. 2
2.2. Coarse-to-Fine Steering Strength Search
- A coarse-to-fine search selects steering strength on 500 development prompts, discarding saturated strata and minimizing a joint overall and stratum-wise parity objective.The deviation vector is held fixed from 2,500 anchor pairs; candidate configurations are evaluated on a coarse grid and, optionally, a finer grid.Source: §2.2 · p. 2
2.3. Inference-Time Conditional Feature Intervention
- A deterministic lexical gate bypasses intervention when a prompt contains an explicit gender keyword; otherwise, steering is applied at each model’s conditioning interface.The method covers token-level, pooled-style, and separate dual-stream conditioning, while generator parameters remain frozen.Source: §2.3 · p. 3
2.4. Baselines and Evaluation Metrics
- The parameter-update-free controls are a neutrality instruction and LEACEpb, which transforms pooled features before broadcasting the change to tokens.LEACEpb is fitted on pooled anchor features and gender labels, with its mixing weight tuned using the parity objective.Source: §2.4 · p. 3
- Evaluation uses a wav2vec 2.0 acoustic-gender proxy and measures utility with UTMOS and Whisper-based WER; Table 1 reports utility on a shared 500-prompt subset.The proxy excludes the child logit when deriving binary gender predictions.Source: §2.4 · p. 3, Table 1 · p. 3
3. EXPERIMENTS
The experiments calibrate operating points on development data and evaluate them on 12,800 held-out prompts. Steering substantially reduces calibration error across four systems, with anchor-count sensitivity and residual stratum disparities also assessed.
3.1. Experimental Setup and Evaluation Protocol
- The test partition contains 12,800 prompts, with calibration parameters optimized on development data and frozen before test evaluation.The development set has 500 items; prompt IDs are disjoint between development and test, though descriptors and templates come from a shared inventory.Source: §3.1 · pp. 3–4
3.2. Operating-Point Calibration
- Selected operating points yield 53.2%, 51.0%, 48.6%, and 59.4% female classifications for Parler-Mini, Parler-Large, PromptTTS++, and VoxInstruct, respectively.Saturated candidates are discarded; in these runs, the stratum term did not change the selected point.Source: §3.2 · p. 4
3.3. Held-Out Evaluation and Robustness Analysis
- Eagg falls from 11.47–28.11 to 0.80–5.86 pp across four systems, with the lowest point-estimate Emacro and Eworst across models.Paired-bootstrap confidence intervals for Emacro reduction exclude zero against both Original and the best baseline for all four models.Source: §3.3 · p. 4
- At fixed operating points, test female rates vary by at most 0.88 pp across anchor counts N=500, 1,500, and 2,500.The sensitivity analysis uses one nested anchor sequence per model.Source: §3.3 · p. 4, Table 3 · p. 4
3.4. Stratum-Level Consistency and Speech Utility
- Parler-Large retains career Eworst of 21.81 pp; relative to Original, UTMOS decreases by at most 0.076 and WER ranges from −4.13 to +3.26 pp.Mean descriptor rates move toward 50% in every model, but descriptor-rate standard deviation widens for both Parler models, narrows for PromptTTS++, and is flat for VoxInstruct; ambiguous outputs average 2.7% under Original and 3.4% under steering.Source: §3.4 · p. 4
4. CONCLUSION
The conclusion reports that model-adaptive steering calibrates acoustic gender output rates in frozen ITTS systems using post-encoder offsets. It reduces calibration errors across three conditioning layouts on 12,800 held-out prompts, while noting residual stratum disparities in Parler-Large and VoxInstruct.
4. CONCLUSION
- Across 12,800 held-out prompts, post-encoder conditioning offsets reduce calibration errors across three layouts at a cost of at most 0.08 UTMOS, with output rates within 0.9 pp down to 500 anchor pairs.Parler-Large and VoxInstruct retain residual stratum disparities.Source: §4 · p. 4
Compliance with Ethical Standards
The ethical statement describes the study’s operational scope and cautions against interpreting its acoustic proxy as validating personal identities.
Compliance with Ethical Standards
- Gender is operationalized as binary F/M for compatibility with available pretrained acoustic models and annotations, not as the full spectrum of gender identity.The authors identify this as an engineering simplification.Source: Compliance with Ethical Standards · p. 5
- The study examines perceived acoustic gender in isolation, omitting intersecting dimensions such as age, ethnicity, and regional dialect.Source: Compliance with Ethical Standards · p. 5
- Benchmarks, lexicons, and models are restricted to English, warranting caution when generalizing to languages with grammatical gender or distinct cultural associations.Source: Compliance with Ethical Standards · p. 5
- The acoustic classifier is a proxy for statistical skews in synthetic outputs and does not assess or validate true personal identities.Source: Compliance with Ethical Standards · p. 5