Personalization Without Stereotyping: Do Speech LLMs Know When to Turn a Deaf Ear?

• Speech LLMs can exploit voice-based demographic cues unavailable from linguistic content, but should use them only when relevant and appropriate.Personalization and stereotyping are two outcomes of sensitivity to speaker characteristics, distinguished by whether demographic cues are relevant to the task.Source: §1 · p. 1
• Existing bias and personalization evaluations can reward or penalize similar demographic-dependent responses because they do not consistently distinguish requested adaptation from stereotyping [28] [2] [49].The paper treats demographic personalization as desirable only when the user explicitly requests it [26].Source: §1 · p. 1
• BASIS-Bench evaluates demographic robustness, unprompted stereotypical behavior, and personalization when explicitly requested, while distinguishing acoustic, backbone, and interaction effects.It also includes Spoken StereoSet+, ELIP-Pair, BASIS-Health, and BASIS-Product, and evaluates eight speech LLMs.Source: §1 · pp. 1–2
• Most models show limited age and gender personalization from speech and stereotypical behavior toward female and elderly speakers, amplified by explicit labels but persisting with transcription.Explicit labels substantially improve personalization, suggesting the main difficulty is translating speech cues into appropriate behavior rather than personalizing at all.Source: §1 · p. 2

• Speech-native assistants have expanded from linguistic content to acoustic and paralinguistic information such as identity, prosody, emotion, and speaking style [14] [15] [21] [25] [31] [39] [44] [46] [47].The paper frames this evolution as reflecting recognition that spoken interaction requires modeling both linguistic content and acoustic characteristics.Source: §2 · p. 2
• Earlier speech-system bias work covers disparities such as transcription and translation, but assessing speech-encoding disparities alone misses other ways demographics can affect speech LLM behavior [3] [8] [13] [18] [23] [24] [30] [33] [43].The paper notes that speech LLMs’ broader capabilities introduce additional ways for demographic information to affect behavior.Source: §2 · p. 2
• Many speech-LLM bias datasets reuse text datasets, emphasizing lexical rather than acoustic cues; the authors argue that fixed semantic content with varied speaker attributes better measures speech-specific bias [11] [28] [29].Choi et al. [11], for example, resynthesize contexts from the text-based BBQ dataset.Source: §2 · p. 2
• Personalization benchmarks test responses to cues such as emotion, accent, age, and gender, while textual-LLM work documents demographic influence even when personalization is unrequested [2] [49] [51] [26] [17].This paper focuses on speech LLMs’ ability to perceive demographic cues, personalize when requested, and avoid unprompted stereotyping.Source: §2 · pp. 2–3

3. BASIS-Bench

• BASIS-Bench focuses primarily on age and gender and combines three evaluation axes with a framework for diagnosing sources of demographic disparity.Figure 1 overviews the axes, tasks, demographic dimensions, and datasets.Source: §3 · p. 3

3.1. Demographic Robustness Axis

• Demographic robustness measures age and gender classification alongside ASR and spoken question-answering disparities across speakers.The evaluation uses Fair-Speech, Spoken StereoSet+ and SD-QA for ASR, and SD-QA and LibriSQA for spoken question answering; scores use normalized WER, LLM-as-judge accuracy, and per-label classification accuracy.Source: §3.1 · p. 3

3.2. Stereotypical Bias Axis

• The stereotypical-bias axis tests whether perceived demographic cues prompt unwarranted stereotypical behavior, using Spoken StereoSet+ and per-demographic ICAT scores.Spoken StereoSet+ is manually filtered to align with the paper’s age- and gender-specific definitions; ICAT combines continuation relevance with stereotype skew and ranges from 0 (maximally biased) to 100 (unbiased).Source: §3.2 · pp. 3–4

3.3. Personalization Axis

• Personalization is treated as appropriate only when explicitly requested; age tasks compare child and adult styles, while gender tasks require selecting a response with the speaker’s gender marker.ELIP-Pair contains 521 questions with child- and adult-tailored responses; BASIS-Health and BASIS-Product assess gender personalization while minimizing reliance on stereotypes.Source: §3.3 · p. 4

3.4. Disparity Sources and Evaluation Modes

• The framework distinguishes acoustic, backbone, and interaction effects, then compares speech/text and implicit/explicit demographic-marker modes to separate cue perception from cue use.Speech-implicit is the target setting; speech-explicit bypasses speech-based demographic inference, while text comparisons isolate backbone behavior and the effect of acoustic information.Source: §3.4 · pp. 4–5

4. BASIS-Bench Evaluation

• Benchmark-eligible models must support speech input; Figure 2 and Appendix Table 7 summarize three axes and four evaluation modes.Appendix Tables 8, 9 and 10 provide additional details for demographic robustness, stereotypical bias and personalization, respectively.Source: §4 · p. 5

4.1. Demographic Robustness Results

• Most models perform robustly on content-based tasks across demographic groups, but gender classification is easier than age classification, with elderly speakers harder overall.DeSTA2.5 has the largest degradation on accented audio; Qwen3-Omni improves over Qwen2.5-Omni on elderly and child classification.Source: §4.1 · p. 5

4.2. Stereotypical Bias Results

• Providing demographics in text substantially raises stereotypical selections, especially for female and elderly speakers, even without instructions to condition on them.The strongest biases span gender and age for Qwen3-Omni, MOSS-Audio, and Voxtral, while Step-Audio2 is particularly biased for gender.Source: §4.2 · pp. 5–6
• In explicit mode, speech generally reduces stereotypical behavior versus text; in implicit mode, it increases elderly-speaker stereotyping in all but AF-Next, though interaction effects are small overall.Implicit-mode age shifts average +4 points for elderly speakers and -3.2 for young speakers; gender shifts are less consistent.Source: §4.2 · p. 6
• Speech-implicit balance can reflect limited cue use rather than fairness: strong classifiers may show little bias until age is explicit, while weak classifiers can appear balanced.MOSS-Audio and Qwen3-Omni classify elderly speakers well yet show little implicit stereotypical behavior, then become the most biased when age is explicit.Source: §4.2 · pp. 6–7

4.3. Personalization Results

• Text-explicit gender personalization is near-ceiling; for age, child personalization is easier than adult personalization, although models default adult-like and show no strong implicit profile preference.Only DeSTA2.5, Qwen3-Omni, and Voxtral exceed 77% for adult personalization; AF-Next and Step-Audio2 score 34.9% and 42.1%, respectively.Source: §4.3 · p. 7
• Speech-implicit gender personalization follows backbone preferences despite reliable classification, while age shifts strongly toward children; adult-persona difficulty also persists with text-explicit prompts.For age, child-speaker accuracy rises by 61.1 points on average versus text-implicit, while adult-speaker accuracy falls by 47.2 points.Source: §4.3 · pp. 7–8
• Gender personalization and stereotypical behavior trade off, while age shows a weaker version of the same pattern that becomes clearer with explicit demographics.Improving demographic perception alone is insufficient; models must learn when demographic information should shape responses and when to ignore it.Source: §4.3 · p. 8

5. Conclusion

• Across eight models, perceiving demographic information does not imply using it appropriately; demographic-dependent behavior often remains shaped by language-model priors.Models also struggle to ignore demographic information when it is irrelevant.Source: §5 · p. 8

Limitations

• The evaluation covers stereotypical bias, but not intersectional bias or all sources of demographic disparities.A comprehensive evaluation requires datasets covering broader forms of bias.Source: Limitations · p. 8
• The binary male/female gender categories are a controlled experimental abstraction, not a comprehensive representation of gender identities and expressions.Source: Limitations · p. 8
• Because gender and age are not always reliably inferable from speech, results measure models’ use of perceived cues, not identification of speakers’ identities.Source: Limitations · p. 8
• Accent is evaluated only for demographic robustness, since appropriate accent-based personalization is difficult to define without assumptions about background, preferences, or cultural identity.Accent-based personalization and bias evaluation are left to future work.Source: Limitations · pp. 8–9
• The manually reannotated Spoken StereoSet labels may reflect shared assumptions or blind spots because the annotators were contributors using the same guidelines.The process enabled consistent application of the criteria but reduced annotator diversity and number; broader validation is proposed.Source: Limitations · p. 9
• Personalization datasets test coarse age-based tone adaptation and binary gender-linked choices, not comprehensive age-aware personalization or gender-aware healthcare reasoning.BASIS-Health’s relevant medical distinction is biological sex, which is not equivalent to perceived gender inferred from speech.Source: Limitations · p. 9
• Synthetic speech may not capture human acoustic variability and naturalness, and its demographic cues may differ from naturally occurring ones.Source: Limitations · p. 9

A.1. Spoken StereoSet+

• Spoken StereoSet+ applies an explicit definition of stereotypical bias and annotation guidelines to filter Spoken StereoSet, yielding a smaller, more consistently defined benchmark.The source dataset’s stereotype labels did not resolve ambiguity about what counts as a stereotype.Source: §A.1 · p. 13

A.1.1 Stereotype and Stereotypical Bias

• Stereotypical bias means harmful stereotyping that uses group membership to draw unwarranted inferences about an individual’s characteristics, abilities, preferences, or social roles.Stereotypes need not be false or inherently harmful.Source: §A.1.1 · p. 13

A.1.2 Annotation Protocol

• The protocol excludes examples confounded by multiple demographics, invalid or indistinguishable continuations, illegal or aggressive content, asymmetric comparisons, or demographic-marker shortcuts.These filters aim to isolate individual demographic variables and the intended stereotype association.Source: §A.1.2 · pp. 13–14
• For age bias, the benchmark excludes legitimate child-related personalization and narrows comparison to male adult versus male elderly speakers to reduce gender confounding.Source: §A.1.2 · p. 14
• For gender bias, the protocol removes child and elderly contexts, harassment or sexist behavior, and cases where a stereotype is only implied rather than directly expressed.Source: §A.1.2 · p. 14

A.1.3 Final Dataset

• The filtered age split contains 67 unique context-continuation examples and 246 test instances for male adult and male elderly speakers.Source: §A.1.3 · p. 14
• The filtered gender split contains 242 examples and 1,434 test instances, with approximately six voices per example.Source: §A.1.3 · pp. 14, 16
• Table 1 lists Spoken StereoSet examples alongside each test set’s target demographic and the annotation issues identified during filtering.Source: Table 1 · p. 15
• The original age split used for ASR contains 11 speakers across elderly, young-adult, and child groups; missing speaker-context pairs were synthesized for symmetric evaluation.Instances with WER > 0 against the whisper-normalized reference were filtered out.Source: §A.1.3 · p. 16

A.2. ELIP-Pair

• ELIP-Pair pairs each of 521 questions with responses for child and adult profiles designed to create a clear stylistic contrast.The adult profile uses formal, expert-oriented style; the child profile uses friendly, humorous style, with concise informativeness in both.Source: §A.2 · p. 16
• The authors reproduced ELIP’s generation pipeline, selecting open-ended ELI5 prompts, manually curating them, removing semantic duplicates, and generating contrastive answers.The resulting benchmark grew from 100 to 521 questions.Source: §A.2 · p. 16
• Speech synthesis used two elderly, two adult, and four child voices per question, with male and female speakers among the adult and elderly voices.Only synthetic voices were cloned.Source: §A.2 · p. 16
• After transcription-fidelity checks and manual review of age-classifier outliers, the dataset contains 3,697 audio files, roughly 460 per speaker.The final splits differ by 17 instances, so text-implicit accuracies do not sum exactly to 100%.Source: §A.2 · p. 17

A.3. BASIS-Health and BASIS-Product

• Gender personalization is formulated as selecting the sole response whose explicit gender marker matches the speaker, reducing ambiguity with stereotypical bias.BASIS-Health uses answers dependent on biological sex; BASIS-Product uses otherwise identical product options differing by gender marker.Source: §A.3 · p. 17
• BASIS-Health contains gender-neutral requests with gender-specific responses, while BASIS-Product uses fictional brands and gender-marked options.The datasets use LLM-as-judge filtering; synthesized questions with non-zero WER were discarded, leaving 1,268 Health and 803 Product samples.Source: §A.3 · p. 17
• Table 7 reports BASIS-Bench results; Tables 8 and 9 cover demographic robustness and Spoken StereoSet+ gender and age results, while Table 10 reports BASIS-Health/Product MCQA accuracy.The supplied captions identify these tables’ scope but do not provide their result values.Source: Table 7 · p. 19, Table 8 · p. 20, Table 9 · p. 20

B.1. Model Inference

• Inference uses vLLM, Hugging Face Transformers, or SGLang by model, with local runs on one A100-80GB GPU and temperature fixed at zero.The LLM-as-judge uses the OpenAI API.Source: §B.1 · p. 22

B.2. Inference Prompts

• Prompts are consistent across models: demographic classification identifies speaker attributes, bias evaluation responds to audio, and personalization explicitly requests demographic adaptation.Age and gender personalization use separate prompts.Source: §B.2 · p. 22

B.3. Evaluation Prompts

• For open-ended SQA, an LLM judge classifies answers as correct or incorrect against the textual question and gold answer, using the VoiceBench prompt.The age-personalization judge classifies responses as suitable for child or adult profiles using response preferences, questions, and reference answers.Source: §B.3 · p. 22
• Table 11 gives the demographic-classification prompt template, which asks an audio classifier to choose between two speaker attributes and answer with one word.The prompt’s attribute and answer choices are placeholders instantiated for each task.Source: Table 11 · p. 22, §B.3 · p. 22
• Table 12 is captioned as presenting prompts for stereotypical-bias evaluation in SpokenStereoSet+, without establishing a particular prompt source or the prompts’ full content.Source: Table 12 · p. 22
• Age and gender personalization prompts distinguish speech and text conditions, with explicit demographic wording added only in explicit conditions.Tables 13 and 15 cover gender and age personalization prompts, respectively.Source: §B.3 · p. 22, Table 13 · p. 22, Table 15 · p. 23
• The speech and text system prompts frame adaptation around demographics perceived from the voice or defined in the question, and the user prompt selects the most appropriate candidate response.The gender prompt asks for the option tailored to a gender word and requires an A-or-B answer.Source: §B.3 · p. 22
• Table 16 contains the LLM-as-judge template for age personalization, while Table 14 lists evaluated speech LLMs and their repositories and licenses.The table captions identify their contents but do not provide the entries.Source: Table 14 · p. 23, Table 16 · p. 24

C.1. Models

• The appendix names Qwen3TTS models for speech synthesis and states that the benchmark models are not redistributed.The custom-voice model is the main TTS model; the base model is used for voice cloning.Source: §C.1 · p. 25

C.2. Datasets Used

• Fair-Speech, SD-QA, LibriSQA, Spoken StereoSet, and EchoMind have distinct license terms, and several are not redistributed.The paper points readers to original sources for Fair-Speech, SD-QA, and LibriSQA; Spoken StereoSet is modified into Spoken StereoSet+, whose license is described separately.Source: §C.2 · p. 25
• The paper identifies Fair-Speech under its custom agreement, SD-QA under Apache-2.0, LibriSQA recordings under CC-BY-4.0, Spoken StereoSet under CC-BY-4.0, and EchoMind under CC-BY-NC-SA-4.0.The textual LibriSQA material has no specific license, and the authors do not redistribute it.Source: §C.2 · p. 25
• EchoMind samples from its synthetic split are used to clone child and elderly speakers, with the dataset’s license taken into account for redistribution.Source: §C.2 · p. 25

C.3. Datasets Created or Modified

• ELIP-Pair and Spoken StereoSet+ are released under CC-BY-NC-SA-4.0.Source: §C.3 · p. 25
• BASIS-Health/Product is released under NAVER LABS Europe’s non-commercial license.Source: §C.3 · p. 25

C.4. Code

• The code, including prompts for data creation, validation, inference, and evaluation, will be distributed under Apache-2.0.Source: §C.4 · p. 25