PaperlayerView exact revision on arXiv

Source-linked AI summary

A Benchmark for Controllable Speaking-Style Captioning in Audio-Language Models

Shreeram Suresh Chandra, Ashwyn Sharma, Liang Lu, Xiutian Zhao, Leda Sari, Björn Schuller, Berrak Sisman

arXiv:2610.33036v2cs.CLcs.AIcs.SDeess.AS

TL;DR

Speaking-style captioning lacks a unified formulation for describing targeted dimensions of speech, and existing approaches may treat style differently. This paper frames it as instruction-conditioned generation and introduces a benchmark; evaluations show instruction-following does not guarantee groundedness, while SSC-specific training improves groundedness and speaker consistency.

  • Problem

    Speaking-style captioning lacks a unified formulation, and differing views of its interacting style dimensions make evaluation and comparison difficult.

    Source: §1 · p. 1

  • Method

    The paper formulates SSC as instruction-conditioned caption generation and introduces StyleInstructCaps, a benchmark with multiple evaluation dimensions.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Strong instruction-following does not guarantee groundedness, while SSC-specific training improves groundedness and speaker consistency.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Instruction-following and groundedness need separate assessment because strong performance on the former does not ensure the latter.

    Source: §6 · p. 8

    LLM-based assessments may remain sensitive to prompt wording, decoding configurations, or stylistic biases despite robustness efforts and adversarial calibration.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Speaking-style captioning (SSC) aims to generate natural language descriptions of how speech is delivered, capturing paralinguistic attributes such as vocal timbre, prosody, and expressivity. Many downstream applications, including conversational AI agents, controllable speech generation, and large-scale audio annotation, require dimension-specific style captions that describe targeted aspects of speech (e.g., speaker traits, emotion, or delivery style), rather than a single undifferentiated description. However, existing work lacks a unified formulation for controllable speaking-style captioning. We address this gap by formulating SSC as an instruction-following audio-language task, where explicit instructions specify the speaking-style dimensions to be described. We introduce StyleInstructCaps, the first standardized benchmark for controllable speaking-style captioning in audio-language models. StyleInstructCaps provides a dataset and evaluation framework that measures metadata groundedness, hallucination, instruction-following ability, generalization to unseen instructions and audio datasets, and speaker-style consistency. Across nine state-of-the-art audio-language models, we find that strong instruction-following ability does not necessarily imply grounded speaking-style understanding, and that SSC-specific instruction-diverse training substantially improves groundedness, controllability, and speaker consistency.

1 Introduction

The introduction frames speaking-style captioning as a subjective, multi-faceted task that lacks a unified formulation and evaluation paradigm. It proposes explicit style instructions and StyleInstructCaps to evaluate controllable captions, reporting that instruction-following does not ensure grounded understanding while SSC-specific training improves performance.

1 Introduction

  • SSC lacks a unified task formulation and evaluation paradigm, while differing views of style make model comparisons difficult.Speaking style spans interacting factors such as identity, affect, prosody, and contextual delivery.Source: §1 · p. 1
  • Explicit style instructions let the same utterance produce targeted descriptions instead of one holistic caption that entangles style dimensions.This makes captions easier to interpret and evaluate systematically.Source: §1 · pp. 1–2
  • StyleInstructCaps evaluates metadata groundedness, hallucination, instruction-following, generalization, and speaker-style consistency.The benchmark addresses a limitation of systems focused narrowly on speaker- or emotion-centric style.Source: §1 · p. 2
  • The 2.7k-hour corpus pairs each utterance with six complementary caption types targeting distinct speaking-style dimensions.The parallel annotations support evaluation of selective attention to stylistic dimensions and integration of multiple cues.Source: §1 · p. 2
  • Strong instruction-following does not guarantee grounded speaking-style understanding, whereas SSC-specific, instruction-diverse training improves groundedness, controllability, and speaker consistency.The evaluation covers nine state-of-the-art audio-language models and SSC-tuned SALMONN variants.Source: §1 · p. 2

2 Related Work

Related work distinguishes open-ended descriptions of how speech is delivered from categorical or content-oriented tasks. It reviews existing datasets and model families, noting the absence of standardized controllable evaluation across stylistic dimensions.

2.1 Speaking-Style Captioning for Paralinguistic Speech Understanding

  • Speaking-style captioning produces open-ended descriptions integrating perceptual cues, unlike fixed-label paralinguistic tasks or content-oriented speech tasks.It describes how speech is delivered, including timbre, prosody, emotion, and delivery style.Source: §2.1 · p. 2

2.2 Datasets and Benchmarks

  • Most existing benchmarks emphasize recognition rather than controllable speaking-style caption generation, limiting consistent comparison and evaluation of selective attention to requested factors.Available resources include expressive-speech datasets and paralinguistic benchmarks, but they do not provide this controllable evaluation setting.Source: §2.2 · pp. 2–3

2.3 Speaking-Style Captioning Models

  • General-purpose audio-language models and specialized SSC systems can generate style descriptions, but prior work rarely standardizes instruction-level controllability across multiple stylistic factors.Specialized approaches include StyleCap, factor-conditioned methods, emotion-focused variants, and frameworks linking captioning with speech synthesis.Source: §2.3 · p. 3

3 Methodology

The methodology defines controllable speaking-style captioning as instruction-conditioned generation, then introduces StyleInstructCaps-DB as its multi-dimensional caption dataset. The benchmark builds on ParaSpeechCaps and supplies six complementary captions for each utterance.

3.1 Problem Formulation

  • Controllable SSC generates a caption describing only the style dimension requested by an explicit instruction, selectively grounding relevant attributes while avoiding unsupported or irrelevant ones.The formulation contrasts this with traditional SSC's single holistic description per utterance.Source: §3.1 · p. 3

3.2 Dataset Description

  • StyleInstructCaps-DB builds on ParaSpeechCaps, which combines human-annotated PSC-base data and automatically annotated PSC-scaled data.The source annotations cover intrinsic and situational speaking-style attributes across speakers and recording conditions.Source: §3.2 · p. 3
  • StyleInstructCaps-DB adds six complementary caption types per utterance, spanning speaker-specific, situational, expressive, linguistic-pragmatic, listener-centric, and overall style.Train and validation captions use Mistral-Small-24B-Instruct, while evaluation captions use Llama-3.1-8B-Instruct to avoid evaluating against captions generated by the same model.Source: §3.2 · p. 3

4 StyleInstructCaps Benchmark

The benchmark evaluates speaking-style captioning across groundedness, hallucination, instruction-following, generalization, and speaker consistency. Its evaluation protocols use metadata and instructions to score captions, and reliability checks compare LLM judges with perturbations, other judges, and human preferences.

4 StyleInstructCaps Benchmark

  • StyleInstructCaps evaluates metadata groundedness, hallucination, instruction-following, generalization to unseen instructions and speech data, and speaker-style consistency.The benchmark is built on StyleInstructCaps-DB.Source: §4 · p. 4

4.1 Metadata Groundedness

  • The groundedness task samples metadata-rich utterances and has an LLM judge compare captions with PSC metadata across speaker, acoustic, and expressive attributes.The MetaSet caps each speaker at 10 utterances and any accent at 25%; the judge reports mean 1–10 similarity across utterances, emphasizing prosodic alignment.Source: §4.1 · p. 4

4.2 Hallucination

  • Hallucination is an unsupported or contradictory speaking-style attribute, scored against speech metadata with both a 0–10 score and a Low, Medium, or High severity label.Non-contradictory descriptive or prosodic interpretations are treated leniently.Source: §4.2 · pp. 4–5

4.3 Instruction-Following

  • Instruction-following evaluation tests whether captions adhere to user-specified style dimensions, with judges scoring compliance and assigning Success or Failure.The benchmark reports both a mean 1–10 IFA score and the percentage of captions judged successful.Source: §4.3 · p. 5

4.4 Generalization to Unseen Instructions

  • Unseen-instruction generalization tests heterogeneous controls beyond the narrative training style, including structured schemas, strict output constraints, ratings, and persona prompts.Two utterances from each of 16 VCTK speakers are paired with all 36 unseen instructions.Source: §4.4 · p. 5

4.5 Generalization to Unseen Speech Dataset

  • Unseen-speech generalization uses 600 GigaSpeech segments paired with Speechcraft captions to assess groundedness and hallucination under varied recording conditions.Speechcraft provides less metadata coverage than PSC.Source: §4.5 · p. 5

4.6 Speaker-Style Consistency

  • Speaker-style consistency is measured through pairwise caption verification and clustering, on an in-domain set and an unseen-speaker VCTK set.The in-domain set contains 400 utterances; UD-VCTK contains 320 utterances from 16 speakers, and both use the same 10 speaker-specific instructions.Source: §4.6 · pp. 5–6

4.7 Are LLM judges reliable for the SSC task evaluation?

  • Human majority votes agreed with the LLM judge panel on 85.9% of caption pairs, and the human ranking recovered the judges' system ordering at Spearman ρ = 0.895.Agreement rose from 73.3% on near-ties to 93.3% on the largest score gaps.Source: §4.7 · p. 7

5 Experiment Setup

The experiments compare open-source audio-language models and SSC-specific SALMONN variants, using LLM judges for groundedness, hallucination, and instruction-following. The evaluation also examines speaker consistency and caption diversity.

5 Experiment Setup

  • Three LLM judges assess metadata groundedness and hallucination, while a single Qwen3-32B run scores instruction-following.Groundedness and hallucination use three runs per judge on StyleInstructCaps-MetaSet and two runs per judge on GigaSpeech and VCTK; instruction-following uses one run due to cost.Source: §5 · p. 7
  • Nine open-source audio-language models are evaluated, with OpenAI-GPT-Audio as a closed-source reference for core groundedness metrics.The study also instruction-tunes SALMONN under three configurations: original ParaSpeechCaps captions, single-task StyleInstructCaps summaries, and multi-task StyleInstructCaps captions.Source: §5 · p. 7
  • The analysis also measures lexical diversity using HD-D, Self-BLEU, Distinct-2, Distinct-3, and caption-length statistics.Preprocessing follows the ParaSpeechCaps setup.Source: §5 · p. 7

6 Results and Discussion

SSC-specific training improves groundedness over general-purpose models, but groundedness, hallucination, and instruction-following do not move together. Results also show that instruction-diverse tuning transfers across domains, while speaker consistency is strongly associated with the SALMONN backbone.

6 Results and Discussion

  • Metadata groundedness and hallucination severity are largely independent across models, so low hallucination alone does not indicate faithful style description.General-purpose models often struggle with SSC despite large-scale training.Source: §6 · p. 7
  • All three SSC-trained SALMONN variants exceed MOSS-Audio, the best general-purpose model, in metadata groundedness.The authors report the largest groundedness gains from SSC-specific training.Source: §6 · p. 8
  • Models with SSC-relevant audio encoders and tasks outperform models exposed only to ASR, which captures speech content rather than delivery.Audio-Flamingo-3 has strong MMAU and MMAR results but low SSC metascore and high hallucination; SALMONN benefits from its BEATs plus Whisper encoder and SSC-relevant tasks.Source: §6 · p. 8
  • On unseen speech, SALMONN variants show no further groundedness gain, while instruction-following varies by output format and model.MOSS-Audio scores highest in groundedness; Audio-Flamingo-3 and Qwen2-Audio drop on key-value slots versus narrative prompts, while MOSS-Audio and Qwen3-Omni remain strong across formats. The unseen speech evaluation uses a GigaSpeech-derived out-of-domain set.Source: §6 · p. 8, Table 3 · p. 8
  • SALMONN-based models yield the most speaker-consistent captions, an advantage already present in the untuned baseline; multitask instruction tuning provides the most balanced and transferable performance across both the ID-balanced set and UD-VCTK.Source: §6 · pp. 8–9

7 Conclusions

StyleInstructCaps frames controllable speaking-style captioning as an instruction-following audio-language task and pairs six complementary captions per utterance with a five-axis evaluation framework. The benchmark also measures speaker-style consistency through verification and clustering in caption-embedding space.

7 Conclusions

  • The benchmark reformulates controllable speaking-style captioning as an instruction-following audio-language task.Explicit instructions specify which speaking-style dimension a model should describe.Source: §7 · p. 9
  • Figure 3 visualizes speaker-style caption embeddings for a six-speaker subset of UD-VCTK using t-SNE.The passage identifies the visualization and its scope but does not state a comparative outcome.Source: Figure 3 · p. 9
  • StyleInstructCaps-DB provides six complementary style captions per utterance, while the benchmark evaluates groundedness, hallucination, instruction-following, generalization, and speaker-style consistency.Its consistency measures test whether captions preserve invariant speaker traits across utterances and whether same-speaker captions cluster coherently.Source: §7 · p. 9

Limitations

The paper identifies constraints in stylistic data coverage and several risks in its LLM-based evaluation framework. Its benchmark primarily tests metadata-grounded captioning, not fully audio-grounded speaking-style understanding.

Limitations

  • Public speech corpora with rich, fine-grained style annotations remain limited in scale and diversity, constraining coverage of real-world expressive speech.The stated coverage boundary spans speakers, domains, recording conditions, and cultural contexts.Source: Limitations · p. 10
  • Open-ended groundedness, hallucination, and instruction-following scores depend on the judge model, prompting strategy, and examples seen during training.The authors average metrics across three judges from different model families, whose system rankings agree closely (pairwise Spearman ρ of 0.864–0.978).Source: Limitations · p. 10
  • LLM assessments may remain sensitive to prompt wording, decoding settings, and stylistic biases despite prompt-robustness measures.Adversarial calibration cannot fully eliminate evaluation instability or hidden biases.Source: Limitations · p. 10
  • The evaluation has a potential closed-loop concern because metadata-derived descriptions inform both reference captions and LLM scoring.The paper says implicit alignment may remain and calls for further study of independent human annotation and audio-grounded verification.Source: Limitations · p. 10
  • The benchmark primarily evaluates metadata-grounded captioning rather than fully audio-grounded speaking-style understanding.References may reflect structured metadata without exhaustively capturing perceptually salient speech properties, so models may optimize for metadata-aligned descriptions instead of holistic expressive-speech representations.Source: Limitations · p. 10

Ethical Considerations

The paper limits its claims to controlled research and benchmarking, noting that paralinguistic understanding is not yet sufficiently reliable for high-stakes use. The supplied appendix passages describe dataset coverage, annotation checks, and StyleInstructCaps construction materials.

Ethical Considerations

  • The authors position results for controlled research and benchmarking, not as substitutes for human judgment in high-stakes paralinguistic assessments.They flag factual accuracy, mental-state inference, and sensitive judgments of speaker intent or affect as settings where misinterpretation could cause harm.Source: Ethical Considerations · p. 10

A Existing Datasets and Benchmarks

  • Table 5 summarizes existing datasets relevant to speaking-style captioning.The supplied passage does not provide the table’s dataset entries.Source: Table 5 · p. 15

B Annotation reliability of PSC style metadata

  • PSC annotation quality is checked through qualified human annotation and agreement filtering, complementary automatic annotation components, and human evaluation of scaled data.PSC-Scaled achieves tag recall comparable to PSC-Base for intrinsic and situational tags.Source: B Annotation reliability of PSC style… · p. 15

C StyleInstructCaps-DB Construction details

  • Construction materials include an EARS metadata example, a prompt template, distributions for metadata and speech properties, caption examples, and dataset-split statistics.These are identified in Figures 4–7 and Tables 7–8.Source: Figure 4 · p. 17, Figure 5 · p. 18, Figure 6 · p. 19

D LLM- judge calibration : Adversarial gaming check

The adversarial check finds that the judge penalizes manipulated captions across groundedness and instruction-following, while hallucination scoring is less consistent for hedged violations. The accompanying human-listener study tests judge preferences against audio-based choices and describes its study design and ranking metrics.

D LLM- judge calibration : Adversarial gaming check

  • For metadata groundedness, all five attack types lower scores; Hedged Hallucination and Cross-Speaker Swap produce the largest reported gaps, 7.46 and 6.41.Oracle captions average approximately 9.5; even Attribute Inversion yields a significant degradation (d = 1.39, p < 0.001).Source: D LLM- judge calibration : Adversarial… · p. 21, Table 9 · p. 22
  • Instruction Parroting reduces instruction-following scores from approximately 9.0 to 2.59, while Attribute Inversion has only a minor effect despite factual corruption.The reported effect sizes are d = 4.23, p < 0.001 for Instruction Parroting and d = 0.34, p < 0.05 for Attribute Inversion.Source: D LLM- judge calibration : Adversarial… · p. 21
  • Attribute Inversion and Cross-Speaker Swap receive the highest hallucination rates, but Hedged Hallucination is not consistently rated as highly severe.Hedged violations are still penalized under metadata groundedness and instruction-following, showing that the three dimensions are assessed as partially decoupled.Source: D LLM- judge calibration : Adversarial… · pp. 21, 23
  • Oracle captions score near ceiling while adversarial captions show statistically significant degradation across all three evaluation axes.The calibration tests whether the judge penalizes adversarial behavior rather than rewarding fluent or stylistically convincing text; Figure 8 presents the analysis.Source: D LLM- judge calibration : Adversarial… · p. 21, Figure 8 · p. 22

E Human-LLM judge correlation study

  • The human-listener study asks whether listeners agree, whether the judge selects the same caption, whether larger judge-score gaps predict agreement, and whether human votes recover the system ranking.The motivation is to test whether metadata-based judge scores align with what listeners perceive in speech.Source: E Human-LLM judge correlation study · p. 23, §E.1 · p. 23

E.1 Study design

  • The study compares two systems’ captions for each of 100 utterances, with 13 retained listeners choosing which better describes the speaking style after hearing the audio.The 14 systems include oracle captions and 13 captioning models; small judge-score-gap items were oversampled.Source: §E.1 · p. 23

E.2 Metrics

  • Listener agreement is assessed with chance-corrected and pairwise measures, while judge agreement compares preferences with the human majority and score-gap calibration tests whether agreement rises with gap size.A Bradley–Terry model fits system strengths to human votes alone, and Spearman’s ρ compares its ranking with each judge-induced ranking.Source: §E.2 · pp. 23–24

E.3 Results

The section validates the LLM judge against listener preferences, tests cross-judge and system-ranking reliability, and reports which system differences are statistically distinguishable. It also documents prompts used for benchmark evaluations.

E.3 Results

  • The judge ensemble matches the human majority on 85.9% of decided items, and human votes recover its system ranking at Spearman ρ = 0.895.Agreement rises from 73.3% on near-ties to 93.3% for the largest score gaps.Source: §E.3 · pp. 24–25, Table 11 · p. 24, Table 12 · p. 24

F Panel of LLM judges: cross-judge agreement

G Pairwise statistical significance of system rankings

H Prompts used

  • The appendix includes prompts for hallucination, instruction-following, unseen-instruction, and speaking-style similarity evaluations.The instruction-following prompts cover context and evaluation framework, then scoring rubric, edge cases, and output format.Source: Figure 9 · p. 29, Figure 10 · p. 30, Figure 11 · p. 31

I Model Training Details

Training results expose a trade-off between task specialization and stylistic breadth, while multi-task instruction tuning and broader data improve several outcomes. The section also reports diversity findings, training choices, and supplementary evaluation and resource details.

I Model Training Details

  • LoRA uses rank r=8, with scaling α=32 in training and α=16 at inference; inference attenuation is reported to dampen specialized updates while preserving general generation ability.The authors report reduced random stylistic drift and stronger anchoring to acoustic evidence.Source: I Model Training Details · p. 34
  • Uniformly mixing six tasks is intended to increase gradient diversity and discourage specialization; single-task tuning isolates supervision but makes memorization easier.Round-robin and scheduled curricula did not improve performance and sometimes degraded it.Source: I Model Training Details · p. 34
  • Training on the 282-hour PSC-base subset overfits limited tasks, yields safer and less expressive captions, and causes difficulty with untuned open-set questions; scaling and roughly 300 prompts per task improve quality, though overfitting persists.The paper argues that capacity control and objective design are needed in addition to data volume.Source: I Model Training Details · p. 35
  • The authors explore auxiliary ASR, gender, and accent objectives and factor-conditioned captioning to encourage acoustically verifiable attributes before free-form descriptions.The factor-conditioned target predicts attributes such as gender, accent, pitch, and speaking rate before the caption.Source: I Model Training Details · p. 35
  • Providing transcripts gives linguistic context for separating lexical content from delivery cues and is reported to improve contextually appropriate style descriptions.The examples include interpreting emphasis, question intonation, and sarcasm markers relative to discourse intent.Source: I Model Training Details · p. 35

J Lexical Diversity

  • Compared with single-task training, multitask tuning improves faithfulness and reduces hallucinations, while reducing repetition (Self-BLEU 0.206 vs. 0.315); its effects on other diversity measures are mixed.HD-D and Distinct-3 are slightly lower, and Distinct-2 is essentially unchanged.Source: J Lexical Diversity · p. 35, Table 20 · p. 36

K LLM Judgement; L Speaker Caption clustering plots

  • Figure 14 pairs an LLM judgment with ground-truth metadata and a generated caption; Figure 15 visualizes all UD-VCTK speaker-style caption embeddings with t-SNE.The captions identify the example and visualization, but do not report an outcome from either figure.Source: Figure 14 · p. 37, Figure 15 · p. 38

M Use of AI assistants; N Licensing

  • The paper reports GPT-5 and Gemini 3 Flash assistance for coding and language-related writing, and lists licenses for VCTK, GigaSpeech, ParaSpeechCaps, and SALMONN.ParaSpeechCaps is CC BY-NC-SA 4.0, which restricts commercial use and requires attribution and share-alike for derivatives.Source: M Use of AI assistants · p. 39, N Licensing · p. 39
Loading 2610.33036v2…