Source-linked AI summary
TVPara: Paralinguistic Captioning of Expressive TV-Series Speech in Complex Acoustic Environments
Ziyang Zhang, Wenyi Yu, Wen Wu, Xuenan Xu, Chao Zhang
TL;DR
LALMs struggle to describe paralinguistics in expressive speech with complex acoustic backgrounds, and existing corpora offer limited matching fine-grained annotations. TVPara builds two paralinguistic captioning tasks from TV-series speech using LALM pre-annotation and human correction. Fine-tuning on the dataset yields a 31.26% average relative improvement, with gains generalizing to an unseen series and cleaner cross-domain speech.
Problem
LALMs struggle with speech combining intense emotion and complex acoustic environments, while existing corpora rarely provide fine-grained captions for both.
Source: §1 · p. 1Method
TVPara is a TV-series paralinguistic captioning dataset with Single-Utterance and Multi-Utterance tasks, annotated through LALM pre-annotation and human correction.
Source: §1 · pp. 1–2, §2.2 · p. 2Results
31.26% average relative improvement in paralinguistic understanding under complex acoustic environments is reported for fine-tuning on TVPara, with gains generalizing to an unseen TV series and cleaner cross-domain speech.
Source: §1 · pp. 1–2Takeaways & Limitations
TVPara provides fine-grained free-form annotations for expressive speech with complex sound, including attribute changes across adjacent utterances that existing corpora often leave uncovered.
Source: §1 · pp. 1–2For Qwen2.5-Omni, predicted volume levels correlate only ρ≈0.03 with measured clip energy before and after fine-tuning, and fine-tuning shifts outputs toward training labels.
Source: §4.1 · p. 4
Abstract
from arXiv · showhide
Large audio-language models (LALMs) struggle to parse paralinguistics (e.g., emotion, pace, pitch) in speech that combines intense emotion with complex acoustic environments. Strengthening this ability calls for training data that reflects such conditions. However, existing corpora rely heavily on studio-recorded speech, offering limited emotional and acoustic diversity. To address this gap, we introduce TVPara, a paralinguistic captioning dataset built from TV series. Speech in TV series often carries diverse emotional expression and co-occurs with scene-specific background sound, providing expressive emotion and complex acoustic environments at scale. Constructed via LALM pre-annotation and human correction, TVPara includes two tasks: Single-Utterance (~492h) and Multi-Utterance (~213h) Paralinguistic Captioning. Fine-tuning open-source LALMs on TVPara yields an average relative improvement of 31.26% in paralinguistic understanding under complex acoustic environments. The gains further generalize to an unseen TV series and to cleaner, cross-domain speech.
1. INTRODUCTION
LALMs struggle with speech combining intense emotion and complex acoustic environments, while existing corpora rarely provide fine-grained captions for both. TVPara uses TV-series speech for two captioning tasks and fine-tuning on it improves paralinguistic understanding, with gains extending beyond the training series and domain.
1. INTRODUCTION
- Existing corpora are often studio-recorded or otherwise clean, while wild-speech corpora with background sound generally provide labels rather than free-form captions.The paper identifies a lack of fine-grained annotations combining expressive emotion and complex acoustic environments.Source: §1 · p. 1
- Fine-tuning open-source LALMs on TVPara yields a 31.26% average relative improvement in paralinguistic understanding under complex acoustic environments, with gains generalizing to an unseen series and cleaner cross-domain speech.The reported average relative improvement is stated in the dataset overview; the generalization scope is also reported there and in the contributions.Source: §1 · pp. 1–2
- TVPara includes about 492 hours of Single-Utterance and 213 hours of Multi-Utterance Paralinguistic Captioning data.The two tasks describe attributes of one utterance and changes across adjacent utterances, respectively.Source: §1 · p. 1, Figure 1 · p. 1
- LALM pre-annotation followed by fine-grained human correction is used to support annotation accuracy.The pipeline combines automatic pre-annotation with human correction.Source: §1 · p. 1
- TVPara pairs expressive emotion and complex acoustic environments in TV-series speech with fine-grained free-form captions.TV-series speech often co-occurs with scene-specific background sound, providing these conditions at scale.Source: §1 · pp. 1–2
- Multi-Utterance Paralinguistic Captioning describes adjacent utterances in order and captures how paralinguistic attributes change across them.The paper notes that most existing corpora annotate utterances in isolation and leave this task uncovered.Source: §1 · pp. 1–2
2. THE TVPara DATASET
TVPara pairs expressive speech and background sound with fine-grained captions, including a task for describing changes across adjacent utterances. Its annotations combine LALM pre-annotation with human correction, and its test splits include an unseen series and held-out audio.
2. THE TVPara DATASET
- TVPara’s single-utterance task captions one utterance, while its multi-utterance task describes adjacent utterances in order and tracks attribute changes.The multi-utterance clips contain 3–5 adjacent utterances from the same episode.Source: §2 · p. 2, §2.2 · p. 2
2.1. Emotion and Background Sound
- 67.2% of TVPara clips have non-neutral emotion versus 2.9% for VoxCeleb, and adjacent utterances differ in emotion category in 55.5% of multi-utterance clips.TVPara clips contain background sound about 4.5 times as often as VoxCeleb clips.Source: §2.1 · p. 2
2.2. Annotation Pipeline
- Each single-utterance caption is LALM-pre-annotated, LLM-polished, and human-corrected against audible cues; after about 50 hours of corrections, the pre-annotation model is fine-tuned for remaining audio.Annotators changed about 38% of initial pre-annotations and about 17% after model fine-tuning.Source: §2.2 · p. 2
- Multi-utterance captions merge the corrected single-utterance captions in time order, and annotators changed only about 4% of a 5-hour reviewed sample.The longer clips are cut from source recordings to preserve scene acoustic continuity.Source: §2.2 · p. 2
2.3. Data Splits
- The unseen-series test set holds out a whole series and its speakers; a separate hold-out test uses continuous episode portions, with no test audio shared with training.The hold-out test contains about 5 hours, leaving about 487 hours for training.Source: §2.3 · pp. 2–3
3. EXPERIMENTAL SETUP
The experiments fine-tune two open-source LALMs on shuffled single- and multi-utterance data, then evaluate on TVPara and cleaner cross-domain speech. Scores are based on attribute-level agreement between generated captions and human-corrected labels.
3.1. Training
- SALMONN and Qwen2.5-Omni are fine-tuned with supervised training on shuffled single- and multi-utterance data, updating the cross-modal adapter and LoRA weights while freezing the audio encoder.One checkpoint serves both tasks.Source: §3.1 · p. 3
3.2. Evaluation
- Evaluation covers TVPara hold-out and unseen-series sets, plus 134 VoxCeleb single-utterance clips representing cleaner cross-domain interview speech.VoxCeleb has light background sound and mostly neutral and calm emotions.Source: §3.2 · p. 3, Table 4 · p. 4
3.3. Metrics
- An LLM judge extracts and compares attributes at the text level against the human-corrected ground-truth label, assigning each attribute a similarity score from 0 to 4, without receiving audio.Source: §3.3 · p. 4
4. RESULTS AND ANALYSIS
Fine-tuning improves paralinguistic captioning on TVPara, with evidence of audio-linked prediction gains for some attributes and a volume limitation for Qwen2.5-Omni. Improvements also extend to an unseen series and cleaner VoxCeleb speech.
4.1. Main Results on TVPara
- 31.26% is the mean per-cell relative improvement of fine-tuning over zero-shot on TVPara, with nearly every gain significant at p<0.01.Both models score low on most attributes in zero-shot, especially on the multi-utterance task.Source: §4.1 · p. 4
- SALMONN’s predicted volume correlates with measured clip energy at ρ=0.85 after fine-tuning versus 0.03 zero-shot, while Qwen2.5-Omni’s described utterance count correlates at ρ=0.83 versus 0.05.These correlations are reported as evidence that some gains reflect audio perception rather than only label-distribution fit.Source: §4.1 · p. 4
- Qwen2.5-Omni’s volume predictions remain at ρ≈0.03 correlation with clip energy before and after fine-tuning, which shifts its output distribution toward training labels instead.The authors attribute this limitation to the model’s smaller audio adapter and a representation that may not encode every paralinguistic attribute.Source: §4.1 · p. 4
4.2. Out-of-Distribution Generalization
- On the unseen series, SALMONN improves all four attributes in both tasks, while Qwen2.5-Omni improves three single-utterance and two multi-utterance attributes; gains are larger with background sound.On VoxCeleb, SALMONN improves all four attributes and Qwen2.5-Omni all but pace; SALMONN volume tracks energy at ρ=0.87 on the unseen series and ρ=0.73 on VoxCeleb.Source: §4.2 · p. 4
5. CONCLUSION
The conclusion presents TVPara as a dataset and two-task framework for expressive, acoustically complex speech, and reports improvements that transfer beyond the training series.
5. CONCLUSION
- TVPara pairs expressive emotion and complex acoustic environments with fine-grained captions, including descriptions of changes across adjacent utterances.The paper identifies limited emotional and acoustic diversity and isolated-utterance annotation as gaps in existing corpora.Source: §5 · p. 4
- 31.26% is the average relative improvement in paralinguistic understanding under complex acoustic environments after fine-tuning the tested open-source LALMs on TVPara.The conclusion reports limited understanding by the two tested models before fine-tuning.Source: §5 · p. 4
- The improved ability transfers to an unseen TV series and to cleaner cross-domain speech.Source: §5 · p. 4
6. COMPLIANCE WITH ETHICAL STANDARDS
TVPara uses scripted performances by professional actors from publicly available official series videos, rather than recordings from private individuals. The authors report that no ethical approval was required and that annotators were paid under local labor laws.
6. COMPLIANCE WITH ETHICAL STANDARDS
- The audio comes from official YouTube channels, and the release contains annotations, source-video links, and clip timestamps—not audio.The speech is scripted performance by professional actors rather than recordings collected from private individuals.Source: §6 · p. 5
- The authors state that the work is not human-subject research and required no ethical approval; commercial annotators were paid in accordance with local labor laws.Source: §6 · p. 5
7. FUNDING ACKNOWLEDGEMENTS
The authors report that the study received no funding and disclose no relevant financial or nonfinancial interests.
7. FUNDING ACKNOWLEDGEMENTS
- The study received no funding, and the authors disclose no relevant financial or nonfinancial interests.Source: §7 · p. 5