PaperlayerView exact revision on arXiv

Source-linked AI summary

Source-Adaptive Data Curation for Bilingual NVV-Aware ASR

Yuang Cao, Qirui Zhan, Jingbin Hu, Ziyu Zhang, Yunxiang Chen, Houdun Liu, Su Feng, Bengu Wu, Lei Xie, Liumeng Xue

arXiv:2609.09929v2eess.AScs.SD

TL;DR

Conventional ASR often omits NVVs, while bilingual tagged transcription requires lexical content, NVV identity, and transcript-relative placement amid supervision gaps. The paper adapts Whisper-medium for unified lexical–NVV decoding and curates public and in-the-wild training data; under the official protocol, FinalScore increases from 33.32 to 53.61.

  • Problem

    Conventional ASR often omits or collapses NVVs, while bilingual NVV-aware ASR must capture lexical content, NVV identity, and transcript-relative placement despite gaps in annotation quality and acoustic diversity.

    Source: §1 · p. 1

  • Method

    The paper adapts Whisper-medium for unified lexical and inline-NVV decoding and combines refined public corpora with NVVs mined and annotated from in-the-wild media.

    Source: §1 · p. 1, §2.2 · p. 2, §2.3 · p. 2

  • Results

    FinalScore improves from 33.32 to 53.61 under the official NVVSpeech Track 1 protocol.

    Source: §1 · p. 1, §4 · p. 4

  • Takeaways & Limitations

    Ablations confirm individual benefits from Data Augmentation, LLM-based Filtering, and Movie NVV Data, while fine-grained categories and English transcription robustness remain unresolved.

    Source: §4 · p. 4, §3.3 · p. 4

    The final training corpus remains Mandarin-dominant at 887.0 versus 291.8 h, and some fine-grained NVV categories remain difficult to recognize.

    Source: §3.2 · p. 4, §3.3 · p. 4

Abstract

from arXiv · show

Nonverbal vocalizations (NVVs), such as laughter, sighs, breaths, and coughs, convey affective and interactional information that conventional automatic speech recognition (ASR) systems often discard. We present a bilingual Mandarin-English system for Track 1 of the NVVSpeech Challenge at ISCSLP 2026, which requires joint transcription of lexical content and 16 NVV categories at their transcript-relative positions. Our NVV-Aware Whisper adapts Whisper-medium through checkpoint-compatible vocabulary remapping, enabling lexical tokens and inline NVV tags to be decoded within a unified autoregressive sequence without expanding the vocabulary. To provide reliable and diverse supervision, we further introduce a source-adaptive data curation strategy that refines public NVV corpora through acoustic augmentation and multimodal LLM filtering, while mining spontaneous NVVs from in-the-wild media through automated preprocessing and annotation. Under the official bilingual evaluation protocol, the proposed system improves final score from 33.32 to 53.61, with ablations confirming the complementary benefits of the proposed data-curation components.

1. Introduction

NVV-aware ASR must jointly capture lexical content, NVV identity, and transcript-relative placement, but existing supervision has quality, balance, and diversity gaps. The paper combines joint decoding with source-adaptive curation and reports a higher official Track 1 FinalScore.

1. Introduction

  • Existing corpora have heterogeneous annotation quality, category imbalance, and limited acoustic diversity; curated data can miss spontaneous events while in-the-wild media lack reliable annotations.These gaps matter for a challenge requiring bilingual transcripts with lexical content, NVV identity, and transcript-relative placement.Source: §1 · p. 1
  • The system remaps vocabulary entries for joint lexical-token and inline-NVV decoding, while curation refines public corpora and mines spontaneous events from in-the-wild media.The resulting complementary sources are combined for bilingual training.Source: §1 · p. 1
  • Under the official NVVSpeech Track 1 protocol, the proposed system improves FinalScore from 33.32 to 53.61, with controlled ablations quantifying contributions of major data-curation components.Source: §1 · p. 1

2. Method

The method formulates audio-only tagged transcription as joint prediction of lexical tokens, NVV categories, and their transcript-relative positions. It adapts Whisper-medium for unified decoding and builds complementary supervision from refined public corpora and annotated in-the-wild media.

2.1. Task Formulation

  • Track 1 predicts an audio-only transcript interleaving lexical tokens with 16 NVV categories at their transcript-relative positions, without visual input or a post-hoc aligner.The fixed inventory includes events such as [breath], [laugh], and [cough].Source: §2.1 · pp. 1–2

2.2. NVV-Aware Whisper

  • Whisper-medium is fully fine-tuned to generate lexical tokens and inline NVV tags in one autoregressive sequence, with timestamp prediction disabled.Sixteen unused BPE entries are remapped to atomic NVV labels, preserving vocabulary size and model dimensions; no separate event-classification or alignment module is used.Source: §2.2 · p. 2

2.3. Source-Adaptive Data Curation

  • Public-corpus refinement normalizes labels, augments waveforms with noise or speed perturbation, and filters examples with a multimodal LLM for acoustic and transcript consistency.Failed samples are discarded rather than automatically repaired, favoring annotation reliability over data quantity.Source: §2.3 · p. 2
  • For in-the-wild media, preprocessing and speech segmentation precede transcript scaffolding; a multimodal LLM then inserts, removes, corrects, or relocates inline NVV tags.Unlike the public-data branch, no waveform augmentation is applied, preserving subtle cues in the original recordings.Source: §2.3 · p. 2
  • Clean Public NVV Data and Movie NVV Data are combined with tag-free speech examples to provide complementary supervision and discourage unsupported NVV-tag generation.The public data provide controlled bilingual supervision, while movie data contribute spontaneous realizations and diverse acoustic contexts.Source: §2.3 · pp. 2–3

3. Experiments

Experiments evaluate source-adaptive data curation under the official bilingual Track 1 protocol, comparing systems with shared model and training settings. The final training corpus substantially improves bilingual FinalScore, while the analyses show language- and category-specific trade-offs and complementary gains from the data-curation components.

3.1. Experimental Setup

  • All systems share the NVV-Aware Whisper architecture, training configuration, decoding strategy, and 178.8 hours of tag-free ASR data; they differ in NVV-labeled training data.The final corpus contains 1,178.8 hours and 401,599 utterances; bilingual Track1Score averages Mandarin and English scores, with F1micro, mNTD, and tag-aware CER/WER measuring event recognition, placement, and transcription.Source: §3.1 · p. 3

3.2. Main Results and Analysis

  • English text-only WER rises from 25.58% to 34.98%, while Mandarin text-only CER changes from 48.15% to 47.38%; English tagged WER rises from 32.38% to 50.91%.The authors attribute the Mandarin gain primarily to NVV recognition and placement with lexical ASR largely preserved; English shows lexical degradation as well as additional NVV-related tagged-transcript errors.Source: §3.2 · pp. 3–4
  • 53.61 vs. 33.32 FinalScore: the proposed system gains 20.29 points, with Mandarin Track1Score rising from 34.05 to 55.45 and English from 32.59 to 51.76.The largest reported gains include improved NVV recognition and placement in both languages.Source: §3.2 · p. 3, Table 2 · p. 4

3.3. Per-Class Analysis

  • The proposed system improves 14 of 16 English NVV categories and 15 of 16 Mandarin categories, but some categories remain difficult and some scores decline.English [gasp], [breath], and [moan] remain at F1 scores of 0.224, 0.330, and 0.340; regressions include English [hiss] and [lipsmack] and Mandarin [snore].Source: §3.3 · p. 4

3.4. Data-Curation Ablation

  • 33.32 to 37.56 (+4.24) with Data Augmentation and then 39.27 (+1.71) with LLM-based Filtering; Movie NVV Data alone scores 46.34, and combining sources reaches 53.61.The combined data improves English from 38.47 to 51.76 and Mandarin from 54.21 to 55.45; the authors describe public-corpus refinement and in-the-wild media as complementary.Source: §3.4 · p. 4, Table 4 · p. 4

4. Conclusion

The bilingual NVV-aware ASR system combines joint lexical–NVV decoding with source-adaptive data curation. It improves the official Track 1 score, while analyses identify persistent category and English-transcription challenges.

  • FinalScore rises from 33.32 to 53.61 under the official NVVSpeech Track 1 protocol.Source: §4 · p. 4
  • Ablations confirm individual benefits from Data Augmentation, LLM-based Filtering, and Movie NVV Data.Source: §4 · p. 4
  • Per-class and text-only analyses show broad NVV gains but persistent challenges in fine-grained categories and English transcription robustness.Source: §4 · p. 4
  • Future work will target improved language balance and a weaker trade-off between NVV recognition and lexical accuracy.Source: §4 · p. 4
Loading 2609.09929v2…