PaperlayerView exact revision on arXiv

Source-linked AI summary

NVV-Locator: From Transcript Tags to Acoustic Boundaries for Fine-Grained Nonverbal Vocalization Grounding

Yuang Cao, Bingshen Mu, Zhennan Lin, Guojian Li, Haoyue Zhan, Jie Liu, Chuan Xie, Qiang Zhang, Liumeng Xue, Lei Xie

arXiv:2609.09940v3eess.AScs.SD

TL;DR

NVV transcript tags provide limited evidence about acoustic boundaries, despite NVVs carrying affective and interactional information. The paper builds a unified taxonomy, timestamp-supervised data, and an expert-refined benchmark, then evaluates a non-autoregressive model for joint grounding. On NVV-TimeBench, it outperforms the evaluated large audio model counterparts across reported recognition and boundary metrics.

  • Problem

    Transcript-level NVV tags preserve event identity and textual context but not waveform onset, offset, or duration, while manual boundary annotation is costly to scale.

    Source: §1 · pp. 1–2

  • Method

    The paper unifies 26 NVV categories, constructs timestamp-supervised data, and proposes non-autoregressive slot filling for joint lexical and NVV grounding.

    Source: §1 · pp. 2–3

  • Results

    71.0% Micro F1 and 70.2% Macro F1, plus 80.4% Macro mIoU and 59.6 ms Macro mMAE, outperform the evaluated large audio model counterparts.

    Source: §4.2 · p. 11, §5 · p. 14

  • Takeaways & Limitations

    The paper presents a scalable approach spanning taxonomy construction, automatic timestamp annotation, benchmark curation, and model formulation.

    Source: §5 · p. 14

Abstract

from arXiv · show

Human speech includes nonverbal vocalizations (NVVs), such as laughter, sighs, breaths, and coughs, which convey affective and interactional information. Existing approaches typically represent NVVs as transcript-level tags, providing limited supervision for their waveform-time boundaries. We present NVV-Locator for fine-grained NVV temporal grounding. We first unify 26 NVV categories across public resources and construct large-scale timestamp-supervised training data through dual-LLM verification, transcript-guided forced alignment, and energy-based boundary refinement. We further introduce NVV-TimeBench, an expert-refined benchmark with 667 utterances and 1,094 events. NVV-Locator uses a non-autoregressive slot-filling architecture to jointly predict lexical timestamps, NVV categories, and event boundaries. On NVV-TimeBench, it achieves 71.0% Micro F1, 70.2% Macro F1, 80.4% Macro mIoU, and 59.6 ms Macro mMAE, outperforming the evaluated large audio model counterparts. Evaluation on an external corpus further demonstrates the cross-corpus generalization of NVV-Locator.

1 Introduction

NVVs convey affective and interactional information, but transcript tags do not preserve their waveform-time boundaries. The paper introduces a taxonomy, timestamp-supervised data and benchmark, and a model for joint lexical and NVV temporal grounding.

1 Introduction

  • Transcript-level NVV tags capture event identity and textual context but omit waveform onset, offset, and duration, while manual timestamp annotation is expensive to scale.Existing transcript-relative positions do not directly represent an NVV’s physical location in audio.Source: §1 · p. 2
  • The paper formulates NVV grounding as joint event recognition and waveform-time localization, supported by a unified 26-category taxonomy and scalable timestamp-supervised data construction.The model jointly predicts lexical timestamps, NVV categories, and NVV boundaries.Source: §1 · p. 2
  • NVV-TimeBench is introduced as a manually corrected benchmark with sub-20 ms inter-annotator boundary discrepancy.It is presented for fine-grained NVV temporal grounding.Source: §1 · p. 2
  • NVV-Locator uses non-autoregressive slot filling for joint lexical and NVV temporal grounding, mitigating temporal inconsistency from autoregressive timestamp decoding.The paper reports strong temporal grounding performance for the model.Source: §1 · p. 3

2 Dataset Construction

Dataset construction combines a unified NVV taxonomy, an automated pipeline for timestamp-supervised data, and an expert-refined benchmark. The pipeline verifies candidate events, estimates coarse regions from transcripts, refines acoustic boundaries, and augments training views.

2.1 A Unified Taxonomy of Nonverbal Vocalizations

  • The 26-category taxonomy harmonizes labels across four public resources, normalizing physically defined events and abstracting lexicalized interjections by function.For example, acoustically comparable throat-clearing labels are merged, while phonetic variants such as [Surprise-ah] and [Surprise-oh] map to [Surprise-X].Source: §2.1 · p. 3

2.2 Automatic Data Construction Pipeline

  • The four-stage pipeline builds timestamp-supervised training data, beginning with candidate acquisition and verification before alignment, boundary refinement, and augmentation.Figure 1 presents the pipeline overview.Source: §2.2 · p. 4, Figure 1 · p. 5

Phase 1: Candidate Data Acquisition and Dual-LLM Verification

  • Candidate instances come from annotated open resources, targeted synthetic data, and speech-only negative controls, then undergo sequential dual-LLM verification for presence, category consistency, and ambiguity.Human spot checks audit a randomly sampled subset of the automated verification decisions.Source: §2.2 · p. 4, Phase 1: Candidate Data Acquisition… · pp. 4–5

Phase 2: Transcript-Guided Coarse Alignment via LLM-ForcedAligner

  • Transcript-guided alignment estimates lexical timestamps, then brackets an NVV between adjacent lexical units as a candidate region rather than its final acoustic boundary.LLM-ForcedAligner estimates character-level timestamps for Chinese and word-level timestamps for English.Source: Phase 2: Transcript-Guided Coarse… · p. 5

Phase 3: Energy-Based Boundary Refinement

Phase 4: Class Balancing and Decoupled Data Augmentation

  • Class balancing and decoupled augmentation use noise injection and time stretching for underrepresented categories, with timestamps scaled alongside stretched audio.Refined utterances are represented as mixed speech, isolated NVV segments, and NVV-removed speech contexts.Source: Phase 4: Class Balancing and Decoupled… · p. 6

2.3 NVV-TimeBench: An Expert-Refined Benchmark for Fine-Grained NVV Temporal Grounding

  • NVV-TimeBench pairs audio and lexical transcripts with canonical-category onset and offset annotations, and evaluates event recognition alongside waveform-time localization.Curation includes expert annotation and review, with five experts independently annotating a random subset and two additional validators checking quality.Source: §2.3 · p. 6
  • 667 utterances and 1,094 target events make up NVV-TimeBench, covering all 26 categories across approximately 112.6 minutes of audio.Events average 0.52 s (median 0.40 s), and durations extend to nearly 8 s; the benchmark is category-diverse rather than strictly class-balanced.Source: §2.3 · p. 6

3 NVV-Locator: A Transcript-Conditioned Model for Fine-Grained NVV Temporal Grounding

NVV-Locator uses a lexical transcript as observed input to jointly estimate lexical boundaries and associated NVV categories and acoustic intervals. Its slot-filling design predicts labels in parallel, with noise-aware training objectives and event-level validation.

3 NVV-Locator: A Transcript-Conditioned Model for Fine-Grained NVV Temporal Grounding

  • NVV-Locator jointly estimates lexical boundaries and associated NVV categories and acoustic onset and offset times from audio and a lexical transcript.The transcript is treated as observed input rather than a generation target.Source: §3 · p. 7

3.1 Non-Autoregressive Slot-Filling Architecture

  • All inserted lexical and NVV slots are supervised without next-token shifting and predicted in parallel, with soft expectation decoding yielding continuous timestamps at sub-bin precision.The model is built on Qwen3-ASR-0.6B and uses a five-slot template after transcript positions.Source: §3.1 · pp. 7–8, Figure 2 · p. 7

3.2 Noise-Robust Training and Optimization Strategies

  • The audio encoder is frozen while the language-model decoder and prediction layer are fine-tuned to limit adaptation under noisy supervision and sparse NVV labels.The corpus may contain residual boundary noise, and NVV events are sparse relative to lexical units.Source: §3.2 · p. 8

Decomposed Slot Optimization and Conditional Masking

  • Separate losses supervise lexical timestamps, NVV types, and NVV timestamps, with NVV boundary loss applied only to valid NVV events.For batches without valid NVV events, event-time loss is zero; per-term normalization prevents dense lexical supervision from numerically dominating sparse NVV objectives.Source: §3.2 · p. 8, Decomposed Slot Optimization and… · p. 9

Event-Level Hungarian Matching Validation

  • Category-aware Hungarian matching selects checkpoints using event-level Macro F1 and boundary MAE over matched pairs rather than token-level accuracy alone.Validation uses a t-IoU threshold of 0.3, more permissive than the t-IoU ≥0.5 criterion used for final benchmarking.Source: Event-Level Hungarian Matching… · pp. 9–10

4 Experiments and Results

The experimental data pipeline yielded 283.20 hours of candidate audio, with 275.71 hours expanded through decoupled augmentation into 551.42 hours of training data. NVV-TimeBench was formed through expert refinement of part of the validation split, and four large audio models were evaluated zero-shot.

4.1 Experimental Setup

  • 551.42 hours of training data were produced by augmenting 275.71 hours of the 283.20-hour candidate corpus; 7.49 hours were reserved for validation and evaluation.Of the reserved audio, 1.88 hours were assigned to expert refinement and 5.61 hours to development validation.Source: Dataset · p. 10

Dataset

  • Four large audio models were evaluated zero-shot on NVV-TimeBench, prompted to output canonical categories and event boundaries under the same evaluation protocol as NVV-Locator.The compared models were Gemini-2.5-Pro, Qwen3-Omni-Instruct (30B), Step-Audio-R1.1, and MOSS-Audio-8B-Instruct.Source: Baselines · p. 10

Baselines

The evaluation measures event recognition and boundary accuracy using event-level F1, boundary error, and IoU. Both micro and macro averages are reported, with macro boundary metrics restricted to categories with matched events.

Baselines

  • Event-Level F1 is the primary event-identification measure, counting a prediction only when its category matches and t-IoU is at least 0.5.The matching criterion requires both category agreement and sufficient temporal overlap.Source: Evaluation Metrics · p. 10

Evaluation Metrics

  • mMAE averages absolute onset and offset errors over successfully matched pairs, while mIoU is an auxiliary measure over those same pairs.Source: Evaluation Metrics · p. 11
  • Micro averages pool category counts, whereas macro F1 averages category-wise F1 and macro mIoU and mMAE include only categories with at least one matched event.Source: Evaluation Metrics · p. 11

4.2 Main Results

NVV-Locator leads the reported aggregate results on NVV-TimeBench, while category-level performance varies with acoustic saliency and boundary ambiguity.

4.2 Main Results

  • 71.0% Micro F1, 70.2% Macro F1, 80.4% Macro mIoU, and 59.6 ms Macro mMAE; NVV-Locator leads aggregate results.Gemini-2.5-Pro is the strongest large audio model baseline, with 49.8% Macro F1, 72.8% Macro mIoU, and 92.6 ms Macro mMAE.Source: §4.2 · p. 11, Table 2 · p. 12

4.3 Fine-Grained Category Analysis

  • Whistle (96.9% F1, 44.2 ms mMAE), Cough (78.1%, 50.0 ms), Sneeze (89.3%, 47.0 ms), and Shush (91.2%, 28.7 ms) combine high F1 with precise boundaries.Their abrupt energy transitions and distinctive harmonic or broadband patterns provide clear localization cues.Source: §4.3 · p. 11
  • Laughter (83.7% F1, 112.6 ms mMAE) and Hum (95.2% F1, 129.4 ms) have high F1 but larger boundary errors; Applause and Yawn show a milder pattern.Periodic, gradually evolving structure makes exact onset and offset less sharply defined.Source: §4.3 · pp. 11–12
  • Lip Smack (21.1% F1), Gasp (16.7%), and Breath (47.7%) remain challenging, while Sniff reaches 85.0% F1 and 40.7 ms mMAE.Ultra-short Lip Smacks can be masked; diffuse respiratory energy and proximity to lexical boundaries increase ambiguity, whereas Sniff has a more stable spectral profile.Source: §4.3 · p. 12

4.4 Ablation Study: Validating the Data Construction Pipeline

The ablation study removes individual data-construction components while keeping the model and training protocol fixed; removing Dual-LLM Verification substantially degrades all three reported metrics.

4.4 Ablation Study: Validating the Data Construction Pipeline

  • Macro F1 is emphasized because it weights categories equally and is more sensitive to degradation in sparse NVV types; mIoU and mMAE assess boundary accuracy.The ablation removes Dual-LLM Verification, Energy-Based Boundary Refinement, or Decoupled Data Augmentation.Source: §4.4 · p. 12
  • Removing Dual-LLM Verification changes Macro F1 from 70.2% to 55.1%, mIoU from 80.4% to 74.9%, and mMAE from 59.6 ms to 73.2 ms.Table 3 reports variants that remove one automatic data-construction component while holding the downstream model architecture and training protocol unchanged.Source: Impact of Dual-LLM Verification (Phase… · p. 13, Table 3 · p. 13
  • Without verification, mislabeled, noisy, overlapping, or acoustically ambiguous samples are more likely to remain, especially for sparse categories.The paper links this noisy supervision to weaker category discrimination and boundary learning.Source: Impact of Dual-LLM Verification (Phase… · p. 13

Impact of Dual-LLM Verification (Phase 1)

Removing Energy-Based Boundary Refinement causes the largest reported localization degradation and also lowers Macro F1.

Impact of Dual-LLM Verification (Phase 1)

  • Without Energy-Based Boundary Refinement, mMAE rises from 59.6 ms to 133.6 ms, mIoU falls from 80.4% to 62.5%, and Macro F1 drops to 51.7%.Transcript-guided lexical gaps can include silence, noise, or speech-adjacent content, overestimating event extent and weakening onset–offset supervision.Source: Impact of Energy-Based Boundary… · p. 13
  • Coarse lexical-gap windows may include silence, background noise, or speech-adjacent content, producing imprecise event boundaries.Energy-based refinement converts those coarse gaps into acoustically grounded NVV boundaries.Source: Impact of Energy-Based Boundary… · p. 13
  • Category-correct predictions with t-IoU below 0.5 count as unmatched under Event-Level F1, affecting both precision and recall.Source: Impact of Energy-Based Boundary… · p. 13

Impact of Energy-Based Boundary Refinement (Phase 3)

Removing Decoupled Data Augmentation produces smaller but consistent declines across Macro F1, mIoU, and mMAE.

Impact of Energy-Based Boundary Refinement (Phase 3)

4.5 Cross-Corpus Generalization Evaluation

NVV-Locator is evaluated zero-shot on WESR-Bench after its labels are mapped to the unified taxonomy. Its results are of comparable overall scale to reported WESR-trained systems, although strict comparison is not possible.

4.5 Cross-Corpus Generalization Evaluation

  • 70.2% Micro F1 and 45.7% Macro F1 were achieved on WESR-Bench in zero-shot evaluation after label mapping to the unified taxonomy.WESR-Bench contains over 900 utterances with diverse vocal events; no dataset-specific fine-tuning was used.Source: §4.5 · pp. 13–14
  • WESR-trained systems reported 70.5–71.4% Micro F1 and 37.7–38.0% Macro F1, but strict comparison is not possible given taxonomy mapping and no target-corpus training.The paper describes the results as comparable in overall scale.Source: §4.5 · p. 14
  • The zero-shot results suggest NVV-Locator retains effective NVV event recognition beyond the NVV-TimeBench distribution.The evaluation is on WESR-Bench after mapping its labels to the unified taxonomy.Source: §4.5 · pp. 13–14

5 Conclusion

The conclusion presents a scalable approach to fine-grained NVV temporal grounding, combining taxonomy and data construction, benchmark curation, and joint recognition and waveform-time localization. The reported results outperform evaluated large audio model counterparts, with ablations and external-corpus evaluation supporting the approach.

5 Conclusion

  • The approach substantially outperforms evaluated large audio model counterparts, while external-corpus evaluation supports cross-corpus generalization.Ablation studies validate the importance of the data construction pipeline for both event recognition and boundary localization.Source: §5 · p. 14
  • The four-stage data construction pipeline combines dual-LLM verification, transcript-guided alignment, energy-based boundary refinement, and decoupled augmentation.The pipeline is used to create automatic timestamp annotations.Source: §5 · p. 14
  • The paper unifies 26 NVV categories and introduces NVV-TimeBench for fine-grained evaluation.Source: §5 · p. 14
  • NVV-Locator is a non-autoregressive slot-filling model for joint NVV recognition and waveform-time localization.Source: §5 · p. 14
Loading 2609.09940v3…