PaperlayerView exact revision on arXiv

Source-linked AI summary

StreamHear: Domain-Adapted Pseudo-Labeling for Semi-Supervised Streaming Speech Recognition

Zefang Liu, Chenyang Zhu, Sangwoo Cho, Xujun Peng, Shi-Xiong Zhang, Sambit Sahu

arXiv:2608.13717v1cs.CLeess.AS

TL;DR

Domain-shifted streaming ASR struggles when labeled in-domain audio is costly despite abundant unlabeled data. StreamHear adapts the streaming student with domain-adapted teacher pseudo-labels and realignment, consistently outperforming supervised fine-tuning across four datasets while narrowing the offline-teacher gap.

  • Problem

    Domain-shifted streaming ASR underperforms, while labeled in-domain audio is costly to prepare and unlabeled audio is abundant.

  • Method

    StreamHear fine-tunes an offline teacher on labeled in-domain data, pseudo-labels unlabeled audio for student training, and realigns words across chunks with prior-regularized dynamic programming.

  • Results

    Across four datasets, StreamHear consistently outperforms supervised student fine-tuning and narrows the gap to the offline teacher.

  • Takeaways & Limitations

    StreamHear provides a single-pass adaptation recipe whose gains remain robust across latency, context, pseudo-label pool, and student-architecture settings.

Abstract

from arXiv · show

Streaming automatic speech recognition (ASR) underperforms on domain-shifted target audio, where labeled in-domain data is costly to prepare while unlabeled audio is abundant. We present StreamHear, a semi-supervised pipeline that adapts a pretrained streaming student by fine-tuning an offline transducer teacher on the labeled training set, generating pseudo-labels on the unlabeled portion, and fine-tuning the student on the mixture. We further introduce a prior-regularized dynamic-programming realignment step that fixes chunk-level word placement using an ASR-hypothesis anchor. Across four datasets spanning financial calls, prepared read speech, and phone-quality dialogue, StreamHear consistently outperforms supervised student fine-tuning and narrows the gap to the offline teacher.

1. INTRODUCTION

StreamHear adapts pretrained streaming ASR to domain-shifted audio through offline-teacher pseudo-labeling and prior-regularized word realignment. Across four datasets, it outperforms supervised student fine-tuning and narrows the gap to the offline teacher.

  • Motivation: Pretrained streaming ASR underperforms on domain-shifted audio, while labeled in-domain fine-tuning is costly and unlabeled audio is abundant.Target challenges include entity-dense vocabulary, accented pronunciations, and narrowband acoustics.
  • StreamHear: StreamHear fine-tunes an offline transducer teacher on labeled in-domain data, pseudo-labels unlabeled audio, and fine-tunes the streaming student on the mixture.The pipeline is designed for adapting cache-aware streaming ASR to a target domain.
  • StreamHear: StreamHear introduces prior-regularized dynamic-programming realignment to redistribute ground-truth words across chunk boundaries using an ASR-hypothesis anchor.This corrects residual placement errors left by CTC Segmentation.
  • Evaluation: Across four datasets, StreamHear consistently outperforms supervised student fine-tuning and narrows the gap to the offline teacher.Ablations confirm robustness across latency and context configurations.

2. RELATED WORK

Prior ASR pseudo-labeling work evolved from early self-training into iterative pipelines and increasingly sophisticated methods for improving label quality. These methods include teacher averaging, filtering, retraining, and audio-aware correction.

  • Pseudo-labeling for ASR: ASR pseudo-labeling progressed from early self-training to iterative pipelines such as IPL and slimIPL.The passage identifies early self-training work and subsequent formalization into iterative pseudo-labeling pipelines.
  • Pseudo-labeling for ASR: Later methods improved pseudo-label quality using EMA teachers, checkpoint-averaged teachers, confidence and uncertainty filtering, incremental retraining, and audio-aware LLM correction.These approaches respectively include MPL, dynamic pseudo-label caches, data filtering, and correctors conditioned on source audio.

3. METHODOLOGY

StreamHear adapts an offline teacher to labeled in-domain data, uses it to pseudo-label unlabeled chunks, and fine-tunes a cache-aware streaming student on the combined data. A prior-regularized dynamic-programming realignment step repairs chunk-level word placement errors caused by VAD boundaries and non-speech tags.

  • StreamHear pipeline: StreamHear sequentially fine-tunes the offline teacher, generates pseudo-labels, and fine-tunes the streaming student on labeled and pseudo-labeled data.The teacher is adapted on DL, pseudo-labels are generated for DU, and MS is trained on DL ∪ D′U.
  • Teacher fine-tuning: Teacher fine-tuning domain-adapts the full-context transducer MT on the labeled in-domain set DL, improving the quality of subsequent pseudo-labels.The adapted teacher transcribes DU once with greedy decoding.
  • Pseudo-label generation: Pseudo-labels may be filtered by retaining the top-K% ranked by sequence-average teacher log-likelihood before student training.The confidence filter is optional.
  • Chunk-level data preparation: Chunk preparation segments recordings with VAD and assigns word timestamps using CTC-Segmentation, but residual placement drift remains near non-speech tags and VAD boundaries.Cache-aware streaming training requires each chunk to contain only words spoken within its time interval.
  • Dynamic-programming realignment: Prior-regularized dynamic programming realigns words by matching ground-truth and ASR-hypothesis sequences, rewarding matches while penalizing mismatches, skips, and chunk displacement.Traceback assigns matched words to hypothesis chunks, preserves unmatched-word assignments, and a two-pass sweep enforces monotonic chunk order.

4. EXPERIMENTS

The experiments cover dataset and chunk-level preparation, training and inference setup, main WER comparisons, and five targeted ablations. The ablations examine alignment correction, pseudo-label pool scaling, context sensitivity, per-latency retraining, and streaming student architecture.

  • The section describes the datasets, chunk-level preparation, and training and inference setup.
  • The main experiments compare WER against pretrained and fine-tuned baselines.
  • Five ablations probe alignment correction, pseudo-label pool scaling, context sensitivity, per-latency retraining, and streaming student architecture.

4.1. Datasets

StreamHear is evaluated on four English speech datasets, including financial calls, global-company earnings calls, professionally transcribed speech, and a proprietary call-center corpus. Long-form earnings-call recordings are standardized and segmented into chunks using VAD, forced alignment, and DP realignment anchored by Whisper hypotheses.

  • Dataset composition: The evaluation covers three public English corpora and one proprietary call-center dataset, including Earnings-21, Earnings-22, SPGISpeech, and BankCall.Earnings-21 contains 44 quarterly calls across nine financial sectors; Earnings-22 contains 121 effective calls from seven world-English regions after excluding four predominantly non-English files.
  • Preprocessing: Earnings-21 and Earnings-22 recordings are standardized to 16 kHz with unified non-speech tags before chunk-level training data are produced.The preprocessing pipeline applies these transformations to long-form earnings-call audio.
  • Preprocessing: Chunk generation combines pyannote-3.0 VAD segmentation, Parakeet-CTC-0.6B CTC-Segmentation forced alignment, and Algorithm 2 DP realignment using a Whisper-Large-v3 hypothesis anchor.VAD keeps each chunk within Whisper’s 30 s context so the anchor covers it in full.

4.2. Experimental Setup

The experiments compare an offline Parakeet teacher with cache-aware streaming FastConformer students, using a shared fine-tuning recipe. Streaming inference uses greedy decoding at right-context RC=1, with WER measured after text normalization and latency defined by lookahead.

  • Models: The offline teacher is Parakeet-TDT-0.6B-v3, while the primary streaming student is Nemotron-Speech-Streaming-EN-0.6B.Both are 0.6B-parameter FastConformer transducers; the student uses cache-aware streaming attention and multi-latency right-context sampling.
  • Inference and metrics: At RC=1, the streaming configuration uses an 80 ms chunk plus 80 ms right-context lookahead, yielding 160 ms of algorithmic latency.Latency is reported as (1+RC) × 80 ms imposed by right-context lookahead.
  • Training: All fine-tuning uses AdamW with learning rate 2 × 10−4, cosine scheduling, 10 epochs, bf16 precision, and SpecAugment.The recipe also uses betas [0.9, 0.98], weight decay 10−3, 10% warmup, minimum learning rate 10−6, and effective batch size 64 for the streaming student.
  • Inference and metrics: Inference uses greedy decoding with a cache-aware streaming attention mask, and WER follows extensive text normalization.Normalization includes bracket removal, number and typography normalization, lowercasing, contraction expansion, spelling and compound normalization, diacritic and filler-word removal, and punctuation removal.

4.3. Experimental Results

StreamHear consistently improves over fine-tuned streaming students across four datasets, including labeled and held-out unlabeled evaluations on three public benchmarks. Its BankCall gains are balanced across customer and agent channels, while pseudo-label transfer requires only a single fine-tuning pass.

  • Overall results: 0.18 to 0.88 percentage points (pp) lower word error rate on labeled test splits across all four datasets versus the fine-tuned student.On the held-out unlabeled split of three public benchmarks, improvements range from 0.44 to 1.85 pp.
  • Overall results: 0.44 to 1.85 pp lower word error rate on held-out unlabeled splits of three public benchmarks versus the fine-tuned student.These results are reported alongside labeled-test improvements of 0.18 to 0.88 pp across four datasets.
  • Method behavior: A single fine-tuning pass transfers pseudo-labels from a domain-adapted offline teacher to the streaming student without iterative refinement or auxiliary neural machinery.On BankCall, the method preserves the intrinsic ∼5.5 pp customer/agent WER gap attributed to acoustic quality rather than labeling asymmetry.
  • BankCall analysis: 0.88 pp gain on BankCall stereo audio is nearly symmetric across customer and agent channels, reaching streaming WERs of 12.55% and 7.00%.The corresponding differences versus the fine-tuned student are −0.90 and −0.85 pp, respectively.

4.4. Ablation Studies

Ablations show that StreamHear benefits from broad pseudo-label use, DP realignment, and a single checkpoint across latency settings. These gains persist with a multilingual student and are accompanied by improved alignment quality.

  • Experimental setup: The ablations use Nemotron-Speech-Streaming-EN-0.6B on Earnings-21 unless otherwise noted, with mean WER (%) over 5 seeds.All experiments use the same normalization pipeline as Table 2.
  • DP realignment: Realignment cuts alignment-quality WER by roughly 9 pp on both datasets while changing chunk count by at most 0.4%.The evaluation uses an independent Parakeet-TDT-0.6B audio anchor, while Whisper-Large-v3 supplies the position prior.
  • Pseudo-label filtering: Larger pseudo-label pools monotonically improve WER on Earnings-21 and Earnings-22; even K=25% outperforms the fine-tuned student.Discarding low-confidence pseudo-labels never helps, indicating that quantity outweighs top-K quality.
  • Student choice: StreamHear outperforms supervised FT with Nemotron-ML by 0.44 pp on labeled test and 2.28 pp on unlabeled data.The multilingual student uses Nemotron-3.5-ASR-Streaming-0.6B, trained at LC=56 with a prompt-conditioned decoder.

5. CONCLUSION

StreamHear adapts a cache-aware streaming ASR student using pseudo-labels from a domain-adapted offline teacher and prior-regularized dynamic-programming realignment. Across four datasets, it consistently outperforms the supervised streaming student.

  • 5. CONCLUSION: StreamHear adapts a cache-aware streaming ASR student with pseudo-labels generated by a domain-adapted offline teacher.The recipe fine-tunes the teacher on the target domain before pseudo-labeling.
  • 5. CONCLUSION: A prior-regularized DP realignment step redistributes ground-truth words across chunk boundaries using an ASR-hypothesis anchor.The realignment addresses word placement across streaming chunks.
Loading 2608.13717v1…