Source-linked AI summary
RoleHear: Pseudo-Labeling for Semi-Supervised Role-Attributed Streaming Speech Recognition
Zefang Liu, Chenyang Zhu, Sangwoo Cho, Xujun Peng, Rongqing Huang, Shi-Xiong Zhang, Sambit Sahu
TL;DR
Streaming ASR produces unattributed words, while real-time applications need inline speaker roles and role labels are costly to obtain. RoleHear trains a cache-aware streaming transducer to emit roles inline using teacher-generated pseudo-labels and MLLM silver labels. On ServiceCall, it beats the Teacher + MLLM cascade on WER and RWER at both role sets, while trailing on F1.
Problem
Streaming ASR produces an unattributed word stream, although real-time applications need each word tied to a conversational role, and role annotations are costly to obtain at scale.
Source: §1 · p. 1Method
Results
On ServiceCall, RoleHear beats the Teacher + MLLM cascade on WER and RWER at both role sets, but trails it on F1.
Source: §4.3 · p. 4Takeaways & Limitations
Abstract
from arXiv · showhide
Streaming automatic speech recognition transcribes at low latency but produces an unattributed word stream, while real-time applications such as customer-service analytics need each word tied to a speaker role. Role attribution is usually a separate stage, either an offline or streaming speaker-diarization model aligned to the words or a text-based large language model applied to the transcript, and role labels for training are scarce. We present RoleHear, a semi-supervised pipeline that makes a single cache-aware streaming transducer emit roles inline through lightweight in-vocabulary role tokens: a fast turn-boundary marker followed by a per-role token a few words later. An offline transducer teacher, fine-tuned into a role-attributed recognizer, relabels unlabeled audio, and a multimodal large language model supplies silver role labels where gold annotations are unavailable. The streaming student is fine-tuned on the mixture in a single pass. Across customer-service calls with gold roles and earnings calls with silver roles, RoleHear attributes roles accurately while preserving transcription quality and narrowing the gap to the offline teacher.
1. INTRODUCTION
RoleHear addresses the need for low-latency transcripts with inline speaker roles, where existing attribution commonly relies on separate stages and role annotations are scarce. It combines decoupled role emission with single-pass pseudo-labeling and MLLM-generated silver labels.
1. INTRODUCTION
- Real-time applications need each word tied to a conversational role inline, while streaming ASR otherwise produces an unattributed word stream.The passage gives live summarization and agent assistance as examples.Source: §1 · p. 1
- Role annotations are costly to obtain at scale, whereas in-domain unlabeled audio is abundant.Source: §1 · p. 1
- RoleHear uses a single cache-aware streaming transducer to emit roles inline, separating a fast turn-boundary marker from a per-role token emitted a few words later.This separates turn detection from role commitment and avoids a separate diarizer.Source: §1 · p. 1
- An offline transducer teacher relabels unlabeled audio, and the streaming student is fine-tuned once on labeled and pseudo-labeled data.The pseudo-labeling pipeline does not iterate.Source: §1 · p. 1
- An MLLM supplies silver role labels for transcript-only corpora by conditioning on chunk audio and surrounding transcript context.The labels are validated against gold labels where available.Source: §1 · p. 1
2. RELATED WORK
Prior role-attributed ASR and diarization approaches target speaker identity, often relying on inventories, serialized decoding, or separate diarizers; streaming pseudo-labeling precedents do not target cache-aware role attribution.
2. RELATED WORK
- RoleHear combines closed-set role tokens on a cache-aware streaming transducer with single-pass pseudo-labeling, without speaker enrollment, clustering, a diarizer, or iterative refinement.The related-work passage contrasts this with speaker-attributed ASR, diarization, streaming pseudo-labeling, and audio-aware LLM correction approaches.Source: §2 · p. 1
3. METHODOLOGY
RoleHear adapts a cache-aware streaming transducer to emit inline role tokens and trains it using teacher-generated pseudo-labels alongside gold and silver role data. An MLLM assigns silver roles from chunk audio and transcript context.
3. METHODOLOGY
- RoleHear adapts a pretrained cache-aware streaming transducer into a role-attributed ASR model through single-pass pseudo-labeling.Source: §3 · p. 1
- Decoupled role emission places a fast turn-boundary marker at the boundary and a per-role token n words later, letting role decisions use in-turn context.Non-negative delay keeps student targets causal; the full-context teacher may place the role token turn-final.Source: §3 · p. 2
- The MLLM labels transcript-only corpora by assigning contiguous role segments from chunk audio, numbered transcript tokens, and neighboring chunk transcripts.The prompt returns role segments with token start indices; stereo contexts restrict roles to the target channel, while mono recordings omit that rule.Source: §3 · p. 2, Figure 1 · p. 2
- The full-context offline teacher is fine-tuned on gold and silver role data, then greedily decodes unlabeled audio once to create role-attributed pseudo-labels.Source: §3 · p. 2
- The streaming student is fine-tuned in a single pass on gold, silver, and teacher-pseudo-labeled data using the standard RNN-T loss.It emits role tokens inline at inference in cache-aware streaming mode, with no additional model.Source: §3 · p. 2
4. EXPERIMENTS
Experiments evaluate RoleHear on earnings calls with silver role labels and customer-service calls with gold labels, then compare its role attribution and transcription quality with baselines. Ablations examine how role-emission delay and right context affect latency and role metrics.
4.1. Datasets
- Evaluation spans two public transcript-only earnings corpora with silver roles and proprietary gold-labeled ServiceCall, which is evaluated with both 3-role and 5-role sets.Table 1 reports split duration in hours and average chunk length in seconds.Source: §4.1 · p. 3, Table 1 · p. 3
4.2. Experimental Setup
- The 0.6B-parameter full-context teacher and cache-aware streaming student are evaluated with WER, role-bucketed RWER, and macro-averaged role F1.The student decodes at RC=1, or 160 ms algorithmic latency; checkpoints are selected on validation RWER, and results average five seeds.Source: §4.2 · p. 3
4.3. Experimental Results
- RoleHear lowers Earnings-21/22 RWER by 4.2/6.5 pp and raises macro F1 by 8.9/13.8 pp with equal-or-better WER; on ServiceCall it beats the cascade on WER and RWER at both role sets.Against the supervised FT Student on ServiceCall, it improves RWER by 3.1 and 4.6 pp and F1 by 2.9 and 2.5 pp.Source: §4.3 · pp. 3–4, Table 2 · p. 4
4.4.1. Role-Emission Delay
- At delay 0, the streaming student reaches 46.1% F1, while delays 1 and 3 tie for best RWER near 31.8%; WER stays flat, so delay 1 is selected.Delay 1 gives the lowest role latency among settings avoiding collapse, approximately 360 ms; the full-context teacher instead uses delay −1 based on Earnings-22 results.Source: Table 3 · p. 4, §4.4.1 · p. 4
4.4.2. Right Context
- At delay 0, increasing RC from 0 to 13 raises F1 from 42.5% to 65.9%, but AL rises from 80 to 1120 ms; delay 1 at RC=0 gives 63.4% F1 at 80 ms.The selected setting is delay 1 with RC=1, yielding 160 ms algorithmic latency.Source: Table 4 · p. 4, §4.4.2 · p. 4
5. CONCLUSION
RoleHear uses one cache-aware streaming transducer to emit conversational roles inline, trained with single-pass pseudo-labels and silver labels where gold annotations are missing. Across earnings and customer-service calls, it preserves transcription quality while narrowing the streaming student’s role-attribution gap to the offline teacher and keeping algorithmic latency low.
5. CONCLUSION
- Across earnings calls with silver roles and customer-service calls with gold roles, adding role tokens is nearly free on transcription and pseudo-labeling narrows the student’s role-attribution gap to the offline teacher.A small role-emission delay keeps algorithmic latency low by deferring only the role token, rather than widening right context and delaying every word.Source: §5 · p. 4
- RoleHear emits roles inline from one cache-aware streaming transducer, using a fast turn-boundary marker followed a few words later by a per-role token.The student learns transcription and role attribution together without a separate diarizer or iterative refinement.Source: §5 · p. 4
- An offline teacher fine-tuned into a role-attributed recognizer relabels unlabeled audio in one pass, while an MLLM supplies silver role labels where gold annotations are missing.The MLLM-generated silver labels are used where gold role annotations are unavailable.Source: §5 · p. 4