PaperlayerView exact revision on arXiv

Source-linked AI summary

Frontend-Assisted Large Audio-Language Models for SLT2026 SmartGlasses Challenge

Yushi Wang, Zeyan Song, Xiaobin Rong, Chengwei Liu, Jun Gao, Jing Lu

arXiv:2610.13282v1eess.AScs.SD

TL;DR

The paper addresses time-stamped, speaker-attributed transcription for challenging two-person conversations, where overlapping speech is often not explicitly separated and many systems use single-channel input. It fine-tunes SoulX-Transcriber and investigates rule-based and neural speech frontends; the most stable final system achieves a test tcpCER of 6.22.

  • Problem

    Many structured-generation ASR systems leave overlapped speech to the backend and are designed for single-channel input, although smart glasses provide multiple microphones with spatial information.

    Source: §I · p. 1

  • Method

    The paper fine-tunes SoulX-Transcriber and investigates multi-channel frontends, including GSS target-speaker extraction and neural speech separation, before transcription.

    Source: §I · p. 1, B. Guided Source Separation Module · p. 2

  • Results

    tcpCER of 6.22 on the test set is achieved by the most stable final submission, the fine-tuned SoulX-Transcriber system.

    Source: B. Experimental Results · p. 3

  • Takeaways & Limitations

    On the validation set, frontend-based target-speaker extraction or separation followed by single-speaker SoulX-Transcriber decoding can further reduce CER, with frontend advantages more evident at higher overlap ratios.

    Source: §IV · p. 3, B. Experimental Results · p. 3

    SoulX-Transcriber can hallucinate repeated tokens or drift in timestamps, and a few problematic samples required approximately 30-second segment decoding instead of full-length inference.

    Source: B. Experimental Results · p. 3

Abstract

from arXiv · show

This paper describes our systems for the Track 1 time-stamped speaker-attributed ASR task of the SmartGlasses Challenge. To handle overlapping speech, background noise and far-field conditions in two-person conversations, we employ SoulX-Transcriber, a multimodal large language model that takes full-length audio as input and generates structured transcriptions. In addition, we investigate the effect of speech frontend processing, including a rule-based Guided Source Separation method and a data-driven speech separation model, both of which improve performance on the validation set. Considering inference stability, our final submission uses the fine-tuned SoulX-Transcriber system, achieving a tcpCER of 6.22 on the test set.

I. INTRODUCTION

The SmartGlasses Track 1 task targets time-stamped, speaker-attributed transcripts of challenging two-person conversations, including overlap and interference. The paper adapts SoulX-Transcriber and investigates multi-channel frontends; its most stable submission achieves a test tcpCER of 6.22.

I. INTRODUCTION

  • Track 1 evaluates time-stamped, speaker-attributed transcription from four-channel smart-glasses audio in two-person conversations with overlap, background interference, and frequent speaker turns.Source: §I · p. 1
  • Existing structured-generation systems generally leave overlapped speech to the backend and often use single-channel input, despite the spatial information available from wearable-device microphones.The paper identifies this as a potential constraint on minimizing WER or CER.Source: §I · p. 1
  • The work fine-tunes an open-source audio-language model on challenge data and investigates rule-based and neural multi-channel frontend processing.Source: §I · p. 1
  • tcpCER of 6.22 on the test set is achieved by the fine-tuned SoulX-Transcriber system, selected for its most stable performance.The authors also investigate frontend processing for target-speaker extraction or speech separation.Source: §I · p. 1

II. SYSTEM DESCRIPTION

The system overview comprises three approaches built around the open-source SoulX-Transcriber: direct decoding of a channel-averaged mixture, GSS extraction, and DPRNN-based separation. The three systems are developed and evaluated independently.

II. SYSTEM DESCRIPTION

  • Three systems use SoulX-Transcriber: direct decoding of the channel-averaged mixture, GSS target-speaker extraction, or DPRNN-based separation into two speaker-specific streams.Source: §II · p. 1
  • System 1 directly applies SoulX-Transcriber to channel-averaged mixture speech, while Systems 2 and 3 add GSS or DPRNN-based processing before the backend.Source: §II · p. 1
  • The three systems are developed and evaluated independently.Source: Figure 1 · p. 1, §II · p. 1

A. SoulX-Transcriber Backbone

SoulX-Transcriber is a Qwen3-Omni-based model that generates time-stamped, speaker-attributed transcripts. Its training combines speaker-aware multi-task pretraining with supervised fine-tuning, and the paper distinguishes the original checkpoint from its challenge-data-fine-tuned version.

A. SoulX-Transcriber Backbone

  • SoulX-Transcriber builds on Qwen3-Omni and inherits long-context audio understanding and autoregressive text generation for time-stamped, speaker-attributed transcription.Source: A. SoulX-Transcriber Backbone · p. 2
  • Training has two stages: speaker-aware multi-task continuous pretraining on pseudo-labeled and simulated conversations, followed by supervised fine-tuning on approximately 1,000 hours of annotated SDR data.The pretraining jointly optimizes speaker-turn prediction, diarization, verification, and ASR; fine-tuning targets instruction following, timestamps, attribution, and transcription robustness.Source: A. SoulX-Transcriber Backbone · p. 2
  • SoulX-Transcriber-pretrained denotes the original open-source model, whereas SoulX-Transcriber-finetuned denotes the checkpoint fine-tuned using the SmartGlasses Challenge training set.Source: A. SoulX-Transcriber Backbone · p. 2

B. Guided Source Separation Module

Guided Source Separation (GSS) is a multi-channel frontend for target-speaker extraction before SoulX-Transcriber. Speaker-attributed activity labels guide separation and beamforming to produce separate tracks for decoding.

B. Guided Source Separation Module

C. Multi-channel Speech Separation Module

The DPRNN-based multi-channel module separates the recording into wearer and partner streams, which SoulX-Transcriber transcribes independently. Its training objective combines complex-spectral, magnitude-spectral, and perceptual losses, with greater weight on the partner stream.

C. Multi-channel Speech Separation Module

  • The DPRNN-based module takes a long-form mixed multi-channel recording and produces separate streams for the smart-glasses wearer and conversation partner.Each stream is independently fed to SoulX-Transcriber for transcription.Source: C. Multi-channel Speech Separation… · p. 2
  • Four-channel STFT inputs are converted to ERB features, fused across channels, and processed by DPRNN blocks that model local and long-range dependencies.The decoder predicts speaker-specific complex time-frequency masks, which are applied to the reference microphone spectrum to reconstruct the streams.Source: C. Multi-channel Speech Separation… · p. 2
  • Training combines complex-spectral MSE, magnitude-spectral MSE, and a time-domain PMSQE loss for each output stream.The complex loss compares real and imaginary parts; the magnitude loss compares magnitude spectrograms.Source: C. Multi-channel Speech Separation… · p. 2
  • The final objective weights the partner-stream loss more heavily; VAD labels also zero inactive target-speaker regions before the streams are transcribed and merged.Source: C. Multi-channel Speech Separation… · p. 2

D. Speaker-Attributed VAD Estimation

Speaker-attributed VAD labels are produced with a multi-channel DiariZen-based diarization model augmented with ChannelAttention.

D. Speaker-Attributed VAD Estimation

  • ChannelAttention adds cross-channel self-attention to early WavLM layers to capture spatial cues, then speaker-embedding clustering turns local activity predictions into global speaker-attributed VAD labels for downstream ASR.Source: D. Speaker-Attributed VAD Estimation · p. 2

A. Datasets and Experimental Setup

The experiments fine-tune SoulX-Transcriber on the challenge training data and train a multi-channel separator using real and simulated room impulse responses. The systems are evaluated on the official Track 1 validation set.

A. Datasets and Experimental Setup

  • SoulX-Transcriber has a 30B-parameter MoE Thinker and a 0.6B-parameter AuT encoder, while DPRNN mc has 4.7M parameters.The challenge training set used for SoulX-Transcriber fine-tuning contains approximately 33 hours of speech.Source: A. Datasets and Experimental Setup · p. 2
  • Table I reports experimental results for the Track 1 TSA-ASR task on the official validation set.Source: Table I · p. 3
  • The separator training uses real CHiME-8 MMCSG RIRs and simulated MCAS RIRs, supplemented with simulated RIRs based on the SmartGlasses microphone layout.Simulation speech and noise data come from the URGENT Challenge.Source: A. Datasets and Experimental Setup · p. 3
  • SoulX-Transcriber is fine-tuned with LoRA at rank 8 and scaling factor α = 32; validation decoding uses temperature 0.2, top-p 0.9, top-k -1, and repetition penalty 1.05.Experiments ran on four NVIDIA 4090 GPUs with 48 GB memory.Source: A. Datasets and Experimental Setup · p. 3

B. Experimental Results

The final submission is fine-tuned SoulX-Transcriber, selected for stable full-length inference and achieving tcpCER 6.22 on the test set. On validation, frontend-assisted systems reduce error rates versus SoulX-Transcriber without frontend assistance, while frontend-related hallucinations remain a concern for the pretrained model.

B. Experimental Results

  • All systems substantially outperform the organizer-provided VibeVoice-ASR baseline, and challenge-data fine-tuned SoulX-Transcriber performs better than its original pretrained version.The authors interpret this as evidence that a small amount of target-domain data is effective for adapting to the challenge recordings.Source: B. Experimental Results · p. 3
  • SoulX-Transcriber occasionally hallucinates through repeated tokens or timestamp drift, and frontend-processed audio tends to increase hallucinations with the pretrained model.The authors suggest frontend distortion and zero-padded non-speech regions may make the processed audio distribution differ from pretraining data.Source: B. Experimental Results · p. 3
  • Frontend processing shows a more evident advantage at higher speech-overlap ratios in the validation-set comparison of chat-level average tcpCER across systems.Fig. 2 groups chats by Low, Mid, and High overlap ratios; five chats with annotation errors were excluded.Source: Figure 2 · p. 3, B. Experimental Results · p. 3
  • Frontend-assisted GSS-SoulX and DPRNN mc-SoulX achieve lower error rates than SoulX-Transcriber without frontend assistance; GSS with oracle VAD performs slightly worse than with estimated VAD.Source: B. Experimental Results · p. 3
  • tcpCER 6.22 on the test set was achieved by the final submission, SoulX-Transcriber-finetuned, selected for its most stable full-length audio performance.Source: B. Experimental Results · p. 3

IV. CONCLUSION

On the validation set, frontend-based target-speaker extraction or speech separation followed by single-speaker decoding with SoulX-Transcriber can further reduce CER in overlapping-speech scenarios.

IV. CONCLUSION

  • Frontend-based target-speaker extraction or speech separation followed by decoding each single-speaker track with SoulX-Transcriber further reduces validation-set CER.The reported sequence pairs frontend processing with per-track decoding.Source: §IV · p. 3
  • The reported CER reduction is on the validation set and concerns speaker-attributed transcription in overlapping-speech scenarios.Source: §IV · p. 3
  • The results indicate potential for combining speech frontend modules with large audio-language ASR models for speaker-attributed transcription.Source: §IV · p. 3
Loading 2610.13282v1…