Source-linked AI summary
AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenario
Yihui Fu, Luyao Cheng, Shubo Lv, Yukai Jv, Yuxiang Kong, Zhuo Chen, Yanxin Hu, Lei Xie, Jian Wu, Hui Bu, Xin Xu, Jun Du, Jingdong Chen
TL;DR
Meeting transcription remains difficult for real-world applications because it combines multiple speech-processing tasks and requires costly, comprehensive annotation, while existing datasets are limited in realism, diversity, and language coverage. AISHELL-4 addresses these gaps with a real-recorded Mandarin meeting corpus and baseline framework, achieving lower speaker-independent and speaker-dependent CER after front-end and diarization processing.
Problem
Meeting transcription remains an unsolved real-world problem, and progress is constrained by the scarcity of datasets with precise annotations, realistic conversational conditions, and language diversity.
Method
AISHELL-4 provides a 120-hour real-recorded Mandarin meeting corpus captured with an 8-channel microphone array, with high-quality multi-aspect transcriptions and an accompanying training and evaluation baseline.
Results
After front-end and diarization processing, CER decreased from 32.56% to 30.49% for speaker-independent evaluation and from 41.55% to 39.86% for speaker-dependent evaluation.
Takeaways & Limitations
AISHELL-4 supplies real multi-channel Mandarin conference data and a baseline framework for research spanning speech enhancement, separation, recognition, and speaker diarization.
Takeaways & Limitations
The speaker diarization baseline assumes each speech frame belongs to only one speaker and does not handle overlapped speech.
Abstract
from arXiv · showhide
In this paper, we present AISHELL-4, a sizable real-recorded Mandarin speech dataset collected by 8-channel circular microphone array for speech processing in conference scenario. The dataset consists of 211 recorded meeting sessions, each containing 4 to 8 speakers, with a total length of 120 hours. This dataset aims to bridge the advanced research on multi-speaker processing and the practical application scenario in three aspects. With real recorded meetings, AISHELL-4 provides realistic acoustics and rich natural speech characteristics in conversation such as short pause, speech overlap, quick speaker turn, noise, etc. Meanwhile, accurate transcription and speaker voice activity are provided for each meeting in AISHELL-4. This allows the researchers to explore different aspects in meeting processing, ranging from individual tasks such as speech front-end processing, speech recognition and speaker diarization, to multi-modality modeling and joint optimization of relevant tasks. Given most open source dataset for multi-speaker tasks are in English, AISHELL-4 is the only Mandarin dataset for conversation speech, providing additional value for data diversity in speech community. We also release a PyTorch-based training and evaluation framework as baseline system to promote reproducible research in this field.
1. Introduction
Meeting transcription estimates who speaks what and when in recordings with multiple speakers and overlap, but remains unsolved for real-world applications. AISHELL-4 addresses data scarcity by releasing a sizeable real-recorded Mandarin meeting corpus with detailed annotations and a baseline framework.
- Motivation: Meeting transcription combines speech front-end processing, activity detection, recognition, speaker identification, and diarization, making it highly challenging.The task must handle multiple speakers and a certain amount of speech overlap.
- Motivation: Precise annotation is costly because meetings contain overlap, quick speaker turns, non-grammatical speech, and noise.Overlapped regions may require repeated listening to identify each involved speaker.
- Data gap: Existing meeting datasets vary in corpus size, speaker and spatial coverage, recording conditions, and content, while almost all public corpora are English-based.Language-specific properties can make English-based solutions suboptimal for other languages.
- Contribution: AISHELL-4 provides 120 hours of real Mandarin meeting recordings captured with an 8-channel microphone array across diverse real-world conditions.The corpus includes varied participant numbers and overlap ratios, with high-quality transcriptions on multiple aspects.
- Contribution: The release includes a baseline training and evaluation system to facilitate research on the complex meeting-transcription problem.The framework accompanies the corpus to support research across different meeting-processing aspects.
2. Previous Works
Earlier conversational and speech-separation datasets provide useful resources but often use limited, synthetic, single-speaker, or English-centered settings. AISHELL-4 is positioned against these gaps through real conference speech and Mandarin coverage.
- Conversational speech corpora: Switchboard and Fisher offer conversational telephone speech, but their 8 kHz recordings and single-speaker setup omit overlapping meeting scenarios.Switchboard contains about 2,500 conversations, while Fisher exceeds 16,000 conversations and 2,000 hours.
- Speech separation corpora: WSJ0-2mix, WHAM!, WHAMR!, and LibriMix use synthetic mixtures with short, fully overlapped segments that mismatch typical real-conversation overlap.Real conversations usually have overlap ratios below 20%.
- Conference conversation corpora: LibriCSS uses read speech played through loudspeakers, whereas CHiME-6 has relatively low recording quality and exactly four participants per meeting.These properties limit how well the datasets represent natural conversation or meeting-level generalization.
- Mandarin resources: Common open-source Mandarin speech datasets are generally near-field recordings without speech overlap, prominent noise, or reverberation.The paper states that no public Mandarin meeting dataset was available to its knowledge.
3. Datasets
AISHELL-4 is organized as a 120-hour Mandarin meeting corpus recorded across varied venues and participant groups. Its session-level metadata and detailed annotations cover speakers, timing, speech content, nonspeech events, and overlap.
- Dataset scale: AISHELL-4 contains 120 hours divided into 107.50 training hours and 12.72 evaluation hours across 191 and 20 sessions.Each session consists of a 30-minute discussion.
- Dataset scale: The training and evaluation sets include 36 and 25 participants, respectively, with balanced gender coverage.Participant counts are reported separately for the two splits.
- Recording conditions: Recordings span 10 conference venues ranging from small to large rooms, with varied wall materials, furnishings, and microphone-speaker distances of 0.6 to 6.0 m.Participants converse naturally around a table-centered microphone array.
- Session diversity: Sessions include 4 to 8 participants and topics spanning medical treatment, education, business, management, industrial production, and daily routines.Average speech overlap is 19.04% for training and 9.31% for evaluation.
- Annotation: Headset signals are aligned with the first array channel, and the higher-quality signal is selected to support manual transcription assisted by automatic speech recognition.Inspectors double-check the labeling results for each session.
- Annotation: TextGrid annotations record session duration, speaker metadata, segment counts, timestamps, transcriptions, nonspeech events, and overlapping versus non-overlapping segments.Nonspeech events include pauses, laughter, coughing, and breathing.
4. Baseline
The baseline combines separately trained diarization, speech separation, and end-to-end ASR modules for meeting transcription. It evaluates speaker-independent and speaker-dependent CER, while assuming each speech frame belongs to only one speaker.
- System overview: The baseline processes meetings with separate speech front-end, speaker diarization, and ASR submodules.During evaluation, diarization provides speaker attribution and utterance boundaries before separation and recognition.
- Speaker diarization: Diarization uses SAD, speaker embeddings, agglomerative clustering, and VBx clustering to assign speaker information.The system adopts a CHiME-6-based diarization design with ResNet embeddings and hierarchical clustering followed by VBx.
- Limitation: The diarization module assumes each speech frame corresponds to only one speaker and therefore does not model overlapped speech.The authors identify overlap-aware diarization as future work.
- Speech separation: The separation front end estimates two masks and applies MVDR beamforming to produce separated outputs.Its network uses three 3084-node LSTM layers followed by a sigmoid mask layer, trained with simulated overlapping and noisy speech.
- Speech recognition: The ASR backend is a sequence-to-sequence transformer with CNN subsampling, an 8-layer encoder, a 6-layer decoder, and joint CTC–cross-entropy training.Training combines simulated data with non-overlapped real-recorded AISHELL-4 speech.
- Evaluation: 32.56% and 41.55% CERs occur without front-end processing, compared with 30.49% and 39.86% after front-end and diarization processing on speaker-independent and speaker-dependent tasks.The evaluation reports CER for both a ground-truth-segmented speaker-independent task and an end-to-end speaker-dependent task.
5. Conclusions
The paper presents AISHELL-4 as a real-recorded Mandarin conference-speech dataset and releases a baseline training and evaluation framework. Experiments show that front-end and diarization processing reduce CER in both evaluation tasks.
- Dataset and framework: AISHELL-4 is a real-recorded, multichannel Mandarin dataset for enhancement, separation, recognition, and speaker diarization in conference scenarios.The recordings were collected in real conference venues and acoustic conditions.
- Dataset and framework: The released training and evaluation framework provides a baseline for research across meeting-processing tasks.The paper describes applications ranging from individual front-end, recognition, and diarization tasks to multimodal and joint optimization research.
- Results: Front-end and diarization processing reduce CER from 32.56% to 30.49% for speaker-independent evaluation and from 41.55% to 39.86% for speaker-dependent evaluation.The reported comparison uses the reference system with and without front-end processing.