Source-linked AI summary
From 2.1M Hours of Raw Broadcast Audio to Finnish Speech SSL: Confidence Filtering and Architectural Analysis at Scale
Yaroslav Getman, Tamás Grósz, Tommi Lehtonen, Mikko Kurimo
TL;DR
Broadcast archives provide scale and variability for SSL but also contain non-speech and multilingual material, and quality-based filtering remains underexplored. This Finnish study filters a 2.1M-hour archive using ASR confidence and compares encoder architectures at matched parameter budgets; it reports improved results over LID-based filtering and prior Finnish SSL baselines.
Problem
Broadcast SSL studies have generally extracted speech rather than systematically selecting it by quality, a question that remains underexplored for noisy archives.
Source: §I · p. 1Method
The study applies VAD, deduplication, and ASR-confidence filtering to Finnish broadcast data, and evaluates four wav2vec 2.0 encoder backends at matched parameter budgets.
Source: §I · pp. 1–2, C. Filtering pipeline · p. 2Results
3–7% relative WER improvements over LID-based filtering were reported despite a smaller retained set, while architectural advantages varied with model scale.
Source: §V · p. 5Takeaways & Limitations
The results support raw broadcast material as a viable speech-SSL source when extensively filtered, with findings scoped to Finnish and guidance for other languages using unconstrained audio.
Source: §V · p. 5
Abstract
from arXiv · showhide
Broadcast archives offer scale and diversity for self-supervised learning (SSL) but contain substantial noisy, non-speech, and multilingual content. Prior work typically relies on voice activity detection or language identification (LID) to extract speech, ensuring presence rather than quality. Whether and how to filter unlabeled speech by quality for SSL, particularly from noisy broadcasts, remains underexplored. We present a large-scale Finnish SSL study starting from 2.1M hours of unconstrained broadcast audio. We propose automatic speech recognition (ASR) confidence as a proxy for pretraining data quality and show 3-7% relative WER improvements over LID-based filtering despite a smaller training set. We further address the lack of controlled architectural comparisons in speech SSL by evaluating Transformer, Macaron-Net, Conformer, and E-Branchformer at matched parameter budgets. Our models improve upon prior Finnish SSL baselines, particularly at Base scale and out-of-domain.
I. INTRODUCTION
Broadcast archives offer scale and variability for speech SSL but contain non-speech and multilingual material, while prior extraction methods generally select speech rather than its quality. This study applies ASR-confidence filtering and compares encoder architectures at controlled parameter budgets for Finnish SSL.
I. INTRODUCTION
- Prior broadcast-data methods primarily extract speech, leaving systematic quality-based selection underexplored, especially for noisy archives.The archives also present limited metadata, non-speech content, and multilingual segments.Source: §I · p. 1
- The study starts with 2.1M hours of unconstrained broadcast material and applies VAD, deduplication, and ASR-confidence filtering to yield 471K hours.The authors describe this as the largest academic monolingual SSL pretraining effort to date.Source: §I · pp. 1–2
- 3–7% relative WER improvements over language-identification-based filtering were achieved despite retaining a smaller training set.The study also reports convolution-augmented encoders yielding up to 37% relative WER reduction over Transformer at Base scale.Source: §I · p. 2
- A parameter-controlled comparison evaluates Transformer, Macaron-Net, Conformer, and E-Branchformer from 95M to 965M parameters.The comparison addresses prior work in which architecture and model size varied together.Source: §I · pp. 1–2
II. DATA
The data combine a large Finnish broadcast archive with three public Finnish speech datasets, processed through a common filtering pipeline. Evaluation includes a manually transcribed KAVI test set designed around real-world broadcast conditions.
II. DATA
- KAVI contributes 2.1M hours of broadcasts from 26 TV channels, 85 radio stations, and online media, spanning 2009–2025.Content ranges from news to entertainment, advertisements, and music, with limited metadata.Source: §II · p. 2
- VoxPopuli, Finnish Parliament, and Lahjoita puhetta add 14.3K, 3.1K, and 3.2K hours, respectively, and undergo the same filtering pipeline.The transcribed portions of Finnish Parliament and Lahjoita puhetta total 4.6K hours and are used for supervised fine-tuning.Source: §II · p. 2
- The KAVI evaluation set contains two manually transcribed one-hour radio broadcasts from stations Nova and Basso.It was created to evaluate real-world broadcast conditions.Source: §II · p. 2
III. METHODOLOGY
The study uses wav2vec 2.0 at three scales and fixes both parameter budgets and pretraining data across encoder backends. It compares four architectures whose blocks differ in how they combine feed-forward, attention, and convolutional processing.
A. Self-supervised pretraining framework
- Base (95M), Large (317M), and X-Large (965M) wav2vec 2.0 models use matched parameter counts and fixed pretraining data across backends.Evaluation spans six in-domain and out-of-domain test sets.Source: A. Self-supervised pretraining framework · p. 2
B. Encoder architectures
- The standard wav2vec 2.0 Transformer block places MHSA before an FFN, with residual connections and layer normalization.Source: B. Encoder architectures · p. 2
- Macaron-Net replaces the Transformer’s single FFN with two half-step FFNs around MHSA, halving inner dimensionality to maintain parameter parity.It is evaluated separately to isolate the dual-FFN configuration from other architectural changes.Source: B. Encoder architectures · p. 2
- Conformer adds a convolution module between the self-attention and final FFN, combining attention-based global dependencies with local patterns.Source: B. Encoder architectures · p. 2
- E-Branchformer processes global MHSA and local convolutional gated-MLP features in parallel, then merges them with depthwise convolution.Source: B. Encoder architectures · p. 2
C. Filtering pipeline
The filtering pipeline first detects speech and removes transcript-level repetition, then uses ASR confidence to select segments by quality. A separate LID baseline retains segments only when both audio- and text-based systems identify Finnish.
C. Filtering pipeline
- VAD produces 1.2M hours of speech segments, divided into chunks of at most 30 seconds.Source: C. Filtering pipeline · p. 2
- Transcript-based deduplication reduces the VAD output from 1.2M to 890K hours by retaining one instance per unique ASR transcript.The method targets repeated broadcast content, while acknowledging that it may reduce acoustic diversity for repeated phrases spoken by different speakers.Source: C. Filtering pipeline · p. 2
- Cchar is the mean log-probability of predicted frame-wise characters, excluding blank tokens, and serves as a proxy for pretraining-data quality.The stated intuition is that a trained ASR model assigns higher CTC log-probabilities to acoustically clear speech than to noisy or irrelevant audio.Source: C. Filtering pipeline · p. 2
- A segment is retained when Cchar ≥ τ, with the threshold selected in Section IV-B.Source: C. Filtering pipeline · p. 3
- The LID baseline retains a segment only when both an audio-based ECAPA-TDNN model and fastText on ASR transcripts identify it as Finnish.The archive contains substantial multilingual material, including approximately 80K hours of English.Source: C. Filtering pipeline · p. 3
A. Experimental setup
Models are pretrained at three scales and fine-tuned for ASR, then evaluated in-domain, out-of-domain, and on challenging broadcast speech.
A. Experimental setup
- Pretraining runs on 512 AMD MI250X GPUs, while ASR fine-tuning uses 128 GPUs; the CNN feature encoder remains frozen during fine-tuning.Fine-tuning uses CTC loss, with learning rates selected by validation performance.Source: A. Experimental setup · p. 3
- Base, Large, and X-Large pretraining runs use different learning rates, per-GPU batch durations, and training steps.They run for 125K, 167K, and 400K steps, respectively.Source: A. Experimental setup · p. 3
- Evaluation uses WER across Finnish Parliament and Lahjoita puhetta in-domain sets, Common Voice and FLEURS out-of-domain benchmarks, and the KAVI broadcast test set.Finnish Parliament includes two distinct test sets.Source: A. Experimental setup · p. 3
B. Data filtering experiments
The filtering experiments compare downstream WER and retained data after VAD and deduplication. ASR confidence filtering at τ = −0.1 is selected for subsequent pretraining.
B. Data filtering experiments
- The comparison trains Base Transformer models on unfiltered, LID-filtered, and ASR-confidence-filtered deduplicated VAD data, using several confidence thresholds.Models are fine-tuned on 4.6K supervised hours and evaluated on four validation sets.Source: B. Data filtering experiments · p. 3
- At similar retention rates (~72%), τ = −0.2 confidence filtering beats LID-only filtering on 3 of 4 benchmarks; τ = −0.1 gives the lowest WER and retains 471K hours.That is 53% of deduplicated data; the figure compares validation WER and retention after thresholds are applied to deduplicated VAD segments.Source: B. Data filtering experiments · p. 3, Figure 1 · p. 3
- The selected τ = −0.1 pipeline reduces 2.1M raw broadcast hours to a 471K-hour pretraining subset through VAD, transcript deduplication, and confidence filtering.LID analysis found approximately 99% Finnish content in the final set, so explicit language filtering was unnecessary.Source: B. Data filtering experiments · p. 3, Figure 2 · p. 4
C. Architecture comparison and depth-width trade-offs
Matched-parameter experiments compare encoder backends and depth-width configurations. Convolution-augmented encoders lead at Base scale, while deeper Transformers and Macaron-Net variants show different outcomes.
C. Architecture comparison and depth-width trade-offs
- The study evaluates alternative backends and depth-width configurations at fixed parameter budgets to analyze architectural modifications and parameter efficiency.Source: C. Architecture comparison and… · p. 4
- Transformer performance saturates at 18 layers; a 24-layer, narrower parameter-matched model shows no improvement on 5 of 6 test sets.At fixed parameter budgets, increased depth may be offset by reduced width.Source: C. Architecture comparison and… · p. 4
- Macaron-Net outperforms Transformers: its 12-layer variant has lower WER than the equivalent Transformer, and its 22-layer variant beats all Transformer variants.Source: C. Architecture comparison and… · p. 4
- Conformer leads on cleaner read-speech benchmarks (CV: 5.69%, FLEURS: 10.56%), while E-Branchformer performs best on broadcast KAVI (22.44%).Because neither consistently dominates, both are scaled to Large and X-Large.Source: C. Architecture comparison and… · p. 4
D. Scaling experiments
Compared with the prior Finnish baseline, the filtered-data Transformer models retain comparable in-domain performance and improve out-of-domain results, especially on broadcast speech. Architectural scaling patterns vary by scale.
D. Scaling experiments
- 9.5–12.8% relative WERR on KAVI accompanies generally improved out-of-domain performance, while in-domain results remain comparable to the baseline across scales.The baseline used 158K hours of mainly clean talk radio; the new pipeline retains 471K hours from unconstrained archives.Source: D. Scaling experiments · p. 4
- The benefit of deeper matched-parameter models does not persist at Large scale: 36-layer Conformer degrades 5.8% on KAVI, and E-Branchformer underperforms 21% on Common Voice versus 24-layer variants.The passage identifies width-depth balance as critical for consistent stability and generalization at larger scales.Source: D. Scaling experiments · p. 4
- At Base and Large scales, convolutional encoders beat Transformer on out-of-domain benchmarks, including 31.0–36.7% WERR on Common Voice; at X-Large, Transformer leads KAVI at 19.54%.At X-Large, Transformer improves over Large across benchmarks, while convolutional encoders show domain-specific degradations.Source: D. Scaling experiments · p. 4
V. CONCLUSION
The study retains 471K hours from 2.1M hours of unconstrained Finnish broadcast audio and finds benefits from confidence filtering and scale-dependent architectural choices.
V. CONCLUSION
- The study retains a high-quality 471K-hour subset from 2.1M hours of unconstrained broadcast archives.Source: §V · p. 4
- Table III reports WER (%) across scales and encoder backends, marking the best result per scale and overall.Source: Table III · p. 5
- 3-7% WER improvements over LID-based filtering are achieved by the confidence-based pipeline, which also removes low-quality speech and foreign content.The retained subset is smaller than the LID-filtered training set.Source: §V · p. 5
- Architectural payoffs depend on scale: deeper and convolution-augmented encoders help at Base, but their advantages diminish near 1B parameters as standard Transformers scale most consistently.Source: §V · p. 5
- The optimized Base models narrow the gap to prior Finnish SSL baselines, while out-of-domain performance continues improving through 1B parameters.The authors suggest larger models remain beneficial for diverse real-world conditions.Source: §V · p. 5