Source-linked AI summary
SHAS: Approaching optimal Segmentation for End-to-End Speech Translation
Ioannis Tsiamas, Gerard I. Gállego, José A. R. Fonollosa, Marta R. Costa-jussà
TL;DR
Real-world end-to-end speech translation needs automatic segmentation, while training corpora provide manual boundaries and existing methods leave a quality gap. SHAS learns frame boundaries from labeled speech and uses probabilistic Divide-and-Conquer splitting; across five language directions, it retains an average 95.7% of manual-segmentation BLEU.
Problem
Real-world speech translation requires automatic audio segmentation, but manual boundaries in corpora are unavailable at inference and existing methods remain behind manual segmentation.
Source: §1 · p. 1Method
Results
Across five language directions, SHAS retains an average 95.7% of manual-segmentation BLEU and improves by 2 BLEU over hybrid methods and.
Source: §5 · p. 3Takeaways & Limitations
SHAS applies to different domains and languages, while lighter versions remain better than existing methods and suggest potential for online settings.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Speech translation models are unable to directly process long audios, like TED talks, which have to be split into shorter segments. Speech translation datasets provide manual segmentations of the audios, which are not available in real-world scenarios, and existing segmentation methods usually significantly reduce translation quality at inference time. To bridge the gap between the manual segmentation of training and the automatic one at inference, we propose Supervised Hybrid Audio Segmentation (SHAS), a method that can effectively learn the optimal segmentation from any manually segmented speech corpus. First, we train a classifier to identify the included frames in a segmentation, using speech representations from a pre-trained wav2vec 2.0. The optimal splitting points are then found by a probabilistic Divide-and-Conquer algorithm that progressively splits at the frame of lowest probability until all segments are below a pre-specified length. Experiments on MuST-C and mTEDx show that the translation of the segments produced by our method approaches the quality of the manual segmentation on 5 language pairs. Namely, SHAS retains 95-98% of the manual segmentation's BLEU score, compared to the 87-93% of the best existing methods. Our method is additionally generalizable to different domains and achieves high zero-shot performance in unseen languages.
1. Introduction
End-to-end speech translation depends on audio segmentation, but manually segmented training and test corpora do not reflect real-world inference, where segmentation must be automatic. SHAS learns segmentation boundaries from labeled speech and improves on existing automatic methods.
1. Introduction
- End-to-end systems combine speech input and translation without the ASR-then-MT cascade, whose shortcomings include error propagation and slow inference.Transformer advances and larger speech translation corpora have enabled competitive end-to-end systems.Source: §1 · p. 1
- End-to-end speech translation needs context-aware segmentation, but real-world inference lacks the manual boundaries supplied by speech translation corpora.Evaluation campaigns have highlighted segmentation’s importance, with top submissions using their own algorithms and achieving significantly better results.Source: §1 · p. 1
- The prior hybrid method improves on length- and pause-based approaches but remains 3–4 BLEU behind manual segmentation, retaining 88.5% of its score.Source: §1 · p. 1
- SHAS learns manual segmentation from labeled speech, using wav2vec 2.0 representations to classify frames and pDAC to split at low-probability frames until segments meet a length limit.Experiments cover five language directions; the multilingual classifier further improves average BLEU retention.Source: §1 · p. 1
2. Background
The background distinguishes speech encoders and audio segmentation strategies by the information they use and the constraints they support. Hybrid methods balance acoustic cues with segment-length limits.
2. Background
- wav2vec 2.0 uses a convolutional feature extractor and Transformer encoder to produce contextualized speech representations after self-supervised pre-training on unlabeled speech.Its multilingual XLS-R version was pre-trained on 128 languages using 436k hours of speech data.Source: §2 · p. 1
- Length-based methods split at fixed durations, while pause-based methods use VAD to detect non-speech frames; neither uses both length and acoustic cues.Pause-based methods can lose syntactic information, while fixed splits may not resemble proper sentences.Source: §2 · pp. 1–2
- Hybrid DAC repeatedly splits at the longest pauses until segments are below a maximum length, whereas STRM uses a minimum and maximum and can segment streaming audio without the full recording.A prior approach also used wav2vec 2.0 as a pause detector by treating predicted padding tokens as non-speech.Source: §2 · p. 2
3. Methodology
SHAS combines a supervised frame classifier with probabilistic Divide-and-Conquer segmentation. The classifier learns manual boundaries from speech, and pDAC uses its frame probabilities to produce bounded segments.
Segmentation Frame Classifier
- The Segmentation Frame Classifier learns which audio frames belong in segments from manually segmented speech, using boundary labels and contextualized wav2vec 2.0 representations.The wav2vec 2.0 encoder is frozen; a Transformer layer and sigmoid output are trained with class-imbalance-weighted binary cross-entropy.Source: §3 · p. 2
Probabilistic DAC
- At inference, SHAS applies pDAC to classifier probabilities, splitting at low-probability frames until segments are shorter than the maximum length.Two offset rolling-window predictions are averaged to reduce context loss; pDAC trims segment ends using a threshold and avoids splits that would produce segments shorter than the minimum.Source: §3 · p. 2
- The pDAC procedure recursively selects candidate split points in ascending probability order and accepts a split when both resulting segments exceed the minimum length.If no candidate satisfies that minimum-length condition, the lowest-probability frame is used.Source: §3 · p. 2
4. Experimental Setup
Experiments evaluate segmentation across five language pairs using TED-talk corpora and a separate parliamentary-talk test domain. Development data tune segmentation parameters, and translated segments are scored against references with BLEU.
Data
- The experiments cover English-to-German and four directions into English, using MuST-C, mTEDx, and Europarl-ST for training, development, or testing.MuST-C and mTEDx use TED talks; Europarl-ST provides a different domain, based on European Parliament talks.Source: §4 · pp. 2–3
Segmentation Frame Classifiers
- Five monolingual segmentation frame classifiers and one multilingual classifier use 20-second random audio segments and XLS-R representations from its 14th layer.Models train for eight epochs; checkpoints are selected by translation quality on development sets.Source: §4 · p. 3
Segmentation methods and algorithms
- Development sets tune SHAS pDAC’s maximum segment length, while the minimum is fixed at 0.2 seconds and the trimming threshold at 0.5.The study also tunes pause-based VAD, length-based, and hybrid DAC/STRM baselines over their stated parameter ranges.Source: §4 · p. 3
Speech Translation Models
- Speech translation uses Joint Speech-to-Text Transformer encoder-decoders, with a MuST-C English-to-German model and a multilingual model for the other directions.Both models use speech at inference and beam search of size five.Source: §4 · p. 3
Evaluation
- Evaluation translates each newly segmented test set, aligns outputs with references using mwerSegmenter, and computes BLEU with SacreBLEU.Source: §4 · p. 3
5. Results
Across five language pairs, SHAS preserves nearly all manual-segmentation BLEU and outperforms existing segmentation methods. It also performs strongly on an unseen domain and in zero-shot language directions, with results depending on classifier and configuration choices.
5. Results
- Pause-based segmentation retains 81.3% of manual BLEU on average, while length-based segmentation reaches 85.3%.The authors suggest less accurate non-English VAD predictions may explain the pause-based result; pause-based segmentation performs better for en-de.Source: §5 · p. 3
- SHAS retains more than 95% of manual BLEU in all language directions and 95.7% on average, improving over existing methods.The comparison averages BLEU across five language pairs; Table 1 covers the main MuST-C and mTEDx results and Europarl-ST cross-domain results.Source: §5 · p. 3, Table 1 · p. 4
- On Europarl-ST test, SHAS achieves 26.4 BLEU, retaining 94.9% of manual segmentation’s BLEU, versus 25.06 BLEU (90.1%) for the best existing approach.Source: §5 · p. 4, Table 1 · p. 4
- Zero-shot BLEU is high and comparable to supervised directions in most cases; all classifiers except English achieve very high average zero-shot performance.The authors attribute this pattern to XLS-R pretraining on all five languages and suggest the lower English-classifier zero-shot performance may reflect its larger training corpus.Source: §5 · p. 4, Table 2 · p. 4
- With an English SFC, early training remains generalizable but is not yet optimal for the supervised direction, illustrating a specialization–generalization trade-off.Figure 2 plots percentage of manual BLEU against training epochs for supervised and zero-shot directions.Source: §5 · p. 4, Figure 2 · p. 4
- A probabilistic STRM version is 0.1 BLEU worse than pDAC, while a compact XLS-R feature-extractor variant is 0.4 BLEU behind.Removing the SFC Transformer layer costs 1.2 BLEU; adding a second Transformer layer brings no further improvement, and lighter variants remain better than existing methods for potential online use.Source: Table 3 · p. 4, §5 · p. 4
6. Conclusions
SHAS surpasses existing segmentation methods across five language directions and narrows the gap with manual segmentation at inference. The method also applies across domains and languages, while lighter versions suggest potential for online speech translation.
6. Conclusions
- SHAS surpasses existing methods in five language directions and significantly reduces the gap with manual segmentation at inference.The method is also applicable to different domains and languages; lighter versions remain better than existing methods, suggesting potential application to online settings.Source: §6 · p. 4