Adaptive Feature Selection for End-to-End Speech Translation

1 Introduction

• Prior work reported that E2E speech translation underperformed cascade methods, while pauses and noise can lengthen inputs and hinder learning and translation performance.Speech information is unevenly distributed, making informative features harder to extract.Source: §1 · p. 1, Figure 1 · p. 1
• AFS selects features between a pretrained ASR encoder and an ST encoder, using ASR pretraining to align selected features with transcriptions.The ASR encoder is frozen for ST training, as shown in the model overview.Source: §1 · pp. 1–2, Figure 2 · p. 2
• AFS extends L0DROP to sparsify speech features, addressing temporal and feature dimensions with distinct gating networks.Unlike prior L0DROP work on MT and summarization, the authors report that sparsity improves ST translation quality.Source: §1 · p. 2
• AFS improves translation by 1.3–1.6 BLEU on average, prunes about 84% of temporal features, and speeds decoding by about 1.4×; on LibriSpeech En-Fr it beats cascade by 0.29 BLEU, reaching 18.56 without augmentation.The authors also report that AFS offers better translation quality than fixed-rate selection and a simple CNN.Source: §1 · p. 2
• The authors argue that sparsification in E2E ST offers follow-up research opportunities beyond efficiency or interpretability, including performance gains.The method assumes that features irrelevant to ASR are also unimportant for ST.Source: §1 · p. 2

2 Background: L0DROP

• L0DROP assigns each encoded source state a scalar gate between 0 and 1 to selectively remove uninformative encoder outputs.The gate controls whether a state is removed or retained.Source: §2 · p. 2
• The gating network associates each state x_i with α_i, which shapes the HardConcrete distribution and controls the gate’s tendency toward zero or one.The trainable parameter w determines α_i from x_i.Source: §2 · p. 2
• L0DROP penalizes gate activity in an upper-bounded likelihood objective; increasing λ encourages more gates near zero.Reparameterized gate sampling makes the method differentiable, and expected gate values are used at inference.Source: §2 · p. 2

• ASR pretraining combines maximum-likelihood and CTC losses, with CTC guiding encoder outputs to align with transcription subwords.The model also uses logarithmic attention penalties for short-range dependency and trainable positional embeddings.Source: §3 · p. 3
• During AFS finetuning, the model excludes CTC loss to relax the alignment constraint and allow more flexible feature adaptation.AFS is placed between the ASR encoder and decoder during finetuning.Source: §3 · p. 3
• AFS adapts L0DROP to sparsify encoder states along temporal and feature dimensions using different gating networks.The feature gate is shared across time steps, while AFS reuses the temporal gate.Source: §3 · pp. 3–4
• ST training uses the pretrained, frozen ASR and AFS modules as a feature extractor and optimizes the translation model with maximum-likelihood loss.The framework requires only pairwise audio-transcription and audio-translation training corpora.Source: §3 · pp. 3–4
• AFS selects transcript-aligned ASR features for a downstream ST encoder, with the ASR encoder and AFS frozen during ST training.The selected features are passed to the ST encoder as its input.Source: §3 · pp. 3–4

4 Experiments

• Experiments use Transformer models on LibriSpeech En-Fr and eight MuST-C translation directions.LibriSpeech En-Fr uses the Augmented LibriSpeech dataset; MuST-C covers English-to-eight-language directions.Source: §4 · p. 4

4.1 Results on MuST-C En-De

• AFS prunes more than 85% of temporal features at λ ≥0.5, while feature sparsity remains 0% and the feature gate instead weights neurons.At λ = 0.1, both AFS variants prune about 60% of temporal features.Source: §4.1 · p. 5, Figure 3 · p. 5
• AFS retains about 95% of ASR accuracy while improving ST by more than 0.8 BLEU; development BLEU peaks at 22.17 for AFSt,f with 85.5% sparsity.The 22.17 BLEU result is on the development set.Source: §4.1 · pp. 5–6, Figure 4 · p. 5
• At k=6, fixed-rate subsampling adds 0.47 BLEU, but suboptimal rates degrade ST and the CNN alternative fails to beat ASR-PT.The authors note that choosing the optimal fixed sampling rate is difficult.Source: §4.1 · p. 6, Figure 5 · p. 6
• AFS variants surpass ASR-PT by 0.9 and 1.71 BLEU, narrowing the cascade gap to -0.14 BLEU on MuST-C En-De.The test results are summarized in Table 1.Source: §4.1 · p. 6, Table 1 · p. 6

4.2 Why (Adaptive) Feature Selection?

• AFS makes training more stable; its gains are larger with more training data, while fixed-rate subsampling performs worse than ASR-PT at data sizes ≤100K.The paper reports no higher data efficiency from feature selection in low-resource settings.Source: §4.2 · pp. 6–7, Figure 6 · p. 6, Figure 7 · p. 7
• AFS shifts cross-attention weights toward larger values, and its retained features are more uniformly distributed across positions than fixed-rate subsampling.Longer-duration words correspond to more retained features under both approaches; AFS uses context information to select features.Source: §4.2 · p. 7

4.3 Results on MuST-C and LibriSpeech

• AFS improves over ASR-PT by 1.34/1.60 average BLEU, prunes 84.5% of temporal features, and averages 1.45× decoding speedup; it narrows the cascade gap to -0.8 BLEU and surpasses Cascade on LibriSpeech En-Fr.The comparisons cover MuST-C and LibriSpeech En-Fr; the tables report results over eight MuST-C languages and LibriSpeech En-Fr, respectively.Source: §4.3 · p. 7, Table 2 · p. 8, Table 3 · p. 8

Speech Translation

• E2E ST directly translates speech without intermediate transcription but has often underperformed cascade systems, especially in low-resource settings.Prior approaches include architecture design, extra training signals, pretraining, distillation, meta-learning, and data augmentation.Source: §5 · p. 8

Speech Feature Selection

• Prior speech-feature reduction includes convolutional modeling, fixed-rate subsampling, and recurrent dynamic subsampling; phoneme-level reduction requires non-trivial recognition and alignment.Earlier dynamic subsampling methods are embedded in recurrent networks and are difficult to adapt to architectures such as Transformers.Source: §5 · pp. 8–9
• AFS applies L0DROP to retain informative speech features, extending it with separate gating networks for temporal and feature dimensions.The approach is described as fully differentiable and independent of specific encoder-decoder architectures.Source: §5 · p. 9

6 Conclusion and Future Work

• AFS filters uninformative features between a pretrained, frozen ASR encoder and the ST encoder, adapting L0DROP to model temporal and feature dimensions.Source: §6 · p. 9
• ∼1.4× decoding speedup accompanies improved translation quality, with ∼84% average temporal sparsity and a narrowed or closed gap to cascading models.The conclusion attributes the gain to AFS filtering speech features.Source: §6 · p. 9
• The paper presents sparsification in E2E ST as yielding substantial translation-quality gains, beyond sparsity’s usual efficiency or interpretability benefits in NLP.Source: §6 · p. 9
• The authors identify adapting AFS to simultaneous speech translation as future work.Source: §6 · p. 9