Source-linked AI summary
Attention-Based Adaptive Policies for Simultaneous Speech-to-Text Translation
Filip Tăşădan, Ema Tomanová, Ondrej Lopuch, Paweł Bilko, Anders Søgaard
TL;DR
Simultaneous speech translation must choose when to emit partial translations while balancing quality and latency, and many adaptive approaches require training on simulated partial input. This paper uses cross-attention to define RFAP and DCAP for offline-trained models; RFAP improves the quality-latency trade-off, while DCAP preserves quality at very low latency.
Problem
Fixed policies cannot judge whether each step has enough speech context for accurate translation, while many adaptive approaches train on simulated partial input, increasing computation and risking train-test mismatch.
Source: §1 · p. 1Method
The paper proposes RFAP and DCAP, attention-based decision policies that use offline-trained speech-to-text models in streaming scenarios.
Source: §1 · p. 1, §3.2 · p. 3Results
RFAP outperforms evaluated policies with gains of up to 4.0 BLEU, while DCAP achieves negative average latency with high translation quality.
Source: §1 · p. 1Takeaways & Limitations
Attention-based policies enable offline models to balance translation quality and latency in real-time streaming.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Simultaneous speech-to-text translation (Simul-S2TT) consists of generating partial translations while the incoming audio frames are processed by the system. However, the streaming nature of this setup creates the challenge of deciding the best moment to perform an accurate translation while minimizing the delay. To address this challenge, we utilize the cross-attention mechanism of the encoder-decoder architecture to find the right alignment between the input speech frames and the target text tokens. In this paper, we propose the Recent Frame Attention Policy (RFAP) and the Dual-Condition Attention Policy (DCAP) that allow offline trained speech-to-text translation models to be used in streaming scenarios without requiring additional training. Results on three different language translation pairs over the CVSS-C corpus show that the RFAP is able to surpass other policies with gains of up to 4.0 BLEU while reducing the translation delay by almost 1 second. Moreover, the DCAP is able to preserve a high translation quality when the latency is very low.
1. INTRODUCTION
Simul-S2TT policies must balance translation quality and latency by deciding when to emit partial translations. The paper proposes RFAP and DCAP for offline-trained models and reports gains for RFAP and high-quality negative-latency translation with DCAP.
1. INTRODUCTION
- Fixed policies such as Wait-k wait a predetermined number of speech segments, but cannot judge whether each step has enough context for accurate translation.Source: §1 · p. 1
- Adaptive policies seek sufficient-context timing, but many built-in approaches train on simulated partial input, adding computation and risking train-test condition mismatch.Source: §1 · p. 1
- Prior offline-model policies use attention-head alignment or beam search to choose when to translate, without additional decision-policy modules.Source: §1 · p. 1
- RFAP and DCAP apply to standard offline speech-to-text models in streaming scenarios.Source: §1 · p. 1
- The reported comparison finds that RFAP outperforms all the other evaluated policies, gaining up to 4.0 BLEU while reducing delay by almost 1 second.Source: §1 · p. 1
- DCAP achieves negative average latency while maintaining high translation quality.Source: §1 · p. 1
2. PRELIMINARY
The preliminary section describes decoder cross-attention as a link between encoded speech and generated text. It defines recent-frame attention as a measure of how much the current token attends to the newest audio, and explains the two-pass procedure used to obtain it.
2. PRELIMINARY
- Cross-attention relates encoded speech tokens to generated text tokens and is used by the adaptive policies to determine when to translate.Source: §2 · p. 2
- At each step, the model receives speech frames and previously generated tokens; speech produces keys and values, while the first-layer query comes from the generated text.Later-layer queries come from the preceding attention layer.Source: §2 · p. 2
- Recent-frame attention sums the weights on the newest input-chunk frames for a target token; focus on recent frames indicates more context is needed, while older-frame focus indicates enough information has accumulated.Source: §2 · p. 2
- The decoder first generates the current token, then runs again with that token appended to extract the cross-attention score used by the decision policies.The second pass uses the same speech input.Source: §2 · p. 2
3. METHOD
RFAP uses recent-frame cross-attention to decide when to READ for more audio or WRITE a target token. DCAP adds attention-change information to make WRITE decisions during both discovery and completion of relevant context.
3.1. Recent Frame Attention Policy (RFAP)
- A recent attention score above α triggers READ; a score below α triggers WRITE, as illustrated by the two decisions in Figures 1 and 2.The threshold α balances translation quality and latency.Source: §3.1 · pp. 2–3, Figure 1 · p. 2, Figure 2 · p. 2
- RFAP’s two-pass inference computes cross-attention only between the next output token and the most recent speech frames, preserving ’s hypothesis while avoiding excessive frame accumulation before WRITE.Source: §3.1 · p. 3
3.2. Dual-Condition Attention Policy (DCAP)
- DCAP extends RFAP with the rate of change in recent-frame attention, addressing RFAP’s quality–delay trade-off by allowing WRITE when new translation-relevant tokens are being read.Source: §3.2 · p. 3
- A nonnegative change in recent attention triggers the Discovery Phase, allowing WRITE when the model detects relevant information in new audio instead of waiting for attention to peak and fall.Source: §3.2 · p. 3
- The Completion Phase triggers WRITE when recent attention falls below α, including when the score descends without activating the rate-of-change condition.Source: §3.2 · p. 3
4. EXPERIMENTS
Experiments on CVSS-C’s FR-EN, DE-EN, and ES-EN pairs evaluate attention-based policies for balancing translation quality and latency. RFAP performs strongly across latency ranges, while DCAP is especially effective at low latency.
4.1. Experimental Setup
- Experiments use CVSS-C, derived from CoVoST 2, for FR-EN, DE-EN, and ES-EN; its input speech is synthetically generated in a single canonical voice.The dataset statistics are presented in Table 1.Source: §4.1 · p. 3, Table 1 · p. 3
- The 55-million-parameter StreamSpeech-based model uses 320 ms input chunks and eight recent speech frames; attention policies use the last cross-attention layer.The encoder produces one speech frame every 40 ms, and the model is adapted for offline speech-to-text translation.Source: §4.1 · p. 3
4.2. Results
- ≈4.0 BLEU: on FR-EN below 500 ms AL, RFAP gains this amount over EDAtt.Table 2 groups policies by AL range and reports the best BLEU within each range.Source: Table 2 · p. 4, §4.2 · p. 4
- For similar translation qualities on FR-EN and ES-EN translations, RFAP decreases latency by around 1 second versus Wait-K and 2 seconds versus LA.Source: §4.2 · p. 4
- On FR-EN, DCAP reaches 22.46 BLEU at 36 ms AL; on ES-EN, it reaches 17.29 BLEU at -311 ms AL.The authors infer that the ES-EN result corresponds to translating almost one 320 ms input chunk ahead of the target reference.Source: §4.2 · p. 4
5. CONCLUSION
The paper proposes RFAP and DCAP, which use cross-attention in offline-trained speech-to-text models for simultaneous translation. Experiments report RFAP’s gains in mid-range latency and DCAP’s low-latency quality, without additional training or complex decision components.
5. CONCLUSION
- RFAP outperforms EDAtt, Wait-k, and Local Agreement by up to 4.0 BLEU in mid-range latency settings, while DCAP achieves negative AL with strong translation quality in low-latency settings.The policies leverage cross-attention in offline-trained models and require no additional training.Source: §5 · p. 4
6. COMPLIANCE WITH ETHICAL STANDARDS
The study used the publicly available CVSS-C corpus, derived from CoVoST 2, and collected no new human-subject data. Ethical approval was therefore not required.
6. COMPLIANCE WITH ETHICAL STANDARDS
- No new data involving human subjects were collected, so ethical approval was not required.The research used the publicly available CVSS-C corpus, derived from CoVoST 2.Source: §6 · pp. 4–5