Source-linked AI summary
Dynamic Lagging using Stable-Prefix Training for Simultaneous Translation
Hieu Hoang, Amittai Axelrod, Matt Post
TL;DR
Cascaded MT systems have limited ability to make contextual decisions about how much of an incoming source prefix to translate. The paper fine-tunes Qwen3-8B on stable prefixes and finds that prefix models dominate most of the latency-quality frontier across the tested language pairs, datasets, and metrics.
Problem
Cascaded MT systems often use coarse wait-k or target-suffix deletion policies that do not respond to source context when deciding how much to translate.
Source: §1 · p. 1Method
Results
The prefix model dominates most of the Pareto frontier across language pairs, datasets, and evaluation metrics; confidence training reduces ECE by 31–46% overall and 48–55% on early source prefixes.
Source: §6 · p. 9Takeaways & Limitations
Training on initial-prefix segmentations generalized to random read schedules and continuations across the tested language pairs, datasets, and metrics.
Source: §6 · p. 9Experiments cover only three English-origin language pairs and one translation engine, so further experimentation is needed to test generalization.
Source: §7 · p. 9
Abstract
from arXiv · showhide
In streaming simultaneous speech translation, the speech translation system is trained to learn a read-write policy that alternates between consuming source words and generating target ones. In a cascaded setting, the output from the speech recognizer is passed to a separate machine translation component, making it more difficult to learn such a policy. Approximations such as fixed wait-k strategies or target-suffix deletion can be employed, but these approaches do not provide the model with a streaming system's flexibility to make contextual read-write decisions. This paper presents a training strategy for a cascaded machine translation system that enables it to dynamically decide how much of the growing source prefix to translate. We achieve this by fine-tuning a large language model (Qwen3-8B) on stable prefixes of the training data, which are produced by pairing every source sentence prefix in the training data with the longest translation of that prefix that is shared with the full source sentence translation. We fine-tune variants of the model on different subsets of the prefixes and compare against wait-k and target-suffix deletion. We also investigate the effect of fine-tuning the target-token generation confidence. Our experiments show that stable prefixes improve the quality-latency tradeoff when translating from English into German, Japanese, and Chinese across a range of test sets.
1 Introduction
Simultaneous translation balances translation quality against latency, while cascaded MT systems often rely on coarse policies that do not respond to source context. The paper trains an MT system on stable translation prefixes so it can learn how much output to commit, improving the latency-quality Pareto frontier across three language pairs and multiple test sets.
1 Introduction
- Simultaneous translation trades off translation quality, highest with the full input, against latency, lowest when translation starts immediately.The section situates fixed schedules such as wait-k among approaches to this tradeoff.Source: §1 · p. 1
- Cascaded MT systems must decide how much input to translate, but standard systems are often limited to wait-k or target-suffix deletion, which ignore source context.A cascade passes ASR text to a separate MT system.Source: §1 · p. 1
- Stable translation prefixes pair each source prefix with the longest shared translation prefix between its partial and full-sentence translations.These pairs are used to fine-tune the MT model to learn how much to trust a source prefix.Source: §1 · p. 1
- Stable-prefix training substantially improves the latency-quality Pareto frontier across EN→DE, EN→JA, and EN→ZH test sets.The evaluations include FLEURS, WMT24++, and CoVoST 2.Source: §1 · pp. 1–2
2 Background
Streaming translation commits output incrementally, whereas cascaded MT can use standard models with limited control over how much they emit. Wait-k and suffix deletion provide coarse controls; this paper instead trains on stable prefixes and uses a tuned confidence threshold.
2 Background
- Streaming translation commits target tokens incrementally, while wait-k offers latency control without considering context.Streaming output cannot be revised once generated.Source: §2 · p. 2
- Cascaded systems pass ASR output to MT and can use specialized streaming models, standard models on chunks, or repeated re-translation.The repeated re-translation approach constrains each new output to the previously generated target prefix.Source: §2 · p. 2
- Standard MT models can use wait-k or target-suffix deletion, but these coarse controls do not account for translation uncertainty or stability at each source prefix.Suffix deletion removes words from the end of a partial translation so the system can wait for more source input.Source: §2 · p. 2
- The paper trains on dynamically determined stable prefixes and uses a tuned confidence threshold to address the limits of wait-k and suffix deletion.The approach extends prefix training with stable prefixes anchored to full-sentence translations.Source: §2 · p. 2
3 Approach
The approach constructs stable target prefixes by comparing translations of source prefixes with the full-sentence translation, then prompts an LLM to continue from previously committed output. This produces monotonically growing targets and a flicker-free inference process.
3.1 Prefix Translation Synthesis
- For each source prefix, the method compares its translation with the full-sentence translation and uses their longest common prefix to identify stable content.The common prefix is measured at word level for EN→DE and character level for EN→JA and EN→ZH.Source: §3.1 · p. 3
- Stable-prefix synthesis makes target prefixes grow monotonically, so later noisier partial translations cannot retract content already agreed with the full-source translation.The German example shows that no partial translation is a prefix of the full-sentence translation until step 4.Source: §3.1 · p. 3, Table 1 · p. 3
- The stable-prefix corpus is merged with full translations at a fixed ratio for fine-tuning, while unstable suffixes of partial translations are removed.As source prefixes lengthen, partial translations may exhibit jitter beyond their stable prefix.Source: §3.1 · p. 3
3.2 LLM Prompting
- At inference, the LLM is prompted on the observed source and re-prompted as it grows, with the previously committed target supplied as a forced prefix.When the source is complete, the model is told it is a complete sentence and emits only the remaining continuation.Source: §3.2 · p. 3
- Because the previous translation is supplied as context but cannot be changed, the prompting architecture is flicker-free by construction.The examples illustrate continued generation as new source chunks arrive.Source: §3.2 · p. 3
4 Experimental setup
The experiments fine-tune Qwen3-8B on self-distilled data in three English-to-target directions, using prefix and continuation training variants. Evaluation measures quality under segmented and incremental input conditions, with validation and checkpoint selection based on full-sentence and prefix performance.
4.1 Models
- Qwen3-8B is the base model used throughout the experiments.Source: §4.1 · p. 3
4.2 Data
- Self-distillation uses the same model to synthesize parallel training data and fine-tunes it to become prefix-aware for streaming translation.The experiments cover EN→DE, EN→JA, and EN→ZH, with evaluation on WMT24++, FLEURS, and CoVoST 2 using COMET-22-DA and MetricX-24-Hybrid-XL.Source: §4.2 · p. 4
4.3 Training and Validation
- The prefix model trains on an equal mixture of prefix and full-sentence examples, while the continuation model adds tasks that continue from an already translated target prefix.Prefix-model checkpoint selection equally weights prefix and full-sentence COMET; training stops after five epochs without validation improvement.Source: §4.3.1 · p. 4, §4.3.2 · p. 4
4.4 Evaluation
- Binary-segmentation evaluation measures COMET as source sentences are split at word boundaries, while incremental evaluation simulates source arriving in chunks and measures latency with Average Lagging.In segmentation evaluation, the prefix translation provides force-decoded context for translating the suffix.Source: §4.4 · p. 5
5 Results
Stable-prefix fine-tuning improves quality–latency trade-offs across the evaluated settings, with confidence-based control generally performing best. It also improves commit/wait calibration and preserves its advantages across additional test sets.
5.1 Finetuning LLM
- Prefix models improve full-sentence COMET and segmented-prefix translation quality over base, but have significantly higher average lag across all three language pairs.Continuation matches base on full-sentence translation and is slightly better than base at segmented positions; prefix sees more source tokens for a given amount of output.Source: Table 2 · p. 5, §5.1 · p. 5
5.2 Pareto frontier
- Fine-tuned models achieve a better quality–latency trade-off under each mechanism, and confidence performs best in most cases, especially with fine-tuned models.Confidence is best across language pairs, models, and latency ranges except base EN→DE; its threshold moves from greedy decoding at τ = 0 to near-offline translation at τ = 1.Source: §5.2 · pp. 6–7, Table 3 · p. 8
5.3 Calibration of commit/wait confidence
- Prefix training reduces ECE against the stable-prefix oracle by 31–46%, and by 48–55% on prefixes with at most 30% of the source visible.Full-sentence-only controls closely match base, while continuation improves less than prefix; generic adaptation alone does not explain the calibration gain.Source: Figure 3 · p. 8, §5.3 · p. 8
5.4 Evaluations on other test sets
- COMET AUC improves for prefix with confidence over base with confidence in all nine dataset–language panels; prefix has the higher quality ceiling, while continuation helps at the lowest-latency frontier.The base confidence sweep controls for the inference mechanism in this comparison; MetricX results support the same conclusions.Source: Table 4 · p. 9, §5.4 · p. 9
6 Conclusion
Stable-prefix training lets a cascaded translation model learn how much target content to commit as source words arrive, improving latency–quality trade-offs across three English-to-target language pairs. Prefix training performs strongly across schedules and datasets, while confidence training improves commit/wait calibration.
6 Conclusion
- Stable prefixes pair each source prefix with the longest translation prefix shared with its full-sentence translation, letting the model learn how much target content to commit.Training mixes these pairs with normal full-sentence data.Source: §6 · p. 9
- Stable-prefix training improves latency–quality trade-offs across EN→DE, EN→JA, and EN→ZH under multiple decoding strategies.Inference uses prefix decoding, which does not modify committed output and is flicker-free by design.Source: §6 · p. 9
- The prefix model dominates most of the Pareto frontier in its latency range, while continuation reaches slightly lower latency at a translation-quality cost.Despite training only on initial-prefix segmentations, prefix performance generalizes to random read schedules and continuations across language pairs, datasets, and metrics.Source: §6 · p. 9
- Confidence training reduces expected calibration error by 31–46% overall and 48–55% on early source prefixes against a FLEURS stable-prefix oracle.A single confidence threshold outperforms wait-k and suffix deletion in AUC across all language pairs and extends the reachable frontier to higher and lower latency.Source: §6 · p. 9
7 Limitations
The experiments cover three English-origin language pairs and one translation engine, so whether the findings generalize remains untested. The accompanying figures present confidence quality–latency frontiers and COMET trade-offs across datasets and language pairs.
7 Limitations
- Experiments cover only three language pairs, all originating in English, and use a single translation engine; generalization requires further testing.Source: §7 · p. 9
- Figure 4 shows confidence quality–latency frontiers on FLEURS for base, prefix, continuation, and full-control adapter models.Source: Figure 4 · p. 10
- Figure 5 lays out COMET quality–latency trade-offs across three datasets and EN→DE, EN→JA, and EN→ZH, comparing wait-k, suffix deletion, and confidence variants.Source: Figure 5 · p. 10
A Stable-prefix construction
Stable-prefix construction iterates over source-word prefixes to create training pairs, using a nondecreasing committed target length. The running maximum prevents the committed target length from shrinking as more source words arrive.
A Stable-prefix construction
- The construction takes a source sentence and translation model as input and outputs prefix training pairs.Source: A Stable-prefix construction · p. 12
- For each source prefix, the procedure adds a training pair containing that prefix and the target prefix through the current committed length.Source: A Stable-prefix construction · p. 13
- The committed target length is updated monotonically using the running maximum, so it never decreases as more source words arrive.The procedure uses the longest common target prefix.Source: A Stable-prefix construction · p. 13
B Reproducibility details
The reproducibility details specify latency and frontier calculations, training settings, and prompt formats for initial and continuing translation.
B Reproducibility details
- Average Lagging measures latency in source-word units using the source words read when each target unit is committed.The target units are whitespace-delimited words for German and characters for Japanese and Chinese.Source: B Reproducibility details · p. 13
- The attainable-envelope AUC compares sweeps over a shared latency interval and denominator, including budgets below a sweep’s minimum attainable latency.For MetricX, the calculation uses minima and the panel’s highest error; the five sweeps are base with wait-k, suffix deletion, or confidence, and prefix and continuation with confidence.Source: B Reproducibility details · p. 13
- The evaluated adapters use LoRA rank 16 and effective batch size 128, with fused AdamW at a constant learning rate of 5 × 10−6.The continuation stage initializes and trains the selected prefix adapter; both stages use gradient clipping at 1.0.Source: B Reproducibility details · p. 13
- Training uses a nominal 20 million source sentences per language direction, but stops early when validation loss stops improving.Table 5 reports updates and processed records; its caption says the records count is updates multiplied by effective batch size 128.Source: B Reproducibility details · p. 13, Table 5 · p. 13
C LLM Prompt Templates
- Prompts use one force-decode turn with the source observed so far and the committed target as a forced prefix when continuing translation.The model is instructed to translate only supported content and emit only the new continuation.Source: C LLM Prompt Templates · p. 13, §C.1 · p. 14
C.1 Prompts
- The prompts distinguish an initial translation from continuation while the source is incomplete and a final continuation after the full sentence arrives.The initial prompt starts with an empty target; the later prompts preserve the committed target and generate only what remains.Source: §C.1 · pp. 13–14
D MetricX evaluation
MetricX rescoring preserves the reported qualitative ordering across the evaluated language pairs and test sets: prefix performs strongest at higher latency, while continuation reaches the lowest-latency band.
D MetricX evaluation
- Prefix is strongest at higher latency, while continuation reaches the lowest-latency band under MetricX.The comparison is shown in Figure 6 and summarized by MetricX frontier AUC in Table 6.Source: D MetricX evaluation · p. 14, Table 6 · p. 14, Figure 6 · p. 15
- The MetricX check re-scores FLEURS, WMT24++, and CoVoST 2 using a reference-based error metric ranging from 0–25, where lower is better.The qualitative ordering carries over between metrics.Source: D MetricX evaluation · p. 14
- Table 6 reports one sweep per column on a shared latency interval for each dataset–language panel, with lower MetricX error preferred.Figure 6 is the MetricX version of Figure 5 and contains the same five sweeps.Source: Table 6 · p. 14, Figure 6 · p. 15