PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning When to Commit from Partial Speech for End-to-End Simultaneous Speech Translation

Hieu Hoang, Amittai Axelrod

arXiv:2610.02612v1cs.CL

TL;DR

Simultaneous speech translation must produce target text before the utterance ends while preserving committed tokens. This paper adapts a speech language model with self-generated prefix supervision and compares streaming decoding protocols. Prefix training improves quality–latency frontiers, with multi-turn training also reducing commit-calibration error, especially at early prefixes.

  • Problem

    Simultaneous speech translation must emit useful target text from partial speech while preserving committed tokens, although direct speech models lack finalized transcripts and explicit word boundaries.

    Source: §1 · p. 1

  • Method

    The model is adapted using prefix supervision derived from its own complete- and partial-waveform translations, without transcripts or human translations.

    Source: §1 · p. 2

  • Results

    Prefix training improves quality–latency frontiers under both protocols; multi-turn training reduces commit-calibration error by 63–68% overall and 68–80% at early prefixes.

    Source: §4.5 · p. 6, §5 · p. 8

  • Takeaways & Limitations

    Confidence provides the broadest and most consistently competitive tested operating range, while multi-turn training improves calibration across the utterance, including at early prefixes.

    Source: §5 · p. 8

    Evaluation uses fixed 2 s blocks and Average Lagging only, without measuring wall-clock decoding latency, throughput, or memory.

    Source: §6 · p. 8

Abstract

from arXiv · show

Simultaneous speech translation must emit useful target text before the source is complete while preserving every committed token. We adapt a full-utterance speech language model using prefix supervision derived from its own complete- and partial-waveform translations, requiring neither transcripts nor human translations. We compare single-turn forced-prefix and multi-turn append-only decoding, use a confidence threshold to control the inference-time quality--latency trade-off, and vary the density of training prefixes with a separate synthesis margin. On FLEURS and CoVoST2 in three language directions, prefix training improves quality--latency frontiers over the unadapted model, and confidence provides the broadest consistently competitive operating range. Multi-turn decoding is generally stronger at low latency; under multi-turn training, commit-calibration error falls by 63--68% overall and 68--80% at early prefixes, whereas single-turn training provides only modest overall calibration gains and no early-prefix improvement. A small synthesis margin sometimes extends the frontier to lower latency, particularly on shorter utterances, while a larger margin degrades translation quality and calibration. Prefix adaptation therefore improves simultaneous speech translation, especially under multi-turn append-only decoding, while synthesis density introduces a non-monotonic quality--latency trade-off.

1 Introduction

Simultaneous speech translation must commit useful target text from evolving acoustic prefixes without revising prior output. This work adapts a speech language model with self-generated prefix supervision and compares streaming protocols and inference-time commitment controls.

1 Introduction

  • Simultaneous translation must balance useful early output against the risk of omissions, reordering, and unstable output while preserving committed tokens.Waiting for the complete utterance incurs sentence-level latency, while translating every incoming block risks errors.Source: §1 · p. 1
  • Direct speech language models must jointly handle recognition and translation uncertainty without explicit word boundaries or finalized transcripts.They are generally optimized for complete inputs and ordinary assistant responses, not irrevocable extensions from partial audio.Source: §1 · p. 1
  • Prefix targets are constructed by comparing translations of partial waveforms with the model’s full-audio translation, using a synthesis margin to control supervision density.The training-time margin is distinct from the inference-time confidence threshold, which controls when the trained model emits.Source: §1 · pp. 1–2
  • Single-turn forced-prefix decoding replays committed target text, whereas multi-turn append-only decoding reuses prior key–value states as audio blocks and target deltas accumulate.The single-turn replay can become computationally expensive for long sequences.Source: §1 · p. 2
  • A confidence threshold controls inference-time quality–latency trade-offs by determining whether the model emits new target text.The threshold is swept to vary the operating point.Source: §1 · p. 2
  • Prefix adaptation improves quality–latency frontiers under both decoding protocols and is associated with substantially better commit calibration under multi-turn decoding.The training targets come from the model’s own complete- and partial-waveform translations, without transcripts or human translations.Source: §1 · p. 2

2 Related Work

Prior work addresses simultaneous translation through fixed or learned read–write policies, streaming speech systems, and methods for stabilizing retranslation output. This paper applies append-only prefix translation directly to waveform prefixes and evaluates commitment calibration under partial speech.

Simultaneous translation policies

  • Fixed wait-k exposes a clear latency knob but cannot adapt delay to sentence-level uncertainty; this work instead thresholds evidence before appending target text.Learned policies use reinforcement learning or oracle-action supervision (Gu et al., 2017; Zheng et al., 2020); wait-k reads an initial prefix before alternating reads and writes (Ma et al., 2019).Source: §2 · p. 2

Streaming speech translation

  • Direct speech translation avoids the changing recognition hypotheses exposed by streaming ASR-to-MT cascades, but must jointly learn acoustic recognition, translation, and scheduling.The paper retains prefix retranslation while constraining output to be append-only and using waveform prefixes rather than reference or ASR transcripts.Source: §2 · p. 2

Output stability and prefix supervision

  • Prior stability methods constrain revisions at inference, whereas this work uses agreement with unconstrained full-audio translation to create training targets and preserve committed output.The cited approaches include target-suffix deletion (Arivazhagan et al., 2020), forced translation (Polák et al., 2022), and agreement between consecutive hypotheses (Liu et al., 2020).Source: §2 · p. 2

Confidence and calibration

  • A complete-utterance model applied to partial speech introduces a train–inference mismatch, motivating direct evaluation of commit calibration.The paper’s focus is adapting one offline speech language model to irrevocable translation from growing audio while holding its full-audio self-translation fixed as the supervision anchor.Source: §2 · p. 3

3 Experimental Setup

The experiments define append-only translation over fixed waveform blocks, synthesize supervision from full- and partial-audio outputs, and compare two decoding protocols. Evaluation uses held-out speech translation benchmarks, threshold-based inference controls, and commit-calibration measures.

3.1 Task and Languages

  • The task covers English speech translated to German, Japanese, and Chinese, with an irrevocable target prefix extended or held at each 2 s audio boundary.The visible translation is the prior commitment followed by any new delta; the final block may be shorter.Source: §3.1 · p. 3

3.2 Translation Prefix Synthesis

  • Prefix supervision compares each partial-audio translation with the full-utterance translation and uses their longest common prefix to determine supervised commitment.Agreement is measured in words for German and characters for Japanese and Chinese; a running maximum makes the chain monotone.Source: §3.2 · p. 3

3.2.1 Synthesis Margin

  • The synthesis margin lets the full-audio translation rescue a less probable token, potentially lengthening prefix supervision at the risk of lower translation quality.At δ = 0, the original prefix algorithm is recovered.Source: §3.2.1 · p. 3

3.3 Decoding Protocols

  • Single-turn prompts replay the committed target as a forced assistant prefix, while multi-turn prompts append alternating audio blocks and target deltas to one causal record.Both protocols preserve prior output; multi-turn decoding reuses prior assistant deltas and reduces redundant computation.Source: §3.3 · pp. 3–4

3.4 Model and Training

  • Both protocols train Qwen2.5-Omni-7B from the same base model, with single-turn examples targeting partial or complete translations and multi-turn records supervising waits and deltas.The decoder uses LoRA adapters; the multi-turn loss covers every assistant delta and closing token in one forward pass.Source: §3.4 · p. 4

3.5 Data

  • Training uses LibriSpeech and Common Voice, with FLEURS and CoVoST 2 as test sets; EN→ZH full-audio outputs often contained conversational boilerplate, especially in Common Voice.Trailing clauses inviting more audio or translation help were removed, with prefix translations truncated accordingly while preserving monotonicity.Source: §3.5 · p. 4

3.6 Validation and Checkpoint Selection

  • Single-turn checkpoints are selected using COMET and attainable COMET over a shared latency grid, while multi-turn selection also considers productive-partial COMET and WAIT-balanced accuracy.Candidates undergo autoregressive development rollouts over the same nine confidence thresholds; single-turn candidates must pass a full-audio quality gate.Source: §3.6 · pp. 4–5

3.7 Inference Mechanisms and Metrics

  • Inference compares a confidence margin against wait-k and target-suffix deletion, measuring translation quality with COMET-22-DA, latency with Average Lagging, and calibration with expected calibration error.Table 2 reports frontier AUC for base and single-turn models over each panel’s shared latency interval.Source: §3.7 · p. 5, Table 2 · p. 5

4 Results

Prefix-trained models improve quality–latency frontiers over the base models in both decoding protocols, with confidence offering a strong operating range. Synthesis-margin effects vary by setting, while multi-turn training yields larger commit-calibration gains than single-turn training.

4.1 Single-turn Forced-prefix Decoding

  • The trained model consistently outperforms the base across all language directions, and confidence gives the highest or tied trained-model AUC in all three directions.Its operating range exceeds those of the other tested mechanisms.Source: §4.1 · p. 5

4.2 Multi-turn Append-Only Decoding

  • Multi-turn training greatly outperforms the base, which can lose translation quality as more source context arrives; confidence also has the best latency range.The comparison uses AUC over each panel’s shared latency interval.Source: §4.2 · p. 5, Table 3 · p. 5

4.3 Synthesis Margin

  • On shorter CoVoST 2 utterances, δ = 0.2 extends the frontier to lower latency, while δ = 0.5 is uncompetitive and can degenerate on FLEURS EN→ZH.The gains from δ = 0.2 vary by language pair and test set; δ = 0.5 produces more severe errors, including unsupported content or repetition.Source: §4.3 · pp. 5–6

4.4 Single-turn vs. Multi-turn

  • Multi-turn models dominate the frontier over their latency range, especially for EN→DE/ZH; at lower latency, δ = 0.2 can outperform single-turn, while single-turn can trade quality for still lower latency.The AUC comparison uses shared latency intervals for each dataset–language panel.Source: §4.4 · p. 6, Table 4 · p. 6

4.5 Commit Calibration

  • Multi-turn prefix training reduces commit ECE by 63–68% overall and 68–80% at early prefixes, while single-turn training reduces overall ECE by 3–11% but worsens early-prefix ECE by 2–5%.Early-prefix calibration matters because low-latency translation decisions use little audio; calibration alone does not explain overall model differences.Source: §4.5 · pp. 6–7

5 Conclusion

Prefix learning adapts a speech language model for flicker-free simultaneous translation without transcripts or human translations. It improves quality–latency frontiers under both decoding protocols, with confidence offering the broadest consistently competitive operating range.

5 Conclusion

  • Prefix learning adapts a speech language model for flicker-free simultaneous translation without transcripts or human translations.The training uses target-prefix supervision derived from the model’s own translations.Source: §5 · p. 8
  • Prefix adaptation improves the quality–latency frontier for both single-turn forced-prefix and multi-turn decoding.Among tested inference mechanisms, confidence provides the broadest and most consistently competitive operating range.Source: §5 · p. 8
  • Under multi-turn training, dense boundary supervision is associated with substantially better confidence calibration across the utterance, including at early prefixes.Single-turn training yields only modest overall gains and no early-prefix improvement; the contrast may help explain stronger low-latency commitments under multi-turn decoding.Source: §5 · p. 8
  • A small synthesis margin sometimes extends low-latency coverage, particularly on shorter utterances, whereas a larger margin degrades translation quality and calibration.Synthesis density therefore introduces a non-monotonic quality–latency trade-off.Source: §5 · p. 8

6 Limitations

The evaluation is bounded by the tested models, languages, benchmarks, and fixed-block setup, as well as by self-teacher targets and automatic assessment. The protocol comparisons, calibration results, and benchmark measurements each have stated limits.

6 Limitations

  • Experiments cover one base model, three English-source directions, and read or prompted speech; transfer to other models, languages, accents, domains, and spontaneous conversation remains untested.The pretraining data were not audited for overlap with evaluation sets, so benchmark contamination cannot be ruled out.Source: §6 · pp. 8–9
  • Stable-prefix targets use lexical longest-common-prefix agreement with the base model’s full-audio translation, so they can inherit its errors and biases without guaranteeing semantic support or correctness.Source: §6 · p. 8
  • Source time advances in fixed 2 s blocks with known utterance boundaries; adaptive segmentation and continuous unsegmented input are not evaluated.Average Lagging measures policy delay only; network delay, wall-clock latency, throughput, memory use, and the expected cache-reuse benefit are unmeasured.Source: §6 · p. 8
  • Translation quality is measured automatically with COMET, without human evaluation of adequacy, fluency, or partial-output usefulness.Monotone output prevents flicker but cannot revise an early erroneous commitment.Source: §6 · pp. 8–9
  • Calibration uses 64 FLEURS oracle chains per direction and base-derived continue-versus-stop labels, so it characterizes those commit contexts and prompt representations rather than generic or out-of-domain calibration.It also does not measure human judgments of semantic certainty.Source: §6 · p. 9
  • Single-turn versus multi-turn comparisons change both causal representation and training-example structure, so their differences cannot be attributed solely to matching commit-decision contexts without a crossed study.Source: §6 · p. 9

A Reproducibility Details

The appendix specifies the data, training setup, latency and frontier calculations, and decoding details used for reproducibility. It also notes that computational efficiency is not empirically measured.

Audio and target units

  • Audio is resampled to 16 kHz and segmented at a fixed 2-second cadence; target prefixes use words for German and characters for Japanese and Chinese.A final audio block shorter than 0.5 seconds is merged with the preceding block.Source: A Reproducibility Details · p. 10

Corpus and splits

  • Final evaluation uses 585 FLEURS utterances and 1,999 usable CoVoST 2 utterances per language direction; training data comprise 139,684 distinct English utterances.The mixture includes LibriSpeech-100, LibriSpeech-360, and Common Voice; every 50th chain is withheld, with controlled selection using at most 400 held-out chains.Source: A Reproducibility Details · p. 10, Table 5 · p. 11

Optimization

  • Training uses Qwen2.5-Omni-7B adapters on decoder projections and MLP layers, with frozen audio and vision towers and a 6,000-update ceiling.The setup uses rank-32 LoRA, learning rate 5×10−6, effective batch size 64, and omits multi-turn records longer than 1,200 tokens.Source: A Reproducibility Details · p. 11, Table 6 · p. 11

Latency measurement

  • Average Lagging uses source blocks read at each target commitment, with a blocks-to-seconds conversion specified separately.An empty output is assigned source-end latency.Source: A Reproducibility Details · p. 11

Frontier evaluation

  • Frontier selection compares attainable COMET across candidates at or below each latency budget, using a shared interval and denominator for the sweeps.Tables 2 and 3 compare base and selected trained models under three decoding mechanisms; Table 4 compares confidence frontiers.Source: A Reproducibility Details · p. 11

Cached decoding and scope

  • Multi-turn append-only decoding retains prior audio and accepted target states, while single-turn forced-prefix decoding replays the committed target at each boundary.The study does not report wall-clock latency, throughput, or memory use, so it does not provide an empirical efficiency comparison.Source: A Reproducibility Details · pp. 11–12

B Translation Prefix Synthesis

Prefix synthesis uses complete-audio translations as anchors and partial-audio decodes to construct monotone target commitments. A synthesis margin can substitute an aligned full-audio token for a greedy partial-audio token.

Self-anchored synthesis

  • Each chain pairs a complete-audio anchor with decodes from proper audio prefixes, using the final prompt for the anchor and streaming prompts for partial inputs.An aligned full-audio token replaces a greedy token when it is within synthesis margin δ of the greedy probability; δ = 0 gives ordinary greedy prefix decoding.Source: B Translation Prefix Synthesis · p. 12

Commitment construction

  • Lexical agreement and its running maximum determine monotone target commitments across audio boundaries.The final boundary commits the complete anchor regardless of earlier agreement.Source: B Translation Prefix Synthesis · p. 12

Normalization

  • Before target-unit segmentation, normalization trims surrounding whitespace and collapses internal whitespace.The procedure applies the same commitment construction to each utterance chain.Source: B Translation Prefix Synthesis · p. 12

C Chinese Self-Teacher Cleanup

The Chinese self-teacher cleanup removes specified invitation-style suffixes from complete-audio translations, then checks that reconstructed commitments remain prefixes of the cleaned anchor. An audit reports how often the rule changed chains and how ambiguous cases were handled.

Cleanup rules

  • The cleaner removes suffixes at sentence, clause, or comma boundaries when conditional wording, topic cues, and invitation cues satisfy specified text-window rules.It also removes direct suffixes equivalent to “you can continue” or “you can tell me anytime,” iterating to a fixed point while preserving retained-translation punctuation.Source: C Chinese Self-Teacher Cleanup · p. 12

Reconstruction checks

  • After cleaning the complete-audio anchor, intermediate commitments are truncated to its length and checked to remain monotone prefixes ending at the full anchor.Anchors with no translation after cleaning are quarantined rather than converted into empty training targets.Source: C Chinese Self-Teacher Cleanup · p. 12

Algorithm 1

  • Algorithm 1 constructs a chain of monotone commitments from a waveform, its audio boundaries, a frozen model, and synthesis margin δ.Its stated output is the commitment sequence for those boundaries.Source: C Chinese Self-Teacher Cleanup · p. 12

Audit

  • The cleanup changes 92,039 of 139,684 EN→ZH chains (65.89%), including 36,684 Common Voice and 55,355 LibriSpeech chains.Manual review covers 20 sampled changed chains and all 15 flagged ambiguous or empty chains; seven legitimate ambiguous cases remain unchanged and eight boilerplate-only chains are quarantined.Source: C Chinese Self-Teacher Cleanup · p. 12

D LLM Prompt Templates

The prompt templates keep a shared system persona and streaming instruction while distinguishing single-turn forced-prefix inputs from multi-turn append-only dialogue. Training and inference preserve each protocol’s corresponding target and cache behavior.

D.1 Shared System Message and Instruction

  • The shared instruction asks the model to translate ongoing English speech into German, emit only newly certain content after each segment, and finish the translation at the end-of-audio marker.The system persona and fixed streaming instruction are shared across the templates; language names change for Japanese and Chinese.Source: §D.1 · p. 13, D LLM Prompt Templates · p. 13

D.2 Single-Turn Forced-Prefix Decoding Template

  • In the single-turn template, each decision contains all audio observed so far and supplies the prior translation as an open assistant prefix.Only the continuation and closing token are generated and supervised; initial partial and complete tasks leave the forced prefix empty.Source: §D.2 · p. 13
  • Single-turn examples show a German target prefix extended from “Wissenschaftler entdeckten” to the completed translation as more audio arrives.At inference, the forced and generated target region is recomputed at each boundary and discarded from the cache, while the visible commitment stays unchanged.Source: §D.2 · p. 13

D.3 Multi-Turn Append-Only Decoding Template

  • In multi-turn decoding, the fixed instruction appears only with the first audio block, each later user turn adds one block, and assistant turns contain only the newly committed delta.An empty assistant turn explicitly represents waiting.Source: §D.3 · p. 13
  • The multi-turn example appends successive German translation deltas across three audio blocks, ending with the final phrase after the end-of-audio marker.The example illustrates the append-only dialogue format.Source: §D.3 · p. 13
  • Multi-turn training masks system, user, instruction, and audio-placeholder tokens while supervising assistant deltas and closing tokens; inference retains the full sequence in one persistent cache.Each decision can attend to earlier audio blocks and committed deltas, but not future audio.Source: §D.3 · p. 13
Loading 2610.02612v1…