Source-linked AI summary
Better Late Than Never: Meta-Evaluation of Latency Metrics for Simultaneous Speech-to-Text Translation
Peter Polák, Sara Papi, Luisa Bentivogli, Ondřej Bojar
TL;DR
Latency metrics for simultaneous speech translation can yield inconsistent assessments, particularly when evaluation uses artificially segmented audio. The paper evaluates metrics across systems and settings, proposes YAAL, LongYAAL, a diagnostic test, and SOFTSEGMENTER, and finds that automatic metrics can approximate actual latency when best practices are followed. It recommends long-form evaluation when feasible because short-form segmentation can distort evaluation.
Problem
Existing simultaneous speech translation latency metrics produce inconsistent assessments, while the reasons for their disagreements and their reliability across short- and long-form settings remain unclear.
Source: §1 · p. 1Method
The paper meta-evaluates latency metrics across systems and regimes, introduces YAAL and a degeneracy test for short-form evaluation, and develops LongYAAL and SOFTSEGMENTER for long-form evaluation.
Source: §1 · p. 1Results
Automatic latency metrics can reliably approximate actual latency when best practices are followed, but artificial input segmentation can make short-form evaluation misleading.
Source: §7 · p. 8Takeaways & Limitations
The authors recommend prioritizing long-form evaluation when feasible and using YAAL with its diagnostic test for short-form systems.
Source: §7 · p. 8The analysis focuses on IWSLT Shared Task systems and high-resource languages, and word-level alignment remains susceptible to errors with disfluencies or noise.
Source: §7 · p. 8
Abstract
from arXiv · showhide
Simultaneous speech-to-text translation systems must balance translation quality with latency. Although quality evaluation is well established, latency measurement remains a challenge. Existing metrics produce inconsistent results, especially in short-form settings with artificial presegmentation. We present the first comprehensive meta-evaluation of latency metrics across language pairs and systems. We uncover a structural bias in current metrics related to segmentation. We introduce YAAL (Yet Another Average Lagging) for a more accurate short-form evaluation and LongYAAL for unsegmented audio. We propose SoftSegmenter, a resegmentation tool based on soft word-level alignment. We show that YAAL and LongYAAL, together with SoftSegmenter, outperform popular latency metrics, enabling more reliable assessments of short- and long-form simultaneous speech translation systems. We implement all artifacts within the OmniSTEval toolkit: https://github.com/pe-trik/OmniSTEval.
1. Introduction
Latency metrics for simultaneous speech translation often produce inconsistent system assessments, and the reasons for those inconsistencies remain unclear. The paper evaluates metrics across systems, language pairs, and short- and long-form regimes, identifying segmentation-related structural bias and introducing tools to address it.
1. Introduction
- Existing latency metrics produce substantially different rankings, raising concerns about valid evaluation and meaningful system comparisons.The IWSLT 2023 comparison includes True Latency, YAAL, and five official metrics.Source: §1 · p. 1, Figure 1 · p. 1
- The paper presents a comprehensive meta-evaluation across diverse systems, language pairs, and short- and long-form settings.Its analysis draws on systems submitted to recent IWSLT simultaneous speech translation shared tasks.Source: §1 · p. 1
- The paper identifies segmentation as a structural source of inconsistent latency evaluations, rather than the simplifying assumptions shared by existing metrics.It also introduces YAAL, a diagnostic test for degenerate policies, SOFTSEGMENTER, and LongYAAL for long-form audio.Source: §1 · p. 1
2. Background
The background describes incremental SimulST systems and distinguishes short-form evaluation on presegmented recordings from long-form evaluation on continuous audio streams. It reviews established short-form latency metrics and the reference-based resegmentation used to apply them to long-form evaluation.
2. Background
- Short-form evaluation divides recordings into a few-second segments, usually following sentence boundaries, and feeds audio chunks incrementally while the system emits translation tokens.The paper assumes incremental systems whose outputs cannot be revised.Source: §2.1 · p. 2, §2 · p. 2
2.1. Short-Form SimulST Latency Metrics
- Short-form metrics differ in how they measure delay: AP, AL, LAAL, DAL, and ATD use distinct assumptions, coverage, or adjustments.LAAL adjusts the AL length ratio to address overgeneration, while DAL penalizes each write with a minimum delay and considers all hypothesis tokens.Source: §2.1 · p. 2
2.2. Long-Form SimulST Latency Metrics
- Long-form evaluation applies short-form metrics to reference-aligned segments of unsegmented audio, but system-provided resegmentation can reduce accuracy and make systems incomparable.Systems may not provide the required timestamps, and relying on their potentially low-quality segmentation can lower evaluation accuracy.Source: §2.2 · p. 2
3. Overcoming the Pitfalls in SimulST Latency Metrics
The section identifies segmentation and tail-word handling as sources of distorted short-form latency estimates, then presents YAAL, SoftSegmenter, and LongYAAL for short- and long-form evaluation.
3.1. The Short-Form Regime
- Oracle segmentation and zero-delay tail-word completion distort short-form evaluation, while including tail words or cutting off at τ(X) can bias latency estimates.The bias can depend on the system's policy.Source: §3.1 · pp. 2–3
3.1.1. Yet Another Average Latency (YAAL)
- YAAL refines LAAL's cutoff to include only words generated strictly before input ends, excluding tail words to improve consistency across segmentation regimes.Source: §3.1.1 · p. 3
3.1.2. The Long-Form Regime
- Long-form audio is more realistic, but short-form metrics do not transfer directly; γ variability, AP convergence, ATD assumptions, and alignment errors motivate new resegmentation and metric extension.Source: §3.1.2 · pp. 3–4
3.1.3. SOFTSEGMENTER
- SoftSegmenter uses softer alignment between hypotheses and reference segments, preserving token delays to prevent future-segment matches and applying character-level similarity after punctuation-aware matching.Source: §3.1.3 · p. 4
3.1.4. Long-Form YAAL (LongYAAL)
- LongYAAL includes emissions beyond segment boundaries but excludes final-stream tail words; for a single-segment stream, it coincides with YAAL.Source: §3.1.4 · p. 4
4. Experimental Settings
The experimental setup spans short- and long-form SimulST data and defines an alignment-based true-latency reference for comparing latency metrics. The two regimes use different alignment tools to estimate that reference.
4.1. Data
- Short-form evaluation uses IWSLT 2022/2023 systems and MuST-C tst-COMMON logs; long-form uses IWSLT 2025 logs across specified language-pair datasets.Reported system statistics exclude incomplete logs, including missing segments, mismatched predictions and delays, or severely hallucinated outputs.Source: §4.1 · p. 4
4.2.1. Metrics
- True latency is the average delay between each target word and its corresponding source word, restricted to aligned words emitted before the end-of-source signal.For a target word aligned to multiple source words, its source delay is the time the speaker finishes the last corresponding source word.Source: §4.2.1 · p. 4
- Metrics are compared pairwise on systems sharing a test set and language pair, with accuracy measuring agreement against true-latency rankings.Accuracy measures ranking agreement, not the magnitude of latency differences.Source: §4.2.1 · p. 5
4.2.2. Implementation
- Short-form true latency uses Montreal Forced Aligner timestamps and awesome-align word mappings; long-form uses WhisperX and resegments hypotheses before alignment.Short-form word mappings retain alignments with probability above 0.5 and exclude punctuation.Source: §4.2.2 · p. 5
5. Short-Form Evaluation
In short-form evaluation, degenerate policies distort automatic latency estimates because current metrics include post-boundary tail words. YAAL and a test comparing expected and observed simultaneous-word fractions support more reliable evaluation.
5. Short-Form Evaluation
- Degenerate systems emit a low-latency prefix, then translate most words offline after the segment boundary; existing metrics count those tail words, while YAAL counts only pre-boundary words.The illustrated degenerate policy quickly emits a few words, then waits until the segment ends before translating most of the sentence.Source: §5 · p. 5, Figure 3 · p. 6
- YAAL compares observed and expected fractions of simultaneously translated words; segments shorter than YAAL latency are treated as having no simultaneous words.The expected fraction assumes words are translated evenly across each segment, using average segment length and YAAL latency.Source: §5 · p. 5
- Across all system pairs, YAAL reaches 98% accuracy while every other metric trails by more than 25% absolute; after filtering degenerate systems, YAAL and LAAL tie, with ATD and AP still 6% and 11% behind.AL and DAL exceed 95% accuracy after filtering; YAAL and LAAL also outperform the other metrics within same-team pairs.Source: Table 3 · p. 6, §5 · p. 6
- Most systems show a linear expected–observed word-proportion relationship; extreme cases differ by 42%–81%, filtering outliers leaves maximum differences of 14%, and gaps above 20% indicate degenerate simultaneous policies.Source: §5 · p. 5
- After removing degenerate systems, YAAL outperforms all metrics and LAAL ties within the 95% confidence interval; YAAL accuracy reaches 90% at 40–240 ms differences and 99% at 110–310 ms.At larger true-latency differences, ATD remains 5–10% behind YAAL/LAAL, while AP lags YAAL by 10–15%.Source: §5 · pp. 6–7, Figure 6 · p. 7
- Short-form evaluation places 41–72% of translations after the segment-end signal across latency regimes, often effectively translating offline.The reported share ranges from 41% in 1–2 s regimes to 72% in 4–5 s regimes; the authors recommend long-form evaluation because artificial boundaries and tail-word handling can misrepresent behavior.Source: Table 4 · p. 7, §5 · p. 7
6. Long-Form Evaluation
Resegmentation substantially improves long-form latency accuracy, and LongYAAL, LongLAAL, and LongDAL are the best-performing long-form metrics. The authors recommend LongYAAL because it evaluates all output words without modification.
6. Long-Form Evaluation
- In the short-form-output comparison against gold segment boundaries, SOFTSEGMENTER yields significantly higher latency-evaluation accuracy than MWERSEGMENTER.The authors caution that short-form outputs may not reflect true long-form system behavior.Source: §6 · p. 7
- LongYAAL, LongLAAL, and LongDAL each exceed 93% accuracy and tie within the 95% confidence interval; LongYAAL is recommended.LongYAAL considers all output words without modification, which may offer better generalization across test sets.Source: Table 6 · p. 7, §6 · pp. 7–8
- Resegmentation raises long-form accuracy: without it, the best AL metric trails StreamLAAL by 17 absolute points and LongYAAL by 29 points.With resegmentation, StreamLAAL and LongLAAL differ by 12 percentage points despite sharing the same metric definition; their resegmentation tools differ.Source: §6 · p. 7, Table 6 · p. 7
- With resegmentation, LongATD and LongAL exceed 92% accuracy, while StreamLAAL reaches 82% and LongAP 71%.The authors attribute LongAP’s low accuracy likely to sensitivity to variable segment lengths.Source: Table 6 · p. 7, §6 · p. 8
- LongYAAL, LongLAAL, and LongDAL show similar accuracy trends as true latency differences grow; their intermediate-gap differences are marginal at 1–2% absolute.LongYAAL reaches 90% accuracy around a 260 ms score difference and nearly perfect accuracy around 440 ms; Figure 7 plots accuracy by true-latency and LongYAAL-value differences, with 95% confidence intervals.Source: §6 · p. 8, Figure 7 · p. 8
7. Conclusions
The conclusions identify artificial segmentation as a source of misleading short-form evaluation and recommend long-form evaluation when feasible. They introduce YAAL for short-form evaluation and LongYAAL with SOFTSEGMENTER for long-form evaluation, while noting limits in study scope and alignment robustness.
7. Conclusions
- LongYAAL with SOFTSEGMENTER addresses long-form evaluation, where existing short-form metrics require resegmentation; it outperforms StreamLAAL and nearly matches short-form YAAL.SOFTSEGMENTER uses word-level alignment, which remains susceptible to errors with disfluencies or noise.Source: §7 · p. 8
- Artificial input segmentation can make short-form evaluation misleading and incentivize degenerate simultaneous policies; up to 70% of words may be generated after a segment ends.The authors recommend prioritizing long-form evaluation when feasible; YAAL and a diagnostic based on expected versus observed proportions of simultaneous words address short-form evaluation.Source: §7 · p. 8
- The analysis centers on IWSLT Shared Task systems and high-resource languages, so findings should be reconfirmed in low-resource settings.The authors note that the systems may not represent the range of techniques or data conditions in broader academic or industrial settings.Source: §7 · p. 8
- Existing latency metrics rely on an oracle policy assuming uniform word timings, no silences, and monotonic alignments, assumptions less valid with longer inputs.The authors say resegmentation helps restore these assumptions only to some extent, and identify alternative paradigms that avoid them as a future direction.Source: §7 · p. 8