Source-linked AI summary
Simultaneous Translation between Sign Languages
Zetian Wu, Bowen Xie, Stefan Lee, Liang Huang
TL;DR
Existing sign-to-sign systems require the full source clip before emitting target signs, limiting live cross-language communication. This paper introduces simultaneous S2S translation with two wait-k regimes and a computation-aware latency metric; averaged across six directions, it reduces ca-Stream-AL by 38% with modest reported quality changes.
Problem
Prior sign-to-sign systems are offline, requiring the full source clip before emitting target signs and limiting live uses such as broadcast interpretation and two-way video calls.
Source: §1 · p. 1Method
Results
Averaged across six S2S directions, streaming reduces ca-Stream-AL by 38%, with a 9% DTW-PA-MPJPE increase and a 2.1 BLEU-4 drop versus the full-sentence baseline.
Source: §8 · p. 9Takeaways & Limitations
At low k, the decoder cannot reorder content it has not observed, creating a latency–anticipation trade-off when sign-language canonical word orders disagree.
Source: §8 · p. 10The latency estimate assumes the video-to-SMPL-X front end runs at ≥25 fps; below that rate, backlog accumulates and ca-Stream-AL becomes a lower bound on perceived latency.
Source: Limitations · p. 10
Abstract
from arXiv · showhide
Deaf and hard-of-hearing (DHH) signers cannot converse in real time across different sign languages today: existing sign-to-sign translation systems run offline, requiring the full source clip before any target sign is emitted. Live use cases - e.g. broadcast interpretation and two-way video calls - instead demand simultaneous output, while the source signer is still signing. We present, to our knowledge, the first simultaneous sign-to-sign (S2S) translation system, with two wait-k regimes: test-time wait-k inference applied directly to a full-sentence model, and a trained wait-k model via stochastic multi-path supervision. We further introduce ca-Stream-AL, a computation-aware latency metric for streaming output. Averaged across six S2S directions on both a smaller human-verified test set and a larger synthetic S2S corpus, our streaming system achieves a 38% ca-Stream-AL reduction while staying within a 9% DTW-PA-MPJPE increase and a 2.1 BLEU-4 drop compared to the full-sentence baseline. A word-order case study probes how the streaming model handles word order mismatch between different sign languages - a consequence of simultaneous translation.
1 Introduction
Existing sign-to-sign systems require the full source clip before producing target signs, limiting live cross-language conversation. This paper introduces simultaneous S2S translation, two wait-k regimes, and a computation-aware latency metric.
1 Introduction
- Sign language is a primary, natural communication medium for many Deaf and hard-of-hearing users, and its visual-spatial grammar carries information written text alone cannot.Live sign interpretation is already provided at major broadcasts such as White House press briefings.Source: §1 · p. 1
- Prior direct S2S systems are offline, requiring the entire source clip before emitting a target sign and ruling out live uses such as broadcasts and two-way video calls.The cascade alternative also incurs substantial latency.Source: §1 · p. 1
- Streaming sign output adds a latency challenge because target chunks take time to play while new source input arrives and later chunks are computed.Existing latency metrics do not fully capture this timing.Source: §1 · p. 1
- The paper presents the first simultaneous S2S system, adapting wait-k at test time and through trained stochastic multi-path supervision, and introduces ca-Stream-AL.The paper also analyzes how streaming handles word-order mismatch between sign languages.Source: §1 · pp. 1–2
- 38% lower ca-Stream-AL, with a 9% DTW-PA-MPJPE increase and a 2.1 BLEU-4 drop versus the full-sentence baseline, averaged across six S2S directions.The comparison covers a small human-verified test set and a larger synthetic S2S corpus.Source: §1 · p. 1
2 Direct Sign-to-Sign Translation
Direct S2S translates a signed utterance into another sign language within one model rather than cascading through text. The adopted pipeline tokenizes source video, generates target sign tokens, and renders them as an avatar.
2 Direct Sign-to-Sign Translation
- Direct S2S maps an utterance in one sign language to another within a single model, unlike a cascade through sign recognition, spoken-language translation, and sign generation.Prior direct systems remain offline; Inan et al. (2025) report back-translation BLEU-4 of 0 in two directions and a best direction of 6.81.Source: §2 · p. 2
- The adopted backbone uses a frozen VQ-VAE to tokenize 25 Hz SMPL-X poses into body, left-hand, and right-hand streams, which a shared transformer processes for T2S and S2S.The transformer is initialized from MBART-large-cc25, with SOKE sign embeddings and language-indicating special-token prefixes.Source: §2 · p. 2
- The pipeline incrementally tokenizes source video, emits target sign tokens after a k-token lag, and decodes them to SMPL-X poses for avatar rendering.Each token spans 160 ms; the same architecture serves six sign-to-sign directions.Source: Figure 1 · p. 2
- The VQ-VAE decoder reconstructs SMPL-X poses, which are rendered to an avatar in parallel with generation.Source: §2 · p. 2
3 Simultaneous Sign-to-Sign Translation
The simultaneous system applies wait-k to sign-token streams and trains a separate wait-k model by sampling source prefixes. Inference re-encodes each observed prefix to retain the backbone’s bidirectional encoder behavior.
3.1 Wait-k Inference
- Each sign token represents four frames (160 ms); wait-k first reads k source tokens, then reveals one additional source token for each generated target token.After the source is exhausted, decoding continues with the full source.Source: §3.1 · pp. 2–3
- Because the encoder is bidirectional, each new source token requires re-encoding the observed prefix so earlier hidden states can update; this scales encoder work as O(|x|^3).The authors report this is acceptable in practice because rendering currently dominates end-to-end latency.Source: §3.1 · p. 3
3.2 Trained Wait-k Model
- Exhaustively optimizing the prefix-to-prefix wait-k objective is impractical for long sign sequences, so training samples m target positions and masks each replica’s loss to its sampled position.Each replica receives the source prefix available at that target position.Source: §3.2 · p. 3
4 Computational-aware Lagging for Stream: ca-Stream-AL
ca-Stream-AL measures streaming sign-output latency on a shared source-time axis, combining schedule-induced lag with computation delay. It extends classical average lagging to account for target-tail duration, per-step processing, and accumulated backlog.
4.1 Classical Average Lagging
- Classical average lagging truncates the target tail at τ, potentially underestimating lag more severely as the target-to-source length ratio increases.For ratios near 1, the omitted tail is typically short and the error small.Source: §4.1 · pp. 3–4
4.2 Stream-AL
- Stream-AL measures signed target-progress deficit against the proportional ideal on a shared source-time axis, including the target tail after source consumption ends.The source-centric timeline uses fixed-duration chunks, with each chunk spanning 160 ms.Source: §4.2 · p. 4, Figure 2 · p. 4, §4.1 · p. 4
4.3 Computation-aware extension.
- Stream-AL assumes instantaneous computation, so the computation-aware extension adds wall-clock delays from encoding, inference, decoding, and rendering.The computation-aware recurrence carries work forward when it exceeds the time available between source arrivals.Source: §4.3 · pp. 5–6
- ca-Stream-AL models each step as starting after both its source chunk arrives and the previous step finishes, so excess computation carries forward as backlog.Its per-step computation includes encoding, transformer inference, decoding, and rendering; the measured pipeline begins at source SMPL-X motion.Source: §4.3 · p. 5
- ca-Stream-AL adds mean computation-induced delay to Stream-AL, separating schedule lag from computation lag and reducing to Stream-AL when computation delay is zero.The delay shifts nominal wait-k progress backward by the accumulated computation lag.Source: §4 · p. 3, §4.3 · p. 5
- ca-Stream-AL excludes upstream regression from raw video to SMPL-X under the assumption it keeps pace with the 25 fps source stream.If regression runs below the source frame rate, backlog may grow and reported ca-Stream-AL is a lower bound on true end-to-end latency.Source: §4.3 · pp. 5–6
5 Experiments
The experiments compare test-time wait-k inference and trained wait-k with a full-sentence baseline across six sign-language directions and two test sets. Across the representative CSL→DGS sweep, Train+TT generally has better quality–latency tradeoffs than TT-only at matched latency.
5.1 Setup
- Each source token spans 160 ms, and the latency sweep tests k values from 1 to 13 source sign tokens.The corresponding wait ranges from approximately 0.16 to 2.08 seconds.Source: §5.1 · p. 6
- The training corpus is a freshly generated back-translation S2S corpus, and the experiment follows the public training configuration of Wu et al. (2026).The backbone is MBART-large-cc25 fine-tuned from public pretrained weights with a frozen VQ-VAE tokenizer.Source: §5.1 · p. 6
- The setup evaluates all six directions among ASL, CSL, and DGS on BT and Strict test sets using DTW-PA-MPJPE, BLEU-4, and ca-Stream-AL.BT uses held-out synthetic-source, gold-target pairs; Strict uses sign-to-sign pairs from the Wu et al. (2026) dataset.Source: §5.1 · p. 6
- Train+TT applies stochastic multi-path supervision with m=5, while TT-only applies wait-k inference to the full-sentence checkpoint without retraining.The full-sentence baseline uses k=|x|.Source: §5.1 · p. 6
5.2 Quality–Latency Pareto Frontier
- At k=7 on CSL→DGS, Train+TT reaches 1.34 s ca-Stream-AL, about 3× below the full-sentence checkpoint's 4.32 s, with DTW-PA-MPJPE trailing by 0.19 and BLEU-4 by 2.4.Across all four Fig. 4 panels, Train+TT dominates TT-only at matched ca-Stream-AL except for a small portion of one panel.Source: §5.2 · p. 7
- Tables 1 and 2 report the best-tradeoff k per direction on BT and Strict, respectively, under Train+TT with m=5.Table 1 defines AL as ca-Stream-AL in seconds, DTW as DTW-PA-MPJPE, and BLEU as BLEU-4.Source: §5.2 · p. 7, Table 1 · p. 7
6 Analysis: Word Order and Streaming Anticipation
At low k, the decoder cannot reorder content it has not yet observed, forcing a choice between waiting and paying latency or anticipating and risking error. A CSL→DGS case study shows streaming works when content order aligns, but copies source order when a short clause requires reordering.
6 Analysis: Word Order and Streaming Anticipation
- At low k, the decoder cannot reorder content it has not yet observed, leaving it to wait and pay latency or anticipate and risk error.The languages differ in canonical order: DGS places sentential negation after the predicate, whereas CSL places it before.Source: §6 · p. 7
- The case study uses two CSL→DGS examples from the Strict subset, decoded by Train+TT at k=3.Each example presents the source signing and gloss alongside predicted target signing and reference DGS signing and gloss.Source: §6 · p. 7
- When content order aligns, the model emits BERG SCHNEE, decomposes “snow-white” into SCHNEE, and drops the source-only “color” marker.The source and target share mountain-then-snow order, so anticipation is unnecessary in this example.Source: §6 · p. 7
- When source and target orders disagree on a short clause, the model emits source-order NEG-NICHTS KALT instead of canonical DGS KALT NEG-NICHTS.At k=3, it emits as soon as the CSL negator arrives; the authors note that the per-direction anticipation lead-time analysis is left to future work.Source: §6 · pp. 7–8
7 Related Work
The paper extends prefix-to-prefix wait-k translation to sign streams and introduces a latency metric for sign output. It distinguishes this setting from prior streaming sign-to-text and text-to-sign work by its sign-stream source and rendered sign target.
Simultaneous machine translation
- The paper extends the prefix-to-prefix framework and wait-k policy from spoken-language MT to sign-token chunks, with stochastic multi-path training for the bidirectional encoder.The policy governs the pipeline from encoding and inference through target-sign decoding and rendering.Source: §7 · pp. 8–9
Duration-aware latency metrics
- ca-Stream-AL adapts computation-aware latency accounting to a source-centric formulation for sign output with fixed-duration chunks.Unlike ATD, it uses a proportional diagonal reference and permits negative values when the target-to-source length ratio is below one.Source: §7 · p. 9
Streaming sign-language translation (sign → text)
- Unlike prior streaming sign-to-text work, this system translates a sign stream into rendered sign chunks emitted in fixed wall-clock durations.That distinction motivates the source-centric, computation-aware ca-Stream-AL metric.Source: §7 · p. 9
Text → sign (T2S) translation
- Text-to-sign systems generate signs from spoken-language text, whereas this work generates sign tokens from a sign-language source under a streaming policy.The cited T2S approaches are described as offline and consuming text.Source: §7 · p. 9
8 Conclusion
The paper presents a simultaneous sign-to-sign system using two wait-k regimes and a computation-aware latency metric. Across six directions, it reduces latency with limited quality changes, while a case study identifies word-order mismatch as a streaming challenge.
8 Conclusion
- The system adapts wait-k through test-time inference on a full-sentence model and trained supervision, and introduces ca-Stream-AL to account for fixed sign-chunk duration.Target-centric AL and CA-AL under-report that playback duration.Source: §8 · p. 9
- Averaged across six S2S directions, the streaming system cuts ca-Stream-AL by 38% while staying within a 9% DTW-PA-MPJPE increase and a 2.1 BLEU-4 drop relative to the full-sentence baseline.Source: §8 · p. 9
- When sign-language word orders disagree, low k forces a trade-off between latency and anticipation because the decoder cannot reorder content it has not observed.The word-order case study identifies this as a central challenge of simultaneous translation.Source: §8 · p. 10
Limitations
The limitations concern the wait-k schedule, training and evaluation scope, and system costs. These constrain latency flexibility, the strength of quality assessment, and how the reported latency applies to deployment.
Limitations
- The fixed unit-slope schedule drifts from the proportional ideal when source/target length ratios differ from 1; ratio-aware scheduling is left to future work.The paper notes this would tighten Stream-AL most on highly asymmetric directions.Source: Limitations · p. 10
- Each Train+TT model commits to one training-time k, so serving a different latency budget requires retraining.Multi-k schemes are left to future work because of additional implementation cost.Source: Limitations · p. 10
- Training uses synthetic back-translation and uniform m=5 position sampling; source-bias interactions and sampling effects remain uncharacterized.Neither m nor the sampling distribution was tuned, and its effects could shift early-segment quality at low k.Source: Limitations · p. 10
- Anticipation analysis covers few cases, and streamed quality lacks native-signer evaluation, which is left to future work.Reported quality uses DTW-PA-MPJPE and a downstream-evaluator BLEU-4; human evaluation would assess perceived choppiness and unnatural early-segment artefacts.Source: Limitations · p. 10
- Rendering contributes about 132 ms per output chunk and dominates per-output cost; render-aware design is outside the paper’s scope.The pipeline runs at approximately 30 fps, compared with about 10 ms combined for encoder and LM decoder steps.Source: Limitations · p. 10
- Re-encoding each source prefix scales encoder work as O(|x|^3), a constraint that binds once rendering speeds up.A causal-mask approximation would amortize the cost but discard the pre-trained backbone’s bidirectional prior.Source: Limitations · p. 10
- The latency estimate assumes pose extraction runs at least 25 fps; below that rate, backlog accumulates and ca-Stream-AL becomes a lower bound.Long-input deployment would need to verify the frame-rate assumption empirically.Source: Limitations · p. 10
- Evaluation covers only ASL, CSL, and DGS, has no external streaming-S2S baseline, and uses greedy decoding rather than nonstreaming beam search.The paper also notes that beam search is ill-defined under partial-source observation.Source: Limitations · p. 11