Source-linked AI summary
ReSCUE: Re-translation with Sentence Commitment for Unsegmented Long-Form Simultaneous Sign Language Translation
Sihan Ren, Gaozheng Li, Yuanshang Quan, Yiming Qin, Fuyi Yang, Chang Liu, Lan Xu, Minye Wu
TL;DR
Existing simultaneous SLT largely assumes pre-segmented, sentence-level inputs, leaving continuous long-form streams underexplored. ReSCUE trains and decodes for streaming video, stabilizes revisable translations, and commits completed sentences online. The authors report state-of-the-art simultaneous SLT performance across latency constraints and validate the approach on long-form streams.
Problem
Most SLT methods remain sentence-level and offline, while existing simultaneous SLT is restricted to pre-segmented inputs, leaving long-form unsegmented streams largely unexplored.
Source: §1 · p. 1Method
ReSCUE combines inference-aware training, stabilized re-translation, and sentence commitment for streaming translation of unsegmented long-form video.
Source: §1 · p. 2Results
Experiments demonstrate state-of-the-art simultaneous SLT performance across diverse latency constraints and validate ReSCUE on realistic long-form streaming scenarios.
Source: §8 · p. 9Takeaways & Limitations
The authors present ReSCUE as a practical path toward deploying SLT in real-world interactive environments and long-form streaming scenarios.
Source: §8 · p. 9Terminal punctuation can be committed prematurely or delayed, risking a sentence boundary inside the signing segment or multiple sentences being treated as one.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Simultaneous Sign Language Translation (SLT) is critical for real-time communication, yet existing methods remain largely confined to sentence-level, offline settings that assume pre-segmented inputs. These assumptions hinder deployment in realistic scenarios involving continuous, unsegmented video streams. We present ReSCUE, a unified framework for simultaneous SLT on unsegmented long-form sign language videos that aligns training and inference with realistic streaming conditions. ReSCUE combines inference-aware training to handle partial inputs, non-signing pauses, and multi-sentence contexts, stabilized re-translation to enable low-latency yet revisable predictions with reduced output flicker, and a sentence commitment mechanism for online segmentation and memory management. Experiments on standard sentence-level benchmarks show that ReSCUE achieves lower latency and the best translation quality under low-latency settings. On long-form unsegmented datasets, ReSCUE approaches the translation quality of oracle offline systems that use ground-truth sentence boundaries, while operating at substantially lower latency, demonstrating its practicality for real-world streaming scenarios.
1 Introduction
Existing SLT systems largely assume offline, pre-segmented sentences, leaving continuous long-form video streams underexplored. ReSCUE aligns training and inference with streaming conditions and evaluates simultaneous translation on sentence-level and long-form data.
Motivation
- Most existing SLT methods focus on sentence-level offline translation, while prior simultaneous SLT remains sentence-level and depends on pre-segmented inputs.The paper identifies long-form, unsegmented settings as largely unexplored, despite mature simultaneous-translation methods in text and speech.Source: §1 · p. 1
Contributions
- ReSCUE combines inference-aware training, stabilized re-translation, and sentence commitment to translate unsegmented long-form video under streaming conditions.The components address partial inputs, pauses and consecutive sentences; stabilize revisable output; and detect sentence completion while managing the input buffer.Source: §1 · p. 2
Evaluation
- On CSL-Daily and Phoenix-2014-T, ReSCUE establishes a new quality–latency Pareto frontier among existing simultaneous SLT methods.The paper also introduces How2Sign-Long and CSL-Story and reports validation of its approach on long-form sign-language videos, with a demo and user study.Source: §1 · p. 2
Motivation
- The paper frames ReSCUE as a step toward practical SLT for real-world interactive environments and long-form streaming scenarios.It identifies live presentations, meetings, classrooms, and public events as settings beyond the reach of systems not designed for continuous, long-duration interaction.Source: §1 · pp. 1–2
2 Related Work
Offline SLT has advanced but generally assumes complete, pre-segmented video. Existing simultaneous SLT work remains limited to sentence-level inputs, leaving long-form unsegmented translation challenges unresolved.
2.1 Offline Sign Language Translation
- Direct video-to-text SLT can reduce latency by removing intermediate recognition stages, but typically assumes pre-segmented inputs and full-video access.Earlier cascaded systems first produce gloss sequences with sign-language recognition, then translate them into text.Source: §2.1 · pp. 2–3
2.2 Simultaneous Translation
- Simultaneous translation commonly uses incremental decoding for monotonic outputs at fixed latency or re-translation to refresh hypotheses as input arrives.Prior commitment methods use auxiliary signals or modules, or a special boundary token.Source: §2.2 · p. 3
- Prior simultaneous SLT methods mostly cannot revise emitted tokens and remain limited to pre-segmented, sentence-level inputs, leaving long-form boundary detection unaddressed.Source: §2.2 · p. 3
3 Method
ReSCUE streams video through a dynamic buffer, training and decoding for partial, pause-containing, multi-sentence inputs. It stabilizes revisable hypotheses and commits completed sentences to manage continuous input.
3.1 Overview
- The streaming pipeline accumulates fixed-length video chunks in a dynamic buffer and re-translates the buffered segment into a revisable current-sentence hypothesis.Committed translations remain fixed and are concatenated with the current hypothesis; sentence commitment advances the input buffer.Source: §3.1 · pp. 3–4
3.2 Inference-Aware Training
- Inference-aware training mixes partial prefixes, silence-augmented instances, and adjacent multi-sentence clips to reflect streaming inputs.Prefix-to-prefix examples truncate video and target text together; silence segments use <SIL>, while neighboring sentences are combined when their temporal gap is under two seconds.Source: Figure 2 · p. 3, §3.2 · pp. 4–5
3.3 Stabilized Re-Translation
- Stabilized re-translation biases search toward the previous hypothesis and masks the last k raw-hypothesis tokens during incomplete input.The bias is disabled after the current prefix diverges from the previous output, allowing corrections; the final translation after the full video is unmasked.Source: §3.3 · p. 5
3.4 Sentence Commitment
- Sentence commitment uses stable terminal punctuation or a <SIL>-only hypothesis to finalize a sentence, append it to history, and advance the buffer.For punctuation-triggered commitment, the next sentence begins after the punctuation token’s arrival time, preserving visual evidence relevant to following sentences.Source: §3.4 · pp. 5–6
4 Long-Form Dataset
The paper introduces How2Sign-Long and CSL-Story for training and evaluating long-form sign language translation. The datasets provide multiple-sentence videos with different sources and annotation processes.
4 Long-Form Dataset
- The two datasets are introduced for both model training and realistic evaluation.Source: §4 · p. 6
- How2Sign-Long reassembles temporally continuous, fully annotated sentence segments into multi-sentence videos with natural non-signing segments.It preserves the original official data splits and sentence-level train/test distribution for comparison with prior work.Source: §4 · p. 6
- CSL-Story is curated from publicly accessible social-media fairy-tale narrations, with Chinese subtitles extracted by OCR and aligned to video.The alignment provides sentence-level temporal boundaries for each sample.Source: §4 · p. 6
5 Experiments
Experiments evaluate ReSCUE on sentence-level and long-form sign language translation, including quality, latency, and stability. ReSCUE improves the quality–latency trade-off in sentence-level comparisons and approaches oracle-reference quality on long-form videos at much lower latency.
5.1 Experiment Setup
- The experiments cover sentence-level CSL-Daily and Phoenix-2014-T and long-form CSL-Story and How2Sign-Long, measuring translation quality, latency, and output stability.Quality uses BLEU and ROUGE-L; latency uses AL and LAAL, and stability uses Normalized Erasure (NE). Long-form training includes silence segments and multi-sentence instances, unlike sentence-level training, which is constrained to prefix-to-prefix instances.Source: §5.1 · p. 7
5.2 Sentence-level SLT Performance
- A 40.4% latency reduction (693 ms vs. 1163 ms) on CSL-Daily accompanies ReSCUE’s low-latency performance, while k = 6 improves BLEU by 19.8% and ROUGE by 29.6% over existing simultaneous works.The high-accuracy configuration remains close to offline CV-SLT, with gaps of 4.22 BLEU and 3.04 ROUGE.Source: §5.2 · p. 7, Figure 4 · p. 7
5.3 Long-form SLT Performance
- ReSCUE achieves comparable translation quality to the oracle reference on How2Sign-Long and CSL-Story while reducing latency from tens of seconds to about 1.6 seconds.The reference uses ground-truth sentence boundaries unavailable in practical inference; direct application of Uni-Sign to long-form videos degrades performance.Source: §5.3 · p. 8, Table 2 · p. 8
5.4 Ablation Studies
- Removing any inference-aware training strategy lowers BLEU, and removing multi-sentence training causes the largest degradation, with higher latency and unstable outputs.Removing silence segments or prefix-to-prefix training also raises NE and can yield abnormally low or negative AL through hallucinated outputs or premature commitments.Source: §5.4 · p. 8
- Increasing β reduces NE but can lower BLEU when revisions are overly constrained; for fixed β, larger k lowers NE at the cost of higher AL.When β > 0.2, increasing k improves BLEU; the authors set β = 0.2 and choose k according to the target latency–quality trade-off.Source: §5.4 · p. 8
- Without sentence commitment, BLEU and AL/LAAL degrade, and the input buffer expands indefinitely as the model treats the stream as one sentence.Source: §5.4 · p. 8
6 Limitations & Discussion
ReSCUE’s long-form setting introduces commitment and silence-detection failure modes beyond standard sentence-level translation, alongside common SLT errors.
6 Limitations & Discussion
- Premature punctuation can commit a sentence before sufficient visual context, causing the next sentence to start mid-segment.Without Prefix-to-Prefix training, complete sentences may be generated from short prefixes; Mask-k can hide unstable early punctuation so it can be revised.Source: §6 · p. 9
- Delayed punctuation can grow the input buffer, overwrite earlier correct translations, and merge multiple sentences.This occurs more often without Multi-sentence training; punctuation near the output end may then be repeatedly hidden by Mask-k.Source: §6 · p. 9
- Incorrect <SIL> predictions can discard useful signing frames, especially near sentence beginnings, producing missing subjects or incomplete translations.The passage describes these errors as less frequent.Source: §6 · p. 9
- ReSCUE may also inherit common SLT errors, including noisy pose estimation and out-of-vocabulary signs.Source: §6 · p. 9
7 User Study
A user study compared ReSCUE with two offline SLT methods; ReSCUE received better subjective quality and fluency ratings and had the fastest response speed.
8 Conclusion
ReSCUE targets simultaneous translation of unsegmented long-form sign-language video, with experiments reporting strong performance across sentence-level and long-form settings. The appendices describe lower latency and stronger low-latency quality on a sentence-level benchmark, and quality close to the oracle reference on long-form datasets.
8 Conclusion
- ReSCUE is presented as operating on unsegmented long-form sign-language videos, aligning training with streaming conditions and using re-translation and sentence commitment for low-latency, stable translation.Source: §8 · p. 9
- Experiments across sentence-level and long-form settings demonstrate state-of-the-art simultaneous SLT performance across diverse latency constraints and validate ReSCUE in realistic streaming scenarios.Source: §8 · p. 9
A.1 Comprehensive Sentence-level SLT performance comparison
- On Phoenix-2014-T, ReSCUE has lower latency and higher translation quality in the low-latency regime than other methods.The passage refers to the extended latency-quality analysis in Fig. 11.Source: §A.1 · p. 14
A.2 Comprehensive SLT Performance on Long-form Datasets
- On CSL-Story and How2Sign-Long, ReSCUE closely approaches the Oracle Reference, a theoretical upper bound that is unattainable in practice.Source: §A.2 · p. 14
A.3 Quantitative Analysis of Sentence Commitment
The appendix analyzes sentence commitment, ablations, trade-off data, pose-based inputs, pre-training, and silence-segment construction and detection. It characterizes commitment as broadly following sentence structure, details the system’s input and training choices, and reports the silence detector’s threshold-sensitivity outcome.
A.3 Quantitative Analysis of Sentence Commitment
- Sentence commitment is evaluated by whether committed-sentence counts roughly match reference counts; the analysis finds broad alignment with sentence structure, with greater variance on How2Sign-Long.Abs. Diff. is a coarse measure rather than precise segmentation quality; only 7.67% of predicted silence tokens led to erroneous commitments across the two long-form test sets.Source: §A.3 · p. 14, Table 4 · p. 14
A.4 Extended Ablation Studies
- The extended long-form ablation reports LAAL as the latency metric because standard AL can yield uninterpretable negative values for overly long predictions.Source: §A.4 · p. 14
A.5 Raw Numerical Results for Trade-off Curves
- Raw trade-off results include Normalized Erasure for output stability and long-form results under different k settings.Source: §A.5 · p. 14
B.1 Pose-based Input Representation
- The pose-only representation partitions skeleton keypoints into semantic patches, including face, hands, and upper body, before embedding them for the Transformer.The representation is intended to preserve local spatial structure and focus on manual and non-manual sign features.Source: §B.1 · p. 15, Figure 6 · p. 15
B.2 Efficiency and Pre-training
- Pose-based inputs are chosen to reduce input size and inference time relative to computationally intensive RGB inputs; the SLT module is initialized from language-specific Uni-Sign-Pose weights.Chinese uses CSL-News pre-training and English uses YouTube-ASL pre-training; the strategy aims to provide motion primitives and semantic alignments before fine-tuning.Source: §B.2 · p. 15
C Silence Segments
- Silence segments are identified offline from hand kinematics and curated from sustained inactivity or safe regions around annotated sentence boundaries.The detector is used for training-data construction only, not at inference or for online translation or commitment.Source: C Silence Segments · p. 16
C.1 Threshold Sensitivity
- On manually annotated CSL-Story videos, the default thresholds of 0.03 speed and 0.65 height achieved the highest recall among nearby threshold choices.Source: §C.1 · p. 16, Table 5 · p. 16
C.2 Full-Video Evaluation
Full-video evaluation finds the silence detector useful but imperfect, with moderate boundary noise causing limited degradation. The accompanying study setup uses multiple translation systems and participant-rated samples.
C.2 Full-Video Evaluation
- The full-video silence detector achieves 0.8989 precision, 0.7268 recall, and 0.8038 F1, while leaving annotation noise for the downstream model to tolerate.Silence is treated as the positive class.Source: §C.2 · p. 17, Table 6 · p. 17
- Under ±15-frame boundary perturbation, BLEU-4 drops by 1.51 points, AL increases by 104 ms, and NE increases by 0.08.Erroneous commitments from predicted silence tokens increase from 5.22% to 5.97%; the mechanism commits on all-silence outputs and Mask-k can suppress premature silence predictions.Source: §C.2 · p. 17
D.1 Setup
- The study presents source video and real-time translated captions to hearing volunteers, who respond after obtaining sufficient information.Three systems—ReSCUE, Uni-Sign, and CV-SLT—process prerecorded signing videos.Source: §D.1 · p. 17
D.2 Sample Composition
- Four signing participants produced 20 videos containing 60 signed sentences, yielding 960 subjective ratings; the same participant pool also produced 480 objective trials.Ratings covered CV-SLT, Uni-Sign, and ReSCUE.Source: §D.2 · p. 17
D.3 Communication Task Protocol (Part I)
The user study evaluates communication and long-form comprehension with participant ratings and response measures. ReSCUE outperforms the two comparison systems in subjective quality and fluency, with significantly lower long-form response time.
D.3 Communication Task Protocol (Part I)
- In Part I, signing and hearing participants rate translation quality and fluency on five-point scales; Part II measures answer-selection accuracy and response time during continuous signing.In Part I, Deaf participants judge responses for relevance and correctness, while hearing participants judge whether translated text is understandable; fluency concerns timely, minimally disruptive interaction.Source: §D.3 · pp. 18–19, §D.4 · p. 19
D.5 Participant Instructions
- Participant instructions use everyday-life topics for Part I and prompted continuous descriptions for Part II, with hearing participants responding or selecting an answer as information becomes sufficient.The tasks are designed to assess communication and translation without predefined sentence boundaries.Source: §D.5 · pp. 19–20, §D.4 · p. 19
D.6 User Study Results
- Participant-level paired tests find that ReSCUE outperforms CV-SLT and Uni-Sign in subjective quality and fluency, with significantly lower long-form response time and competitive selection accuracy.The user-study results also report higher quality and fluency ratings, high accuracy, and faster response times.Source: Table 8 · p. 20, Table 7 · p. 20, §D.7 · p. 20
D.8 Consent, Compensation, and Review Considerations
- Participants gave written informed consent, could withdraw without penalty, and received compensation of at least four times the local minimum hourly wage.The study reports minimal foreseeable risk beyond ordinary communication and video-recording privacy considerations, and says it did not require IRB review under applicable policy.Source: §D.8 · pp. 20–21
E.1.1 Construction Strategy.
- How2Sign-Long retains 31,086 original training samples and adds 129,140 silence frames and 28,935 adjacent multi-sentence pairs.The construction strategy retains atomic samples for sentence-level supervision and adds non-signing intervals and context transitions.Source: §E.1.1 · p. 21, §E.1.2 · p. 21
E.1.3 Testing.
- How2Sign-Long testing concatenates adjacent sentences within a two-second continuity threshold, producing 297 long-form videos averaging 53.69 seconds.The test set includes ambiguous boundaries and silence segments longer than 10 seconds, with the longest lasting 36 seconds.Source: §E.1.3 · p. 21
E.2 CSL-Story
- CSL-Story supplies 750 manually corrected sentence annotations from OCR-extracted subtitles, averaging 63 sign-language sentences per video.Its mixed training pool adds 92 atomic narrative samples, 583 multi-sentence samples, and 471 silence frames.Source: §E.2 · p. 21, §E.2.1 · p. 22
E.2.2 Testing.
- The CSL-Story test set contains 12 continuous videos, with densely packed sentence boundaries and long silence spans that stress online segmentation and commitment.34.9% of adjacent sentence pairs are fewer than 10 frames apart; six silence segments exceed 10 seconds, with the longest reaching 21 seconds.Source: §E.2.2 · p. 22
E.2.3 Additional Training Analysis
The additional analyses examine whether a small CSL-Story dataset creates overfitting concerns, while the following appendix section describes dynamic generation of inference-aware training instances.
E.2.3 Additional Training Analysis
- Because CSL-Story is relatively small, the authors identify overfitting as a concern and use a jointly trained CSL-Story and CSL-Daily checkpoint for the main experiments.They conduct two additional analyses to examine whether this affects their conclusions.Source: §E.2.3 · p. 22
- The jointly trained checkpoint achieves 21.91 BLEU and 49.48 ROUGE on CSL-Daily and remains competitive with the CSL-Daily-only checkpoint.This comparison suggests that joint training does not simply overfit to CSL-Story.Source: §E.2.3 · p. 22, Table 9 · p. 22
- Training only on CSL-Story yields higher quality on CSL-Story than joint training, supporting its use as a standalone training resource.Table 10 reports performance for joint and CSL-Story-only training.Source: §E.2.3 · p. 22, Table 10 · p. 22
F Dynamic Construction of Inference-Aware Training Instances
- Dynamic training starts from sentence-level samples paired with complete translations and pure-silence samples paired with a single <SIL> token.Each iteration independently samples transformations controlled by prefix, multi-sentence, and silence probabilities.Source: F Dynamic Construction of… · p. 23
- Prefix truncation trains on proportionally truncated video and text, while multi-sentence extension appends a following-sentence prefix and any intervening silence.Silence augmentation adds surrounding non-signing video and corresponding <SIL> tokens; for prefix-truncated samples it is restricted to the clip beginning to avoid future-signing leakage. The transformations generate varied instances across epochs.Source: F Dynamic Construction of… · p. 23
G Inference Speed and Buffer Analysis
This section covers ReSCUE’s streaming inference and buffer characteristics, along with related methodological and responsible-deployment considerations.
G Inference Speed and Buffer Analysis
- Buffer length remained well controlled on both long-form datasets, with a worst-case maximum of 525 frames; a maximum-buffer-length safeguard can serve as a fallback.The buffer is cleared when sentence commitment is triggered, so inference cost is mainly determined by active-buffer length.Source: G Inference Speed and Buffer Analysis · p. 24, Table 11 · p. 24
- 115.42 ms average and 225.00 ms 95th-percentile runtime were measured across both long-form datasets; at the maximum 525-frame buffer, runtime was 1248.09 ms.Measurements used a single NVIDIA RTX 3090 GPU; with 15-frame chunks arriving every 500 ms, most translation steps finish before the next chunk arrives.Source: G Inference Speed and Buffer Analysis · p. 24
H Broader Impacts
- Deployment requires caution: machine translations may mislead in high-stakes settings, should not replace qualified interpreters without oversight, and require privacy protections and broader community-centered evaluation.The paper calls for informed consent, privacy-preserving storage and access control, disclosure that outputs are machine-generated, and evaluation across sign languages, variants, styles, and signer demographics.Source: H Broader Impacts · p. 24
I.1 Successful Cases
- A successful case illustrates Mask-k preventing premature commitment as the translation grows across successive inference steps.The example progresses from “The mulch side to keep weeds down where I walk” to a longer translation before ending with an unfinished phrase.Source: §I.1 · p. 25
NeurIPS Paper Checklist
- The paper reports no error bars for its main automatic SLT results, although Appendix E.6 reports participant-level paired statistical tests for the user study.Source: NeurIPS Paper Checklist · p. 34
16. Declaration of LLM usage
- The paper answers yes on declaring LLM use, identifying a multimodal language model in the SLT module.The paper further details the pose-based input representation and pretrained backbone in Appendix C.Source: §10 · p. 37