PaperlayerView exact revision on arXiv

Source-linked AI summary

SignRAG: Unified Retrieval-Augmented Gloss-Free Sign Language Translation

Zhi Rao, Yucheng Zhou, Qianran Sun, Yiqing Huang, Longcan Yuan, Jiayi Hou, Chengwen Yao, Lin Cheng, Donghui Sun, Xiaoxin Chen, Jun Wan

arXiv:2610.11371v1cs.CLcs.CVcs.MM

TL;DR

Gloss-free SLT pretraining has largely targeted encoder–decoder models, limiting direct applicability to decoder-only LLMs. SignRAG combines hierarchical pretraining, target-domain retrieval, and retrieval-aware reinforcement fine-tuning. It reports new state-of-the-art results across multiple benchmarks, including a gloss-free lead over gloss-supervised methods on CSL-Daily.

  • Problem

    Existing gloss-free SLT pretraining largely targets encoder–decoder models, while replacing them with a stronger LLM can let the language branch dominate and under-train the visual pathway.

    Source: §1 · p. 1

  • Method

    SignRAG stages sign representation learning before joint sign-encoder and LLM pretraining, adds target-domain retrieval cues, and uses RUG-RFT to combine translation-quality and retrieval-utility rewards.

    Source: §1 · p. 2

  • Results

    SignRAG achieves new state-of-the-art performance across multiple SLT benchmarks.

    Source: §5 · p. 10

  • Takeaways & Limitations

    On CSL-Daily, SignRAG surpasses gloss-supervised methods across all reported metrics, which the authors describe as a first for gloss-free SLT.

    Source: §5 · p. 10

    Million-scale retrieval remains a limitation; coarse-to-fine retrieval may discard candidates that score highly under exhaustive token-level matching.

    Source: §E.5 · p. 28

Abstract

from arXiv · show

Contemporary decoder-only large language models (LLMs) have demonstrated strong capabilities across a wide range of domains. However, existing pretraining paradigms for gloss-free sign language translation (SLT) are largely designed around conventional encoder-decoder pretrained language models, which limits their direct applicability to decoder-only LLMs. To address this limitation, we propose SignRAG, a unified framework combining hierarchical pretraining, target-domain retrieval augmentation, and retrieval-aware reinforcement fine-tuning. Hierarchical pretraining first learns linguistically grounded sign representations and then jointly aligns the sign encoder with an LLM, mitigating cross-modal optimization imbalance. For downstream adaptation, SignRAG complements parameter-based fine-tuning with a target-domain retrieval gallery that provides instance-specific translation cues. To ensure that retrieved contexts are used appropriately, we further introduce Retrieval Utility-Guided Reinforcement Fine-Tuning (RUG-RFT), which combines translation-quality and retrieval-utility rewards to encourage beneficial retrieval use while suppressing harmful reliance. Experiments on multiple SLT benchmarks establish new state-of-the-art performance. In particular, to the best of our knowledge, SignRAG is the first gloss-free approach to outperform gloss-supervised methods across all reported metrics on CSL-Daily. Our code has been released at \href{https://github.com/shahelaojieraozhi/SignRAG}{GitHub}, together with models of different sizes to support future academic research.

1 INTRODUCTION

SignRAG addresses the challenge of applying decoder-only LLMs to gloss-free SLT with hierarchical pretraining, target-domain retrieval, and retrieval-aware reinforcement fine-tuning. It reports new state-of-the-art performance across four benchmarks and surpasses gloss-supervised methods on all CSL-Daily metrics.

1 INTRODUCTION

  • Strong LLM language branches can dominate optimization and leave the weaker visual pathway under-trained, motivating semantically grounded sign pretraining.The introduction frames this as Cross-Modal Optimization Imbalance and notes that strong language priors can induce shortcut learning.Source: §1 · pp. 1–2
  • Hierarchical pretraining first learns linguistically grounded sign representations, then jointly pretrains the sign encoder, multimodal projector, and LLM.The Pretrain→SFT variant alone already surpasses prior state-of-the-art methods under both gloss-free and gloss-based settings.Source: §1 · p. 2
  • A target-domain retrieval gallery complements parameter-based SFT with related sign instances and translations as instance-specific cues.It is built exclusively from the target-domain training split; retrieval reduces generated-to-ground-truth semantic distance from 0.234 to 0.206.Source: §1 · p. 2
  • RUG-RFT combines translation-quality and retrieval-utility rewards to encourage useful retrieval and suppress harmful reliance on misleading contexts.Retrieval dropout is also introduced during retrieval-augmented SFT to improve robustness when retrieved information is absent.Source: §1 · p. 2
  • SignRAG consistently outperforms gloss-free state-of-the-art methods on four benchmarks and surpasses gloss-supervised methods across all metrics on CSL-Daily.The authors describe this as the first gloss-free SLT approach to achieve the CSL-Daily result.Source: §1 · p. 2

2 RELATED WORK

Prior SLT work spans gloss-supervised pipelines and gloss-free visual-language approaches; SignRAG examines large-scale sign pretraining and retrieval-augmented reinforcement fine-tuning under an LLM decoder without gloss annotations.

2 RELATED WORK

  • Prior gloss-free SLT uses vision-language learning, LLM decoding, or cross-modal alignment, while retrieval-based SLT mainly targets gloss-based settings; SignRAG studies retrieval-augmented fine-tuning without gloss annotations.The passage also distinguishes encoder-only from end-to-end pretraining and notes the predominance of T5-based encoder–decoder models.Source: §2 · p. 3

3 METHOD

SignRAG organizes gloss-free SLT into hierarchical pretraining, target-domain retrieval-augmented SFT, and retrieval-aware reinforcement fine-tuning. These stages learn linguistically grounded sign representations, supply instance-specific translation cues, and regulate retrieval use.

3.1 HIERARCHICAL PRETRAINING

  • Stage I aligns skeletal sign representations with text; Stage II jointly trains the grounded sign encoder, projector, and LoRA-adapted LLM on large-scale sign–text corpora.Stage I combines bidirectional InfoNCE contrastive learning with skeleton-to-text translation.Source: §3.1 · pp. 3–4

3.2 RETRIEVAL-AUGMENTED SFT

  • For each target dataset, training-split sign features and paired translations form a gallery; frame-level matching retrieves fine-grained cues, while dropout trains with or without retrieval.The gallery is constructed from the training split, while validation and test samples are queries; frame-level correspondences are intended to preserve semantically relevant matches.Source: §3.2 · pp. 4–5

3.3 RETRIEVAL UTILITY-GUIDED REINFORCEMENT FINE-TUNING

  • RUG-RFT combines sentence-level translation quality and retrieval contribution rewards under GRPO, which estimates advantages from group reward statistics without a separate value model.It addresses the lack of explicit SFT supervision for whether retrieved contexts help generation.Source: §3.3 · p. 5

3.3.1 RETRIEVAL-UTILITY REWARD

  • Retrieval utility compares the same generated output under retrieval-augmented scoring and a frozen SFT retrieval-free reference; positive gain indicates increased confidence with retrieval.Using the frozen SFT checkpoint for retrieval-free likelihood prevents artificially lowering that likelihood to increase the reward.Source: §3.3.1 · p. 6

3.3.2 QUALITY REWARD AND REWARD COMPOSITION

  • The composed reward pairs a ground-truth-based sentence-level quality score with positive retrieval utility, balancing their magnitudes through a scaled utility term.The quality score may be BLEU or ROUGE; the scaling factor is adjusted using recent-batch exponential averages, with a stabilizer for early training.Source: §3.3.2 · p. 6

4 EXPERIMENT

SignRAG is evaluated on four SLT benchmarks using large-scale pretraining, target-domain retrieval, and reinforcement fine-tuning. The experiments report strong benchmark results and analyze the contributions of the components, retrieval quality, model scale, and retrieved-context examples.

4.1 EXPERIMENTAL SETUP

  • Evaluation covers CSL-Daily, PHOENIX-2014T, How2Sign, and OpenASL, with BLEU-1/2/3/4 and ROUGE-L reported.Retrieval galleries are built from each dataset’s training split; validation and test samples are queries only.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

  • SignRAG outperforms gloss-free state-of-the-art methods on all four benchmarks and surpasses gloss-supervised methods across metrics on CSL-Daily.PHOENIX-2014T results use no additional pretraining data, and gains on English datasets are more moderate; the authors note incomplete YouTube-ASL pretraining data may limit effectiveness.Source: Table 1 · p. 7, §4.2 · p. 8

4.1 EXPERIMENTAL SETUP

  • The setup uses Qwen3-8B, with two pretraining stages followed by SFT and RFT; retrieval uses K = 3.Checkpoints are selected by validation BLEU-4.Source: §4.1 · pp. 7–8

4.3 ABLATION STUDY

  • Pretraining, retrieval, and RUG-RFT cumulatively raise CSL-Daily scores to 62.34 ROUGE-L and 32.91 BLEU-4; combining quality and retrieval-utility rewards performs best.The component comparison begins with pretraining, adds retrieval, then RUG-RFT; the two reward types also improve performance independently.Source: Table 2 · p. 8, §4.3 · p. 8

4.4 ANALYSIS

  • Across retrieved-context counts, retrieval F1 and translation performance first improve and then decline as less-relevant contexts are added.BLEU-4 results in this analysis are from SFT models.Source: Figure 3 · p. 9, §4.4 · p. 9
  • BLEU-4 rises from 22.72 to 28.58 and 32.91 as the Qwen3 backbone scales from 0.6B to 4B and 8B under the same protocol.Source: Figure 3 · p. 9, §4.4 · p. 9
  • In the CSL-Daily case study, SignRAG’s retrieved contexts supply topical and lexical cues that preserve reference details Geo-Sign misses.The cited examples include “Traditionally” and “lucky money”; an appendix bad case shows noisy retrieval can mislead the model.Source: Table 3 · p. 9, §4.4 · p. 9

5 CONCLUSION

SignRAG combines staged sign pretraining, target-domain retrieval, and retrieval-aware reinforcement fine-tuning, achieving reported state-of-the-art results across SLT benchmarks. The supplied material also describes its related-work context, similarity objectives, distributed retrieval implementation, and research reproducibility resources.

5 CONCLUSION

  • SignRAG achieves new state-of-the-art performance across multiple SLT benchmarks and surpasses existing gloss-supervised methods across all reported metrics on CSL-Daily.Future work will explore more efficient and robust retrieval, plus knowledge transfer across datasets and languages.Source: §5 · p. 10

AI USE STATEMENT

  • The authors used generative AI for writing and language polishing, not for research or experimental design, data analysis, or result interpretation.They say all AI-assisted text was reviewed and revised by the authors, who take responsibility for the manuscript.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The authors plan to release source code, pretrained checkpoints from different stages, and models of different sizes to support verification and future research.The statement says these resources will be released upon acceptance.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 SIGN LANGUAGE TRANSLATION

  • Early SLT often relied on gloss supervision, while gloss-free SLT maps sign videos directly to spoken-language sentences; costly gloss annotations limit scalability.The passage notes that a performance gap from gloss-based methods remains in earlier gloss-free work.Source: §A.1 · p. 16

A.2 RL FOR RETRIEVAL-ENHANCED REASONING

  • Prior retrieval-based SLT mainly uses gloss guidance, while prior RAG reinforcement learning targets knowledge-intensive reasoning rather than sign-input translation.SignRAG instead combines a sign-to-text retrieval gallery with retrieval-utility-guided reinforcement fine-tuning.Source: §A.2 · p. 16

B.2 ATTENTION-WEIGHTED SKELETON–TEXT SIMILARITY

  • Skeleton and text features are projected into a shared D-dimensional space and normalized; row-wise softmax produces attention weights for global skeleton–text similarity.The similarity uses a trainable temperature, and text-to-skeleton similarity is computed analogously.Source: §B.2 · p. 17

B.3 BIDIRECTIONAL INFONCE LOSS

  • A bidirectional InfoNCE objective is applied to batch-level similarity matrices from paired samples, using temperature τ′.Source: §B.3 · p. 17

B.4 MEMORY-BOUNDED DISTRIBUTED RETRIEVAL

  • Distributed retrieval combines CPU-resident gallery shards, block-wise similarity computation, and exact distributed top-K aggregation.For fixed chunk sizes and sequence lengths, resident GPU storage is independent of total gallery size, but CPU storage grows with the local gallery and retrieval still evaluates every eligible pair.Source: §B.4 · pp. 17–18

C MORE IMPLEMENTATION DETAILS

The implementation details cover the computing setup, sign keypoints, staged pretraining, retrieval-augmented supervised fine-tuning, and retrieval-utility reinforcement fine-tuning. Training retrieval filters out the query and identical target translations to avoid a shortcut unavailable at inference.

C.1 COMPUTING INFRASTRUCTURE

  • Experiments used four NVIDIA RTX 4090D GPUs with 48GB VRAM, an AMD EPYC 7502 processor, Ubuntu 20.04, Python 3.10, CUDA 11.8, transformers 4.51.3, and PyTorch 2.1.2.Source: §C.1 · p. 19

C.2 KEYPOINT EXTRACTION.

  • Each frame retains 77 keypoints: 9 body joints, 21 joints per hand, 8 mouth landmarks, and 18 facial landmarks.The keypoints are extracted using the off-the-shelf MMPose estimator.Source: §C.2 · p. 19

C.3 PRETRAINING(STAGE I): SIGN-TEXT PRE-TRAINING.

  • Stage I trains a compact skeleton–text model for 200 epochs, using three-layer Transformer encoders and mBART-initialized text components.For CSL-News, the vocabulary is reduced to 11,540 tokens; visual and textual contrastive features have dimension 1024.Source: §C.3 · p. 20

C.4 PRETRAINING(STAGE II): LARGE-SCALE SIGN MLLM PRETRAINING.

  • Stage II uses Qwen3-8B and a GELU MLP projector to map 512-dimensional visual features into the LLM’s 4096-dimensional embedding space.LoRA is applied to attention query and value projections; training runs for 50 epochs with cross-entropy loss.Source: §C.4 · p. 20

C.5 SFT: RETRIEVAL-AUGMENTED SUPERVISED FINE-TUNING.

  • Downstream SFT switches to SLT datasets, combines retrieved texts with sign features, and trains for 10 epochs; Figure 4 distinguishes prompts with retrieved contexts from those without them.The SFT stage attaches new LoRA modules after merging the pretrained LoRA parameters into the base LLM.Source: §C.5 · p. 20, Figure 4 · p. 20
  • Retrieval ranks gallery samples by visual similarity, excludes the query itself and, during training retrieval, entries with identical reference translations, then returns up to K contexts.The gallery entries contain a sign sample, its translation, and sample ID.Source: §C.5 · p. 21, §C.7 · p. 21

C.6 RFT: RETRIEVAL UTILITY-GUIDED REINFORCEMENT FINE-TUNING.

  • RFT samples eight responses per input and computes group-relative advantages from sentence-level quality and retrieval-utility rewards.It runs for two epochs; very short target sentences receive final punctuation before reward computation to reduce BLEU-4 training-signal instability.Source: §C.6 · p. 21

C.7 IDENTICAL-TEXT FILTERING IN TRAINING RETRIEVAL.

  • Identical-text filtering prevents training retrieval from directly supplying a query’s target sentence, which would create a copying shortcut unavailable at dev/test inference.Only two CSL-Daily test samples share identical translations with the training set; the passage says this indicates gains are not driven by direct same-translation matches.Source: §C.7 · p. 21, Table 5 · p. 21

D.1 ANALYSIS OF DIFFERENT PRETRAINING STRATEGIES.

Hierarchical pretraining substantially outperforms direct end-to-end pretraining on CSL-Daily. The analysis also reports a gap between direct VideoLLaMA2 adaptation and SignRAG-SFT, and describes why staged sign representation learning matters when training with a strong LLM.

D.1 ANALYSIS OF DIFFERENT PRETRAINING STRATEGIES.

  • 29.72 BLEU-4 and 58.78 ROUGE-L are reported for hierarchical pretraining, versus 9.53 BLEU-4 and 38.70 ROUGE-L for direct end-to-end pretraining on CSL-Daily.The comparison evaluates downstream SFT; hierarchical pretraining first learns a language-aware sign encoder, then jointly trains the sign encoder, projector, and LLM.Source: Table 6 · p. 22, §D.1 · p. 22

D.2 DIRECTLY FINE-TUNING A VIDEO LLM.

  • Directly fine-tuning VideoLLaMA2 (7B) on sign–text pairs shows a substantial performance gap from the SignRAG-SFT model.The authors attribute the gap in part to general-purpose video encoders’ difficulty modeling fine-grained sign cues, including handshape, motion, location, and non-manual signals.Source: §D.2 · p. 22, Figure 5 · p. 23

D.3 WHY DOES HIERARCHICAL PRETRAINING BENEFIT SIGN MLLMS?

  • Directly coupling a sign encoder with a powerful LLM can perform worse than a T5-based architecture, while pretraining the sign encoder before joint training improves performance.The authors explain that mature LLM language priors can dominate when sign representations are weak; their staged approach first aligns sign semantics, then integrates the sign encoder with the projector and LLM. The analysis is based on a relatively limited experimental setting.Source: §D.2 · p. 22, §D.3 · pp. 23–24

D.4 DATASET STATISTICS.

The dataset statistics section reports the available YouTube-ASL subset used for pretraining, alongside the broader dataset statistics in Table 7.

D.4 DATASET STATISTICS.

  • 434,953 of the 530,161 reported YouTube-ASL samples were available to the authors, or 82.04% of the original dataset.Some source-video links were invalid, so the available subset was used for pretraining.Source: §D.4 · p. 24, Table 7 · p. 24

D.5 METRIC OVERFITTING ANALYSIS.

SignRAG-RFT is evaluated on chrF and BERTSim in addition to the BLEU- and ROUGE-based rewards it optimizes. It improves over SignRAG-SFT on all four reported metrics.

D.5 METRIC OVERFITTING ANALYSIS.

  • SignRAG-RFT improves over SignRAG-SFT on ROUGE-L, BLEU-4, chrF, and BERTSim.These complementary metrics suggest that the gains are not limited to direct BLEU/ROUGE-oriented optimization.Source: §D.5 · p. 24
  • The analysis adds chrF and BERTSim to assess performance beyond the sentence-level BLEU and ROUGE rewards optimized by RUG-RFT.BERTSim is a BERT-based semantic similarity score computed with bert-base-chinese.Source: §D.5 · p. 24
  • Table 8 reports ROUGE-L, BLEU-4, and BERTSim under the abbreviations R-L, B@4, and BSim, respectively.The table compares SignRAG-SFT (SFT) with SignRAG-RFT (RFT).Source: Table 8 · p. 24

D.6 TRANSFERABILITY ACROSS CHINESE SLT BENCHMARKS.

Pretrain→SFT transfers to CE-CSL, while a qualitative case illustrates RFT recovering from misleading retrieved context. The efficiency analysis reports online inference overhead relative to a hardware-controlled baseline.

D.6 TRANSFERABILITY ACROSS CHINESE SLT BENCHMARKS.

  • 24.44 BLEU-4 and 54.55 ROUGE-L surpass the strongest CE-CSL baselines by 2.37 and 2.54 points, respectively.Pretrain→SFT outperforms Uni-Sign and Geo-Sign across all reported metrics.Source: §D.6 · p. 25, Table 9 · p. 25

D.7 FAILURE CASE (HARMFUL RELIANCE ON RETRIEVED CONTEXTS).

  • In the failure case, Pretrain→SFT translates correctly, SignRAG-SFT follows an incorrect retrieved context, and SignRAG-RFT recovers the correct translation.Table 10 highlights misleading or incorrectly generated content in red.Source: §D.7 · p. 25, Table 10 · p. 25

E EFFICIENCY, COMPUTATIONAL COST, AND RETRIEVAL SCALABILITY

  • Efficiency comparisons use Qwen3-8B on four RTX 4090D GPUs and control hardware, batch size, precision, and decoding strategy against Pretrain→SFT.The analysis focuses on SignRAG's incremental cost relative to this hardware-controlled baseline.Source: E EFFICIENCY, COMPUTATIONAL COST, AND… · p. 25

E.1 ONLINE SLT INFERENCE EFFICIENCY

  • Against Pretrain→SFT, SignRAG-SFT raises CSL-Daily BLEU-4 from 29.72 to 31.22, while adding 19.29 ms/sample and 5.75 GiB/GPU.Throughput falls from 23.42 to 16.13 samples/s; gallery search is performed offline and excluded from deployed generation.Source: §E.1 · pp. 25–26, Table 11 · p. 26

E.2 OFFLINE RETRIEVAL EFFICIENCY ON CSL-DAILY

Offline retrieval cost rises with gallery size on CSL-Daily and OpenASL. The analysis also reports full-test-set retrieval and combined retrieval-and-generation time on CSL-Daily.

E.2 OFFLINE RETRIEVAL EFFICIENCY ON CSL-DAILY

  • As the CSL-Daily gallery grows from 1K to 18.4K samples, average retrieval latency rises from 3.49 to 23.88 ms/query.Retrieval throughput falls from 286.67 to 42.02 queries/s, while the authors describe the cost as practical at CSL-Daily scale.Source: §E.2 · p. 26, Table 12 · p. 26
  • For 1,176 CSL-Daily test queries against 18,401 training samples, full-gallery retrieval takes 64.70 s; combined retrieval and generation takes 137.59 s.The combined evaluation corresponds to approximately 117.00 ms/sample and 0.153 GPU-hours across four GPUs; precomputed retrieval leaves deployed online latency at 61.98 ms/sample.Source: §E.2 · p. 26

E.3 RETRIEVAL SCALABILITY ON OPENASL

  • On OpenASL's 95,888-sample gallery, retrieval takes 705.85 ms/query for 256 queries and 715.90 s for all 971 test queries.The full-gallery test-set rate is 737.29 ms/query; the authors report feasibility at approximately 100K samples while noting that cost grows substantially with gallery size.Source: §E.3 · p. 27, Table 13 · p. 27

E.4 TRAINING COST

The complete SignRAG training pipeline is reported at 413.99 GPU-hours. The discussion describes million-scale retrieval as unevaluated and flags an accuracy–efficiency trade-off for coarse-to-fine retrieval.

E.4 TRAINING COST

  • The complete SignRAG pipeline requires 413.99 GPU-hours, including 106.67 for Stage I pretraining, 277.50 for Stage II, 24.04 for SFT, and 5.78 for RFT.The reported setup uses CSL-News for pretraining and CSL-Daily for downstream fine-tuning.Source: §E.4 · p. 27, Table 14 · p. 27

E.5 DISCUSSION ON MILLION-SCALE RETRIEVAL

  • Retrieval is demonstrated on galleries of approximately 18K and 96K samples, but not on BOBSL's approximately 1.2 million sentences.The authors state that BOBSL is unavailable to them and avoid extrapolating without direct experimental evidence.Source: §E.5 · p. 27
  • Coarse-to-fine retrieval could shortlist candidates before token-level matching, but a global representation may discard samples highly scored by full matching.The authors do not claim that this strategy preserves exhaustive-retrieval accuracy; evaluation should cover latency, Top-K preservation, retrieval quality, and downstream SLT across candidate-pool sizes.Source: §E.5 · pp. 27–28

F MORE CASES

  • The additional CSL-Daily examples compare reference translations, top-3 retrieved texts, and Geo-Sign and SignRAG outputs.The cases provide a broader qualitative view of translation performance across examples.Source: F MORE CASES · p. 28, Table 15 · p. 28
Loading 2610.11371v1…