LLMs are Good Sign Language Translators

• SLT requires understanding visual and linguistic cues, while limited paired sign-text data exacerbates the challenge and leaves effective cross-modal solutions an open question.The introduction notes that this remains open despite advances in architectures, sign representations, and training methods.Source: §1 · p. 1
• The authors keep the off-the-shelf LLM frozen to preserve pretrained multilingual knowledge, since training it directly on small SLT datasets could risk forgetting that knowledge.The paper asks whether sign videos can be treated as a form of language and translated by a frozen LLM.Source: §1 · p. 1
• The framework regularizes sign videos toward discrete tokens and a character-to-word hierarchy, hypothesizing that language-like representations improve LLM understanding.This design is motivated by the discrete vocabulary and hierarchical structure of spoken languages.Source: §1 · p. 2
• VQ-Sign converts videos into discrete character-level tokens, CRA maps them to word-level tokens, and sign-text alignment further narrows the gap to text tokens.The resulting language-like sign sentences are fed to a frozen LLM with a prompt for the desired translation language.Source: §1 · p. 2
• SignLLM achieves state-of-the-art gloss-free results on two popular SLT datasets while harnessing off-the-shelf, frozen LLMs.The authors describe it as a first step toward effectively using LLMs for SLT.Source: §1 · p. 2

• Earlier SLT work improves architectures and visual representations, adds pretraining or joint tasks, and includes gloss-free approaches that avoid gloss annotations during training.Related work also notes datasets such as How2Sign and BOBSL with large sign and text vocabularies.Source: §2 · p. 2
• LLMs trained on large web-scale text corpora show generalization across text tasks, including multilingual translation.Source: §2 · p. 2
• SignLLM is presented as the first approach to use an off-the-shelf frozen LLM for SLT by converting videos into language-like sign tokens.Its overview depicts character-level tokens reorganized into word-level tokens before a prompted LLM generates the translation.Source: §2 · p. 3, Figure 1 · p. 3

3.1. SignLLM Overview

• SignLLM turns video X into language-like sign sentence W by composing discrete character-level tokens into word-level tokens, aligning them to text, and feeding W to a prompted frozen LLM.Source: §3.1 · pp. 3–4

3.2. Vector-Quantized Visual Sign Module

• VQ-Sign quantizes sign-video features into character-level tokens and learns its discrete codebook through context prediction rather than reconstructing high-dimensional video clips.The tokens correspond to short clips and may have little meaning individually, while sequences can convey clearer meaning.Source: §3.2 · pp. 4–5

3.3. Codebook Reconstruction and Alignment

• CRA reduces repeated-token redundancy while preserving a speed cue, then uses optimal transport to form a compact, low-entropy word-level codebook from character tokens.When repetitions exceed the sequence average, preprocessing inserts a token as a “slowing down” sign; codebook size balances entropy reduction against increased size.Source: §3.3 · pp. 5–6
• Sign-text alignment minimizes MMD between sign and text embedding spaces, applying the loss to both character-level and word-level tokens to narrow their distribution gap.Source: §3.3 · pp. 6–7

3.4. Training and Inference

• Training first pretrains VQ-Sign with context prediction and constructs and aligns codebooks, then fine-tunes with the LLM frozen using VQ, MMD, and text-similarity losses.Inference transforms video features into character- and word-level tokens, projects the sign sentence to LLM embedding space, and prompts the LLM for translation.Source: §3.4 · p. 7

4.1. Implementation Details

• The implementation uses frozen LLaMA-7B-16bit and a character-level codebook of 256 vectors, each 1024-dimensional.SignLLM is fine-tuned for 20 epochs.Source: §4.1 · p. 7

4.2. Datasets and Evaluation Metrics

• Evaluation uses Phoenix-2014T and CSL-Daily dev and test sets, with BLEU-1 to BLEU-4 and ROUGE-L metrics.The datasets are German and Chinese sign-language benchmarks, respectively.Source: §4.2 · p. 7

4.3. Main Results

• SignLLM improves every reported metric over prior gloss-free methods on both Phoenix-2014T and CSL-Daily.The comparisons are presented in Tables 1 and 2.Source: §4.3 · p. 7, Table 1 · p. 8, Table 2 · p. 8

4.4. Ablation Study

• The LLaMA-based approach performs better than the no-LLM baseline, while using the smaller T5 model leads to worse performance.The no-LLM baseline replaces the LLM with a trainable lightweight mBART text generator.Source: §4.4 · p. 7, Table 3 · p. 8
• Component ablations include feeding character-level tokens directly to the LLM and omitting the sign-text alignment loss.Table 5 covers ablations of SignLLM’s main components.Source: §4.4 · p. 8, Table 5 · p. 8
• In two qualitative examples, SignLLM preserves sentence semantics more accurately than GFSLT-VLP, whose second output adds redundant, erroneous content.Figure 2 marks correct translations in blue and incorrect translations in red.Source: §4.4 · p. 8, Figure 2 · p. 8

• SignLLM achieves state-of-the-art gloss-free results on two popular SLT benchmarks.Source: §5 · p. 8
• VQ-Sign and CRA impart language-like characteristics to sign-video representations, while sign-text alignment improves semantic compatibility.Source: §5 · p. 8
• The framework harnesses off-the-shelf, frozen LLMs for sign language translation.Source: §5 · p. 8