PaperlayerView exact revision on arXiv

Source-linked AI summary

SignRR: Retrieve and Refine Real Motion for Sign Language Production

Fidel Omar Tito Cruz, Angie Sanchez Marquina, Summy Farfan, Gissella Bejarano

arXiv:2608.28568v1cs.CV

TL;DR

Sign language production must preserve both realistic articulation and coherence across continuous signing, challenges that generative and retrieval-only methods address incompletely. SignRR retrieves real sign segments and refines their full assembly with a part-aware Residual VQ-VAE. On PHOENIX14T and CSL-Daily, it achieves state-of-the-art back-translation performance among non-GT methods while maintaining competitive pose quality.

  • Problem

    Generative methods lack observed signing references, while retrieval-based assembly can introduce sequence-wide rhythm and style inconsistencies.

  • Method

    SignRR retrieves motion from a gloss-motion dictionary and refines the full assembled sequence with a part-aware Residual VQ-VAE.

  • Results

    SignRR achieves state-of-the-art back-translation performance among non-GT methods on PHOENIX14T and CSL-Daily while maintaining competitive pose quality.

  • Takeaways & Limitations

    Retrieve-and-refine combines realistic articulation from real segments with learned refinement of sequence coherence.

  • Takeaways & Limitations

    SignRR does not capture fine-grained non-manual cues such as mouthings or eye movements and depends on the sign inventory in its dictionary.

Abstract

from arXiv · show

Sign language production (SLP) aims to generate continuous signing motion from spoken language, often through gloss-to-pose generation. Prior work mainly follows two paradigms. Generative models synthesize motion from a learned prior or from noise, without reference to an observed signing instance, making rare hand configurations and signer-specific articulation difficult to preserve. Retrieval-based methods reuse real, well-articulated motion segments, but concatenating segments from different signers and co-articulation contexts can introduce rhythm and style inconsistencies across the full sequence, not only at segment boundaries. These limitations suggest a complementary solution: use retrieval to provide realistic articulation, and use learned refinement to impose the global coherence that retrieval alone lacks. We therefore propose retrieve-and-refine, a paradigm that starts from real retrieved motion and refines it into a globally coherent signing sequence rather than generating motion from scratch. Our framework, SignRR, initializes motion from a dictionary of real sign segments and refines the full sequence with a part-aware Residual VQ-VAE, where residual quantization preserves fine hand articulation and temporal length differences are handled in the latent space. Experiments on PHOENIX14T and CSL-Daily show that SignRR achieves state-of-the-art back-translation performance while maintaining competitive pose quality.

1 Introduction

Sign language production commonly uses gloss-to-pose generation, but generative and retrieval-based approaches each struggle to preserve realistic articulation and global sequence coherence. SignRR addresses this by retrieving real motion and refining the assembled sequence with a part-aware Residual VQ-VAE.

  • Motivation: Gloss-to-pose generation maps intermediate glosses to continuous signing pose sequences, reducing cross-lingual alignment difficulty.Glosses preserve signing order and make pose generation more tractable than directly mapping spoken words to signs.
  • Existing paradigms: Generative methods can produce temporally averaged, under-articulated motion, particularly in the hands, because they synthesize from learned priors or noise without observed signing references.Hands are among the most informative cues in sign language, making this limitation especially relevant.
  • Existing paradigms: Retrieved segments provide realistic articulation, but differences in signer and speed can create rhythm mismatches across the full assembled sequence, not only at boundaries.This motivates refinement that reasons over the complete sequence rather than only reconstructing junctions.
  • SignRR: SignRR introduces retrieve-and-refine, combining real retrieved motion with learned refinement to produce coherent signing rather than generating motion from scratch.The paradigm uses retrieval for natural articulation and refinement for sequence coherence.
  • SignRR: SignRR initializes motion from a gloss-motion dictionary and refines the full assembled sequence with a part-aware Residual VQ-VAE.The framework is designed to preserve articulation while resolving inconsistencies introduced during assembly.
  • Results: SignRR achieves state-of-the-art results on PHOENIX14T and CSL-Daily across standard sign-language-production metrics, with ablations validating its components.The reported evaluation covers both benchmark datasets and includes component ablations.

2 Related Work

Prior sign-language-production research spans generative, retrieval-based, and discrete motion-representation approaches. SignRR combines real-segment retrieval with full-sequence refinement to address sequence-level inconsistencies that local stitching cannot resolve.

  • Sign Language Production: Earlier sign-language-production systems progressed from manually specified avatar lexicons to data-driven gloss-to-pose regression and discrete motion generation.Regression methods often produce temporally averaged poses and under-articulated hands, motivating vector-quantized alternatives.
  • Retrieval-based Sign Production: Retrieval-based methods assemble real segments from gloss-motion dictionaries, but prior approaches primarily optimize or refine individual junctions.This leaves inconsistencies across the assembled sequence insufficiently addressed.
  • Retrieval-based Sign Production: SignRR treats retrieval as initialization and refines the complete assembled motion to target sequence-level rhythm and articulation inconsistencies.The approach specifically targets problems that local stitching cannot resolve.
  • Motion Representations for SLP: Residual vector quantization uses cascaded codebooks to encode successive residuals, enabling progressive fidelity without codebook explosion.Prior motion-generation work also indicates benefits from multiple residual layers and separate body-part representations.
  • Motion Representations for SLP: SignRR applies part-aware residual quantization to retrieved sequences, unlike prior sign-production systems that generally use single-layer per-part quantization for generation from a clean learned prior.The paper positions this design for assembled motion from different signers and co-articulation contexts.
  • Pose-Conditioned Human Generation: Pose-conditioned human-generation studies use pose to control appearance synthesis, whereas SignRR focuses on producing the pose sequence itself.The resulting pose representation can support later human-appearance generation.

3 Method

SignRR retrieves real motion segments, assembles them into an initial sequence, and refines the complete sequence with a part-aware Residual VQ-VAE. Its method combines curated gloss-motion retrieval, part-specific residual quantization, latent length alignment, and full-sequence decoding.

  • Gloss-Motion Dictionary: SignRR builds a gloss-motion dictionary from continuous training sequences, preserving co-articulated transitions rather than isolated sign realizations.A pretrained CorrNet+ recognizer supplies ground-truth-constrained boundaries; candidates are filtered by motion smoothness and duration regularity, retaining S diverse representatives per gloss.
  • Part-aware Residual VQ-VAE: The refinement model encodes body, right-hand, and left-hand streams separately, reflecting the importance of preserving fine handshape, location, and movement patterns.Part-specific encoders use distinct temporal modeling choices, with wider dilation for body motion and larger latent dimensions and codebooks for hand streams.
  • Part-aware Residual VQ-VAE: Residual vector quantization uses cascaded codebooks to encode successive residuals, helping preserve fine pose details before the part-wise latents are combined.The quantized outputs from all residual layers are summed for each part and concatenated into the full latent representation.
  • Latent Alignment: A gloss encoder predicts the assembled-to-target length difference, and the quantized latent is linearly resampled to the target temporal resolution with aligned endpoints.At inference, the predicted target length replaces the unknown ground-truth length; the decoder then produces the final pose sequence.
  • Training Objective: Training concatenates one retrieved segment per gloss and uses the corresponding continuous sequence as supervision, with reconstruction, velocity, commitment, and length-prediction losses.The reconstruction loss weights hands above the body, while the length term updates only the gloss-encoder and predictor branch.
  • Inference: Assembly and Refinement: At inference, dynamic programming selects one candidate segment per gloss by minimizing boundary-smoothness and segment-quality costs.The selected path is assembled into the initial motion sequence before refinement.

4 Experiments and Results

SignRR is evaluated on PHOENIX14T and CSL-Daily against prior methods, with experiments covering benchmark performance, qualitative quality, ablations, robustness, and efficiency. It achieves the strongest back-translation results among non-GT methods while balancing pose accuracy, and refinement is the primary source of improvement.

  • Benchmark comparison: SignRR achieves the best back-translation performance among non-GT methods on PHOENIX14T and CSL-Daily.On PHOENIX14T, it improves over G2P-DDM across WER, BLEU-1, BLEU-4, and ROUGE; on CSL-Daily, it leads all reported back-translation metrics.
  • Benchmark comparison: On PHOENIX14T test, SignRR reaches WER 80.98, BLEU-1 28.78, BLEU-4 10.96, and ROUGE 28.15.It improves test BLEU-4 from 6.67 to 10.96 over the retrieval-based Sign Stitching baseline and outperforms Sign-IDD on all back-translation metrics.
  • Benchmark comparison: SignRR ranks second on MPJPE while achieving the strongest back-translation scores, indicating a balance between pose accuracy and intelligibility.The same pattern appears on both PHOENIX14T and CSL-Daily: the lowest MPJPE does not correspond to the best back-translation performance.
  • Benchmark comparison: On CSL-Daily test, SignRR reaches BLEU-1 15.45, BLEU-4 1.57, and ROUGE 15.46, leading non-GT methods on every back-translation metric.The dataset’s ground-truth back-translation ceiling is BLEU-4 5.06, limiting the absolute scale of this metric.
  • Ablation studies: The part-aware encoder and three residual codebooks provide the strongest overall back-translation trade-off in the ablations.Removing separate body and hand streams lowers BLEU-1 from 28.78 to 27.89 and ROUGE from 28.15 to 26.94; Nq=3 gives the strongest BLEU-1, CHRF, and ROUGE.
  • Ablation studies: Using ground-truth length improves MPJPE from 53.70 to 47.61 and FID from 1.21 to 1.16 but lowers BLEU-1 from 28.78 to 27.07.The results show that better frame-level alignment to the reference does not necessarily improve intelligibility under back-translation.
  • Ablation studies: Refinement drives most of the improvement: with dynamic-programming selection, it raises BLEU-1 from 22.06 to 28.78 and reduces MPJPE from 68.25 to 53.70.Refinement also reduces sensitivity to retrieval seeds, while dynamic programming adds smaller gains when refinement is enabled.
  • Analysis: Real retrieved motion outperforms synthetic periodic motion even with decoder cross-attention, reaching BLEU-1 28.78 versus 25.06.Cross-attention substantially improves synthetic motion, but does not consistently benefit real retrieved motion.

5 Limitations

SignRR has several scope and evaluation limitations, including incomplete modeling of facial cues, dependence on dictionary coverage, and indirect quality assessment.

  • SignRR models body and hand articulation but omits fine-grained facial cues such as mouthings and eye movements.These cues can carry grammatical and lexical information in signed languages.
  • Unseen glosses are mapped to the closest dictionary entry by edit distance, limiting the current system's open-vocabulary coverage.Future alternatives include external lexical resources or a generative fallback.
  • Evaluation uses back-translation recognition performance as an indirect proxy rather than direct judgments from fluent signers.Signer evaluation could assess perceptual naturalness, articulation quality, and linguistic intelligibility more directly.

6 Conclusion

SignRR retrieves real sign segments and refines their assembled motion with a part-aware Residual VQ-VAE to improve coherence while preserving realistic articulation. It achieves state-of-the-art back-translation performance among non-GT methods on two benchmarks, with competitive pose quality.

  • SignRR retrieves dictionary segments, assembles them, and refines the full sequence with a part-aware Residual VQ-VAE.Retrieval preserves realistic articulation, while refinement improves sequence coherence.
  • SignRR achieves state-of-the-art back-translation performance among non-GT methods on PHOENIX14T and CSL-Daily.The framework maintains competitive pose quality on both datasets.
  • Ablations show that full-sequence refinement is important and improves the quality of retrieved assemblies.

A Keypoint Representation

The appendix describes a 61-keypoint 3D representation and a dictionary-based retrieval pipeline that filters, scores, and dynamically selects real motion segments before refinement.

  • A Keypoint Representation: Each frame uses 61 3D keypoints, giving D = 61×3 = 183 pose values.The representation includes body, right-hand, and left-hand keypoints.
  • B Gloss-Motion Dictionary Construction: The gloss-motion dictionary stores multiple real segments per gloss, extracted using CorrNet+ boundaries and the known gloss order.The alignment requires sentence-level gloss annotations rather than frame-level boundaries.
  • B Gloss-Motion Dictionary Construction: Candidate filtering discards segments with durations too far from each gloss's typical duration, reducing poorly aligned examples.
  • B Gloss-Motion Dictionary Construction: Remaining candidates are scored by motion smoothness and duration regularity, and the top S segments are retained for each gloss.Smoothness is measured using average frame-to-frame acceleration; lower values indicate smoother motion.
  • C Dynamic Programming Segment Selection: Dynamic programming jointly balances segment quality and boundary continuity when selecting one candidate per gloss.Boundary descriptors compare endpoint poses and local velocities, while the algorithm runs in O(NS^2) time.
  • C Dynamic Programming Segment Selection: Out-of-dictionary glosses are replaced with the closest dictionary entry under Levenshtein edit distance.PHOENIX14T has 19 development and 22 test unseen gloss types; CSL-Daily has none.
  • C Dynamic Programming Segment Selection: The selected segments are concatenated in order and passed to the RVQ-VAE as the initial motion for refinement.

D Implementation Details

The implementation uses dynamic programming for retrieval and a part-aware residual-quantized refinement model with separate body and hand streams, plus gloss-conditioned length prediction.

  • Dynamic programming segment selection: Dynamic programming resolves out-of-dictionary glosses, computes trellis costs, stores minimizing predecessors, and backtracks the lowest-cost path.
  • Refinement model: The refinement model uses separate body, right-hand, and left-hand encoders with three residual codebooks per part and a shared decoder.The decoder outputs D = 183 pose dimensions.
  • Gloss encoder and length prediction: A two-layer Transformer gloss encoder predicts a length offset, with target lengths clamped to [16,Tmax] at inference.Tmax is 300 for PHOENIX14T and 352 for CSL-Daily.
  • Training objective: Training uses λv = 0.1, λc = 0.02, and λℓ = 0.01, with hand reconstruction weights of 2.0 versus 1.0 for the body.
  • Dictionary configuration: The implementation retains at most S = 25 representative segments per gloss and builds dictionaries containing 9,311 PHOENIX14T and 32,500 CSL-Daily segments.Average candidate counts are 8.61 and 16.26, respectively, though the passage truncates the final unit.

E Synthetic Motion Generation

The appendix constructs semantically neutral synthetic motion from simple trajectories applied to selected hand keypoints, then smooths and filters the resulting sequences. This setup tests whether real retrieved motion is necessary for effective refinement.

  • Trajectory construction: Synthetic sequences add linear, sinusoidal, or quadratic displacement trajectories to a randomly selected reference frame.The trajectories use a uniformly sampled step size and produce 2D displacement vectors over the sequence length.
  • Trajectory construction: The displacements are applied to 46 hand keypoints covering both wrists, palms, and finger joints.The selected right- and left-hand keypoints carry most of the movement during signing.
  • Temporal assembly: Adjacent synthetic segments receive 10 interpolated boundary frames before concatenation, avoiding duplicated boundary frames and preserving smooth transitions.The resulting sequences have lengths comparable to the originals, with variation of ±20 frames per video.
  • Filtering and resampling: Frames violating hand-height or arm-length constraints are removed as physically implausible, and valid frames are resampled to the original sequence length.The hand-height ceiling uses keypoint 9 plus an offset of 0.6 units, while elbow positions must remain within 20% of torso length.
  • Purpose: The synthetic sequences are smooth and semantically neutral, lacking sign-specific articulation and meaningful gestures.This makes them suitable for testing whether real retrieved motion contributes to effective refinement.

F Robustness to Dictionary Segmentation

SignRR remains relatively stable when gloss boundaries used to build its dictionary are perturbed, although performance gradually decreases with larger offsets. Uniform segmentation also provides a recognizer-free baseline with measurable performance.

  • Boundary perturbation: At ±16-frame boundary variation, BLEU-4 decreases by only 0.42.Perturbed boundaries preserve gloss order, require at least three frames per segment, and keep the sequence endpoints fixed.
  • Boundary perturbation: Performance decreases gradually as dictionary boundary variation increases.Each perturbation setting rebuilds the dictionary and retrains SignRR using the corresponding boundaries.
  • Recognizer-free baseline: Uniform segmentation achieves BLEU-4 10.58 without CorrNet+ boundary predictions.A sequence of length L containing N glosses is divided uniformly into N segments before dictionary construction and training.

G Inference Time

SignRR inference combines CPU dynamic-programming segment selection with one GPU refinement pass. On PHOENIX14T, it requires 87.8 ms per sequence and is substantially faster than G2P-DDM.

  • Component timing: SignRR requires 87.8 ms per sequence, comprising 54.6 ms for segment selection and 33.1 ms for refinement.Measurements use batch size 1 on a single NVIDIA A100 GPU, with dynamic-programming selection running on the CPU.
  • Model comparison: SignRR has inference time comparable to Sign-IDD and is about 25× faster than G2P-DDM.The comparison uses the released implementations of the baselines with 10 DDIM and 100 diffusion steps, respectively.
Loading 2608.28568v1…