PaperlayerView exact revision on arXiv

Source-linked AI summary

SeRV: Semantic-Aligned Residual Vector Quantization for American Sign Language Generation

Hongyu Wu, Xu Wu, Tianhao Wu, Jiawei Yu, Phuc Nguyen, Jian Liu, Yi Wu

arXiv:2609.05742v1cs.CVcs.AI

TL;DR

Text-to-3D ASL generation faces limited paired motion data and tokenizers that lack semantic alignment with text. SeRV combines semantic-aligned residual quantization, hierarchical generation, and a reconstructed YouTube-ASL benchmark; it reports state-of-the-art pose accuracy on How2Sign and YouTube-ASL.

  • Problem

    Text-to-3D ASL generation is limited by scarce paired motion data and reconstruction-focused tokenizers that do not align tokens with text meaning.

    Source: §1 · p. 1

  • Method

    SeRV combines sentence-level and token-level motion–text alignment in a residual tokenizer with a hierarchy-aware generator and a reconstructed YouTube-ASL motion-text benchmark.

    Source: §1 · pp. 1–2

  • Results

    SeRV achieves state-of-the-art pose accuracy on How2Sign and YouTube-ASL and the best result on five of six reported metrics across the two datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support generating semantically consistent, fine-grained 3D ASL motion directly from text with text-predictable residual tokens.

    Source: §1 · p. 1, §5 · p. 8

    The autoregressive generator predicts tokens sequentially, limiting inference parallelism and making long-sequence generation inefficient.

    Source: §6 · pp. 8–9

Abstract

from arXiv · show

American Sign Language (ASL) generation remains challenging due to limited paired text-ASL motion data and the difficulty of learning motion representations both precise for reconstruction and predictable from linguistic input. Existing methods rely on motion tokenizers optimized for reconstruction, without explicit semantic supervision from paired text. As a result, the learned tokens remain limited in supporting semantically consistent and fine-grained ASL motion generation. To address this limitation, we propose SeRV (Semantic-Aligned Residual Vector Quantization), a semantic-aligned RVQ tokenizer for ASL generation. SeRV learns a semantically structured residual token space by combining sentence-level motion-text alignment with token-level text-conditioned supervision. Building on this tokenizer, a Hierarchical GPT predicts residual motion tokens in a coarse-to-fine manner, generating structurally coherent and semantically aligned 3D ASL motion. We further construct a large-scale reconstructed 3D ASL motion-text benchmark by recovering paired 3D motion from YouTube-ASL videos. Experiments across 375 hours of ASL video show that SeRV achieves state-of-the-art pose accuracy on both How2Sign and YouTube-ASL datasets, while producing semantically consistent 3D ASL motion directly from text.

1 Introduction

SeRV addresses semantic and data limitations in text-to-3D ASL generation with a text-aligned residual tokenizer, a hierarchy-aware generator, and a reconstructed YouTube-ASL benchmark. It reports state-of-the-art pose accuracy on both benchmarks and the best result on five of six reported metrics.

1 Introduction

  • Existing reconstruction-only tokenizers lack alignment to text meaning, while limited 3D ASL resources constrain scale, vocabulary, and motion diversity.The paper notes that fine-grained hand differences distinguish signs, but distinct meanings can map to nearly identical tokens.Source: §1 · p. 1
  • SeRV combines sentence-level and token-level motion–text alignment to make residual tokens precise for reconstruction and predictable from text.Sentence-level alignment supervises the coarsest tokens, while token-level alignment supervises the fully quantized latent.Source: §1 · pp. 1–2
  • SeRV achieves the best result on five of six reported metrics across How2Sign and YouTube-ASL, with state-of-the-art pose accuracy on both benchmarks.The six metrics include body and hand DTW-JPE, where lower is better, and SiBLEU-4, where higher is better.Source: Figure 1 · p. 2, §1 · p. 2
  • Hierarchical GPT predicts residual tokens progressively using per-token text features and previously decoded lower-level motion tokens.The design follows SeRV’s coarse-to-fine hierarchy to produce structurally coherent, semantically consistent motion.Source: §1 · p. 2
  • The authors construct a large-scale reconstructed 3D ASL motion-text benchmark from YouTube-ASL videos, with quality-based cleaning and filtering.They describe it as the largest reconstructed 3D ASL motion-text benchmark to date.Source: §1 · p. 2
  • MaDiS could not be evaluated on the larger YouTube-ASL benchmark under the same protocol because its code and checkpoints were unavailable.The paper identifies MaDiS as concurrent work.Source: §1 · p. 2

2 Related Work

Prior sign-generation work spans intermediate-representation pipelines, direct continuous generation, and discrete motion-token methods. SeRV addresses limitations in existing resources and tokenization with semantic-aligned residual quantization.

2 Related Work

  • Sign-generation methods include cascaded pipelines, direct continuous pose or video generation, and discrete-token approaches for sign motion.The surveyed discrete approaches include pose-level and video tokenizers, as well as methods for 3D motion and sign-video generation.Source: §2 · pp. 2–3
  • How2Sign provides 80+ hours of in-studio ASL, while YouTube-ASL contains 984 hours of in-the-wild video from 2,500+ signers.The datasets differ in recording setting and visual diversity.Source: §2 · p. 3
  • SeRV and a reconstructed 3D ASL motion–text benchmark target limitations in existing resources and seek residual tokens that are precise for reconstruction and predictable from text.The benchmark is constructed by recovering 3D sign motion from YouTube-ASL videos and curating it at scale.Source: §2 · p. 3

3 Methodology

The methodology turns 3D ASL motion into residual discrete tokens, adds sentence- and token-level text alignment, and trains a coarse-to-fine generator to predict those tokens from text.

3.1 Preliminaries of RVQ-VAE

  • Residual vector quantization repeatedly quantizes each latent residual with a codebook, then sums the quantized levels for decoding into motion.Each selected code index is a discrete token, and the RVQ-VAE objective includes motion reconstruction and a per-level embedding constraint.Source: §3.1 · p. 3

3.2 SeRV Model Architecture

  • SeRV adds sentence-level alignment of level-0 motion with transcript semantics and token-level late interaction between text tokens and fully quantized motion latents.Each valid token is matched to its closest token in the other modality; both alignment losses accompany the standard RVQ objective.Source: §3.2 · pp. 3–4

3.3 Hierarchical GPT

  • Hierarchical GPT predicts SeRV tokens autoregressively over time and RVQ levels, generating coarse tokens before residual refinements conditioned on text and previously generated tokens.The shared motion decoder converts the predicted multi-level tokens back into an ASL motion sequence.Source: Figure 3 · p. 5, §3.3 · p. 5

3.2 SeRV Model Architecture

  • SeRV training combines the standard RVQ tokenizer objective with global and token-level semantic alignment losses.The global loss aligns coarse level-0 motion with sentence-level text, while the token-level loss encourages fine-grained interaction between text tokens and quantized motion latents.Source: Figure 2 · p. 4, §3.2 · p. 5

4 Performance Evaluation

Experiments on How2Sign and reconstructed YouTube-ASL compare SeRV with prior generators, assess reconstruction and qualitative motion, and test semantic alignment and hierarchical generation. SeRV improves benchmark pose accuracy, while the ablations support both semantic supervision and preservation of the residual hierarchy.

4.1 Experimental Methodology

  • Evaluation uses How2Sign annotations and filtered reconstructed YouTube-ASL SMPL-X motion, with DTW-JPE for sequence-level errors and SiBLEU-4 for token-level consistency.Hand Avg. averages left- and right-hand errors; tokenizer reconstruction instead uses PA-MPJPE on frame-aligned motion.Source: §4.1 · pp. 5–6

4.2 Comparison with State-of-the-Art Methods

  • On YouTube-ASL, SeRV lowers SOKE’s body and hand DTW-JPE by 8.1% and 9.9%, and raises SiBLEU-4 from 2.18 to 3.13; on How2Sign, it has the lowest body and hand errors.MaDiS reports higher How2Sign SiBLEU-4 than SeRV (4.47 vs. 4.12), but is unavailable for evaluation on YouTube-ASL; general motion generators adapted to sign perform substantially worse than sign-specific methods.Source: Table 1 · p. 6, §4.2 · p. 6

4.3 Qualitative Comparison

  • In qualitative examples, SeRV produces more accurate fine-grained handshapes and more plausible upper-body posture with fewer self-contact artifacts than SOKE.Examples include six sentences, with three each from How2Sign and YouTube-ASL.Source: Figure 4 · p. 7, §4.3 · p. 7

4.4 Ablation Studies

  • With batch size 64, SeRV improves Recall@1, Recall@5, and Recall@10 in both text-to-sign and sign-to-text retrieval on both datasets.Text inputs use frozen T5 global embeddings and sign motions use pooled latents from the frozen tokenizer encoder.Source: Table 3 · p. 8, §4.4 · p. 8
  • Across both datasets, combining sentence-level and token-level alignment performs best across metrics, with semantic supervision improving generation accuracy and SiBLEU-4.The gains are larger on hand motion, and all variants use the same generator.Source: Table 4 · p. 8, §4.4 · p. 8
  • On both datasets, Hierarchical GPT outperforms flat autoregressive and two-stage baselines on body and hand motion by predicting residual tokens coarse-to-fine.All compared generator designs use the same SeRV tokenizer.Source: Table 5 · p. 8, §4.4 · p. 8

5 Conclusion

SeRV uses sentence-level and token-level motion–text alignment to learn text-predictable residual tokens for 3D ASL generation. With Hierarchical GPT and a reconstructed YouTube-ASL benchmark, experiments report state-of-the-art performance on How2Sign and YouTube-ASL.

5 Conclusion

  • SeRV combines sentence-level and token-level motion–text alignment to learn residual tokens that are precise for reconstruction and predictable from text.The method is designed for text-to-3D ASL generation.Source: §5 · p. 8
  • Hierarchical GPT predicts residual tokens across time and quantization depth while preserving the coarse-to-fine hierarchy, alongside a reconstructed 3D ASL motion-text benchmark from YouTube-ASL.Source: §5 · p. 8
  • Experiments on How2Sign and YouTube-ASL report state-of-the-art performance with consistent body- and hand-motion accuracy gains, and ablations support both semantic alignment and hierarchy-aligned generation.Source: §5 · p. 8

6 Limitations

The current framework has limitations in decoding efficiency, hand-specific linguistic supervision, non-manual expression modeling, and input modality.

6 Limitations

  • Sequential autoregressive token generation limits inference parallelism and makes long-sequence generation inefficient.The paper identifies masked, semi-autoregressive, and diffusion-based generation as possible directions for improving decoding speed while preserving temporal coherence.Source: §6 · pp. 8–9
  • The objective does not explicitly supervise linguistic handshape correctness, palm orientation, or sign-level articulatory categories.The paper suggests hand-specific objectives, higher-resolution hand tokenization, or dedicated hand refinement modules.Source: §6 · p. 9
  • The representation does not sufficiently model facial expression and emotion, which are essential non-manual components of sign language.The paper suggests incorporating facial expression parameters, affective cues, and emotion-aware supervision.Source: §6 · p. 9
  • The current setting uses text alone, leaving speech, prosody, glosses, reference videos, and signer-specific style prompts as possible future inputs.The paper frames these modalities as extensions for more controllable and expressive sign generation.Source: §6 · p. 9

A.1 Additional Controlled Ablation Studies

The controlled ablations examine the contributions of semantic objectives and the behavior of the residual hierarchy under matched experimental conditions.

A.1 Additional Controlled Ablation Studies

  • The experiments isolate the contributions of semantic objectives and analyze the residual representation hierarchy.Source: §A.1 · p. 10
  • The controlled setting is intended to distinguish objective effects from changes in experimental setup.The stated controls hold the listed protocol factors constant across compared variants.Source: §A.1 · p. 10
  • Compared variants use the same model architecture, data split, training schedule, and evaluation protocol unless otherwise specified.Source: §A.1 · p. 10

A.1.1 Controlled Tokenizer-Objective Ablation

The tokenizer-objective ablation finds that semantic supervision improves reconstruction and generation, with the combined objectives performing best across the reported datasets.

A.1.1 Controlled Tokenizer-Objective Ablation

  • The tokenizer variants share the RVQ architecture, codebooks, decoder, temporal downsampling ratio, reconstruction objective, data split, and training protocol; each semantic objective has weight 0.005 when enabled.Each tokenizer is paired with a Hierarchical GPT trained from scratch using identical generator settings.Source: §A.1.1 · pp. 10–11
  • 4.12 SiBLEU-4 on How2Sign is the combined-objective score, while Full SeRV achieves the best reconstruction and generation results across all reported YouTube-ASL metrics.On How2Sign, the combined objectives also produce the lowest body and hand DTW-JPE.Source: §A.1.1 · p. 11
  • On How2Sign, adding Lglobal raises SiBLEU-4 from 3.20 to 3.53, while adding Ltok raises it to 3.80 versus the RVQ-only baseline.Source: §A.1.1 · p. 11

A.1.2 Effect of Cumulative-Latent Supervision

Token-level supervision on increasingly complete cumulative latents improves semantic retrieval and generation quality, while later residual levels primarily refine the coarse representation.

A.1.2 Effect of Cumulative-Latent Supervision

  • The ablation applies Ltok to Level 0, the cumulative latent from Levels 0–1, or the full cumulative latent from all RVQ levels.The cumulative latent sums the contributions of RVQ levels through level k.Source: §A.1.2 · p. 11
  • 4.12 SiBLEU-4 and 14.52 T2S R@1 are obtained with full-latent supervision, up from 3.70 and 12.65 with Level-0 supervision.With Levels 0–1 supervision, the corresponding values are 3.91 SiBLEU-4 and 13.80 T2S R@1.Source: §A.1.2 · p. 11
  • Hierarchical GPT conditions each refinement level on text and lower-level motion tokens, making cumulative supervision compatible with the staged generation process.Source: §A.1.2 · p. 11
  • Similar temporal dispersion for Level 0 and the full latent suggests later residual levels mainly refine the coarse-level representation rather than independently encoding word-level alignment.Source: §A.1.2 · p. 11

A.1.3 Cumulative Contribution of RVQ Levels

The cumulative RVQ-level analysis finds that the first level carries most retrieval signal, while later levels improve reconstruction and modestly strengthen retrieval.

A.1.3 Cumulative Contribution of RVQ Levels

  • 13.61/7.79 T2S/S2T R@1 at Level 0 captures most of the cross-modal retrieval signal.The measures rise to 14.52/8.42 as residual levels are added.Source: §A.1.3 · p. 11
  • 20.84/8.06 to 16.23/6.13 body/hand PA-MPJPE shows that adding residual levels substantially improves reconstruction.Source: §A.1.3 · p. 11
  • The levels divide labor coarsely: Level 0 carries dominant motion structure and most text-related information, while later levels refine geometry and slightly strengthen alignment.Source: §A.1.3 · p. 11

A.1.4 Semantic-Loss Weight Sensitivity

The appendix tests semantic-loss weighting and perceptual outcomes, documents the reconstructed YouTube-ASL resource and its audit, and specifies evaluation and generation procedures.

A.1.4 Semantic-Loss Weight Sensitivity

  • λsem = 0.005 provides the strongest downstream generation performance, while 0.0005 slightly favors reconstruction and 0.05 degrades both.All model components and training settings were fixed while varying λsem; the selected setting yields body/hand DTW-JPE of 6.51/9.22 and SiBLEU-4 of 4.12.Source: §A.1.4 · pp. 11–12, Table 9 · p. 12

A.2 Human Perceptual Evaluation

  • SeRV ranks first in 58.9% of valid comparisons and has mean rank 1.53, versus 23.3%/2.19 for SOKE and 17.8%/2.28 for S-MotionGPT.Five participants ranked outputs for 30 videos; the 129 valid rankings assess perceptual motion similarity, not linguistic correctness, sign intelligibility, or grammatical accuracy.Source: Table 10 · p. 12, §A.2 · p. 12

A.3.2 Human Audit Protocol

  • 97.0% of 400 audit ratings judge reconstructed motion at least partially correct, but residual errors include finger articulation, occlusion, trajectories, timing, and smoothness.Four raters assessed 100 sampled clips against their source videos using hand shape, finger articulation, arm trajectory, and temporal alignment.Source: Table 12 · p. 13, §A.3.2 · p. 13

A.3.1 Data Processing Statistics; A.4 Comparison with Existing 3D Sign Resources

  • The processed YouTube3D resource contains 236,537 ASL–text pairs from 8,119 videos, totaling 295.1 hours, addressing the limited scale of existing ASL resources.The resource comparison concerns paired 3D sign-language motion; the YouTube-ASL processing statistics report retained clips and hours.Source: §A.4 · p. 13, §A.5 · pp. 13–14, §A.3.1 · p. 13

A.5 Data Preparation

  • YouTube-ASL preparation reconstructs upper-body pose, hand articulation, and facial expression, then filters multi-signer, unidentifiable, or low-confidence clips.Each paired motion sequence uses 133 SMPL-X parameters: 3D rotations for 11 upper-body and 30 hand joints, plus 10 facial-expression coefficients.Source: §A.5 · p. 14

A.6 Evaluation Metric Definitions

  • DTW-JPE aligns variable-length sequences with dynamic time warping, PA-MPJPE compares frame-aligned poses after Procrustes alignment, and SiBLEU-4 scores motion-token n-grams.Source: §A.6 · pp. 14–15

A.7 Implementation details.

  • SeRV training combines RVQ reconstruction with sentence-level and token-level text alignment, then Hierarchical GPT predicts frozen-tokenizer residuals in coarse-to-fine order.The tokenizer uses three RVQ levels with 512-dimensional latents and a 512-entry codebook per level; higher-level decoders refine lower-level representations.Source: §A.7 · p. 15

A.9 Additional Qualitative Results on Ablation Study

  • Qualitative comparisons report that SeRV preserves tokenizer motion details, especially hand articulation, and better captures hand information in generated signing motions.Source: Figure 5 · p. 17, Figure 6 · p. 17

Related papers

Loading 2609.05742v1…