PaperlayerView exact revision on arXiv

Source-linked AI summary

Signing at Scale: Learning to Co-Articulate Signs for Large-Scale Photo-Realistic Sign Language Production

Ben Saunders, Necati Cihan Camgoz, Richard Bowden

arXiv:2203.15354v1cs.CV

TL;DR

Large-scale sign production needs to move beyond under-articulated sequences and constrained vocabularies to continuous photo-realistic signing. The paper aligns dictionary signs with FS-NET and synthesizes video with SIGNGAN; evaluations report signer comprehension and improvements over baselines.

  • Problem

    Existing deep-learning sign production produces under-articulated skeleton sequences from constrained vocabularies, limiting its reach across large discourse domains.

    Source: §1 · p. 1

  • Method

    The system translates speech to gloss, uses FS-NET to align dictionary signs into continuous poses, then uses SIGNGAN and a keypoint-based loss to synthesize photo-realistic video.

    Source: §1 · p. 2, §6 · p. 8

  • Results

    SIGNGAN outperformed baselines in quantitative metrics, human evaluation, and native-signer comprehension; PHOENIX14T back translation improved 43% over baselines.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The approach demonstrates large-scale signing from unconstrained discourse that native signers can understand.

    Source: §6 · p. 8

    The approach remains limited by text-to-gloss translation performance in large-scale domains, where gloss annotations are scarce.

    Source: §6 · p. 8

Abstract

from arXiv · show

Sign languages are visual languages, with vocabularies as rich as their spoken language counterparts. However, current deep-learning based Sign Language Production (SLP) models produce under-articulated skeleton pose sequences from constrained vocabularies and this limits applicability. To be understandable and accepted by the deaf, an automatic SLP system must be able to generate co-articulated photo-realistic signing sequences for large domains of discourse. In this work, we tackle large-scale SLP by learning to co-articulate between dictionary signs, a method capable of producing smooth signing while scaling to unconstrained domains of discourse. To learn sign co-articulation, we propose a novel Frame Selection Network (FS-Net) that improves the temporal alignment of interpolated dictionary signs to continuous signing sequences. Additionally, we propose SignGAN, a pose-conditioned human synthesis model that produces photo-realistic sign language videos direct from skeleton pose. We propose a novel keypoint-based loss function which improves the quality of synthesized hand images. We evaluate our SLP model on the large-scale meineDGS (mDGS) corpus, conducting extensive user evaluation showing our FS-Net approach improves co-articulation of interpolated dictionary signs. Additionally, we show that SignGAN significantly outperforms all baseline methods for quantitative metrics, human perceptual studies and native deaf signer comprehension.

1. Introduction

Large-scale sign language production must generate continuous, co-articulated signing across broad discourse domains. The paper combines dictionary-sign alignment with photo-realistic video synthesis and evaluates both signer perception and translation performance.

1. Introduction

  • Earlier deep-learning SLP either concatenated isolated signs without natural co-articulation or generated continuous sequences that were under-articulated and struggled beyond weather domains.These limitations motivate learning co-articulation between dictionary signs.Source: §1 · p. 2
  • FS-NET learns a temporal alignment path to select frames from dictionary signs that best represent continuous signing, with transformer cross-attention supervised by Dynamic Time Warping.The method models co-articulation between dictionary signs via monotonic alignment to continuous sequences.Source: §1 · p. 2
  • SIGNGAN generates photo-realistic continuous sign videos from skeleton poses, while a keypoint-based loss improves hand synthesis and avoids motion-blur issues.The method also models a multi-modal distribution of signing videos in different styles.Source: §1 · p. 2
  • Back translation on PHOENIX14T improved 43% over baselines, and deaf users overwhelmingly preferred FS-NET to baseline SLP methods.User evaluation on mDGS showed FS-NET improved the natural signing motion of interpolated dictionary sequences.Source: §1 · p. 2
  • SIGNGAN outperformed state-of-the-art synthesis methods on C4A in quantitative and human-perception studies and was more understandable than skeletal sequences to deaf users.The comprehension comparison is with skeletal sequences previously used to represent sign.Source: §1 · p. 2
  • The paper presents its system as the first SLP model to produce large-scale sequences from unconstrained discourse at a level understandable to native deaf signers.The stated target is photo-realistic continuous signing for large discourse domains.Source: §1 · pp. 1–2

2. Related Work

Related work spans sign recognition, translation, production, and pose-conditioned synthesis. The paper situates its approach against limitations in co-articulation, signing realism, hand synthesis, and style diversity.

2. Related Work

  • Sign-language technology shifted from recognition [29] toward translation, while avatar-based production used rules-based co-articulation that did not generalize to unseen sequences [50].Source: §2 · p. 2
  • Deep-learning SLP first concatenated isolated signs, while continuous regression methods [49] exhibited under-articulation; this work instead learns temporal alignment.Source: §2 · p. 2
  • Skeleton representations have been associated with reduced deaf comprehension, while earlier photo-realistic signer systems produced only low-resolution isolated signs [46].The paper describes its output as high-resolution, continuous video from spoken-language input across unrestricted domains.Source: §2 · p. 2
  • Prior pose-conditioned synthesis addressed whole bodies, faces, and hands,, [67], but accurate hands in full-body sign synthesis remained a gap.The paper proposes a keypoint-based loss for hand synthesis; FaceGAN addressed high-resolution faces, not sign-language hand interaction.Source: §2 · pp. 2–3
  • Motion-transfer work and few-shot appearance methods left limited research on novel poses and produced only a single style at inference.Source: §2 · p. 3
  • Co-articulation is the influence of one phonetic element on another across intervening elements; this work models sign duration, boundaries, and transitions through alignment to continuous signing.Figure 2 depicts the pipeline from gloss-based dictionary signs through FS-NET alignment to SIGNGAN video generation.Source: §2 · p. 3, Figure 2 · p. 3

3. Large-Scale Photo-Realistic SLP

The section presents a three-stage route from spoken language to glosses, co-articulated skeleton poses, and style-conditioned photo-realistic signing video. FS-NET aligns interpolated dictionary signs to continuous signing, while SIGNGAN synthesizes the video from pose.

3. Large-Scale Photo-Realistic SLP

  • The overall method targets unconstrained spoken-language input and produces continuous photo-realistic sign video through its gloss, pose, and video-generation stages.A style image accompanies the pose sequence at the final stage.Source: §3 · p. 3

3.1. Text to Gloss

  • The pipeline translates spoken language into glosses with an encoder-decoder transformer, then uses the gloss sequence to retrieve ordered dictionary signs.Dictionary examples can expand the vocabulary; neighboring signs are linearly interpolated into a continuous pose sequence before FS-NET alignment.Source: §3.1 · pp. 3–4, §3.2 · p. 4

3.2. Gloss to Pose

  • FS-NET learns a monotonic temporal alignment between dictionary-sign frames and continuous signing, selecting or skipping frames to form the co-articulated pose sequence.Its alignment is trained with a Dynamic Time Warping path; the method assumes frames are removed rather than added, based on Q >> T for 88% of mDGS sequences.Source: §3.2 · p. 4

3.3. Pose to Video

  • SIGNGAN generates signer video conditioned on pose and a style image, with style controlling signer-specific appearance while pose remains signer-invariant.Its objective combines multi-scale adversarial, feature-matching, perceptual reconstruction, and hand-keypoint losses; the keypoint discriminator compares generated- and real-hand keypoints.Source: §3.3 · p. 5, Figure 3 · p. 5

4. Experiments

The experiments evaluate large-scale SLP on mDGS and PHOENIX14T, then assess co-articulation, comprehension, and photo-realistic video synthesis. User studies and synthesis comparisons favor the proposed methods, while text-to-gloss performance remains lower on the larger-domain mDGS protocols.

4.1. Experimental Setup

  • The setup uses mDGS free-flowing signing and PHOENIX14T weather broadcasts, with C4A videos for photo-realistic generation.The mDGS protocol contains 40,230 segmented sentence, gloss, and video examples; gloss variants are either included or removed in two protocols.Source: §4.1 · p. 6

4.2.1 Text to Gloss

  • Text-to-gloss translation scored 21.93 BLEU-4 on PHOENIX14T, outperforming (20.23) but trailing (23.17), while results were considerably lower on mDGS-V and mDGS.The authors link the lower scores to the larger domain and note that scaling translation to larger vocabularies requires further research.Source: §4.2.1 · p. 6, Table 1 · p. 6

4.2.2 Gloss to Pose

  • FS-NET productions were unanimously preferred to the baseline for understandability on both mDGS and PHOENIX14T, and trimmed-sign productions were preferred 40% of the time for smoother transitions.For trimmed signs, participants reported equal preference 13% of the time; untrimmed dictionary sequences were unanimously preferred to isolated sequences.Source: §4.2.2 · p. 7, Table 4 · p. 7, Table 5 · p. 7

4.2.3 Pose to Video

  • On C4A, SIGNGAN outperformed all baselines, particularly for Hand SSIM and FID; the hand keypoint loss improved performance, especially Hand SSIM.The hand discriminator performed poorly for SSIM and Hand SSIM, which the authors attributed to blurred hands.Source: §4.2.3 · pp. 7–8, Table 6 · p. 7, Table 7 · p. 8
  • SIGNGAN outputs were preferred for body synthesis by 96.2% on average and hand synthesis by 95.6% on average; Vid2vid was the strongest contender, with SIGNGAN preferred 85% of the time.The perceptual study included 46 participants, 28% of whom were signers.Source: §4.2.3 · p. 8, Table 8 · p. 8
  • Deaf participants rated synthesized videos 3.9 out of 5 for comprehension, compared with 3.2 for corresponding skeletal sequences.For FS-NET productions, participants recognized an average of 4.8 of the five signs shown in each video.Source: §4.2.2 · p. 7, §4.2.3 · p. 8

4.3. Qualitative Evaluation

  • The qualitative evaluation presents example photo-realistic frames in Figure 4, with additional qualitative evaluation in the supplementary materials.Source: §4.3 · p. 8

5. Potential Negative Societal Impact

The authors caution that SLP remains under development and should not yet be relied upon. Incorrect sign-language output could be harmful, particularly in emergency settings.

  • SLP remains under development and should not yet be relied upon, particularly because incorrect sign-language content could cause harm in emergency settings.Source: §5 · p. 8

6. Conclusion

The paper presents FS-NET for co-articulating dictionary signs and SIGNGAN for photo-realistic video generation, with a keypoint-based loss for hand synthesis. The authors report favorable co-articulation, comprehension, and baseline-comparison results, while identifying text-to-gloss performance as a limitation for large-scale domains.

  • FS-NET co-articulates dictionary signs through temporal alignment, while SIGNGAN generates photo-realistic sign videos and a keypoint-based loss improves hand synthesis despite motion blur.Source: §6 · p. 8
  • The authors report scaling to unconstrained domains, native-signer understanding, improved realism and preference over baselines, and SIGNGAN gains in quantitative, human-evaluation, and deaf-signer-comprehension comparisons.Source: §6 · p. 8
  • Large-domain SLP remains limited by text-to-gloss performance; the authors identify limited gloss annotations and the need for improved architectures and datasets.They also identify spatial co-articulation between dictionary signs as future work.Source: §6 · p. 8

Appendices

The appendix describes the mDGS translation protocols: free-flowing discourse is segmented into aligned German sentences, glosses, and videos. It also notes that dataset use requires permission from the University of Hamburg.

meineDGS (mDGS) Translation Protocol

Related papers

Loading 2203.15354v1…