Source-linked AI summary
Multi-channel Transformers for Multi-articulatory Sign Language Translation
Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, Richard Bowden
TL;DR
Sign-language translation must capture information from asynchronous manual and non-manual articulators, while prior approaches often depend on expensive gloss supervision. The paper proposes a multi-channel Transformer with inter- and intra-channel contextual modeling and channel-specific anchoring, achieving competitive PHOENIX14T performance without gloss reliance. This removes the gloss dependency that limits scaling to larger unannotated datasets.
Problem
Previous SLT approaches largely focused on manual features, did not consider multiple articulators, and relied on expensive gloss annotations that limit data availability.
Method
The paper extends Transformers with multi-channel attention for asynchronous manual and non-manual sources, modeling inter- and intra-channel context while preserving channel-specific information through anchoring losses.
Results
Experiments on RWTH-PHOENIX-Weather-2014T show competitive or on-par translation performance against state-of-the-art approaches, without relying on gloss information.
Takeaways & Limitations
Removing gloss dependence supports future scaling to larger datasets where gloss information is unavailable, including broadcast footage.
Abstract
from arXiv · showhide
Sign languages use multiple asynchronous information channels (articulators), not just the hands but also the face and body, which computational approaches often ignore. In this paper we tackle the multi-articulatory sign language translation task and propose a novel multi-channel transformer architecture. The proposed architecture allows both the inter and intra contextual relationships between different sign articulators to be modelled within the transformer network itself, while also maintaining channel specific information. We evaluate our approach on the RWTH-PHOENIX-Weather-2014T dataset and report competitive translation performance. Importantly, we overcome the reliance on gloss annotations which underpin other state-of-the-art approaches, thereby removing future need for expensive curated datasets.
1 Introduction
Sign language conveys meaning through interacting manual and non-manual articulators, but prior computational work largely emphasized manual features. This paper introduces a multi-channel Transformer that models asynchronous articulators without gloss-level supervision while achieving competitive translation performance.
- Sign languages combine manual features, such as hand shape and motion, with non-manual features including facial expressions, mouthings, and body pose.
- Manual features alone do not capture the full context of sign meaning, because manual and non-manual features affect each other when used together.
- Prior literature predominantly focused on manual features, while sign-language channels also introduce complexity because they are asynchronous.
- The proposed Multi-channel Transformer models hand shape, upper body pose, and mouthings through inter- and intra-channel contextual relationships.
- The approach combines multiple articulatory channels with anchoring losses to preserve channel-specific information and avoid gloss-level supervision.
- The authors report competitive continuous SLT performance on PHOENIX14T, on par with models trained using additional gloss annotation.
2 Literature Review
The literature review identifies two central limitations in sign-language translation: dependence on costly gloss annotations and insufficient modeling of multiple articulators. Earlier systems used recognition or tokenization strategies that often relied on gloss-trained representations.
- Sign-language processing matters for accessibility, sign linguistics, language deprivation, and human-computer interaction.
- Recognition addresses only part of communication because sign languages have distinct grammar and require reordering and mapping into the target spoken language.
- Earlier translation systems commonly separated recognition from translation or treated translation as text-to-text processing.
- Several prior approaches achieved translation results using gloss-based tokenization or pretrained gloss representations, while simpler non-gloss tokenization performed worse.
- The paper targets two shortcomings: glosses are expensive and limit data availability, and previous SLT work had not considered multiple articulators.
3 Background on Neural Machine Translation
Neural machine translation learns a conditional mapping from source sequences to target sequences using encoder-decoder architectures and attention. The paper extends Transformer attention to asynchronous multi-channel sources while preserving channel-specific information.
- Machine translation learns the conditional probability p(Y|X), mapping a source sequence X to a target sequence Y.
- RNN encoder-decoder models encode source sentences into context representations, but fixed-size context creates long-term dependency and information-bottleneck problems.
- Attention conditions target-word prediction on a weighted sum of source representations, with weights learned from decoder-state and encoder-output relevance.
- In dot-product attention, queries represent the decoder hidden state, while keys and values represent encoder outputs.
- Transformers use self-attention and learned linear projections to obtain queries, keys, and values without relying on recurrent processing.
- The proposed multi-channel attention refines each source-channel representation using other channels while anchoring channel-specific information, and adapts encoder-decoder attention to multiple sources.
4 Multi-channel Transformers
Multi-channel Transformers extend the Transformer for asynchronous source channels by modeling within-channel and cross-channel context while preserving channel-specific information. The architecture combines channel-specific attention, multi-channel attention, decoder integration, and translation and anchoring losses.
- Architecture: The architecture targets sequence-to-sequence learning where source information is distributed across several asynchronous channels.In sign language translation, these channels represent manual and non-manual features.
- Encoder: Channel-wise self-attention models contextual relationships within each source channel before feed-forward processing.Each channel uses individual self-attention and feed-forward layers to retain channel-specific representations.
- Encoder: Multi-channel encoder attention models relationships between self-attended channels by scoring each channel against the concatenated keys and values of the others.Concatenation occurs over the time axis, allowing source channels with different token counts.
- Decoder: Multi-channel decoder attention replaces standard encoder-decoder attention by scoring target queries against keys from all source channels and averaging their channel-specific outputs.The resulting representations are passed through a feed-forward layer, and stacked decoder layers condition target-token generation.
- Loss Functions: Training combines translation loss with channel anchoring loss to preserve channel-specific information during encoding.Anchoring uses classifier predictions as ground truth for token-level cross-entropy losses, while weighting coefficients control the relative importance of the losses.
5 Implementation and Evaluation Details
The evaluation uses PHOENIX14T with hand-shape, mouthing, and upper-body-pose channels, alongside specified optimization, decoding, and translation-evaluation procedures.
- Dataset: The model is evaluated on PHOENIX14T, described as the only publicly available large-vocabulary continuous sign-language-translation dataset aimed at vision-based research.
- Sign Channels: Three articulatory channels represent hand shapes, mouthings, and upper-body pose.Hand-shape and mouthing channels use 1024-dimensional CNN features, while channel predictions anchor their representations.
- Sign Channels: Upper-body pose uses 2D skeletal information lifted to 3D positions for 50 joints, without anchoring loss because no prior pose subunit classes were available.
- Training: Training uses Adam with batch size 32, learning rate 10^-3, weight decay 10^-3, Xavier initialization, and single-headed scaled dot-product attention.The networks are trained from scratch without dropout.
- Decoding: Beam width and length-penalty α are selected on the development set for test-time beam-search decoding.Beam width ranges from 0 to 10 and α ranges from 0 to 5.
- Metrics: Translation performance is measured with BLEU and ROUGE, reporting mean and standard deviation across 10 repetitions for BLEU-4 and ROUGE.The best setup is also reported according to development-set BLEU-4.
6 Experiment Results
Experiments compare single-channel, naive fusion, and Multi-channel Transformer approaches for sign language translation. The proposed architecture outperforms early and late fusion at matched parameter counts, while anchoring losses improve performance and enable larger networks to approach gloss-based results.
- 6.1 Single Channel Baselines: Single-channel experiments establish translation baselines across three articulators and network sizes from 32x64 to 256x512.All networks use two encoder and decoder layers.
- 6.1 Single Channel Baselines: Hand is the best-performing channel across network setups, while 128x256 provides the preferred configuration for subsequent experiments.The authors attribute this choice to limited training samples and over-fitting concerns.
- 6.1 Single Channel Baselines: 19.52 vs. 16.44 dev BLEU-4: gloss-trained CNN features outperform all single-articulator models.The 128x256 setup has better mean performance than the 256x512 setup despite the latter’s best individual development and test results.
- 6.2 Early and Late Fusion of Sign Channels: Early fusion concatenates channel features, whereas late fusion combines channel-specific processing for multi-channel translation benchmarks.Early fusion models contextual relationships implicitly through the transformer architecture.
- 6.2 Early and Late Fusion of Sign Channels: Hands-and-Mouth early fusion slightly improves over single-channel non-gloss models, but early-fusion performance worsens as more features increase model size.Late fusion improves in all tested scenarios; the authors associate early-fusion degradation with over-fitting on the small dataset.
- 6.3 Multi-channel Transformers: At matched parameter counts, the Multi-channel Transformer outperforms both early and late fusion; anchoring losses improve 128x256 models and bring 256x512 models near gloss-feature performance.The anchoring loss preserves channel-specific information during contextual modeling and regularizes translation loss.
- 6.3 Multi-channel Transformers: The best multi-channel model surpasses several previous methods and performs on par with the single-gloss model and competitively with transformer-based state-of-the-art approaches.Those transformer approaches use larger models and gloss supervision at multiple levels.
7 Conclusion
The paper presents a multi-channel transformer for modeling inter- and intra-channel relationships among asynchronous sign articulators. On RWTH-PHOENIX-Weather-2014T, it achieves competitive or state-of-the-art-comparable performance without relying on gloss information.
- 7 Conclusion: The architecture models contextual relationships among asynchronous channels within the transformer while preserving channel-specific information.The approach targets multi-channel sign language translation.
- 7 Conclusion: Experiments on RWTH-PHOENIX-Weather-2014T demonstrate performance on par with or competitive with the state-of-the-art.The conclusion identifies removal of gloss dependence as the more important outcome.
- 7 Conclusion: Removing gloss dependence supports future scaling to larger datasets, including broadcast footage without gloss annotations.This is presented as future work rather than an evaluated result.