Source-linked AI summary
Neural Sign Language Translation based on Human Keypoint Estimation
Sang-Ki Ko, Chang Jo Kim, Hyedong Jung, Choongsang Cho
TL;DR
Sign language translation must bridge challenging visual and linguistic information, while high-quality training data is difficult to collect. This paper introduces a Korean sign language dataset and a keypoint-based sequence-to-sequence model, then verifies the approach through experiments.
Problem
Sign language translation is challenging because it interprets visual information, including body movement and facial expression, as natural language sentences, while large datasets are difficult to collect.
Method
The paper constructs the KETI Korean sign language dataset and translates videos using human keypoints with object 2D normalization and a sequence-to-sequence model.
Results
The proposed idea is verified through various experiments using the sign language dataset.
Takeaways & Limitations
The KETI dataset provides a large-scale Korean resource, and human keypoints are used as high-level, lower-dimensional features for sign language translation.
Takeaways & Limitations
Future work is needed to translate gloss sequences into natural language sentences, potentially by separating video-to-gloss and gloss-to-language annotation.
Abstract
from arXiv · showhide
We propose a sign language translation system based on human keypoint estimation. It is well-known that many problems in the field of computer vision require a massive amount of dataset to train deep neural network models. The situation is even worse when it comes to the sign language translation problem as it is far more difficult to collect high-quality training data. In this paper, we introduce the KETI (short for Korea Electronics Technology Institute) sign language dataset which consists of 14,672 videos of high resolution and quality. Considering the fact that each country has a different and unique sign language, the KETI sign language dataset can be the starting line for further research on the Korean sign language translation. Using the KETI sign language dataset, we develop a neural network model for translating sign videos into natural language sentences by utilizing the human keypoints extracted from a face, hands, and body parts. The obtained human keypoint vector is normalized by the mean and standard deviation of the keypoints and used as input to our translation model based on the sequence-to-sequence architecture. As a result, we show that our approach is robust even when the size of the training data is not sufficient. Our translation model achieves 93.28% (55.28%, respectively) translation accuracy on the validation set (test set, respectively) for 105 sentences that can be used in emergency situations. We compare several types of our neural sign translation models based on different attention mechanisms in terms of classical metrics for measuring the translation performance.
1. Introduction
Sign language translation is difficult because it must map complex visual information—including body movement and facial expression—into natural language. The paper addresses this challenge with a Korean dataset, human-keypoint translation system, normalization, and experiments.
- Sign language translation requires interpreting body movement, hand shape, and facial expression as natural language sentences.
- The paper introduces the first large-scale Korean sign language dataset for translation research.
- The proposed system translates sign videos using 2D coordinates of human keypoints estimated from the signer.
- The paper presents an effective normalization technique for preprocessing human-keypoint coordinates.
- The authors verify the proposed ideas through experiments with the sign language dataset.
2. Related Work
Related work spans hand-language recognition, sentence-level sign-language recognition, weakly annotated translation corpora, and deep-learning sequence models. These studies motivate sequence-to-sequence approaches for mapping spatiotemporal sign representations to spoken or written language.
- Alphabetic hand-language recognition is simpler than sign-language recognition because each letter maps to a single hand shape.
- Sentence-level ASL recognition has been studied with Hidden Markov models, including systems using gloves and systems operating without gloves.
- RWTH-PHOENIX-Weather corpora provide weakly annotated German sign-language data from televised weather forecasts with gloss and natural-language annotations.
- Deep-learning approaches have addressed hand pose estimation, signer extraction, and sequence modeling for sign-language recognition.
- Sequence-to-sequence models use encoder-decoder architectures with recurrent cells for sequence generation tasks.
- Prior work formalized sign-language translation as learning mappings from spatiotemporal sign representations to spoken or written language.
3. KETI Sign Language Dataset
The KETI dataset targets Korean sign language in emergency situations and combines high-definition recordings, multiple signers and views, natural-language annotations, and gloss sequences. It is intended to support sign-language recognition and translation under limited data.
- The KETI dataset focuses on Korean sign language used by hearing-impaired people in emergency situations.
- The dataset contains 14,672 full-HD videos recorded at 30 frames per second from front and side camera angles.
- Recordings include 524 signs performed by fourteen hearing-impaired signers, with guide videos used to reduce sign ambiguity.
- Each of 105 emergency-related signs has five Korean natural-language sentence annotations.
- All sign videos are additionally annotated with gloss sequences, whose units transcribe individual signs.
- Table 1 presents ten annotation examples, while showing only two sentences per example because of space limitations.
4. Our Approach
The approach translates sign-language videos into natural-language sentences by extracting and normalizing human keypoints, augmenting frame sequences, and decoding with an attention-based sequence-to-sequence model.
- Human Keypoint Representation: The system extracts 124 signer keypoints from the body, hands, and face using OpenPose, excluding lower-body points.The selected representation contains 12 upper-body, 42 hand, and 70 facial keypoints.
- Feature Vector Normalization: Object 2D normalization standardizes keypoint coordinates using their mean and standard deviation to reduce signer-dependent visual variance.The method separately normalizes x- and y-coordinate vectors before concatenating them into the final feature vector.
- Frame Skip Sampling for Data Augmentation: The model randomly samples 10 to 50 keyframes, producing input feature dimensions of 248 × |V | for each video.Frame-skip sampling begins with an evenly spaced baseline and adds random offsets while preserving likely key moments.
- Attention-based Encoder-Decoder Network: An encoder-decoder architecture based on RNNs such as GRUs or LSTMs maps input sequences to output sentences by predicting each next word conditionally.The decoder uses the previously predicted word, its prior hidden state, and an encoder-derived context vector.
- Attention-based Encoder-Decoder Network: Bahdanau attention computes a dynamic context vector as a weighted combination of encoder hidden states using encoder-decoder alignment scores.The alignment function measures how well encoder and decoder hidden states match.
5. Experimental Results
Experiments evaluate preprocessing, augmentation, sampled frames, attention, batch size, annotation type, and feature choice for Korean sign language translation. Results favor normalized human keypoints, moderate augmentation, and models that generalize to independent signers.
- Experimental setup: The study uses Korean POS-level tokenization with Kkma and trains PyTorch models using Adam, dropout, gradient clipping, and scheduled learning-rate changes.The reported implementation uses 50 epochs, initial learning rate 0.001, hidden-state dimension 256 for sequence-to-sequence models, and batch size 128.
- Annotation type: The experiments compare sentence-level and gloss-level annotations, with gloss-level translation expected to perform better because each sign maps to a unique gloss sequence.Natural-language annotations contain multiple sentences per sign video, whereas gloss sequences are unique.
- CNN comparison: CNN-extracted features performed significantly worse than human keypoint features, although VGGNet-16 generalized best among the CNN models on the test set.The paper attributes the CNN disadvantage to many trainable parameters and weaker handling of subtle image variance for unseen signers.
- Attention mechanisms: The Transformer generalized better to the independent-signer test set, while Luong attention performed better on the more similar validation set.The comparison concerns four encoder-decoder architectures specialized for machine translation tasks.
- Data augmentation: 50 was the optimal augmentation factor for both validation and test sets, whereas no random frame augmentation caused severe overfitting.An augmentation factor of 100 produces 943,200 training samples from 9,432 videos, but larger factors do not always improve generalization.
- Number of sampled frames: 40 sampled frames gave the best validation performance, while 50 sampled frames gave the best test performance.The authors motivate frame selection partly by computational cost and declining sequence-to-sequence performance with longer inputs.
- Keypoint ablation: Both-hands keypoints were the most important among the evaluated hand, face, and body keypoints, while face keypoints reduced performance in all cases.The authors associate the face-keypoint result partly with the imbalanced number of keypoints, including 70 from the face.
6. Conclusions
The paper introduces a Korean sign language dataset and a sequence-to-sequence translation model using human keypoints, while identifying data scale and keypoint detection as future improvement areas.
- The authors introduce a manually annotated Korean sign language dataset and a neural translation model based on sequence-to-sequence architectures.The dataset addresses the scarcity of large sign language datasets, while the model uses human keypoints estimated by OpenPose as lower-dimensional features.
- The proposed system successfully trains a sign language translator using human keypoints estimated from sign language videos.The approach extracts high-level, lower-dimensional features from videos to support training when sufficient sign language data is difficult to collect.
- Improved data augmentation and stronger keypoint detectors are identified as future directions for improving translation performance.The paper specifically mentions spatial video augmentation and detectors such as Mask R-CNN and AlphaPose.
- Expanding the KETI dataset with more signers and recording environments is identified as an important next step.