PaperlayerView exact revision on arXiv

Source-linked AI summary

AUTSL: A Large Scale Multi-modal Turkish Sign Language Dataset and Baseline Methods

Ozge Mercanoglu Sincan, Hacer Yalim Keles

arXiv:2008.00932v2cs.CV

TL;DR

Large-vocabulary sign language recognition remains difficult because signs combine hand, body, and facial articulations under realistic visual variation. The paper introduces AUTSL, a multimodal Turkish Sign Language dataset and user-independent benchmark, and evaluates CNN–LSTM baselines with pooling and temporal attention. The best baseline reaches 62.02% accuracy on the user-independent benchmark, while performance is higher on random splits and Montalbano.

  • Problem

    Large-vocabulary sign language recognition remains challenging in realistic settings because signs combine hand, body, and facial articulations.

  • Method

    The paper builds AUTSL with RGB, depth, and skeleton data, then evaluates CNN–LSTM and BLSTM baselines augmented with feature pooling and temporal attention.

  • Results

    62.02% accuracy was achieved by the best baseline on the user-independent AUTSL benchmark, compared with up to 95.95% on random splits and 96.11% on Montalbano.

  • Takeaways & Limitations

    AUTSL provides a public benchmark for evaluating sign recognition under signer-independent and varied-background conditions.

  • Takeaways & Limitations

    Similar signs can still be misclassified, although the correct class is usually among the model’s top-3 or top-5 predictions.

Abstract

from arXiv · show

Sign language recognition is a challenging problem where signs are identified by simultaneous local and global articulations of multiple sources, i.e. hand shape and orientation, hand movements, body posture, and facial expressions. Solving this problem computationally for a large vocabulary of signs in real life settings is still a challenge, even with the state-of-the-art models. In this study, we present a new largescale multi-modal Turkish Sign Language dataset (AUTSL) with a benchmark and provide baseline models for performance evaluations. Our dataset consists of 226 signs performed by 43 different signers and 38,336 isolated sign video samples in total. Samples contain a wide variety of backgrounds recorded in indoor and outdoor environments. Moreover, spatial positions and the postures of signers also vary in the recordings. Each sample is recorded with Microsoft Kinect v2 and contains RGB, depth, and skeleton modalities. We prepared benchmark training and test sets for user independent assessments of the models. We trained several deep learning based models and provide empirical evaluations using the benchmark; we used CNNs to extract features, unidirectional and bidirectional LSTM models to characterize temporal information. We also incorporated feature pooling modules and temporal attention to our models to improve the performances. We evaluated our baseline models on AUTSL and Montalbano datasets. Our models achieved competitive results with the state-of-the-art methods on Montalbano dataset, i.e. 96.11% accuracy. In AUTSL random train-test splits, our models performed up to 95.95% accuracy. In the proposed user-independent benchmark dataset our best baseline model achieved 62.02% accuracy. The gaps in the performances of the same baseline models show the challenges inherent in our benchmark dataset. AUTSL benchmark dataset is publicly available at https://cvml.ankara.edu.tr.

1 Introduction

The paper frames sign language recognition as a difficult computer-vision task and introduces AUTSL to address the shortage of realistic, large-scale Turkish Sign Language data. It also proposes deep-learning baselines and user-independent evaluation.

  • Motivation: Sign recognition must capture local and global articulations across hands, arms, face, posture, and temporal repetition.Similar gestures can differ through facial expression or repetition count, while signer, duration, illumination, and background variation add difficulty.
  • 11?

2 Related Works

Earlier sign language recognition work progressed from handcrafted features and conventional classifiers toward deep neural networks, but realistic large-scale datasets remained limited. Recent datasets expanded vocabulary, samples, or signer counts, while often retaining plain backgrounds that constrain real-life applicability.

  • Conventional methods: Traditional systems extracted handcrafted features such as SIFT and HOG, then used SVMs, K-NN, HMMs, or DTW for recognition.These approaches separated feature extraction from classification or sequence matching.
  • Deep learning methods: Deep learning methods increasingly use CNNs for feature extraction, sometimes with segmentation networks and recurrent models for temporal information.CNN features are commonly followed by recurrent neural networks rather than only fully connected layers.
  • Datasets: Most earlier datasets were small in signs, signers, or samples, while newer large-scale datasets improved one or more of these dimensions.The literature includes datasets with large vocabularies, many samples, or many signers, but their characteristics differ substantially.
  • Datasets: AUTSL addresses a gap by providing a large-scale Turkish dataset with many backgrounds and challenging variation rather than predominantly plain recording environments.Its dataset design emphasizes realistic conditions for practical sign language recognition.
  • Multimodal methods: Multimodal fusion combines inputs such as RGB, depth, intensity, articulated pose, and audio, with prior work reporting gains from fusing modalities at multiple scales.Some systems fuse CNN-extracted streams through fully connected layers.
  • Attention methods: Attention mechanisms assign importance to spatial locations or temporal information and have been incorporated into recurrent and visual models for sign-related tasks.Examples include attention in bidirectional RNNs for translation, image captioning, and ASL fingerspelling recognition.

3 AUTSL Dataset

AUTSL is a multimodal, isolated Turkish Sign Language dataset designed around diverse recording conditions and user-independent evaluation. Its 226 signs, 43 signers, 38,336 samples, varied backgrounds, and similar-sign cases create a challenging benchmark.

  • Dataset composition: AUTSL contains 226 signs, 38,336 samples, and recordings from 43 signers using RGB, depth, and skeleton modalities captured with Microsoft Kinect v2.The dataset targets large-scale multimodal isolated sign recognition.
  • Sign diversity and difficulty: The selected signs cover varied hand shapes and movements while retaining signs that can be difficult to distinguish because of similar performance characteristics.Some signs involve hands occluding one another or the face.
  • Recording conditions: The dataset includes 20 different backgrounds spanning indoor and outdoor settings, with added variation from camera field-of-view changes and background objects.These changes increase appearance variation across recordings.
  • Dataset composition: AUTSL has balanced sign distribution, although signer sample counts differ because some signers were recorded repeatedly with different clothes or backgrounds.Figure 2 presents distributions across signs and signers.
  • Benchmark protocol: The user-independent benchmark uses 36 signers for training and validation and seven unseen signers for testing, with 27,676 training, 4,884 validation, and 5,776 test samples.The test set contains nine backgrounds, including three absent from training and validation.

4 The Methods

The baseline methods combine CNN-based spatial feature extraction with recurrent sequence modeling, while testing feature pooling, temporal attention, and bidirectional recurrence. The models support RGB and RGB-D inputs and are trained end-to-end without explicit segmentation.

  • Model components: Feature Pooling Modules use multiple dilation rates to represent spatial features at multiple scales and preserve contextual clues.The parallel convolutional outputs are concatenated after processing with different dilation rates.
  • Model components: Temporal attention forms a context vector from a weighted sum of all hidden states, assigning greater contribution to more informative frames.Attention weights are normalized with softmax, so the weights across frames sum to 1.
  • Model components: CNNs extract spatial features from each video frame, and LSTM-based models capture temporal relationships across frames.
  • Baseline models: The baseline family compares unidirectional LSTM, bidirectional LSTM, feature pooling, and temporal attention configurations.Five deep neural networks are separately trained end-to-end to assess these components.
  • Baseline models: The models operate on RGB or RGB-D data without explicit segmentation, using parallel VGG-based streams and late fusion for RGB-D inputs.Depth frames are repeated across three channels so they can be processed by pretrained VGG models.

5 Results and Discussion

The evaluation uses AUTSL and Montalbano, with AUTSL tested both under user-independent and random train-validation-test splits. The main AUTSL benchmark holds out signers for testing.

  • Evaluation setup: AUTSL user-independent experiments train and validate on 36 signers and test on the remaining 7 signers.
  • Evaluation setup: Randomly selected training, validation, and test sets are also used to measure performance in a user-dependent setting.
  • Evaluation setup: The baseline models are evaluated on both the new AUTSL dataset and the Montalbano Italian gesture dataset.

5.1 Evaluation Metric

The paper evaluates recognition using top-1, top-3, and top-5 recognition rates, reflecting whether the true sign is the model’s best prediction or appears among its highest-ranked predictions.

  • Recognition rates: Top-1 recognition rate is the proportion of samples whose predicted label equals the true label.The metric counts one when p(i) = y(i), and zero otherwise, across n samples.
  • Recognition rates: Top-N recognition rate measures how often the true class label appears among the model’s top-N predictions.Because similar AUTSL signs can be confused, the evaluation includes top-3 and top-5 recognition rates.

5.2 Experiment Results on AUTSL

AUTSL experiments show that multimodal inputs and temporal attention substantially improve recognition, while user-independent evaluation remains difficult because similar signs and dynamic backgrounds cause errors.

  • User-independent evaluation: 39.31% recognition was reached by fusing RGB and depth, compared with 23% for vanilla CNN + LSTM using RGB alone in user-independent evaluation.The authors attribute the gain to depth information for signs moving toward or away from the camera.
  • Prediction analysis: 66.64% of RGB-D predictions contained the true sign in the top five, compared with 59.13% in the top three, revealing confusion among similar signs.Top-3 and top-5 recognition rates increased substantially over top-1 performance.
  • RGB-D ablations: 18.49% top-1 improvement came from temporal attention on RGB-D data, whereas feature pooling alone improved balanced-test recognition by 1.95%.Adding feature pooling after attention produced a further 2.22% top-1 improvement, yielding the CNN + FPM + LSTM + Attention baseline.
  • RGB ablations: 44.89% top-1 recognition was obtained with RGB-only CNN + FPM + LSTM + Attention, up from 23.00% for the vanilla model.Attention contributed more than 19%, while feature pooling added 2.75%; RGB-D still exceeded RGB by 15.13% in the balanced test.
  • Confusion analysis: 10 of approximately 17 test samples of “dede” were confused with “dolu” by the best RGB-D model.The confusion matrix shows that particular visually similar sign pairs remain difficult.
  • Attention visualization: Temporal attention highlights sign-relevant motion and discards initial and ending neutral frames, supporting the quantitative accuracy gains.Spatial and temporal visualizations also show that dynamic background people can redirect attention and cause misclassification.

5.3 Experiment Results on Montalbano

The authors evaluate their best CNN + FPM + BLSTM + Attention model on the Montalbano isolated gesture dataset, using RGB and RGB-D inputs. It achieves competitive state-of-the-art performance, with depth improving the reported accuracy.

  • Montalbano contains 20 Italian gestures performed by 27 users, with approximately 14,000 samples across training, validation, and test splits.The videos provide RGB, depth, user segmentation, and skeleton modalities from Microsoft Kinect recordings.
  • The Montalbano videos are converted into isolated samples and cropped around the signers’ upper bodies using shoulder-center skeleton coordinates.Frames are resized to 400x400 pixels during preprocessing.
  • The experiments use batch size 16 and an initial learning rate of 1e-4 for Montalbano.These settings differ from the AUTSL configuration because Montalbano videos have a fixed number of frames.
  • 95.46% accuracy is achieved using RGB data, while 96.11% accuracy is achieved using RGB-D data.The evaluated model is CNN + FPM + BLSTM + Attention.

6 Conclusion

The paper introduces AUTSL as a large-scale public Turkish Sign Language benchmark with varied recording conditions and user-independent evaluation, alongside deep-learning baselines. Results show strong performance on Montalbano but substantial difficulty on visually similar signs and unconstrained backgrounds, motivating further robustness work.

  • AUTSL is a large-scale isolated Turkish Sign Language dataset with varied backgrounds, multiple signers, and a user-independent benchmark.The dataset is presented as a public benchmark for evaluating recognition models.
  • The benchmark includes RGB-D and RGB baseline models built from CNN + LSTM architectures, incrementally augmented with feature pooling, temporal attention, and BLSTM variants.The best results are obtained with RGB-D input and CNN + FPM + BLSTM + Attention.
  • The models achieve competitive results against state-of-the-art approaches on Montalbano using both RGB and RGB-D data.The conclusion reports comparisons with state-of-the-art methods on that dataset.
  • Visually similar signs are misclassified, while varied backgrounds from unconstrained settings substantially degrade performance.The authors provide spatial and temporal attention visualizations supporting these observations.
  • Future work targets more robust spatial and temporal attention for dynamic backgrounds and more discriminative training for similar signs.These directions are proposed to improve classification performance on the AUTSL benchmark.
Loading 2008.00932v2…