PaperlayerView exact revision on arXiv

Source-linked AI summary

BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues

Samuel Albanie, Gül Varol, Liliane Momeni, Triantafyllos Afouras, Joon Son Chung, Neil Fox, Andrew Zisserman

arXiv:2007.12131v2cs.CV

TL;DR

Recognising continuous signing remains challenging because co-articulation distinguishes natural signing from isolated signs. This paper uses visual keyword spotting of mouthings to annotate BSL signs in broadcast video, and reports that its models exceed state of the art on MSASL and WLASL.

  • Problem

    Recognising naturally produced continuous signing remains challenging because co-articulation makes signs visually different from isolated dictionary examples.

    Source: §1 · pp. 1–2

  • Method

    The paper uses visual keyword spotting to align subtitle words with signers’ mouthings in broadcast video and automatically annotate localized signs.

    Source: §1 · p. 2

  • Results

    The BSL-1K approach exceeds state of the art on both MSASL and WLASL benchmarks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    BSL-1K, coupled with 3D CNN training, supports high-performance large-vocabulary sign recognition and provides beneficial initialization for ASL benchmarks.

    Source: §6 · p. 14

    Sign recognition failures are driven by strong visual similarity in sign production.

    Source: §5.2 · p. 14

Abstract

from arXiv · show

Recent progress in fine-grained gesture and action classification, and machine translation, point to the possibility of automated sign language recognition becoming a reality. A key stumbling block in making progress towards this goal is a lack of appropriate training data, stemming from the high complexity of sign annotation and a limited supply of qualified annotators. In this work, we introduce a new scalable approach to data collection for sign recognition in continuous videos. We make use of weakly-aligned subtitles for broadcast footage together with a keyword spotting method to automatically localise sign-instances for a vocabulary of 1,000 signs in 1,000 hours of video. We make the following contributions: (1) We show how to use mouthing cues from signers to obtain high-quality annotations from video data - the result is the BSL-1K dataset, a collection of British Sign Language (BSL) signs of unprecedented scale; (2) We show that we can use BSL-1K to train strong sign recognition models for co-articulated signs in BSL and that these models additionally form excellent pretraining for other sign languages and benchmarks - we exceed the state of the art on both the MSASL and WLASL benchmarks. Finally, (3) we propose new large-scale evaluation sets for the tasks of sign recognition and sign spotting and provide baselines which we hope will serve to stimulate research in this area.

1 Introduction

The paper targets continuous BSL recognition, where co-articulation makes signs difficult to recognize, by using mouthing cues and subtitles to scale annotation. It presents BSL-1K, recognition and spotting evaluations, and baselines.

1 Introduction

  • Continuous sign recognition remains challenging because co-articulation between neighboring signs makes naturally produced signs visually different from isolated dictionary examples.The paper frames large-scale data as a path forward, noting parallels with speech recognition and lip reading.Source: §1 · pp. 1–2
  • Subtitle alignment is challenging because signs may be brief, subtitles may be delayed, English and sign word order can differ, and words may be unmouthed or signed differently by context.The paper uses abundant broadcast footage with an overlaid interpreter and subtitles as the basis for alignment.Source: §1 · p. 2
  • Visual keyword spotting addresses alignment by searching for a known word in a short window, an easier task than directly lip-reading arbitrary isolated mouthings.Large video datasets with aligned audio transcriptions enable training visual keyword-spotting models.Source: §1 · p. 2
  • The paper introduces BSL-1K, a large-scale dataset of British Sign Language signs with a 1K-sign vocabulary, annotated using signers’ mouthing cues.Visual keyword spotting detects mouthings to obtain annotations from video.Source: §1 · p. 2
  • BSL-1K-trained models recognize co-articulated BSL signs, transfer to other sign languages and benchmarks, and exceed the state of the art on MSASL and WLASL.The paper also proposes new evaluation datasets for sign recognition and sign spotting, with baselines for both tasks.Source: §1 · p. 3

2 Related Work

Prior sign-language datasets leave a gap in large-scale, co-articulated BSL data, while earlier work has not combined subtitle-based weak supervision with mouthing cues for precise sign localisation. The related work also situates BSL-1K among evolving recognition methods and earlier uses of mouth information.

Sign language datasets

  • BSL datasets are scarce, and the linguistic corpus offers fine-grained annotations but covers only a fraction of its source videos; BSL-1K provides more annotations than previous public datasets in a large-domain co-articulated setting.Earlier datasets were limited in size, vocabulary, or signers; large-vocabulary MSASL and WLASL contain isolated signs, while realistic continuous datasets have limited discourse domains.Source: §2 · pp. 3–4

Sign language recognition

  • Recognition research has moved from hand-crafted features and pose toward 3D CNNs, with I3D outperforming pose-based counterparts in recent work; this paper also explores pose distillation for pretraining.Prior sequence models include HMMs and LSTMs, while translation work is limited to RWTH-Phoenix weather discourse and training-set size.Source: §2 · p. 4

Using the mouth patterns

  • BSL-1K combines weakly aligned subtitles and mouthing cues for sign localisation, bringing together ideas used separately in prior work and extending earlier subtitle-and-mouth alignment.Keyword spotting precisely localises signs; the work gathers signs over 1,000 hours, compared with 30 hours in.Source: §2 · pp. 4–5, Figure 1 · p. 5
  • Earlier mouth-shape work mainly recognises gestures [3, 38], while mouthings have received less attention through viseme categories; uses an open/closed-mouth classifier rather than precise keyword spotting.The comparison also distinguishes scale: this work covers 1,000 hours of signing versus 30 hours in.Source: §2 · pp. 4–5

3 Learning Sign Recognition with Automatic Labels

This section describes how subtitle-aligned broadcast footage and visual keyword spotting are used to build BSL-1K, a large-scale dataset of localized British Sign Language signs. It also describes signer-independent splits and human validation of the automatic annotations.

3.1 Finding probable signing windows in public broadcast footage

  • The source comprises 1,412 BBC episodes with 1,060 hours of continuous BSL signing and about 9.5 million subtitle words, yielding 1,350 initial vocabulary queries.Words were selected when present in subtitles and listed in both BSL signbank and sign BSL; the footage features forty professional interpreters translating program content.Source: §3.1 · pp. 5–6

3.2 Precise sign localisation through visual keyword spotting

  • Visual keyword spotting uses subtitle-centered windows and mouthing cues to localize target signs, rather than requiring general lip reading.Subtitle proposals are noisy because spoken words may not be signed or may be misaligned by seconds. Since mouthings can be partial and noncontinuous, the method searches for one target among a small candidate set; it pads subtitles by four seconds on each side and uses a 0.6-second sign window, a 0.04-second stride, and a 0.5 confidence threshold.Source: §3.2 · pp. 6–7

3.3 BSL-1K dataset construction and validation

  • The pipeline localizes approximately 280k signs from 2.4 million subtitle candidates, then forms signer-independent splits and a final 1,064-word vocabulary.Episodes are grouped by forty signers and split into 32 training, four validation, and four test signers; each vocabulary word has at least one training instance with confidence 0.8 or higher.Source: §3.3 · pp. 7–8
  • In an initial human check, 70% of 1,000 sampled automatic annotations were marked correct, supporting the localization approach.The final manually verified test set contains 2,103 correct annotations covering 334 of the 1,064 vocabulary signs; reported experimental test results use this verified set.Source: §3.3 · p. 8

4 Models and Implementation Details

The section describes the models used for mouth-based sign collection, sign recognition and spotting, and pose-distillation pretraining.

4.1 Visual keyword spotting model

  • The visual keyword spotter combines face-cropped video features with a fixed-length P2G keyword embedding to estimate framewise keyword probabilities and localize detections.It uses a 512 × 512 SSD face detector, BiLSTMs, and a sigmoid output; clips are thresholded by maximum posterior, and 0.6-second temporal non-maximum suppression removes duplicates. The P2G variant improves on the G2P encoder-decoder in the P2G baseline.Source: §4.1 · p. 9

4.2 Sign recognition model

  • The sign recogniser uses an RGB-only I3D model trained on 16-frame clips, with all parameters fine-tuned when pretrained weights are used.Videos are resized to 224 × 224; training uses cross-entropy, SGD with momentum 0.9, batch size 4, and a 50-epoch learning-rate schedule, plus colour, scale, and horizontal-flip augmentation and 0.5 dropout. At test time, it centre-crops, applies a sliding window with stride 8, and averages classification scores for a video-level prediction.Source: §4.2 · pp. 9–10

4.3 Video pose distillation

  • The pose-distillation network trains a 16-frame video embedding to regress 130 OpenPose keypoints, encouraging it to encode pose and dynamic gesture information.It assumes one person per frame and is trained on part of BSL-1K’s training set; architectural and training details are deferred to Appendix B.Source: §4.3 · p. 10

5 Experiments

Experiments evaluate mouthing-threshold and model ablations, establish BSL-1K recognition and spotting baselines, and test transfer to ASL benchmarks. The reported results show benefits from larger automatically annotated training sets, multimodal cues, video pose distillation, and BSL-1K pretraining.

5.1 Ablations for the sign recognition model

  • 70.61% accuracy on the manually validated test set was achieved by training on the larger, noisier BSL-1Km.5 set at a 0.5 mouthing threshold.The lower threshold yields more sign instances but potentially noisier annotations; the remaining ablations use BSL-1Km.8 for faster iterations, while the final model returns to BSL-1Km.5.Source: Table 3 · p. 11, §5.1 · p. 11
  • I3D significantly outperformed Pose→Sign in per-instance accuracy, at 65.57% versus 49.66%.Pose→Sign uses a 2D ResNet on dynamic pose images extracted with OpenPose, while I3D operates on video.Source: §5.1 · p. 11
  • Using all cues yielded 65.57% accuracy, outperforming face-only I3D at 42.23%, while the table reports that face and body-and-hand cues both contribute.The experiments compare face, body-and-hand, and combined cues for pose-based models, and mask video pixels to isolate regions for I3D.Source: §5.1 · pp. 11–12, Table 4 · p. 12
  • Video pose distillation reached 70.38%, above sign-recognition pretraining at 69.90% and action-recognition pretraining at 69.00%.Other reported initialisations were gesture recognition at 46.93% and random initialisation at 39.80%; pose distillation encourages attention to human-keypoint dynamics.Source: §5.1 · p. 12, Table 5 · p. 12

5.2 Benchmarking sign recognition and sign spotting

  • The BSL-1K benchmarks use an I3D model trained on BSL-1Km.5, pose-distillation initialisation, and random temporal offsets of up to 4 frames.For spotting, 0.6-second windows centred on verified instances are positive; windows near subtitle keywords missed by the visual pipeline are excluded, and mAP over 334 classes is reported at IoU 0.5.Source: §5.2 · pp. 12–13, Table 6 · p. 13
  • Qualitative recognition failures were commonly associated with strong visual similarity in sign production.The examples also identify gesture similarity and mouthing as failure modes.Source: Figure 3 · p. 13, §5.2 · p. 14

5.3 Comparison with the state of the art on ASL benchmarks

  • BSL-1K pretraining boosted ASL performance, surpassing state-of-the-art top-1 accuracy results of 46.82% and 64.71%.The comparison uses 64-frame inputs and reproduces the Kinetics-pretrained I3D baseline because models from were unavailable.Source: §5.3 · p. 14, Table 7 · p. 14

6 Conclusion · APPENDIX

The paper presents BSL-1K, built through visual keyword spotting with weakly aligned subtitles, and reports its use for large-vocabulary sign recognition and ASL benchmark initialization. The appendix provides additional results and details on the pose distillation model and dataset.

6 Conclusion

  • The authors demonstrate visual keyword spotting with weakly aligned subtitles for automatically annotating continuous sign language videos.Source: §6 · p. 14
  • BSL-1K combines a large-scale dataset of co-articulated signs with 3D CNN training for high-performance, large-vocabulary sign recognition.Source: §6 · p. 14
  • The recognition model also provides beneficial initialization for ASL benchmarks.Source: §6 · p. 14
  • The paper provides ablations and baselines for sign recognition and sign spotting.Source: §6 · p. 14
  • A proposed future direction is using the automatic annotations and recognition model for sign language translation.Source: §6 · p. 14

APPENDIX

  • The appendix contains additional results, details about the video pose distillation model, and information about the BSL-1K dataset.Source: APPENDIX · p. 19

A Additional Results

The appendix presents a qualitative analysis and complementary experiments on recognition settings, model ensembling, transfer, and alternative cues.

A Additional Results

  • Additional results cover qualitative analysis and experiments on mouthing search windows, recognition frame counts, test-time mouth masking, part-model ensembles, transfer to co-articulated datasets, and alternative cues.The experiments are organized in Sections A.2–A.7; the passage does not report their findings.Source: A Additional Results · p. 19

A.1 Qualitative analysis … A.4 Masking the mouth region at test time

The appendix presents qualitative examples of mouthing-based annotations and sign predictions alongside ablations of search-window and annotation-frame choices. Its mouth-masking test shows how recognition performance changes when mouth cues are unavailable.

A.2 Size of the search window for visual keyword spotting

  • Eight-second search windows yield the strongest test-set performance and are used in the main-paper experiments; shorter windows risk missing words, while longer ones reduce annotation quality.Source: §A.2 · p. 19, Table A.1 · p. 22

A.1 Qualitative analysis

  • For “happy” and “important,” the top 24 videos by mouthing confidence show visually similar manual features, suggesting mouthing cues can help automatically annotate training samples.Source: §A.1 · p. 19, Figure A.1 · p. 20

A.3 Temporal extent of the automatic annotations

  • Twenty frames before the mouthing peak balance training variation against irrelevant frames, while most experiments use 24 frames and Table 6 uses 20.Mouthing confidence is highest at the end of mouthing, so frames preceding that peak are used for sign-classification training.Source: §A.3 · pp. 19, 22, Table A.2 · p. 22
  • The I3D model takes 16 consecutive frames as input, with the input slice selected randomly during training.Source: §A.3 · p. 22

A.4 Masking the mouth region at test time

  • Masking the mouth only at test time drops performance from 65.57% to 34.74%, while masked-mouth training raises it to 46.75%, indicating reliance on mouth cues.Source: §A.4 · p. 22

A.5 Late fusion of part-specific models.

Late fusion of face-crop and mouth-masked I3D models improves sign recognition over full-frame input, suggesting complementary signing cues are useful, but requires pose estimation and two models.

A.5 Late fusion of part-specific models.

  • 68.55% from late-fusing face-crop and mouth-masked I3D models exceeds 65.57% for full-frame input, suggesting complementary manual and non-manual cues help.The models were trained on the BSL-1Km.8 subset with available pose estimates.Source: §A.5 · pp. 22–23, Table A.4 · p. 23
  • Late fusion requires human-pose computation and training two separate models, so the authors use it only for experiments.Source: §A.5 · p. 23

A.6 Transferring BSL-1K pretrained model to other datasets

BSL-1K pretraining improved transfer to the co-articulated Phoenix and BSL-Corpus datasets, and the authors conclude it provides strong initialization across isolated and co-articulated datasets in ASL, BSL, and DGS.

A.6 Transferring BSL-1K pretrained model to other datasets

  • For isolated ASL datasets MSASL and WLASL, mouth-masked pretraining and retaining weights for words shared with BSL-1K each gave slight improvements.Mouth masking forced the model to attend entirely to manual features; the study also reports co-articulated transfer experiments on Phoenix and BSL-Corpus with and without BSL-1K pretraining.Source: §A.6 · p. 24
  • 5.6 WER improvement over Kinetics pretraining was obtained on Phoenix using CTC because sign-gloss alignments for short-clip I3D training were unavailable.Source: §A.6 · p. 24
  • 24.4% vs 12.8% accuracy with/without BSL-1K pretraining was obtained on a BSL-Corpus subset of 6k annotations covering 966 signs.The authors used an 8:1:1 train/val/test split and found that retaining classification weights for words common to BSL-Corpus and BSL-1K gave small gains over random initialization.Source: §A.6 · p. 24
  • The authors conclude that BSL-1K provides strong initialization for isolated and co-articulated datasets in ASL, BSL, and DGS.Source: §A.6 · p. 24

A.7 Dataset expansion through other cues and additional baselines … C.1 Sign verification tool

Alternative annotation and dataset-expansion strategies did not improve performance, while the appendix details the pose-distillation setup and annotator verification interface. The pose model predicts per-frame human keypoints from video clips.

A.7 Dataset expansion through other cues and additional baselines

  • Without keyword-spotting localization, the subtitle-based method produced extremely noisy annotations and a model that significantly underperformed the main-paper baselines.The method searched subtitles for signs and selected candidate temporal windows by maximizing ROC area for positively and negatively labelled bags.Source: §A.7 · p. 24
  • The exemplar-SVM expansion was difficult to calibrate, despite exploring the original parameters and other settings.The expanded dataset used confident positives from the strongest pretrained model and windows not containing the keyword as negatives.Source: §A.7 · pp. 24–25
  • Tripling the dataset with episode-specific exemplar SVMs trained on confident positives did not boost model performance.The signs were selected by the strongest pretrained model at confidence greater than 0.8; temporal windows excluding the keyword served as negatives.Source: §A.7 · pp. 24–25

B Video Pose Distillation

  • The pose-distillation model uses an I3D backbone on 16-frame, 224×224 clips and projects 1024-dimensional embeddings to 4160 pose-coordinate predictions.The predictions correspond to xy coordinates for 130 OpenPose keypoints [12] trained on COCO, normalized to [0, 1] and penalized with L2 loss.Source: B Video Pose Distillation · p. 25
  • Pose distillation trains on one-minute segments from each BSL-1K training episode for 100 epochs using Lookahead.Training uses minibatches of 32, learning rate 0.1 reduced tenfold after 50 epochs, and weight decay 0.001.Source: B Video Pose Distillation · p. 25

C.1 Sign verification tool

  • The sign-verification tool lets annotators review keyword-spotting candidates at reduced speed and mark each as correct, incorrect, or unsure.The tool was used to verify or reject signs found by the proposed method in the test set.Source: §C.1 · p. 25

C.2 Dataset source material · C.3 Dataset vocabulary

The source-material section lists BBC broadcast shows and their occurrence counts, while the vocabulary section enumerates 1,064 words for BSL-1K.

C.2 Dataset source material

  • Countryfile (266) and Holby City (261) have the highest counts among the listed BBC shows.The source-material inventory also lists Great British Railway Journeys (122) and Great British Menu (110).Source: §C.2 · p. 25
  • The listed source shows include entries with counts as low as 1 for Home Comfort At Christmas and 3 for Our Food Uk 4.Other low-count entries include Two Greedy Italians (4) and Hairy Bikers Bakeation (6).Source: §C.2 · pp. 25–26

C.3 Dataset vocabulary

  • The BSL-1K vocabulary is formed by 1,064 words.Source: §C.3 · p. 26
  • The vocabulary list includes entries such as “abortion,” “afghanistan,” “apple,” “birmingham,” “britain,” and “comfort-.”These are examples from the listed vocabulary, not an exhaustive selection.Source: §C.3 · p. 26
  • The list includes entries such as “dance,” “deaf,” “doctor,” “england,” “family,” “football,” “london,” and “research.”These examples appear among the listed words from “able” through “rocket.”Source: §C.3 · p. 27
  • Later vocabulary entries include “school,” “sign,” “speech,” “train,” “video,” “wheelchair,” and “yesterday.”The supplied list runs from “roll” through “yesterday.”Source: §C.3 · p. 28

Related papers

Loading 2007.12131v2…