Source-linked AI summary
BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues
Samuel Albanie, Gül Varol, Liliane Momeni, Triantafyllos Afouras, Joon Son Chung, Neil Fox, Andrew Zisserman
TL;DR
Continuous sign recognition is hindered by scarce large-scale annotations and the visual difficulty of co-articulated signs. The paper uses weakly aligned subtitles and mouthing-based visual keyword spotting to build BSL-1K, then trains recognition models that exceed state of the art on MSASL and WLASL while introducing new evaluation sets and baselines.
Problem
Continuous sign recognition needs large-scale annotated data, but co-articulation complicates recognition and sign annotation is difficult to scale.
Method
The paper uses subtitle-derived temporal windows and visual keyword spotting of signer mouthings to automatically localise signs in broadcast footage.
Results
The resulting BSL-1K dataset supports strong co-articulated BSL recognition models that exceed the state of the art on MSASL and WLASL.
Takeaways & Limitations
BSL-1K also provides pretraining for other sign-language benchmarks and accompanies new evaluation sets and baselines for recognition and spotting.
Takeaways & Limitations
The approach assumes mouthing provides a sufficiently frequent localisation cue, while recognition failures can arise from strong visual similarity between signs.
Abstract
from arXiv · showhide
Recent progress in fine-grained gesture and action classification, and machine translation, point to the possibility of automated sign language recognition becoming a reality. A key stumbling block in making progress towards this goal is a lack of appropriate training data, stemming from the high complexity of sign annotation and a limited supply of qualified annotators. In this work, we introduce a new scalable approach to data collection for sign recognition in continuous videos. We make use of weakly-aligned subtitles for broadcast footage together with a keyword spotting method to automatically localise sign-instances for a vocabulary of 1,000 signs in 1,000 hours of video. We make the following contributions: (1) We show how to use mouthing cues from signers to obtain high-quality annotations from video data - the result is the BSL-1K dataset, a collection of British Sign Language (BSL) signs of unprecedented scale; (2) We show that we can use BSL-1K to train strong sign recognition models for co-articulated signs in BSL and that these models additionally form excellent pretraining for other sign languages and benchmarks - we exceed the state of the art on both the MSASL and WLASL benchmarks. Finally, (3) we propose new large-scale evaluation sets for the tasks of sign recognition and sign spotting and provide baselines which we hope will serve to stimulate research in this area.
1 Introduction
Continuous sign recognition requires models that handle co-articulated signs, but progress is limited by scarce large-scale annotated data. The paper addresses this gap by using mouthing cues and weakly aligned subtitles to build BSL-1K and demonstrate its value for recognition.
- Continuous signing is difficult because co-articulation makes signs visually different from isolated dictionary examples.
- The paper designs a scalable method for generating large continuous-signing datasets from broadcast video and subtitles.
- Visual keyword spotting aligns subtitle words with signer mouthings to localise signs despite subtitle misalignment and differing word order.
- BSL-1K provides high-quality annotations for a 1K-sign British Sign Language vocabulary.
- The resulting models exceed the state of the art on both MSASL and WLASL benchmarks.
- The paper also introduces evaluation datasets and baselines for sign recognition and sign spotting.
2 Related Work
Prior sign-language datasets are constrained by limited scale, vocabulary, signer diversity, or laboratory settings. This work combines weakly aligned subtitles with visual keyword spotting and scales the approach to 1,000 hours of signing.
- Existing datasets cover multiple sign languages, but BSL resources remain comparatively scarce.
- Earlier datasets were limited in video volume, vocabulary size, or number of signers, while newer isolated-sign benchmarks reached 1K and 2K signs.
- Continuous datasets have often used laboratory recordings, restricted vocabularies, or narrow domains such as weather broadcasts.
- Prior recognition pipelines used hand, body, face, and mouth features with HMM or LSTM sequence models.
- The paper combines weakly aligned subtitles and mouthing-based localisation, a combination the authors identify as previously unreported.
- The approach scales collection to 1,000 hours of signing rather than the 30 hours considered in closely related work.
3 Learning Sign Recognition with Automatic Labels
BSL-1K is constructed from broadcast BSL footage by using subtitle occurrences to propose candidate windows and visual keyword spotting to localise signs precisely. The resulting annotations are filtered, split by signer, and partly human-validated.
- The collection pipeline sources probable sign clips from subtitles, precisely localises signs, and annotates a 1K-sign vocabulary.
- The source comprises 1,412 BBC episodes containing 1,060 hours of continuous BSL signing by forty professional interpreters.
- Approximately 9.5 million subtitle words yield an initial vocabulary of 1,350 words linked to BSL dictionaries.
- Subtitle-derived windows are noisy because words may not be signed and subtitle timing may not align with sign production.
- Mouthing is used as a localisation cue because visual keyword spotting searches for a known token within a short candidate window rather than performing general lip reading.
- Padding subtitle windows by four seconds on each side was empirically selected as a good choice.
- Approximately 280k localised signs are obtained from 2.4 million candidate subtitles, followed by signer-based splitting and vocabulary filtering.
- The final vocabulary contains 1,064 words, and an initial human assessment marked 70% of 1k sampled signs correct.
4 Models and Implementation Details
The paper uses visual keyword spotting for annotation, I3D-based spatio-temporal networks for sign recognition, and pose distillation to improve representation learning. Training and inference use standard video-classification procedures with multi-frame inputs.
- The modelling pipeline includes visual keyword spotting, sign recognition and spotting architectures, and an initialization method for recognition.
- The keyword spotter combines face-cropped visual features with a fixed-length keyword representation, BiLSTMs, and a sigmoid output.
- The sign recognizer uses an I3D spatio-temporal convolutional network that outputs probabilities over sign categories.
- Training uses cross-entropy with SGD and momentum, scheduled learning-rate reductions, 50 epochs, and video augmentations.
- Inference centre-crops videos, applies a sliding window with stride 8 frames, and averages classification scores.
- Pose distillation trains a video embedding to regress 130 OpenPose keypoints across 16-frame inputs, encouraging dynamic gesture information.
5 Experiments
The experiments examine cue selection, annotation-threshold trade-offs, pose distillation, benchmarking, and transfer from BSL-1K to ASL benchmarks.
- Ablations: 70.61% accuracy was obtained on the manually validated test set when training used the larger BSL-1Km.5 set despite noisier annotations.BSL-1Km.5 uses a 0.5 mouthing threshold, whereas BSL-1Km.8 uses 0.8; the larger set provided the best performance.
- Ablations: 65.57% accuracy using all cues, including body and hands, outperformed the face-only I3D result of 42.23%.The comparison evaluates individual visual regions and their combination on a BSL-1Km.8 subset.
- Ablations: 70.38% was achieved with video pose distillation, exceeding sign-recognition pretraining from sign recognition at 69.90% and action recognition at 69.00%.The pose-distillation network uses 16-frame inputs to learn body-motion-aware cues before fine-tuning.
- Benchmarking: The strongest model combines BSL-1Km.5 training, pose-distillation initialisation, and random temporal offsets of up to 4 frames.It was used to establish baselines for both sign recognition and sign spotting on BSL-1K.
- Benchmarking: Sign spotting locates instances of a target sign in untrimmed footage, with 0.6-second positive windows around verified instances and exclusions around missed subtitle keywords.The evaluation covers 334 sign categories and approximately one positive instance every 1.5 hours of continuous footage.
- Transfer to ASL: 46.82% and 64.71% top-1 accuracy were reported on ASL benchmarks after BSL-1K pretraining, outperforming the state-of-the-art results.The transfer evaluates BSL-pretrained I3D features on MSASL and WLASL.
6 Conclusion
The paper concludes that visual keyword spotting can automatically annotate continuous signing at scale, enabling high-performance recognition and useful initialization for ASL benchmarks.
- Visual keyword spotting with weakly-aligned subtitles automatically annotates continuous sign-language videos.
- BSL-1K couples a large-scale co-articulated-sign dataset with 3D CNN training for high-performance large-vocabulary recognition.
- BSL-1K-pretrained models provide beneficial initialization for ASL benchmarks.
- The paper provides ablations and baselines for sign recognition and sign spotting, with sign-language translation identified as a potential future direction.
A Additional Results
The supplementary section provides complementary experiments, pose-distillation details, and additional information about the BSL-1K dataset.
- Additional results cover qualitative analysis, mouthing search windows, frame counts, mouth masking, part-specific ensembles, dataset transfer, and alternative cues.
- The supplement also describes the video pose distillation model and the BSL-1K dataset.
A.1 Qualitative analysis
The qualitative analyses examine temporal search and clip-length choices for mouthing-based annotation, alongside representative localisation and recognition examples. Masking experiments further indicate substantial reliance on mouth cues.
- Temporal window selection: 8-second search windows provided the strongest test-set performance for annotations generated by visual keyword spotting.Shorter windows increased missed words, while excessively long windows reduced annotation quality.
- Clip duration: 20 frames before the mouthing peak offered a compromise between training variation and excluding irrelevant frames.The experiments otherwise used 24 frames, while the combined ablation model used 20.
- Qualitative examples: Qualitative samples show automatically localised training signs for “happy” and “important” and correctly classified test examples for “orange” and “business”.The top 24 videos were visualised for each keyword or sign according to mouthing or classification confidence.
- Mouth masking: Masking the mouth at test time reduced performance from 65.57% to 34.74%, while mouth-masked training improved it to 46.75%.The results show both strong reliance on mouth cues and partial recovery when the model is trained to use other regions.
A.5 Late fusion of part-specific models.
Late fusion combines models specialised for mouth-specific and body-specific cues. This complementary arrangement outperformed a single full-frame model, though it required additional pose computation and separate model training.
- A.5 Late fusion of part-specific models: 68.55% from late fusion of face-crop and mouth-masked I3D models exceeded the 65.57% full-frame result.The ensemble averaged classification scores from models specialising in different body regions.
- A.5 Late fusion of part-specific models: The part-specific ensemble combines a mouth-focused model with a body-focused model to separate complementary manual and non-manual signing cues.Table A.4 describes the models as trained on a BSL-1K subset with available pose estimates.
- A.5 Late fusion of part-specific models: The late-fusion procedure adds complexity because it requires human-pose computation and training two separate models.The authors therefore use it only for experimental purposes.
A.6 Transferring BSL-1K pretrained model to other datasets
BSL-1K pretraining transfers to isolated and co-articulated datasets across BSL, ASL, and DGS. The reported experiments show gains on Phoenix and BSL-Corpus, as well as state-of-the-art results on MSASL and WLASL.
- A.6 Transferring BSL-1K pretrained model to other datasets: BSL-1K pretraining improved Phoenix performance by 5.6 WER over Kinetics pretraining.Because Phoenix lacks sign-gloss alignment for short I3D clips, the authors adapted I3D with a simple CTC loss.
- A.6 Transferring BSL-1K pretrained model to other datasets: 24.4% versus 12.8% accuracy was obtained on BSL-Corpus with versus without BSL-1K pretraining.The split used 6k annotations covering 966 signs, with an 8:1:1 train/validation/test ratio.
- A.6 Transferring BSL-1K pretrained model to other datasets: The authors conclude that BSL-1K provides strong initialisation for co-articulated and isolated datasets across ASL, BSL, and DGS.For ASL transfer, mouth-masking pretraining was used to emphasise manual features, and shared sign classes retained classification weights where applicable.
A.7 Dataset expansion through other cues and additional baselines
Additional dataset-expansion strategies were tested against the keyword-spotting pipeline. Subtitle-based labelling without keyword-spotting localisation was noisy and underperformed, while tripling the dataset with exemplar SVMs did not improve performance.
- A.7 Dataset expansion through other cues and additional baselines: Without keyword-spotting localisation, subtitle-based annotations were extremely noisy and the resulting model underperformed all main-paper baselines.The alternative searched subtitles and selected temporal windows using positive and negative bags around target words.
- A.7 Dataset expansion through other cues and additional baselines: Tripling the dataset with exemplar-SVM expansion produced no performance boost.The authors selected confident positives above 0.8 and used windows without the keyword as negatives, but found SVM confidence calibration difficult.
- A.7 Dataset expansion through other cues and additional baselines: The pose-distillation implementation uses an I3D backbone with 16-frame, 224×224-pixel inputs and projects embeddings to per-frame pose predictions.The model replaces the original classification head with a linear projection for pose regression.
- A.7 Dataset expansion through other cues and additional baselines: Annotators verified or rejected keyword-spotting detections using a tool that supports reduced-speed viewing and correct, incorrect, or unsure labels.This verification procedure was applied to signs found in the test set.
C.2 Dataset source material
The dataset source material comprises BBC broadcast television programmes with recorded occurrence counts, alongside a 1,064-word BSL-1K vocabulary and a manual whole-sign verification tool.
- BBC broadcast TV programmes supplied the source material, with occurrences recorded for each listed show.The listed programmes include Countryfile, Holby City, and multiple cooking and travel shows.
- The source list spans programmes with occurrence counts ranging from 1 to 266.Home Comfort At Christmas has 1 occurrence, while Countryfile has 266.
- Manual annotation used a Whole-Sign Verification Tool.The appendix figure presents a screenshot of this tool.
- The BSL-1K vocabulary contains 1,064 words.The supplied vocabulary list includes terms such as abortion, communicate, research, and sign.