PaperlayerView exact revision on arXiv

Source-linked AI summary

Watch, read and lookup: learning to spot signs from multiple supervisors

Liliane Momeni, Gül Varol, Samuel Albanie, Triantafyllos Afouras, Andrew Zisserman

arXiv:2010.04002v1cs.CV

TL;DR

The paper asks how to spot isolated dictionary signs within continuous, co-articulated signing despite sparse labels, limited vocabulary, and domain differences. It combines sparse annotations, subtitles, and dictionary videos in a multiple-instance contrastive framework, achieving state-of-the-art BSL-1K performance and introducing BslDict.

  • Problem

    Sign spotting must identify and localise isolated signs in continuous signing, but available corpora have sparse labels and dictionaries differ substantially from co-articulated footage.

  • Method

    The framework combines sparse annotations, subtitle-based weak supervision, and dictionary lookup using Multiple Instance Learning and Noise Contrastive Estimation.

  • Results

    The model achieves 0.170 mAP on BSL-1K, outperforming the previous state-of-the-art 0.160 mAP.

  • Takeaways & Limitations

    BslDict and the unified framework support low-shot sign spotting, novel-sign evaluation, and applications such as mining dense annotations.

  • Takeaways & Limitations

    Dictionary-based spotting is constrained by isolated-versus-co-articulated domain differences, slower dictionary performances, few examples per sign, and multiple variants per keyword.

Abstract

from arXiv · show

The focus of this work is sign spotting - given a video of an isolated sign, our task is to identify whether and where it has been signed in a continuous, co-articulated sign language video. To achieve this sign spotting task, we train a model using multiple types of available supervision by: (1) watching existing sparsely labelled footage; (2) reading associated subtitles (readily available translations of the signed content) which provide additional weak-supervision; (3) looking up words (for which no co-articulated labelled examples are available) in visual sign language dictionaries to enable novel sign spotting. These three tasks are integrated into a unified learning framework using the principles of Noise Contrastive Estimation and Multiple Instance Learning. We validate the effectiveness of our approach on low-shot sign spotting benchmarks. In addition, we contribute a machine-readable British Sign Language (BSL) dictionary dataset of isolated signs, BSLDict, to facilitate study of this task. The dataset, models and code are available at our project page.

1 Introduction

The paper targets sign spotting across continuous signing using isolated dictionary queries, despite sparse labels, limited vocabulary, and a severe domain gap. It combines sparse annotations, subtitles, and dictionary examples in a unified learning framework.

  • 1 Introduction: Large continuous-signing corpora have sparse labels and limited vocabulary, unlike the densely labelled datasets that enabled spoken-language keyword spotting.This limits direct transfer of keyword-spotting successes to sign language.
  • 1 Introduction: Sign spotting identifies whether and where an isolated dictionary sign appears in continuous, co-articulated signing.The task has practical applications including video indexing, linguistic study, student feedback, accessibility, and dataset construction.
  • 1 Introduction: Dictionary examples broaden vocabulary but differ from target signing in articulation, speed, shot count, and sign variants, so direct training performs poorly.These differences create a substantial domain shift between isolated dictionary signs and continuous signing.
  • 1 Introduction: The proposed framework combines sparse annotations, subtitle-based weak supervision, and dictionary lookup through Multiple Instance Learning and Noise Contrastive Estimation.The goal is a representation invariant across isolated and co-articulated signing.
  • 1 Introduction: The paper contributes BslDict, a machine-readable BSL isolated-sign dictionary dataset, and reports state-of-the-art performance on the BSL-1K sign-spotting benchmark.It also evaluates a low-shot co-articulated sign-spotting setting and studies mining and sign-language false friends.

2 Related Work

The work builds on sign recognition, spotting, datasets, multiple-instance learning, domain adaptation, and low-shot localization. Its distinction is combining weak supervision, dictionary videos, sparse labels, and explicit handling of sign variants for co-articulated signing.

  • Sign language spotting: Earlier sign-spotting methods used handcrafted features and full supervision, whereas this work learns representations from multiple weak supervisory cues.The paper uses an I3D architecture as a foundational component.
  • Sign language spotting: Unlike related domain-adaptation and alignment work, the paper targets few-shot dictionary examples and labeling co-articulated signing rather than isolated-sign recognition.It also explicitly accounts for dialect variations in dictionary examples.
  • Sign language datasets: BslDict complements BSL-1K and other BSL datasets by providing a handful of examples per sign across a 9K vocabulary rather than a 1K vocabulary.Its broad lexicon supports sign-spotting applications beyond the coverage of existing corpora.
  • Multiple instance learning: The framework extends Multiple Instance Learning by incorporating dictionaries, sparse labels, subtitle supervision, sign variants, and more than two orders of magnitude more weakly supervised data than earlier studies.This differentiates the method from prior subtitle-association approaches.
  • Low-shot action localization: The framework shares temporal-ordering ideas with low-shot action localization but addresses sign spotting rather than general semantic video matching.The paper emphasizes the narrower sign-specific objective.

3 Learning Sign Spotting Embeddings from Multiple Supervisors

The method learns sign embeddings from sparse continuous-sign labels, subtitles, and isolated dictionary videos. Multiple-instance contrastive learning handles ambiguous alignments and sign variants, while the implementation jointly uses BSL-1K and BslDict data.

  • 3 Learning Sign Spotting Embeddings from Multiple Supervisors: The task assumes subtitled continuous videos, sparse localized sign annotations, and a curated dictionary of isolated signing instances.The model learns a representation that maps video segments to vectors for sign spotting.
  • 3.1 Integrating Cues through Multiple Instance Learning: The representation must bridge isolated-versus-co-articulated distribution shift, multiple variants, weak subtitle alignment, and sparse localization.These conditions prevent straightforward fully supervised positive-pair construction.
  • 3.1 Integrating Cues through Multiple Instance Learning: Multiple Instance Learning forms positive bags expected to contain at least one matching segment-dictionary pair and negative bags expected to contain none.This accommodates uncertain dictionary variants and subtitle locations.
  • 3.1 Integrating Cues through Multiple Instance Learning: Watch and Lookup uses annotated foreground segments with corresponding dictionary examples, while Watch, Read and Lookup adds subtitle-derived background bags to extend vocabulary coverage.Negative bags use excluded matches and non-overlapping words across batch items.
  • 3.1 Integrating Cues through Multiple Instance Learning: The loss generalizes InfoNCE using cosine similarity, positive and negative bags, and a temperature hyperparameter.The objective is a Noise Contrastive Estimation formulation for multiple-instance training.
  • 3.2 Implementation details: The embedding architecture uses an I3D trunk and a three-layer MLP, projecting 1024-dimensional trunk outputs to 256-dimensional sign embeddings.The trunk consumes 16-frame, 224×224 clips and is initialized through joint pretraining on BSL-1K and BslDict.
  • 3.2 Implementation details: Training samples annotated foreground clips, ten additional background clips, and dictionary entries for subtitle words in each minibatch.Minibatches contain 128 continuous-signing sequences with corresponding dictionary entries.
  • 3.2 Implementation details: Table 1 contrasts BSL-1K’s large number of annotated signs with BslDict’s large vocabulary size and reports videos, vocabulary, and signer counts.The datasets serve complementary supervision roles.

4 Experiments

The experiments evaluate Watch-Read-Lookup for low-shot sign spotting using BSL-1K and the contributed BslDict dataset, then examine its applications in variant identification, dense annotation, and cross-language similarity discovery.

  • 4.1 Datasets: The evaluation uses BSL-1K, whose 1,064-sign vocabulary is sparsely annotated over more than 1,000 hours, with R@5 and mAP measuring retrieval and temporal localization.The unseen protocol trains on 800 signs and tests on the remaining 264, while the full dictionary vocabulary remains available.
  • 4.1 Datasets: BslDict contributes 14,210 isolated-sign clips covering 9,283 BSL signs and 148 signers, with 1–10 examples per sign in the 1,064-sign subset.The dictionary supplies broad lexical coverage despite domain differences from continuous signing.
  • 4.3 Ablation Study: Using the full 9k BslDict vocabulary significantly improves unseen-sign results when continuous annotations cover only 800 words.
  • 4.3 Ablation Study: Increasing the number of negative pairs by increasing batch size improves performance before saturating; the final Watch-Read-Lookup model uses batch size 128 because of high memory requirements.
  • 4.3 Ablation Study: Watch-Read-Lookup outperforms classification, single-instance InfoNCE [50], and Watch-Lookup baselines in both seen and unseen protocols.The framework combines multiple positive dictionary candidates with subtitle-based supervision in a multiple-instance formulation.
  • 4.3 Ablation Study: 0.170 mAP surpasses the previous state of the art, 0.160 mAP, on the BSL-1K sign spotting benchmark.
  • 4.4 Applications: The method identifies matching sign variants, localizes subtitle-query signs with 83.08% accuracy, and retrieves visually similar BSL and ASL signs, including “faux amis.”Dense annotation queries only words appearing in subtitles, while cross-language retrieval compares BslDict with WLASL.

5 Conclusions

The paper concludes that visual sign dictionaries and multiple supervisory signals support sign spotting in continuous signing. It also identifies applications for scaling automatic annotation of sign language datasets.

  • 5 Conclusions: The approach combines sparse continuous-sign annotations, subtitles, and dictionary examples through multiple-instance contrastive learning.
  • 5 Conclusions: The authors identify sign variant identification, dense annotation, and cross-language “faux amis” discovery as potential applications for scaling sign language dataset annotation.
  • APPENDIX: The appendix supplies additional qualitative and experimental results plus detailed explanations of Watch-Read-Lookup training.

A Qualitative Results

The project webpage provides qualitative demonstrations of sign spotting and its proposed applications.

  • A Qualitative Results: The video illustrates sign spotting, sign variant identification, annotation densification, and “faux amis” identification between languages.

B Additional Experiments

The appendix reports complementary experiments addressing variability, sampling, architecture, and inference choices.

  • B Additional Experiments: Additional experiments examine random-seed variance, class balancing, domain-specific layers, language-aware negative sampling, test-time stride, mouthing thresholds, and trunk architectures.

B.1 Variance of results

The supplementary experiments assess result variance, sampling choices, domain-specific layers, synonym-aware negatives, and temporal stride. Results support stable performance and favor simple shared components but require a one-frame stride for precise spotting.

  • B.1 Variance of results: Three-seed repetitions produce results consistent with the main paper’s experiments.Means and standard deviations are reported for the repeated Table 2 and Table 3 experiments.
  • B.2 Class-balanced sampling: Class-balanced sampling does not significantly affect performance across tested batch sizes.The strategy maximizes negative pairs by including one labelled sample per word, while limiting batch size to the number of sign classes.
  • B.3 Domain-specific layers: Domain-specific MLP layers provide no observed gain, so the method uses one shared MLP for continuous and dictionary domains.The comparison addresses the speed and co-articulation differences between continuous signing and isolated dictionary signs.
  • B.4 Language-aware negative sampling: Discarding synonym pairs yields only marginal gains, so synonym-aware negative sampling is omitted for simplicity.The default setup associates each sign with a single word, which can incorrectly create negatives for synonymous signs such as “happy” and “content”.
  • B.5 Effect of the sliding window stride: A one-frame sliding-window stride is critical for precise sign spotting because eight-frame steps can skip discriminative moments.The window size is 16 frames, while typical co-articulated signs last 7–13 frames and may be shorter.

B.6 Mouthing confidence threshold at training

The study tests how confidence thresholds on automatically mouthing-derived BSL-1K annotations affect training-set construction. The lowest tested threshold performs best, apparently benefiting from the larger available training set.

  • B.6 Mouthing confidence threshold at training: A 0.5 mouthing-confidence threshold performs best, despite producing noisier annotations.Lower thresholds enlarge the training set, and the result agrees with the prior BSL-1K conclusion.

B.7 Trunk network architecture: S3D vs I3D

The supplementary comparison evaluates S3D against the I3D trunk used throughout the experiments. S3D provides no improvement and overfits more, so I3D remains the feature-extraction backbone.

  • B.7 Trunk network architecture: S3D vs I3D: S3D does not improve sign-recognition performance over I3D and overfits the training set more, so I3D is retained.The comparison uses a 1064-way BSL-1K classification setup with the same hyperparameters for both architectures.

C Training Details

This section specifies the training implementation through architectural details, positive/negative bag sampling, and experimental infrastructure.

  • C Training Details: The training-details section covers architecture, positive/negative bag sampling, and the infrastructure used for the main-paper experiments.

C.1 Architectural details

The framework combines a frozen I3D feature extractor with a trainable MLP and uses Multiple Instance Learning to form positive and negative bags from sparse labels, subtitles, and dictionary examples.

  • C.1 Architectural details: The model uses a two-stage embedding architecture: an I3D trunk followed by a three-layer MLP, with the trunk frozen while the MLP is trained using the proposed loss.The same embedding layers and parameters process dictionary videos and continuous signing videos.
  • C.1 Architectural details: Re-initializing batch-normalization layers raises 1064-way dictionary training accuracy from 48.09% to 78.94%, whereas domain-specific batch normalization reaches 62.73%.The comparison is reported for per-instance top-1 accuracy on the training dictionary videos.
  • C.2 Positive/Negative bag sampling formulations: Watch-Lookup forms positive bags by pairing labelled continuous segments with dictionary sign variations, or dictionary entries with their corresponding labelled segments.Negative bags pair labelled segments or dictionary sets with examples associated with different words; each positive bag assumes at least one matching sign variation.
  • C.2 Positive/Negative bag sampling formulations: Watch-Read-Lookup extends Watch-Lookup with subtitle words, treating the labelled word as foreground and unlabelled subtitle words as background to create additional positive bags.Background-anchored bags pair continuous background segments with matching dictionary entries, while dictionary-anchored bags pair dictionary entries with continuous segments whose subtitles contain the background word.
  • C.2 Positive/Negative bag sampling formulations: The full negative-bag construction anchors on foreground segments, foreground dictionary entries, background continuous segments, and background dictionary entries, then combines these sets for MIL-NCE training.The supplementary material gives precise set definitions and implementation details for the bag formulations.
Loading 2010.04002v1…