PaperlayerView exact revision on arXiv

Source-linked AI summary

SignMatch: Matching Dictionary Signs to Continuous Sign Language Video

Ryan Wong, Youngjoon Jang, Liliane Momeni, Gül Varol, Andrew Zisserman

arXiv:2609.01886v1cs.CV

TL;DR

SignMatch addresses visual matching of dictionary signs to corresponding segments in continuous signing video. It learns a prototype-organized embedding from annotated continuous signing, maps dictionary examples into it, and transfers across datasets, tasks, and sign languages.

  • Problem

    The paper addresses how to visually identify and localise dictionary signs within continuous signing video using handshape and motion relative to the body.

  • Method

    The approach learns a prototype-organized embedding from temporally annotated continuous signing and maps isolated dictionary examples into the same space for matching.

  • Results

    The representation transfers across datasets, tasks, and American, British, Spanish, and unseen sign languages, supporting dictionary retrieval, sign matching, and automatic annotation.

  • Takeaways & Limitations

    Dictionary-driven visual matching can bridge isolated sign resources and continuous sign-language corpora for sign-centric search, navigation, and annotation.

  • Takeaways & Limitations

    The work leaves open how prototype-space geometry relates to handshape, location, movement, and orientation, and how to reduce citation-form–co-articulation differences.

Abstract

from arXiv · show

The objective of this paper is to match dictionary sign videos to corresponding signs in continuous signing videos, where a match is defined by the visual similarity alone - the handshape and motion relative to the body. To achieve this, we learn a prototype-structured sign embedding space from continuous video annotated with signs, where each learnable prototype corresponds to a sign class. Isolated dictionary videos are then mapped into this sign space, enabling the matching between dictionary exemplars and continuous sign instances. This design supports direct dictionary-guided sign matching through embedding similarity and naturally extends to unseen signs using only dictionary exemplars. Experiments on ASL-Citizen dictionary retrieval, ChaLearn OSLWL dictionary-to-continuous sign matching, and using BOBSL's CSLR2 evaluation for automatic sign annotation demonstrate strong generalisation across datasets, tasks and sign languages. Without benchmark-specific supervision, the learned representation transfers effectively across American, British, and Spanish Sign Languages, outperforming prior methods on all three benchmarks. Project page: https://www.robots.ox.ac.uk/~vgg/research/signmatch/

1 Introduction

The paper matches isolated dictionary signs to corresponding segments in continuous signing using visual production, then learns a prototype-structured embedding from richly annotated continuous video. Dictionary examples are mapped into this space, enabling scalable annotation and generalisation across unseen signs, tasks, datasets, and languages.

  • Problem: The task is to identify and temporally localise continuous-video segments that visually correspond to dictionary sign examples.Matching is based on visual production rather than sign meaning, especially handshape and hand movement relative to the body.
  • Applications: Visual sign matching supports dictionary lookup, scalable dataset annotation from isolated examples, sign-based video search, and translation between sign dictionaries.Unseen dictionary signs can be enrolled by embedding them and used to label corresponding locations in continuous signing.
  • Approach: The method learns a sign embedding space from temporally annotated, co-articulated continuous video rather than sparse dictionary exemplars.Multiple sign instances and signers provide variation in execution speed, signer identity, and contextual co-articulation.
  • Approach: Learnable sign prototypes organize the embedding space, while a learned mapping places isolated dictionary videos in the same space for direct matching.This decouples representation learning from dictionary alignment and supports matching through embedding similarity.
  • Generalisation: The learned representation generalises to unseen signs and entirely new sign languages, supporting a language-agnostic representation of sign production.The stated scope includes American, British, and Spanish Sign Languages while remaining agnostic to semantics.

2 Related Work

Prior work progresses from handcrafted temporal matching to deep, weakly supervised representations for spotting signs in continuous video. Dictionary-based spotting extends lexical coverage through isolated exemplars but must overcome the domain gap between canonical dictionary signs and co-articulated continuous signing.

  • Sign spotting from continuous signing: Sign spotting has evolved from handcrafted features with dynamic time warping and hierarchical sequential patterns to deep spatiotemporal representations trained with weakly aligned subtitles [2] [21] [26].
  • Mouthing-based supervision for sign spotting: Mouthing cues provide scalable supervision because signers often mouth corresponding spoken words, supporting sign language translation, recognition, and large-scale annotation mining [14].BSL-1K [2] combines weakly aligned subtitles with visual keyword spotting over mouth movements [25] to recover sparse sign labels from broadcast footage.
  • Dictionary-based sign spotting: Dictionary-based sign spotting uses isolated exemplars as visual queries, but dictionary and continuous signs differ in articulation speed, co-articulation, context, and signer variation.Cross-domain embedding methods [26] address this mismatch.
  • Dictionary retrieval and sign-centric lookup: Dictionary retrieval systems support sign-centric lookup, including webcam retrieval of candidate glosses in Gloss-Finder and isolated sign recognition evaluated with recallat-K in ASL Citizen [9].These methods focus on retrieving dictionary entries from performed signs rather than requiring users to know written translations.
  • Sign language corpora and automatic annotations: The field increasingly relies on continuous corpora with weak subtitle-based supervision, shifting from small carefully annotated datasets such as the BSL Corpus toward larger broadcast- and web-scale collections.

3 The Sign Matching Model

The model matches isolated dictionary signs to co-articulated segments through a two-stage prototype-structured embedding space. It learns sign geometry from annotated continuous signing, aligns dictionary videos into that fixed space, and compares embeddings by cosine similarity.

  • Training: Training uses two stages: prototype-based representation learning from co-articulated signing, followed by dictionary alignment into the learned space.This division supports generalisation to previously unseen dictionary data.
  • Inference: At inference, dictionary and continuous-video embeddings are compared using cosine similarity, with high similarity indicating a likely occurrence of the queried sign.The trained models localise dictionary signs within continuous signing videos.
  • Stage 1: Prototype Space: Stage 1 assigns learnable prototypes to sign classes, encouraging visually similar signs to occupy nearby but discriminative regions without requiring strictly distinct prototypes.Prototype relationships define the structured embedding geometry.
  • Stage 1: Prototype Space: Stage 1 trains embeddings against prototype-induced soft targets using temperature-scaled distributions and detached-target KL divergence.The backbone, projection head, and prototypes are jointly optimised while preserving prototype similarity structure.
  • Stage 2: Dictionary Alignment: Stage 2 freezes the Stage 1 backbone and prototypes, learning only a dictionary projection network to map isolated signs into the preserved co-articulated embedding geometry.Overlapping 32-frame clips with stride 16 capture longer dictionary articulations before aggregation.

4 Training details

The model is trained in two stages: first, prototype-based sign embeddings learn from annotated BSLCorpus segments, then a dictionary projection network aligns isolated dictionary videos to the frozen embedding space. Pseudo-labels from larger multilingual corpora extend supervision and improve transfer from BSL to ASL.

  • Stage 2 training: Stage 2 aligns isolated BSL dictionary videos by encoding sliding 32-frame windows with the frozen Stage 1 backbone and training only a two-layer LSTM dictionary projection network.The backbone and prototypes remain frozen, while the projection network is trained with dictionary examples labeled by Stage 1 id-gloss classes.
  • Pseudo-label training on larger corpora: Phase 2 applies the Phase 1 model to unlabelled continuous corpora, generating dictionary-guided pseudo-labels for BSL and ASL data before retraining on the expanded supervision.The procedure uses BSL SignBank and BSLDict for BSL, ASLDict for ASL, and extends optimization to 125,000 steps for the larger pseudo-labelled corpus.
  • From BSL to ASL: Phase 2 pseudo-label training further improves ASL-Citizen dictionary retrieval after Phase 1 already exceeds backbone features by 3.75 DCG, demonstrating transfer from BSL supervision to unseen ASL.Phase 2 uses pseudo-labels generated on larger multilingual corpora with corresponding dictionary resources; Table 4 reports the iterative-training effect.

5 Applications and Generalisation

The frozen Phase 2 representation is evaluated zero-shot across dictionary retrieval, cross-lingual sign matching, and automatic annotation. It transfers across datasets and languages while remaining effective without benchmark-specific or subtitle-based supervision.

  • Dictionary-to-dictionary retrieval: The RGB model achieves state-of-the-art ASL-Citizen retrieval across DCG, Recall@1, and Recall@5 without training on ASL-Citizen, surpassing directly trained models.Retrieval uses normalized dictionary embeddings and no isolated-sign training or finetuning on ASL-Citizen.
  • Dictionary-based sign matching: The method achieves state-of-the-art official average F1 on ChaLearn OSLWL, outperforming baseline and competition results while localizing Spanish signs without Spanish supervision.Queries use one dictionary exemplar, and detections come from similarity matching over sliding temporal windows.
  • Automatic data labelling: On BOBSL, dictionary-based retrieval significantly improves WER, mIoU, and mean F1 over prior annotation approaches without mouthings, subtitle alignment, or translation supervision.The comparison uses the CSLR2 protocol and includes dictionary- and subtitle-based annotation approaches.
  • Automatic data labelling: The method outperforms the multi-stage framework of [26] across all metrics using learned dictionary embeddings and sliding-window matching without subtitle-driven optimisation.Prior work combines sparse annotations, subtitles, dictionary supervision, MIL, and NCE, whereas this approach transfers from isolated to co-articulated signing more simply.
  • Automatic data labelling: Removing subtitle-based dictionary filtering changes performance only marginally for the proposed method, unlike prior approaches that rely heavily on subtitle information.Subtitle filtering restricts retrieval to dictionary entries matching normalized or lemmatized subtitle words; no-subtitle retrieval searches the full candidate set.
  • Cross-lingual generalisation: A BSL-only model outperforms prior ASL-specific baselines on ASL-Citizen, while adding ASL training improves retrieval; embeddings also expose cross-lingual visual similarities and potential faux amis.The cross-lingual retrieval patterns include visually similar signs with different meanings and signs with related or equivalent meanings.

6 Conclusions

The paper introduces sign matching as a purely visual task that localises dictionary signs in continuous co-articulated signing video without an intermediate textual representation. Its representation transfers across datasets, tasks, and sign languages while supporting dictionary-guided automatic annotation.

  • 6 Conclusions: Sign matching localises dictionary signs in continuous co-articulated signing video using visual similarity alone, without an intermediate textual representation.The approach decouples representation learning from dictionary alignment by learning a visually organised embedding space from continuous sign corpora, then aligning isolated dictionary exemplars to it.
  • 6 Conclusions: The learned representation transfers across ASL-Citizen retrieval, ChaLearn OSLWL sign matching, and BOBSL automatic annotation evaluated using CSLR2 labels.It generalises beyond the initial supervised vocabulary to new datasets, tasks, and sign languages.
  • 6 Conclusions: The model generalises from BSL and ASL supervision to Spanish Sign Language sign matching while supporting large-scale automatic annotation through dictionary-guided retrieval.These capabilities demonstrate transfer beyond the initial supervised vocabulary and across sign languages.

SignMatch - Supplementary Material

The supplementary material details the pseudo-labelling pipeline, qualitative dictionary-matching examples, subtitle-guided filtering, target-type analysis, and limitations with future directions.

  • A. Pseudo-labelling: Section A describes the pseudo-labelling pipeline used to scale training to large continuous sign language corpora.
  • B. Qualitative analysis: Section B presents qualitative examples of dictionary matching within continuous signing and analyzes subtitle-guided dictionary filtering.
  • C. Training targets: Section C evaluates soft versus hard training targets and analyzes their effects on the learned prototype space.
  • D. Limitations and future directions: Section D discusses the method’s limitations and future directions.

A. Scaling Sign Matching for Automatic Sign Annotation

The approach scales sign matching from manually aligned BSLCorpus examples by automatically discovering dictionary-labeled sign instances in large subtitle-aligned corpora. Subtitle-filtered dictionary retrieval produces pseudo-labels that expand supervision for Phase 2 retraining.

  • Initial Supervision: Manually aligned BSLCorpus sign instances paired with BSL SignBank identifiers and temporal boundaries provide initial supervision for learning the shared embedding space.These paired dictionary and continuous-signing examples support the paper’s two-stage training procedure.
  • Automatic Scaling: The model scales training by discovering additional sign instances in BOBSL and BSL and ASL portions of YouTube-SL-25 using BSL SignBank, BSLDict, and ASLDict entries.This reduces reliance on exclusively manual annotations and supplies dictionary queries across BSL and ASL data.
  • Retraining: The resulting temporally localized sign segments, dictionary labels, and confidence scores are combined with manual data for Phase 2 retraining.The expanded supervision is used together with dictionary videos from BSL SignBank, BSLDict, and ASLDict.

B. Qualitative Analysis

Qualitative examples show that SignMatch localizes dictionary labels within continuous signing and retrieves visually similar candidate signs. These results also reveal ambiguity between signs with similar production-level features, motivating additional linguistic and articulatory cues.

  • Qualitative retrieval: For a query sign, the correct dictionary sign BIRD is ranked first and localized, while the visually similar SWINDON also receives a high similarity score.Both candidates match the same temporal region, showing that the representation can surface multiple plausible signs sharing production-level features.
  • Temporal label matching: The model localizes dictionary labels to temporal intervals in continuous signing and illustrates each matched segment with representative keyframes.Figure 6 includes the PT:PRO3SG label, whose pointing-sign meaning is clarified by the accompanying subtitle.
  • Limitations and future cues: Distinguishing visually similar signs may improve by incorporating linguistic context, fingerspelling, and mouthing information.These additional cues are proposed to refine retrieval performance beyond visual production features alone.

Subtitle Filtering for Automatic Annotation

Subtitle-guided dictionary filtering has mixed effects on automatic sign annotation: it can suppress false positives and resolve visual ambiguities, but it can also remove valid candidates and correct detections. These effects arise because subtitles may be temporally misaligned and need not provide a one-to-one lexical transcription of signing.

  • Qualitative comparison: The retrieval model itself is often the dominant factor, and subtitle filtering can produce similar predictions to unfiltered retrieval for some instances.The qualitative examples show that filtering effects vary substantially across instances.
  • Limitations: Subtitle filtering behaves inconsistently because subtitles may be temporally misaligned with signing and may not lexically transcribe the signed utterance one-to-one.Signers may omit words, add concepts, or use different lexical choices; subtitle text also reflects spoken-language grammar and vocabulary.
  • Qualitative comparison: Subtitle constraints suppress correct detections in several examples, whereas another example shows filtering recovering a more accurate prediction.Examples 1–4 show reduced annotation coverage; Example 5 demonstrates a beneficial constraint.
  • Qualitative comparison: Filtering can resolve ambiguities that visual evidence alone cannot distinguish, including the highly similar glosses month and minute.The subtitle constraint suppresses a plausible but incorrect visual match in the illustrated case.
  • Qualitative comparison: Subtitle filtering can suppress false-positive matches and resolve visually ambiguous glosses, but it can also suppress correct detections and valid dictionary candidates.Table 10 shows filtering helping recover a more accurate prediction and distinguish visually similar glosses such as month and minute, while other examples lose correct detections.

C. Ablation: Training with Soft vs. Hard Targets

Soft targets outperform one-hot cross-entropy targets across BOBSL CSLR2 metrics and produce a more discriminative prototype space with smaller groups of visually similar signs.

  • Ablation: Training with Soft vs. Hard Targets: The ablation replaces prototype-induced soft targets with one-hot targets in both training stages, substituting cross-entropy for KL divergence.All remaining training settings are unchanged.
  • Ablation: Training with Soft vs. Hard Targets: Soft-target training performs better across all BOBSL CSLR2 metrics than hard cross-entropy targets under otherwise identical settings.The comparison uses the pose backbone without subtitle filtering.
  • Prototype space analysis: At cosine similarity 0.9, cross-entropy groups prototypes from over one hundred sign classes together, whereas soft targets form smaller groups of visually similar signs.This indicates that the soft-target objective preserves more visually meaningful prototype structure.

D. Limitations and Future Directions

The learned representation captures visual relationships between signs across datasets and sign languages, motivating future work to characterize its geometry and address citation-form versus co-articulated signing. Proposed directions include phonological analysis of prototype neighborhoods and data-level synthesis or augmentation of faster dictionary-sign variants.

  • Transfer across datasets and sign languages, together with the learned prototype-space structure, indicates that the representation captures visual relationships between signs.
  • Future work could characterize prototype-space geometry using production parameters such as handshape, location, movement, and orientation, including comparisons with phonological annotations.
  • Another direction is reducing citation-form and co-articulation differences by synthesizing or augmenting faster dictionary-sign variants alongside embedding-level alignment.
Loading 2609.01886v1…