Transferring Cross-domain Knowledge for Video Sign Language Recognition

Motivation

• WSLR datasets remain small because expert annotation is required, with existing datasets averaging only 10 to 50 instances per word.Abundant subtitled sign-news videos may provide additional data, but they are not directly usable for supervised WSLR.Source: §1 · p. 1

Challenges

• Transferring news-sign knowledge is difficult because subtitles provide weak, noisy labels and news signs span 9-16 frames versus about 60 frames in isolated-sign videos.The mismatch concerns both annotation quality and gesture speed.Source: §1 · pp. 1–2

Method

• The method extracts news signs with a base WSLR model and jointly trains a classifier on news and isolated signs to coarsely align their domains.It then stores aligned news-sign class centroids in prototypical memory.Source: §1 · p. 2
• Prototypical memory learns domain-invariant descriptors from shared visual concepts, while memory-augmented temporal attention emphasizes class-specific features and suppresses common gestures.The design targets hand gestures, body movements, and other concepts shared across domains.Source: §1 · p. 2

Results

• 12% on WLASL and 6% on MSASL: the method outperforms state-of-the-art WSLR methods in recognition accuracy and achieves 28.1 AP@0.5 for automatic sign-word localization.These results cover both recognition and localization from sign-news sentences.Source: §1 · p. 2

Positioning

• The paper frames its contribution as semi-supervised learning from weakly and noisily labeled data.Sign-news subtitles provide the weak supervision considered by this line of work.Source: §2 · p. 2

WSLR methods

• Earlier WSLR methods use hand-crafted features, HMM or DTW temporal modeling, 2D CNNs with recurrent networks, 3D CNNs, and body keypoints.These approaches represent successive strategies for spatial, temporal, or spatiotemporal modeling.Source: §2.1 · p. 2

Web data

• Web-data methods learn visual representations from accessible videos using concept detection, Q-learning-based selection and labeling, or large-scale pretraining.The paper distinguishes these efforts from its cross-domain transfer setting.Source: §2.2 · p. 2

Prototypes

• Prototypical networks classify by measuring distances between test embeddings and class prototypes, providing a nearest-neighbor partition of embedding space.This prototype principle motivates the paper’s use of class centroids in external memory.Source: §2.3 · pp. 2–3

External memory

• External memory supplies additional offline information to neural models during training and testing and has been applied beyond language tasks, including visual tracking and image captioning.The paper adapts this memory idea to cross-domain sign recognition.Source: §2.3 · p. 3

Isolated signs

• The labeled WSLR dataset is denoted D_s = {(I_i, L_i)}_i=1^N, where each I_i is an RGB video and L_i is a K-class label.Each isolated-sign video has dimensions l × h × w × 3.Source: §3.1 · p. 3

Sign news

• The complementary sign-news dataset is denoted D_n = {(S_i, T_i)}_i=1^M, pairing an RGB video S_i with subtitle tokens T_i.The news videos average 300 frames, compared with about 64 frames for isolated-sign videos.Source: §3.1 · p. 3

Motivation

• Samples from the same sign class share visual concepts such as hand gestures and body movements, making these concepts suitable for cross-domain representation.The method therefore targets features that remain invariant across news and isolated-sign domains.Source: §3.2 · p. 3

Domain alignment

• News signs are extracted and jointly trained with isolated signs so the two domains become coarsely aligned in embedding space.The aligned news representations are then used to construct prototypes in external memory.Source: §3.2 · p. 3

Descriptor learning

• For each isolated sign, the model measures correlations with memory contents to learn a domain-invariant descriptor from news-sign prototypes.This descriptor provides a class-specific common feature for subsequent recognition.Source: §3.2 · p. 3

Extracting news signs

• The base classifier selects the highest-scoring sliding window for each subtitle-associated sign class and discards windows below threshold ϵ.The resulting clips form the news-sign set collected for each class.Source: §3.3.1 · p. 3
• Domain differences cause the classifier to produce false negatives, so many valid news signs are missed during localization.The isolated-sign and news-sign feature distributions differ substantially before alignment.Source: §3.3.2 · p. 3

Prototypical memory

• Each class prototype is the mean feature embedding of its coarsely aligned news-sign samples, stored as a cell in prototypical memory.The memory provides a compact representation of news signs for later retrieval of related visual concepts.Source: §3.3.3 · pp. 3–4
• Although sign news is abundant, domain gaps yield relatively few extracted samples, and further reducing the gap can make clusters less concentrated and classification less accurate.Using the adapted classifier to recollect clips does not increase the number of news-sign samples because its WSLR performance decreases.Source: §3.3.3 · p. 4

Descriptor construction

• The descriptor-learning stage generates a class-specific common feature from prototypical memory for an isolated sign feature sequence.The isolated feature X and memory M contain frame-level features and class prototypes, respectively.Source: §3.4 · p. 4
• Because isolated-sign and prototype features occupy different spaces, separate projection matrices map them into a common embedding space before correlation is computed.The correlation uses a normalized dot product after projection.Source: §3.4 · p. 4
• The method reweights memory features according to their correlations with the isolated sign, then maps the result back as a residual and max-pools it into descriptor P.A perturbation matrix compensates for errors during domain alignment, and Wu maps the intermediate representation back to the input space.Source: §3.4 · pp. 4–5

Descriptor use

• The resulting descriptor is used to learn isolated-sign representations through a memory-augmented attention module.The descriptor provides the representation on which the subsequent temporal-attention operation is based.Source: §3.4 · p. 5

3.5. Memory-augmented Temporal Attention

• The resulting attention focuses classification on discriminative visual concepts while suppressing common gestures such as raising or putting down the hands.These demonstrating gestures occur across isolated sign videos and can make WSLR classification harder.Source: §3.5 · p. 5
• The module compares the prototypical-memory descriptor with isolated-sign features across temporal regions to capture salient information.The descriptor and sign representation are projected into a shared space, producing pairwise similarities that define the attention map.Source: §3.5 · p. 5
• It embeds isolated-sign features, aggregates them using the attention map, and maps them back to the original feature space as attended representations.The attended representation is formed through low-dimensional channel aggregation and a lifting projection before complementing the sign representation.Source: §3.5 · p. 5

3.6. Optimization

• Binary cross-entropy is computed from each sample’s class probabilities and labels across the batch.N denotes batch size, K the number of sign classes, p_ij the probability for sample i and class j, and y the sample label.Source: §3.6 · p. 5

4.2. Qualitative Results

• Memory-augmented temporal attention filters out starting and ending gestures for “jacket” and identifies two signing peaks for repeated “brown” gestures.The attention module selects representative frames as sign signatures, which are visually consistent with manually identified news-sign frames.Source: §4.2 · p. 6, Figure 4 · p. 6

4.4.1 Comparison of Recognition Performance

• Directly adding extracted news words to training harms performance in most cases, although MSASL200 shows a minor improvement.This result demonstrates the difficulty of transferring knowledge across the domain gap while supporting the usefulness of the collected news signs.Source: §4.4.1 · p. 7, Table 2 · p. 7
• The proposed method surpasses the previous state-of-the-art I3D model on both WLASL and MSASL.Both methods use the same I3D backbone, so the reported improvement is attributed to knowledge transferred from news words.Source: §4.4.1 · p. 7
• The method achieves a large-margin advantage on WLASL, where each class has only 13–20 examples compared with 40–50 for MSASL.The authors connect this result to more effective representation learning under WLASL’s scarce-data regime.Source: §4.4.1 · p. 7

4.4.2 Word-level Classifier as Temporal Localizer

• The word-level classifier provides higher temporal-localization performance than I3D across the evaluated tIoU levels.The authors use this approach to provide an option for automatic temporal annotations of sign-word boundaries.Source: §4.4.2 · p. 8, Table 3 · p. 8

4.5. Analysis and Discussions

• Ablations show that coarse domain alignment improves performance, while using aligned news signs as prototypical memory achieves the best setting.Using only isolated-sign features lacks cross-domain knowledge, and averaging isolated and news features produces noisy centroids.Source: §4.5 · p. 8

• The proposed method leverages subtitled sign news to improve sign language recognition and automatically localize sign words, reducing laborious labeling.Source: §5 · p. 8
• It jointly aligns isolated and news signs, stores class centroids in prototypical memory, and learns domain-invariant descriptors for recognition.Source: §5 · p. 8
• Temporal attention emphasizes class-specific features while suppressing features shared across different signs.Source: §5 · p. 8
• The model outperforms state-of-the-art methods, while the reported tables evaluate coarse domain alignment and sign-news-based external memory settings.Source: §5 · p. 8, Table 4 · p. 8, Table 5 · p. 8