PaperlayerView exact revision on arXiv

Source-linked AI summary

SignDino: Self-Supervised Sign Language Representation Learning via Temporal-Axis Self-Distillation

Junyi Hu, Zhewen He, Haomian Huang, Zhenhua Li, Zhifei Li, Yi Fang

arXiv:2609.06296v1cs.CVcs.CL

TL;DR

Sign-video SSL must represent distinct articulators and how they evolve over time. SignDino applies DINO-style student–teacher learning to tracked anatomical streams, improving over the strongest public SSL baseline across several sign tasks.

  • Problem

    Sign-video SSL must capture where linguistic evidence is articulated and how it evolves, without relying on gloss labels, translations, or hand-defined sub-sign units.

    Source: §1 · p. 1

  • Method

    SignDino independently encodes tracked left-hand, right-hand, and face streams, applying temporal DINO distillation, masked-frame prediction, feature spreading, and Gram anchoring.

    Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 4

  • Results

    SIGNDINO improves over the strongest public SSL baseline across translation, ISLR, fingerspelling, and phonological probing; frozen features also exceed the previous live-fine-tuned translation baseline.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The released framework, code, and checkpoints provide a strong public self-supervised baseline for sign-language video.

    Source: §5 · p. 9

    Tracker dropouts under occlusion or fast motion can corrupt stream inputs, and the validity mask removes only the worst cases.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Self-supervised sign language representation learning must model two properties not central to natural-image SSL: signs are produced by a small set of anatomically distinct articulators, and their meaning depends on the temporal organisation of those articulators. We introduce SignDino, a self-supervised sign-video encoder that moves the DINOv3 student--teacher recipe from the spatial domain of image crops to the temporal domain of tracked sign streams. Each video is decomposed into left-hand, right-hand, and face streams by a detector-first YOLOv8n+ByteTrack pipeline. A frozen DINOv3 ViT-B/16 embeds each per-frame anatomical crop, while lightweight temporal Transformers, not the image backbone, form the student and EMA teacher. They are trained by temporal DINO self-distillation, frame-level masked-token prediction in the style of iBOT, KoLeo feature spreading, and Gram anchoring of the frame-to-frame similarity structure. This design keeps strong image-level visual primitives fixed and learns only how articulator states evolve across time. We evaluate on sign-to-English translation, isolated sign recognition, and fingerspelling detection benchmarks. Across these tasks, SignDino provides a strong public self-supervised representation and shows competitive or state-of-the-art performance under matched downstream evaluation.

1 Introduction

Sign language requires representations that track distinct articulators and their temporal coordination, while learning without glosses, translations, or hand-defined sub-sign units. SignDino adapts DINO’s local–global distillation to tracked anatomical streams and evaluates the framework across sign tasks.

1 Introduction

  • The temporal view treats a shorter crop of an articulator as meaningful within its trajectory, rather than as another spatial view of one frame.Figure 1 depicts temporal global and local crops along tracked left-hand, right-hand, and face streams, with iBOT masks in local crops.Source: §1 · p. 1, Figure 1 · p. 2
  • The paper evaluates SignDino on sign-to-English translation, isolated sign recognition, and fingerspelling detection under the SHUBERT protocol.Its pretext task differs from SHUBERT’s precomputed-cluster supervision by using continuous temporal student–teacher distillation.Source: §1 · p. 2
  • SignDino transposes DINO to temporal crops, using per-frame articulator tokens, frame masking, and Gram anchoring to constrain clip structure.The teacher observes a longer temporal crop and the student shorter, partially masked crops, matching prototype and frame-similarity structure.Source: §1 · pp. 1–2
  • Left-hand, right-hand, and face streams are encoded independently, with fusion deferred to downstream heads.A detector-first YOLOv8n and ByteTrack pipeline supplies the anatomical crops.Source: §1 · p. 2

2 Related Work

Related work spans self-supervised learning across modalities, sign-specific representation learning, and multi-stream sign-video pipelines.

SSL across modalities and the DINO family

  • DINO-style image SSL builds on student–teacher learning; DINOv2 and DINOv3 scale the recipe, while iBOT adds masked-patch prediction and VideoMAE adapts MAE to video.SIGNDINO freezes a DINOv3 encoder and reapplies DINO, iBOT, KoLeo, and Gram objectives along tracked streams’ time axes.Source: §2 · p. 2

Sign-language representation learning

  • Prior sign SSL uses reconstruction or sign–text alignment; SHUBERT predicts clustered hidden units, whereas SIGNDINO uses continuous student–teacher distillation over temporal crops.SHUBERT’s streams use masked cluster prediction on 984 hours of YouTube-ASL.Source: §2 · p. 2

Multi-stream pipelines, detector-first tracking, fingerspelling

  • Related sign-video work includes multi-stream architectures, ByteTrack for Kalman-filling dropped frames, and phonological decomposition resources such as ASL-Lex 2.0 and ASL Citizen.Fingerspelling is described as a dense-temporal-cue sub-task.Source: §2 · p. 3

3 SignDino

SignDino encodes left-hand, right-hand, and face streams independently, keeping the image backbone frozen while learning temporal representations. Its training adapts DINO-style distillation to temporal crops, masked frames, and within-clip similarity structure.

3 SignDino

  • Three independently trained streams encode left hand, right hand, and face, with fusion postponed to downstream heads and only temporal encoders and heads receiving gradients.A frozen DINOv3 backbone supplies per-frame crop embeddings.Source: §3 · p. 3, §3.2 · p. 3

3.1 Multi-Stream Feature Pre-Processing

  • YOLOv8n detects hands and faces, while ByteTrack associates hand identities and interpolates dropouts of up to K frames.The pipeline masks entries when tracking fails for more than K consecutive frames; Kalman-filled tracking reaches approximately 95% per-frame hand detection on How2Sign.Source: §3.1 · p. 3

3.2 Self-Supervised Training of SignDino

  • Each temporal Transformer projects frozen 768-dimensional embeddings to 384 dimensions and uses six pre-LN blocks with absolute-frame sinusoidal positions.Validity masks prevent detector-missed frames from being attended; CLS and per-frame tokens feed different losses and downstream heads.Source: §3.2 · p. 3
  • Training samples two global crops of 64–96 frames and eight local crops of 10–32 frames, while the student masks frame positions and the teacher sees only globals.Padding and validity masks exclude padded positions from attention, iBOT masking, and Gram loss.Source: §3.2 · pp. 3–5
  • Stage 1 combines crop-level DINO distillation, masked-frame iBOT prediction, and KoLeo regularisation that spreads embeddings across the unit sphere.The iBOT term predicts the teacher's prototype assignment at frames whose input the student did not see.Source: §3.2 · pp. 4–5
  • Stage 2 adds Gram anchoring that matches student and frozen-teacher frame-to-frame cosine-similarity matrices within the same temporal crop.This preserves which frames are similar or dissimilar inside a clip.Source: §3.2 · pp. 5–6
  • For downstream use, layer-weighted per-frame representations from all three streams are concatenated and projected to the task input dimension.The layer weighting uses tokens from every Transformer layer.Source: §3.2 · p. 6

4 Experiments

The experiments compare SIGNDINO with SHUBERT under a matched public-data protocol, varying the upstream representation while holding downstream settings as constant as possible. Results span translation, isolated sign recognition, phonological probing, fingerspelling detection, and ablations of the model and training choices.

4 Experiments

  • The matched protocol uses SHUBERT's public-data setting and varies the upstream representation; pre-training uses a decontaminated source split of approximately 984 YouTube-ASL hours.Clips overlapping the OpenASL evaluation set are removed and replaced with non-overlapping ASL videos from YouTube-SL-25.Source: §4 · p. 6, §4.1 · p. 6

4.2 Sign Language Translation

  • 17.9/24.5 BLEU on How2Sign/OpenASL beats SHUBERT by +1.7/+1.3 BLEU, respectively, under live fine-tuning.Frozen-feature SIGNDINO also exceeds SHUBERT on both translation datasets; on zero-shot FLEURS-ASL, it reaches 5.3 BLEU versus SHUBERT's 4.7.Source: §4.2 · pp. 6–7, Table 1 · p. 7

4.4 Phonological Feature Probing

  • The phonological probe uses 16 feature-specific classifiers to diagnose whether SIGNDINO represents sign-internal categories, without training or selecting the representation.The probed ASL-Lex 2.0 attributes include handshape, selected fingers, location, contact, and motion.Source: §4.4 · p. 7

4.3 Isolated Sign Language Recognition

  • Full fine-tuning reaches Recall@1 0.704 (+0.054) on ASL Citizen and 0.593 (+0.053) on Sem-Lex; rank-1 LoRA improves both by +0.02.On WLASL2000, LoRA reaches 66.85/64.20 P-I/P-C, exceeding Uni-Sign's 63.52/61.32 P-I/P-C by +3.3 P-I; the frozen-feature variant does not exceed SHUBERT.Source: §4.3 · p. 7, Table 2 · p. 8

4.5 Fingerspelling Detection

  • Full fine-tuning reaches 0.43 mIoU on ASL-STEM-Wiki, versus SHUBERT's 0.40 under rank-1 LoRA.SIGNDINO reaches 0.41 versus 0.40 under rank-1 LoRA; detection uses per-frame binary classification followed by thresholding, smoothing, and interval extraction.Source: Table 2 · p. 8, §4.5 · p. 8

4.6 Ablations

  • Frame-only masking leads the tested strategies; right-hand removal is most damaging (-8.1 BLEU), and further ablations favor randomized crops, layer-weighted pooling, and ByT5.Teacher prototype entropy stays above 0.91; layer weights concentrate on deeper layers for ISLR and fingerspelling but spread wider for translation, while the fusion head's Face query attends 0.57 to itself for translation.Source: §4.6 · pp. 8–9
  • Removing the Gram anchor drops How2Sign BLEU from 17.9 to 16.5, while a short teacher window costs 2.1 BLEU and raw detections cost 1.5 BLEU.Raw detections also cost 0.094 ASL Citizen R@1; the passage attributes this to sparser, noisier crops from unfilled single-frame detector failures, with ISLR more sensitive.Source: §4.6 · pp. 8–9, Table 4 · p. 9

5 Conclusion

SIGNDINO applies the DINOv3 teacher–student recipe to tracked hand and face streams along the temporal axis. Under the SHUBERT public-data protocol, it improves on the strongest public SSL baseline across translation, isolated sign recognition, fingerspelling, and phonological probing.

5 Conclusion

  • SIGNDINO transposes DINOv3 teacher–student learning onto tracked left-hand, right-hand, and face streams, using DINO, iBOT, KoLeo, and stage-2 Gram anchoring.The front end combines YOLOv8n–ByteTrack crops with a frozen DINOv3 image encoder.Source: §5 · p. 9
  • Under the SHUBERT public-data protocol, SIGNDINO improves over the strongest public SSL baseline across translation, ISLR, fingerspelling, and phonological probing.The conclusion also reports that frozen-feature SIGNDINO exceeds the previous live-fine-tuned baseline on translation.Source: §5 · p. 9

Limitations

The paper identifies practical limits in tracking, the frozen image backbone, stream coverage, language scope, and preprocessing cost. It also explains why the DINOv3 EMA schedule is retained despite variable token counts.

Limitations

  • Occlusion or fast motion can cause long ByteTrack dropouts that corrupt hand and face streams; the validity mask removes only the worst cases.The frozen DINOv3 backbone was trained on web images rather than signing video, and stream-specific fine-tuning is left for future work.Source: Limitations · p. 9
  • The three-stream design omits explicit body-pose context, and experiments cover only ASL or ASL-to-English benchmarks; extension to other sign languages remains future work.Adding a 14-d upper-body stream is described as a one-config-line change in the fusion module.Source: Limitations · p. 9
  • Detector-tracker preprocessing and frozen-image-encoder embedding impose a non-trivial one-off cost, though the full upstream budget remains below the strongest comparable SSL baseline.The temporal SSL stages are modest by comparison once embeddings are cached.Source: Limitations · p. 9

A Default Hyperparameter Card

  • The DINOv3 EMA momentum schedule is retained for variable-length crops because the averaged SSL losses keep per-step gradient magnitude approximately invariant to token count.The stated token-count range is 208–448, and the schedule tracks parameter movement per step rather than information flow per token.Source: A Default Hyperparameter Card · p. 13

B Qualitative Cluster Analysis of SIGNDINO Features

The qualitative analysis clusters frozen SIGNDINO embeddings from held-out videos to inspect what the streams represent. The visualized clusters span recurring facial expressions and poses, as well as handshapes across identities, backgrounds, and skin tones.

B Qualitative Cluster Analysis of SIGNDINO Features

  • The analysis applies k-means to L2-normalized per-frame CLS vectors from fresh How2Sign and OpenASL videos, then visualizes crops nearest selected centroids.Cluster labels are descriptive only and are not used in downstream evaluation.Source: B Qualitative Cluster Analysis of… · p. 13
  • Face clusters recur across identities and reflect expression or pose, while hand clusters capture recurring handshapes across backgrounds and skin tones.The authors interpret these patterns as evidence that SIGNDINO internalizes sub-lexical articulator structure rather than identity or background.Source: B Qualitative Cluster Analysis of… · p. 13

C Phonological Feature Recognition

SIGNDINO’s phonological probe outperforms SHUBERT across the 16 ASL-Lex features on both datasets. A descriptive t-SNE probe shows translation-trained features grouping occurrences of semantically related English words.

C Phonological Feature Recognition

  • The probe averages stream-fused isolated-sign features and uses 16 linear heads to predict ASL-Lex phonological attributes, reporting top-1 Recall@1 per feature.Its purpose is diagnostic: it tests sub-lexical structure without training the representation or informing model selection.Source: C Phonological Feature Recognition · pp. 13–14
  • 87.64 vs 79.38 R@1 on Sem-Lex and 91.85 vs 85.02 R@1 on ASL Citizen: SIGNDINO beats SHUBERT across all 16 features.No feature regresses; gains are largest on several temporally articulated and fine-handshape categories.Source: C Phonological Feature Recognition · p. 14, Table 6 · p. 15

D Cross-Sentence t-SNE of Translation-Trained Features

  • The visualization projects time-aligned, per-word fused features from 2,400 sampled sentences to probe their alignment with English-token semantics.The sample draws from How2Sign and three ISLR datasets.Source: D Cross-Sentence t-SNE of… · p. 15
  • Family nouns and visual-perception verbs cluster in the t-SNE projection, while weather nouns and motion verbs occupy distinct regions.“Sleep” lies between the visual-perception and motion verb groups.Source: D Cross-Sentence t-SNE of… · p. 15
  • The t-SNE is descriptive and is not used for model selection or a quantitative claim.It offers a sentence-level view of semantic structure after the SLT objective is applied to the SSL representation.Source: D Cross-Sentence t-SNE of… · p. 15

E Reproducibility Checklist

The checklist documents data processing, training, evaluation, and release details for reproducibility. Its diagnostics report prototype use, layer weighting, cross-stream attention, and pre-training compute.

E Reproducibility Checklist

  • The checklist describes source-data decontamination, anatomical-crop preprocessing, task-specific evaluation protocols, released artifacts, and seed and software settings.It also retains privacy-aware face-processing options and reports preprocessing choices so users can audit retained visual cues.Source: E Reproducibility Checklist · pp. 15–17

F Training and Downstream Diagnostics

  • The diagnostics examine prototype collapse, encoder-layer use, and whether fusion-head attention aligns with articulator roles across downstream tasks.The reported patterns are diagnostic rather than task-performance results.Source: F Training and Downstream Diagnostics · p. 17

F.1 Prototype entropy (cluster non-collapse)

  • At Stage 2 convergence, normalized prototype entropy remains about 0.92 for hand streams and 0.94 for face, above the collapse regime.Entropy declines from 1.000 at initialization; the face stream remains higher throughout.Source: §F.1 · p. 18, Table 9 · p. 18

F.2 Layer-weighted-sum head weights

  • Hand-stream layer weights peak at L=5, while face weights peak earlier at L=3–4 and are more evenly distributed across layers.ISLR and fingerspelling assign more weight to hand layers L=5+6 than translation does.Source: §F.2 · p. 18, Table 10 · p. 18

F.3 Cross-stream attention at the most-active fusion layer

  • Fusion attention patterns differ by task: SLT face queries favor face, ISLR hands attend to each other, and fingerspelling attention is more diffuse across streams.For fingerspelling, face-query attention is 0.34/0.33/0.33, while hand self- and cross-attention strengthen.Source: §F.3 · pp. 18–19, Table 11 · p. 19

F.4 Pre-Training Compute Budget

  • The three stream encoders train sequentially on the same 8-GPU pool, with the frozen-DINOv3 embedding cache reducing per-step backbone work.The compute-budget table reports the full upstream schedule and one-off cache preprocessing cost.Source: §F.4 · p. 19

F.5 SSL Data-Scaling Curve

The data-scaling scan varies the size of the decontaminated YouTube-ASL pre-training corpus while holding the training and evaluation setup fixed. Frozen-feature How2Sign results improve throughout the scanned range.

F.5 SSL Data-Scaling Curve

  • Across the scan, the temporal sampler, embedding cache, downstream recipe, and SSL schedule remain fixed; only the source subset size changes.Evaluation uses frozen SIGNDINO encoders and the same downstream ByT5-Base head.Source: §F.5 · p. 19, Table 13 · p. 19
  • The scan compares 10% and 100% of the decontaminated YouTube-ASL corpus, with a five-point sweep extending the same scale range.Intermediate subsets are video-disjoint random samples intended to preserve speaker and topic distributions.Source: Table 13 · p. 19, Table 14 · p. 19, §F.5 · p. 19
  • Doubling pre-training hours yields roughly 1 BLEU improvement across the scanned range, and the curve is not yet flat at 100% of the public corpus.The table reports frozen-feature How2Sign BLEU and BLEURT as a function of source-corpus pre-training budget.Source: Table 14 · p. 19

F.6 Additional Ablations

The ablations compare design choices under matched evaluation settings, including pooling, fine-tuning, crop sampling, visual backbones, and translation decoders. The ethics statement identifies signer privacy and the absence of a fine-grained per-signer fairness breakdown.

F.6 Additional Ablations

  • The ablation suite also tests masking strategy and holds the common training contract and How2Sign test evaluation fixed unless noted.The appendix lists masking, pooling, fine-tuning, crop-length, backbone, and decoder comparisons.Source: §F.6 · pp. 19–20, Table 15 · p. 20
  • Raw frozen DINOv3 embeddings yield 9.4 BLEU/41.2 BLEURT, 7.4 BLEU below the weighted-sum default; last-layer-only pooling recovers most, but not all, of the gap.The ablation supports the learnable weighted sum over all encoder layers as the default pooling.Source: §F.6 · p. 20, Table 16 · p. 20
  • Live fine-tuning reaches 17.9 BLEU versus 16.8 BLEU frozen, while the frozen variant is recommended when downstream compute is constrained.The live setting is the recommended operating point for the headline result.Source: Table 1 · p. 20, §F.6 · p. 20, Table 17 · p. 20
  • The crop-length scan reports a directionally consistent randomization trend that strengthens with the FLOP gap, not a single strongly significant gap.The reported iso-FLOP gaps are about 1.9–2.4 combined-seed standard deviations and the low-FLOP gap about 2.8.Source: §F.6 · p. 20, Table 18 · p. 21
  • Across five frozen DINO-family backbones, How2Sign BLEU spans 16.5–18.0; DINOv3 ViT-L/16 adds only +0.1 over ViT-B/16 at 3.5× the parameters.The authors retain ViT-B/16 as the compute-aware default.Source: §F.6 · p. 21, Table 19 · p. 21
  • SIGNDINO exceeds SHUBERT by +1.4–+1.7 BLEU across tested decoders; ByT5-Base is best for both, and larger decoder capacity does not guarantee better quality.At matched capacity, ByT5-Base also beats mT5-Base by +0.8 BLEU and mBART-large by +1.1 BLEU for SIGNDINO.Source: §F.6 · p. 21, Table 20 · p. 21

G Ethics Statement

  • The evaluation uses public ASL research corpora, but a fine-grained per-signer fairness breakdown is left for future work.The authors also recommend Deaf-community review of production deployments using the released artifacts.Source: G Ethics Statement · p. 22

H Sample How2Sign Translations

The section presents a qualitative How2Sign translation comparison and describes an ASL phonological-feature probe that tests spatial grounding, manual articulation, and temporal movement.

I ASL Phonological Feature Classification Details

  • The probe uses the ASL-Lex 2.0 feature set and retains SHUBERT’s feature vocabulary to preserve benchmark comparability.The feature values are used across 16 classification heads.Source: I ASL Phonological Feature… · p. 22

H Sample How2Sign Translations

I ASL Phonological Feature Classification Details

  • The phonological probe groups features to test spatial grounding, fine manual articulation, and dynamic temporal information.Location and contact probe spatial grounding; handshape, selected fingers, thumb position, flexion, and spread probe manual articulation; movement and wrist twist probe temporal information.Source: I ASL Phonological Feature… · p. 23

J Discussion

SIGNDINO’s live fine-tuning gain is smaller than SHUBERT’s, while the authors recommend its frozen variant under constrained compute. The discussion frames temporal distillation and Gram anchoring as complementary to SHUBERT’s cluster prediction.

J Discussion

  • SIGNDINO moves DINO’s local–global consistency to temporal crops of tracked articulators, with Gram anchoring preserving which clip frames are similar.The discussion links this frame-similarity structure to downstream fingerspelling and continuous-sign translation.Source: J Discussion · p. 23
  • SIGNDINO and SHUBERT differ in image extractors and SSL objectives; combining chunk-level cluster prediction with frame-level Gram anchoring is left for future work.The authors say matched-protocol ablations should first isolate each contribution.Source: J Discussion · p. 23
  • +1.1 BLEU (16.8→17.9) for SIGNDINO’s unfrozen temporal encoders versus SHUBERT’s +2.6 BLEU (13.6→16.2); SIGNDINO starts from a higher frozen-feature point.The authors also attribute the smaller gain to SIGNDINO’s lighter temporal encoder and Gram anchoring, among other factors.Source: J Discussion · p. 23
  • The authors recommend SIGNDINO’s frozen variant for constrained-compute deployment, where the gap to live fine-tuning is smaller than for discrete-cluster SSL baselines.They argue that Gram anchoring preserves frame-to-frame structure used by the downstream head and decoder, leaving less new signal for live downstream gradients.Source: J Discussion · p. 24
Loading 2609.06296v1…