PaperlayerView exact revision on arXiv

Source-linked AI summary

A Reverse Sign Language Dictionary: Open-Vocabulary Sign Recognition from Continuous Signing via Video Captioning and Description Retrieval

Santiago Poveda-Gutiérrez, Hideki Nakayama, Mayumi Bono

arXiv:2609.03788v1cs.CVcs.CL

TL;DR

Most ISLR uses fixed gloss inventories, leaving continuous signing and signs unseen in training outside that framing. This paper captions sign-level clips into procedural descriptions and retrieves matching vocabulary entries. Fine-tuning improves retrieval, including on unseen classes, and the authors identify captioning quality as a remaining gap.

  • Problem

    Most ISLR assumes isolated clips and a fixed gloss inventory, while natural signing is continuous and the lexicon is never fully covered by training.

    Source: §I · p. 1

  • Method

    The pipeline captions sign-level clips with an open-weight vision-language model, then retrieves the closest target description using a multilingual sentence encoder.

    Source: A. Pipeline · p. 1

  • Results

    On TS1, +13.0pp top-10 over frozen-ViT and +44.5pp over untrained; each training step improves every test set, with top-5/top-10 comparable to I3D on two of three eligible sets.

    Source: A. Vision-adapted fine-tuning matches… · p. 2

  • Takeaways & Limitations

    The paper presents open-vocabulary, description-based lookup from continuous signing without gloss supervision as an approach not previously applied to JSL.

    Source: §I · p. 1

    Evidence is limited by the small, single-language corpus, one training seed, and signer and recording-session context shared between training and test.

    Source: §IV · p. 3

Abstract

from arXiv · show

Isolated Sign Language Recognition (ISLR) is conventionally cast as closed-set classification over gloss labels, which cannot generalize to signs unseen in training and ties every deployment to a gloss-annotated lexicon. We instead recognize signs extracted from continuous signing by (1) captioning a sign-level clip into a free-form procedural description of the articulation with an open-weight vision-language model, and (2) retrieving the closest entry from a vocabulary of target descriptions with a multilingual sentence encoder: a reverse sign language dictionary that needs no gloss supervision and admits an open vocabulary. On 1,300 sign-level segments from a Japanese Sign Language (JSL) dialogue corpus annotated with procedural descriptions (against a 2% top-10 chance floor over the 503-entry target vocabulary), fine-tuning the captioner substantially improves seen-class retrieval: language and vision tower fine-tuning raises top-10 retrieval on seen classes from 4.5% (untrained) to 49%, becoming statistically indistinguishable from a standard supervised closed-set classifier (I3D) on two of the three test sets where a closed-set classifier can be evaluated at all. More importantly, unseen-class retrieval also improves significantly over the untrained pipeline (11.5% -> 21.0% top-10, p=0.0094), a regime in which the closed-set classifier cannot participate. A matcher-side empirical upper-bound analysis shows the sentence encoder already recovers close to 100% of paraphrased gold descriptions, locating a gap in captioning quality that we aim to address in future work. To our knowledge this is the first description-based, open-vocabulary sign lookup from continuous signing without gloss supervision, and the first for JSL.

I. INTRODUCTION

Most ISLR assumes isolated clips and a fixed gloss inventory, while natural signing is continuous and training never covers the full lexicon. The paper proposes open-vocabulary retrieval from continuous signing without gloss supervision.

I. INTRODUCTION

  • Natural signing is continuous, coarticulated, and fast, averaging approximately 0.5 seconds per sign in the corpus, while training never covers the full lexicon.Most ISLR research instead assumes isolated, deliberately produced clips and a fixed gloss inventory.Source: §I · p. 1
  • Prior sign spotting is gloss-supervised and closed-vocabulary, while no prior method is known to perform gloss-free, open-vocabulary lookup from continuous signing or target JSL.Zero-shot description recognition has been studied for isolated dictionary-style productions.Source: §I · p. 1
  • The paper contributes a modular captioning-and-retrieval pipeline, unseen-class and view-generalization evaluation, and a controlled study comparing two fine-tuning regimes.The study reports that vision-tower adaptation yields the larger improvement and significantly improves unseen-class retrieval over the untrained baseline.Source: §I · p. 1

A. Pipeline

The pipeline captions a sign-level clip with a vision-language model, then retrieves the closest target description using a multilingual sentence encoder. Its stages are swappable, and the matcher and vocabulary remain unchanged in the reported results.

A. Pipeline

  • An open-weight vision-language model turns each sign-level clip into a procedural description of its articulation.The paper uses InternVL3, 8B, and prompts for features such as hand shape, location, and movement.Source: A. Pipeline · p. 1
  • A multilingual sentence encoder embeds the caption and ranks precomputed vocabulary entries by cosine similarity, returning the top-k matches.Captioning and matching are conducted in Japanese, the language of the ground-truth descriptions.Source: A. Pipeline · p. 1
  • Both pipeline stages are swappable; fine-tuning updates captioner adapters, while the matcher and vocabulary remain untouched in the reported results.Vision-adapted fine-tuning additionally updates a low-rank adapter on the captioner’s vision tower.Source: Figure 1 · p. 2

B. Data and evaluation protocol

The evaluation uses Japanese procedural descriptions of conversational JSL signs and tests retrieval across seen classes, unseen classes, and changes in viewpoint. It compares retrieval with a closed-set classifier and a matcher upper bound.

B. Data and evaluation protocol

  • The JSL dialogue subset contains 1,300 sign-level segments from five signers and two prefectures, with semi-frontal and side views and 503 unique descriptions.Each segment has a free-form Japanese articulation description that can differ from its dictionary citation form.Source: B. Data and evaluation protocol · p. 2
  • Retrieval accuracy is measured at top-1, top-5, and top-10 against the full vocabulary, with Wilson 95% confidence intervals and exactly one gold description per query.Testing uses two active-signer single-view crops of each segment.Source: B. Data and evaluation protocol · p. 2
  • TS1 tests unseen segments from seen classes, TS2 tests singleton descriptions with no training segments, and TS3.1/TS3.2 test cross-view generalization on the same 100 descriptions and segments.TS2 is inaccessible to closed-set classifiers; sharing the TS3 sets controls for class difficulty.Source: B. Data and evaluation protocol · p. 2
  • The comparisons include a per-test-set fine-tuned I3D closed-set classifier and a matcher upper bound using lexical, structural, and free-form paraphrases of gold descriptions.The paraphrases are fed directly to the matcher to isolate matcher error from captioning error.Source: B. Data and evaluation protocol · p. 2

C. Fine-tuning: two regimes

The study compares two otherwise matched fine-tuning regimes: one freezes the vision tower, while the other also adapts its linear layers. Both use the same training schedule, data rules, prompts, and decoding as the untrained pipeline.

C. Fine-tuning: two regimes

  • Regime A trains only the language model’s LoRA adapters and multimodal projector, while Regime B additionally targets the vision transformer’s linear layers with LoRA.The vision tower is frozen in Regime A and adapted in Regime B.Source: C. Fine-tuning: two regimes · p. 2
  • Both regimes fine-tune InternVL3-8B with LoRA using one uninterrupted nine-epoch cosine schedule and per-test-set training pools with identical train/test exclusion rules.Prompts and decoding match the off-the-shelf pipeline.Source: C. Fine-tuning: two regimes · p. 2
  • Each test set’s Regime A/B pair is a matched control differing only in whether the vision tower is adapted.Data, schedule, and seed are identical between each pair.Source: C. Fine-tuning: two regimes · p. 2

A. Vision-adapted fine-tuning matches I3D at top-5/top-10, beats it at top-1, and improves unseen-class retrieval

Vision-adapted fine-tuning improves retrieval across test sets, with top-5/top-10 results comparable to I3D on two of three eligible sets and significantly better unseen-class top-10 retrieval than the untrained pipeline. I3D cannot be evaluated on TS2 because its classes have no training segments.

A. Vision-adapted fine-tuning matches I3D at top-5/top-10, beats it at top-1, and improves unseen-class retrieval

B. Why LoRA-on-ViT instead of full ViT fine-tuning

At this data scale, LoRA-on-ViT at a moderate matched learning rate is the best fine-tuning recipe; full vision-tower tuning trades capacity gains against collapse or slower language-adapter learning. The matcher recovers 90–100% of paraphrased gold descriptions, pointing to captioning rather than retrieval as the remaining gap.

B. Why LoRA-on-ViT instead of full ViT fine-tuning

  • At matched learning rate, added vision capacity helps by +9.0pp (p = 0.025), but the lower rate needed for full ViT fine-tuning costs −15.0pp (p = 1.0×10−4).The lower rate also slows the language model’s adapters.Source: B. Why LoRA-on-ViT instead of full ViT… · p. 3
  • At this data scale, LoRA-on-ViT with a moderate matched learning rate is the best recipe; full-tower fine-tuning at the same rate collapses to one distinct caption across 200 clips at epoch 1.The recipe used approximately 800 unique training segments, and the authors interpret the result as evidence for a data-limited ceiling.Source: B. Why LoRA-on-ViT instead of full ViT… · p. 3
  • The matcher recovers the gold description 90–100% of the time when given a paraphrase, locating the remaining gap in captioning rather than retrieval.Source: B. Why LoRA-on-ViT instead of full ViT… · p. 3

IV. LIMITATIONS AND FUTURE WORK

The evaluation is limited by modest sample sizes, a single training seed, shared signers and recording-session context, and a small single-language corpus. Fine-tuned captions also concentrate on training descriptions; planned work targets this collapse and tests broader transfer.

Limitations.

  • Each configuration uses one random seed, and test segments share signers and recording sessions with training, so measured accuracy may partly reflect context rather than sign recognition transfer.All five signers appear in both training and test; sample sizes are n = 100–200, with an average Wilson CI half-width of ±5.7pp across 36 cells.Source: §IV · p. 3

Captioner output collapse

  • On TS1, over 90% of fine-tuned captions are near-verbatim training-description copies, versus 0% before fine-tuning, regardless of vision-tower adaptation.The authors interpret this as overfitting at the current data scale, rather than a property of vision-tower adaptation specifically.Source: §IV · p. 3

Interpretation and next steps

  • For unseen TS2 classes, training raises top-5/top-10 while top-1 remains 0.0% for both regimes because copied training descriptions cannot be the exact gold string.The authors say semantically close captions can still place the gold description in the shortlist.Source: §IV · p. 3
  • Planned work includes cross-lingual evaluation, a CLIP-style contrastive baseline, large-scale masked video pre-training, paraphrase-augmented targets, and video-aware retrieval components.Source: §IV · p. 3
Loading 2609.03788v1…