SignTrace: Describe a Sign, Find the Word

1 Introduction

• Learners may remember a sign’s movement without knowing its word, while conventional lookup begins with a known word.Reverse lookup must bridge informal observations and dictionary descriptions while retaining distinguishing details.Source: §1 · p. 1
• SignTrace coordinates LLMs across dictionary preparation, query interpretation, and candidate comparison.Enriched entries, parallel action extraction and dictionary-style rewriting, seven retrieval channels, and LLM scoring connect movement descriptions to dictionary candidates.Source: §1 · pp. 1–2
• The system combines an operational lookup service with evaluation of its retrieval architecture.It was released for user trials and received positive informal feedback; baseline comparisons, component analyses, and error analysis examine enrichment, reranking, and remaining limitations.Source: §1 · p. 2

2.1 Sign language dictionaries and form-based access

• SignTrace complements form-based dictionary access with free-text movement descriptions and inspectable candidate entries.Multiple query views accommodate incomplete or ambiguous wording.Source: §2.1 · p. 2

2.2 Reverse dictionaries and sign-video retrieval

• SignTrace retrieves discrete dictionary entries from text, unlike cross-modal sentence-to-video retrieval or continuous sign-language translation.Reverse dictionaries retrieve words from descriptions of meanings, whereas this task describes visible form rather than a definition.Source: §2.2 · p. 2

2.3 Query expansion, fusion, and LLM reranking

• SignTrace applies query expansion to sign-dictionary movement descriptions, supplementing the original query and structured restatement with a rewritten description.The controls examine the rewrite channel but do not isolate writing style from other changes introduced by rewriting.Source: §2.3 · p. 2
• Reciprocal rank fusion combines retriever rankings, while SignTrace’s reranker scores candidate action matches in a shared prompt.The evaluation fixes query representations, reranking model, and a 20-candidate window; it assesses neither candidate-order sensitivity nor repeated-call stability.Source: §2.3 · pp. 2–3

3.1 Description-to-entry retrieval

• The task returns a ranking of up to K entries from a movement-description query, with a designated dictionary target for each benchmark query.The study uses N = 6,699 entries and K = 60.Source: §3.1 · p. 3

3.2 Four layers of document representation

• Each indexed document combines source action wording, generated visual re-description, structured visual summaries and tokens, and written senses.The generated layers make visually relevant details available in several formulations while the first layer preserves dictionary wording.Source: §3.2 · p. 3

3.3 Dictionary annotation

• Structural annotations and free-text visual descriptions cover all 6,699 entries, and dictionary enrichment preceded retrieval evaluation.Structural annotation model identifiers are recorded, but free-text description identifiers and full prompts are unavailable; no independent annotation-accuracy estimate is available.Source: §3.3 · p. 3

4.1 Overview

• SignTrace enriches dictionary entries, extracts and rewrites each query in parallel, retrieves candidates through seven channels, fuses a 60-entry pool, and reranks its first 20.Figure 1 shows the extracted observation also reaching the reranker directly.Source: §4.1 · p. 3, Figure 1 · p. 4

4.2 Structured action extraction

• Action extraction records hand count, handshape, finger states, orientation, location, contact, movement, two-hand relation, and action sequence, while keeping unspecified properties unknown.It separates observed actions from guesses about meaning; discrete features support structural comparison while the original wording retains expressions transformations may lose.Source: §4.2 · p. 4

4.3 Parallel dictionary-style rewriting

• A parallel 60–120-character dictionary-style rewrite supplies a separate text channel and meaning hints, while reranking uses the original extraction rather than rewrite guesses as new observations.The rewrite prompt prohibits adding unspecified details.Source: §4.3 · p. 5

4.4 Five textual retrieval channels

• Five BM25 views cover extracted action fields, their restatement, movement terms and steps, the raw query, and the dictionary-style rewrite; only T and Td retain all positive-scoring entries.The other three textual views contribute only their first 20 entries.Source: §4.4 · p. 5

4.5 Structural and meaning-hint channels

• The structural channel compares hand count, finger states, movement families, body regions, orientation, and contact; the meaning-hint channel votes from up to ten ordered guesses.Structural comparisons handle two-hand assignment and viewpoint ambiguity; meaning guesses are not intended as action-match scoring evidence.Source: §4.5 · p. 6

4.6 Candidate fusion

• Head-weighted fusion boosts each channel’s top three entries and forms a 60-entry pool; query 490’s target rose from recall rank 13 to final rank 1.The example target received votes from five channels.Source: §4.6 · p. 6, Figure 3 · p. 7

4.7 Action-grounded reranking

• The reranker scores the first 20 candidates for form compatibility against the extracted observation, emphasizing handshape and individual fingers; candidates outside its window cannot be promoted into leading positions.An inherited static-image caution can inappropriately instruct the reranker to ignore movement on textual queries; malformed extraction separately triggers an original-query fallback.Source: §4.7 · p. 7

5.1 Evaluation dataset

• The dataset contains 500 Chinese movement-description queries paired with distinct targets; 200 targets had previously been selected within a 60-entry retrieval pool, introducing selection bias.The queries were derived from dictionary action descriptions and annotations, and only movement descriptions were submitted as input.Source: §5.1 · pp. 7–8, Table 2 · p. 8

5.2 Evaluation metrics and statistical analysis

• Hit@1, Hit@3, Hit@5, Hit@9, Hit@20, and Pool@60 assess fixed designated-target rankings or coverage; paired tests are exploratory and do not adjust for dictionary-derived wording or prior query selection.Alternative entries with similar meanings do not count as matches, and candidate text may retain semantic cues.Source: §5.2 · pp. 8–9

5.3 Experimental configuration

• At most six queries ran concurrently; all 500 received two DeepSeek calls and one Gemini call, with no HTTP errors, retries, or reranking failures.The batch lasted 19 minutes 29 seconds; reported per-query processing time excludes worker waiting and browser or public-service network overhead.Source: §5.3 · p. 9

5.4 Baselines and component analyses

• Raw-query BM25, seven-channel retrieval, and final SignTrace rankings separate first-stage retrieval from reranking; BM25 lists were not Gemini-reranked, so no final-ranking advantage over a matched reranker is established.Offline controls vary dictionary enrichment, remove individual channels or the fusion head bonus, and hold generated query representations fixed.Source: §5.4 · p. 9

6.1 Retrieval performance

• 94.0% Hit@1 and 97.4% Hit@9 were achieved on 500 queries; mean reciprocal rank was 0.9540 at a maximum depth of 60.Table 3 reports final performance on the complete benchmark, with proportions based on 500 queries and no attempted query excluded.Source: Table 3 · p. 10, §6.1 · p. 10

6.2 Effect of reranking

• 22.2 percentage points: Gemini raised first-result hits from 359 to 470 on the same seven-channel candidate lists; 114 outcomes improved and three worsened.The paired comparison reports a conservative interval of 16.8–27.2 points and a Holm-adjusted exact p-value of 1.29 × 10−29.Source: §6.2 · p. 10, Table 4 · p. 11
• At Hit@9, reranking added 12 hits (475 to 487); seven-channel recall added 23 net first-20 hits (465 to 488) over enriched raw-query BM25.Pool coverage rose from 485 to 497 with seven-channel recall; reranking leaves Hit@20 and Pool@60 unchanged, so it improves ordering among available candidates.Source: §6.2 · p. 11, Table 5 · p. 11

6.3 Dictionary enrichment and channel contributions

• 333 first-result hits followed adding structured summaries and visual tokens, versus 249 with source descriptions alone; first-20 coverage rose from 403 to 465.Because queries also derive from dictionary annotations, the comparison cannot separate added action information from similarity between query and index wording.Source: §6.3 · p. 11, Table 6 · p. 11
• Removing the rewrite text channel lowered first-result hits from 359 to 324 and first-20 coverage from 488 to 479; removing the head bonus lowered first-result hits to 271.Removing S or G increased first-result hits but reduced first-20 coverage; the selected fusion was therefore not optimal on every first-stage metric.Source: §6.3 · pp. 11–12, Table 7 · p. 12, Figure 5 · p. 12

6.4 Error analysis and query length

• Thirteen targets fell outside the first nine final results; 12 of these failures were constrained by candidate selection, while the remaining target entered the reranking window at rank 16 and finished at rank 13.First-result hits were 40/45 for 13–24-character queries, 286/307 for 25–40, and 144/148 for 41–64. These post hoc strata do not establish a causal effect of length.Source: §6.4 · pp. 12–13

6.5 Processing time and reliability

• 13.37 seconds was the median query-processing time under six concurrent queries; preparation and local recall had a 2.69-second median, versus 10.68 seconds for reranking.The measured durations exclude batch scheduling wait and browser/network overhead outside the experimental pipeline.Source: §6.5 · p. 13, Table 8 · p. 13, Figure 6 · p. 13
• 863,291 tokens across 1,000 DeepSeek calls and 2,965,848 across 500 Gemini calls were API-reported input-plus-output counts; all three extraction parse failures still ranked first.The parse-failure cases used the original-query fallback and a successful rewrite; token and elapsed-time figures do not imply monetary or browser-latency estimates.Source: §6.5 · pp. 13–14

7 Practical Value and Trial Use

• Learners can describe remembered handshape, contact, and movement, then inspect dictionary entries, illustrations, and meanings to check whether a candidate fits.Conventional word-to-sign lookup remains available for revisiting known vocabulary.Source: §7 · p. 14
• The deployed service makes the movement-description lookup workflow available to users for trials.The paper reports positive informal feedback, but no recruitment protocol, participant-level outcomes, or controlled usability measurements.Source: §7 · p. 14
• The reported trial feedback is an early observation, not a controlled usability measurement.Source: §7 · p. 14

8.1 Architectural contributions

• Gemini reranking raises complete-system first-result hits from 359 to 470, while enrichment raises raw-query BM25 first hits from 249 to 333.Enrichment improves access to relevant entries; reranking improves the order of candidates within its window.Source: §8.1 · p. 14

8.2 Limitations

• Dictionary-derived wording and prior selection of 200 queries mean the benchmark does not establish generalization to descriptions independently produced by users.The wording and action details may resemble indexed descriptions, and prior retrieval-based selection introduces additional selection bias.Source: §8.2 · p. 14
• The test lacks controlled comparisons across signers, dialects, video conditions, and levels of sign-language experience.Its single designated entry per query also means it does not assess all acceptable alternatives or signs outside the dictionary.Source: §8.2 · pp. 14–15

8.3 Future work

• Further evaluation should use independently written observer descriptions and involve Deaf signers and sign-language educators in study design and assessment.Proposed measures include lookup success, identification time, acceptable alternatives, and errors from incomplete descriptions.Source: §8.3 · p. 15
• Candidate-window expansion should be tested on currently missed targets and a broader fixed sample, reporting accuracy, latency, and cost together.Source: §8.3 · p. 15
• Adaptation to another sign language requires its own dictionary, action descriptions, tokenization, feature vocabularies, and prompts; performance there requires new evaluation.The reported experiments cover only the Chinese dictionary evaluated here.Source: §8.3 · p. 15

• Across 500 dictionary-derived queries over 6,699 entries, SignTrace achieves 94.0% Hit@1, 97.4% Hit@9, and mean reciprocal rank 0.9540.Source: §9 · p. 15
• The architecture combines LLM-based dictionary enrichment, action extraction, query rewriting, seven-channel retrieval, and reranking.Source: §9 · p. 15
• The system has been released for user trials and received positive informal feedback, but independent user studies remain necessary to measure everyday lookup success and learning benefits.Component analyses and failure analysis address enriched descriptions, reranking, candidate selection, and action comparison.Source: §9 · p. 15

• Technical development was contributed by Zengji Tu, Xingye Zhu, Ningjing Wang, and Tingyi Huang; all authors contributed to evaluation-data preparation.Those four authors and Yangjunfeng Zhu contributed to system testing and iterative evaluation; Dai Wan provided supervision and technical guidance.Source: Declarations · p. 15
• The research received no external funding, and the authors declare no financial or non-financial competing interests.Source: Declarations · p. 15
• Quantitative evaluation uses constructed descriptions and dictionary entries, with no human-participant experiment or participant-level trial data reported.Informal trial feedback is described separately in Section 7.Source: Declarations · p. 16
• Supplementary materials include evaluation queries, rankings, processing times, analysis scripts, prompts, and scoring rules, but not the source dictionary, illustrations, full annotations, or private API logs.End-to-end replication requires access to the dictionary corpus and model services.Source: Declarations · p. 16

Appendix A Statistical Methods

• The paired accuracy difference is (w −l)/n, with exact two-sided testing conditioned on discordant queries; correctness ties remain in the accuracy denominator.Here, w and l count queries correct only for the first and second method, respectively.Source: Appendix A · p. 16
• Difference intervals use separate 97.5% two-sided Clopper–Pearson intervals for win and loss probabilities, combined for at least 95% simultaneous coverage.Holm correction controls the four displayed exact tests as one comparison family, not the broader development history.Source: Appendix A · p. 16
• These tests and intervals characterize uncertainty conditional on the evaluation dataset, not dependence from shared dictionary content or representative sampling of users.Source: Appendix A · p. 16

Appendix B Experimental Configuration and Model Usage

• Configuration tables record requested and returned API model identifiers, but those strings do not independently verify fixed model weights.Endpoints may be mutable; disabling application caches does not exclude provider-side prompt caching, and supplementary configuration documents record evaluation settings.Source: Table B1 · p. 16, Appendix B · p. 17
• Table B2 reports API token usage for all 1,500 calls, with output counts including reasoning tokens counted within completion usage and no monetary conversion.Source: Table B2 · p. 17

Appendix C Supplementary Retrieval Results

• Table C1 reports recall and final target-hit counts and percentages at recorded cutoffs using the complete 500-query denominator.Source: Table C1 · p. 17
• Table C2 lists the 13 queries whose final target rank exceeds nine; absent targets are outside the 60-entry candidate pool, while targets beyond 20 retain their recall rank.Source: Table C2 · p. 17
• Table C3 partitions all 500 queries by Unicode-character length, reporting hit counts, percentages, and Hit@1 intervals for post hoc comparisons across different queries.Source: Table C3 · p. 17