PaperlayerView exact revision on arXiv

Source-linked AI summary

DiscoSign: Discourse-Aware Text to Sign Language Gloss Translation

Vasileios Baltatzis, Mert Inan, Connor Gillis, Raja Kushalnagar, Lorna Quandt, Leah Findlater, Colin Lea

arXiv:2609.02796v1cs.CL

TL;DR

Existing text-to-sign systems largely translate sentences independently, leaving discourse coherence under-modeled. DiscoSign combines modular LLM-based processing with explicit discourse constraints and tailored metrics, and reports stronger discourse modeling than sentence-level and context-only approaches while retaining competitive single-sentence quality.

  • Problem

    Sentence-level text-to-sign systems lack mechanisms to maintain discourse-level consistency, while standard metrics do not assess pragmatic appropriateness or discourse coherence.

  • Method

    DiscoSign uses modular LLM-based translation with explicit state registries, programmatic constraints, ASL-focused instructions, and discourse-specific evaluation metrics.

  • Results

    Explicit discourse modeling substantially outperforms sentence-level and context-only approaches while maintaining competitive single-sentence gloss translation quality.

  • Takeaways & Limitations

    The framework establishes a systematic foundation for discourse-level text-to-sign gloss translation with metrics for spatial coreference, QAC appropriateness, and concept-gloss consistency.

  • Takeaways & Limitations

    The evaluation focuses on English-to-ASL gloss translation, and the Aesop’s Fables dataset lacks ground-truth ASL annotations for reference-based discourse evaluation.

Abstract

from arXiv · show

Sign language processing systems have traditionally operated at the sentence level, ignoring critical discourse phenomena fundamental to sign language comprehension. We introduce DiscoSign, a computational approach for discourse-aware text to sign language gloss translation grounded in linguistic research. We address three key phenomena within our modular Large Language Model (LLM)-based translation framework: (i) spatial coreference resolution, where entities maintain consistent spatial locations throughout discourse; (ii) Question-Answer Clauses (QACs), pseudocleft structures serving specific discourse functions; and (iii) concept-gloss consistency, ensuring stable mappings between English concepts and American Sign Language (ASL) signs. Traditional translation metrics fail to capture discourse-level quality, so we introduce a suite of novel evaluation metrics designed to assess each dimension of discourse coherence addressed by our framework. Experiments on sentence-level and discourse-level datasets show that our approach for discourse-aware processing significantly improves spatial consistency and entity tracking relative to sentence-only translation, while maintaining competitive single-sentence gloss translation quality. Our work establishes the first systematic framework for discourse-level text to sign language gloss translation with corresponding evaluation methodology.

1 Introduction

Sentence-level text-to-sign systems lack mechanisms for maintaining entity, concept, and rhetorical consistency across discourse. DiscoSign introduces a discourse-aware framework with explicit coherence constraints and evaluation components for spatial coreference, QAC usage, and concept-gloss mappings.

  • Motivation: Sentence-level systems translate each sentence independently, failing to track discourse-level consistency across entities, concepts, and rhetorical choices.These errors can propagate through downstream sign production because glosses serve as an intermediate representation.
  • Discourse phenomena: Spatial coreference requires entities to retain consistent spatial locations throughout discourse, while QAC restructuring depends on discourse context and information structure.ASL may use spatial indices such as IX-3p:i and IX-3p:j, and context-sensitive structures such as I WANT WHAT? COFFEE.
  • Discourse phenomena: Concept-gloss consistency requires repeated concepts to map to stable signs rather than alternating among different glosses across sentences.The paper illustrates this with COMPUTER, MACHINE, and TECHNOLOGY as competing mappings for “computer.”
  • Framework: DiscoSign presents the first systematic discourse-aware text-to-gloss framework, using explicit state registries and programmatic constraints for three discourse phenomena.The framework addresses spatial coreference resolution, pseudocleft usage, and concept-gloss consistency.
  • Contributions: The paper develops ASL-centric instruction design, novel discourse-level metrics, and ablation analyses covering spatial indexing, QAC appropriateness, and concept-gloss consistency.The instruction and evaluation components are designed to assess discourse coherence rather than only sentence-level translation quality.
  • Results: The reported gains hold for both proprietary and open-weight LLM backbones.The analysis examines individual and combined instruction components and their relative importance for translation quality.

2 Related Work

Prior sign-language NLP work addresses linguistic phenomena, LLM-based translation, and production considerations, but standard evaluation metrics inadequately capture discourse coherence and pragmatic appropriateness. DiscoSign responds with metrics tailored to discourse-level properties of signed languages.

  • Sign-language processing: Prior work studies spatial-agreement coreference, non-manual markers, signing-avatar production, and other linguistic structures in sign-language processing.These strands address complementary linguistic and system-level challenges.
  • LLMs for sign-language translation: Recent LLM approaches span gloss-free visual-to-text translation, vocabulary sharing, and semantically aware training strategies.The related work positions LLMs as an emerging direction in sign-language processing.
  • Evaluation: BLEU and ROUGE emphasize lexical overlap and miss visual-spatial and discourse properties such as flexible word order and omitted function words.chrF, COMET, and SilverScore address some word-order or semantic limitations but do not evaluate pragmatic appropriateness or discourse coherence.
  • Evaluation: DiscoSign addresses these evaluation gaps with novel metrics targeting discourse-level phenomena specific to signed languages.The proposed suite complements conventional translation metrics with discourse-focused assessment.

3 Discourse-Aware Sign Language Gloss Translation Framework

DiscoSign is a discourse-aware text-to-gloss framework that maintains cross-sentence coherence by coordinating spatial coreference, QAC usage, and concept-gloss mappings through explicit state and verification. Its modular design translates each sentence in one LLM call while carrying discourse registries forward as constraints.

  • Framework overview: DiscoSign introduces a discourse-aware framework for text-to-gloss translation that maintains cross-sentence coherence across three sign-language discourse phenomena.These are spatial coreference, Question-Answer Clauses, and concept-gloss consistency.
  • Spatial Coreference Module (SCM): Spatial coreference resolution tracks entities across sentence boundaries and preserves their assigned spatial indices and directional relationships.Alice and Bob, for example, remain associated with established locations such as IX-3p:i and IX-3p:j.
  • Integrated Processing: Each sentence is processed in a single LLM call using accumulated registries as hard constraints, followed by verification and registry updates for subsequent sentences.The unified state consists of spatial assignments S, concept-gloss mappings L, and QAC decisions Q.
  • Question-Answer Clause Module (QACM): QAC processing evaluates discourse context to determine when topic-comment restructuring is appropriate and avoids QACs at discourse beginnings or after genuine questions.The module records sentence-level QAC decisions and considers causal explanations, contrastive elements, and focal points.
  • Concept-Gloss Consistency Module (CGCM): Concept-gloss processing maps known concepts to previously registered glosses and assigns vocabulary-compliant glosses to novel concepts.Synonyms or paraphrases such as “car” and “vehicle” can therefore resolve to one consistent gloss, such as CAR.

4 Evaluation Suite

The evaluation suite introduces metrics for discourse coherence in text-to-sign gloss translation, including spatial consistency, QAC appropriateness, and concept-gloss consistency. It evaluates these dimensions across sentence-level and discourse-level datasets.

  • Novel metrics evaluate discourse-coherence dimensions that traditional translation metrics cannot capture.The suite provides a systematic evaluation of cross-sentence phenomena in text-to-sign language gloss translation.
  • Spatial Coreference Accuracy (SCA): SCA measures the proportion of spatial references passing consistency and coreference-correspondence checks.It excludes first definitions and evaluates directional-verb indices independently.
  • Question-Answer Clause Appropriateness (QACAp): QACAp evaluates whether inserted Question-Answer Clauses occur in linguistically suitable discourse contexts.The metric focuses on precision for used QACs, considering contextual markers and whether the sentence directly answers a genuine question.
  • Concept-Gloss Consistency (CGC): CGC measures whether repeated English concepts maintain consistent ASL gloss mappings throughout discourse.The metric checks prior occurrences and treats repeated concepts across multiple sentences as the evaluation set.
  • Evaluation datasets: The evaluation datasets include sentence pairs or texts, with annotations and discourse-level coverage identified for each dataset.Table 1 records average sentences per example, expert-annotated reference gloss availability, and whether discourse phenomena are present.

5 Experiments and Findings

Experiments evaluate DiscoSign on sentence-level and discourse-level datasets using baseline, context-aware, and explicitly discourse-modeled configurations. The framework preserves sentence-level quality while substantially improving discourse coherence metrics, with human evaluation supporting SCA and CGC validity.

  • Experimental setup: Experiments span sentence-level translation accuracy and discourse-level coherence assessment across three datasets, including Aesop’s Fables for multi-sentence narratives.The systems use sentence-only, context-aware, and proposed configurations with ASLLRP SignBank vocabulary constraints.
  • Sentence-level translation quality: The proposed framework achieves competitive or superior sentence-level performance on both ASL STEM Wiki and the licensed dataset.Lower absolute ASL STEM Wiki scores reflect scientific vocabulary requiring extensive fingerspelling.
  • Sentence-level translation quality: ASL STEM Wiki chrF: 54.8 vs 41.9 and COMET: 0.81 vs 0.73; licensed dataset chrF: 67.2 vs 65.2 and COMET: 0.90 vs 0.88 in back-translation.These improvements are reported alongside differences between gloss-level and back-translation evaluation.
  • Discourse-level translation quality: Explicit discourse modeling reaches SCA: 0.84, CGC: 0.97, and QACAp: 0.76, exceeding context-only gains from SCA: 0.29 → 0.48, CGC: 0.59 → 0.68, and QACAp: 0.70 → 0.72.Traditional chrF and COMET remain comparable despite differing discourse coherence.
  • Discourse-level translation quality: With Qwen3.6-35B-A3B, the framework improves SCA by +0.49 and CGC by +0.13 over the sentence-level baseline while again scoring highest on discourse metrics.Traditional metrics again separate the conditions poorly, and the qualitative example preserves both Weasel’s gloss and spatial index.
  • Ablation studies: Ablations show module-specific effects: disabling SCM drops SCA from 0.81 to 0.71, disabling CGCM drops CGC from 0.97 to 0.64, and disabling QACM lowers QACAp from 0.76 to 0.71–0.72.The full module combination provides the most balanced performance across discourse-specific metrics.
  • Context window and human evaluation: Increasing context improves discourse metrics, with SCA rising from 0.40 → 0.84 and CGC from 0.78 → 0.97, while chrF remains ∼42 and COMET ∼0.78.With one prior sentence, the structured framework exceeds the context-aware baseline with full access on SCA: 0.57 vs 0.48.
  • Context window and human evaluation: SCA correlates significantly with human ratings, while CGC shows significant combined correlation and QACAp correlations are non-significant.The evaluation used two ASL-fluent evaluators rating 32 stories and 152 sentences on 1–5 Likert scales.

6 Conclusion

DiscoSign establishes discourse-aware text-to-gloss translation as a novel task and provides a framework with specialized modules and evaluation metrics for three discourse phenomena. Experiments show explicit discourse modeling outperforms sentence-level and context-only approaches.

  • 6 Conclusion: DiscoSign formalizes spatial coreference, question-answer clauses, and concept-gloss consistency with specialized modules and corresponding evaluation metrics.The framework is presented as a foundation for linguistically authentic sign language translation systems.
  • 6 Conclusion: Explicit discourse modeling substantially outperforms both sentence-level and context-only approaches in discourse-aware text-to-gloss translation.The conclusion identifies this result as establishing a foundation for more linguistically authentic sign language translation systems.

7 Limitations

The framework has implementation, evaluation, and scope limitations, including dependence on LLM behavior, deterministic QAC decisions, and omission of non-manual markers. Its modular design and registry state support future alternatives and downstream NMM generation.

  • LLM instruction-following dependencies may limit reproducibility as model versions change.
  • The framework is implementation-agnostic, allowing modules to use rule-based systems, trained neural models, or human experts instead of LLMs.
  • Deterministic QAC trigger rules cannot represent contexts where both QAC and non-QAC renderings are natural.
  • Evaluation is limited to English-to-ASL gloss translation, although the considered phenomena occur across signed languages.
  • The Aesop’s Fables dataset lacks ground-truth ASL annotations, requiring automatic metrics and back-translation for discourse-level evaluation.
  • The framework omits non-manual markers such as facial expressions and prosody, while its registries contain information relevant to future NMM generation.

8 Ethical Considerations

The work emphasizes collaboration with Deaf and Hard-of-Hearing communities and describes datasets selected without personally identifiable or offensive material.

  • Developing sign-language systems without meaningful Deaf and Hard-of-Hearing community engagement risks producing technology that does not serve their interests.
  • The authors collaborated with signing-community members to ground the approach in authentic linguistic practices.
  • The study uses ASL STEM Wiki, a licensed dataset, and Aesop’s Fables, covering educational, scientific, and traditional narrative content.
  • The datasets contain no personally identifiable information or offensive material.

A Translation Prompts

The appendix specifies prompts for sentence-level, context-aware, and registry-based translation, with sequential processing, vocabulary constraints, and explicit discourse-state management.

  • A Translation Prompts: All conditions require structured JSON containing glosses, spatial mappings, directional-verb mappings, QAC usage, and concept-to-gloss mappings.
  • A Translation Prompts: Every gloss must come from the provided vocabulary, with fingerspelling as the only fallback; story sentences are processed sequentially.
  • A.1 Sentence-Level Baseline: The sentence-level baseline translates each sentence independently without prior context and with minimal instructions.
  • A.2 Context-Aware Baseline: The context-aware baseline adds previously translated sentences and ASL translations but provides no explicit guidance for using that context.
  • A.3 Proposed Approach: The proposed approach combines detailed discourse instructions, a configurable context window, and structured registries accumulated from prior outputs.
  • A.3 Proposed Approach: The spatial registry preserves entity-index assignments and constrains directional-verb source and target indices across sentences.
  • A.3 Proposed Approach: The QAC registry restricts rhetorical questions by discourse position and information structure, including causal explanation, topicalization, and emphasis.
  • A.3 Proposed Approach: The concept-gloss registry enforces stable mappings and vocabulary compliance, while post-processing verifies outputs and corrects registry violations.

B Human Evaluation Details

Human evaluation sampled Aesop’s Fables translations across automated score ranges and assessed discourse phenomena with ASL-fluent evaluators, while revealing agreement and ceiling-effect challenges.

  • Study Design: 32 stories comprising 152 sentences were sampled across low, medium, and high automated SCA, CGC, and QAC score ranges.
  • Study Design: Two ASL-fluent evaluators independently rated spatial coreference accuracy, concept-gloss consistency, and QAC appropriateness on 1–5 scales with N/A options.
  • Inter-Rater Agreement: SCA agreement was moderate, with weighted κ = 0.43, Spearman ρ = 0.49 (p < 0.001), and 64% of ratings within one point.
  • Inter-Rater Agreement: CGC and QAC ratings were near-ceiling, leaving insufficient variance for meaningful inter-rater agreement statistics.
  • Correlation Analysis: Story-level correlations were computed with Spearman statistics after mean aggregation because sentences within a story are not independent.
  • QAC Subjectivity: QAC appropriateness correlations were non-significant because rule-based triggers do not cover the broader range of acceptable native-signing judgments.
  • QAC Subjectivity: QAC presence agreement varied across evaluators, with ρ = 0.85 for one evaluator and ρ = 0.39 for the other.
  • CGC Ceiling Effects: CGC had significant combined correlation, ρ = 0.48 (p = 0.007), but near-ceiling human ratings limited its discriminative power.
Loading 2609.02796v1…