PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond BLEU: A Case for Redefining Sign Language Translation Benchmarks

Oline Ranum, Edward Fish, Simon Hadfield, Richard Bowden

arXiv:2609.03734v1cs.CLcs.AI

TL;DR

The paper asks whether BLEU-4 reliably measures sign understanding in low-resource multimodal SLT, where target-side regularities can substitute for visual grounding. It introduces an open-weight-LLM QA protocol for salient content preservation and validates it against human rankings. The protocol changes the field’s picture: gloss-free systems are largely indistinguishable on Phoenix-2014T, while the gloss-supervised system leads by 9.3 QA points despite BLEU-4 ranking it second.

  • Problem

    BLEU-4 may not reliably measure sign understanding because low-resource SLT permits target-language regularities and spurious correlations to influence scores.

  • Method

    The paper develops an open-weight-LLM QA protocol that extracts salient content, generates questions and distractors, and evaluates content preservation against human rankings.

  • Results

    The five gloss-free systems are within noise on Phoenix-2014T, while the gloss-supervised system leads by 9.3 QA points where BLEU-4 ranks it second.

  • Takeaways & Limitations

    Salient-content QA provides a different view of SLT performance by emphasizing content transfer and reducing reliance on target-side structures that inflate BLEU-4.

  • Takeaways & Limitations

    The protocol’s scope remains tied to current content-transfer benchmarks; as SLT matures, more detailed benchmarks will be needed.

Abstract

from arXiv · show

BLEU-4 is the standard metric for evaluating sign language translation (SLT), but spoken-language metrics may not adequately reflect sign language proficiency. The multimodal, low-resource context of SLT allows models to exploit spurious correlations and spoken-language priors, rather than learning stronger sign representations. In this paper, we evaluate the relationship between spatio-temporal understanding and BLEU-4 across six SLT models on Phoenix-2014T and CSL-Daily, showing that gains in BLEU-4 are not on their own evidence of better sign language understanding. This work introduces an alternative inspired by language-learning assessment, using an open-weight-LLM QA protocol that measures salient content preservation. It aligns more closely with human rankings and is six to seven times more paraphrase-invariant than BLEU-4. Applied to SLT, this protocol targets content transfer, is more robust to train-test overlap, and gives a different picture of the field: the five gloss-free systems are largely within noise of one another on Phoenix-2014T, while the gloss-supervised system stands 9.3 points higher, a gap invisible to BLEU-4.

1. Introduction

SLT evaluation relies on BLEU-4, but low-resource multimodal conditions and non-isomorphic source-target mappings can let systems exploit target-language regularities instead of visual signing. The paper therefore proposes evaluating whether salient content is preserved, rather than only whether reference phrasing is reproduced.

  • Motivation: BLEU-4 measures reference n-gram reproduction, which may reward spoken-language priors while overlooking the visual language signal.SLT datasets cover narrow linguistic scopes, making target-language regularities particularly exploitable.
  • Motivation: Signing encodes information simultaneously and densely, while spoken-language grammatical elements may be absent, creating many valid translations and reducing BLEU’s reliability.The source-target mapping is therefore highly non-isomorphic.
  • Contribution: The paper shows that BLEU-4 can retain substantial scores after strong input corruptions and disproportionately reward components weakly connected to visual language.These findings motivate reassessing whether BLEU gains indicate stronger sign understanding.
  • Contribution: The proposed QA-based framework evaluates salient content preservation, is validated against semantic shifts and human judgment, and is applied to six SLT models on two benchmarks.The framework is designed to assess comprehension-like content transfer rather than exact phrasing.

2. Background & Related Work

SLT spans recognition, production, and translation, but its principal benchmarks are limited and controlled, while BLEU remains contested as an evaluation metric. Recent work motivates alternatives including multimodal-aware metrics and LLM-based QA evaluation.

  • SLT Scope: Sign language modeling covers recognition, production, and translation over video or pose inputs with gloss or text supervision.Glosses are written labels for individual signs and capture only a limited portion of signed content.
  • SLT Scope: Early SLT systems generated text through gloss sequences, whereas newer gloss-free systems map visual inputs directly to text with end-to-end transformers and pretrained language models.This contrast frames the evaluation challenge across different supervision regimes.
  • Benchmarks: Phoenix14T contains 8,257 German Sign Language videos from 9 interpreters, while CSL-Daily contains 20,654 videos covering 2,000 daily-life phrases.Both datasets are controlled, limiting domain variability, signer diversity, and naturalness.
  • Benchmarks: The study evaluates six state-of-the-art models on Phoenix14T and CSL-Daily, including five gloss-free systems and one gloss-supervised system.The benchmarks are used partly because they provide gloss annotations for attribution analysis.
  • Evaluation Alternatives: Prior work challenges BLEU through multimodal metrics, evidence of hallucination from reduced visual dependence, and findings that training-like test examples inflate BLEU.LLM-based QA frameworks have also shown competitive alignment with human judgment in translation evaluation.

3. Preliminaries: Correlations and Attributions of BLEU-4 in SLT

Analyses of corruption robustness and POS attribution show that BLEU-4 can preserve scores without the visual signal and allocate substantial credit to function words with limited correspondence to signing. These patterns make current BLEU differences difficult to interpret as sign proficiency.

  • Input Corruption: BLEU-4 retention under spatial and temporal corruption is used to test whether systems depend on visual information.The corruption conditions include masking hands or face, shuffling frames, and adding Gaussian noise.
  • Input Corruption: Replacing the visual representation with noise leaves every model at 1.2 BLEU-4 points or below, establishing a noise floor.Despite this floor, masking articulators or shuffling frames often leaves scores far above it.
  • Input Corruption: 81%, 83%, and 80% of baseline remain for CiCo, SignCL, and GFSLT-VLP respectively when both hands are masked on Phoenix-2014T.Face masking leaves every model at 53–68% on Phoenix-2014T and 63–86% on CSL-Daily.
  • Ranking Implications: 7.3±0.9 BLEU-4 points remain for the five gloss-free systems under both-articulator masking, nearly three times the 2.5-point baseline spread separating the six systems.Thus, performance obtained while reading none of the signing exceeds the margins used to rank systems.
  • POS Attribution: 37% of BLEU-4 is attributable to function words, despite their much lower frequency in gloss than in text for both Phoenix-2014T and CSL-Daily.The attribution is 3.65 points for German adpositions and 1.81 points for Chinese particles.

4. Implications for SLT Evaluation

Because function words and surface order can survive loss of visual information, the paper argues for metrics centered on salient content preservation. Its proposed framework is intended to tolerate paraphrase, discount target-side structure, and probe comprehension through content questions.

  • Why BLEU Falls Short: BLEU-4 can reward structures with limited or absent correspondence to the visual signal by more than the margin separating current systems.Function words are especially recoverable from concentrated target-side vocabularies, unlike broad, long-tailed content words.
  • Why BLEU Falls Short: Across both datasets and every corruption condition, coverage is retained better than 4-gram precision, while function-word coverage exceeds content-word coverage.The surviving output therefore predominantly reflects what target-side priors can supply.
  • Proposed Direction: A salient-content metric can tolerate paraphrasing, discount surface structure that survives visual corruption, and interpret occlusion-related drops as lost content.These properties shift evaluation toward what the translation conveys rather than exact wording.
  • Proposed Direction: Receptive sign-language assessment motivates querying conveyed content and checking whether responses recover the input’s salient information.This perspective defines the paper’s central evaluation questions about scalable measurement and target-side regularities.
  • Diagnostic Analyses: Figure 2 compares gloss and text POS distributions with BLEU-4 attribution, while Figure 3 tracks content and function markers by retained word order and coverage.Together, the figures separate what survives corruption from how it is scored.

5. Measuring Content Preservation

The protocol evaluates whether translations preserve salient content rather than reproduce reference wording, using LLM-generated, quality-controlled questions. Validation tests wording invariance, meaning sensitivity, and agreement with human judgments across multiple benchmarks.

  • Evaluation Protocol: The QA metric scores a candidate by the fraction of reference-derived multiple-choice questions it answers correctly from the candidate alone.Questions are generated from semantic content units and filtered for answerability, non-world-knowledge dependence, and ambiguity.
  • Evaluation Protocol: Question-bank size scales with sentence complexity, yielding 3.2 questions per OpusParcus reference versus 11.1 for PAWS-X.The protocol extracts content across entity, action, attribute, quantity, time, location, relation, negation, and polarity categories.
  • Evaluation Protocol: 88.6% and 94.6% of content lemmas are covered by QA banks for PHOENIX14T and CSL-Daily, respectively.Coverage is the fraction of a reference’s content lemmas reappearing in its generated QA items.
  • Validation: QA is more wording-invariant than BLEU-4, reaching an SNR of 12.5 versus 2.1 on WMT19.On WMT21, the corresponding SNR values are 23.0 for QA and 3.1 for BLEU-4.
  • Validation: Across OpusParcus quality bands, QA scores rise from 9 to 72 while BLEU-4 rises from 9 to 14.The metric scores are compared against human-rated similarity bands from low similarity to clean paraphrase.
  • Validation: QA awards meaning-flipped PAWS-X pairs 55.7% versus 95.4% for true paraphrases, testing sensitivity to semantic changes under similar wording.The comparison uses paired sentences whose surface overlap remains high despite meaning inversion.
  • Validation: QA matches or exceeds BLEU-4 in Kendall concordance with human MT-system rankings in every evaluated campaign.The clearest separation occurs in 2013–2015, when systems are weak enough for content transfer to differ.

6. QA-based Evaluation for SLT

The QA protocol evaluates salient content preservation and robustness, offering a more stable and less overlap-sensitive view of SLT than BLEU-4. On Phoenix-2014T, it separates the gloss-supervised system from five gloss-free systems that BLEU-4 does not distinguish clearly.

  • Spoken-language validation: QA outperforms BLEU-4 across SNR, paraphrase-quality correlation, and meaning-inversion AUC in spoken-language robustness tests.SNR measures invariance, Spearman correlation measures agreement with human paraphrase-quality ratings, and AUC measures separation of paraphrases from meaning-flipped pairs.
  • Efficiency and stability: 22–24 minutes are required for QA-bank construction per benchmark, while evaluating an additional system costs roughly 2–3 GPU minutes.Bank construction includes content extraction, two QA-generation passes, and three quality-control gates; answering is up to an order of magnitude cheaper.
  • Efficiency and stability: σ = 0.17–0.34 percentage points on Phoenix-2014T and 0.11–0.26 on CSL-Daily across ten regenerated banks, versus bootstrap half-widths of 2.4–2.7 and 1.4–2.0 points.Question-bank regeneration contributes substantially less variation than which test sentences are sampled, and a QA gap below roughly half a point is within run-to-run noise.
  • Exposure to target-side regularities: 37% of BLEU-4 comes from function words, whereas QA allocates 95% of Phoenix-2014T credit and over 99% of CSL-Daily credit to content.Function-word recall remains relatively resilient under visual corruption, creating a pathway for BLEU-4 gains without stronger sign comprehension.
  • Exposure to target-side regularities: 32% per likeness point is BLEU-4’s sensitivity for the most training-like Phoenix-2014T sentences, compared with 6.5% for QA.Across Phoenix-2014T and CSL-Daily, BLEU-4 accelerates as references resemble training targets, while QA responds much less.
  • Saliency-oriented system ranking: 9.3 points separate the gloss-supervised system from the gloss-free systems on Phoenix-2014T, while all ten gloss-free confidence-interval pairs overlap within 3.6 QA points.The separation is invisible to BLEU-4, which ranks the gloss-supervised system second; QA accuracy ranges from 52.6 to 65.5% on Phoenix-2014T and 11.6 to 61.6% on CSL-Daily.

7. Conclusion

The paper argues that BLEU-4 differences do not reliably indicate better sign understanding, whereas QA-based content preservation reveals a distinct evaluation picture. The authors therefore advocate measuring salient, grounded discourse content until more detailed benchmarks mature.

  • BLEU-4 differences at the current state of the art are not evidence of better sign understanding.BLEU-4 rewards target-side words and its corruption-surviving credit exceeds the 2.5-point spread separating the reproduced systems.
  • QA evaluation is stable, content-focused, and more resistant to the factors that inflate BLEU-4.It places 95% of credit on content and remains within half a point across reevaluations at 2–3 GPU-minutes per additional system.
  • The five gloss-free systems cannot be told apart on Phoenix-2014T, while the gloss-supervised system leads by 9.3 QA points.BLEU-4 ranks the gloss-supervised system second, so the QA result changes the field’s apparent ordering.
  • The paper recommends focusing on recovery of salient, grounded discourse content until content transfer saturates and more detailed benchmarks replace it.This is presented as the authors’ interim focus as SLT matures.

8. Supplementary Material

The supplementary material documents the datasets, reproduced checkpoints, preprocessing, corruption procedures, and evaluation implementation. It also specifies how visual perturbations and target-side replacement are applied across model families and datasets.

  • Datasets: The experiments use Phoenix-2014T and CSL-Daily, controlled SLT benchmarks with limited domain variability, signer diversity, and naturalness.They are used because gloss annotations provide a proxy for signed content in attribution analysis.
  • Models and checkpoints: All six systems are reproduced from released checkpoints rather than retrained, with five gloss-free systems drawn from a common benchmark pipeline.One checkpoint per model and dataset is used with standardized data and evaluation settings.
  • Evaluation setup: Retention is reported relative to each model’s own undistorted baseline after inference-time corruption over the full test split.This makes architectures with different absolute scores comparable.
  • Corruptions: Spatial masking removes face or hand pixels using segmentation masks and is repeated with filled bounding boxes to eliminate residual shape cues.The segmentation masks preserve the outline of the removed articulator, motivating the harsher bounding-box variant.
  • Corruptions: Noise replacement removes usable visual evidence, so remaining scores measure target-side priors alone.The replacement point differs between gloss-free and other model families because their architectures expose different interception points.

B.4. Retention under both metrics

Under QA, the two datasets show sharply different dependence on visual channels, while BLEU-4 attribution reveals substantial credit for function words and concentrated target-side vocabulary. The section also defines implementation and uncertainty procedures for the metric analyses.

  • Retention: 64–80% of QA content transfer remains after masking both hands on Phoenix-2014T, versus 4–16% on CSL-Daily.Face masking retains 62–70% on Phoenix-2014T and 40–68% on CSL-Daily, while masking both articulators reduces Phoenix-2014T to 16–35%.
  • Retention: 8–15% and 1–6% remain under the two noise floors, representing residue attributable to target-side priors alone.The datasets reverse which articulator is load-bearing under fixed architectures, indicating that grounding depends on benchmark affordances.
  • BLEU-4 attribution: A concentrated function-word vocabulary lets n-gram matching obtain substantial credit from fewer than one hundred types, while visual content is broader and rarer.The paper argues that this asymmetry makes BLEU-4 vulnerable to target-side priors.
  • QA protocol: The QA pipeline generates content units, multiple-choice questions, repaired distractors, quality-control gates, and passage-only answers.Question correctness is evaluated at the instance level because questions sharing a reference and candidate are correlated.
  • Uncertainty: Confidence intervals cover finite-test-set variation with the question bank fixed, while bank regeneration is reported separately.On a common standard-error basis, the instance term is three to nine times the bank term, depending on the system.

F.1. Evaluation of content unit extraction

The supplementary evaluation examines whether extracted content units and generated questions are grounded, answerable, and useful for scoring. Human checks find generally strong extraction and high judge–model agreement, while also exposing occasional over-crediting of vague translations.

  • Content units: A content unit is a reference span naming an isolated entity, action, attribute, quantity, time, location, relation, negation, or polarity.Articles, copulas, and conjunctions are not treated as standalone units.
  • Content units: About 80% of extracted spans receive an appropriate size assessment across 50 spans from four datasets and three languages.The evaluation reports some over- and under-segmentation, with undersegmentation appearing in the longest references.
  • Question validation: A usable question must test the selected unit, be answerable from the reference, assume only stated relations, and be well formed.The judge evaluates all four properties before an item is admitted.
  • Question validation: The distractor procedure separates clearly wrong, also-correct, synonymous, wrong-kind, and implausible options, while retaining a not-stated sentinel.This prevents an item from being solvable merely because distractors are obviously irrelevant.
  • Human agreement: 92.0% agreement (κ = 0.88) is observed between one judge and the model on system translations.The judge answered 13 of 25 Phoenix-2014T items and 3 of 25 CSL-Daily items, reflecting different answerability rather than different agreement.
  • Human agreement: All four judge–model disagreements were model over-credits in which the judge selected not stated.The model sometimes reproduced a pipeline key when the translation only vaguely expressed the information.
  • Content coverage: Gold answers emphasize content, carrying 66.6% of Phoenix-2014T reference content lemmas versus 51.9% of function lemmas.Per-POS scoring mass is concentrated on numerals, adjectives, and nouns, with pronouns handled through participant extraction.

H. Per-language Validation Results

Across seven PAWS-X languages, QA consistently outperforms BLEU-4 on meaning-flip discrimination, with the largest advantages where BLEU-4 is weakest.

  • +0.171 QA advantage appears in English, compared with +0.250 and +0.255 in Japanese and Korean.BLEU-4 is near chance in Japanese and Korean, with scores of 0.545 and 0.534.
  • QA correlates more strongly than BLEU-4 with human meaning judgments in OpusParcus, reaching ρ = 0.53 versus ρ = 0.12 in Russian.The passage reports per-language correlations ranging from these values across the evaluated languages.
  • QA wins in all 7 PAWS-X languages on meaning-flip discrimination.The table defines ROC-AUC as the probability that paraphrases score above flipped pairs, with 0.5 indicating blindness to the flip.

I. Run-to-Run Stability: Sources and Per-System Numbers

Repeated QA evaluations vary mainly because deliberate question sampling changes the admitted question bank, while the scoring stages are highly reproducible and preserve system ordering.

  • 71.7% of admitted Phoenix-2014T items and 77.9% of CSL-Daily items recur between two runs.The corresponding Jaccard similarities are 0.56 and 0.64.
  • Question generation repeats identically for only 57.6% and 62.8% of calls, making deliberate sampling the dominant variability source.Greedy extraction, distractor generation, and answering are substantially more reproducible; batching contributes secondary variation.
  • The modal QA system ordering is reproduced in every repeat on both benchmarks, with mean Kendall τ = 1.00.This reproducibility holds despite overlapping confidence intervals for the closest Phoenix-2014T pair, SignCL and GFSLT-VLP.
  • On CSL-Daily, the admitted set changes by one reference in two of ten repeats, moving BLEU-4 by at most 0.01 points.QA spread therefore arises from which questions pass the gates rather than from instability in scoring.

J. Training-Likeness: Exclusions and Cumulative View

The training-likeness analysis estimates metric sensitivity while excluding degenerate windows, then examines cumulative score retention across increasingly training-like test instances.

  • Training-Likeness Estimator: The local sensitivity estimator regresses per-instance score on training-likeness and reference length, using βm(ℓ) as the fixed-length sensitivity.The sliding window spans 12 likeness points and advances by 2.
  • Metric Interpretation: Sentence BLEU-4 is used as the local sensitivity normalizer, differing from reported corpus BLEU-4 by 1–2 points.The distinction matters because corpus BLEU-4 is not a simple mean and its normalization shifts with n-gram statistics.
  • Exclusions: Exact training-sentence copies and unsupported windows are excluded from local sensitivity curves rather than interpolated.Excluded cases include ℓ = 100 references, windows with fewer than 40 instances, and windows where one likeness value exceeds half the mass.
  • Cumulative View: The cumulative analysis sorts test instances from least to most training-like and plots Rm(f) = Cm(f)/Cm(1) after within-model normalization.This measures how much of each model’s published score is accounted for by nested prefixes of the test set.
  • Per-instance Distributions: QA and sentence BLEU-4 agree at corpus level but diverge across instances: BLEU-4 clusters near zero, while Phoenix-2014T QA spans the range with a mode at 100.QA = 0 occurs in 7–10% of Phoenix-2014T instances versus 28–67% on CSL-Daily.
Loading 2609.03734v1…