PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Term Evaluation in Machine Translation: Variation Matters

Nicolas Dahan, Ziqian Peng, François Yvon, Rachel Bawden

arXiv:2609.08779v1cs.CL

TL;DR

Terminology evaluation in MT often treats one target form as correct, penalizing the valid variation that human translators use. This paper evaluates English–French scientific MT with glossary accuracy, consistency, and a cross-term variation diagnostic, finding that glossary constraints improve conventional scores but suppress valid variation. The authors therefore support variation-aware evaluation that checks whether target-side variation mirrors source-side variation.

  • Problem

    Existing terminology metrics penalize acceptable variation and cannot assess non-glossary variants or whether preferred-term–variant relationships are preserved across translation.

  • Method

    The study analyzes two English–French NLP scientific corpora translated by four MT systems using glossary-based accuracy, translation consistency, and cross-term variation evaluation.

  • Results

    MT systems produce less target-side variation than human translators; consistency rankings vary by metric, and glossary constraints improve accuracy and consistency while degrading cross-term variation.

  • Takeaways & Limitations

    Variation-aware evaluation should condition consistency penalties on whether target-side variation mirrors structurally legitimate source-side variation.

  • Takeaways & Limitations

    The analysis is limited to English–French NLP scientific text, two corpora, medium-size 8B–22B models, and zero-shot evaluation.

Abstract

from arXiv · show

Terminology evaluation in machine translation (MT) usually assumes a single correct target form per source term. However, human translators routinely introduce variation that current metrics penalize as inconsistency. We examine how to account for this variation in document-level MT evaluation of English-French scientific translation, combining glossary-based accuracy, translation consistency, and a new cross-term variation (CTV) diagnostic measure that tests whether variation relationships are preserved across languages. Based on analyses of two parallel corpora, translated by four MT systems, we find that (1) MT systems generate less target-side variation than human translators; (2) transfer patterns strongly depend on the variation type; (3) consistency rankings vary with the choice of metric; and (4) constraining MT with a glossary improves accuracy and consistency but degrades CTV by suppressing valid variation. We argue for variation-aware evaluation that conditions consistency penalties on whether target-side variation mirrors source-side variation.

1 Introduction

Terminology evaluation in scientific MT must account for valid terminological variation rather than assuming one correct target form. The paper introduces complementary metrics for accuracy, consistency, and cross-term variation, finding that glossary constraints improve conventional scores while suppressing valid variation.

  • Motivation: Terminological variation includes acronyms, reductions, lexical substitutions, and morphosyntactic reformulations of a concept.These variants arise naturally through stylistic alternation, register adaptation, and avoidance of repetition in specialized discourse.
  • Evaluation gap: Existing glossary-based evaluation cannot assess non-glossary variants that arise dynamically in running texts.Simply enlarging a glossary cannot exhaustively enumerate reorderings, reductions, and context-dependent acronyms.
  • Approach: The study evaluates English–French scientific MT using term-level accuracy, translation consistency, and cross-term variation across two NLP corpora translated by four MT systems.Cross-term variation tests whether relationships between preferred terms and variants are preserved across languages.
  • Findings: Human translators introduce more target-side lexical variation than MT systems, while variation types differ in how consistently they transfer across languages.Reductions transfer consistently, whereas acronym behavior depends strongly on the corpus and translation guidelines.
  • Findings: Consistency measures can produce divergent system rankings because their definitions differ.This makes metric choice consequential when comparing MT systems.
  • Implication: Glossary constraints improve accuracy and consistency but degrade cross-term variation by suppressing acceptable target-side variation.The results motivate evaluation that conditions consistency penalties on whether target-side variation mirrors source-side variation.

2 Related Work

Prior terminology metrics largely reward uniform translation, although specialized discourse contains meaningful variation and human translations often preserve or increase it. Existing neural and glossary-based methods therefore miss document-level term relationships or penalize acceptable diversification.

  • Existing metrics: Segment-level neural metrics estimate overall quality but do not track per-concept cross-document consistency or preferred-term–variant relationships.Their semantic focus rarely penalizes valid variation, but it also limits diagnosis of terminological inconsistency.
  • Glossary methods: Glossary-based methods measure whether MT produces expected target terms, using exact, partial, lemma-based, and document-level occurrence matching.These methods remain tied to glossary entries and therefore have limited coverage of non-glossary variants.
  • Consistency methods: Consistency metrics reward reuse of a dominant translation and assume that every occurrence of a term should receive the same translation.This assumption penalizes stylistic alternation that may be deliberate and well motivated in scientific writing.
  • Linguistic evidence: Terminological variation is a fundamental property of specialized discourse, and translation studies report that human translations broadly preserve or accentuate source-side variation.Prior English–German evidence also finds fewer variant translations in MT than in human translation and post-editing.

3 Methodology

The methodology extends terminology analysis beyond glossary entries by detecting, labeling, aligning, and classifying source–target term occurrences, including dynamic variants. It combines concordancer-based detection with alignment and a targeted LLM fallback to produce annotated term pairs.

  • Pipeline: The pipeline processes parallel documents through term detection, source-side variation labeling, source–target alignment, and target-side labeling.It canonicalizes occurrences by removing inflectional and other non-terminological variation.
  • Term detection: Static and dynamic variant detection expands coverage beyond exact glossary entries through lexical, syntactic, morphological, and contextual transformations.The concordancer uses offline transformation rules and runtime dependency patterns.
  • Variation labeling: Source occurrences receive variation labels relative to preferred terms using the Cárcamo et al. typology.Glossary-listed forms are labeled with few-shot prompting, while non-lexicalized matches inherit labels from the formation pattern that triggered detection.
  • Coverage: Exact-form matching covers about 62–63% of term occurrences, while lemma matching raises coverage to about 82–87%.Additional concordancer matches are dominated by lexical and reduction variants.
  • Alignment: Bertalign performs sentence alignment and SimAlign assembles target spans for each source term occurrence.When span similarity falls below 0.8, ChatGPT replaces the aligned span; validation found 96% accuracy across 100 sampled alignments.

4 Evaluation Framework

The evaluation framework measures term-level accuracy, translation consistency, and cross-term variation as complementary views of terminology handling. Its diagnostic design distinguishes uniformity from source–target variation transfer and does not by itself assess semantic translation quality.

  • Evaluation dimensions: The framework evaluates term-level accuracy, translation consistency, and cross-term variation as three complementary measurements.CTV tests whether variation relationships between preferred terms and variants are preserved in translation.
  • Term-level accuracy: Exact Match measures the proportion of term occurrences whose translations match glossary entries.It applies to occurrences whose canonical forms map to preferred or alternative glossary terms.
  • Translation consistency: Translation consistency evaluates the target translations of repeated canonical forms using TCR-EM, TCR-T, and TCR-PR.The metrics differ in whether they require glossary agreement, consecutive agreement, or agreement with the first occurrence.
  • Translation consistency: The choice of consistency definition can change system rankings because TCR-EM, TCR-T, and TCR-PR operationalize consistency differently.TCR-EM is more restrictive because it requires both glossary accuracy and consistency, whereas TCR-T and TCR-PR focus on uniformity.
  • Cross-term variation: CTV measures whether source variation types such as GRAPH, EXP, RED, and LEX transfer to corresponding target forms.It evaluates cross-term relationships rather than treating each preferred term or variant independently.
  • Interpretation: The framework is diagnostic rather than a direct measure of translation quality.A translation can be consistently wrong or inconsistently appropriate, so consistency and accuracy do not fully determine quality.

5 Experimental Setup

The experiments use two English–French NLP-domain parallel corpora, a manually curated bilingual glossary, and four multilingual LLM-based MT systems. Each system is evaluated without terminology guidance and with glossary-preferred terms injected into the prompt.

  • Corpora: The analysis covers the ParaNLP and IWSLT2023 English–French parallel corpora.ParaNLP contains 6,272 parallel sentences from 32 papers, while IWSLT2023 contains revised transcripts and translations of 10 ACL 2022 presentations.
  • Terminological resource: The terminological resource contains 1,642 NLP concepts with expert-validated preferred English and French realizations.The glossary was manually curated from domain-specific resources and published by INIST.
  • Models: Four multilingual models are evaluated: Llama3.1-8B-Instruct, Qwen3-8B, EuroLLM-9B-Instruct, and EuroLLM-22B-Instruct.The human translation is used as the reference condition.
  • Scope: GRAPH and MORPH/COMBO variants have constrained evaluation coverage because source-to-target mappings are variable or less predictable.MORPH and COMBO are excluded from some transfer analyses because they involve grammar-dependent or simultaneous transformations.
  • Translation settings: The evaluation compares baseline translation with glossary-guided translation using preferred terms and expected translations in the prompt.The comparison diagnoses whether systems exploit supplied glossary information.

6 Analysis

The analysis shows that terminological variation differs across corpora and variation types, while current metrics can penalize appropriate variation. Glossary prompting improves accuracy and consistency but lowers CTV, revealing a trade-off between uniformity and variation transfer.

  • Variation patterns: Concepts average 1.37 distinct surface forms in ParaNLP and 1.14 in IWSLT2023, with 77.7% and 88.3% realized by a single form.Variation is concentrated among frequent concepts, while preferred terms account for 72–75% of occurrences.
  • Variation patterns: RED and GRAPH each account for 7–9% of source variation, LEX for 6–8%, and MORPH, EXP, and COMBO together for less than 3%.Because 38–43% of concepts are hapaxes, scores requiring multiple occurrences miss much terminology.
  • Reference and MT: Reference translations show more target-side variation than MT in ParaNLP, where 34.4% of concepts have more target than source realizations.The corresponding IWSLT2023 rate is 15.8%, indicating register-dependent variation patterns.
  • Reference and MT: Human references use target-side lexical variants for 24.98% of IWSLT2023 preferred-term cases and 15.68% of ParaNLP cases.For example, reported speech receives both parole rapportée and discours rapporté in one ParaNLP sentence.
  • Variation transfer: Variation transfer depends strongly on type: RED preservation reaches 83–86%, while GRAPH preservation ranges from 50–60% in ParaNLP to 14–29% in IWSLT2023.In IWSLT2023, source acronyms are often spelled out as French phrases rather than retained as acronyms.
  • Metric coverage: The three dimensions together evaluate 96.8% of ParaNLP occurrences and 97.5% of IWSLT2023 occurrences, while CTV uniquely evaluates hapax variants through their relationship to preferred terms.EM coverage drops sharply for variants, especially MORPH and COMBO, and GRAPH coverage differs across corpora.
  • Metric behavior: Consistency metrics penalize contextually appropriate variation, such as the use of hauteur and mélodique for pitch across 14 occurrences.The penalty is especially pronounced for LEX and GRAPH terms compared with invariant terms.
  • Metric behavior: Each TCR variant ranks a different system first on ParaNLP, showing that consistency rankings depend on metric definition.The reference also has lower CTV in ParaNLP because its greater target-side variation creates more opportunities for structural mismatch.

7 Conclusion

The study shows that terminology evaluation must account for valid variation in scientific translation. Human translators produce more target-side variation than MT systems, while glossary constraints improve accuracy and consistency but suppress variation captured by CTV.

  • Human translators introduce more target-side lexical variation than MT systems on academic prose, and current consistency metrics penalize this standard practice.
  • Current metrics inadequately capture variation: glossary exact matching favors preferred terms, consistency penalizes acceptable diversification, and consistency rankings diverge across operationalizations.
  • Cross-term variation (CTV) recovers structural variation signal that glossary accuracy and consistency measures miss.
  • Glossary-constrained MT improves accuracy and consistency but degrades CTV by suppressing acceptable target-side variation preserved by the unconstrained baseline.

8 Limitations

The findings are scoped to English–French NLP scientific translation using two corpora and medium-size zero-shot LLMs, with several pipeline and coverage constraints. The analysis also documents variation-transfer behavior and a small residual set of unevaluated terms.

  • Scope boundaries: The experiments cover one language pair, one domain, two corpora, and medium-size 8B–22B LLMs evaluated under zero-shot conditions.The authors state that broader experimental coverage is needed to confirm generalizability.
  • Metric assumptions: CTV assumes each source variation type should be preserved, although legitimate cross-lingual adaptation may produce a different target type.The paper gives acronym expansion for target-language clarity as an example.
  • Evaluation coverage: Neural and reference-free metrics are not tested for terminological variation, leaving their document-level inconsistency-diagnosis capability open.The paper notes that these metrics commonly operate at segment level.
  • Pipeline constraints: Pipeline results depend on a closed LLM variation classifier, alignment tools, and language-specific heuristic resources, with reported classifier and alignment error rates.Manual validation reports 84% type accuracy and 11% recall for COMBO; an alignment audit estimates a 4% error rate.
  • Coverage and variation: CONCORDANCER extends recognition beyond exact and lemma matching, retrieving remaining variants and identifying out-of-glossary terms primarily through lexical and reduction variants.Exact-form and lemma baselines retrieve 61.9% and 82.2% of ParaNLP terms, and 62.7% and 86.6% of IWSLT2023 terms.
  • Variation transfer: Most concepts use one canonical surface form, but a long tail has multiple variants; transfer varies by type, with RED most stable and GRAPH highly corpus-dependent.Baseline NO-VAR preservation is 81–85% on ParaNLP versus 71–75% on IWSLT2023, while RED→RED is 83–86% across both corpora.
  • Variation transfer: Glossary constraints increase NO-VAR preservation while reducing NO-VAR→LEX transfer, selectively suppressing preferred-term variation.NO-VAR→NO-VAR rises to 90–93% on ParaNLP and 79–91% on IWSLT2023, while NO-VAR→LEX falls to 3–5% and 5–16%.
  • Residual evaluation gaps: Human references alternate between contextually appropriate equivalents, whereas baseline MT systems often repeat one form, and some terms remain unevaluated because they lack glossary or CTV relationships.Unevaluated occurrences comprise 3.2% of ParaNLP and 2.5% of IWSLT2023 occurrences.

A.8 TCR Metric Divergence Example

Consistency and variation metrics expose different aspects of terminology quality. Their scores vary by metric, corpus, system, and variation type, while glossary guidance improves EM and TCR but often lowers CTV.

  • TCR divergence: For 5 neural network occurrences, TCR-EM scores 0.40, TCR-PR 0.60, and TCR-T 0.50.
  • TCR divergence: On ParaNLP baseline, TCR-EM favors the reference at 82.99, TCR-PR favors Llama at 89.31, and TCR-T favors E9 at 87.93.Each TCR variant ranks a different system first.
  • CTV results: CTV rankings reverse across corpora: the reference leads IWSLT2023 at 86.67, while Llama leads ParaNLP at 74.63 and the reference scores 66.06.The paper attributes this asymmetry to target-side amplification.
  • CTV results: RED preservation is consistently high, whereas GRAPH and LEX coherence vary substantially by corpus and system.RED reaches 82–100% on IWSLT2023 and 81–87% on ParaNLP; LEX spans 50–86% and 57–72%, respectively.
  • Glossary-constrained evaluation: Base+terms raises MT EM by 10–17 points on IWSLT2023 and 6–10 points on ParaNLP, with the largest gains for NO-VAR terms.MORPH and COMBO remain at 0%.
  • Glossary-constrained evaluation: Base+terms raises NO-VAR TCR-PR to 91–95% on IWSLT2023 and 93–95% on ParaNLP, while variant-category gains are smaller.
  • Glossary-constrained evaluation: Base+terms lowers CTV for most MT systems, especially on ParaNLP from 72–75% to 67–69%, despite occasional GRAPH improvements on IWSLT2023.RED coherence decreases on IWSLT2023 for the reported systems.

B Manual Evaluation of LLM-based Variation Labeling

The LLM-based variation classifier performs reliably for most single-type distinctions but struggles with compound variations. This weakness has limited downstream impact because COMBO cases are rare.

  • Manual evaluation: The classifier achieves 84% overall type recall and 77% subtype accuracy across 49 evaluable annotated pairs.
  • Manual evaluation: GRAPH, LEX, RED, EXP, and MORPH each reach 100% type-level recall, with GRAPH and LEX subtype accuracy at 86% and 89%.
  • Compound variations: COMBO recall is 11%, with 1 of 9 gold compound instances recovered because the classifier predicts a single type instead of a combination.The systematic error involves acronym-plus-ellipsis cases labeled as pure GRAPH rather than GRAPH+EXP.
  • Compound variations: COMBO oversimplification lowers GRAPH precision to 67% because seven compound pairs are absorbed into the GRAPH class.
  • Error analysis: The remaining subtype errors are limited to two GRAPH1-to-GRAPH5 cases and two LEX1-to-LEX2 cases.
  • Overall assessment: The classifier is reliable for most encountered variation types, and COMBO errors have limited downstream impact because compound variations are a small dataset fraction.

C Manual Evaluation of Alignment Accuracy

A manual evaluation assessed whether automatically aligned target spans correctly realized source terms across the pipeline’s system outputs and human references. The aligner achieved high accuracy, with errors limited to isolated lexical-choice and span-boundary cases.

  • The evaluation sampled 100 alignments stratified across nine outputs, including four MT systems under two conditions and the human reference.Each MT condition contributed eleven alignments, while the reference contributed twelve; empty target spans were excluded.
  • 96% of sampled alignments were judged correct (96/100).Accuracy was 98% for baseline (43/44), 95% for base+terms (42/44), and 92% for the human reference (11/12).
  • The four errors comprised three lexical-choice errors and one span-boundary error.Examples included adjacent-word captures and omission of automatique from reconnaissance automatique de la parole.
  • No systematic failure mode was observed because all four errors affected single occurrences.Accuracy was also stable across variation types, with no error among ten sampled dynamic variants.
  • The pipeline uses Bertalign and SimAlign for alignment, with an LLM fallback for low-confidence spans.When assembled-span cosine similarity falls below 0.8, ChatGPT returns the term realization in the target sentence and replaces the SimAlign span.

E Reproducibility Statement

The reproducibility materials document the LLM prompts, model usage, inference hardware, computational budget, and supporting typology and evaluation tables. They cover alignment, translation, glossary-guided translation, and variation classification components.

  • The variation classifier and alignment-disambiguation step use OpenAI’s gpt-4.1-mini.The paper states that this usage is intrinsic to the contribution and described in the methodology.
  • All MT inference ran on a single NVIDIA H100 96GB GPU, requiring approximately 10 minutes across four systems, two conditions, and 42 documents.Inference used vLLM with pre-generated prompts; the labeling pipeline issued approximately 5,000 gpt-4.1-mini requests.
  • The supplementary materials include variation typology, classifier-label evaluations, alignment-error examples, and per-category evaluation tables.Tables cover EM accuracy, multiple TCR variants, CTV variation preservation, and term coverage under baseline and base+terms conditions.
Loading 2609.08779v1…