Source-linked AI summary
SalamandraTA at WMT 2026 Terminology Shared Task: Hard Examples Are Better Teachers
Xixian Liao, Maite Melero
TL;DR
Terminology-aware translation must follow prescribed glossary terms, but much glossary supervision is redundant because models already produce the required terms. The paper filters training data to retain model-glossary contradictions, builds that data through two synthetic pipelines, and evaluates the resulting public model in a document-level WMT26 submission. Hard-example selection raises term accuracy from 78.7% to 89.9% at fixed data volume, while the official system reaches 94.2% term success and 74.6 chrF++.
Problem
Terminology-aware translation must produce prescribed glossary terms, not merely a meaning-preserving translation.
Method
The paper filters glossary-annotated data to retain examples where the model’s unconstrained translation misses at least one required term.
Results
78.7% to 89.9% term accuracy is achieved by replacing easy examples with hard examples at identical data volume.
Takeaways & Limitations
Hard examples provide concentrated supervision for glossary adherence, and the resulting data supports the public SalamandraTA-7b-instruct v3.0 release and BSC submission.
Takeaways & Limitations
The controlled study covers one data source, three test directions, one data volume, and a single reference model for defining hardness.
Abstract
from arXiv · showhide
Terminology-aware translation asks for more than a correct translation: the output must use the exact terms a glossary prescribes. The standard recipe, fine-tuning on glossary-annotated translation pairs, hides an inefficiency: for most examples the glossary prescribes exactly what the model would have produced anyway, so they teach nothing about following a glossary. We therefore keep only the examples where the model's own translation contradicts the glossary. In a controlled study at fixed data volume, this selection alone raises term accuracy from 78.7% to 89.9%. The filtered data, built by a two-way synthetic pipeline on open models, is part of the instruction-tuning mixture of our public release SalamandraTA-7b-instruct v3.0, which, used exactly as released and wrapped in a document-level inference pipeline, forms the BSC submission to the WMT26 Terminology Shared Task Track 1. At the official WMT26 evaluation, our system achieves 94.2% term success at 74.6 chrF++, with only two of the twenty-two submissions outperforming it on both metrics. On last year's benchmark, it also surpasses our GRPO-based system, despite being trained solely with ordinary supervised fine-tuning.
1 Introduction
Terminology-aware translation must use prescribed terms, not merely preserve meaning. The paper argues that training should focus on examples where the model’s default translation contradicts the glossary.
- Motivation: Terminology-aware translation requires prescribed glossary terms even when a generic translation preserves the meaning.The paper illustrates this with a Spanish translation of “pull request.”
- Motivation: Glossary-annotated training is often redundant because the model already produces the prescribed term.The useful supervision is concentrated in examples where the glossary differs from the model’s default choice.
- Contribution: 78.7% to 89.9% term accuracy results when the share of hard examples rises from none to all at fixed training volume.The controlled experiment varies only the proportion of hard examples.
- Submission: The BSC submission uses SALAMANDRATA-7B-INSTRUCT v3.0 with terminology data built by a two-way synthetic pipeline and a document-level glossary pipeline.The model is used as released, with per-chunk glossaries and post-editing protection for prescribed terms.
- Contributions: The paper presents redundant-data reduction, synthetic terminology-data construction, and a document-level Track 1 submission as its three contributions.The first contribution reports an eleven-point term-accuracy gain at fixed data volume.
2 Related work
Prior work has imposed terminology constraints during decoding, training, or reward optimization. This paper instead selects already-labelled examples according to whether the model’s unconstrained output contradicts the prescribed term, using open synthetic-data pipelines.
- Terminology control: Terminology constraints have moved from lexically constrained decoding into training and reward optimization.The paper situates its approach after constrained beam search, annotated training, and GRPO-based optimization.
- Contribution: The paper’s novelty is selecting only training examples whose terminology the model gets wrong, rather than changing how the model is trained.This distinguishes the approach from the authors’ previous GRPO system.
- Synthetic data: The synthetic pipeline differs by running in both directions: expanding glossaries into documents and mining documents for glossaries.It is built entirely on openly licensed models and data.
- Data selection: Unlike prior selection approaches based on continuous difficulty scores, this filter uses binary, self-referential hardness from the model’s own unconstrained output.Selection is performed from an already-labelled pool without querying an annotator.
3 Data: two synthetic pipelines
The terminology data combines two synthetic pipelines: one generates parallel documents from glossary terms, while the other extracts aligned terminology from parallel text. Both retain examples that the model mistranslates.
- Glossary-to-document pipeline: One pipeline seeds Gemma-4-31B with bilingual medical glossaries and generates parallel texts around the terms.The glossary source is MeSpEn.
- Document-to-glossary pipeline: The second pipeline extracts aligned term pairs from EMEA sentence pairs and assembles the sentences into documents.Its documents combine consecutive sentences at multiple granularities.
- Data scale: Together the pipelines produce 33,615 instances covering 94 directed language pairs across 29 languages.The instances comprise 12,479 from MeSpEn and 21,136 from EMEA.
- Glossary-to-document pipeline: Generated documents group semantically similar hard-term seeds and require repeated main terms to receive identical translations.This mirrors terminology consistency requirements across document contexts.
- Hard-example filter: The shared filter translates candidates without terminology prompts and discards examples where every required term already appears.Only candidates with at least one missed term survive, and the filter is computed once against v2.0.
4 Controlled study: does hardness matter?
The controlled study isolates hard-example selection as the only changing factor and finds that concentrating training on glossary contradictions substantially improves term accuracy without reducing translation quality.
- Experimental design: The experiment varies the hard-example share from 0% to 100% while holding training data volume and other training conditions fixed.Six mixtures include 0%, 25%, 50%, 75%, and 100% hard examples plus a random baseline.
- Selection effect: 89.9% term accuracy with only hard examples versus 78.7% with none yields an 11.2-point gain at identical data volume.The difference is statistically significant, with 95% CI [+9.5, +12.8] and p < 0.0001.
- Dose of difficulty: Hard-example gains generally increase with difficulty, although the 50% versus 25% difference is within noise on average.The 25% mixture delivers 71% of the full effect, while the average difference is +0.3 points with p = 0.55.
- Direction-specific effects: The largest easy-to-hard gain occurs for en→ru, while en→es rises strictly with hard share and en→de saturates earliest.The gains are +15.7 points for en→ru and +10.8 for en→es; en→de reaches 93.0 with 75% hard examples versus 92.3 with pure hard examples.
- Translation quality: 48?
5 Document-level submission pipeline
The submission translates full documents chunk by chunk, narrows each chunk’s glossary to relevant entries, and selectively post-edits low-scoring segments while protecting terminology accuracy.
- Pipeline motivation: Full documents create context-window pressure and oversized-glossary distraction, motivating staged document-level processing.The 8,192-token window must contain the document, glossary, and generated translation.
- Layout-preserving chunking: Documents are split at line breaks into roughly 500-word chunks, translated separately, checked automatically, and re-translated line by line when needed.Chunks are re-joined so the submitted layout matches the source by construction.
- Glossary filtering: Each chunk receives only glossary entries whose source terms occur verbatim or through matching lemmas.This reduces the corpus-level glossary to entries the model must act on.
- QE-guided post-editing: Low-scoring segments are post-edited by the same released model using its instruction-tuned post-editing prompt.CometKiwi supplies the segment scores, keeping the submission within one model.
- Terminology safeguard: Post-editing is accepted only when it raises CometKiwi without lowering document-level term accuracy.This guards against fluency edits that remove repeated prescribed terms or replace their wording.
6 Results
The released SalamandraTA system performs strongly on the official WMT26 evaluation and surpasses the earlier GRPO-based system on last year’s benchmark, though that comparison is not fully controlled.
- Official WMT26 results: 94.2% lemmatised term success at 74.6 document chrF++ places the system among the strongest WMT26 Track 1 submissions.Only COZY and Agentic-OPUS improve on both axes simultaneously among twenty-two submissions.
- Comparison with GRPO: The released v3.0 model improves over the GRPO system from 67.3 to 69.4 average chrF and from 91.3% to 94.0% average term accuracy.The comparison uses last year’s benchmark and involves no reinforcement learning, task-specific adaptation, or test-time domain access for v3.0.
- Comparison caveat: The GRPO comparison is a system-level result rather than a controlled ablation because the systems differ in terminology data and broader instruction mixtures.The paper therefore does not isolate which training component accounts for the improvement.
7 Discussion
The discussion identifies tensions and open questions around terminology adherence, while suggesting that hard-example filtering may transfer beyond terminology tasks. It also leaves the mechanism behind the filter’s effect unresolved.
- Many-to-one glossaries can make quality and terminology adherence disagree, because smoothing repeated prescribed targets improves quality scores while damaging adherence.The passage reports a 0.13 higher CometKiwi score for the smoothed variant and positive post-editing deltas for edits that damage adherence.
- Hard-example filtering may transfer to style guides, register constraints, and do-not-translate lists where naturally occurring supervision is similarly redundant.
- The paper leaves open whether the filter’s behavioural account holds mechanistically and what in the model implements it.
- Table 1 compares WMT26 submissions using lemmatised exclusive term success and document chrF++ across directions and averaged scores.
8 Conclusion
The paper presents a public SalamandraTA release and its BSC WMT26 submission, combining filtered terminology training with document-level inference. The controlled study attributes an eleven-point term-accuracy gain to filtering, while the released system reaches 94.2% term success at 74.6 chrF++.
- The BSC WMT26 Track 1 submission uses SalamandraTA-7B-Instruct v3.0, terminology data filtered by a single keep-what-the-model-gets-wrong rule, and document-level inference with per-chunk glossaries.
- An eleven-point term-accuracy gain at fixed data volume is reported for the hard-example filter.
- 94.2% term success at 74.6 chrF++ is reported for the released system in the WMT26 official evaluation.
Limitations
The WMT26 evaluation mixes terminology-adherence failures with retrieval failures because corpus-level glossaries must be narrowed to chunks before translation. The controlled study and training data also have limited scope.
- WMT26 errors mix adherence failures with retrieval failures because corpus-level glossaries are narrowed by surface and lemma matching, so escaped inflections never reach the model.
- The controlled study covers one data source, three test directions, one data volume, and a single reference model defining hardness.
- The filter is static, computed once against v2.0 before training, while dynamic re-estimation during training was out of scope.
- The terminology data is biased toward medical seed resources, whereas the test domains differ.
- Very long documents require chunking at inference, which can lose cross-chunk context for consistent term choice.
A Synthetic-data pipeline details
The paper builds two synthetic terminology datasets around glossary terms that SALAMANDRATA already misses, then filters them for textual and semantic quality. The resulting corpora span varied text granularities and language coverage while preserving terminology difficulty as the selection criterion.
- Two synthetic pipelines: MeSpEn generates source–translation text around hard glossary-term groups, whereas EMEA extracts aligned term pairs from an existing parallel corpus.The pipelines therefore differ in whether terminology drives generation or is extracted from corpus text.
- Hard-case selection: Both pipelines retain only terminology cases that SALAMANDRATA gets wrong, removing terms or sentences the model already translates correctly.MeSpEn selects mismatched glossary terms, while EMEA keeps sentences containing at least one missed gold term.
- Quality filtering: LaBSE-based semantic filtering and rule checks remove records with missing terms, degenerate formatting, or source–target meaning divergence.MeSpEn requires term-pair and full-text similarity of at least 0.80; EMEA requires sentence-pair similarity of at least 0.88.
- Granularity and construction: Synthetic examples cover single sentences, paragraphs, and multi-paragraph documents, with EMEA instances reconstructed from consecutive corpus sentences.Document examples preserve newline layout, and their glossaries combine the term pairs from their component sentences.
- Experimental pools: The ablation samples 4,000 records per direction from hard and easy pools, with a random baseline drawn from their union.The pools are counted after quality and LaBSE filtering and before subsampling.
D Ablation: full numbers
The ablation reports per-direction term accuracy with chrF values and documents the fixed instruction-tuning mixture shared across runs.
- Table 4: Table 4 reports term accuracy and chrF for every direction and every hard-example mixture evaluated in Figure 2.Accuracy is micro-averaged over 1,589 term occurrences, while chrF is averaged across the three directions.
- Table 4: Table 4 places chrF in parentheses alongside term accuracy for each per-direction result.This presentation compares terminology adherence with translation quality in the same table.
- Table 5: Table 5 defines the fixed background mixture shared by all instruction-tuning runs.It includes per-direction sampling caps for high-resource corpora and full use for smaller datasets.