Source-linked AI summary
Reasoning about In-Context Samples for Machine-Translation
Maxime Bouthors, Josep Crego, François Yvon
TL;DR
The paper asks whether explicit reasoning over retrieved translation-memory exemplars can improve LLM-based machine translation. It distills fragment and draft traces from a teacher model, then trains a student to use them, finding consistent gains over standard k-shot and drafting approaches across languages and domains.
Problem
The paper investigates whether LLMs can reliably identify useful translation fragments in retrieved exemplars and whether those fragments improve MT across exemplar counts, domains, and languages.
Method
A teacher LLM generates silver parallel fragments and drafts from retrieved exemplars, and a fine-tuned student reproduces fragments as intermediate reasoning traces for translation.
Results
Fragment-based MT consistently improves translation scores across six languages and ten domains, outperforming standard k-shot prompting and drafting-based approaches.
Takeaways & Limitations
Fragment extraction provides a robust reasoning signal that improves translation quality even with noisy fragments and generalizes to domains unseen during fine-tuning.
Takeaways & Limitations
The evaluation remains limited in scale, and reasoning-trace quality depends strongly on potentially suboptimal teacher-generated silver fragments.
Abstract
from arXiv · showhide
Large Language Models (LLMs) can be trained to perform chain-of-thoughts reasoning in order to improve the reliability of their responses. In this work, we investigate how explicit reasoning can be leveraged for LLM-Based Machine Translation (MT) with in-context samples. We introduce a novel fragment-based reasoning framework in which the model first extracts parallel source-target fragments from retrieved similar exemplars, and uses these fragments as intermediate reasoning traces to produce the final translation. To train our model, we distill silver fragments and drafts from a large teacher model. Our experiments with the Qwen3 model family, over 6 languages, including up to 5 domains per language, demonstrate that fragment-based MT significantly outperforms alternative methods like standard k-shot or basic drafting.
1 Introduction
The paper frames fragment-based reasoning as a different way to use translation-memory exemplars: identify reusable source-target fragments and recombine them during translation. It asks whether LLMs can extract useful fragments, improve MT, remain robust across exemplar counts, and generalize across domains and languages.
- In-context learning supplies task-specific lexical, terminological, and stylistic context that can guide LLM-based translation.
- The proposed framework models reasoning around editing one or more close translation-memory exemplars rather than simulating a human translator’s general chain of thought.Translation memories support reuse of validated terminology and phraseology in specialized translation.
- Translation-memory reasoning decomposes into retrieving exemplars, matching source segments to target equivalents, recombining them, and optionally revising the draft.The paper focuses mainly on matching and, to a lesser extent, recombination.
- The pipeline retrieves exemplars, has a teacher perform fragment extraction, and trains a student to reproduce the extraction as a reasoning trace.The reported findings include improvements across metrics, robustness to retrieved-exemplar count, and generalization to unseen domains.
- The study asks whether LLMs can identify actual parallel fragments, whether fragments improve MT, how exemplar count affects gains, and whether benefits vary across domains or languages.
2 Related Work
Related work connects the paper to retrieval-augmented, edit-based, and reasoning-based MT. Prior approaches use exemplars or intermediate reasoning for context, editing, analysis, drafting, or revision, whereas this work emphasizes fragment matching and recombination.
- Retrieval-Augmented Machine Translation: Retrieval-augmented MT uses similar, inspectable exemplars to improve translation quality and provide domain-adaptation and transparency benefits.
- Retrieval-Augmented Machine Translation: Edit-based MT patches existing translations by identifying content to keep, adapt, or retranslate, and works especially well with very close exemplars.
- Large Reasoning Machine Translation: The paper differs from prior approaches by focusing on matching and recombining fragments from retrieved exemplars as the reasoning process.
- Large Reasoning Models: Large Reasoning Models generate intermediate chain-of-thought text before producing final answers, using prompting, supervised fine-tuning, or reinforcement learning.
- Large Reasoning Machine Translation: Reasoning-based MT studies post-editing, translator introspection, auxiliary pre-translation tasks, and combinations of translation subtasks distilled from teachers.
3 Method
The method distills exemplar-based reasoning traces from a strong teacher and trains a student to extract, adapt, recombine, and translate with those fragments. Ablations separately test fragments, drafts, and ordinary in-context translation.
- Reasoning process: Given retrieved exemplars, the process decomposes the source into minimal semantic units and assigns each a copied, adapted, or newly generated translation.
- Reasoning process: The method forms a draft by recombining fragment translations, adding inserts or reformulations to fill gaps between them.
- Method challenges: The decomposition, matching, and recombination stages are difficult because segmentation is ambiguous, alignments are fuzzy, and recombination may require structural changes.
- Silver-fragment distillation: A strong teacher LLM generates silver fragments and drafts from source sentences augmented with retrieved translation-memory exemplars.These artificial annotations extend training data with supervision for simulated reasoning.
- Student training: A student LLM is fine-tuned to generate reasoning tokens before the final translation, producing the Fragments+Draft model.
- Ablations: The study compares Fragments+Draft with fragments-only, draft-only, and baseline variants trained on the same artificial data.
4 Data and Metrics
Experiments use multilingual, multidomain parallel data with retrieved in-domain exemplars and an unseen-domain test set. Translation quality is evaluated with BLEU, COMET, and MetricX.
- Data: The corpus covers five language pairs linking English with European languages and includes domains from German, French, Polish, Ukrainian, and Spanish data.
- Data: The training set contains 160k samples from 16 domain-language combinations, with each instance using between k = 0 and k = 3 retrieved examples.
- Data: Retrieval combines BM25 and Levenshtein distance over the training pool, excluding development and test splits.
- Evaluation splits: Each domain has a 1,000-example test set and 100-example development set, while GNOME provides an unseen English-French surprise test set.
- Metrics: Translation quality is measured with BLEU, COMET, and MetricX.
5 Experimental Settings
The experiments use Qwen3-32B as a teacher and Qwen3-8B as the student, comparing fine-tuning and inference settings with and without reasoning. A fine-tuned translation-only baseline is included, while the teacher’s reasoning mode is excluded from the large test set because of its cost.
- Models: Qwen3-32B serves as the teacher model, while the smaller Qwen3-8B serves as the student model.The teacher uses a long prompt with detailed instructions and an illustration; the student uses a minimal prompt because of deployment and fine-tuning costs.
- Training: Each model observes k = 0, 1, 2, 3 exemplars equally during training, with 20% of samples containing empty reasoning traces.The empty traces enable inference with thinking enabled or disabled, and all fine-tuning runs last 2 epochs.
- Baselines: The fine-tuned baseline generates translations without any thinking process.An instruct Qwen3-8B model without fine-tuning is also considered, although the supplied passage ends before describing its evaluation setup.
- Baselines: The instruct model’s reasoning often requires thousands of tokens, so it is evaluated only with reasoning turned off on the large multi-domain test set.This restriction is motivated by the high cost of running reasoning over the full test set.
6 Results and Analysis
Across the evaluated settings, fragment-based reasoning improves translation quality and generalizes beyond the fine-tuning domain, while drafting alone often hurts performance. Extracted fragments are usually faithful to source spans, although generated drafts and fragment extraction remain imperfect.
- 6.1 Fragment-based MT: Fragment-based translation significantly improves translation quality across all considered metrics, with or without drafting.The improvement is reported across configurations and values of k.
- 6.1 Fragment-based MT: Drafting alone seems to degrade translation quality, whereas reasoning gains appear for every number of retrieved exemplars k.The authors suggest inaccurate drafts or poor use of draft information may explain the degradation.
- 6.6 Domain and language analysis: On the unseen GNOME domain, fragment settings produce results similar to in-domain tests, suggesting that fragment-based reasoning generalizes to new domains.On GNOME, drafting also yields gains with and without fragments.
- 6.3 Assessing fragment quality: Source-fragment precision is high but recall is about 80%, with lower recall largely reflecting omitted punctuation or function words.Inferred fragments achieve higher source recall than silver fragments after supervised fine-tuning.
- 6.4 Silver vs. inferred fragments and drafts: Silver fragments have an extraction rate of approximately 70%, compared with about 35% for student fragments, because students sometimes generate rather than copy fragments.The extraction-rate difference does not affect overall translation quality.
- 6.4 Silver vs. inferred fragments and drafts: Fragment unigrams cover 81% of silver drafts and 98% of inferred drafts, while generated drafts score approximately 30 BLEU versus approximately 38 for silver fragments.The results indicate that inferred drafts mostly recombine fragments and that drafting is unlikely to help in this setting.
- 6.6 Domain and language analysis: COMET gains are substantially higher when exemplar coverage is low than when it is high, across all values of k.High coverage usually indicates a close exemplar that already supports strong baseline translation; fragment extraction helps more when irrelevant content surrounds useful information.
7 Conclusions and Outlook
The paper introduces fragment-based reasoning for LLM-based MT and finds consistent gains across six languages and ten domains. It identifies low draft quality and reliance on supervised distillation as important boundaries, while leaving reinforcement learning, larger exemplar sets, and traceability studies for future work.
- Conclusions: Fragment-based reasoning augments LLM-based MT with exemplar matching and recombination, consistently improving translation scores over standard k-shot prompting and drafting-based approaches.The experiments cover six languages and ten domains.
- Outlook: The approach currently relies on supervised distillation from a large teacher model to learn reasoning traces.
- Outlook: Future work includes reinforcement learning for intermediate reasoning quality, increasing the number of retrieved exemplars beyond three, and studying traceability with translators.
Limitations
The evaluation is limited in scale despite covering multiple language pairs and domains, and the approach depends on available bilingual retrieval resources and teacher-generated silver fragments.
- Evaluation covers multiple language pairs and domains but remains limited in scale, motivating broader validation of generalization and robustness.The authors call for more diverse languages, domains, and translation conditions.
- The approach assumes translation memories or large bilingual corpora from which relevant exemplars can be retrieved.Such resources may be unavailable for low-resource languages, emerging domains, or highly specialized applications.
- Reasoning-trace quality depends on teacher-generated silver fragments, whose errors can propagate to student models and degrade translation quality.The authors identify malformed fragments, hallucinated content, incomplete extractions, and incorrect bilingual correspondences as possible errors.
Ethical Statement
The paper reports no ethical issues with this work.
- The authors state that there are no ethical issues with this work.
A Datasets
The study builds multilingual training data by selecting high-quality, challenging samples across domains, retrieving similar exemplars, and training Qwen3-8B variants with distinct reasoning prompts.
- Data sources and filtering: The corpus comprises datasets spanning various domains, with French-English and German-English data drawn from previously prepared resources and other datasets downloaded from OPUS.OpusCleaner removes empty and excessively short or long sentences before further filtering.
- Data selection: 10K samples per domain are selected using a quality score combining COMETKiwi parallel quality with source-sentence difficulty from sentinel-src-25.The squared COMETKiwi term gives greater weight to parallelism and avoids difficult low-quality training samples.
- Retrieval setup: The selected domain subsets are concatenated into a 160K multilingual dataset, with each instance receiving uniformly sampled in-domain exemplars from k = 0 to k = 3.Retrieval is performed for training, validation, and test samples while excluding development and test splits from the retrieval pool.
- Training setup: Student models are trained with Qwen3-8B supervised fine-tuning using LoRA for two epochs and an effective batch size of 32.The reported configuration uses rank 16, α = 16, dropout 0.05, learning rate 2e-4, and AdamW.
- Teacher prompting: The teacher’s fragment-extraction prompt withholds the reference translation so target fragments are extracted from exemplars rather than copied from the reference.The same prompt generates both silver fragments and a draft.
- Student prompting: Student prompts differ in their thinking process, defining baseline, draft, fragments, and draft-plus-fragments model variants through supervised training data.The baseline prompt directly requests a translation without extra information.
E Detailed results per domain
Detailed results are reported across 16 domains and languages for five Qwen3-8B model variants, with separate tables for BLEU, COMET, and MetricX and an illustration of translation traceability.
- Per-domain evaluation: Scores and gains vary across the 16 domains and languages, although the paper primarily presents aggregated results.The full BLEU, COMET, and MetricX results are provided separately for models I, B, D, F, and F+D.
- Traceability: Figure 9 illustrates traceability in fragment-based translation by linking spans in generated translations to the source sentence and retrieved exemplars.
- Prompt configurations: Figures 4–8 document the prompts used for teacher fragment and draft generation and for training the baseline, draft, fragment, and draft-plus-fragments students.The teacher prompt disables thinking, while the student figures specify the corresponding training prompts.
- Per-domain evaluation: Tables 7–11 report BLEU, COMET, and MetricX scores for the pretrained, baseline, drafting, fragment-based, and draft-plus-fragments models.Each table covers the multi-domain test sets for one model variant.