Source-linked AI summary
Qwen-MT-Lumen at WMT2026: Instruction-Specialized Translation via Quality-Aware Training and On-Policy Distillation
Hao Wang, Yangyang Liu, Xiaohu Zhao, Heng Liu, Zifu Shang, Tianhao Li, Ruize Gao, Jialong Tang, Liangying Shao, Haoran Wei, Baosong Yang, Linlong Xu, Longyue Wang, Weihua Luo
TL;DR
Generic translation models can struggle to satisfy fine-grained requirements for style, terminology, and structured output. Lumen combines instruction-specialized adaptation, online preference optimization, on-policy distillation, and structure-aware inference; on WMT26 test sets, its 14B system scores 4.60 overall and is comparable to leading proprietary systems on translation quality and instruction following.
Problem
Generic LLM-based machine translation systems often struggle to reliably follow fine-grained instructions on style, terminology, contextual ambiguity, and structured formats.
Source: §1 · p. 1Method
Lumen combines CPT and SFT, online M2PO with LLM direct-assessment rewards, on-policy distillation from a larger teacher, and structure-aware inference with targeted postchecking.
Source: §4 · p. 8Results
4.60 overall mean on WMT26 test sets under the absolute 1–5 rubric, with translation-instruction following at 4.49 and translation quality at 4.71; performance is comparable to leading proprietary systems on both aspects.
Source: §1 · p. 2Takeaways & Limitations
A 14B translation backbone can achieve performance comparable to leading proprietary systems in translation quality and translation-instruction following.
Source: §1 · p. 2
Abstract
from arXiv · showhide
We present Lumen, The open-source model is available at, our submission to the WMT 2026 General Machine Translation Shared Task. We focus on instruction-specialized translation, where systems must produce accurate translations while satisfying fine-grained requirements on style, terminology, and structured output. Lumen follows a four-stage pipeline. First, continued pre-training and supervised fine-tuning jointly strengthen multilingual translation competence and the ability to follow translation-specific instructions, such as terminology, style, and format constraints. Second, we extend the originally offline M^{2}PO preference-optimization framework into an online preference-optimization stage: the current policy samples candidate translations, an LLM judge assigns continuous direct-assessment scores, and the resulting multi-pair preferences are used to update the model. Third, on-policy distillation transfers stronger translation-instruction-following behavior from a larger teacher model to the Qwen3-14B student, improving translation-instruction adherence while retaining a 14B primary translation backbone. Finally, structure-aware inference and postprocessing preserve required formats, while selective repair addresses clear structural or decoding failures before submission. We participate in 23 translation directions. Under an absolute 1--5 LLM-judge rubric that evaluates translation-instruction following and translation quality separately, Lumen reaches an overall mean score of 4.60. Despite using a 14B translation backbone, it achieves performance comparable to leading proprietary systems in both translation quality and translation-instruction following.
1 Introduction
Lumen targets the gap between generic translation quality and reliable adherence to fine-grained translation instructions. Its four-stage system combines instruction-specialized adaptation, online preference optimization, on-policy distillation, and structure-aware inference, reaching an overall mean of 4.60 and performance comparable to leading proprietary systems.
1 Introduction
- Generic LLM translation systems often struggle with fine-grained style, terminology, ambiguity, and structured-format instructions needed in practical applications.The paper identifies reliable instruction following as a gap for deploying LLM-based machine translation in real-world scenarios.Source: §1 · p. 1
- CPT on WMT-relevant domains and directions followed by SFT on translation and instruction data builds translation competence and instruction following.The instruction data covers voice, glossaries, structured translation, and style and expression requirements.Source: §1 · p. 2
- Online M2PO refreshes candidates from the current policy, uses an LLM judge’s direct-assessment scores, and converts rankings into multi-pair supervision.This extends M2PO from optimization over a fixed preference set to an online procedure.Source: §1 · p. 2
- On-policy distillation transfers a larger teacher’s instruction-following behavior to the 14B student while retaining the student as the translation backbone.The student samples trajectories for token-level distribution alignment.Source: §1 · p. 2
- Structure-aware inference preserves markup and protected content, while rule-based checks trigger deterministic repairs or selective rewrites for clear failures.Source: §1 · p. 2
- 4.60 overall mean under the absolute 1–5 LLM-judge rubric, with performance comparable to leading proprietary systems in translation quality and instruction following.Lumen’s scores were 4.49 for translation-instruction following and 4.71 for translation quality.Source: §1 · p. 2
2.1 System Overview
Lumen uses four successive stages, progressing from multilingual translation and instruction adaptation to preference optimization, distillation, and structure-aware inference.
2.1 System Overview
- CPT and SFT adapt the Qwen3-14B backbone to multilingual translation and translation-specific instruction following.Source: §2.1 · p. 2
- Online M2PO uses on-policy candidates and LLM-based direct-assessment rewards to improve translation fidelity.Source: §2.1 · p. 2
- On-policy distillation transfers instruction-following behavior from a larger frozen teacher to the translation-specialized 14B model.Source: §2.1 · p. 2
- Structure-aware decoding and targeted postchecking preserve required formats and repair clear decoding failures.Source: §2.1 · p. 2
2.2 CPT and SFT Adaptation
The first stage adapts Qwen3-14B through full-parameter continued pre-training and supervised fine-tuning, with multilingual coverage and translation-specific instruction data.
2.2 CPT and SFT Adaptation
- CPT on multilingual text and general instruction data strengthens the translation foundation; SFT then jointly develops translation quality and translation-specific instruction following.Both phases update all model parameters.Source: §2.2 · pp. 2–3
2.2.1 Training Data
- The CPT and SFT mixtures combine multilingual and task-specific translation data with general and translation-specific instruction data across 23 WMT26 directions.Instruction data covers formal or informal voice, glossary constraints, structured translation, and style and expression requirements; most such examples are produced with an LLM-assisted synthesis and filtering pipeline.Source: §2.2.1 · pp. 3–4
2.2.2 Adaptation Parameterization
- Both adaptation phases update all parameters rather than using parameter-efficient adapters, and the resulting CPT+SFT model feeds the subsequent RL and OPD stages.Source: §2.2.2 · p. 4
2.3 Preference Alignment with Online M2PO
Online M2PO extends fixed-dataset preference optimization by repeatedly sampling and ranking candidates from the current policy, then using ranked pairs to update it.
2.3 Preference Alignment with Online M2PO
- Online M2PO generates fresh candidates from the current policy, ranks them with an LLM judge, and immediately uses the resulting preferences for policy updates.This avoids reliance on a static preference dataset while retaining multi-pair supervision.Source: §2.3 · p. 4
2.3.1 Online M2PO
- For each input, the policy samples eight candidates, ranks them by direct-assessment score, and forms four non-overlapping pairs by coupling opposite ends of the ranking.The reward assesses meaning preservation and grammatical quality; its continuous values preserve intermediate candidate ordering and set pair margins.Source: §2.3.1 · p. 4
2.3.2 Training Configuration
- M2PO training uses LoRA adapters, with eight prompt groups per optimization step and 64 candidate sequences in total.The adapters have rank 128 and α = 256 and are merged into the full model after training; preference temperature β is 0.1.Source: §2.3.2 · p. 4
2.4 On-Policy Distillation for Translation Instruction Following
On-policy distillation addresses residual failures on fine-grained translation instructions by using a frozen, larger teacher to supervise the translation-specialized 14B student. The student samples its own translations, and training minimizes reverse KL between student and teacher token distributions along those trajectories.
2.4 On-Policy Distillation for Translation Instruction Following
- The frozen Qwen3-32B-Instruct teacher supervises the post-M2PO Qwen3-14B student to address residual errors on style, terminology, and output-format constraints.The online M2PO reward assesses translation quality rather than instruction compliance.Source: §2.4 · p. 5
2.4.1 Distillation Objective
- For each instruction input, the student samples a translation and the teacher supplies token-level distributions for that same trajectory.This differs from conventional sequence-level distillation over a fixed teacher-generated dataset.Source: §2.4.1 · p. 5
- Training minimizes reverse KL along student-generated trajectories, encouraging probability mass on the teacher’s preferred continuations.The teacher thus supervises distributions on trajectories where the student makes errors, rather than only fixed teacher outputs.Source: §2.4.1 · p. 5
- The student and teacher share the Qwen3 tokenizer and vocabulary, allowing token-level KL computation without vocabulary alignment.The distributions are conditioned on the input and the preceding student-generated tokens.Source: §2.4.1 · p. 5
2.4.2 Training Configuration
- The full student is fine-tuned with the reverse-KL objective using AdamW, a 1 × 10−5 learning rate, and global batch size 256.The learning rate is lower than for SFT because OPD refines the output distribution rather than teaching translation from scratch.Source: §2.4.2 · p. 5
- Student trajectories are sampled at temperature 1.0, and teacher and student use distillation temperature 1.0.For each trajectory, the frozen teacher supplies token-level distributions in one forward pass.Source: §2.4.2 · p. 5
2.5 Structure-Aware Inference and Postcheck
Stage 4 preserves input structure and screens for clear decoding failures after translation. Its conservative processing repairs only objective mechanical defects or routes flagged outputs for regeneration, rather than acting as a general-purpose post-editor.
2.5 Structure-Aware Inference and Postcheck
- Stage 4 follows translation with structure-aware processing and a conservative postcheck, targeting required input structure and clear decoding failures rather than general editing.Decoding uses greedy search with vLLM.Source: §2.5 · p. 5
- HTML text nodes and designated attributes, JSON string leaves, and long plain-text chunks are translated and reassembled while preserving markup, keys, and protected content.Protected content includes URLs, e-mail addresses, and numbers.Source: §2.5 · p. 5
- Checks flag length-ratio, structural, repetition, script-coverage, or source-copy anomalies; only unambiguous mechanical defects are repaired, while other flagged outputs are regenerated.Outputs that pass all checks are retained unchanged, and a failed check alone does not establish a translation error.Source: §2.5 · pp. 5–6
3 Experiments
On WMT26, Lumen’s 14B system is comparable to leading proprietary systems, with instruction following distinguishing systems more than translation quality. Across the cumulative ablation, training stages and postcheck improve both aspects, with different stages contributing most to different dimensions.
3.1 Experimental Setup and Evaluation
- Under the absolute 1–5 LLM-judge protocol, the system is evaluated on official WMT26 test sets across 23 directions and compared with four reference LLMs.The judge scores instruction following and translation quality separately; pass rate is the proportion scoring at least 4.Source: §3.1 · pp. 6–7
3.2 Main Results
- Lumen scores 4.60 overall; its 14B system is comparable to leading proprietary systems, and translation quality scores surpass Gemini-3.1-Pro.Instruction following is comparable to GPT-5.5 and Qwen3.7-Max-Think; quality scores across systems are uniformly high, so systems separate mainly on instruction following.Source: §3.2 · p. 7
3.3 Per-Direction Results
- Across the 23 directions, instruction following is near-ceiling on most, with exceptions concentrated in large test sets; quality is mostly ≥4.6 and dips on several lowest-resource directions.The lower-scoring directions differ across the two aspects, suggesting they capture different failure modes.Source: Table 3 · p. 6, §3.3 · p. 7
3.4 Ablation Study
- CPT+SFT raises instruction following from 1.14 Avg and 0.6% Pass to 4.08 Avg and 69.5% Pass, and quality from 1.09 Avg and 1.0% Pass to 4.23 Avg and 74.8% Pass.Online M2PO then raises quality from 4.23 to 4.55 Avg, while OPD raises instruction following from 4.13 to 4.38 Avg and quality from 4.55 to 4.67 Avg; postcheck further raises them to 4.49 and 4.71 Avg, respectively.Source: §3.4 · pp. 7–8, Table 4 · p. 7
3.5 Analysis
- The stages have complementary effects: CPT+SFT establishes instruction-based translation, online M2PO contributes the largest later quality gain, and OPD contributes the largest later instruction-following gain.Postcheck recovers a small tail of structurally invalid or degenerate outputs; instruction following and quality also fail on different directions, supporting separate evaluation.Source: §3.5 · p. 8
4 Conclusion
Lumen combines instruction-specialized adaptation, online preference optimization, on-policy distillation, and structure-aware inference with targeted postchecking. On WMT26 it achieves a 4.60 overall mean and performance comparable to leading proprietary systems despite its 14B translation backbone.
4 Conclusion
- Lumen’s four-stage pipeline combines instruction-specialized CPT+SFT, online M2PO preference optimization, on-policy distillation, and structure-aware inference with targeted repair.Distillation transfers instruction-following behavior from a Qwen3-32B-Instruct teacher to a 14B student; postchecking targets clear structural or decoding failures.Source: §4 · p. 8
- Under a separate-aspect absolute 1–5 LLM-judge rubric, Lumen’s 14B system reaches an overall mean of 4.60 and performs comparably to leading proprietary systems in translation quality and translation-instruction following.Source: §4 · p. 8
- The results support a translation-specialized 14B system that combines quality-oriented preference optimization with a dedicated mechanism for instruction following.The reported comparison is to leading proprietary systems.Source: §4 · p. 8