Source-linked AI summary
Task-Preserving Multilingual Instruction Tuning for Open-Ended QA in WMT 2026 MIST
Aleix Sant, Jordi Luque, Carlos Escolano
TL;DR
Multilingual instruction data can expand language coverage, but uniform translation may alter tasks or constraints. This paper adapts Qwen3-8B with LoRA using shared-task and auxiliary multilingual data. Scores improve on target-language open-ended QA and summarisation, alongside declines in context-grounded QA and non-target languages.
Problem
Translating English instruction examples offers scalable multilingual coverage, but may alter task constraints and corrupt supervision.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The reported trade-offs highlight the need to preserve task coverage alongside language coverage during multilingual adaptation.
Source: §9 · p. 6Gemma-4 is a single reference-free judge whose scores are not validated against human annotations or a second evaluator.
Source: Limitations · p. 6
Abstract
from arXiv · showhide
This paper describes our submission to the WMT 2026 Multilingual Instruction Shared Task. We adapt Qwen3-8B with LoRA using organiser-provided data, EuroAlpaca, a multilingual instruction dataset created through task-preserving localisation, and CrossEuroAlpaca, a cross-lingual augmentation that assigns different source and target languages to instructions, contexts and responses. Our system targets open-ended question answering (QA-OEG) in ten shared-task languages represented in the auxiliary data. Under matched decoding, adaptation improves the mean Gemma-4 judge score on target-language QA-OEG by 7.8% relative to the base model. On an additional evaluation using the Aya Evaluation Suite across all shared-task languages, it yields relative gains of 41.5% in macro-averaged ROUGE-L and 2.9% in BERTScore-F1, with larger improvements in target languages. However, these gains are task- and language-specific. Gemma-4 scores improve for QA-OEG and summarisation in the target languages but decline for context-grounded QA there and across all three tasks in the non-target languages.
1 Introduction
Multilingual instruction tuning must broaden language coverage without changing the tasks or constraints that examples supervise. The system adapts Qwen3-8B for open-ended QA in ten languages, improving target-language QA-OEG scores while showing task- and language-specific regressions.
1 Introduction
- Directly translating every field of English instruction examples can alter tasks or constraints and corrupt the supervision signal.Instruction resources are unevenly distributed across languages, motivating scalable coverage expansion.Source: §1 · p. 1
- The system focuses on open-ended QA in Czech, German, English, Finnish, Italian, Portuguese, Russian, Slovak, Spanish and Turkish, while evaluating the full test set for generalisation.The broader evaluation includes languages and tasks less represented in the training mixture.Source: §1 · p. 1
- The system applies LoRA to Qwen3-8B and combines organiser-provided data with EuroAlpaca and CrossEuroAlpaca auxiliary supervision.EuroAlpaca uses task-preserving localisation; CrossEuroAlpaca assigns different source and target languages across example fields.Source: §1 · p. 1
- 7.8%: the adapted model improves target-language QA-OEG judge scores over the identically decoded base, but gains coexist with declines in context-grounded QA and non-target languages.The introduction characterizes the effects as task- and language-specific.Source: §1 · p. 1
2 Shared Task
The shared task evaluates multilingual instruction following across summarisation and two question-answering formats, including monolingual and cross-lingual prompts.
2 Shared Task
- MIST covers context-grounded QA, open-ended QA and summarisation, with examples that can require responses in a language different from the input.Context-grounded QA uses a supplied passage; open-ended QA does not, while summarisation translates English documents into target languages.Source: §2 · p. 2
- The test set contains 12,775 prompts across 24 languages: 8,640 context-grounded QA, 2,359 open-ended QA and 1,776 summarisation instances.The examples include monolingual and cross-lingual cases.Source: §2 · p. 2
- The system is optimised for open-ended QA in the ten response languages covered by its auxiliary data.Source: §2 · p. 2
3 Related Work
Related work addresses multilingual transfer, task-preserving localisation of instruction data and model-based evaluation when reference answers are unavailable.
2 Related Work
- Multilingual instruction tuning can support cross-lingual transfer, while outcomes remain sensitive to training-data language and task composition (Muennighoff et al., 2023; Singh et al., 2024; Üstün et al., 2024; Sant et al., 2026).Controlled comparisons report that multilingual tuning can approach or match separate monolingual adapters, even with limited per-language supervision (Chen et al., 2024a; Shaham et al., 2024).Source: §3 · p. 2
- Task-preserving localisation protects or rewrites fields when uniform translation would invalidate task-dependent linguistic forms (Chen et al., 2024b; Li et al., 2024; Sant et al., 2026).The paper uses EuroAlpaca's task-valid localised examples as auxiliary supervision rather than data produced by uniform machine translation.Source: §3 · p. 2
- When reference answers are unavailable, model-based judges assess open-ended responses against prompts and rubrics, but may have uneven language calibration (Liu et al., 2023; Zheng et al., 2023).The paper treats judge scores as diagnostic evidence and complements them with reference-based Aya Evaluation Suite metrics.Source: §3 · p. 2
4 Training and Validation Data
The training mixture combines repeated shared-task examples with multilingual localised and cross-lingual instruction data, using held-out MIST samples for validation.
4 Training and Validation Data
- The final mixture combines repeated MIST training examples with EuroAlpaca and CrossEuroAlpaca, while held-out MIST samples are reserved for validation and early stopping.The MIST split is deterministic and stratified by language, task and source dataset.Source: §4 · pp. 2–3
4.1 MIST
- Four repetitions of the cleaned MIST training set yield 85,776 effective samples and raise MIST's mixture share from 1.8% to 7.0%.The repetition maintains regular exposure to supervision matching the shared-task formats.Source: §4.1 · p. 3
4.2 EuroAlpaca
- EuroAlpaca contains 51,760 examples in each of 11 languages, including a Turkish version created for this work.Its pipeline decides whether translation is safe, preserves task-critical content or rewrites examples, then checks coherence and language quality.Source: §4.2 · p. 3
4.3 CrossEuroAlpaca
- CrossEuroAlpaca uses aligned localisations of the same underlying example to create examples with different source and target languages across fields.It retains 40,672 examples safe for field-wise translation, excludes 11,088 language-dependent examples, and balances 5,176 examples across each of 110 directed language pairs.Source: §4.3 · p. 3
5 System Description
The system applies LoRA to Qwen3-8B and specifies its training setup and inference configuration for the shared task.
5.1 Model and Adaptation
- LoRA adapts Qwen3-8B by training 15.3M of 8.20B parameters while leaving the base model frozen.It targets all attention query, key, value and output projections, with rank 16, α = 32 and dropout 0.05.Source: §5.1 · p. 3
5.2 Training
- Training uses AdamW, bfloat16, gradient clipping at 1.0, no weight decay, and a peak learning rate of 1.5 × 10^-5.The effective batch size is 128 across four 64 GB H100 GPUs; early stopping monitors validation loss with patience 5.Source: §5.2 · p. 3
5.3 Inference
- Inference uses Qwen’s native chat template with thinking disabled, a 16,384-token prompt cap, and greedy decoding with repetition penalty 1.08.Generation is limited to 1,024 new tokens; five-beam decoding was discarded after occasionally causing language and script drift.Source: §5.3 · pp. 3–4
6 Evaluation
The evaluation compares base and adapted models across shared-task formats and target-language status, and uses a multilingual reference-based suite as a complementary assessment.
6.1 Shared-task Test Analysis
- The shared-task analysis compares off-the-shelf and LoRA-adapted Qwen3-8B under matched inference, by task and target-language status; Gemma-4 scores exclude French, which was not in the test set.The shared-task test data include the ten target languages, while French appeared in the original MIST data and auxiliary data.Source: §6.1 · p. 4
- A Q8-quantised Gemma-4-31B3 model serves as a reference-free judge, scoring responses from one to five for task completion, compliance, grounding, factuality, fluency and visible reasoning.Acceptance rate is the share of responses scoring four or five.Source: §6.1 · p. 4
- Repetition penalty 1.08 was selected using Gemma-4 scores on released test prompts, so the decoding analysis is test-aware.The base model uses the same penalty for comparison.Source: §6.1 · p. 4
- The target QA-OEG uncertainty estimate clusters each underlying prompt with its ten language versions and uses 50,000 bootstrap resamples for a percentile-based 95% confidence interval.The subset contains 1,000 instances from 100 prompts.Source: §6.1 · p. 4
6.2 Aya Evaluation Suite
- The Aya Evaluation Suite assessment covers 27 overlapping languages and reports macro-averaged ROUGE-L and BERTScore-F1 with language-level bootstrap confidence intervals.Both models use identical context length, batch size and greedy decoding conditions.Source: §6.2 · p. 4
7 Results
The adapted model improves target-language QA-OEG scores, but results vary by task and language coverage. Aya evaluation shows broad gains, while official shared-task scores vary across target languages.
7.3 Aya Evaluation Suite
- On Aya, BERTScore-F1 improves in all 27 evaluated languages, while ROUGE-L improves in 21; macro-average gains are larger across the 11 auxiliary-data languages.Every language in both reported subsets improves on both metrics.Source: Table 3 · p. 5, §7.3 · p. 5
7.1 Target QA-OEG
- Across tested repetition penalties, the adapted model beats the base model; its target QA-OEG score peaks at 1.08 before declining at 1.10.The advantage persists across tested penalties, not only at the selected value.Source: Table 4 · p. 5, §7.1 · p. 5
- 3.596 to 3.876 mean Gemma-4 judge score (+7.8%) under matched decoding; the paired difference is +0.280 (95% cluster-bootstrap CI [+0.133, +0.429]).Acceptance rose from 49.6% to 62.6%, a 13.0 percentage-point gain.Source: §7.1 · pp. 4–5
- Nine of ten target languages improve, with the largest gains in Czech, Slovak and Russian; English is the only decline.The aggregate gain is distributed across most target languages rather than concentrated in one or two.Source: §7.1 · p. 5
7.2 Task and Language Trade-offs
- At repetition penalty 1.08, target-language adaptation improves QA-OEG and summarisation but substantially weakens context-grounded QA, the largest regression; non-target scores fall across all three tasks.Pooling all test instances therefore favours the base model.Source: Table 5 · p. 5, §7.2 · p. 5
7.3 Aya Evaluation Suite
- Aya language-level bootstrap intervals exclude zero for all reported macro-average differences; six ROUGE-L regressions occur outside the target set, while BERTScore-F1 improves in each.Semantic similarity improves consistently, although lexical overlap does not improve in every language.Source: §7.3 · p. 5
7.4 Official Shared-Task Results
- Official QA-OEG scores are 62.37 across 24 evaluation languages and 73.63 on the ten targets; the system ranks second in English, Portuguese and Spanish, and third in Italian.It is less competitive in Czech and Finnish; these official scores are not directly comparable to Gemma-4 results because the official metric uses a majority-vote constraint-rule pass rate across three LLM judges.Source: Table 6 · p. 6, §7.4 · p. 6
8 Analysis
Adaptation improves open-ended QA and summarisation in target languages but weakens context-grounded QA there and all three tasks outside the target set. Aya reference-similarity gains complement, but do not negate, that task-specific regression; official team rank measures a different comparison.
Task- and language-specific results
- Target-language QA-OEG and summarisation improve, while context-grounded QA weakens and all three tasks decline outside the target set.Target-language gains therefore do not guarantee preserved performance across task formats or languages not covered by the auxiliary data.Source: §8 · p. 6
Complementary evaluation
- Aya shows reference-similarity gains in BERTScore-F1 across 27 languages and ROUGE-L across 21, without contradicting context-grounded QA regression.Aya evaluates reference similarity on open-ended responses, whereas QA-context requires grounding in a supplied passage.Source: §8 · p. 6
Official evaluation
- English ranks second among teams despite a small Gemma-4 decline, because official evaluation compares submitted systems under a different metric.The analysis instead measures adaptation gains over the base model.Source: §8 · p. 6
Possible explanation and mitigation
- MIST supplies 7.0% of effective training rows, but experiments do not isolate data-source effects, leaving the proposed composition explanation unconfirmed.With base parameters frozen, regressions are consistent with reversible adapter-induced interference; balanced task sampling, QA-context replay, and task-aware checkpoint selection are suggested responses.Source: §8 · p. 6
9 Conclusion
The paper presents a Qwen3-8B system adapted with LoRA on MIST, EuroAlpaca, and CrossEuroAlpaca for open-ended QA in ten languages. Results combine target-language gains with regressions in context-grounded QA and non-target languages, underscoring the need to preserve task as well as language coverage.
9 Conclusion
- The system adapts Qwen3-8B with LoRA on MIST, EuroAlpaca, and CrossEuroAlpaca, targeting open-ended QA in ten languages.The conclusion reports Gemma-4 gains on target-language QA-OEG and summarisation, alongside broader reference-based response-quality gains in Aya.Source: §9 · p. 6
- Target-language QA-OEG and summarisation improve, but context-grounded QA declines there and scores fall across all three tasks outside the target-language set.The conclusion calls for preserving task coverage alongside language coverage during multilingual adaptation.Source: §9 · p. 6
Limitations
Gemma-4 is the sole reference-free judge, and its scores are not validated against human annotations or another evaluator.
Limitations
- Gemma-4 is the only reference-free judge used for these scores.The paper reports no human-annotation or second-evaluator validation.Source: Limitations · p. 6
- The scores are not validated against human annotations.They are also not checked against a second evaluator.Source: Limitations · p. 6
- Judge scores may reflect uneven language coverage, rubric interpretation, or response-style preferences.These are stated as possible influences, not established causes of the results.Source: Limitations · p. 6
A LLM-as-a-Judge Prompt
The appendix prompt gives Gemma-4 a multilingual judging role and specifies its scoring rubric and JSON response format.
A LLM-as-a-Judge Prompt
- The appendix presents the prompts sent to Gemma-4, with placeholders replaced for each test instance.The user prompt includes task and question-language metadata and a candidate-answer placeholder.Source: A LLM-as-a-Judge Prompt · p. 8, User prompt template · p. 8
System prompt
- The system prompt requires valid JSON only and forbids Markdown.Source: System prompt · p. 8
User prompt template
- The user prompt frames Gemma-4 as a strict multilingual judge assessing how well a candidate follows the prompt and solves the task, without a reference answer.Judging is to rely on the prompt and candidate answer.Source: User prompt template · p. 8
- General rules prioritize explicit prompt constraints and supported content, while penalizing task mismatch, unnecessary verbosity, and malformed requested formats.Source: User prompt template · p. 8
- Language rules require the requested output language and penalize wrong language, script, mixing, spelling, grammar, or unnatural wording.Minor wording issues should not be over-penalized when the answer is correct, fluent, and in the right language.Source: User prompt template · p. 8
- Content rules reward faithful summaries and direct, context-grounded answers while penalizing omissions, unsupported claims, and irrelevant output.Source: User prompt template · p. 8
- The rubric assigns integer scores from 1 to 5, accepts only scores 4 or 5, and requires a JSON object with score, acceptability, and reason.Source: User prompt template · p. 8