PaperlayerView exact revision on arXiv

Source-linked AI summary

EuroAlpaca: Task-Preserving Localisation of Instruction Data for European Languages

Aleix Sant, Jordi Luque, Carlos Escolano

arXiv:2609.05043v1cs.CL

TL;DR

Machine translation can scale multilingual instruction tuning while corrupting task-critical constraints and required outputs. EuroAlpaca introduces task-preserving localisation and a near-parallel resource, paired with European-IFEVAL, and shows that preserving task semantics reverses the instruction-following degradation caused by direct MT.

  • Problem

    Directly translating English instruction data can preserve sentence meaning while corrupting task-critical constraints, field roles and required outputs.

  • Method

    EuroAlpaca routes examples through field-wise translation with preservation decisions or task-level reconstruction, then validates target-language and cross-field coherence across 50 European languages.

  • Results

    Direct MT reduces European-IFEVAL accuracy by 29.8% relative to the unadapted baseline, whereas EuroAlpaca improves it by 12.9% and also achieves the highest Aya ROUGE-L and F-BERT scores.

  • Takeaways & Limitations

    Multilingual instruction tuning should evaluate both target-language response quality and verifiable instruction following, while preserving task function rather than fluency alone.

  • Takeaways & Limitations

    The pipeline’s localisation quality in low-resource varieties is bounded by Gemma-4’s prior exposure and knowledge of those languages.

Abstract

from arXiv · show

Machine translation (MT) offers a scalable way to extend English instruction-tuning data to multiple languages, but it can distort task-critical constraints and required outputs, creating corrupted training examples and degrading models trained on such data. We introduce EuroAlpaca, a task-preserving localisation pipeline and near-parallel resource covering 50 European languages and regional varieties, together with European-IFEval, a multilingual benchmark for verifiable instruction following. Depending on the example, our pipeline applies field-wise MT while preserving task-critical content or reconstructs a task-equivalent target-language instance, followed by validation of cross-field coherence and target-language consistency. Across LoRA experiments with four LLMs, training on directly translated data improves ROUGE-L and F-BERT on the Aya Evaluation Suite, but reduces accuracy on European-IFEval by 29.8% relative to the unadapted baseline. In contrast, adaptation with EuroAlpaca improves accuracy by 12.9% over the same baseline, reversing the degradation caused by direct MT, while also achieving the highest ROUGE-L and F-BERT scores on Aya. These results show that preserving task semantics is essential for multilingual instruction tuning.

1 Introduction

English-centric instruction data can be scaled through machine translation, but sentence-level fidelity does not ensure preservation of task-critical constraints and output relations. EuroAlpaca addresses this by routing examples through field-wise translation or task-level localisation, then validating the resulting target-language data.

  • Motivation: Machine translation scales multilingual instruction data but can preserve sentence meaning while corrupting the task encoded by the complete example.This problem affects tasks involving linguistic form, field roles, or required outputs.
  • Motivation: Task-preserving localisation constructs target-language instruction–input–output triples that preserve the task, constraints, difficulty and answer relation without requiring literal surface translation.
  • Approach: The proposed pipeline annotates tasks and domains, selects translation or localisation, preserves designated fields, and validates cross-field coherence and target-language consistency.It starts from 51,760 Alpaca Cleaned examples and produces a near-parallel resource.
  • Resources and evaluation: EuroAlpaca covers English and 50 European languages and regional varieties, while European-IFEVAL evaluates verifiable instruction following alongside Aya’s open-ended generation tasks.
  • Results: Direct MT improves reference similarity but reduces instruction-following accuracy, whereas EuroAlpaca improves both evaluation dimensions across four language-specific fine-tuning experiments.

2 Related Work

Prior multilingual instruction resources largely extend English data through translation, templating, curation or generated responses. Related evaluation work increasingly distinguishes reference similarity from compliance with objectively verifiable constraints.

  • Synthetic instruction data: Self-Instruct, Alpaca and Alpaca Cleaned established scalable synthetic instruction data with instruction, optional input and expected output fields.
  • Multilingual resources: Okapi, MITS and Bactrian-X extend Alpaca- or Dolly-style data across multiple languages using translation, ranking, or target-language response generation.
  • Instruction evaluation: IFEval evaluates objectively verifiable instruction compliance, while M-IFEval, Marco-Bench-MIF and XIFBench extend multilingual constraint-based evaluation.
  • From translation to localisation: Direct translation is scalable but can introduce translationese, knowledge–language mismatches and corruption in language-dependent tasks.
  • From translation to localisation: EuroAlpaca differs by treating multilingual adaptation as task localisation rather than translation alone, using routing and task-preserving transformations.

3 Task-Preserving Localisation

Task-preserving localisation represents each record as an instruction, optional input and output whose task relation must remain valid. A hierarchical pipeline routes examples between field-wise MT with preservation decisions and task-level reconstruction, followed by consistency validation.

  • Problem formulation: A source record x = (i, c, o) is transformed into a target-language record while preserving the task relation; literal equivalence of every field is neither necessary nor sufficient.
  • Routing: Sample-level routing selects field-wise MT or task-level localisation, and MT-routed samples receive TRANSLATE or PRESERVE policies for each non-empty field.
  • Handling regimes: The operational taxonomy distinguishes examples requiring target-language rewriting from those where specific fields should remain unchanged.
  • Handling regimes: The three handling regimes are direct translation of all non-empty fields, field-preserving translation, and task-level reconstruction when field-wise MT would invalidate the task.
  • Pipeline stages: Gemma-4 supports semantic annotation, policy prediction, task-preserving rewriting and validation, while NLLB-3.3B translates fields assigned TRANSLATE.
  • Pipeline stages: Final validation checks contradictions, wording, language errors, calques, mixed-language content and instruction–response mismatches while preserving task-required non-target spans.

4 Generated Resources

EuroAlpaca is a near-parallel multilingual instruction resource aligned by source identifier and task intent across 50 European languages and regional varieties. European-IFEVAL is a separate harmonised benchmark assembled from existing multilingual IFEval-style resources.

  • Near-parallel instruction data: EuroAlpaca covers 50 European languages and regional varieties in addition to an annotated English source.
  • Near-parallel instruction data: Records retain source identifiers and task intent while allowing lexical or structural differences when literal equivalence would invalidate the task.
  • Language coverage: The language set spans diverse language families, scripts and resource levels, including national, regional and minority varieties.
  • Language coverage: This diversity enables controlled multilingual instruction-following experiments while keeping the underlying supervision aligned.
  • European-IFEVAL: European-IFEVAL is separate from the localisation pipeline and combines existing resources into a common evaluation format for the same 50 target languages.
  • European-IFEVAL: Its records harmonise prompts, instruction-checker identifiers, arguments, sample identifiers and target-language response constraints for common scoring.

5 Experimental Setup

The experiments compare three supervision conditions across four multilingual models and 50 languages, using LoRA adapters and complementary generation and instruction-following evaluations.

  • Models and training design: Four multilingual 3–4B models are evaluated across 50 languages, producing 200 model–language pairs and 600 language-specific LoRA fine-tuning runs.The unadapted checkpoint serves as an additional baseline.
  • Supervision conditions: The three supervision conditions are Direct MT, Task-Preserved, and final EUROALPACA after validation.Direct MT translates all fields uniformly, whereas the other conditions incorporate routing, preservation, localisation, or validation.
  • Controlled comparisons: Training and evaluation settings remain fixed within each model–language pair, so differences primarily reflect the supervision data.Additional comparisons use Alpaca-derived subsets of Okapi, MITS, and Bactrian-X on shared languages and benchmarks.
  • Evaluation: Aya measures open-ended generation with ROUGE-L and FBERT, while EUROPEAN-IFEVAL measures strict compliance with verifiable instructions.Aya covers 36 languages with reference data, whereas EUROPEAN-IFEVAL covers all 50 languages.
  • Evaluation: Scores are macro-averaged across available non-English languages within each model and then averaged across the four models.This aggregation produces model-level and overall language-averaged comparisons.

6 Results

The results show that direct translation can improve reference similarity while reducing instruction-following accuracy, whereas task-preserving processing and validation improve both evaluation dimensions.

  • Reference similarity versus instruction following: Direct MT improves Aya ROUGE-L and FBERT for nearly all models but substantially reduces EUROPEAN-IFEVAL accuracy for every model.The exception is a small FBERT decrease for Llama; the contrast shows that reference similarity does not necessarily preserve the underlying task.
  • Final EUROALPACA results: EUROALPACA surpasses Direct MT by 0.020 in Aya ROUGE-L, 0.010 in Aya FBERT and 0.132 in EUROPEAN-IFEVAL accuracy at the macro-average.It achieves the highest scores across all models and metrics and wins all 200 model–language comparisons against Direct MT.
  • Task preservation: Task-Preserved improves ROUGE-L, FBERT and EUROPEAN-IFEVAL accuracy over Direct MT for all models and outperforms Direct MT in 199 of 200 comparisons.It also slightly exceeds the off-the-shelf mean.
  • Final validation: Final validation adds 0.016 Aya ROUGE-L, 0.008 Aya FBERT and 0.037 EUROPEAN-IFEVAL accuracy over Task-Preserved at the macro-average.EUROALPACA outperforms Task-Preserved in 180 of 200 comparisons, with one tie.
  • Existing resources: On shared-language comparisons, EUROALPACA beats Okapi and MITS on nearly all reported metrics but trails Bactrian-X in EUROPEAN-IFEVAL accuracy, 0.385 versus 0.401.The external-resource comparisons use differing shared-language subsets and therefore should not rank resources globally.
  • Cross-language alignment: Okapi and EUROALPACA have similar embedding-similarity scores, 0.867 versus 0.860, while Bactrian-X is lower at 0.800.The paper interprets embedding similarity as cross-language consistency rather than task validity or overall dataset quality.

7 Discussion

The results show a divergence between reference similarity and verifiable instruction following: direct MT can improve reference-based scores while reducing constraint-based accuracy. Task-preserving localisation and validation address this gap, though cross-resource comparisons do not isolate the effect of localisation alone.

  • Direct MT improves ROUGE-L and FBERT on Aya but decreases EUROPEAN-IFEVAL accuracy by 0.092.This divergence shows why multilingual instruction data requires both reference- and constraint-based evaluation.
  • Task-Preserved training gains 0.095 accuracy over Direct MT, while final validation adds a further gain of 0.037.The ablation attributes these gains to routing, field preservation, task-level localisation, cross-field consistency and target-language enforcement.
  • EUROALPACA outperforms Okapi and MITS on shared-language subsets, while Bactrian-X achieves higher EUROPEAN-IFEVAL accuracy on its language set.EUROALPACA nevertheless achieves cross-language embedding similarity close to Okapi’s and higher than Bactrian-X’s.
  • Embedding similarity is the mean cosine similarity between BGE-M3 representations of aligned instruction–input–output records across the common 11-language intersection.
  • Differences in external-resource construction prevent attributing performance gaps solely to task-preserving localisation.

8 Conclusion

EUROALPACA is an LLM-guided pipeline and near-parallel resource for task-preserving instruction-data localisation across 50 European languages and regional varieties. Across four LLMs, it reverses the instruction-following degradation caused by direct MT while also improving open-ended generation metrics.

  • EUROALPACA combines policy prediction, policy-guided MT, task-level rewriting and target-language and cross-field coherence validation.
  • Direct MT reduces EUROPEAN-IFEVAL accuracy by 29.8% relative to the unadapted baseline, whereas EUROALPACA improves it by 12.9%.EUROALPACA also improves open-ended generation metrics and outperforms direct MT in all 200 model–language pairs on instruction following.

Limitations

The study is limited to European languages and regional varieties, and its model-assisted processing may inherit errors or biases. Results also use a single training seed and non-comprehensive external-resource comparisons.

  • The study covers only European languages and regional varieties, although the approach could apply to other regions.
  • Model-assisted labels, policies, localisations and validation edits may inherit errors or biases, especially where Gemma-4 has limited exposure to low-resource varieties.The authors also note that a larger, systematic error analysis would better assess quality across languages and processing stages.
  • A single training seed prevents estimation of optimisation variance, while external-dataset comparisons are restricted to matched model–language cells.Those comparisons should not be interpreted as comprehensive evaluations of the external resources.

Data Availability Statement

EUROALPACA and EUROPEAN-IFEVAL were not publicly available at submission because of double-blind review. The authors plan to release both datasets, construction code and intermediate annotations upon acceptance.

  • EUROALPACA and EUROPEAN-IFEVAL were withheld during double-blind review and were planned for release upon acceptance.
  • The planned release includes complete data-construction code and intermediate annotations from semantic labels through validation decisions and post-edit metadata.

AI-Generated Content Disclosure

The manuscript discloses generative-AI assistance and presents taxonomies and routing analyses used in constructing the dataset. These materials cover task and domain labels, routing reasons, and field policies.

  • Generative AI assisted paraphrasing, manuscript clarity and grammar, and coding, with all generated content reviewed and validated by the authors.
  • The source examples are organized with predefined task and domain taxonomies, whose primary-label distributions are shown in Figures 4 and 5.
  • Table 9 reports reasons for routing 11,088 examples to task-level localisation.
  • Table 10 reports field policies for 40,672 examples passing the whole-sample translation gate, counting empty inputs as preserved.

A.2 Examples Across Task-Preserving Handling Regimes

The paper documents task-preserving handling regimes, multilingual coverage, evaluation resources, and shared implementation settings for language-specific fine-tuning. Examples use translated templates with explicit target-language response requirements.

  • Handling regimes: Table 11 contrasts failure cases with task-preserving target-language constructions for examples requiring task-level localisation.
  • Handling regimes: Direct field-wise translation preserves tasks in some examples, while others require preserving labels, code, literal tokens or fixed-language outputs.
  • Resource coverage: EUROALPACA covers 50 European languages and regional varieties, with internal resource tiers reflecting digital-resource availability rather than language vitality or sociolinguistic status.
  • Evaluation resources: EUROPEAN-IFEVAL provides language-level provenance information, including URLs, source tiers, missing-sample counts and source-language answer-constraint counts.
  • Evaluation resources: EUROPEAN-IFEVAL accuracy is reported across 50 languages, while Aya ROUGE-L and FBERT results are reported across 36 languages.
  • Implementation: All language-specific runs share one optimisation and adapter configuration, with a 1,600-token limit for ga, gd, is, mk, mt and sq.
Loading 2609.05043v1…