PaperlayerView exact revision on arXiv

Source-linked AI summary

Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generation

Lester James V. Miranda, Ivan Vulić, Anna Korhonen

arXiv:2604.11290v2cs.CL

TL;DR

Multilingual SFT data is scarce, while teacher selection is often based on model size rather than demonstrated multilingual teaching effectiveness. The paper evaluates 10 models across 6 languages using PG-SCORE, combining synthetic-data quality with student performance, and finds that data characteristics and teacher–student family matching matter more than scale alone. It also identifies generation strategies that can improve results for less-resourced languages.

  • Problem

    Less-resourced languages lack high-quality prompt-response data, while teacher models are often selected arbitrarily or by size despite uncertain multilingual effectiveness.

  • Method

    The study evaluates 10 language models across 6 languages, generating over 1.4M SFT examples and finetuning 240 students while combining intrinsic data-quality metrics with extrinsic student performance in PG-SCORE.

  • Results

    Model scale does not significantly predict teacher effectiveness; prompt diversity, response fluency, and length capture over 93% of intrinsic-quality variance and predict student performance with R2=0.664.

  • Takeaways & Limitations

    Matching teacher and student model families is a reliable selection heuristic, while Respond or Translate can improve synthetic-data generation for less-resourced languages.

  • Takeaways & Limitations

    The evaluation covers six languages, and Translate assumes English prompts that can be meaningfully translated while risking culture-localization problems and translationese artifacts.

Abstract

from arXiv · show

Synthesizing supervised finetuning (SFT) data from language models (LMs) to teach smaller models multilingual tasks has become increasingly common. However, teacher model selection is often ad hoc, typically defaulting to the largest available option, even though such models may have significant capability gaps in non-English languages. This practice can result in poor-quality synthetic data and suboptimal student downstream performance. In this work, we systematically characterize what makes an effective multilingual teacher. We measure intrinsic measures of data quality with extrinsic student model performance in a metric we call Polyglot Score; evaluating 10 LMs across 6 typologically diverse languages, generating over 1.4M SFT examples and training 240 student models. Among the models tested, Gemma 3 27B and Aya Expanse 32B emerge as consistently effective teachers across different student base model families. Further analyses reveal that model scale alone does not significantly predict teacher effectiveness; instead, data qualities such as prompt diversity, length, and response fluency capture over 93.3% of variance in intrinsic data quality and predict student performance. Finally, we provide practical recommendations, including matching the model families of teacher-student pairs and translating from or responding to existing prompts, which can yield improvements for less-resourced languages. We hope that our work advances data-centric research in multilingual synthetic data and LM development.

1 Introduction

The paper addresses scarce multilingual SFT data and the ad hoc selection of teacher models by introducing a systematic evaluation of multilingual synthetic data generation. It combines intrinsic data-quality measures with downstream student performance in the POLYGLOT SCORE and derives practical teacher-selection recommendations.

  • Motivation: Scarce prompt-response pairs for less-resourced languages make human-generated multilingual SFT data a development bottleneck.Synthetic generation is presented as a way to reduce the human effort required to create language-specific training data.
  • Motivation: Existing work often defaults to the largest benchmark-performing teacher models, although strong overall performance may not imply strong multilingual capabilities.The paper frames this practice as problematic for multilingual synthetic data generation.
  • Approach: The study evaluates 10 language models across 6 typologically diverse languages and three generation methods: Generate, Translate, and Respond.The methods respectively create new pairs from few-shot examples, translate English prompts, or answer target-language prompts.
  • Approach: POLYGLOT SCORE combines intrinsic synthetic-data quality with extrinsic multilingual student performance to assess teacher effectiveness for each target language.Intrinsic measures include diversity, base-model perplexity, and reward-model response quality; extrinsic evaluation covers cultural understanding, mathematical reasoning, and general chat.
  • Findings: The experiments generate over 1.4M SFT examples and finetune 240 students, with Gemma 3 27B consistently ranking among the top three teachers across student families.The Gemma family also outperforms Llama 3.1 and IBM Granite in the reported comparisons.
  • Recommendations: Data-generation recommendations include matching teacher and student model families and using existing prompts or English translation for less-resourced languages.The reported gains vary by teacher model, and the paper also releases code, data, and models.

2 Evaluating Language Models as Multilingual Teachers

The evaluation framework measures multilingual teacher quality through synthetic-data construction, intrinsic quality and diversity metrics, student finetuning, and downstream multilingual evaluation. These components are combined into a single PG-SCORE for comparing teachers across languages and model families.

  • POLYGLOT SCORE: PG-SCORE combines a teacher’s intrinsic synthetic-data quality with the extrinsic performance of a student finetuned on that data.The score is designed to provide a single comparison of both data quality and student outcomes.
  • Seed data: Seed datasets aggregate publicly available multilingual instruction-tuning resources for each target language and include English instruction data for translation scenarios.The listed sources include the Aya Collection, WildChat 4.8-M, EuroBlocks-SFT, Magpie-Align, Tülu 3 SFT, HelpSteer3, and GSM8K.
  • Synthetic data generation: Teachers generate prompt-response datasets using Generate, Translate, or Respond methods conditioned on few-shot examples, translated English prompts, or existing target-language prompts.Each method produces a synthetic dataset of N prompt-response pairs.
  • Data quality and diversity: Intrinsic evaluation measures prompt and response diversity, response perplexity, and prompt-response quality using embedding-based and model-based metrics.Lower perplexity indicates greater fluency and naturalness, while M-Prometheus provides reward-model quality scores.
  • POLYGLOT SCORE: PG-SCORE uses z-score-normalized intrinsic and extrinsic metrics, with higher values indicating better synthetic-data quality and student performance.Equal weighting is used as the baseline for the combined score.
  • Student model performance: Students are finetuned on teacher-generated data and evaluated on cultural understanding, general chat, and mathematical reasoning tasks.The task suites are Global-MMLU Lite, M-RewardBench, and M-GSM.

3 Experiments: Evaluating LMs and PG-SCORE Generalization

The experiments evaluate 10 teacher LMs across six languages using PG-SCORE, then test whether teacher rankings generalize across student base models and data-generation methods.

  • Experimental Setup: 10 state-of-the-art LMs were evaluated across 6 typologically diverse languages, using balanced generation methods and repeated random seeds.Students were finetuned from OLMo 3 7B on teacher-generated data.
  • Teacher Effectiveness: Gemma 3 27B achieved the highest average PG-SCORE at 0.726, followed by Aya Expanse 32B at 0.706.Both outperformed Llama 3.1 70B Instruct, which scored 0.140.
  • Teacher Effectiveness: Smaller Gemma teachers also ranked highly: Gemma 3 12B scored 0.595 and Gemma 3 4B scored 0.469.Llama 3.1 70B Instruct ranked ninth despite being the largest evaluated model.
  • Cross-Language Results: Teacher performance varied by language, with German and Spanish scoring highest across models while Arabic was challenging for most teachers.The results suggest that language-specific factors influence teacher effectiveness.
  • Generalization Across Base Models: Gemma 3 27B and Aya Expanse 32B remained among the top three teachers across alternative student bases, while ranking correlations ranged from ρ=0.57 to ρ=0.87.Family-matched Gemma teacher-student pairs achieved at least +20.5% higher PG-SCORE than the worst pair.
  • Data Generation Methods: Generate performed best for German, whereas Respond or Translate were more effective for less-resourced Arabic and Indonesian.The study therefore used an equal mix of the three methods in its core teacher comparison.

4 Analysis: What Makes a Good Polyglot Teacher?

The analysis tests whether model scale and benchmark strength explain multilingual teacher quality, then examines which intrinsic data characteristics predict student performance.

  • Do Stronger Models Make Better Teachers?: Neither parameter size nor average multilingual benchmark performance significantly predicted PG-SCORE.A 1-unit increase in log(Param. Size) corresponded to a non-significant 0.053 increase in PG-SCORE.
  • Intrinsic Data Quality: Prompt diversity, prompt length, response fluency, and response quality characterize important dimensions of effective multilingual teacher data.The principal-component loadings associate good teachers with diverse, high-quality responses and diverse, longer prompts.
  • Intrinsic Data Quality: The first four principal components explained over 93.3% of the variance in intrinsic data quality metrics.PC1 captured response perplexity and distinctiveness, while PC2 captured prompt diversity and length.
  • Predicting Student Performance: Intrinsic-metric principal components predicted multilingual student performance with R2 = 0.664 and RMSE = 0.440.The result indicates that a simple linear model can use intrinsic metrics to predict extrinsic performance.

5 Discussion: Towards a Recipe for Multilingual Synthetic Data Generation

The discussion turns the experiments into practical guidance: prioritize generated-data quality, match teacher and student model families, and adapt generation methods to language resource levels.

  • Teacher Selection: Prompt diversity, response fluency, and length capture over 93% of intrinsic data-quality variance and predict student performance with R2=0.664.These metrics offer a cheaper alternative to full student-training runs for screening teacher candidates.
  • Teacher-Student Pairing: Matching teacher and student model families is a reliable heuristic when the optimal teacher is unknown.Gemma-with-Gemma and Llama-with-Llama pairings outperformed a mismatched baseline by at least 20%.
  • Language-Dependent Generation: Generate is recommended for high-resource German, while Respond or Translate can improve results for less-resourced Arabic and Indonesian.For truly low-resource languages, the authors recommend combining synthetic generation with targeted data collection.
  • Held-Out Language Demonstration: A held-out Tagalog recipe based on PG-SCORE analyses improved performance on an unseen Filipino-centric benchmark.Each recommendation component produced observable performance gains in the supplementary demonstration.

6 Related Works

Prior multilingual synthetic-data work used varied generation strategies and often evaluated pipeline components separately; this paper evaluates core strategies jointly to derive teacher-selection guidance.

  • Multilingual Synthetic Data Generation: Prior datasets such as Bactrian-X, MultiAlpaca, and xP3 were created using Translate, Generate, and Respond strategies, respectively.These projects motivated examining the three core multilingual synthetic-data generation strategies in a common setup.
  • Multilingual Synthetic Data Generation: The study tests Generate, Translate, and Respond in isolation to provide empirically grounded guidance for selecting teacher LMs.The intended recipe is designed to apply across generation methods.
  • Evaluating the Synthetic Data Pipeline: Earlier evaluations often isolated intrinsic or extrinsic pipeline measures or focused exclusively on English tasks.This work instead combines intrinsic data quality with extrinsic multilingual student performance in PG-SCORE.

7 Conclusion

The paper evaluates multilingual teacher models using intrinsic data-quality measures and extrinsic student performance, identifying factors beyond model size that contribute to teacher effectiveness.

  • The evaluation combines intrinsic synthetic-data quality with extrinsic multilingual student performance to assess teacher effectiveness.The study measures prompt and response diversity, response perplexity, reward-model quality, and student performance.
  • Teacher effectiveness depends on properties beyond model size or benchmark performance, including prompt-response diversity, fluency, and language representation.
  • The paper proposes practical recommendations for constructing multilingual synthetic-data generation recipes.These recommendations are intended to support future development of more inclusive language technologies through higher-quality synthetic data.

Limitations

The study’s findings are limited by its six-language evaluation scope, and broader validation remains necessary. Its translation-based method also depends on suitable English prompts and can inherit localization and translationese problems.

  • The evaluation covers six carefully selected languages, so validating the findings across a broader language sample remains future work.The selection reflects benchmark availability and prior theoretical work on principled test-language selection.
  • Translate assumes access to English prompts that can be meaningfully translated into target languages.
  • Translation-based generation inherits limitations involving culture-specific references and translationese artifacts.

Ethics Statement

The paper situates multilingual synthetic-data generation within established Generate, Translate, and Respond approaches and releases resources to support future research. It also documents benchmark, dataset, model, and training settings underlying the evaluation.

  • Prior work and ethics context: Most surveyed multilingual synthetic-data methods fall into Generate, Translate, or Respond, the three approaches tested in this work.The survey indicates that these categories cover the majority of approaches used in multilingual synthetic-data generation.
  • Released resources: The POLYGLOT collection provides synthetic datasets and student models generated using the best teacher across target languages.The collection includes datasets produced by each teacher and 8B students finetuned with Gemma 3 27B.
  • Data and evaluation setup: Student finetuning and multilingual evaluation use specified computational, framework, benchmark, formulation, and trial settings.The reported setup includes Unsloth, Grace Hopper GH200 Superchips, Lighteval, multiple-choice evaluation, and three random-seed trials.

F Full Results for Intr. and Extr. Metrics

The supplementary materials provide complete data-quality and student-performance results, seed-data statistics, teacher-model details, and analyses of protocol choices affecting PG-SCORE.

  • Full results: Tables 11 and 12 report data-quality metrics and student-model results for every teacher across all target languages.
  • Dataset and model details: The seed dataset statistics document the multilingual examples used to bootstrap synthetic-data generation.
  • Dataset and model details: Teacher-model details compare 10 models across providers, sizes, multilingual capabilities, and licensing terms.
  • Full results: The supplementary tables quantify PG-SCORE changes from family-matched teacher-student pairs and teacher-language-specific generation methods.These comparisons use the OLMo 3 7B and equal-mix baselines, respectively.
  • Ablations: Ablation analyses examine evaluation-protocol aspects that may affect teacher-model PG-SCORE.

G.1 Effect of Data Scale on Student Model Performance

Student performance improves as synthetic-data scale increases, but gains diminish beyond 10k examples. The study therefore uses 10k synthetic examples per language as sufficient for reliably computing PG-SCORE without inflating it through sample count.

  • Results: 10k synthetic examples per language are sufficient to reliably compute PG-SCORE without inflating the metric by increasing sample count.The experiment evaluates 1k, 5k, 10k, 25k, and 50k SFT instances using Gemma 3 27B across Arabic, German, and Indonesian.
  • Results: Student performance improves with more synthetic data, but gains diminish beyond 10k examples.This pattern is reported for average student performance as a function of SFT-instance count.
  • Scope of the scale finding: Scaling beyond 10k instances can still improve performance on broader task distributions such as FILBENCH.The paper notes continued gains from 10k to 25k instances on FILBENCH, whose task diversity differs from the main experimental benchmarks.

I.3 Analysis: Ablation Experiments

The ablations show that teacher selection and teacher–student family matching improve held-out Tagalog performance more than simply increasing data scale. Native-speaker inspection further links higher PG-SCORE with stronger cultural accuracy, while fluency is not monotonic across teachers.

  • Experimental setup: The ablation evaluates additive interventions on held-out Tagalog using FILBENCH, including teacher choice, family matching, data scale, and student model scale.The setup compares synthetic-data mixes and reports model performance on the held-out language.
  • Curation of publicly-available data vs. Synthetic data generation: 0.5pp separates publicly available Tagalog SFT data from GPT-4o-generated synthetic data, indicating no significant synthetic-data advantage with a non-optimal teacher.The authors note that some public datasets may themselves be semi-synthetic, complicating the comparison.
  • Matching teacher and student model families: Family-matched teacher–student pairs achieve at least +20.5% higher PG-SCORE than mismatched pairs.The authors attribute this likely to shared tokenization schemes and architectural similarities that facilitate knowledge transfer.
  • Increase data scale: 0.21pp is the modest gain from increasing synthetic instances from 10k to 25k, smaller than gains from teacher selection and family matching.The continued improvement on FILBENCH’s diverse task distribution suggests saturation points may depend on the task.
  • Instance-level qualitative inspection of generated data: Higher PG-SCORE teachers produce higher-quality Tagalog data overall, with cultural accuracy increasing monotonically with PG-SCORE.Gemma 3 27B has the highest PG-SCOREtl (1.049) and the strongest ratings on both fluency and cultural accuracy.
  • Instance-level qualitative inspection of generated data: Aya Expanse receives a lower fluency score than GPT-4o, 2.480 vs. 2.800, despite having a higher PG-SCOREtl.Gemma 3 27B exceeds both other teachers by at least 1.3 on fluency and 0.9 on cultural accuracy.
Loading 2604.11290v2…