PaperlayerView exact revision on arXiv

Source-linked AI summary

X-Instruction: Aligning Language Model in Low-resource Languages with Self-curated Cross-lingual Instructions

Chong Li, Wen Yang, Jiajun Zhang, Jinliang Lu, Shaonan Wang, Chengqing Zong

arXiv:2405.19744v1cs.CLcs.AI

TL;DR

Low-resource-language responses lag, and translated English instructions can contain errors or miss language-specific knowledge. X-Instruction generates English instructions for multilingual texts, then refines and diversifies the resulting cross-lingual samples. Across evaluated languages, tuned models surpass translation and distillation baselines, even exceeding ChatGPT, and can follow instructions in the output language without further tuning.

  • Problem

    Low-resource-language responses lag behind high-resource-language responses, while high-quality instruction data are limited and direct translation can introduce errors and miss cultural or language-specific knowledge.

    Source: §1 · p. 1

  • Method

    X-Instruction generates English instructions for multilingual-corpus texts used as responses, then refines candidate samples and diversifies the final dataset.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    On four instruction-following benchmarks, X-Instruction-tuned models surpassed translation and distillation baselines and even exceeded ChatGPT in the evaluated languages.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Models tuned on cross-lingual instruction samples can follow instructions in the output language without further tuning.

    Source: §1 · p. 2, §6 · p. 9

    Open-ended generation was evaluated only in single-turn conversations because of limited OpenAI account quota, and results may differ on multi-turn benchmarks.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models respond well in high-resource languages like English but struggle in low-resource languages. It may arise from the lack of high-quality instruction following data in these languages. Directly translating English samples into these languages can be a solution but unreliable, leading to responses with translation errors and lacking language-specific or cultural knowledge. To address this issue, we propose a novel method to construct cross-lingual instruction following samples with instruction in English and response in low-resource languages. Specifically, the language model first learns to generate appropriate English instructions according to the natural web texts in other languages as responses. The candidate cross-lingual instruction tuning samples are further refined and diversified. We have employed this method to build a large-scale cross-lingual instruction tuning dataset on 10 languages, namely X-Instruction. The instruction data built using our method incorporate more language-specific knowledge compared with the naive translation method. Experimental results have shown that the response quality of the model tuned on X-Instruction greatly exceeds the model distilled from a powerful teacher model, reaching or even surpassing the ones of ChatGPT. In addition, we find that models tuned on cross-lingual instruction following samples can follow the instruction in the output language without further tuning.

1 Introduction

Low-resource-language responses lag behind, and translating English instructions can introduce errors and miss cultural or language-specific knowledge. X-Instruction builds cross-lingual samples from multilingual texts, and its tuned models outperform translation and distillation baselines, even exceeding ChatGPT in the evaluated languages.

1 Introduction

  • Low-resource-language responses are inferior to high-resource-language responses, with limited pretraining data and a lack of high-quality instruction data among the proposed explanations.Teacher distillation may fail when the teacher responds poorly in the target language; suitable annotators are also hard to find, while direct translation can introduce errors and miss cultural nuances.Source: §1 · pp. 1–2
  • The pipeline generates English instructions for texts in other languages, then iteratively refines candidate samples and diversifies the final dataset across instruction-embedding clusters.The candidates use multilingual texts as responses, retaining native expressions and language-specific knowledge.Source: §1 · p. 2
  • On four instruction-following benchmarks, X-Instruction-tuned models surpassed translation and distillation baselines and even exceeded ChatGPT in the evaluated languages.The dataset contains 320k samples across 10 languages, including five medium-resource and five low-resource languages.Source: §1 · p. 2
  • Models tuned on cross-lingual instruction samples can follow instructions in the output language without further tuning.The dataset is built for 10 languages, including five low-resource languages.Source: §1 · p. 2

2 Related Works

Prior multilingual instruction work uses translation or teacher distillation, while self-alignment iteratively generates and filters samples. X-Instruction instead mines multilingual corpora using English generation capabilities.

2 Related Works

  • Instruction generation commonly uses powerful-model distillation or human labeling; self-alignment methods iteratively generate and filter instructions without strong models or extensive annotation.The cited self-alignment approaches are Bai et al. (2022), Sun et al. (2023), and Li et al. (2023c).Source: §2 · p. 2
  • Self-alignment is difficult to apply when language models cannot generate high-quality instructions in unseen languages.X-Instruction avoids this weakness by exploiting stronger generation in English.Source: §2 · pp. 2–3
  • Multilingual instruction tuning has used English-to-target translation, while Li et al. (2023b) translated 67k instructions and distilled language-specific outputs from ChatGPT.The latter approach showed better responses and multilingual abilities than tuning on translated samples.Source: §2 · p. 3
  • Unlike distillation from stronger models, X-Instruction mines high-quality samples from neural multilingual corpora and applies to languages unsupported by teacher models..Source: §2 · p. 3

3 Method

X-Instruction pairs an instruction in one language with an output in another, then generates, refines, and diversifies samples drawn from multilingual corpora.

3 Method

  • The pipeline generates candidate English instructions for multilingual-corpus texts, discards inappropriate samples, then diversifies the remainder across instruction-embedding clusters.These are the three stages illustrated for X-Instruction sample construction.Source: §3 · p. 3

3.1 X-Instruction Definition

  • An X-Instruction sample pairs an instruction in language a with an output in a different language b; the optional input is ignored for simplicity.The example uses an English instruction and a Urdu output.Source: §3.1 · p. 3

3.2 X-Instruction Generation

  • The model learns from seed samples to generate English instructions given texts in other languages, treating some multilingual texts as suitable responses.This yields candidate cross-lingual instruction-following samples.Source: §3.2 · p. 3

3.3 X-Instruction Refinement

  • Each refinement iteration trains an evaluator on synthetic three-level ratings: vanilla outputs are best, mismatched outputs worst, and modified or flawed outputs intermediate.The evaluator is used to identify inappropriate samples and select better cross-lingual examples.Source: §3.3 · p. 3
  • As iterations proceed, selected samples join the seed data for tuning, improving model outputs and enabling the evaluator to find better samples.The default number of refinement iterations is three.Source: §3.3 · p. 3

3.4 X-Instruction Diversification

  • The method embeds instructions, clusters them with k-means, and samples equal numbers from each cluster to reduce dominance by a few domains.The cluster-based sampling is intended to increase dataset diversity.Source: §3.4 · p. 4, Figure 2 · p. 4

4 X-Instruction Dataset

This section presents X-Instruction, a 320k-sample cross-lingual dataset for 10 languages, and describes its source data, instruction diversity, and quality review.

4 X-Instruction Dataset

  • Using LLaMA-2-7B, the authors applied their automatic pipeline to build X-Instruction, a 320k-sample dataset spanning 10 languages.They plan to make the codes and data public after review.Source: §4 · p. 4

4.1 Seed Data

  • Seed data combine first-turn Open Assistant conversations and highest-quality responses with Google-translated outputs from 3k English samples per language, supplementing scarce human-labeled data.Source: §4.1 · p. 4

4.2 Unsupervised Multilingual Corpus

  • The multilingual corpus uses CulturaX web texts, filtered from mC4 and OSCAR, with 1M texts sampled per language due to computation limits.Source: §4.2 · p. 4

4.3 Statistics

  • Figure 3 shows the 16 most common root verbs and their top direct noun objects; the authors report broad diversity in the verb–noun instructions.Source: Figure 3 · p. 4, §4.3 · p. 4

4 X-Instruction Dataset

  • Table 2 reports 32k cross-lingual instruction samples for each language.Source: Table 2 · p. 5

4.4 Quality

  • More than 80% of reviewed samples were valid; reported issues included navigation-bar content (11.3%) and incorrect URLs or HTTP status codes (3.8%).The review randomly sampled 200 instructions and corresponding texts per language.Source: §4.4 · pp. 4–5, Table 3 · p. 5

5 Experiments

Across four benchmarks in three low-resource languages, X-Instruction outperforms translation and distillation baselines, with X-Instruction13B reaching a 67.66% win rate against ChatGPT. Evaluations also find stronger responses in human judgments, retained performance under output-language prompting, and gains on multilingual understanding tasks.

5.2 Evaluation Results from GPT-4

  • On four benchmarks across three low-resource languages, GPT-4 evaluations show X-Instruction13B achieved a 67.66% win rate against ChatGPT and X-Instruction outperformed Alpaca-MT and Bactrian-X across all benchmarks.The cited passage says Bactrian-M models reach 15.8%, not that X-Instruction improves by that amount.Source: §5.2 · pp. 5–6, Table 4 · p. 6
  • X-Instruction models using 3k labeled seed examples per language achieve the highest average win rate across ten languages, and average 6.9 GPT-4 score across languages.The average score is 1.3 higher than Bactrian-X; Bactrian-M scores 5.6.Source: §5.2 · p. 6, Table 5 · p. 6
  • Under output-language zero-shot prompting, average win rate drops only 5.7%, while X-Instruction13B surpasses ChatGPT at every WizardLM difficulty level.Average response quality falls by 0.7 and remains 90.6% of the vanilla evaluation; Figure 5 covers Easy, Medium, and Hard prompts.Source: §5.2 · pp. 6–7, Figure 5 · p. 7
  • X-Instruction13B beats Bactrian-M in every Vicuna category and ChatGPT in eight categories, with fermi the exception.The authors suggest native multilingual corpus content may contribute to its commonsense, writing, and knowledge performance; they attribute weaker code and math results to relevant corpora filtered during language identification.Source: §5.2 · p. 7, Figure 4 · p. 7

5.3 Evaluation Results from Human

  • Human evaluators prefer X-Instruction responses to ChatGPT and Bactrian-M13B in three low-resource languages, consistent with GPT-4 evaluations.The human comparison uses randomized answer order to reduce position bias.Source: §5.3 · p. 8, Figure 6 · p. 8

5.4.1 Iteration of Refinement

  • More refinement iterations improve Urdu and Bengali win rates, with response-quality gains nearly saturated at three iterations.The comparison uses 32k cross-lingual samples on four benchmarks.Source: §5.4.1 · p. 8, Figure 7 · p. 8

5.4.2 Data Quantity and Quality

  • Response quality rises with more Urdu samples and approaches saturation at 32k; higher-rated samples also yield better responses.These results use samples from the third refinement iteration.Source: §5.4.2 · p. 8

5.4.3 Multilingual Tasks

  • On zero-shot in-context NLI and reasoning tasks, X-Instruction improves base-model performance by 3.1%, versus 1.8% for Bactrian-X.The authors conclude that X-Instruction tuning enhances multilingual language-understanding abilities.Source: §5.4.3 · pp. 8–9, Table 7 · p. 8

6 Conclusion

The paper presents X-Instruction, a pipeline and dataset for cross-lingual instruction tuning, and reports improved response generation and output-language instruction following without further tuning.

6 Conclusion

  • Models tuned on X-Instruction generate better responses and follow instructions in the output language without further tuning.The dataset was constructed for 10 languages.Source: §6 · p. 9
  • Future work could focus on directly generating high-quality instruction-following samples in low-resource languages.The authors hope the work brings more attention to instruction construction for these languages.Source: §6 · p. 9

Limitations

The authors identify constraints in the model used to create samples, the number of languages covered, and the evaluation setting.

Limitations

  • X-Instruction samples were excavated with LLaMA-2-7B, whose English-dominant vocabulary makes multilingual-corpus processing less efficient.The authors note that more powerful base models could produce higher-quality samples.Source: Limitations · p. 9
  • Limited computation resources restricted this study's application of the method to 10 languages.Other languages are left for future investigation.Source: Limitations · p. 9
  • Open-end generation was evaluated only in single-turn conversations, and results might differ on multi-turn dialogue benchmarks.The authors cite their limited OpenAI account quota as the reason for this evaluation scope.Source: Limitations · p. 9

Ethical Considerations

The paper notes possible cultural bias and toxic content in web-derived samples, alongside implementation and seed-data details.

Ethical Considerations

  • Web-derived instruction-tuning samples may contain cultural bias and toxic content; rigorous corpus cleaning or safety evaluation during refinement are possible mitigations.The authors leave these mitigation approaches for future investigation.Source: Ethical Considerations · p. 9

A Hyperparameters

  • Training used AdamW at an initial learning rate of 1e-5 for 3 epochs, with cosine scheduling, mixed precision, and ZeRO.Batch size was 32 for most models and 8 when training samples numbered under 3k; response generation used nucleus sampling with p = 0.9 and T = 0.7. The experiments ran on 8 A100 80GB GPUs.Source: A Hyperparameters · pp. 11–12

B.1 Seed Data

  • No Open Assistant seed samples were found for the five low-resource languages, so all their seed outputs were translated from English samples.Table 8 reports statistics for the cross-lingual instruction-tuning seed data with English instructions; low-resource languages are marked with a star.Source: §B.1 · p. 12, Table 8 · p. 12

B.2 Datasets

The appendix describes evaluation datasets and reports dataset-length statistics for multilingual instruction-tuning baselines.

B.2 Datasets

  • Vicuna and WizardLM include diverse instruction categories, which the authors say supports test-set richness and reduces evaluation-bias risk.The section points to the four open-end generation datasets detailed in Table 9.Source: §B.2 · p. 12, Table 9 · p. 12
  • Table 10 reports the average lengths of baseline multilingual instruction-tuning datasets across ten languages.The passage supplies the table's scope but no length values.Source: Table 10 · p. 12, §B.2 · p. 12

C.1 Detailed Results from GPT-4

The section reports detailed comparisons of X-Instruction models with ChatGPT and examines how diversification and instruction categories relate to performance. The provided results describe the comparisons’ scope and selected patterns, but not enough values to state specific benchmark winners.

C.1 Detailed Results from GPT-4

  • X-Instruction13B is compared with ChatGPT across four datasets, and Table 11 reports GPT-4 evaluations of multilingual models on 10 languages across two benchmarks.Figure 8 covers four benchmarks in three low-resource languages; Table 11 includes win rates to ChatGPT and GPT-4 scores, with zero-shot results marked separately.Source: §C.1 · p. 12, Figure 8 · p. 13, Table 11 · p. 13

C.2 Diversification

  • With 32k final samples, tuned-model output quality drops at 2,000 clusters; the authors set 1,000 clusters as the default.The authors suggest the drop may reflect smaller inter-cluster distances and less diversity in sampled data.Source: §C.2 · p. 12

C.3 Improvement over Seed Model

  • For Turkish, Swahili, and Urdu, X-Instruction7B improves performance over the seed model on prompts from all categories.The passage notes especially strong improvement for a category but is truncated before naming it.Source: §C.3 · p. 12

D.1 Quality Evaluation

The appendix shows examples of valid and invalid cross-lingual instructions, discusses response quality and evaluation procedures, and documents prompt, language-code, and dataset-license information.

D.1 Quality Evaluation

  • Invalid examples contain inappropriate instructions, but their generated English instruction semantics remain related to the source text.The quality-review interface asks annotators to read the English instruction and corresponding text before answering two questions.Source: §D.1 · p. 13, Figure 10 · p. 15, Figure 11 · p. 16

D.2 Instruction Following

  • X-Instruction13B often gives more detailed, coherent responses, but a code-generation example offers suggestions instead of code, possibly reflecting limited multilingual-corpus code data.The examples span math, roleplay, knowledge, and code-generation instructions.Source: §D.2 · p. 13, Figure 14 · p. 18, Figure 15 · p. 19

E.1 Quality Evaluation

  • Five annotators evaluated X-Instruction samples, with payment of $0.1 per evaluated sample.The review interface is illustrated in Figure 11.Source: §E.1 · p. 13, Figure 11 · p. 16

E.2 Response Comparison

  • Three evaluators chose their preferred response in pairwise comparisons and received $0.2 per annotation.The comparison interface is illustrated in Figure 12.Source: §E.2 · p. 14, Figure 12 · p. 17

F Prompt Template in GPT-4 Evaluation

G Additional Information about Language Code

H Dataset License

  • X-Instruction is built from CulturaX, which is filtered from mC4 and OSCAR, and its license follows the mC4 and OSCAR licenses.The passage identifies ODC-BY4 for mC4 and Creative Commons CC0 for OSCAR.Source: H Dataset License · p. 14

Related papers

Loading 2405.19744v1…