Source-linked AI summary
Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback
Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan A. Rossi, Thien Huu Nguyen
TL;DR
Open-source multilingual instruction tuning has largely focused on English, popular languages, and SFT, leaving RLHF insufficiently studied for broader language coverage. Okapi addresses this gap with an RLHF-based framework, multilingual data, and evaluation resources spanning 26 languages. Experiments report benefits of RLHF over SFT, while identifying low-resource languages and broader language coverage as remaining challenges.
Problem
Existing open-source instruction-tuned LLMs mainly cover English and popular languages, while multilingual work has used SFT without examining RLHF.
Method
Okapi builds an open-source RLHF-based multilingual instruction-tuning framework with translated instructions, response rankings, benchmarks, and fine-tuned models across 26 languages.
Results
RLHF generally performs better than SFT across multilingual tasks, base models, and language groups, with average improvement reaching 2.5% on HellaSwag with LLaMA.
Takeaways & Limitations
Okapi provides multilingual RLHF models and resources, while the weaker gains for low-resource languages identify a continuing research challenge.
Takeaways & Limitations
The study covers 26 languages and uses only BLOOM and LLaMA base models with 7B parameters, leaving broader language and model coverage for future work.
Abstract
from arXiv · showhide
A key technology for the development of large language models (LLMs) involves instruction tuning that helps align the models' responses with human expectations to realize impressive learning abilities. Two major approaches for instruction tuning characterize supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF), which are currently applied to produce the best commercial LLMs (e.g., ChatGPT). To improve the accessibility of LLMs for research and development efforts, various instruction-tuned open-source LLMs have also been introduced recently, e.g., Alpaca, Vicuna, to name a few. However, existing open-source LLMs have only been instruction-tuned for English and a few popular languages, thus hindering their impacts and accessibility to many other languages in the world. Among a few very recent work to explore instruction tuning for LLMs in multiple languages, SFT has been used as the only approach to instruction-tune LLMs for multiple languages. This has left a significant gap for fine-tuned LLMs based on RLHF in diverse languages and raised important questions on how RLHF can boost the performance of multilingual instruction tuning. To overcome this issue, we present Okapi, the first system with instruction-tuned LLMs based on RLHF for multiple languages. Okapi introduces instruction and response-ranked data in 26 diverse languages to facilitate the experiments and development of future multilingual LLM research. We also present benchmark datasets to enable the evaluation of generative LLMs in multiple languages. Our experiments demonstrate the advantages of RLHF for multilingual instruction over SFT for different base models and datasets. Our framework and resources are released at https://github.com/nlp-uoregon/Okapi.
1 Introduction
Okapi addresses the limited multilingual accessibility of open-source instruction-tuned LLMs and the lack of RLHF-based multilingual instruction tuning. It develops an open-source framework and resources across 26 languages, finding benefits from RLHF over SFT while identifying low-resource languages as challenging.
- Existing open-source LLMs mainly support instruction following in English and other popular languages, limiting access for many language communities.
- Prior multilingual instruction-tuning work used SFT בלבד, leaving RLHF unexplored for improving multilingual LLM performance.
- Okapi introduces the first open-source RLHF-based instruction-tuned LLM framework for multiple languages, covering 26 diverse languages.
- Experiments show RLHF generally outperforms SFT for multilingual instruction tuning across different base models and datasets.
- The study highlights low-resource languages as a challenging area requiring greater research focus.
- The framework creates multilingual instruction, response-ranking, benchmark, and fine-tuned-model resources to support training and evaluation.
2 Data Preparation
Okapi prepares multilingual instruction-tuning resources by generating English instructions, translating them into 26 languages, and producing ranked responses for RLHF. The framework emphasizes consistent cross-language content and includes languages spanning high-, medium-, and low-resource categories.
- Data preparation pipeline: Okapi prepares instruction, ranking, and evaluation data through English instruction generation, translation, ranking-data production, and evaluation-data creation.
- Language coverage: The framework covers 26 non-English languages, including 8 high-resource, 11 medium-resource, and 7 low-resource languages.Marathi, Gujarati, and Kannada are among the languages described as receiving limited NLP attention.
- Instruction translation: Okapi translates 158K English instructions into 26 target languages with ChatGPT, using prompts designed to preserve programming code and instruction context.
- Instruction translation: Unlike approaches that translate only instructions and inputs, Okapi directly translates each instruction, input text, and associated output.
- Ranking data production: For ranking data, the model generates multiple responses for each instruction-input pair, then ranks them to train a reward model using contrastive learning.
- Ranking data production: ChatGPT ranks multilingual responses through two turns: translating target-language examples into English, then ranking the translated responses for correctness, coherence, and naturalness.This process produces ranked responses for 42K sampled instructions.
3 Reinforcement Learning with Human Feedback
Okapi develops each target-language RLHF model through supervised fine-tuning, reward-model training, and reinforcement learning. PPO then optimizes the SFT model using reward signals while penalizing large deviations from the initial policy.
- Training pipeline: Okapi develops each target-language model in three stages: supervised fine-tuning, reward-model training, and reinforcement learning.
- Supervised fine-tuning: Supervised fine-tuning trains a multilingual pretrained base model on target-language instruction data using an autoregressive objective.
- Reward-model training: The target-language reward model assigns scalar appropriateness scores to prompt-response pairs using response-ranked datasets.
- Reinforcement learning: PPO further fine-tunes the SFT model to align it with human preferences by maximizing mean reward.
- Reinforcement learning: The RL objective combines reward maximization with a KL penalty that discourages large deviations from the initial SFT policy.
4 Experiments
Okapi evaluates SFT and RLHF instruction tuning with BLOOM 7B and LLaMA 7B across translated multilingual benchmarks. RLHF generally outperforms SFT, although gains vary by dataset and language-resource group.
- Experimental Setup: Okapi compares SFT and RLHF using BLOOM 7B and LLaMA 7B across translated ARC, HellaSwag, and MMLU datasets.The evaluation also reports base-model references and, for BLOOM, comparison with BLOOMZ.
- Results: RLHF is generally better than SFT across tasks, base models, and language-resource groups.The reported average improvement reaches 2.5% on HellaSwag with LLaMA.
- Results: 2.5% is the reported maximum average improvement over all languages on HellaSwag with LLaMA when comparing RLHF with SFT.
- Results: RLHF gains are more substantial on ARC and HellaSwag than on MMLU because the generated instructions better match general knowledge and basic inference than professional knowledge.The instructions therefore do not meaningfully activate or complement the skills assessed by MMLU.
- Results: Performance is highest for high-resource languages, followed by medium-resource and low-resource languages across base models.For BLOOM, RLHF improvement is least pronounced for low-resource languages.
- Results: BLOOM models trained on 158K generated instructions outperform BLOOMZ across almost all languages on ARC, HellaSwag, and MMLU; RLHF is 4.8% better on average for HellaSwag.
5 Related Work
Related work covers multilingual model tuning and multilingual evaluation, spanning monolingual and multilingual pre-trained models, downstream fine-tuning, and instruction tuning. It identifies limited multilingual RLHF research and a shortage of evaluation datasets beyond English.
- Multilingual Tuning: Prior multilingual NLP work includes monolingual models and multilingual models designed to transfer knowledge across languages.
- Multilingual Tuning: Fine-tuning pre-trained language models supports strong performance across multilingual tasks and zero-shot cross-lingual transfer.
- Instruction Tuning: Instruction tuning trains generative pre-trained models on instruction data to improve instruction following and response alignment with human expectations.
- Instruction Tuning: SFT is common in instruction tuning, while RLHF is less explored in open-source LLMs because reward models require ranked response data.
- Instruction Tuning: Contemporary multilingual instruction-tuning work uses SFT, while RLHF has not been studied for non-English instruction tuning.
- Multilingual Evaluation: Multilingual evaluation is hindered by scarce datasets, and widely used benchmark datasets support evaluation only in English.
6 Conclusion
Okapi is presented as the first open-source framework for RLHF-based multilingual instruction tuning, addressing data scarcity with instruction and response-ranked resources in 26 languages. Experiments report benefits of RLHF for multilingual fine-tuning and identify low-resource languages as challenging.
- Okapi is presented as the first open-source framework for RLHF-based instruction tuning of LLMs in multiple languages.
- The framework introduces instruction and response-ranked data in 26 diverse languages to support multilingual training frameworks.
- Experiments reveal benefits of RLHF for multilingual fine-tuning and highlight low-resource languages as challenging for future research.
Limitations
The work has three stated limitations: incomplete language coverage, reliance on BLOOM and LLaMA 7B base models, and automatically generated and translated development data.
- Okapi covers 26 languages, but many other languages, especially low-resource languages, are not included.
- The system uses only BLOOM and LLaMA base models with 7B parameters.
- The development process automatically generates instructions in English and translates them into multiple languages.