Source-linked AI summary
Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities
Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Iida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, Naoaki Okazaki
TL;DR
The paper asks whether English-trained LLMs can be adapted to Japanese efficiently while clarifying how training scale, vocabulary expansion, and parallel data affect performance. It builds Swallow by expanding Llama 2’s vocabulary and continually pre-training on Japanese data, finding broad Japanese gains, monotonic improvement with more data, and stronger translation from parallel-corpus mixing. Vocabulary expansion improves efficiency without harming most task accuracy, except summarization.
Problem
English-trained LLMs perform worse in languages with different writing systems, while bilingual pre-training from scratch requires enormous computational resources and continual-pre-training effectiveness remains incompletely characterized.
Method
Swallow expands Llama 2’s vocabulary with Japanese characters and undergoes continual pre-training primarily on Japanese corpora, with English replay data included to help prevent forgetting.
Results
Continual pre-training substantially improves Japanese abilities, especially knowledge-intensive question answering, with performance increasing alongside training data and surpassing Japanese models trained from scratch.
Takeaways & Limitations
Vocabulary expansion improves computational efficiency with little performance impact except summarization, while adding parallel sentences improves Japanese-English translation without hurting other tasks.
Takeaways & Limitations
Swallow may produce harmful, offensive, or biased content because it is trained on datasets derived from publicly available online sources.
Abstract
from arXiv · showhide
Cross-lingual continual pre-training of large language models (LLMs) initially trained on English corpus allows us to leverage the vast amount of English language resources and reduce the pre-training cost. In this study, we constructed Swallow, an LLM with enhanced Japanese capability, by extending the vocabulary of Llama 2 to include Japanese characters and conducting continual pre-training on a large Japanese web corpus. Experimental results confirmed that the performance on Japanese tasks drastically improved through continual pre-training, and the performance monotonically increased with the amount of training data up to 100B tokens. Consequently, Swallow achieved superior performance compared to other LLMs that were trained from scratch in English and Japanese. An analysis of the effects of continual pre-training revealed that it was particularly effective for Japanese question answering tasks. Furthermore, to elucidate effective methodologies for cross-lingual continual pre-training from English to Japanese, we investigated the impact of vocabulary expansion and the effectiveness of incorporating parallel corpora. The results showed that the efficiency gained through vocabulary expansion had no negative impact on performance, except for the summarization task, and that the combined use of parallel corpora enhanced translation ability.
1 Introduction
The paper investigates continual pre-training from English LLMs as a resource-efficient route to stronger Japanese capabilities, addressing open questions about data scale and adaptation methods. Swallow improves Japanese performance broadly, while vocabulary expansion and parallel corpora offer distinct efficiency and translation benefits.
- Main findings: Swallow improved performance in almost all Japanese tasks across all model sizes, with Japanese task gains reaching approximately 70%.Figure 1 reports relative performance changes compared with Llama 2.
- Motivation and approach: Continual pre-training from English LLMs addresses the resource demands of bilingual pre-training while investigating how English knowledge transfers to Japanese.The study examines model sizes, Japanese training-data amounts, and evaluations across Japanese and English tasks.
- Efficiency and transfer methods: Vocabulary expansion improved Japanese text-generation efficiency by up to 78% through a 56.2% token reduction without compromising downstream accuracy except in summarization.The study evaluates its impact on both Japanese and English performance.
- Efficiency and transfer methods: Mixing Japanese-English parallel corpora with plain text improved machine-translation accuracy without affecting other task performance.The result identifies parallel-corpus mixing as a targeted method for enhancing translation ability.
- Main findings: Swallow achieved the highest Japanese performance among models developed in Japan as of December 2023, using fewer computational resources than Japanese models trained from scratch.The comparison supports cross-lingual continual pre-training as an efficient alternative to training from scratch.
- Main findings: Continual pre-training was especially effective for Japanese question answering, and Japanese performance improved monotonically as the amount of Japanese training data increased.These findings address the study’s questions about task sensitivity and training-data scale.
2 Related work
Prior work uses continual pre-training and vocabulary expansion to adapt LLMs across languages, but comprehensive evidence on their effects remains limited. Parallel corpora have shown promise for cross-lingual transfer and translation, while their role in continual pre-training is less established.
- Continual pre-training adapts pre-trained LLMs using additional target-task or target-language text, but its effects across model and data scales lacked comprehensive evaluation.
- Vocabulary expansion adds target-language tokens to reduce sequence length and generation cost, but its performance impact in continual pre-training remains insufficiently understood.
- Existing Japanese vocabulary-expansion studies generally omit English evaluation or base-model comparisons, limiting conclusions about multilingual effects.
- Parallel-corpus pre-training has been reported to promote cross-lingual transfer, while parallel sentences in instruction tuning can improve translation efficiency.
3 Continual pre-training
Swallow adapts Llama 2 through Japanese-focused continual pre-training, vocabulary expansion, and mixed-language corpora. The evaluation covers Japanese and English tasks across multiple task types.
- Swallow continually pre-trains Llama 2 after vocabulary expansion using corpora primarily consisting of Japanese, then evaluates Japanese and English capabilities.
- The models retain Llama 2’s Transformer decoder architecture and core architectural dimensions during continual pre-training.
- AdamW, cosine learning-rate decay, warmup, weight decay, gradient clipping, and Flash Attention 2 were used for training efficiency.
- English replay data was added alongside Japanese data because full-parameter continual pre-training can cause forgetting of previously learned knowledge.
- The continual-pre-training corpus sampled approximately 100B tokens from Japanese and English sources, with Japanese text forming the majority.
- Vocabulary expansion added Japanese vocabulary to the existing Llama 2 vocabulary to improve computational efficiency for Japanese text.
- The expanded vocabulary added 11,176 subwords, producing a vocabulary size of 43,176 after BPE construction and initialization from Llama 2 subwords.
- Evaluation used Japanese and English datasets spanning question answering, reading comprehension, summarization, arithmetic reasoning, commonsense reasoning, and machine translation.
4 Results
Continual pre-training substantially improves Japanese performance, especially on question answering, while causing smaller English declines and task-dependent translation changes. Performance rises monotonically with additional Japanese training data, and Swallow outperforms Japanese models trained from scratch.
- Swallow’s average Japanese-task score surpasses Llama 2 by approximately 7 points, while English scores are 2–5 points lower.
- Japanese question answering improves by up to 75%, while Japanese arithmetic reasoning improves by 36–63%.
- English question answering and arithmetic reasoning decline by 6–23%, whereas automatic summarization changes by less than 5%.
- English-to-Japanese translation improves by 15–41%, while Japanese-to-English translation declines by 4–13%.
- Continual pre-training improves Japanese question answering by shifting many NIILC answers from incorrect to correct, with the reverse shift uncommon.
- Swallow’s average score is 8.4 to 17.4 points higher than Japanese models trained from scratch.
- Japanese benchmark scores increase monotonically as continual-pre-training data grows, with the largest gain occurring during the initial 20B tokens.
5 Analysis
Vocabulary expansion has mostly minor performance effects but harms summarization, while parallel corpora substantially improve translation. Neither intervention consistently improves non-translation tasks across model sizes.
- Vocabulary expansion has a minor overall effect on Japanese capabilities, with question-answering changes of approximately ±10% and no consistent direction across model sizes.
- Automatic summarization declines by about 5–15% with vocabulary expansion.
- The parallel-corpus experiments use Swallow¬VE as the baseline and incorporate approximately 22 million Japanese-English parallel sentences from JParaCrawl 3.0.
- Effective parallel-corpus configurations include mixed next-token prediction and two-stage translation-instruction training.
- Parallel-corpus training improves translation by 9–24% for English-to-Japanese and 14–51% for Japanese-to-English.
- For tasks other than translation, parallel-corpus effects remain within ±15% without consistent improvement or decline across model sizes.
6 Conclusions
The study finds that continual pre-training substantially improves Japanese abilities, especially knowledge-intensive question answering, while outperforming Japanese LLMs trained from scratch. Performance also rises with more training data; vocabulary expansion improves efficiency with little performance effect except on summarization, and parallel data improves translation without harming other tasks.
- Continual pre-training significantly boosts Japanese abilities, particularly on knowledge-intensive question answering tasks.
- Japanese performance improves in line with increases in training data.
- Vocabulary expansion improves computational efficiency with little performance impact except for summarization.
- Mixing parallel sentences with plain text improves Japanese-English translation performance without hurting other tasks.
- Continual pre-trained models outperform Japanese LLMs trained from scratch.
7 Ethical considerations
Swallow shares common LLM limitations, including stale information, non-factual generation, hallucinations, and potentially harmful, offensive, or biased content from web-derived training data.
- Swallow cannot update information after pretraining and may generate non-factual advice or hallucinations.
- Training on publicly available online sources may produce harmful, offensive, or biased content.
8 Reproducibility statement
The study reports that its models and benchmark datasets are publicly available, making the reported experiments straightforward to reproduce. The Swallow Corpus was built from refined Japanese Common Crawl text.
- All continually pre-trained Swallow models and benchmark datasets are publicly available for reproducing results in Tables 4, 5, and 9.
- The Swallow Corpus contains about 312.1 billion characters extracted and refined from 21 Common Crawl snapshots collected between 2020 and 2023.
B Ratio of training data
The study uses a 9:1 Japanese-to-English training ratio and distributed training infrastructure for Swallow models. It expands the vocabulary with Japanese subwords and formats parallel sentences for next-token or translation-instruction training.
- Ratio of training data: A 9:1 Japanese-to-English ratio was selected because it slightly outperformed a 5:5 ratio on average Japanese-task performance after approximately 20B tokens.
- Distributed training: Swallow training uses distributed parallelism because 7B-model training on a single GPU is constrained by memory and time.
- Distributed training: Training used mixed-precision bfloat16 on multiple NVIDIA A100 nodes interconnected with InfiniBand HDR.
- Distributed training: 3D parallelism combines data, tensor, and pipeline parallelism to improve computational efficiency and memory utilization.
- Distributed training: The Distributed Optimizer reduces memory usage by distributing optimizer states across data-parallel processes without increasing communication cost.
- Distributed training: Topology-aware mapping places tensor-parallel workers within nodes because NVLink communication is faster than inter-node communication.
- Distributed training: 1F1B pipeline parallelism improves memory efficiency by limiting activation storage to at most the number of pipeline stages in micro-batches.
- Distributed training: Sequence parallelism reduces redundant memory usage for Layer-Norms and Dropouts without communication overhead when combined with tensor parallelism.
F Issues in evaluating natural language inference tasks
Natural language inference scores can be misleading because severe class imbalance makes results fluctuate with the alignment of predicted and ground-truth majority classes. The evaluation section therefore frames these benchmarks as requiring distribution-aware interpretation.
- Class imbalance: In three datasets, the most frequent predicted class accounts for over 95% of predictions, causing scores to depend strongly on majority-class alignment.This imbalance affects both ground-truth and Swallow-7b predictions.
- Score instability: Swallow-7b’s learning curves fluctuate by about 40 points in two datasets as the majority predicted class changes.The fluctuations reflect transitions in prediction distributions rather than stable changes in inference ability.
- Cross-model pattern: Prediction imbalance is not unique to Swallow, appearing in other 7b and 13b models as well.This cross-model pattern strengthens the concern that raw scores alone are unreliable for these datasets.
- Evaluation scope: The Japanese evaluation suite spans multiple-choice and open-ended question answering, reading comprehension, summarization, and arithmetic reasoning tasks.The listed benchmarks include JCQA, JEMHQA, JSQuAD, NIILC, XL-Sum, and MGSM.
H Comparison of models trained from scratch and other continual pre-training models
The comparison evaluates Japanese performance across pre-trained, continually pre-trained, and other model families using the Japanese evaluation results. Training metadata distinguishes vocabulary expansion, continual pre-training, and training from scratch, while noting that listed data sources are only the major sources.
- Evaluation comparison: Table 9 compares pre-trained and continually pre-trained models, including models developed in Japan, on Japanese evaluation datasets.The supplied passage identifies Table 9 as the source of these evaluation results.
- Models trained from scratch: Some models trained from scratch outside Japan outperform Llama 2 on Japanese performance.The passage specifically names Mistral v0.1, Qwen-7B, and Qwen-14B.
- Continual pre-training models: Continually pre-trained versions of several strong base models achieve higher average scores than Swallow.The named models are japanese-stablelm-base-gamma-7b, nekomata-7b, and nekomata-14b.
- Training categories: The training-detail abbreviations distinguish VE for vocabulary expansion, CT for continual pre-training, and PT for pre-training from scratch.These abbreviations organize the model descriptions in the comparison tables.
- Training data: Models labeled Ja use Japanese data, En models primarily use English data, and Ja & En models use both Japanese and English data.The training-detail labels describe major data sources rather than every corpus used.