PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca

Yiming Cui, Ziqing Yang, Xin Yao

arXiv:2304.08177v3cs.CLcs.HCcs.LG

TL;DR

Open-source LLaMA models have limited Chinese support, while the cost of training and deployment constrains accessible research. The paper extends the vocabulary with Chinese tokens and applies continued pre-training, LoRA, and instruction fine-tuning. The resulting models improve Chinese understanding and generation, show competitive C-Eval performance, and are released with supporting resources.

  • Problem

    LLaMA has limited native Chinese support, while the costs of training and deployment hinder transparent and accessible research.

  • Method

    The paper extends LLaMA’s vocabulary with 20,000 Chinese tokens, continues pre-training with Chinese data, and uses LoRA and instruction fine-tuning to build Chinese LLaMA and Alpaca.

  • Results

    The proposed models significantly improve Chinese understanding and generation over original LLaMA and achieve competitive performance on C-Eval against models several times larger.

  • Takeaways & Limitations

    The released models and training resources support further open research on Chinese language modeling and adaptation to other languages.

  • Takeaways & Limitations

    The models may generate harmful or misaligned content and may remain suboptimal because training was constrained by computing power and data availability.

Abstract

from arXiv · show

Large Language Models (LLMs), such as ChatGPT and GPT-4, have dramatically transformed natural language processing research and shown promising strides towards Artificial General Intelligence (AGI). Nonetheless, the high costs associated with training and deploying LLMs present substantial obstacles to transparent, accessible academic research. While several large language models, such as LLaMA, have been open-sourced by the community, these predominantly focus on English corpora, limiting their usefulness for other languages. In this paper, we propose a method to augment LLaMA with capabilities for understanding and generating Chinese text and its ability to follow instructions. We achieve this by extending LLaMA's existing vocabulary with an additional 20,000 Chinese tokens, thereby improving its encoding efficiency and semantic understanding of Chinese. We further incorporate secondary pre-training using Chinese data and fine-tune the model with Chinese instruction datasets, significantly enhancing the model's ability to comprehend and execute instructions. Our experimental results indicate that the newly proposed model markedly enhances the original LLaMA's proficiency in understanding and generating Chinese content. Additionally, the results on the C-Eval dataset yield competitive performance among the models with several times the size of ours. We have made our pre-trained models, training scripts, and other resources available through GitHub, fostering open research for our community. Chinese LLaMA series: \url{https://github.com/ymcui/Chinese-LLaMA-Alpaca} and Chinese Llama-2 series: \url{https://github.com/ymcui/Chinese-LLaMA-Alpaca-2}

1 INTRODUCTION

The paper targets LLaMA and Alpaca’s limited native support for Chinese, proposing vocabulary expansion, efficient adaptation, and instruction-following models for Chinese tasks. It reports improved Chinese understanding and generation, competitive C-Eval performance, and publicly released resources.

  • Motivation: LLaMA and Alpaca provide open-source platforms for research but offer limited native support for Chinese language tasks.Their vocabularies contain only a few hundred Chinese tokens, hindering Chinese encoding and decoding efficiency.
  • Contributions: 20,000 additional Chinese tokens extend LLaMA’s vocabulary to improve Chinese encoding efficiency and understanding.The vocabulary extension is presented as a central contribution of the proposed Chinese LLaMA models.
  • Contributions: LoRA enables efficient training and deployment of the Chinese LLaMA and Alpaca models without excessive computational costs.The method is intended to make adaptation more accessible to researchers with limited resources.
  • Evaluation: The proposed models are evaluated on Chinese instruction-following and natural language understanding tasks, showing substantial improvements over original counterparts.The evaluation focuses on Chinese language tasks.
  • Resources: The authors publicly release models, training scripts, and other resources to foster further research and collaboration.The release also encourages adaptation of LLaMA and Alpaca models to other languages.

2 CHINESE LLAMA AND CHINESE ALPACA

The paper adapts LLaMA for Chinese by extending its tokenizer and continuing pre-training, then uses LoRA and supervised fine-tuning to build Chinese LLaMA and Chinese Alpaca models. The resulting tokenizer reduces sequence length, while instruction fine-tuning targets user-directed responses.

  • Background: LLaMA is a decoder-only transformer language model whose training data consists mainly of English and limited European-language material.Its Chinese generation ability is limited despite basic Chinese understanding.
  • Chinese Vocabulary Extension: The original tokenizer represents many Chinese characters as 3–4 byte tokens, increasing sequence length and reducing Chinese encoding and decoding efficiency.The original vocabulary covers fewer than a thousand Chinese characters.
  • Chinese Vocabulary Extension: The method trains a 20,000-token Chinese tokenizer and merges it with LLaMA’s vocabulary into a 49,953-token Chinese LLaMA tokenizer.The merged tokenizer is then used to adapt the model’s embeddings and language-model head.
  • Chinese Vocabulary Extension: Approximately half as many tokens are generated by the Chinese LLaMA tokenizer compared with the original tokenizer.With fixed context length, the model can accommodate about twice as much information and generation is twice as fast.
  • Parameter-Efficient Fine-Tuning: LoRA freezes pre-trained weights and adds trainable low-rank matrices, reducing trainable parameters and memory requirements.The paper applies LoRA during both pre-training and fine-tuning, primarily in attention and MLP layers.
  • Supervised Fine-Tuning and Chinese Alpaca: Chinese Alpaca is produced by fine-tuning Chinese LLaMA on instruction-following datasets so it generates responses autoregressively from prompted instructions.The loss is calculated only on the output portion, and the paper uses a prompt template without a separate input field.

3 EXPERIMENTAL SETUPS

The experimental setup combines LoRA-based pre-training and instruction fine-tuning, using Chinese corpora and instruction data across Chinese LLaMA and Alpaca models.

  • Pre-training: LoRA is applied throughout pre-training and fine-tuning to reduce computational costs while training Chinese LLaMA and Alpaca models.The setup applies LoRA to attention and MLP components, with embeddings and language-model heads also trainable during pre-training.
  • Pre-training: The basic Chinese LLaMA-7B model uses two-stage pre-training, first adapting new Chinese embeddings, then training embeddings, LM heads, and LoRA parameters.The transformer encoder parameters remain fixed during stage 1 to minimize disturbance to the original model.
  • Pre-training: 20GB of general Chinese text is used for basic-model pre-training, while the Plus version expands the corpus to 120GB with CommonCrawl and encyclopedia data.The expanded data is intended to improve understanding of fundamental concepts.
  • Hyperparameters: Table 2 lists pre-training hyperparameters, while Table 3 lists instruction fine-tuning hyperparameters for Chinese Alpaca.The Plus version uses a larger LoRA rank than the basic version, with learning rate and batch size also adjusted.
  • Instruction fine-tuning: Instruction fine-tuning uses approximately 2M to 3M examples for basic models and approximately 4M to 4.3M examples for Plus models.The Plus dataset emphasizes STEM and scientific disciplines, while Alpaca-33B additionally uses OASST1 data.
  • Instruction fine-tuning: Crawled instruction data is generated through a simplified self-instruct template specifying target domains and instruction types without requiring seed tasks.The data is obtained from ChatGPT through the gpt-3.5-turbo API, with templates and code provided on GitHub.

4 RESULTS ON INSTRUCTION-FOLLOWING TASKS

The evaluation uses GPT-4 ratings, manual checks, and a 200-sample set spanning ten instruction-following and language tasks. Results show complementary strengths: larger Alpaca-33B performs better on reasoning, coding, and ethics, while data-rich Plus models perform better on generation and dialogue.

  • 4.1 TASK DESIGN AND EVALUATION METHOD: GPT-4 rates 200 samples across ten tasks on a normalized 100-point scale, with manual checks used to adjust inaccurate ratings.The tasks include question answering, reasoning, literature, entertainment, translation, multi-turn dialogue, coding, and ethics.
  • 4.2 EXPERIMENTAL SETUPS FOR DECODING: Decoding uses a 2048-token context, 512-token maximum output, temperature 0.2, top-k 40, top-p 0.9, and repetition penalty 1.1.Multi-turn dialogue and generation tasks use temperature 0.5 instead of 0.2.
  • 4.2 EXPERIMENTAL SETUPS FOR DECODING: The decoding hyperparameters were not tuned separately for each task, because the authors sought to maintain a balanced evaluation view.The authors note that these values may not be optimal for every testing scenario.
  • 4.3 RESULTS: The main comparison covers Chinese Alpaca-Plus-7B, Chinese Alpaca-Plus-13B, and Alpaca-33B using GPT-4-rated results that are comparable only within this model combination.Alpaca-33B uses the original FP16 model, whereas the Plus models use 8-bit quantized versions.
  • 4.3.1 MULTI-TURN DIALOGUE: Plus models consistently improve multi-turn dialogue over the basic model, despite having several times fewer parameters.The authors suggest that ingesting more training data may matter more for dialogue quality than simply increasing parameter size.
  • 4.3.2 TEXT GENERATION: Alpaca-33B performs worse than Alpaca-Plus-7B and Alpaca-Plus-13B on text generation, which provide more detailed and appropriate letter-style responses.The authors associate the Plus models’ richer content with their substantially larger training-data exposure.
  • 4.3.3 NUMERICAL CALCULATION AND REASONING: Alpaca-33B significantly improves numerical reasoning over both Plus models, correctly solving weight comparison, commonsense counting, and sequence-pattern examples.The observations indicate that model size is vital for numerical reasoning tasks.
  • 4.3.4 CODING / 4.3.5 ETHICS: Alpaca-33B yields significant improvements in coding and ethics, while the Plus models remain stronger in text generation and multi-turn dialogue.The authors conjecture that larger models better handle high-level reasoning, whereas more training data supports richer and more diverse content.

5 RESULTS ON NATURAL LANGUAGE UNDERSTANDING TASKS

The models are evaluated on C-Eval using next-token probabilities mapped to answer labels, with results compared across model families, sizes, and prompting settings. Chinese LLaMA moderately improves over original LLaMA, while Alpaca variants show stronger gains and competitive leaderboard performance.

  • Evaluation setup: C-Eval covers four categories, 52 disciplines, and nearly 14K samples; evaluation uses 1,346 validation and 12,342 test examples.The task requires predicting one of four option labels.
  • Evaluation setup: The evaluation maps next-token probabilities to A–D labels through a verbalizer and selects the label with maximum probability.LLaMA receives examples directly, whereas Alpaca uses the instruction prompt template.
  • Comparisons to original LLaMA: Chinese LLaMA moderately improves over original LLaMA on C-Eval, but gains are not consistent across settings.LLaMA-Plus does not significantly outperform Chinese LLaMA and is inferior in the 13B setting.
  • Comparisons to original LLaMA: Alpaca models significantly outperform their LLaMA counterparts in both zero-shot and 5-shot settings.Alpaca-Plus also significantly improves over basic Alpaca, unlike the corresponding LLaMA comparison.
  • Prompting effects: LLaMA generally performs better with 5-shot prompting, whereas Alpaca generally performs better zero-shot.The authors associate this contrast with LLaMA’s need for task-structure information and Alpaca’s prior instruction training.
  • Scope: The reported C-Eval observations are based solely on that dataset, so their generalizability to other datasets requires further investigation.The authors plan more comprehensive testing of LLaMA and Alpaca behaviors.
  • Comparisons to other models: Chinese-Alpaca-33B and Chinese-Alpaca-Plus-13B achieve competitive performance among open-source leaderboard models several times larger.The authors report a +5.8 average-score improvement for Alpaca-13B, from 30.9 to 36.7, after using their own evaluation implementation.

6 EFFECT OF DIFFERENT QUANTIZATION METHODS

The study compares 2-bit through 8-bit quantization with FP16 using perplexity on Chinese text corpora. Higher-bit quantization preserves perplexity more closely while reducing deployment cost, creating a memory–speed–quality trade-off.

  • Experimental setup: Quantized Alpaca-Plus-7B, Alpaca-Plus-13B, and Alpaca-33B models are evaluated at 2, 3, 4, 5, 6, and 8 bits against FP16.Perplexity is measured on Chinese text corpora using llama.cpp.
  • Results: Figure 4 reports perplexities for different quantization methods and notes that the 33B model has higher perplexity because it was trained on less data.The comparison spans the tested quantization levels and the original FP16 model.
  • Results: 8-bit quantization has almost the same or lower perplexity than FP16 while using half the model size.The authors identify it as a good option for personal-computer deployment.
  • Results: 6-bit models achieve perplexities comparable to 8-bit models, offering a better balance between deployment cost and quality.Quantization level is explicitly tied to memory usage and inference speed.

7 CONCLUSION

The report extends LLaMA for Chinese understanding and generation by expanding its vocabulary, then uses instruction-data fine-tuning to produce Chinese Alpaca models. Evaluations report improved Chinese capabilities and competitive C-Eval performance relative to substantially larger models.

  • Conclusion: The approach adds 20K Chinese tokens to LLaMA’s vocabulary to increase Chinese encoding efficiency.The report identifies the original Chinese vocabulary as a limitation.
  • Conclusion: Supervised fine-tuning with instruction data builds Chinese Alpaca models with improved instruction-following capabilities.The fine-tuning is performed on top of Chinese LLaMA.
  • Conclusion: Evaluation of 200 annotated samples across ten task types used GPT-4, and the proposed models significantly outperformed original LLaMA on Chinese understanding and generation tasks.The report also tested the models on C-Eval.
  • Conclusion: On C-Eval, the proposed models achieved significant improvements and competitive performance relative to models several times larger.This conclusion summarizes the reported validation and leaderboard findings.

LIMITATIONS

The report identifies limitations involving harmful outputs, insufficient training, brittleness, incomplete evaluation coverage, and deployment scalability. These constraints leave room for improvement in robustness, capability, accessibility, and breadth of validation.

  • Safety: The models may still generate harmful or misaligned content despite rejecting unethical queries.The authors attribute this possibility to training-data bias or difficulty discerning appropriate outputs in some contexts.
  • Training: Computing-power and data constraints may make training insufficient for optimal performance, leaving room to improve Chinese understanding.This is presented as a limitation of the current training regime.
  • Robustness: The models may be brittle under adversarial inputs or rare language phenomena, producing inconsistent or nonsensical outputs.The limitation concerns robustness across unusual or challenging inputs.
  • Evaluation: The comprehensiveness and appropriateness of current LLM evaluation benchmarks require further study.The authors call for more diverse and comprehensive datasets and benchmarks.
  • Scalability and efficiency: Despite LoRA and quantization, model size and complexity can hinder deployment for users with limited computational resources.The authors identify accessibility and widespread adoption as affected boundaries.

A APPENDIX

This appendix presents baseline results for Chinese-LLaMA-2 and Chinese-Alpaca-2, using settings mostly identical to Chinese-LLaMA.

  • Baseline results are presented for Chinese-LLaMA-2 and Chinese-Alpaca-2.
  • The reported models are Chinese-LLaMA-2 and Chinese-Alpaca-2.
  • Most experimental settings are identical to those used for Chinese-LLaMA.

A.1 C-EVAL

The section presents C-Eval results in Table 10, covering both the valid and test sets.

  • Results on C-Eval are presented in Table 10.
  • Table 10 reports results for the C-Eval valid set.
  • Table 10 reports results for the C-Eval test set.

A.2 CMMLU

The section presents CMMLU results in Table 11, specifically for the CMMLU test sets.

  • Results on CMMLU are presented in Table 11.
  • Table 11 reports results on CMMLU test sets.
  • The CMMLU evaluation is organized as a test-set results table.

A.3 LONGBENCH

The section evaluates Chinese and code-task performance on LongBench, a benchmark designed to test long-context ability, using models supporting either 16K or 64K context.

  • Results on LongBench are presented in Table 12 for Chinese and code tasks.
  • LongBench is specifically designed to test the long context ability of large language models.
  • The evaluation uses the Chinese subsets of LongBench, including code tasks.
  • Models marked 16K were fine-tuned using Positional Interpolation, which supports 16K context.
  • Models marked 64K were fine-tuned using YaRN, which supports 64K context.
  • LongBench covers single-document QA, multidocument QA, summarization, few-shot learning, code completion, and synthetic tasks.
Loading 2304.08177v3…