PaperlayerView exact revision on arXiv

Source-linked AI summary

Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languages

Shih-Cheng Huang, Pin-Zu Li, Yu-Chi Hsu, Kuang-Ming Chen, Yu Tung Lin, Shih-Kai Hsiao, Richard Tzong-Han Tsai, Hung-yi Lee

arXiv:2310.04799v3cs.CL

TL;DR

Most open-source LLMs have English-focused capabilities, while preference alignment in a target language can require costly training and target-language feedback. The paper subtracts base-model weights from chat-model weights and adds the resulting vector to a continually pretrained model; evaluations report gains across conversational tasks and settings.

  • Problem

    Most open-source LLMs have language capabilities constrained to English, and aligning models with human preferences in other languages can require target-language preference data and complex training.

    Source: §1 · pp. 1–2

  • Method

    The chat vector is the difference between a chat model’s and its base model’s weights, added to a continually pretrained target-language model.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Across instruction following, toxicity, and multi-turn dialogue evaluations, chat-vector integration after continual pre-training outperformed direct pretraining on LLaMA-2-chat.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that chat-vector arithmetic offers a more efficient alternative to reimplementing RLHF in the target language.

    Source: §1 · p. 2, §6 · p. 9

    The authors note that some issues require further investigation, without specifying them in the supplied passage.

    Source: §7 · p. 10

Abstract

from arXiv · show

Recently, the development of open-source large language models (LLMs) has advanced rapidly. Nevertheless, due to data constraints, the capabilities of most open-source LLMs are primarily focused on English. To address this issue, we introduce the concept of $\textit{chat vector}$ to equip pre-trained language models with instruction following and human value alignment via simple model arithmetic. The chat vector is derived by subtracting the weights of a pre-trained base model (e.g. LLaMA2) from those of its corresponding chat model (e.g. LLaMA2-chat). By simply adding the chat vector to a continual pre-trained model's weights, we can endow the model with chat capabilities in new languages without the need for further training. Our empirical studies demonstrate the superior efficacy of the chat vector from three different aspects: instruction following, toxicity mitigation, and multi-turn dialogue. Moreover, to showcase the adaptability of our approach, we extend our experiments to encompass various languages, base models, and chat vectors. The results underscore the chat vector's simplicity, effectiveness, and wide applicability, making it a compelling solution for efficiently enabling conversational capabilities in pre-trained language models. Our code is available at https://github.com/aqweteddy/ChatVector.

1 Introduction

The paper addresses the English-centered language capabilities of open-source LLMs and the difficulty of aligning models with human preferences in a target language. It proposes adding a chat vector to a continually pretrained model and evaluates the approach across conversational tasks and settings.

1 Introduction

  • Target-language RLHF requires human feedback and preference data, while its training process also presents technical, memory, and computational challenges.Even DPO requires preference data collected in the target language.Source: §1 · p. 2
  • The method subtracts a base model’s weights from its chat model’s weights, then adds that chat vector to a continually pretrained target-language model.This replaces the conventional continual-pretraining, supervised-fine-tuning, and RLHF sequence with continual pre-training plus vector addition.Source: Figure 1 · p. 1, §1 · p. 2
  • Across instruction following, toxicity, and multi-turn dialogue evaluations, adding the chat vector after continual pre-training outperformed direct pretraining on LLaMA-2-chat.The evaluation focused primarily on Traditional Chinese and also examined multiple target languages, models, and chat vectors.Source: §1 · p. 2
  • The authors report that the resulting model responds precisely in the target language, including when answering questions or declining inappropriate requests.The paper evaluates instruction following, toxicity, and multi-turn dialogue, and extends tests across languages, base models, and chat vectors.Source: §1 · p. 2

2 Related Work

Related work covers human-preference training methods and parameter-space approaches for changing model behavior. The paper situates its chat vector within both lines of work.

2.1 Human Preference Training

  • RLHF combines supervised fine-tuning, reward-model training, and PPO, and has been used to improve instruction following and mitigate toxic or harmful generation [1, 2].Later approaches include DPO, IPO, and KTO, which optimize preferences without learning a reward function or using the same preference inputs [3, 4, 5].Source: §2.1 · p. 3

2.2 Task Vector

  • Task vectors are differences between fine-tuned and pretrained weights; addition or negation can shape model behavior without further fine-tuning [6].Other work uses negative task vectors for hallucination mitigation, composes parameter-efficient modules, or interpolates models trained on different rewards [7, 8, 9].Source: §2.2 · p. 3
  • Because the principles underlying task vectors remain limited, subsequent work has investigated the effectiveness of task arithmetic [10, 11].Source: §2.2 · p. 3

3 Methodology

The method first continually pre-trains a base model on target-language corpora, then adds a chat vector derived from a chat model’s weights. This combines target-language modeling with instruction-following and alignment behavior.

3.1 Continual Pre-training (CP)

  • Continual pre-training uses target-language corpora with a causal language modeling objective, predicting each token from the preceding context.The loss is defined over the continual pre-training data and input sequence.Source: §3.1 · pp. 3–4

3.2 Chat Vector

  • The chat vector is the weight difference between a chat-enhanced model and its corresponding pre-trained base model.The method denotes these model weights as θchat and θPLM, respectively.Source: §3.2 · p. 4
  • Adding the chat vector element-wise to the continually pre-trained model yields a model intended to follow target-language instructions and align with criteria such as helpfulness and harmlessness.The resulting model weights are denoted θchat_new.Source: §3.2 · p. 4

4 Experimental Setup

The experiments describe Traditional Chinese training data, evaluation datasets and scoring procedures, and baseline configurations that vary model, language, and chat-vector choices.

4.1 Training Dataset

  • Training uses a 3.1B-token Traditional Chinese corpus and about 80,000 prompt-response pairs, with the fine-tuning data limited to single-turn examples.The corpus draws on publicly available sources across news, education, Wikipedia, academic abstracts, government reports, dictionaries, and scientific articles; the fine-tuning data also includes translation and summarization examples.Source: §4.1 · p. 4

4.2 Evaluation Dataset

  • Evaluation covers translated Chinese and Korean Vicuna questions, Chinese toxicity prompts, Chinese safety scenarios, and Traditional Chinese multi-task knowledge questions.The toxicity set includes challenging and non-challenging prompts; the safety evaluation uses seven typical scenarios.Source: §4.2 · pp. 4–5

4.3 Evaluation Metrics

  • GPT-4 scores model responses from 0 to 10 across several criteria, while toxicity and safety are assessed with Perspective API scores and a GPT-3.5 function call.The GPT-4 criteria include usefulness, relevance, correctness, detail, and language use; model scores are averaged.Source: §4.3 · p. 5

4.4 Baselines

  • Baselines compare continual pre-training and fine-tuning with chat-vector additions, and extend experiments to other chat vectors, a Mistral-based model, and Korean.The listed chat-vector variants include LLaMA2-chat, xwin-13b, and tulu2-dpo-13b; Breeze is the continually pre-trained Mistral-based model.Source: §4.4 · p. 5

5 Experimental Result

Across instruction following, safety, forgetting, multi-turn dialogue, and model/language variants, adding a chat vector after continual pre-training generally restores or improves chat capabilities. The vector’s magnitude and target-language output remain important qualifications.

Chat Vector Enables Pre-trained Models to Follow Instructions

The Magnitude of The Chat Vector Could Severely Affect The Performance

  • The chat-vector magnitude affected performance: half magnitude may help a fine-tuned Chinese-LLaMA 13B model but did not help its continually pretrained counterpart, and choosing an optimal coefficient remains open.The section also concludes that the vector augments instruction following independently of the continual-pretraining model’s native language.Source: The Magnitude of The Chat Vector Could… · p. 7

5.3 Safety Prompts

  • Chat-vector models performed significantly better in four safety fields and comparably in others, while continual-pretraining models showed higher toxicity than models with chat vectors.The reported safety fields were Insult, Unfairness and Discrimination, Crimes and Illegal Activities, and Privacy and Property.Source: §5.3 · p. 7, §5.2 · p. 7, Table 3 · p. 7

5.4 Catastrophic Forgetting

  • Adding a chat vector did not significantly reduce TMMLU+ scores, and it preserved language abilities better across English and Traditional Chinese than fine-tuning on Traditional Chinese data.The authors interpret these findings as evidence that the approach mitigates catastrophic forgetting while largely preserving acquired knowledge.Source: Knowledge Retention · p. 8, Language Ability · p. 8, Table 4 · p. 8

5.5 Case Study of Multi-turn Conversations

  • In a multi-turn dialogue case study, the chat-vector model remembered the user’s lightning-bolt instruction, while continual pre-training and fine-tuning without the vector did not.The fine-tuning data did not include multi-turn conversations.Source: §5.5 · p. 8, Figure 3 · p. 8

5.6 Versatility of Chat Vector

  • Results extended to other chat vectors and base models: xwin and tulu2-dpo vectors worked with Traditional Chinese LLaMA, and a Mistral-instruct-0.2 vector outscored Breeze-Instruct.The Mistral result used Breeze-7B, a Mistral-based model continually pretrained in Traditional Chinese; its embedding and LM head layers were excluded and the vector coefficient was 0.5.Source: §5.6 · p. 9, Table 6 · p. 9
  • Applying the LLaMA2 chat vector to a Korean continual-pretraining model enabled instruction following, supporting use across languages.The experiments focused primarily on Traditional Chinese, with Korean as an additional example.Source: §5.6 · p. 9

6 Conclusion

The paper presents chat vectors as a way to add chat capabilities to LLMs in new languages through continual pre-training and simple model arithmetic. It reports lower alignment costs than approaches involving continual pre-training, supervised fine-tuning, and human-preference training.

6 Conclusion

  • The approach combines continual pre-training with a chat vector to add instruction following, human-value alignment, safe responses, and multi-turn dialogue in a new language.It relies on continual pre-training and arithmetic operations rather than continual pre-training, supervised fine-tuning, and human-preference training.Source: §6 · p. 9
  • The authors describe the approach as more efficient and scalable because it significantly reduces the cost of aligning models with human preferences.The conclusion frames this as a way to imbue LLMs with chat capabilities across multiple languages.Source: §6 · p. 9
  • The conclusion presents the method as applicable to multiple languages, using English-based pre-trained models as its starting point.The conclusion specifically describes continual pre-training of an English-based model before integrating the chat vector.Source: §6 · p. 9

7 Limitation

The paper identifies target-language output, evaluation coverage, and comparison with RLHF as areas needing further investigation. Its results are mainly based on automatic evaluations.

English Response

  • Adding the chat vector sometimes led to English rather than target-language responses; reducing its magnitude mitigated this for one setting but weakened instruction following and toxicity mitigation in another.For Chinese-LLaMA 13B, continual pre-training plus the full chat vector yielded Chinese responses in 20.9% of Safety Prompts outputs.Source: English Response · p. 10

Lack of Human Evaluation

  • The results rely mainly on automatic evaluations, which may miss nuances that human evaluators could identify.The authors plan to add human evaluation in future work.Source: Lack of Human Evaluation · p. 10

Direct Comparison with RLHF

  • Because the authors could not obtain LLaMA2-chat’s RLHF training data to build a controlled baseline, whether chat vectors can completely replace RLHF remains unclear.The unavailable data prevented precise control of the training data for a direct RLHF comparison.Source: Direct Comparison with RLHF · p. 10

A.1 CP + chat vector →SFT

Adding the chat vector before SFT modestly improved performance on the Traditional Chinese Vicuna benchmark compared with direct SFT.

A.1 CP + chat vector →SFT

  • The chat-vector model incorporated before SFT modestly outperformed the direct SFT model on the Traditional Chinese Vicuna benchmark.The authors describe this as a modest improvement and call for further exploration.Source: §A.1 · p. 12
  • The reported comparison is between incorporating the chat vector before SFT and using direct SFT.The passage does not provide a numerical score or effect size.Source: §A.1 · p. 12
  • The authors identify further investigation of this training paradigm as future work.Source: §A.1 · p. 12

A.2 Instruction Following Ability Evaluation Prompt

The evaluation prompt is adapted to the target language and asks GPT-4 to compare an assistant answer with a standard answer. The Traditional Chinese Vicuna results are reported as GPT-4 evaluation scores.

A.2 Instruction Following Ability Evaluation Prompt

  • The GPT-4 evaluation prompt is translated and adjusted for the target language.Source: §A.2 · pp. 12–13
  • Table 8 reports GPT-4 evaluation scores on the Traditional Chinese Vicuna benchmark.The supplied caption does not include the scores or identify a best-performing model.Source: Table 8 · p. 13
  • GPT-4 rates the assistant answer against a standard answer for helpfulness, relevance, accuracy, detail, and target-language use.The prompt asks for a score followed by an explanation and instructs the evaluator to avoid order bias.Source: §A.2 · p. 13

A.3 Safety Prompt GPT-3.5-Turbo Evaluation Prompt

The appendix reports English toxicity results after adaptation, case studies of safety responses, and an ongoing role-play attack evaluation. It also gives details of the GPT-3.5-Turbo safety prompt and training setup.

A.3 Safety Prompt GPT-3.5-Turbo Evaluation Prompt

  • The GPT-3.5-Turbo evaluation prompt asks whether a response is safe under its scenario and requests a reason.Its instructions say responses need not agree with offensive or unsafe prompts; the prompt uses function calling for structured evaluation.Source: §A.3 · p. 13
  • The safety prompt represents its judgment with a Boolean safety field and a textual reason.The reason is specified as explaining why the response is safe or unsafe.Source: §A.3 · p. 13

A.4 Real Toxicity Prompt in English

  • Chat-vector models generated less toxic English words even after fine-tuning or continual pretraining on another language.Table 9 reports English Real Toxicity Prompt scores from Perspective API.Source: §A.4 · pp. 13–14, Table 9 · p. 15

A.5 Case Study of Toxicity Mitigation and Safety

  • In three safety case studies, chat-vector configurations refused unsafe requests, while continual-pretraining and fine-tuning without the vector produced illicit advice.The examples concern obtaining bank passwords, saying dirty words, and obtaining drugs.Source: §A.5 · p. 14, Table 10 · p. 15, Table 11 · p. 16

A.6 Role Play Instruction Attack

  • The role-play attack study randomly selected 1,000 Safety Prompts examples, while evaluation across attack fields remained ongoing.Table 13 is captioned as reporting unsafe rates under role-play instruction attacks, but no rates are supplied here.Source: §A.6 · p. 14, Table 13 · p. 17

A.7 Training Details

  • Continual pretraining used 23,063 steps with a cosine learning-rate schedule, and supervised fine-tuning ran for three epochs.Training used AdamW and 32 Tesla V100 32GB GPUs.Source: §A.7 · p. 14

Related papers

Loading 2310.04799v3…