PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving the Robustness of Large Language Models via Consistency Alignment

Yukun Zhao, Lingyong Yan, Weiwei Sun, Guoliang Xing, Shuaiqiang Wang, Chong Meng, Zhicong Cheng, Zhaochun Ren, Dawei Yin

arXiv:2403.14221v2cs.CL

TL;DR

LLMs can respond inconsistently to semantically equivalent instructions, and systematic analysis and solutions remain limited. This paper proposes instruction-augmented fine-tuning followed by consistency alignment using self-rewards. Experiments report improved consistency and ROUGE scores across compared Vicuna and Llama 2 models, with Vicuna-13B + SFT (IA) + CAT surpassing GPT-4 in the reported setting.

  • Problem

    Current LLMs may respond inconsistently to semantically equivalent instructions, while quantitative analysis and systematic solutions remain lacking.

    Source: §1 · p. 1

  • Method

    The framework pairs paraphrased instructions with original responses during SFT, then uses self-rewards to align generated responses for consistency without external human preference resources.

    Source: §1 · p. 2, §4.1 · p. 4, §4.2 · p. 4

  • Results

    Consistency and ROUGE scores improve after SFT and further with CAT across compared Vicuna and Llama 2 models; Vicuna-13B + SFT (IA) + CAT surpasses GPT-4 in the reported setting.

    Source: §5.2 · p. 6

  • Takeaways & Limitations

    The framework is presented as a plug-in for continuously improving existing LLMs without additional human guidance or external reward models.

    Source: §6 · p. 9

    Self-rewards may not improve a model when its alignment performance is poor, and LLM-generated paraphrases may be less diverse than end-user instructions.

    Source: §7 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) have shown tremendous success in following user instructions and generating helpful responses. Nevertheless, their robustness is still far from optimal, as they may generate significantly inconsistent responses due to minor changes in the verbalized instructions. Recent literature has explored this inconsistency issue, highlighting the importance of continued improvement in the robustness of response generation. However, systematic analysis and solutions are still lacking. In this paper, we quantitatively define the inconsistency problem and propose a two-stage training framework consisting of instruction-augmented supervised fine-tuning and consistency alignment training. The first stage helps a model generalize on following instructions via similar instruction augmentations. In the second stage, we improve the diversity and help the model understand which responses are more aligned with human expectations by differentiating subtle differences in similar responses. The training process is accomplished by self-rewards inferred from the trained model at the first stage without referring to external human preference resources. We conduct extensive experiments on recent publicly available LLMs on instruction-following tasks and demonstrate the effectiveness of our training framework.

1. Introduction

Current LLMs can answer semantically equivalent instructions inconsistently, while quantitative analysis and systematic solutions remain lacking. The paper proposes a two-stage consistency-alignment framework and reports robustness and instruction-following generalization improvements on public LLMs.

1. Introduction

  • Inconsistent responses to semantically equivalent instructions remain a robustness problem, and prior work lacks quantitative analysis and a systematic solution.The paper notes that such inconsistencies hinder practical applications.Source: §1 · p. 1
  • The proposed framework combines instruction-augmented supervised fine-tuning with response consistency alignment to improve LLM robustness.The first stage augments training with paraphrased instructions paired with original responses; the second uses consistency scores to form response preference pairs.Source: §1 · p. 2
  • Experiments on Vicuna and Llama 2 models show robustness improvements and better instruction-following generalization with consistency self-alignment.The experiments cover Vicuna-7B, Vicuna-13B, Llama2-7B, and Llama2-13B.Source: §1 · p. 2
  • The framework uses self-rewards to improve LLM performance without external human preference resources or external reward models.Source: §1 · p. 2

2. Related Work

Related work covers instruction tuning, prompting, and instruction-data construction, alongside research on robustness to instruction changes. The paper identifies a remaining lack of quantitative analysis and corresponding solutions for generation robustness.

2. Related Work

3. Robustness on Instruction Following

The section defines robustness as consistency across equivalent instructions and evaluates recent LLMs using consistency metrics. GPT-4, ChatGPT, and Vicuna-13B are more robust on those metrics, while smaller models leave room for improvement.

3. Robustness on Instruction Following

  • Robustness is defined as expected consistency between responses to equivalent paraphrases, with higher consistency indicating greater robustness.The response-similarity function is assessed by an LLM, which classifies responses as similar or contradicted.Source: §3 · p. 3
  • CR measures consistency between answer pairs under Q, while MCR reports the rate of maximum-consistent answers among generated answers.The section names Consistency Rate (CR) and Maximum Consistency Rate (MCR) as its metrics.Source: §3 · p. 3
  • GPT-4, ChatGPT, and Vicuna-13B emerge as more robust on CR and MCR, with smaller models leaving room to improve.The comparison concerns responses to diverse linguistic representations.Source: Figure 2 · p. 3, §3 · p. 4
  • The preliminary evaluation sampled 490 Super Natural Instructions questions, created 10 paraphrases per question, and used GPT-4 to verify response consistency.The resulting test set contained 4,900 unique questions spanning diverse topics.Source: §3 · p. 3

4. Training Large Language Models via Consistency Alignment

The framework first broadens instruction-following through paraphrased instructions, then uses model-generated consistency rewards to distinguish responses that better align with human expectations.

4. Training Large Language Models via Consistency Alignment

  • The framework combines instruction-augmented supervised fine-tuning with consistency alignment training to improve instruction-following robustness.The first stage targets generalization across instructions; the second uses automatic feedback to distinguish similar responses rather than merely imitate them.Source: §4 · p. 4

4.1. Instruction Augmented Supervised Fine-Tuning

  • Instruction augmentation paraphrases each original task instruction while keeping its input-output instances unchanged.The authors use language models to generate up to n similar instructions per task because scaled human-written similar-instruction datasets are unavailable.Source: §4.1 · p. 4
  • Paraphrasing precision was 93% for Vicuna-7B, 95% for ChatGPT, and 95% for GPT-4.These are the reported precision values for the three models, respectively.Source: §4.1 · p. 4
  • The SFT stage randomly samples m instances per task, combines each instruction and input as a question, and trains against the target output.Its objective is a standard supervised fine-tuning loss.Source: §4.1 · p. 4

4.2. Response Consistency Alignment Training

  • For each input, the trained model generates responses under multiple task instructions and scores each response for answer type and correctness.The answer-type check distinguishes whether a response matches the expected type; correctness uses a common language-model definition.Source: §4.2 · p. 4
  • Consistency alignment trains on response pairs ranked by self-rewards, favoring more aligned responses and reducing the probability of lower-reward responses.The method uses offline optimization of <good, bad> pairs.Source: §4.2 · pp. 4–5

5. Experiments

Across Vicuna and Llama 2, instruction augmentation and consistency alignment improve consistency and ROUGE scores; CAT + SFT(IA) also outperforms GPT-4 in the reported setting. Human evaluation likewise favors CAT + SFT(IA) over the compared trained models.

5.1. Experimental Setup

  • The experiments use Super-Natural Instructions, four Vicuna and Llama 2 models, and separate paraphrased-instruction and standard test sets.Training uses up to 100 instances per task; the method reuses SFT-stage samples for CAT rather than adding data.Source: §5.1 · pp. 5–6

5.2. Main Results

  • CAT adds consistency and ROUGE gains across the compared Vicuna and Llama 2 models; Vicuna-13B + CAT + SFT(IA) surpasses GPT-4 in this setting.The authors report consistent method comparisons on test set II, where Vicuna-13B achieves the best performance.Source: §5.2 · pp. 6–7, Table 4 · p. 7, Table 5 · p. 7

5.3. Detailed Analysis

  • Combining correctness and expected-answer-type rewards improves Vicuna-13B ROUGE over correctness-only rewards; rewards from vanilla Vicuna-13B degrade CAT performance below SFT.The reported comparisons are on test set I; the table compares reward types and reward-model sources.Source: Table 6 · p. 7, §5.3 · p. 7
  • ROUGE scores are higher with λ set to 0.5 or 1 than 0, but increasing λ further reduces performance.The authors attribute the decline to learning less from negative generations and losing some benefits of consistency alignment.Source: §5.3 · pp. 7–8, Figure 4 · p. 8
  • Increasing augmented instructions to 10 improves consistency and ROUGE, while further increases degrade performance as each task receives fewer training instances.The comparison holds training-set size and other hyperparameters fixed.Source: Figure 5 · p. 8, §5.3 · p. 8

5.4. Human Evaluation

  • Human evaluation favors CAT + SFT(IA) over vanilla and SFT(IA), with more wins than losses and more responses judged as expected answer types and correct.For CAT + SFT(IA), 86% of responses scored 1 or 2 and 44% scored 2, versus 56% and 14% for vanilla.Source: Table 7 · p. 8, §5.4 · p. 8

6. Conclusion

The paper presents a two-stage framework combining instruction-augmented supervised fine-tuning and consistency alignment, and reports its effectiveness on Vicuna and Llama 2. It frames the approach as a plug-in for improving existing LLMs without external human guidance or reward models.

6. Conclusion

  • The framework combines instruction-augmented supervised fine-tuning and consistency alignment training, evaluated on Vicuna and Llama 2 instruction-following tasks.The conclusion says the paper separately verifies the effectiveness of the SFT(IA) and CAT modules.Source: §6 · p. 8
  • Human-evaluation wins, ties, and losses are determined by comparing response scores for expected answer type and correctness.Responses receive scores of 0, 1, or 2.Source: §6 · p. 8
  • The authors frame the method as a plug-in for continually improving existing LLMs, using decomposed self-rewards without external human guidance or reward models.They state that the self-rewards help generate more robust and accurate responses.Source: §6 · p. 9

7. limitations

The authors identify self-reward quality and instruction diversity as limitations of their approach.

7. limitations

  • Poor alignment of the model supplying self-rewards may prevent further improvement, and LLM-generated paraphrases may be less diverse than end-user instructions.The authors plan experiments on smaller, weaker models and datasets using instructions collected from end-users.Source: §7 · p. 9

8. Ethics Statement

The authors report no explicit ethical considerations because their models and datasets are public, while noting unresolved questions about encoded biases.

8. Ethics Statement

  • The effect of training on biases encoded by the models remains unclear, despite the use of public models and datasets.The authors acknowledge that LLMs may encode problematic biases.Source: §8 · p. 9
Loading 2403.14221v2…