PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenChat: Advancing Open-source Language Models with Mixed-Quality Data

Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, Yang Liu

arXiv:2309.11235v2cs.CL

TL;DR

SFT treats mixed-quality data equally, while RLFT requires costly preference labels. OpenChat introduces C-RLFT, which uses data-source classes as coarse rewards and learns a class-conditioned policy through RL-free supervised learning. openchat-13b achieves the highest average performance among 13b open-source models and is the only fine-tuned model to surpass its base model on average AGIEval accuracy.

  • Problem

    SFT treats mixed-quality training data uniformly, whereas RLFT requires costly high-quality pairwise or ranking-based preference feedback.

  • Method

    C-RLFT uses data-source classes as coarse-grained reward labels, trains a class-conditioned policy, and solves the optimal policy with reward-weighted supervised learning.

  • Results

    openchat-13b achieves the highest average performance among 13b open-source language models and is the only fine-tuned model whose average AGIEval accuracy surpasses the base model.

  • Takeaways & Limitations

    OpenChat provides a lightweight, RL-free way to fine-tune open-source models using mixed-quality data without preference labels.

  • Takeaways & Limitations

    The method assumes data-source classes reflect quality, which may be overly simplistic, and its demonstrated focus is instruction following rather than reasoning.

Abstract

from arXiv · show

Nowadays, open-source large language models like LLaMA have emerged. Recent developments have incorporated supervised fine-tuning (SFT) and reinforcement learning fine-tuning (RLFT) to align these models with human goals. However, SFT methods treat all training data with mixed quality equally, while RLFT methods require high-quality pairwise or ranking-based preference data. In this study, we present a novel framework, named OpenChat, to advance open-source language models with mixed-quality data. Specifically, we consider the general SFT training data, consisting of a small amount of expert data mixed with a large proportion of sub-optimal data, without any preference labels. We propose the C(onditioned)-RLFT, which regards different data sources as coarse-grained reward labels and learns a class-conditioned policy to leverage complementary data quality information. Interestingly, the optimal policy in C-RLFT can be easily solved through single-stage, RL-free supervised learning, which is lightweight and avoids costly human preference labeling. Through extensive experiments on three standard benchmarks, our openchat-13b fine-tuned with C-RLFT achieves the highest average performance among all 13b open-source language models. Moreover, we use AGIEval to validate the model generalization performance, in which only openchat-13b surpasses the base model. Finally, we conduct a series of analyses to shed light on the effectiveness and robustness of OpenChat. Our code, data, and models are publicly available at https://github.com/imoneoi/openchat and https://huggingface.co/openchat.

1 INTRODUCTION

Open-source LLM fine-tuning commonly relies on mixed-quality data, while SFT treats examples uniformly and RLFT requires costly preference feedback. OpenChat introduces C-RLFT to use coarse data-source quality labels, achieving strong benchmark performance and generalization.

  • SFT methods often treat mixed-quality conversations equally, although sub-optimal data may negatively impact learning.
  • RLFT methods can align models with desired behaviors but typically require substantial high-quality pairwise or ranking-based human feedback.
  • OpenChat applies C-RLFT to non-pairwise SFT data by using different data sources as coarse-grained reward labels and learning a class-conditioned policy.
  • openchat-13b surpasses previous 13b open-source language models and can outperform gpt-3.5-turbo across Alpaca-Eval, MT-bench, and Vicuna-bench.

2 PRELIMINARIES

SFT adapts pretrained LLMs through supervised likelihood training but treats all examples uniformly, making mixed-quality datasets problematic. RLFT maximizes rewards under KL regularization, yet depends on costly high-quality preference signals.

  • Supervised Fine-tuning (SFT): SFT fine-tunes a pretrained language model on conversation pairs using maximum likelihood estimates.
  • Supervised Fine-tuning (SFT): Because SFT treats training data uniformly, sub-optimal examples in inexpensive conversation datasets can cause performance degeneration.
  • Reinforcement Learning Fine-tuning (RLFT): RLFT models rewards from human preferences or classifiers and fine-tune the language model to maximize those rewards.
  • Reinforcement Learning Fine-tuning (RLFT): KL-regularized RLFT constrains the fine-tuned policy to remain close to the pretrained model, helping avoid distribution collapse.
  • Reinforcement Learning Fine-tuning (RLFT): Effective RLFT requires considerable costly pairwise or ranking-based human preference feedback.

3 OPENCHAT

OpenChat uses source classes to encode coarse quality information in mixed-quality datasets and trains a class-conditioned policy with a class-conditioned reference policy. Its optimal policy can be learned through reward-weighted supervised regression, avoiding ongoing pretrained-policy loading during training.

  • Setting: OpenChat considers a small expert dataset and a larger sub-optimal dataset without preference labels.
  • 3.1 CLASS-CONDITIONED DATASET AND REWARDS: Different data sources become class labels in Dc = {(x_i, y_i, c_i)}, producing a class-conditioned behavior policy πc(y|x, c).
  • 3.1 CLASS-CONDITIONED DATASET AND REWARDS: Coarse rewards assign expert data value 1 and sub-optimal data value α, with α < 1, favoring higher-quality responses.
  • 3.2 FINE-TUNING VIA C(ONDITIONED)-RLFT: C-RLFT fine-tunes πθ(y|x, c) and regularizes it against the class-conditioned reference policy πc rather than the pretrained policy π0.
  • 3.2 FINE-TUNING VIA C(ONDITIONED)-RLFT: The optimal policy is learned through a simple reward-weighted regression objective using Dc, without requiring accurate reward labels.
  • 3.2 FINE-TUNING VIA C(ONDITIONED)-RLFT: After initialization from π0, C-RLFT no longer needs π0 loaded during training, saving considerable computation resources.

4 EXPERIMENTS

The experiments evaluate OpenChat on mixed-quality ShareGPT data using established instruction-following benchmarks and automatic evaluators. OpenChat achieves leading results among 13b open-source models and preserves generalization on AGIEval.

  • Experimental Setups: ShareGPT contains approximately 70k conversations, including around 6k GPT-4-generated expert conversations and the remaining GPT-3.5 conversations.
  • Experimental Setups: The evaluation uses AlpacaEval, MT-bench, Vicuna-bench, and zero-shot English AGIEval tasks to assess instruction following and generalization.
  • Experimental Setups: Automatic evaluation uses AlpacaEval's alpaca eval gpt and GPT-4 for MT-bench and Vicuna-bench, with GPT-3.5 and Claude-2 added to examine self-enhancement bias.
  • Experimental Setups: Win rate awards 1 point for a win, 0.5 for a tie, and 0 for a loss; MT-bench uses scores from 1 to 10, while AGIEval reports accuracy.
  • Main Results: OpenChat achieves the highest win-rate scores in AlpacaEval, MT-bench, and Vicuna-bench among the evaluated 13b open-source models.
  • Main Results: OpenChat achieves the highest MT-bench score, exceeding open-source models with much larger parameter counts, including llama-2-chat-70b.

5 ANALYSIS

The analysis examines mixed-data quality, component contributions, representation differences, inference conditioning, and robustness to data-size changes. Results show that OpenChat benefits from quality-aware conditioning and remains robust to variation in data size.

  • Data Quality: GPT-4 conversations contain more high-quality conversations and exhibit a higher average quality than GPT-3.5 conversations in ShareGPT.
  • Ablation Studies: Only SFT on GPT-4 data performs much better than only SFT on the entire ShareGPT dataset, while OpenChat outperforms all only-SFT variants.The comparison indicates that both coarse-grained rewards and the class-conditioned policy contribute to performance.
  • Representation Analysis: OpenChat and only-SFT representations form multiple clusters across GPT-4 and GPT-3.5 conversations, reflecting diverse conversation domains.Representations are obtained by mean pooling final-layer token outputs and mapped to two dimensions with UMAP.
  • Inference Conditioning: Replacing the GPT-4 inference prompt with the GPT-3.5 prompt causes a substantial performance decline.The result supports the model’s ability to distinguish data-source quality through its class-conditioned policy.
  • Data-Size Effects: OpenChat’s average performance declines only modestly when one data class is subsampled from 60% to 100%.GPT-4 data-size variation has a more pronounced effect than GPT-3.5 data-size variation despite GPT-4 being much smaller, underscoring the importance of expert data.

6 RELATED WORKS

Related work includes supervised fine-tuning methods that construct instruction datasets and reinforcement-learning fine-tuning methods that use preference signals. OpenChat differs by seeking to exploit mixed-quality data without treating all examples uniformly or requiring preference labels.

  • Supervised Fine-tuning for LLMs: Supervised fine-tuning methods improve base language models using instruction-following demonstrations and increasingly large instructional conversation datasets.Examples include Alpaca, Koala, WizardLM, UltraChat, Baize, and Vicuna.
  • Supervised Fine-tuning for LLMs: Unlike prior SFT studies that treat training data uniformly, OpenChat aims to maximize the use of mixed-quality data.
  • Reinforcement Learning Fine-tuning for LLMs: RLHF trains reward models from human preference feedback and then uses PPO to maximize the resulting reward for the target language model.DPO and RRHF provide simpler alternatives through one-stage reward maximization or ranking loss.
  • Reinforcement Learning Fine-tuning for LLMs: OpenChat uses easily collectible mixed-quality training data without preference labels to fine-tune language models.

7 CONCLUSION AND FUTURE WORK

OpenChat introduces Conditioned-RLFT for fine-tuning open-source language models with mixed-quality data. The authors report strong benchmark performance while identifying coarse quality assumptions and reasoning ability as areas for future improvement.

  • Conclusion: OpenChat presents Conditioned-RLFT to advance open-source language models using mixed-quality data.
  • Conclusion: openchat-13b delivers the highest average performance among all 13b open-source language models on extensive benchmarks.The reported advantages include simplicity, RL-free training, and minimal reward quality requirements.
  • Future Work: The assumption that data-source differences adequately represent quality may be overly simplistic, and coarse-grained rewards could be refined.
  • Future Work: Future work includes applying OpenChat to improve reasoning abilities beyond its primary focus on instruction following.

ETHICS STATEMENT

The study frames open-source language models as accessible and transparent platforms that broaden participation in AI research and support collaboration.

  • Ethics Statement: Open-source language models democratize AI research by broadening accessibility and providing inclusive, transparent platforms.
  • Ethics Statement: These models stimulate innovation, foster a diverse research community, and facilitate broader discussion and faster collaboration.

REPRODUCIBILITY STATEMENT

OpenChat publicly releases its training code, data, and model weights, while evaluating alternative positions for class-conditioned prompt tokens during training.

  • The training code, data, and openchat-13b model weights are publicly available through GitHub and Hugging Face.
  • The study tests class-conditioned prompt tokens placed before the speaker, before the assistant, or at the beginning.
  • Prompt tokens inserted in every conversation turn perform similarly whether placed before the speaker or before the assistant.

B DERIVATION OF THE OPTIMAL POLICY IN C-RLFT

The derivation formulates C-RLFT as a constrained KL-regularized policy optimization problem and solves it using KKT conditions. The surrounding study evaluates models across instruction-following benchmarks and checks evaluator consistency.

  • B DERIVATION OF THE OPTIMAL POLICY IN C-RLFT: C-RLFT seeks an optimal KL-regularized conditional policy under a normalization constraint.
  • B DERIVATION OF THE OPTIMAL POLICY IN C-RLFT: The constrained optimization problem is solved by constructing a Lagrangian and applying Karush-Kuhn-Tucker conditions.
  • B DERIVATION OF THE OPTIMAL POLICY IN C-RLFT: The derivation obtains a closed-form optimal policy whose normalization term ensures a valid probability distribution.
  • C DETAILS OF BENCHMARKS: The study uses AlpacaEval, MT-bench, Vicuna-bench, and AGIEval to assess instruction following and generalization.
  • D EXPERIMENTAL SETUP: Table 4 specifies the base models, context lengths, fine-tuning methods, and datasets used for OpenChat and comparison models.
  • E EVALUATORS CONSISTENCY: Automatic evaluations use GPT-4, GPT-3.5, and Claude-2, with model scores showing a Pearson correlation of r = 0.91 between GPT-3.5 and Claude-2.

F AGIEVAL RESULTS

AGIEval is evaluated with official zero-shot prompts and answer matching, while evaluator-consistency analyses compare GPT-4, GPT-3.5, and Claude-2. The supplied passages also identify comparison results for MT-bench, AlpacaEval, and Vicuna-bench.

  • F AGIEVAL RESULTS: AGIEval uses official zero-shot prompts and answer matching, with conversation models receiving their corresponding conversation templates.
  • H EVALUATORS CONSISTENCY: The evaluator-consistency analysis compares GPT-4, GPT-3.5, and Claude-2 across model outputs.
  • EVALUATION RESULTS: Table 7 reports MT-bench comparisons and states that only openchat-13b followed the user’s request in the second turn.
  • EVALUATION RESULTS: Table 8 reports AlpacaEval comparisons in which openchat-13b provides a reasonable estimate for a complex hypothetical question.
  • EVALUATION RESULTS: Table 9 reports Vicuna-bench comparisons in which only openchat-13b and wizardlm-13b-v1.2 answer the math problem correctly.
Loading 2309.11235v2…