PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenChat: Advancing Open-source Language Models with Mixed-Quality Data

Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, Yang Liu

arXiv:2309.11235v2cs.CL

TL;DR

SFT treats mixed-quality data equally, while RLFT requires costly preference labels. OpenChat introduces C-RLFT, which uses data-source classes as coarse rewards and learns a class-conditioned policy through RL-free supervised learning. openchat-13b achieves the highest average performance among 13b open-source models and is the only fine-tuned model to surpass its base model on average AGIEval accuracy.

  • Problem

    SFT treats mixed-quality training data uniformly, whereas RLFT requires costly high-quality pairwise or ranking-based preference feedback.

    Source: §1 · p. 2

  • Method

    C-RLFT uses data-source classes as coarse-grained reward labels, trains a class-conditioned policy, and solves the optimal policy with reward-weighted supervised learning.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 4

  • Results

    openchat-13b achieves the highest average performance among 13b open-source language models and is the only fine-tuned model whose average AGIEval accuracy surpasses the base model.

    Source: §1 · p. 3, §4.2 · p. 7, §7 · p. 9

  • Takeaways & Limitations

    OpenChat provides a lightweight, RL-free way to fine-tune open-source models using mixed-quality data without preference labels.

    Source: §1 · p. 2, §7 · p. 9

    The method assumes data-source classes reflect quality, which may be overly simplistic, and its demonstrated focus is instruction following rather than reasoning.

    Source: §7 · p. 10

Abstract

from arXiv · show

Nowadays, open-source large language models like LLaMA have emerged. Recent developments have incorporated supervised fine-tuning (SFT) and reinforcement learning fine-tuning (RLFT) to align these models with human goals. However, SFT methods treat all training data with mixed quality equally, while RLFT methods require high-quality pairwise or ranking-based preference data. In this study, we present a novel framework, named OpenChat, to advance open-source language models with mixed-quality data. Specifically, we consider the general SFT training data, consisting of a small amount of expert data mixed with a large proportion of sub-optimal data, without any preference labels. We propose the C(onditioned)-RLFT, which regards different data sources as coarse-grained reward labels and learns a class-conditioned policy to leverage complementary data quality information. Interestingly, the optimal policy in C-RLFT can be easily solved through single-stage, RL-free supervised learning, which is lightweight and avoids costly human preference labeling. Through extensive experiments on three standard benchmarks, our openchat-13b fine-tuned with C-RLFT achieves the highest average performance among all 13b open-source language models. Moreover, we use AGIEval to validate the model generalization performance, in which only openchat-13b surpasses the base model. Finally, we conduct a series of analyses to shed light on the effectiveness and robustness of OpenChat. Our code, data, and models are publicly available at https://github.com/imoneoi/openchat and https://huggingface.co/openchat.

1 INTRODUCTION

Open-source LLM fine-tuning commonly relies on mixed-quality data, while SFT treats examples uniformly and RLFT requires costly preference feedback. OpenChat introduces C-RLFT to use coarse data-source quality labels, achieving strong benchmark performance and generalization.

  • SFT methods often treat mixed-quality conversations equally, although sub-optimal data may negatively impact learning.Source: §1 · pp. 1–2
  • RLFT methods can align models with desired behaviors but typically require substantial high-quality pairwise or ranking-based human feedback.Source: §1 · p. 2
  • OpenChat applies C-RLFT to non-pairwise SFT data by using different data sources as coarse-grained reward labels and learning a class-conditioned policy.Source: §1 · p. 2
  • openchat-13b surpasses previous 13b open-source language models and can outperform gpt-3.5-turbo across Alpaca-Eval, MT-bench, and Vicuna-bench.Source: §1 · p. 2

2 PRELIMINARIES

SFT adapts pretrained LLMs through supervised likelihood training but treats all examples uniformly, making mixed-quality datasets problematic. RLFT maximizes rewards under KL regularization, yet depends on costly high-quality preference signals.

Supervised Fine-tuning (SFT)

  • SFT fine-tunes a pretrained language model on conversation pairs using maximum likelihood estimates.Source: §2 · p. 3
  • Because SFT treats training data uniformly, sub-optimal examples in inexpensive conversation datasets can cause performance degeneration.Source: §2 · p. 3

Reinforcement Learning Fine-tuning (RLFT)

  • RLFT models rewards from human preferences or classifiers and fine-tune the language model to maximize those rewards.Source: §2 · p. 3
  • KL-regularized RLFT constrains the fine-tuned policy to remain close to the pretrained model, helping avoid distribution collapse.Source: §2 · p. 3
  • Effective RLFT requires considerable costly pairwise or ranking-based human preference feedback.Source: §2 · p. 3

3 OPENCHAT

OpenChat uses source classes to encode coarse quality information in mixed-quality datasets and trains a class-conditioned policy with a class-conditioned reference policy. Its optimal policy can be learned through reward-weighted supervised regression, avoiding ongoing pretrained-policy loading during training.

Setting

  • OpenChat considers a small expert dataset and a larger sub-optimal dataset without preference labels.Source: §3 · p. 3

3.1 CLASS-CONDITIONED DATASET AND REWARDS

  • Different data sources become class labels in Dc = {(x_i, y_i, c_i)}, producing a class-conditioned behavior policy πc(y|x, c).Source: §3.1 · p. 4
  • Coarse rewards assign expert data value 1 and sub-optimal data value α, with α < 1, favoring higher-quality responses.Source: §3.1 · p. 4

3.2 FINE-TUNING VIA C(ONDITIONED)-RLFT

  • C-RLFT fine-tunes πθ(y|x, c) and regularizes it against the class-conditioned reference policy πc rather than the pretrained policy π0.Source: §3.2 · p. 4
  • The optimal policy is learned through a simple reward-weighted regression objective using Dc, without requiring accurate reward labels.Source: §3.2 · p. 5
  • After initialization from π0, C-RLFT no longer needs π0 loaded during training, saving considerable computation resources.Source: §3.2 · p. 5

4 EXPERIMENTS

The experiments evaluate OpenChat on mixed-quality ShareGPT data using established instruction-following benchmarks and automatic evaluators. OpenChat achieves leading results among 13b open-source models and preserves generalization on AGIEval.

Experimental Setups

  • ShareGPT contains approximately 70k conversations, including around 6k GPT-4-generated expert conversations and the remaining GPT-3.5 conversations.Source: §4.1 · p. 5
  • The evaluation uses AlpacaEval, MT-bench, Vicuna-bench, and zero-shot English AGIEval tasks to assess instruction following and generalization.Source: §4.1 · p. 5
  • Automatic evaluation uses AlpacaEval's alpaca eval gpt and GPT-4 for MT-bench and Vicuna-bench, with GPT-3.5 and Claude-2 added to examine self-enhancement bias.Source: §4.1 · p. 5
  • Win rate awards 1 point for a win, 0.5 for a tie, and 0 for a loss; MT-bench uses scores from 1 to 10, while AGIEval reports accuracy.Source: §4.1 · pp. 5–6

Main Results

  • OpenChat achieves the highest win-rate scores in AlpacaEval, MT-bench, and Vicuna-bench among the evaluated 13b open-source models.Source: §4.2 · p. 6
  • OpenChat achieves the highest MT-bench score, exceeding open-source models with much larger parameter counts, including llama-2-chat-70b.Source: §4.2 · p. 7

5 ANALYSIS

The analysis examines mixed-data quality, component contributions, representation differences, inference conditioning, and robustness to data-size changes. Results show that OpenChat benefits from quality-aware conditioning and remains robust to variation in data size.

Data Quality

  • GPT-4 conversations contain more high-quality conversations and exhibit a higher average quality than GPT-3.5 conversations in ShareGPT.Source: §5.1 · p. 7

Ablation Studies

  • Only SFT on GPT-4 data performs much better than only SFT on the entire ShareGPT dataset, while OpenChat outperforms all only-SFT variants.The comparison indicates that both coarse-grained rewards and the class-conditioned policy contribute to performance.Source: §5.2 · p. 8

Representation Analysis

  • OpenChat and only-SFT representations form multiple clusters across GPT-4 and GPT-3.5 conversations, reflecting diverse conversation domains.Representations are obtained by mean pooling final-layer token outputs and mapped to two dimensions with UMAP.Source: §5.3 · p. 8

Inference Conditioning

  • Replacing the GPT-4 inference prompt with the GPT-3.5 prompt causes a substantial performance decline.The result supports the model’s ability to distinguish data-source quality through its class-conditioned policy.Source: §5.3 · pp. 8–9

Data-Size Effects

  • OpenChat’s average performance declines only modestly when one data class is subsampled from 60% to 100%.GPT-4 data-size variation has a more pronounced effect than GPT-3.5 data-size variation despite GPT-4 being much smaller, underscoring the importance of expert data.Source: §5.4 · p. 9

6 RELATED WORKS

Related work includes supervised fine-tuning methods that construct instruction datasets and reinforcement-learning fine-tuning methods that use preference signals. OpenChat differs by seeking to exploit mixed-quality data without treating all examples uniformly or requiring preference labels.

Supervised Fine-tuning for LLMs

  • Supervised fine-tuning methods improve base language models using instruction-following demonstrations and increasingly large instructional conversation datasets.Examples include Alpaca, Koala, WizardLM, UltraChat, Baize, and Vicuna.Source: §6 · p. 9
  • Unlike prior SFT studies that treat training data uniformly, OpenChat aims to maximize the use of mixed-quality data.Source: §6 · p. 9

Reinforcement Learning Fine-tuning for LLMs

  • RLHF trains reward models from human preference feedback and then uses PPO to maximize the resulting reward for the target language model.DPO and RRHF provide simpler alternatives through one-stage reward maximization or ranking loss.Source: §6 · p. 9
  • OpenChat uses easily collectible mixed-quality training data without preference labels to fine-tune language models.Source: §6 · p. 9

7 CONCLUSION AND FUTURE WORK

OpenChat introduces Conditioned-RLFT for fine-tuning open-source language models with mixed-quality data. The authors report strong benchmark performance while identifying coarse quality assumptions and reasoning ability as areas for future improvement.

Conclusion

  • OpenChat presents Conditioned-RLFT to advance open-source language models using mixed-quality data.Source: §7 · p. 9
  • openchat-13b delivers the highest average performance among all 13b open-source language models on extensive benchmarks.The reported advantages include simplicity, RL-free training, and minimal reward quality requirements.Source: §7 · pp. 9–10

Future Work

  • The assumption that data-source differences adequately represent quality may be overly simplistic, and coarse-grained rewards could be refined.Source: §7 · p. 10
  • Future work includes applying OpenChat to improve reasoning abilities beyond its primary focus on instruction following.Source: §7 · p. 10

ETHICS STATEMENT

The study frames open-source language models as accessible and transparent platforms that broaden participation in AI research and support collaboration.

Ethics Statement

  • Open-source language models democratize AI research by broadening accessibility and providing inclusive, transparent platforms.Source: ETHICS STATEMENT · p. 10
  • These models stimulate innovation, foster a diverse research community, and facilitate broader discussion and faster collaboration.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

OpenChat publicly releases its training code, data, and model weights, while evaluating alternative positions for class-conditioned prompt tokens during training.

B DERIVATION OF THE OPTIMAL POLICY IN C-RLFT

The derivation formulates C-RLFT as a constrained KL-regularized policy optimization problem and solves it using KKT conditions. The surrounding study evaluates models across instruction-following benchmarks and checks evaluator consistency.

B DERIVATION OF THE OPTIMAL POLICY IN C-RLFT

C DETAILS OF BENCHMARKS

D EXPERIMENTAL SETUP

E EVALUATORS CONSISTENCY

  • Automatic evaluations use GPT-4, GPT-3.5, and Claude-2, with model scores showing a Pearson correlation of r = 0.91 between GPT-3.5 and Claude-2.Source: E EVALUATORS CONSISTENCY · p. 16

F AGIEVAL RESULTS

AGIEval is evaluated with official zero-shot prompts and answer matching, while evaluator-consistency analyses compare GPT-4, GPT-3.5, and Claude-2. The supplied passages also identify comparison results for MT-bench, AlpacaEval, and Vicuna-bench.

F AGIEVAL RESULTS

H EVALUATORS CONSISTENCY

EVALUATION RESULTS

  • Table 7 reports MT-bench comparisons and states that only openchat-13b followed the user’s request in the second turn.Source: Table 7 · p. 18
  • Table 8 reports AlpacaEval comparisons in which openchat-13b provides a reasonable estimate for a complex hypothetical question.Source: Table 8 · p. 19
  • Table 9 reports Vicuna-bench comparisons in which only openchat-13b and wizardlm-13b-v1.2 answer the math problem correctly.Source: Table 9 · p. 20

Related papers

Loading 2309.11235v2…