Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models

Zixiang Chen; Yihe Deng; Huizhuo Yuan; Kaixuan Ji; Quanquan Gu

Source-linked AI summary

Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models

Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, Quanquan Gu

arXiv:2401.01335v3Jan 2, 2024cs.LGcs.AIcs.CLstat.ML

TL;DR

The paper asks whether a weak LLM can improve without additional human-annotated data. It proposes SPIN, which iteratively trains a model against responses generated by its previous versions, and reports substantial gains across benchmarks, including improvement from 58.14 to 63.16 on the HuggingFace Open LLM Leaderboard. The method’s fixed human target distribution imposes a performance ceiling, while reducing synthetic-data requirements remains future work.

Problem
Existing alignment methods require substantial human-annotated data, motivating fine-tuning methods that improve weak LLMs without acquiring additional annotations.
Method
SPIN starts from a supervised fine-tuned model and iteratively distinguishes human responses from synthetic responses generated by its previous iterations through self-play.
Results
SPIN consistently improves performance across benchmarks, raising the average score from 58.14 to 63.16 on the HuggingFace Open LLM Leaderboard and from 5.94 to 6.78 on MT-Bench.
Takeaways & Limitations
Self-play lets an LLM self-improve using its supervised fine-tuning data without additional human data or feedback from stronger LLMs.
Takeaways & Limitations
SPIN’s fixed human-generated target distribution imposes a ceiling on fine-tuned LLM performance, and synthetic-data generation requires substantial resources.

Abstract

from arXiv · show

Harnessing the power of human-annotated data through Supervised Fine-Tuning (SFT) is pivotal for advancing Large Language Models (LLMs). In this paper, we delve into the prospect of growing a strong LLM out of a weak one without the need for acquiring additional human-annotated data. We propose a new fine-tuning method called Self-Play fIne-tuNing (SPIN), which starts from a supervised fine-tuned model. At the heart of SPIN lies a self-play mechanism, where the LLM refines its capability by playing against instances of itself. More specifically, the LLM generates its own training data from its previous iterations, refining its policy by discerning these self-generated responses from those obtained from human-annotated data. Our method progressively elevates the LLM from a nascent model to a formidable one, unlocking the full potential of human-annotated demonstration data for SFT. Theoretically, we prove that the global optimum to the training objective function of our method is achieved only when the LLM policy aligns with the target data distribution. Empirically, we evaluate our method on several benchmark datasets including the HuggingFace Open LLM Leaderboard, MT-Bench, and datasets from Big-Bench. Our results show that SPIN can significantly improve the LLM's performance across a variety of benchmarks and even outperform models trained through direct preference optimization (DPO) supplemented with extra GPT-4 preference data. This sheds light on the promise of self-play, enabling the achievement of human-level performance in LLMs without the need for expert opponents. Codes are available at https://github.com/uclaml/SPIN.

1 Introduction

The paper asks whether a weak LLM can improve without additional human-annotated data and proposes SPIN, an iterative self-play fine-tuning method. SPIN distinguishes human responses from responses generated by earlier model iterations and improves benchmark performance.

Motivation: Alignment methods such as SFT and RLHF rely on costly human-annotated data.This motivates methods that use existing human data more efficiently.
SPIN: SPIN starts from a supervised fine-tuned model and trains it to distinguish human responses from responses generated by its previous iteration.The previous model generates responses for SFT prompts, while the new model learns to distinguish them from human responses.
SPIN: SPIN eliminates the need for expert annotators, including humans or more advanced LLMs such as GPT-4.The method uses self-play between model iterations rather than direct supervision from an expert opponent.
Results: SPIN outperforms continued SFT on its own dataset, which reaches a performance plateau or diminished evaluation scores.SPIN consistently improves the model across successive iterations while using only a 50k subset of Ultrachat200k.
Results: SPIN improves zephyr-7b-sft-full’s average score from 58.14 to 63.16 on the HuggingFace Open LLM Leaderboard and from 5.94 to 6.78 on MT-Bench.It also reports 10%+ score improvements on GSM8k and TruthfulQA and results comparable to models trained with an additional 62k preference dataset.

2 Related Work

The related work situates SPIN within self-play and synthetic-data approaches for improving LLMs while reducing reliance on costly human-authored data. SPIN applies self-play to LLM enhancement and uses synthetic data as a proxy for human data.

Self-Play: Self-play trains agents through interactions with copies of themselves, increasing the challenge and complexity of the learning environment.Prior work established self-play in multiagent reinforcement learning, including AlphaGo Zero.
Self-Play: SPIN applies self-play to enhance LLMs and aims to convert a weak model into a strong one without additional human-annotated data.The paper presents this as a new application of self-play beyond its established use in multiagent reinforcement learning.
Synthetic Data for LLMs: Synthetic data is increasingly used as a proxy for human data because human-crafted data is effective but costly to acquire at sufficient scale.The related work discusses synthetic data in the context of supervised fine-tuning for tasks including code generation and mathematical reasoning.

3 Problem Setting and Preliminaries

The paper formalizes LLM responses as conditional distributions and reviews SFT and RL fine-tuning. SFT aligns model predictions with high-quality labeled responses, while RL fine-tuning maximizes rewards under KL regularization but requires a reward function typically learned from preference data.

LLM Setup: An LLM generates a response y from conditional distribution pθ(·|x) given prompt x.The model generates response tokens autoregressively from previously generated tokens.
Supervised Fine-Tuning: SFT minimizes negative log-likelihood on prompts from q(·) and high-quality responses from pdata(·|x).The labeled responses define the target conditional distribution for fine-tuning.
Supervised Fine-Tuning: SFT reaches its minimum when pθ(y|x) aligns perfectly with pdata(y|x).The resulting model is expected to generate responses resembling the high-quality responses in the training distribution.
RL Fine-Tuning: RL fine-tuning maximizes expected reward while using KL regularization to keep pθ close to reference model pref.The regularization parameter λ controls deviation and helps reduce the risk of mode collapse.
RL Fine-Tuning: RL fine-tuning requires a reward function, typically trained from preference data compiled through human or strong-AI evaluations.Such preference-data collection demands substantial resources.

4 Method

SPIN iteratively improves a supervised fine-tuned LLM by contrasting human responses with synthetic responses from the previous model iteration. Its self-play objective trains a discriminator-like main player, updates the opponent model, and repeats this process without additional human or AI feedback.

4 Method: Further SFT can be ineffective or harmful, while unavailable human or AI feedback prevents constructing preference data for RL fine-tuning.A persistent quality gap between ground-truth and model-generated responses motivates iterative synthetic-data training.
4.1 Self-Play Fine-Tuning (SPIN): SPIN treats the new model as a main player distinguishing human responses from responses generated by the previous model as opponent.Both players are versions of the same LLM from different iterations.
4.1 Self-Play Fine-Tuning (SPIN): At each iteration, the previous LLM generates synthetic responses, the main player is trained, and the resulting model is copied as the next opponent.This creates an iterative sequence of opponent and main-player models.
Training the Main Player: The main player assigns higher values to human-distribution responses and lower values to responses from the opponent distribution.Its objective maximizes the expected value gap between pdata and pθt over prompts and paired responses.
Training the Main Player: SPIN uses logistic loss because it is non-negative, smooth, and has an exponentially decaying tail, preventing excessive growth in the scoring function.The loss is chosen from convex, decreasing alternatives described by the method.
Updating the Opponent Player: The ideal opponent update yields a model distribution matching the closed-form opponent selection, with θt+1 obtained by optimizing the SPIN objective.The end-to-end objective uses human responses y and previous-model responses y′ sampled from pθt(·|x).
Relation to DPO: SPIN differs from DPO by iteratively matching pθ(y|x) to pdata(y|x) through self-play rather than relying on a preference dataset.Only the logistic-loss version of SPIN becomes similar to DPO’s training objective.

5 Theoretical Analysis

Under monotonicity and convexity assumptions, SPIN’s global optimum aligns the LLM distribution with the target data distribution. Its iterative updates move probabilities toward that target, with λ controlling opponent updates and training stability.

SPIN’s global optimum is achieved only when the LLM distribution matches the target data distribution, under Assumption 5.1.Assumption 5.1 requires a monotonically decreasing convex loss with ℓ′(0) < 0.
Theorem 5.2 establishes sufficiency at distributional alignment and non-optimality away from alignment for an appropriately chosen λ.
For logistic loss, SPIN increases probabilities below the target distribution and decreases probabilities above it, driving iterative refinement toward alignment.
λ controls opponent-player updates: smaller values produce larger changes, while larger values improve stability as the model approaches the target distribution.λ acts as the regularization parameter controlling deviation of the opponent player.

6 Experiments

Experiments evaluate SPIN across the HuggingFace Open LLM Leaderboard and related benchmarks using synthetic responses generated from a supervised-fine-tuned model. SPIN improves performance across iterations, matches DPO at iteration 0, and surpasses it from iteration 1 while iterative training outperforms simply adding epochs.

SPIN markedly enhances performance across diverse benchmarks and breaks the performance limit of supervised fine-tuning without introducing new human-annotated data.
Model and datasets: Synthetic responses are generated from zephyr-7b-sft-full using a randomly sampled 50k prompts from Ultrachat200k.The base model derives from Mistral-7B and was previously fine-tuned on Ultrachat200k.
Benchmark performance: 2.66%: iteration 0 improves the average leaderboard score, followed by a further 1.32% gain at iteration 1.Iteration 0 exceeds 5% improvement on TruthfulQA and 10% on GSM8k; iteration 1 particularly improves Arc Challenge and TruthfulQA.
Comparison with DPO: SPIN at iteration 0 achieves average performance comparable to DPO despite using existing SFT data, and iteration 1 surpasses DPO on the leaderboard benchmark.DPO uses approximately 62k preference examples, whereas SPIN uses 50k SFT examples.
Ablation studies: Further epochs within iteration 0 yield only modest gains, and extending iteration 0 does not reach the performance achieved by iteration 1.The analysis reports stability rather than degradation with longer training, but identifies a ceiling within a single iteration.

7 Conclusion and Discussion

SPIN uses self-play to iteratively align a language model with human-demonstration data without additional human data or stronger-model feedback. The paper reports broad benchmark gains, while noting that a fixed human target distribution imposes a performance ceiling.

SPIN fine-tunes a main LLM to distinguish previous-iteration responses from target human data, then iteratively aligns its distribution with that target.
SPIN improves performance across diverse benchmarks without additional human data or feedback from stronger LLMs.
A fixed target distribution generated by humans imposes a ceiling on the fine-tuned LLM’s performance.The paper identifies dynamically changing the target distribution and reducing synthetic-data volume as future directions.

Impact Statement

SPIN is presented as a self-play approach for improving LLM capabilities with synthetic data, without human preference annotations. The paper frames this as a potential contribution to downstream applications.

Impact Statement: SPIN uses synthetic data to enhance LLMs without requiring human preference annotations.The paper positions this as its contribution to LLM development.
Impact Statement: The paper suggests that SPIN-generated synthetic data may further augment training for various language models.
Impact Statement: SPIN is described as improving LLM capabilities and opening avenues for downstream applications.

B.2 Training Overhead

SPIN adds generation overhead because it creates synthetic data, while its fine-tuning cost remains computationally comparable to SFT and DPO. Training time increases when successive iterations use doubled datasets.

Training Overhead: 6.69s generation and 10s training are required per 64 examples.These measurements use 8xA100 (80G) GPUs.
Training Overhead: SPIN’s fine-tuning process remains computationally equal to SFT and DPO, with most added cost coming from synthetic-data generation.
Training Overhead: Table 2 reports generation and training times across different SPIN iterations.
Training Overhead: Training time doubles from iteration 1 to iteration 3 because each later iteration combines 50k previous synthetic examples with 50k current examples.

B.3 Additional Experiment Result for SPIN+DPO

SPIN checkpoints can be further improved with DPO using additional GPT-4-evaluated preference data. This combined procedure produces a reported average enhancement of 0.89%.

Additional Experiment Result for SPIN+DPO: DPO further improves the SPIN iteration-3 checkpoint using chosen and rejected responses evaluated by GPT-4.
Additional Experiment Result for SPIN+DPO: Table 3 compares SPIN+DPO across HuggingFace Open LLM Leaderboard datasets with multiple baselines.
Additional Experiment Result for SPIN+DPO: SPIN followed by DPO yields an average enhancement of 0.89% over the last SPIN iteration.The DPO stage uses 62k UltraFeedback Binarized preference examples.
Additional Experiment Result for SPIN+DPO: The largest reported gain is on TruthfulQA, at around 5%.

B.4 Further Experiment Results

Further experiments show that SPIN improves performance across Open LLM Leaderboard, MT-Bench, Big-Bench, and OpenBookQA without major degradation. Theoretical analysis states that the target data distribution is the unique global optimum condition.

B.4 Further Experiment Results: SPIN improves Open LLM Leaderboard performance, whereas another epoch of SFT on the same data degrades performance.
B.4 Further Experiment Results: The additional evaluations cover MT-Bench, Big-Bench, OpenBookQA, and Open LLM Leaderboard tasks.
B.4 Further Experiment Results: SPIN steadily improves scores on MT-Bench, Big-Bench, and OpenBookQA without significant degradation.
B.4 Further Experiment Results: 6.57 is SPIN’s MT-Bench score, surpassing vicuna-13b-v1.5.
B.4 Further Experiment Results: Higher SPIN iterations generally produce more concise responses that more closely resemble the ground-truth completion.
B.4 Further Experiment Results: SPIN robustly outperforms the base SFT model across all MT-Bench evaluation aspects from iteration 1.
Theory: When the current model matches the target distribution, it is a global optimum of the SPIN objective and remains unchanged by the update.
Theory: If the current model does not match the target distribution, it is not a global optimum of the SPIN objective.

C.2 Proof Theorem 5.4

The proof characterizes the logistic-loss objective’s global optimum by reducing it to a Jensen–Shannon-divergence expression and identifying the equality condition. This establishes the closed-form optimizer used to prove the theorem under the constraint set.

Auxiliary lemmas: The logistic-loss term aℓ(t) + bℓ(−t) is minimized at t = log(a/b).The derivative is negative below log(a/b) and positive above it.
Auxiliary lemmas: Lemma C.2 gives a closed-form solution for the objective when the constraint set F_t is ignored.Its derivation rewrites the objective using the Jensen–Shannon divergence between p+ and p−.
Auxiliary lemmas: The Jensen–Shannon divergence is non-negative and equals zero exactly when p+ and p− are identical.The corresponding minimum value is log 2 − JSD(p+∥p−).
Theorem 5.4: If the closed-form solution belongs to F_t, it is also the solution of the constrained optimization problem.This observation supplies the key step for proving Theorem 5.4.