PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models

Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, Quanquan Gu

arXiv:2401.01335v3cs.LGcs.AIcs.CLstat.ML

TL;DR

Can weak language models be strengthened without collecting additional human annotations? SPIN addresses this through iterative self-play, fine-tuning a supervised model to distinguish human responses from its earlier outputs. Experiments show improved performance across benchmarks.

  • Problem

    The challenge is to strengthen weak LLMs without acquiring additional human-annotated data, given existing alignment methods’ substantial annotation requirements.

    Source: §1 · p. 1

  • Method

    SPIN iteratively fine-tunes a supervised model to distinguish existing human responses from responses generated by its previous iteration.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    SPIN improves performance across benchmarks, raising the base model’s HuggingFace Open LLM Leaderboard average score from 58.14 to 63.16.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Existing human demonstrations can support further LLM improvement without new human data or feedback from stronger models.

    Source: §7 · p. 8

    SPIN targets a fixed human-generated data distribution, which the authors identify as imposing a ceiling on fine-tuned model performance.

    Source: §7 · p. 8

Abstract

from arXiv · show

Harnessing the power of human-annotated data through Supervised Fine-Tuning (SFT) is pivotal for advancing Large Language Models (LLMs). In this paper, we delve into the prospect of growing a strong LLM out of a weak one without the need for acquiring additional human-annotated data. We propose a new fine-tuning method called Self-Play fIne-tuNing (SPIN), which starts from a supervised fine-tuned model. At the heart of SPIN lies a self-play mechanism, where the LLM refines its capability by playing against instances of itself. More specifically, the LLM generates its own training data from its previous iterations, refining its policy by discerning these self-generated responses from those obtained from human-annotated data. Our method progressively elevates the LLM from a nascent model to a formidable one, unlocking the full potential of human-annotated demonstration data for SFT. Theoretically, we prove that the global optimum to the training objective function of our method is achieved only when the LLM policy aligns with the target data distribution. Empirically, we evaluate our method on several benchmark datasets including the HuggingFace Open LLM Leaderboard, MT-Bench, and datasets from Big-Bench. Our results show that SPIN can significantly improve the LLM's performance across a variety of benchmarks and even outperform models trained through direct preference optimization (DPO) supplemented with extra GPT-4 preference data. This sheds light on the promise of self-play, enabling the achievement of human-level performance in LLMs without the need for expert opponents. Codes are available at https://github.com/uclaml/SPIN.

1 Introduction

SPIN strengthens a supervised fine-tuned LLM through iterative self-play, reusing existing human demonstrations without additional annotations. Experiments show benchmark improvements, while theoretical analysis establishes convergence when the model distribution matches the target data distribution.

1 Introduction

  • The motivating problem is whether a weak LLM can improve without additional human-annotated data, given the substantial annotation requirements of existing alignment methods.The aim is to extract more value from the existing fine-tuning dataset rather than acquire further human annotations.Source: §1 · p. 1
  • SPIN iteratively fine-tunes an LLM to prefer existing human demonstrations over responses generated by its previous version, which serves as its self-play opponent.Starting from a supervised fine-tuned model, each updated model becomes the next response-generating opponent; no additional human or stronger-model annotator is required.Source: §1 · p. 2
  • SPIN’s theoretical convergence condition is equality between the LLM’s response distribution and the target data distribution.Unlike DPO (Rafailov et al., 2023), SPIN requires no extra human preference data. Its GAN-like mechanism uses different iterations of the same LLM as generator and discriminator.Source: §1 · p. 2
  • 58.14 to 63.16 is SPIN’s average-score improvement on the HuggingFace Open LLM Leaderboard.MT-Bench (Zheng et al., 2023) improves from 5.94 to 6.78, with 10%+ score improvements on GSM8k and TruthfulQA. Results are comparable to models trained with an additional 62k preference dataset (Tunstall et al., 2023a).Source: §1 · p. 2
  • 50k examples from Ultrachat200k (Ding et al., 2023) support successive SPIN improvements, whereas continued SFT on its own dataset plateaus or worsens.Experiments use zephyr-7b-sft-full (Tunstall et al., 2023a), a supervised fine-tuned model based on Mistral-7B (Jiang et al., 2023).Source: §1 · p. 2
  • SPIN avoids additional human binary feedback and extra reward models, and operates with a single LLM rather than separate weak and strong models.Singh et al. (2023) use synthetic data with binary feedback, whereas SPIN avoids additional human binary feedback or an extra reward model. Burns et al. (2023) require both a weak supervisor and a strong model.Source: §1 · p. 2

2 Related Work

SPIN applies self-play to strengthen LLMs without additional human-annotated data. Unlike prior synthetic-data approaches that rely on more advanced models, it generates data directly from the target model.

Self-Play

  • The authors position SPIN as the first application of self-play to LLM enhancement, to their knowledge, strengthening weak models without additional human-annotated data.The approach follows the self-play paradigm used by AlphaGo Zero.Source: §2 · pp. 2–3
  • AlphaGo Zero demonstrated exceptional performance against human players through self-play, where agents learn by interacting with copies of themselves.Such interactions introduce increasing challenge and complexity, and self-play's effectiveness is well established in multiagent reinforcement learning.Source: §2 · p. 2

Synthetic Data for LLMs

  • SPIN generates synthetic data from the target model itself, unlike prior approaches that use more advanced models to generate training data.Synthetic data offers an alternative to costly human-crafted data. Prior research also uses LLM rephrasing to augment synthetic data for SFT (Yu et al., 2023; Liu et al., 2023).Source: §2 · p. 3

3 Problem Setting and Preliminaries

The section frames language modeling as conditional, autoregressive response generation and reviews two fine-tuning approaches. SFT fits high-quality labeled responses, while RL fine-tuning uses rewards and KL regularization but requires a suitable reward function, typically learned from resource-intensive preference data.

3 Problem Setting and Preliminaries

  • An autoregressive LLM samples responses conditioned on a prompt, generating each token using the prompt and preceding response tokens.Prompts and responses are sequences of tokens from a predetermined vocabulary.Source: §3 · p. 3

3.1 Supervised Fine-Tuning

  • SFT adapts pre-trained models to downstream tasks by minimizing negative log-likelihood on high-quality labeled responses; its minimum occurs when model and data distributions match.Prompts follow q(·), and labeled datasets are relatively small compared with pre-training data. The resulting model is expected to generate responses resembling the task’s high-quality training data.Source: §3.1 · p. 3

3.2 RL Fine-Tuning

  • RL fine-tuning maximizes response rewards while KL regularization constrains deviation from a reference model, typically initialized from SFT.The positive regularization parameter λ controls deviation from the reference model. This constraint helps prevent excessive deviation and reduces the risk of mode collapse.Source: §3.2 · p. 3
  • Finding a good reward function is RL fine-tuning’s primary challenge, typically requiring training on a resource-intensive preference dataset.Preference evaluations can come from human annotators through RLHF (Christiano et al., 2017; Bai et al., 2022a) or strong AI agents through RLAIF (Bai et al., 2022b).Source: §3.2 · pp. 3–4

4 Method

SPIN refines a supervised fine-tuned LLM through iterative self-play using existing human demonstrations and model-generated responses, without additional human or AI feedback. Its discrimination and KL-regularized opponent updates combine into one end-to-end training objective.

4 Method

  • A persistent quality gap between SFT demonstrations and model responses motivates synthetic-data refinement when further SFT is ineffective and additional feedback is unavailable.Reapplying SFT to the same dataset can worsen performance, while obtaining preference data for RL fine-tuning requires human or AI feedback.Source: §4 · p. 4

4.1 Self-Play Fine-Tuning (SPIN)

  • SPIN trains successive versions of the same LLM to distinguish human demonstrations from responses generated by the previous iteration.The previous model generates responses to SFT prompts; the newly trained model is copied to become the next opponent.Source: §4.1 · pp. 4–5
  • A convex, decreasing logistic loss trains the discriminator to score human responses above model responses while avoiding the unbounded objective associated with linear loss.The score-gap formulation is motivated by integral probability metrics. Logistic loss is nonnegative, smooth, and has an exponentially decaying tail, helping prevent excessive score magnitudes.Source: §4.1 · p. 4
  • The opponent maximizes the main player’s scores under KL regularization, and both players’ updates combine into one end-to-end training objective.KL regularization limits deviation from the previous model to stabilize self-play. The chosen function class makes the learned LLM exactly realize the ideal opponent update.Source: §4.1 · pp. 4–5

4.2 Comparison between SPIN and DPO

  • Unlike DPO (Rafailov et al., 2023), SPIN inherently iterates toward the target response distribution using SFT pairs rather than preference triples.SPIN can precede RL fine-tuning and accepts convex, decreasing losses; only logistic loss yields a DPO-like objective. Unlike Xu et al. (2023) and Yuan et al. (2024), its self-assessment requires no intermediate reward or preference feedback.Source: §4.2 · pp. 5–6

5 Theoretical Analysis

Under loss and representability assumptions, SPIN’s theoretical analysis identifies target-distribution matching as its optimization stopping condition. Logistic-loss updates tend to correct probability mismatches, with λ controlling update size and training stability.

5 Theoretical Analysis

  • The loss must be convex and monotonically decreasing, with ℓ′(t) ≤ 0 everywhere and ℓ′(0) < 0.These assumptions hold for correlation, hinge, exponential, and logistic losses.Source: §5 · p. 6
  • Matching the target distribution is globally optimal for every λ ≥ 0; any mismatch fails global optimality for an appropriately chosen λ.The theorem assumes the target distribution is representable by the LLM and the loss satisfies Assumption 5.1. Under these conditions, distribution matching is the optimization’s stopping condition.Source: §5 · p. 6
  • With logistic loss, SPIN updates tend to increase probabilities underestimated relative to the target distribution and decrease overestimated probabilities.The characterization assumes the prescribed update lies within the LLM family and the next iterate globally minimizes the SPIN objective.Source: §5 · p. 6
  • Smaller λ produces larger opponent updates, while larger λ limits changes and improves training stability as the model approaches the target distribution.The update is controlled by 1/λ, consistent with λ’s role in KL regularization controlling opponent deviation.Source: §5 · p. 7

6 Experiments

SPIN improves Open LLM Leaderboard performance beyond its SFT starting model and surpasses a DPO comparator while reusing existing SFT data. Ablations show that larger synthetic datasets help and that additional epochs cannot replace iterative training.

6.1 Experiment Setup

  • 50k sampled Ultrachat200k prompts support SPIN training from zephyr-7b-sft-full, with two training epochs per iteration.Synthetic datasets contain 50k examples at iteration 0 and 100k at iterations 1–3, combining newly generated data with synthetic data from the preceding iteration. The starting model is Mistral-7B (Jiang et al., 2023) fine-tuned on Ultrachat200k.Source: §6.1 · p. 7
  • Six Open LLM Leaderboard datasets assess commonsense reasoning, multitask language understanding, human falsehood mimicry, and mathematical problem solving.Evaluation uses the Language Model Evaluation Harness library.Source: §6.1 · p. 7

6.2 SPIN Effectively Improves Benchmark Performance

  • SPIN improves leaderboard performance across iterations and surpasses DPO from iteration 1 while using existing SFT data rather than new preference data.The DPO comparator, zephyr-7b-beta, uses approximately 62k GPT-4-evaluated preference examples. SPIN already matches its average performance at iteration 0 using 50k SFT examples.Source: Figure 2 · p. 7, §6.2 · pp. 7–8, §6 · p. 7
  • 2.66% average-score improvement occurs at iteration 0, followed by another 1.32% at iteration 1 on the Open LLM Leaderboard.Iteration 0 improves TruthfulQA by more than 5% and GSM8k by more than 10%. Later iterations deliver diminishing gains, approaching zero in the last iteration.Source: §6.2 · p. 7, Figure 2 · p. 7

6.3 Ablation Studies

  • 1% is the maximum improvement from further SFT epochs, while SPIN improves as synthetic training size increases.The study evaluates nested synthetic datasets of 14k, 26k, and 50k examples after one SPIN epoch, comparing against additional SFT epochs on Ultrachat200k.Source: §6.3 · p. 8
  • Extending iteration 0 training maintains stable performance but cannot reach iteration 1 performance, demonstrating the need for iterative training.The largest gains occur during the first two epochs; subsequent epochs provide only modest improvements without degrading performance.Source: §6 · p. 7, §6.3 · p. 8

7 Conclusion and Discussion

SPIN uses iterative self-play to improve LLMs without additional human data or stronger-model feedback. Its fixed human-generated target distribution imposes a performance ceiling, motivating research into dynamic targets and reduced synthetic-data requirements.

Conclusion

  • SPIN improves performance across diverse benchmarks, outperforming models trained with additional human data or AI feedback.These improvements require neither additional human data nor feedback from stronger LLMs.Source: §7 · p. 8
  • SPIN iteratively aligns an LLM with the target data distribution by learning to distinguish previous-iteration model responses from target responses.The current LLM serves as the main player, while the previous iteration supplies the opponent responses.Source: §7 · p. 8

Limitation and Future Work

  • SPIN’s optimization converges if and only if the model distribution matches pdata, while the fixed human-generated target distribution imposes a performance ceiling.Dynamically changing the target distribution is proposed as a direction for exceeding this ceiling, potentially toward super-human performance.Source: §7 · p. 8
  • Reducing the required volume of synthetic data is another future direction because generating it demands resources.Source: §7 · p. 8

Impact Statement … B.1 Hyperparameters and Implementation Details

SPIN aims to enhance LLMs without human preference annotations through synthetic data and self-play between consecutive model iterations. The appendices connect this approach to related learning frameworks and specify training, data-generation, and evaluation settings.

Impact Statement

  • SPIN-generated synthetic data can augment language-model training without human preference annotations, potentially broadening downstream applications.The impact statement emphasizes improved LLM capabilities and potential technological and societal applications.Source: Impact Statement · p. 9

A Further Related Work

B.1 Hyperparameters and Implementation Details

  • Table 1 documents dataset-specific few-shot example counts and evaluation metrics for the HuggingFace Open LLM Leaderboard.The supplied caption identifies the evaluation scope but provides no numerical settings or performance results.Source: Table 1 · p. 15
  • 5e-7 is the peak learning rate for iterations 0 and 1, decreasing to 1e-7 for iterations 2 and 3.Training uses RMSProp without weight decay, global batch size 64, 10% warmup, and bfloat16 precision. The Alignment Handbook codebase incorporates DeepSpeed ZeRO-3 (Rajbhandari et al., 2020) and FlashAttention-2 (Dao, 2023) to reduce training cost.Source: §B.1 · p. 15
  • 0.1 is the default β, increased to 5.0 at iteration 3 near convergence; the maximum sequence length is 2048 tokens.Synthetic-data generation uses Accelerate for distributed multi-GPU inference with global batch size 64. Prompts follow the instruction/response template used in Taori et al. (2023), and Ultrachat200k contributes only first-round prompt/completion pairs.Source: §B.1 · p. 16

B.2 Training Overhead

SPIN’s additional overhead mainly comes from generating synthetic data; fine-tuning remains computationally equal to SFT and DPO. Generation takes less time than training, and combining synthetic datasets doubles training time.

B.2 Training Overhead

  • SPIN’s additional computational overhead mainly comes from synthetic-data generation, while fine-tuning remains computationally equal to SFT and DPO.Source: §B.2 · p. 16
  • 6.69s generation and 10s training per 64 examples were measured for SPIN on a machine with 8xA100 (80G) GPUs.Generation takes less time than training at each iteration.Source: Table 2 · p. 16, §B.2 · p. 16
  • 2× training time from iter 1 to iter 3 results from combining 50k previous-iteration and 50k current-iteration synthetic examples into a double-sized dataset.The time estimates assume that 50k examples are generated per iteration.Source: §B.2 · p. 16

B.3 Additional Experiment Result for SPIN+DPO

DPO further improves the SPIN checkpoint using additional preference data, yielding a 0.89% average improvement and around 5% improvement on TruthfulQA.

B.3 Additional Experiment Result for SPIN+DPO

  • 62k preference examples from UltraFeedback Binarized train DPO for two epochs starting from SPIN iteration 3.The dataset contains chosen and rejected responses evaluated by GPT-4. SPIN itself requires only SFT data and can precede additional preference-based fine-tuning.Source: §B.3 · p. 16
  • 0.89% average improvement over the SPIN checkpoint is reported after DPO fine-tuning across HuggingFace Open LLM Leaderboard datasets.The comparison uses zephyr-7b-sft-full as its base model.Source: Table 3 · p. 16, §B.3 · p. 16
  • Around 5% improvement on TruthfulQA is reported when DPO further fine-tunes the SPIN checkpoint.The authors identify this benchmark's improvement as particularly significant.Source: §B.3 · p. 16

B.4 Further Experiment Results … C.1 Proof of Theorem 5.2

Further experiments show SPIN improves benchmark performance and example response quality, while repeated SFT degrades performance. Theorem 5.2’s proof establishes optimality and unchanged updates when model and data distributions match, and nonoptimality otherwise for sufficiently small positive λ.

B.4 Further Experiment Results

  • SPIN improves zephyr-7b-sft-full using synthetic data, whereas another epoch of SFT on Ultrachat200k degrades its Open LLM Leaderboard performance.Table 4 reports SPIN’s task-level performance across iterations; Table 5 reports the additional-SFT comparison.Source: §B.4 · p. 16, Table 4 · p. 17, Table 5 · p. 17
  • 6.57 on MT-Bench (Zheng et al., 2023) puts SPIN above vicuna-13b-v1.5, alongside improvements across additional benchmarks without major degradation.Additional evaluations cover Big-Bench-Hard causal judgment, sports understanding, and formal fallacies, plus OpenBookQA (Mihaylov et al., 2018). Metrics include MT-Bench average score, Big Bench accuracy under standard few-shot CoT, and OpenBookQA 1-shot acc_norm as used in Anil et al. (2023).Source: Table 6 · p. 17, §B.4 · p. 17

B.5 Generation Examples

  • SPIN’s generation examples improve in quality over the SFT checkpoint, while higher iterations typically produce more concise responses resembling ground-truth completions.The concision comparison is against iteration 0.Source: §B.5 · p. 17
  • Starting at iteration 1, SPIN outperforms the base SFT checkpoint across every evaluated MT-Bench aspect.Figure 6 compares performance across question types and SPIN iterations.Source: §B.4 · p. 17, Figure 6 · p. 18

C.1 Proof of Theorem 5.2

  • Matching the data distribution makes theta_t a global optimum of the SPIN objective and leaves the next iterate unchanged.Symmetry and Jensen’s inequality, using convexity of ℓ from Assumption 5.1, establish L_SPIN(theta, theta_t) ≥ ℓ(0) = L_SPIN(theta_t, theta_t).Source: §C.1 · p. 19
  • When model and data distributions differ, theta_t is not a global optimum for sufficiently small positive λ.The necessity argument uses ℓ′(0) < 0 to obtain g(λ) < ℓ(0) and chooses theta_* whose policy matches p_data.Source: §C.1 · p. 20

C.2 Proof Theorem 5.4

The proof of Theorem 5.4 uses auxiliary lemmas to obtain an unconstrained logistic-loss optimum, then shows that this optimum also solves the constrained problem because it belongs to F_t.

C.2 Proof Theorem 5.4

  • Lemma C.1 shows that aℓ(t) + bℓ(−t) has its unique minimum at t = log(a/b) for positive a and b under logistic loss.The derivative is negative below log(a/b) and positive above it, establishing the minimum.Source: §C.2 · p. 20
  • log 2 - JSD(p+∥p−) is the unconstrained logistic objective’s global minimum, with JSD nonnegative and zero exactly when p+ and p− coincide.Lemma C.2 provides a closed-form solution when the constraint set F_t is ignored.Source: §C.2 · pp. 20–21
  • Theorem 5.4 establishes global optimality for (4.2) by showing that the unconstrained optimizer f* belongs to F_t.Feasibility lets the closed-form solution from Lemma C.2 also solve the constrained optimization problem.Source: §C.2 · p. 22

Related papers

Loading 2401.01335v3…