PaperlayerView exact revision on arXiv

Source-linked AI summary

Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMs

Yibo Wang, Hai-Long Sun, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, Lijun Zhang

arXiv:2601.08198v1cs.CLcs.LG

TL;DR

SPIN addresses scarce annotated data but can become unstable as current advantages vanish and can misalign training rewards with generation likelihoods. T-SPIN adds historical advantages through triplet-based self-play and uses entropy-constrained reference-free optimization, outperforming SPIN and matching or exceeding supervised fine-tuning with 25% of its samples.

  • Problem

    SPIN faces unstable optimization when current advantages vanish and misalignment between reference-policy rewards and generation likelihoods.

  • Method

    T-SPIN jointly optimizes current and historical advantages over annotated, synthetic, and proto-synthetic responses, while entropy constraints remove the need for a reference policy.

  • Results

    T-SPIN outperforms SPIN and SFT across evaluated tasks, with particularly strong gains in math and instruction-following and stable iterative performance.

  • Takeaways & Limitations

    Using 25% annotated samples, T-SPIN achieves performance comparable to or better than supervised fine-tuning on some tasks.

  • Takeaways & Limitations

    T-SPIN regenerates synthetic responses for all prompts each iteration, and its fixed annotated data may not suit shifting target distributions.

Abstract

from arXiv · show

Recently, self-play fine-tuning (SPIN) has been proposed to adapt large language models to downstream applications with scarce expert-annotated data, by iteratively generating synthetic responses from the model itself. However, SPIN is designed to optimize the current reward advantages of annotated responses over synthetic responses at hand, which may gradually vanish during iterations, leading to unstable optimization. Moreover, the utilization of reference policy induces a misalignment issue between the reward formulation for training and the metric for generation. To address these limitations, we propose a novel Triplet-based Self-Play fIne-tuNing (T-SPIN) method that integrates two key designs. First, beyond current advantages, T-SPIN additionally incorporates historical advantages between iteratively generated responses and proto-synthetic responses produced by the initial policy. Even if the current advantages diminish, historical advantages remain effective, stabilizing the overall optimization. Second, T-SPIN introduces the entropy constraint into the self-play framework, which is theoretically justified to support reference-free fine-tuning, eliminating the training-generation discrepancy. Empirical results on various tasks demonstrate not only the superior performance of T-SPIN over SPIN, but also its stable evolution during iterations. Remarkably, compared to supervised fine-tuning, T-SPIN achieves comparable or even better performance with only 25% samples, highlighting its effectiveness when faced with scarce annotated data.

1 Introduction

Self-play fine-tuning adapts LLMs with limited annotated data by iteratively generating responses, but SPIN can become unstable and misalign rewards with generation. T-SPIN addresses these issues with historical advantages and entropy-constrained, reference-free optimization, achieving strong performance with scarce data.

  • Self-play fine-tuning progressively refines an LLM by having a main player distinguish expert-annotated responses from self-generated responses.
  • SPIN can become unstable when annotated and synthetic responses converge, because its advantage vanishes and the objective becomes policy-independent.
  • SPIN’s reference-policy reward can misalign training rewards with the log likelihood that determines generation priority.
  • T-SPIN combines current and historical advantages using annotated, synthetic, and proto-synthetic responses, preserving an optimization signal when current advantages diminish.
  • T-SPIN significantly improves performance, particularly in math and instruction-following, while remaining stable across iterations.
  • Using only 25% of annotated samples, T-SPIN matches or exceeds supervised fine-tuning on some tasks.

2 Preliminary

Supervised fine-tuning aligns a policy with annotated responses but typically requires extensive high-quality data. Self-play reduces this data burden through iterative synthetic responses, while SPIN faces instability and training-generation misalignment.

  • Supervised fine-tuning: Supervised fine-tuning maximizes the log likelihood of high-quality annotated responses to align the policy with target data.
  • Supervised fine-tuning: The supervised fine-tuning optimum matches the annotated-response distribution but typically demands extensive high-quality samples.
  • Self-play fine-tuning: Self-play fine-tuning combines annotated samples with continuously refined self-generated responses instead of relying on a fixed annotated dataset alone.
  • Self-play fine-tuning: At each iteration, SPIN samples a synthetic response from the previous policy and updates the current policy using annotated–synthetic response pairs.
  • Self-play fine-tuning: SPIN’s objective uses a convex decreasing loss over annotated and synthetic responses, with the previous policy supplying the reference reward.
  • Self-play fine-tuning: When synthetic responses equal annotated responses, SPIN’s advantage vanishes, making every policy optimal and potentially causing instability or performance degradation.
  • Self-play fine-tuning: Because SPIN’s reward includes the previous policy, higher reward does not necessarily imply higher generation probability.

3 Triplet-based self-play fine-tuning

T-SPIN extends self-play fine-tuning with triplet-based historical comparisons and entropy-regularized, reference-free opponent updates. These designs target stable iterative refinement while aligning the training objective with generation.

  • Main-player update: T-SPIN compares annotated, current synthetic, and initial proto-synthetic responses to model both current and historical advantages.The proto-synthetic response serves as an anchor for measuring progress from the initial policy.
  • Stable evolution: The historical advantage remains meaningful as current advantages diminish, preventing the objective from degenerating into a policy-independent constant.Because the proto-synthetic response remains fixed, the historical comparison continues to provide an optimization signal.
  • Opponent-player update: The opponent player maximizes expected confidence over generated responses while entropy regularization stabilizes its optimization.The regularization parameter α controls the entropy constraint, and the opponent objective has a closed-form solution.
  • Reference-free formulation: T-SPIN uses the policy-only confidence c_t+1(x, y′) = α log πθ(y′|x), avoiding explicit optimization and reference-policy dependence.This confidence function is chosen so the policy is the optimal solution of the entropy-constrained opponent objective.
  • End-to-end objective: The unified T-SPIN loss combines the annotated-versus-synthetic and synthetic-versus-proto-synthetic comparisons using the current policy likelihoods.Its expectation samples prompts, annotated responses, current synthetic responses, and initial-policy proto-synthetic responses.
  • Comparison with SPIN: T-SPIN differs from SPIN through historical-advantage optimization, reference-free reward formulation, and only marginal additional computational and memory costs.The additional proto-synthetic generation occurs once at the beginning rather than at every iteration.

4 Experiments

Experiments across diverse benchmarks compare T-SPIN with SPIN, SFT, and ablated variants, showing stronger performance, more stable iteration, reward-generation alignment, and robustness to hyperparameters.

  • Comparisons to SPIN and SFT: T-SPIN is evaluated against SPIN and SFT across mathematical, logical, knowledge, commonsense, and instruction-following tasks using Zephyr-7B and Mistral-7B.The experiments use 50k annotated samples for self-play methods and assess 10 tasks across multiple capabilities.
  • Comparisons to SPIN and SFT: T-SPIN improves average performance, including 14.82 points on GSM8K and 28.32 points on IFEval, while SPIN degrades after peaking at iteration 3.T-SPIN remains stable across iterations, unlike SPIN's subsequent performance decline.
  • Aligning training rewards and generation log-likelihoods: SPIN produces higher rewards but lower log-likelihoods for annotated responses, with nearly half of samples showing this mismatch; T-SPIN aligns the training reward with generation likelihood.SPIN's reference-policy reward differs from its generation metric, whereas T-SPIN uses a reference-free formulation.
  • Fine-tuning with less annotated data: T-SPIN with 50k annotated responses reaches a 42.56% average score, exceeding SFT's 42.01% with 200k annotated responses.With the same 50k data amount, T-SPIN scores 42.56% versus 40.65% for SFT.
  • Historical advantages: Removing historical advantages causes slower improvement and instability, while T-SPIN reaches 42.56% at iteration 1 and maintains consistent task-level gains.The ablated variant plateaus at 41.64% at iteration 3 and shows declines on GSM8K, MATH, and IFEval.
  • Reference-free reward function: T-SPIN outperforms the reference-policy variant across all iterations and is relatively insensitive to varied α and β values.The tested α and β values are 0.1, 0.3, 0.5, and 1.0.

5 Related work

Related work situates self-play fine-tuning within broader self-play learning and addresses the use of generated data to reduce reliance on costly human annotation.

  • Self-play fine-tuning for LLMs: Self-play learning trains models by competing against previous versions, with applications in multi-agent reinforcement learning and game-playing.Recent self-play fine-tuning adapts this framework to large language models.
  • Data generation in LLMs: LLM training depends heavily on high-quality annotated data, but limited availability and expensive expert annotation motivate alternative data-generation strategies.Recent work uses advanced language models to generate training data.

6 Conclusion and discussion

The paper identifies instability and training-generation misalignment in SPIN, proposes T-SPIN to address them, and reports stable, stronger performance with limited annotation; future work must handle generation cost and distribution shift.

  • Conclusion: SPIN suffers from unstable optimization as current advantages vanish and from misalignment between reference-based rewards and generation likelihoods.These issues motivate the design of T-SPIN.
  • Conclusion: T-SPIN combines historical advantages with an entropy constraint that enables reference-free rewards and aligns training with generation.Historical advantages compare newly generated responses with proto-synthetic responses from the initial policy.
  • Conclusion: T-SPIN maintains stable evolution, outperforms competing methods, and exceeds SFT while using 25% of its annotated data.The reported conclusion covers experiments across a variety of tasks.
  • Limitations and future work: T-SPIN currently regenerates synthetic responses for every prompt and keeps annotated data fixed, limiting efficiency and adaptation to non-stationary target distributions.The authors identify filtering valuable samples and online fine-tuning under distribution shift as future directions.

A.1 Proof of Proposition 1

The proof derives the optimal policy under an entropy-constrained objective and identifies it as a valid policy solution. The normalization term is independent of the policy.

  • The proof begins by deriving the optimal solution of the entropy-constrained objective.
  • The entropy is defined as the expected negative log-probability under the policy.
  • The normalization term Z(x) depends on the prompt x but is independent of policy πθ.
  • The derived πθ∗ is valid because its probabilities are nonnegative and it is identified as the optimal policy.

A.2 Proof of Theorem 1

The theorem proof derives gradients using expectations over annotated, current synthetic, and initial-policy responses, then specializes the result to logistic and exponential losses. The surrounding material introduces task metrics and experiment comparisons.

  • A.2 Proof of Theorem 1: The gradient derivation takes expectations over prompts, annotated responses, current-policy responses, and initial-policy responses.
  • A.2 Proof of Theorem 1: For logistic loss, ℓ(x) = −log σ(x) and ℓ′(x) = −σ(x), which are substituted into the general gradient expression.
  • Experiments: Table 3 specifies the number of few-shots and evaluation metrics for each task.
  • A.2 Proof of Theorem 1: For exponential loss, ℓ(x) = exp(−x) and ℓ′(x) = −exp(−x), yielding a corresponding specialized expression.
  • Experiments: The experiments include implementation details, task introductions, method comparisons, and ablation studies.

B.1 Implementation details

The experiments use the Alignment Handbook and Accelerate, train with RMSProp without weight decay, and run on eight H100 GPUs under fixed sequence, warmup, and batch-size settings.

  • The experiments use the Alignment Handbook as the codebase and Accelerate for distributed training.
  • Training uses RMSProp with no weight decay, a maximum sequence length of 2048, and warmup equal to 10% of total steps.
  • All experiments run on 8 H100 GPUs with 80GB memory, using batch size 8 per GPU.

B.2 Tasks

The evaluation covers reasoning, language understanding, instruction following, and related tasks, with task-specific few-shot and metric settings. Performance is compared across iterations on GSM8K and IFEval and summarized across ten tasks.

  • Tasks: GSM8K evaluates mathematical reasoning using grade-school word problems requiring 2 to 8 solution steps.
  • Comparisons: Table 4 compares T-SPIN, SPIN, and SFT, with SPIN and T-SPIN trained on 50k annotated examples and SFT on the complete dataset.
  • Comparisons: Figure 6 compares T-SPIN and SPIN on GSM8K and IFEval over 5 iterations and shows average scores across 10 tasks.
  • Tasks: WinoGrade is a large-scale benchmark designed to increase the scale and hardness of the original WSC task.
  • Tasks: BBH contains 23 challenging tasks spanning algorithmic reasoning, language understanding, world knowledge, and multi-step logic.
  • Tasks: IFEval evaluates instruction following with 500 prompts covering 25 verifiable instruction types.
  • Evaluation: All tasks are evaluated with the Language Model Evaluation Harness using the default settings specified in Table 3.

B.3 Experimental results on Mistral-7B

On Mistral-7B, T-SPIN improves across iterations, surpasses SFT by iteration 3, and remains more stable than SPIN. SPIN instead shows degradation in later iterations.

  • T-SPIN achieves a notable performance improvement over iterations and outperforms SFT at iteration 3.The comparison uses 10 different tasks and fewer annotated samples for T-SPIN than SFT.
  • T-SPIN exhibits better performances and stable improvements than SPIN across iterations.SPIN suffers performance degradations, whereas T-SPIN maintains stable refinement.
  • At iteration 3, SPIN degrades relative to iteration 2, while T-SPIN maintains steady improvements throughout the iterations.The comparison is shown for GSM8K, IFEval, and average score.
  • Figure 7 tracks annotated-sample and synthetic-sample log-likelihoods and their gap during training at iterations 0, 1, and 2.The gap is defined as log πθ(y|x) − log πθ(y′|x).

C Case study

The case study illustrates T-SPIN’s response evolution across iterations, while the accompanying implementation computes current and historical advantages for its combined loss.

  • Case study: The case study presents annotated and synthetic responses generated at each T-SPIN iteration.Examples are provided to illustrate the model’s progressive evolution.
  • Case study: Across iterations 1–4, the response develops from explicit questions and details toward a more generalized gratitude letter.Later responses use placeholders such as [Organization Name] and [insert type of disaster].
  • Implementation: The T-SPIN loss uses current advantage between real and synthetic samples and historical advantage between synthetic and proto-synthetic samples.The advantages are computed as policy_real_logps − policy_generated_logps and policy_generated_logps − policy_proto_logps.
  • Implementation: The loss scales both advantages by alpha, applies negative log-sigmoid losses, and combines them as current_loss + beta * history_loss.Alpha is the regularization coefficient and beta is the trade-off parameter.
Loading 2601.08198v1…