PaperlayerView exact revision on arXiv

Source-linked AI summary

Aligning Large Language Models via Fully Self-Synthetic Data

Shangjian Yin, Zhepei Wei, Xinyu Zhu, Wei-Lin Chen, Yu Meng

arXiv:2510.06652v1cs.CL

TL;DR

Human-feedback alignment relies on costly demonstrations and preference labels, while AI-feedback approaches can still depend on external models or datasets. SAO generates prompts, responses, and preferences itself through persona role-play and self-judgment. The authors report chat-benchmark gains, preserved or improved objective-task performance, and promising scaling results.

  • Problem

    Human demonstrations and preference labels are costly to collect, and existing self-improvement or AI-feedback approaches can retain dependencies on external models, data, or reward systems.

    Source: §1 · p. 1

  • Method

    SAO has the model generate persona-based prompts and responses, self-rank responses, and optimize preferences using those rankings.

    Source: §1 · p. 1

  • Results

    SAO improves subjective chat benchmarks while maintaining or improving objective performance; Gemma-2-9B-it scores 74.41 versus its 74.28 baseline on the Open LLM Leaderboard.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The authors present SAO as a practical self-improvement approach with promising scaling through larger synthetic datasets and iterative optimization.

    Source: §1 · p. 2, §5 · p. 8

    Experiments are limited to models below 10 billion parameters because of resource constraints and demonstrate effectiveness with simple prompt templates.

    Source: §6 · pp. 8–9

Abstract

from arXiv · show

Traditional reinforcement learning from human feedback (RLHF) for large language models (LLMs) relies on expensive human-annotated datasets, while Reinforcement Learning from AI Feedback (RLAIF) also incurs significant costs, requiring the collection of diverse prompts and corresponding responses, often necessitating external reward models or proprietary models like GPT-4 to annotate preference pairs. In this work, we introduce Self-Alignment Optimization (SAO), a fully self-synthetic framework for LLM alignment, where all training data, including prompts (i.e., user queries), responses, and preferences, are generated by the model itself. Specifically, SAO first instructs the LLM to engage in persona role-play and generate diverse prompts and responses, which are then self-evaluated for preference optimization. Extensive experiments demonstrate that SAO effectively enhances the model's chat capabilities on standard benchmarks like AlpacaEval~2.0, while maintaining strong performance on downstream objective tasks (e.g., question-answering, math reasoning). Our work provides a practical solution for self-improvement in aligning LLMs, and the code for reproducing our results is available at: https://github.com/SJY8460/SAO.

1 Introduction

SAO addresses the cost and external dependencies of preference-data alignment by generating prompts, responses, and preference judgments within the model. The paper reports improved chat benchmark results, maintained objective-task performance, and promising scaling and self-judging findings.

1 Introduction

  • Human demonstrations and preference labels are costly to collect, while RLAIF can still require proprietary models, specialized reward models, or data filtering.Prior self-improvement methods also retain external dependencies such as human-labeled templates, existing prompt datasets, or external reward models.Source: §1 · p. 1
  • SAO generates persona-based prompts, paired responses, and self-judged rankings, then uses preference optimization to fine-tune the model.This fully self-synthetic approach eliminates costly training-data collection and annotation.Source: §1 · p. 1
  • Across chat benchmarks, SAO is competitive with GPT-4 and strong open-weight models; Gemma-2-9B-it gains 18.1% LC and 27.9% WR on AlpacaEval 2.0.On MT-Bench, its average rises from 8.41 to 8.66; on Arena-Hard, its WR rises from 40.8% to 54.3%.Source: §1 · pp. 1–2
  • On the Open LLM Leaderboard, Gemma-2-9B-it averages 74.41 across benchmarks versus its 74.28 baseline, without further training.The passage contrasts this with 70.38 for Gemma-2-9B-it trained on UltraFeedback using the same training algorithm.Source: §1 · p. 2
  • Self-synthesized prompts improve WR by 16.46% over UltraFeedback prompts under the same SAO steps, and the authors attribute SAO's effectiveness largely to strong self-judging.The reported self-judging ability surpasses ArmoRM-Llama3-8B-v0.1 and GPT-4-level feedback within the SAO framework.Source: §1 · p. 2

2 Related Work

The related work situates SAO among synthetic-data and persona-based approaches, then distinguishes its integration of LLM judging with instruction-following ability.

2.1 Synthetic Data for LLMs

  • Unlike synthetic-data approaches that typically use more advanced models, this work generates training data directly from the target model.The authors frame this as streamlining synthesis and reducing dependence on external resources.Source: §2.1 · p. 2

2.2 Persona Roleplay

  • Persona role-play is presented as decompressing distributed world knowledge into diverse texts, and Persona Hub supplies personas without requiring seed corpora.The work leverages Persona Hub personas in customized synthesis prompts.Source: §2.2 · p. 2

2.3 LLM-as-a-Judge

  • SAO integrates LLM-as-a-Judge preference optimization with general instruction-following skills, distinguishing it from work focused on improving judge capability.The related-work passage describes judging as also used for evaluation, reward-model training, and data curation.Source: §2.3 · pp. 2–3

3 Self-Alignment Optimization

SAO builds a preference-optimization dataset from persona-generated prompts and paired responses, which the model ranks itself. It then optimizes the model using those self-generated preferences, with SimPO selected for datasets that frequently contain longer responses.

3.1 Diverse Prompt Generation

  • Each persona produces one prompt from a role template, with persona resources randomly sampled from Persona-Hub.The one-question-per-persona constraint is intended to ensure prompt diversity.Source: §3.1 · p. 3

3.2 Pairwise Response Generation

  • For each generated prompt, the model produces two responses for comparative evaluation.Generating additional responses could potentially improve performance, but would increase computational costs and evaluation time.Source: §3.2 · p. 3

3.3 Self-Judgment

  • The model self-judges the paired responses by ranking their relevance and quality against the prompt.The ranking yields superior and inferior responses for preference optimization.Source: §3.3 · p. 3

3.4 Dataset Construction

  • The synthetic dataset aggregates generated prompts and ranked responses across personas for preference optimization.It lets the model learn from its own generated and ranked responses across diverse personas.Source: §3.4 · p. 3

3.5 Preference Optimization

  • SimPO is used for preference optimization because its length normalization suits datasets that frequently contain longer responses.Its length-normalization technique captures token-level information.Source: §3.5 · p. 3
  • The optimization objective uses self-generated preferences to guide the model toward desired outcomes.SimPO includes a target reward margin γ > 0 to enforce a minimum reward difference between winning and losing responses.Source: §3.5 · pp. 3–4

4 Experiments

SAO improves chat-benchmark performance while generally preserving or slightly improving downstream objective-task performance. Analyses find gains from dataset scaling and iterative optimization, and identify self-feedback and judging quality as important factors.

4.1 Experiment Setting

  • Experiments use Gemma-9B-it as the base, apply similar fine-tuning to Llama-3-8B-Instruct, and by default use 60k synthetic samples, one epoch, global batch size 128, and four A100 GPUs.Source: §4.1 · p. 4

4.2 Evaluation Metrics and Baselines

  • Evaluation covers AlpacaEval 2.0, Arena-Hard, MT-Bench, and six Open LLM Leaderboard datasets, with vanilla, externally trained, and self-improvement baselines.AlpacaEval uses GPT-4-Turbo or Qwen2-72B-Instruct as annotators; Arena-Hard and MT-Bench use GPT-4 evaluation.Source: §4.2 · p. 4

4.3 Main Results

  • Gemma-2-9B-it-SAO reaches 69.2% LC and 66.0% WR on AlpacaEval 2.0 with GPT-4-Turbo-1106, while its MT-Bench score rises from 8.41 to 8.66 and Arena-Hard WR from 40.8% to 54.3%.Llama-3-Instruct-8B-SAO also improves on AlpacaEval 2.0 and Arena-Hard, while maintaining its 7.84 MT-Bench average score.Source: §4.3 · pp. 4–5, Table 1 · p. 5, Table 2 · p. 5
  • Gemma-2-9B-it-SAO scores 74.41 versus 74.28 for its baseline across Open LLM Leaderboard benchmarks, while externally labeled SimPO models show overall declines.Llama-3-8B-Instruct-SAO scores 68.20 versus 68.19 for its baseline; the cited comparison reports decreases for Gemma and Llama SimPO models.Source: §4.3 · p. 5, Table 3 · p. 6

4.4 Analysis and Ablations

  • 74.06% WR at 10k examples rises from the vanilla model’s 39.25%, while performance stabilizes around 72% at larger dataset sizes.LC reaches 76.02% with 60k examples; the analysis reports diminishing returns beyond 10k.Source: Figure 1 · p. 6, §4.4 · p. 6
  • 86.46% WR after the second iteration exceeds 74.06% after the first, while LC increases to 74.13%.In iterative optimization, the improved model generates new preference data for further training.Source: Figure 2 · p. 6, §4.4 · p. 6
  • SimPO reaches 74.04% WR on the generated dataset, and persona role-play yields 72.30% WR with 0.7% prompt repetition versus 62.50% and 45.7% without it.The synthetic dataset tends toward prompts under 200 words and responses mainly 400–800 words, compared with UltraFeedback.Source: Table 4 · p. 6, §4.4 · p. 7, Table 5 · p. 7
  • Self-feedback achieves 74.04% WR versus 52.80% with GPT-4o feedback, and Gemma judging outperforms LLaMA judging across generation configurations.Self-feedback gains persist across three independent evaluators, while LLaMA-feedback WR ranges from 32.67% to 38.39%.Source: Table 6 · p. 7, §4.4 · pp. 7–8, Table 8 · p. 8

5 Conclusion

SAO aligns LLMs using self-generated synthetic data and preference judgments. Experiments report gains on subjective benchmarks, maintained or improved objective-task performance, and promising scaling through larger datasets and iterative optimization.

5 Conclusion

  • SAO aligns LLMs with human preferences using only self-generated synthetic data.The model generates and judges instruction-preference pairs, avoiding costly external annotations.Source: §5 · p. 8
  • SAO achieves substantial gains on subjective evaluation benchmarks while maintaining or improving downstream objective-task performance.Source: §5 · p. 8
  • Dataset-size scaling and iterative optimization show promising scalability, with improved models generating higher-quality preference data in subsequent rounds.Source: §5 · p. 8

6 Limitations

The study's reported limitations concern model scale and prompt design: experiments use models below 10 billion parameters, and the approach has been tested with simple prompt templates.

6 Limitations

  • The experiments are limited to models smaller than 10 billion parameters because of resource limitations.Source: §6 · p. 8
  • The authors anticipate scaling SAO to larger models could yield greater performance enhancements.Source: §6 · pp. 8–9
  • Although SAO has demonstrated effectiveness with simple prompt templates, more complex templates remain to be investigated.Source: §6 · p. 9

A Impact Statements

The impact statement presents SAO as a way to reduce the burden of external supervision while cautioning that fully self-synthesized datasets may contain inaccuracies or hallucinations.

A Impact Statements

B Experiments on Small-Sized LLMs

The small-model experiments show different outcomes across the two reported backbones: SAO degrades Llama-3.2-Instruct-3B but improves Gemma-2-2B-it. The authors hypothesize that weaker models may lack sufficient judgment ability for self-alignment.

B Experiments on Small-Sized LLMs

C Repetition in Generated Prompts Under Random Generation

Table 11 identifies the five most repeated prompts in a 10k synthesized dataset generated without persona role-play. Figure 4 shows the persona-instruction and pair-wise response-ranking prompts.

C Repetition in Generated Prompts Under Random Generation

  • Table 11 presents the top five repeated prompts from a 10k synthesized dataset generated without persona role-play.The caption does not specify the prompts or their repetition counts.Source: Table 11 · p. 12

D Prompt Template

  • The supplied example is labeled “Persona Instruction Example.”Source: D Prompt Template · p. 13
  • The persona instruction asks the model to guess a prompt that a given persona might ask it to perform.The example persona is a political analyst specializing in El Salvador’s political landscape.Source: D Prompt Template · p. 13
  • Figure 4 shows an instruction for generating a prompt based on a specified persona and a separate prompt for ranking responses.The ranking criteria are modified from Shen et al. (2024).Source: Figure 4 · p. 13

E Synthesized Examples

Table 12 is identified as an example of the synthetic dataset.

E Synthesized Examples

E.1 Model Outputs Before and After SAO Training

Table 13 is identified as showing outputs from Gemma2-9B-it and Gemma2-9B-it-SAO.

E.1 Model Outputs Before and After SAO Training

  • Table 13 presents outputs from Gemma2-9B-it and Gemma2-9B-it-SAO.Source: Table 13 · p. 15
Loading 2510.06652v1…