PaperlayerView exact revision on arXiv

Source-linked AI summary

Condor: Enhance LLM Alignment with Knowledge-Driven Data Synthesis and Refinement

Maosong Cao, Taolin Zhang, Mo Li, Chuyu Zhang, Yunxin Liu, Haodong Duan, Songyang Zhang, Kai Chen

arXiv:2501.12273v1cs.CLcs.AI

TL;DR

As high-quality SFT data becomes harder to obtain, Condor generates training examples from world-knowledge tags and refines responses through self-reflection. The authors report improved subjective chat capabilities and performance surpassing many existing methods and official RLHF models using a small amount of synthetic data.

  • Problem

    High-quality human-annotated SFT data is becoming depleted, while synthetic data quality varies and existing synthesis approaches can lack prompt diversity or reliable quality.

    Source: §1 · pp. 1–2

  • Method

    Condor synthesizes QA data from a World Knowledge Tree and uses Self-Reflection Refinement to improve generated responses.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2.1 · p. 4

  • Results

    Condor-generated data significantly improves subjective chat capabilities across models, and fine-tuned models surpass many existing methods and official RLHF models using a small amount of synthetic data.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The authors report that models can achieve self-iteration through synthetic data and identify further scaling potential for synthetic data.

    Source: §1 · p. 2, §6 · p. 9

    The authors identify multi-round iterative synthesis, greater synthetic-data diversity, and potential hallucinations as issues requiring further exploration.

    Source: §7 · p. 9

Abstract

from arXiv · show

The quality of Supervised Fine-Tuning (SFT) data plays a critical role in enhancing the conversational capabilities of Large Language Models (LLMs). However, as LLMs become more advanced, the availability of high-quality human-annotated SFT data has become a significant bottleneck, necessitating a greater reliance on synthetic training data. In this work, we introduce Condor, a novel two-stage synthetic data generation framework that incorporates World Knowledge Tree and Self-Reflection Refinement to produce high-quality SFT data at scale. Our experimental results demonstrate that a base model fine-tuned on only 20K Condor-generated samples achieves superior performance compared to counterparts. The additional refinement stage in Condor further enables iterative self-improvement for LLMs at various scales (up to 72B), validating the effectiveness of our approach. Furthermore, our investigation into the scaling for synthetic data in post-training reveals substantial unexplored potential for performance improvements, opening promising avenues for future research.

1 Introduction

Condor addresses the difficulty of obtaining high-quality SFT data by synthesizing data from world-knowledge tags and refining responses. The authors report improved subjective chat capabilities across models and describe further refinement and scaling investigations.

1 Introduction

  • High-quality SFT data is increasingly difficult to obtain as existing human-annotated data depletes and synthetic data quality varies.Prior synthesis approaches also face seed-prompt dependence, limited diversity, and uncertain quality.Source: §1 · pp. 1–2
  • Condor-generated data significantly improves subjective chat capabilities across model families.The comparison uses CompassJudger-1-32B.Source: Figure 1 · p. 1, §1 · p. 2
  • Condor combines a World Knowledge Tree with Self-Reflection Refinement to generate diverse questions and higher-quality SFT responses.The authors describe the pipeline as scalable and able to incorporate up-to-date world-knowledge tags.Source: §1 · p. 2
  • The authors identify Condor Void, Condor Refine, and investigations of dataset scaling and self-iteration as contributions.Condor Void uses knowledge inspiration for diverse questions; Condor Refine iteratively revises responses.Source: §1 · p. 2

2 Related Work

Prior synthetic-data and self-iteration methods use seed datasets, specialized models, or task-specific feedback. Condor instead uses tags for data generation and the model’s own capabilities for broader self-improvement.

2.1 Synthesizing Instruction Tuning Data

  • Earlier synthesis methods rely on seed data or model-specific chat templates, while agent-based generation can require multiple concurrent models.WizardLM expands an initial question set, Self-Reward builds on seed data, and Magpie uses chat templates.Source: §2.1 · p. 2
  • Condor uses tags rather than a seed dataset and a single model for generation, supporting data diversity, scalability, and lower costs.Source: §2.1 · p. 2

2.2 LLM Self-Iteration

  • Some self-iteration methods remain close to seed instructions or depend on reward models, while others target specific domains.Condor claims self-improvement across human-preference datasets using the model’s own capabilities.Source: §2.2 · pp. 2–3

3 Methodology

Condor uses one model to synthesize QA data from a structured knowledge tree and then refine its responses. Task and difficulty variation broadens the questions, while self-generated critiques guide response revision.

3.1 Pipeline Overview

  • The pipeline uses one model to synthesize an initial QA dataset, DV, then iteratively optimize responses to produce DR.The two stages are data synthesis and data refinement.Source: §3.1 · p. 3

3.2 Condor Void

  • Condor Void builds synthetic data from world knowledge using a World Knowledge Tree and QA-pair generation.The resulting dataset is called DV.Source: §3.2 · p. 3

3.2.1 World Knowledge Tree

  • The World Knowledge Tree organizes broad root tags and detailed leaf tags into a hierarchy, augmented with trending-topic tags.The tree is designed to update over time and contains over 8,400 tags.Source: §3.2.1 · p. 4

3.2.2 Q&A Pair Generation

  • Question generation varies across seven chat tasks and three difficulty levels, yielding approximately 200k Condor Void QA pairs.The model generates questions under tag themes, encouraging knowledge extrapolation.Source: §3.2.2 · p. 4

3.3 Condor Refine

  • Condor Refine prompts the model to identify response strengths and weaknesses, then use its suggestions to revise answers while preserving effective elements.The refined dataset contains approximately 200,000 examples from DV.Source: §3.3 · p. 4

4 Experiments and Results

Condor Refine improves human-preference performance while retaining knowledge-QA capability, and comparisons and scaling experiments report gains across methods, models, and data settings. The experiments also examine model size, task difficulty, data scale, and self-iteration.

4.1 Training and Evaluation Settings

  • Experiments use eight human-preference benchmarks, report average normalized scores, and assess knowledge-QA on ground-truth-based datasets.GPT-4o judges the main results; CompassJudger-1-32B is used for ablation and scaling experiments.Source: §4.1 · p. 4

4.2 Data Synthesis Settings

  • The main experiments train on 200K samples synthesized by Qwen2.5-72B; additional 200K-sample sets from Qwen2.5-7B support comparisons and self-interaction experiments.A single model is used throughout each data-synthesis and refinement pipeline.Source: §4.2 · p. 5

4.3 Main Results

  • Condor-trained models surpass the official model on almost all human-preference benchmarks, achieve comparable or better knowledge-QA results, and outperform other synthesis methods.The knowledge-based benchmarks were not specifically included in Condor’s synthetic data; the synthesis-method comparison used Qwen2.5-7B-Base and GPT-4o judging.Source: §4.3 · pp. 5–6, Table 1 · p. 5, Table 2 · p. 5

4.4 Ablation Study

  • Training on Condor Refine improves subjective evaluation across Qwen, InternLM, and Llama; the reported Qwen2.5-7B score rises from 56.9% to 63.3%.The reported improvements for InternLM2 and Llama are about 10%; the 14B and 32B models improve more than the 7B variant.Source: §4.4 · p. 6
  • Combining all three task-difficulty levels yields a 0.34% average improvement, and harder tasks contribute to greater performance improvements.The study compares training data containing tasks of varying difficulty.Source: §4.4 · p. 6

4.5 Scaling of Condor Data

  • As training data increases, the average score across eight benchmarks consistently improves; adding tasks has a greater influence than adding WKT tags.The model retains a high proportion of performance when trained on only a small data proportion, despite fluctuations on some benchmarks.Source: §4.5 · p. 7, Figure 4 · p. 8, Figure 5 · p. 8

4.6 Self Iteration

  • Both Qwen2.5-7B and 72B achieve self-improvement over RLHF models, with the 72B model’s improvement slightly greater.Scores are reported across eight human-preference benchmarks.Source: §4.6 · p. 8

5 Analysis and Discussion

Condor training improves all evaluated sub-capabilities, with the largest reported increases in Creation, QA, and Chat. Its questions have a similar or broader distribution than Magpie’s, while the analysis attributes most knowledge acquisition to pretraining.

5.1 Which capability is improved the most?

  • Condor improves all evaluated sub-capabilities, with the largest score increases in Creation, QA, and Chat.These dimensions align with core aspects of human-preference capability.Source: §5.1 · p. 8, Figure 6 · p. 8

5.2 How large is the synthetic data coverage?

  • A t-SNE comparison of 200K questions per dataset finds Condor’s distribution similar to or broader than Magpie’s, indicating question diversity.The authors note that Magpie’s method can feed back a model’s own training data, making its distribution somewhat close to that model’s original SFT data.Source: §5.2 · p. 8, Figure 7 · p. 9

5.3 What does SFT actually enhance?

  • The analysis concludes that SFT has minimal impact on intrinsic knowledge; Condor primarily refines the use of existing knowledge to answer questions.It attributes most knowledge acquisition to pretraining and states that this refinement can be achieved with relatively little high-quality data.Source: §5.3 · p. 9

6 Conclusion

Condor uses synthetic data for SFT and explores scaling and self-iteration. The authors report that fine-tuned models surpass many existing methods and official RLHF models using a small amount of this data.

6 Conclusion

  • Condor-generated data lets fine-tuned models surpass many existing methods and official RLHF models using a small amount of synthetic data.Source: §6 · p. 9
  • The paper explores scaling synthetic data and self-iteration, reporting that models can achieve self-iteration through synthetic data.Source: §6 · p. 9
  • Condor is a two-stage data synthesis engine for generating high-quality supervised fine-tuning data.Source: §6 · p. 9

7 Limitations

The authors identify open questions in iterative synthetic-data generation and diversity, and flag LLM-generated hallucinations as a potential risk.

7 Limitations

  • Multi-round iterative synthetic data and ways to further increase synthetic-data diversity remain to be explored.Source: §7 · p. 9
  • Hallucinations produced by LLMs in synthetic data are a potential risk that the authors say needs further attention.Source: §7 · p. 9
  • The authors state that addressing these issues could enhance synthetic-data quality.Source: §7 · p. 9

A.1 Detailed Calculation Method for the Subjective Evaluation Scores.

The appendix maps subjective-chat benchmark scores with different ranges onto a common 0–100 scale, then averages the mapped scores.

A.1 Detailed Calculation Method for the Subjective Evaluation Scores

  • Scores from CompassArena, AlpacaEvalv2, and ArenaHard already use a 0–100 range and require no mapping.Source: §A.1 · p. 13
  • FoFo and Followbench scores are multiplied by 100, while AlignBenchv1.1 and MTBench101 scores are multiplied by 10 to map them onto 0–100.Source: §A.1 · p. 13
  • WildBench scores are shifted by adding 100 and then divided by 2 before averaging.Source: §A.1 · p. 13
  • The final Average score is the mean of the dataset scores after mapping them to the 0–100 range.Source: §A.1 · p. 13

A.2 Detailed Information for Sub-capabilities Improvement of Condor

This appendix describes how benchmark subscores are aggregated by capability and documents elements of Condor’s training-data construction, including task coverage and prompts for generating and refining examples.

A.2 Detailed Information for Sub-capabilities Improvement of Condor

  • Capability scores are formed by aggregating dataset-provided subscores for each capability dimension.Source: §A.2 · p. 13
  • Table 10 reports performance by tag count and notes that the tag pool contains 4,249 Chinese and 4,296 English tags, sampled proportionally for training.Source: Table 10 · p. 15
  • Table 11 reports performance as tasks are added cumulatively, beginning with role-playing and ending with creation.Source: Table 11 · p. 15
  • The question-synthesis prompt requests exactly three context-rich questions in a specified domain and theme, with distinct easy, medium, and hard levels.Source: §A.2 · p. 16
  • The response-critic prompt asks for strengths, weaknesses, and suggestions in a prescribed format.Source: §A.2 · p. 16
  • The appendix groups chat capability into seven tasks, including role-playing, daily chat, domain QA, material processing, format control, views, and creation.Source: §A.2 · p. 20
  • The remaining listed tasks are offering views and creating new content such as articles, stories, poetry, and designs.Source: §A.2 · p. 20

Related papers

Loading 2501.12273v1…