Language Models as Continuous Self-Evolving Data Engineers

1 Introduction

• Acquiring high-quality data is increasingly challenging as available sources are depleted and human expert annotation is costly and time-consuming.The introduction also notes continuing demand for new high-quality data.Source: §1 · p. 1
• Prior approaches either depend on external-model supervision or do not cover the full post-training data-construction lifecycle.The introduction contrasts teacher-generated synthetic data with iterative self-training approaches.Source: §1 · p. 2
• For lower-quality seed data, LANCE generates related instructions and responses; for higher-quality data, it creates flawed responses to form preference pairs.New data is filtered and reviewed before use in self-training.Source: §1 · p. 2
• Across varied tasks, iterative processing on a small dataset raised average performance while individual evaluation metrics remained stable or improved.The reported findings support continuous improvement through autonomous data processing.Source: §1 · p. 2
• LANCE autonomously generates, cleans, reviews, and preference-annotates training data, reducing the time and expense of post-training data preparation.The introduction describes an end-to-end process intended to reduce human involvement and reliance on external models.Source: §1 · p. 2

2.1 Post-Training

• Post-training commonly uses supervised fine-tuning and preference learning, both of which depend heavily on human-annotated data.These methods support instruction-following, alignment with preferences, and skill enhancement.Source: §2.1 · p. 2

2.2 Data Augmentation for LLMs

• Data augmentation approaches include teacher-model generation, human-LLM annotation, and reasoning-trace collection, but may depend on external models, human effort, or ground-truth tasks.Such dependencies can increase costs and limit scalability.Source: §2.2 · pp. 2–3

2.3 Self-Evolving LLMs

• Self-evolving approaches use models to generate instruction data, act as reward models, or self-play, while LANCE constructs targeted data from seed examples without large-scale given data or external reward models.The passage characterizes some prior approaches as requiring substantial input or being limited by data distribution.Source: §2.3 · p. 3

3.1.1 Supervised Fine-Tuning

• SFT trains on instruction-response pairs by minimizing negative log-likelihood, with the loss minimized when the model matches the dataset’s response distribution given each instruction.Source: §3.1.1 · p. 3

3.1.2 Reinforcement Learning from Human Feedback

• DPO uses preferred and dispreferred responses to minimize a preference-driven likelihood-ratio loss without requiring a reward model.The reference model is the initial SFT model, and β controls deviation from that reference policy.Source: §3.1.2 · p. 4

3.2 Initialization

• The seed data mixes a small labeled dataset for sampling new data with a review dataset containing instruction-response prompts, rationales, and scores.An initial model is SFT-trained on this seed data to establish instruction-following and review capabilities.Source: §3.2 · p. 4

3.3.1 Review Seed data

• The model reviews seed examples across tasks and topics to identify data deficiencies, using scores and rationales for each review.Review scores are extracted and averaged for subsequent decisions.Source: §3.3.1 · p. 4

3.3.2 Reward-Based Generation

• For examples below the review threshold, the model generates new instructions and responses; examples meeting the threshold can seed intentionally flawed responses for preference pairs.The threshold is typically set to 7.0; generated instructions explore the original examples’ topics, and response pairs are sorted by reward values.Source: §3.3.2 · pp. 4–5

3.3.3 Data Cleaning and Annotation

• Generated data is filtered by length and ROUGE-L similarity, and reward-qualified instruction data and preference pairs are added to their respective training datasets.Preference pairs designate preferred and dispreferred responses by comparing their reward values.Source: §3.3.3 · p. 5

3.4 Language Models as Continuous Self-Evolving Data Engineers

• Each iteration generates preference and supervised datasets, applies SFT and then DPO, and uses the resulting model in the next round.The process starts with SFT on seed data and repeats through the final model.Source: §3.4 · p. 5

4.1 Experimental Setup

• Experiments use Qwen2-7B and Qwen2-7B-Instruct with seed data from UltraChat and OpenAssistant Conversations, and evaluate reasoning, knowledge, and mathematical benchmarks.The seed set includes 3,184 UltraChat and 5,632 OpenAssistant examples; Figure 3 tracks average scores by iteration and compares against a 50k-example non-iterative Self-Instruct baseline.Source: §4.1 · pp. 5–7, Figure 3 · p. 7

4.2 LANCE Improves Benchmark Performance

• 3.64 average-score improvement for Qwen2-7B and 1.75 for Qwen2-7B-Instruct over SFT; LANCE often ranks first or second across benchmarks.Table 1 reports methods at their optimal iteration rounds; other abilities generally improve or remain largely unchanged.Source: Table 1 · p. 6, §4.2 · p. 7
• LANCE improves over iterations, while SPIN declines after its first round and I-SHEEP remains 1.18 points below LANCE on Qwen2-7B.Qwen2-7B-Instruct fluctuates before reaching its highest performance at the final iteration; the authors suggest multiple rounds may generate sufficiently high-quality data for further gains.Source: §4.2 · p. 7

4.4 Ablation Studies

• Without DPO-related components, gains are slower and performance declines after iteration four; without SFT-related components, progress becomes unstable.The authors characterize SFT as providing stability and DPO as driving faster, more sustainable improvement.Source: Table 3 · p. 8, §4.4 · p. 8

4.3 LANCE Improves Math Skills

• On Qwen2-7B, LANCE gains 6.38 accuracy on MATH and 3.60 on Olympiad Bench, outperforming SPIN’s 1.60 Olympiad Bench gain.LANCE also shows cross-lingual transfer on MGSM despite English-only training and improves math reasoning from generic seed data; the authors suggest deliberate reasoning during generation and review may help with complex problems.Source: §4.3 · p. 8

• LANCE enables LLMs to autonomously generate, clean, review, and annotate data with preference information, reducing post-training construction time and cost.The conclusion presents this as a continuous self-evolving data-engineering paradigm.Source: §5 · p. 8
• LANCE improves performance across tasks and works both for post-training pretrained models and for further post-training fully trained models.The conclusion says it outperforms other self-evolution methods.Source: §5 · p. 8
• LANCE enhances mathematical capabilities even when trained on general-purpose datasets, while aligning generated data with human preferences and reducing resource requirements for high-quality data creation.Source: §5 · pp. 8–9

• LANCE substantially improves mathematical reasoning, but improvement on knowledge-dependent tasks remains limited.Without external supervision signals, self-evolved data cannot introduce knowledge beyond the model’s existing capacity.Source: §6 · p. 9
• Enhancing strongly knowledge-dependent abilities remains an open challenge; the authors consider self-evolution more suitable for weakly knowledge-dependent capabilities.Source: §6 · p. 9
• Multiple rounds of sampling, reviewing, and annotation introduce significant computational overhead, despite the framework using a small seed dataset.The authors identify this cost as needing further optimization for scalability and practical use in resource-constrained environments.Source: §6 · p. 9

7 Ethic Statement

• The datasets and models used are publicly available and adhere to their respective open-source licenses.The authors restrict their use to academic research purposes.Source: §7 · p. 9
• Misuse of the methodology could potentially generate unethical data, a concern shared with other data-generation methods.The authors identify bias and ethical challenges in AI-generated content as ongoing topics for community discussion and research.Source: §7 · p. 9

A Data Distribution Visualization

• The visualization uses embeddings from 1,000 examples each of the seed, synthetic SFT, and DPO datasets, reduced with t-SNE.The embeddings were extracted with stella_en_400M_v5.Source: A Data Distribution Visualization · p. 12, Figure 4 · p. 12
• LANCE-generated synthetic data encompasses the seed distribution and explores new regions in the embedding space.The authors say this broader distribution is valuable for model generalization.Source: A Data Distribution Visualization · p. 12
• The SFT dataset has the widest distribution range among the three datasets, which the authors attribute to new instructions introducing diversity and complexity.They suggest instruction generation plays a critical role in data diversity and coverage.Source: A Data Distribution Visualization · p. 12

B Evaluation Setting

• The evaluation settings table records few-shot example counts, evaluation-configuration versions, and the frameworks used for each benchmark.Shot counts vary by benchmark, with examples including 0, 4, or 6; tools include LM Evaluation Harness and OpenCompass.Source: B Evaluation Setting · p. 12
• The listed configuration versions and evaluation tools are intended to support reproducibility and consistency across benchmark assessments.Source: B Evaluation Setting · p. 12

C Hyperparameter Settings

• Sampling uses threshold V = 7.0 and sample size K = 4; SFT and DPO each train for one epoch with different learning rates and batch configurations.Sampling uses top-p 0.9, temperature 0.7, and token limits of 10–4096; SFT uses learning rate 3e-5, batch size 2, and gradient accumulation 2, while DPO uses 5e-6, batch size 1, gradient accumulation 8, and β = 0.2. Experiments used two NVIDIA RTX A6000 GPUs with 48GB VRAM each.Source: C Hyperparameter Settings · pp. 12–13, Table 5 · p. 13

D Iteration details

• Only LANCE continuously improves Qwen2-7B across iterations and sustains improvement over multiple iterations on Qwen2-7B-Instruct, reaching its final-iteration peak after a fluctuation.I-SHEEP rises in the first two iterations, declines in the third, and rebounds to its peak in the fourth, but remains behind LANCE; other methods on Qwen2-7B-Instruct progress only in the first iteration.Source: D Iteration details · p. 13

E Model Evolution Steps

• Without DPO-related components, Qwen2-7B shows only slow and unstable iterative improvement; the authors emphasize the complete pipeline for robust, consistent gains.For Qwen2-7B-Instruct, performance falls in iterations two and three before rebounding notably in iteration four.Source: E Model Evolution Steps · p. 13

F Ablation Details

• Removing SFT-related components significantly degrades GSM8k performance.The authors attribute this likely to novel instructions generated during SFT data construction, which enhance mathematical and logical reasoning.Source: F Ablation Details · p. 13
• An incomplete LANCE pipeline leads to unstable performance improvements or even declines.Source: F Ablation Details · p. 13
• DPO data construction adds no new instructions, which may explain its limited impact on mathematical reasoning.Source: F Ablation Details · p. 13

G Case Study

• Table 7 reports that LANCE consistently gains performance across iterations and outperforms other baselines.Source: Table 7 · p. 14
• The SFT example replaces a disconnected recipe-to-wine transition with a Thai curry instruction linking wine pairing to the dish’s flavors.Source: G Case Study · p. 14
• Generated SFT data adds rationales connecting Riesling’s sweetness and acidity, and Gewürztraminer’s aromatic intensity, to the dish’s flavors.Source: G Case Study · pp. 15–16
• Figure 6 illustrates DPO pair construction: LANCE keeps the high-quality seed response as preferred and generates an intentionally suboptimal dispreferred response.Source: G Case Study · pp. 16–17, Figure 6 · p. 17