PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Autonomous Agentic Data Engineering for Model Specialization

Yujie Luo, Xiangyuan Ru, Jingsheng Zheng, Jingjing Wang, Yuqi Zhu, Jintian Zhang, Runnan Fang, Kewei Xu, Ye Liu, Zheng Wei, Jiang Bian, Zang Li, Shumin Deng

arXiv:2605.30407v2cs.CLcs.AIcs.IRcs.LG

TL;DR

LLMs often struggle to adapt to specialized domains, and existing curation methods rely on human-designed workflows. This paper evaluates autonomous end-to-end data engineering, finding that GPT-5.2 achieves a 57.29% average gain through iterative data optimization.

  • Problem

    LLMs often struggle with specialized tasks, while existing LLM-based data curation relies on human-designed workflows rather than autonomous end-to-end data engineering.

  • Method

    The paper formalizes Agentic Data Engineering as a closed-loop paradigm in which an LLM autonomously curates training data to specialize a fixed student model using feedback.

  • Results

    57.29% average gain: GPT-5.2 consistently achieves stronger and more stable specialization through iterative, feedback-driven data optimization across Science, Code, and Finance.

  • Takeaways & Limitations

    Agentic Data Engineering emerges as a measurable capability through which LLM agents can autonomously author data curricula for model specialization.

  • Takeaways & Limitations

    Evaluation focuses on QA tasks, while iterative agents require substantial computation and exhibit unavoidable run-to-run variance.

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated strong performance on general tasks, while often struggling to adapt to specialized domains without high-quality domain-specific data. Existing LLM-based data curation methods primarily rely on human-designed workflows, leaving it unexamined whether LLMs can autonomously execute an end-to-end data engineering pipeline for model specialization. We formalize Autonomous Agentic Data Engineering, a novel task designed to evaluate LLMs as autonomous data engineers that drive model specialization through end-to-end data curation. We frame data as an optimizable component and study agents that plan, generate, and iteratively optimize training data across multiple domains, guided by post-training performance improvement. Experiments show that autonomous LLM data engineers yield substantial gains, as GPT-5.2 constructs a training curriculum that improves a student model by 57.29%, entirely through iterative, agent-driven data adaptation. By illuminating both potential and bottlenecks, our study establishes autonomous data engineering as a measurable capability and charts a path toward agent-driven model specialization (Code will be released at https://github.com/zjunlp/DataAgent).

1 Introduction

Specialized-domain adaptation remains difficult when training data do not reflect downstream tasks, motivating autonomous agents that independently curate and optimize data for model specialization. The paper formalizes this capability, evaluates it across domains and agent settings, and provides an execution and performance-based evaluation environment.

  • Motivation: LLMs perform strongly on general tasks but often struggle to adapt when training data inadequately reflect specialized downstream tasks.This motivates improving domain-specific training data for specialized adaptation.
  • Motivation: Existing LLM-based methods use LLMs as data generators within human-designed workflows, while adapting handcrafted recipes to new domains requires extensive configuration.The paper positions modern LLM agents as an alternative because of advances in complex reasoning and code generation.
  • Task formulation: Autonomous Agentic Data Engineering tasks LLMs with independently completing the entire training-data curation pipeline, including strategy planning, domain specification, prompt design, and data synthesis.Teacher and student models are held fixed, and capability is evaluated by post-training improvement of the student model.
  • Experimental scope: The study evaluates mainstream LLMs across Science, Code, and Finance using One-Shot and closed-loop Iterative Agent settings, both from scratch and with initial seed data.The analysis also examines iterative optimization, data quality, and failure modes toward specialization.
  • Contributions: The paper introduces a controlled, budget-limited execution and evaluation environment covering the full data curation pipeline, with external feedback and performance-based evaluation.This environment makes end-to-end data engineering a measurable capability of LLM agents.

2 Agentic Data Engineering

Agentic Data Engineering is formalized as a closed-loop process in which an LLM autonomously curates teacher-generated data to specialize a fixed student model, with curation strategies optimized using post-training feedback. The framework evaluates agents under controlled inputs, budgets, standardized task interfaces, and private-set performance gains, including One-Shot and Iterative Agent settings.

  • Framework: Agentic Data Engineering casts data curation as an optimization problem in which an LLM searches for strategies that maximize the specialized student model’s post-training performance.The teacher and student models remain fixed across tasks, isolating the contribution of agent-driven data curation.
  • Task Definition: The agent receives task and dataset information, a teacher-call budget, a fixed student model, and standardized fine-tuning and inference interfaces, then outputs teacher-generated training data as submission.json.The generated code must create all instances through teacher-model API calls rather than writing them directly into the file.
  • Task Definition: Private-set performance gain measures end-to-end data engineering capability after the student is fine-tuned on the agent’s submission data.The environment enforces fixed teacher API-call and wall-clock budgets and provides standardized interfaces for teacher calls, student fine-tuning, and public-set evaluation.
  • Benchmark Construction: Tasks span Science, Code, and Finance reasoning domains and use deterministic rule-based evaluation with separate public and private test sets to support iterative feedback and final assessment.The private test set is hidden for final evaluation, while the public test set is visible for procedural feedback during iterative optimization.
  • Agent Settings: The study compares One-Shot generation, which produces a submission in one pass, with Iterative Agent optimization, which continuously improves data through environmental feedback.The Iterative Agent uses Draft, Debug, and Repair operations to implement strategies, fix execution errors, and address invalid submissions.

3 Experiments

Experiments evaluate autonomous data engineering by measuring student-model post-training gains and submission efficiency under training-based settings. Iterative optimization and seed data improve specialization, with GPT-5.2 reaching a 57.29% average relative gain in the from-scratch regime.

  • Evaluation Metrics: Student-model post-training performance gain directly measures agent effectiveness, with positive values indicating specialization and negative values indicating degradation.All tasks use their source benchmark’s official evaluation metric, evaluated by accuracy; absolute accuracy is also reported as a complementary measure.
  • Evaluation Metrics: MATS counts average trial attempts yielding submission.json files whose validated filtered submissions retain at least 1,000 instances.Results report MATS, where lower is better, and relative performance Gain (%) over the base Llama-3.1-8B-Instruct model, where higher is better.
  • Experimental Settings: Agents are evaluated both from scratch and with a seed pool of 1,000 raw questions guiding synthesis and exploration.Experiments use a 50,000-call teacher-API budget per task, with limits of 5,000 calls per attempt, 3 hours per execution, and 12 hours per run.
  • Main Results: 57.29% is GPT-5.2’s average relative gain with iterative optimization from scratch, up from 40.73% for the corresponding one-shot setting.Iterative agents consistently outperform one-shot agents, while a single one-shot error produced DeepSeek-V3.1’s -4.58% drop on Code.
  • Main Results: 57.65% is DeepSeek-V3.1’s gain with iterative optimization and seed data, compared with 12.50% in the one-shot from-scratch baseline.The results indicate that weaker models benefit more from sophisticated agent frameworks, while even the fragile one-shot from-scratch setting generally produces positive average gains.

4 Further Analysis

Further analysis shows that iterative agents improve scores through repeated data adaptation, with greedy selection stabilizing progress and diversity emerging as the primary optimization target. Fully autonomous data engineering can surpass human-designed pipelines, but remains limited by data-diversity gaps, submission failures, and weak safeguards against distribution and semantic errors.

  • Iterative Improvement: Public, private, and final scores rise across iterations, with substantial gains usually appearing within 8 to 15 iterations before performance plateaus.The plateau indicates diminishing returns as the agent approaches the boundaries of its data awareness and cognitive capacity.
  • Iterative Improvement: Greedy public-score selection retains the best historical submission, making the final performance curve more stable despite synthetic-data variance and occasional catastrophic regressions.Public and private performance are largely aligned across iterations, supporting public-score selection as a robustness mechanism.
  • Data Optimization: Iteration primarily improves instruction and response diversity, based on diagnostics covering instruction difficulty, instruction diversity, response quality, response diversity, and response perplexity.The analysis evaluates six intrinsic metrics using GPT-4o, embedding similarity, the Skywork reward model, and LLaMA-3.1-8B.
  • Human Involvement: GPT-5.2’s fully autonomous pipeline surpasses the human-designed DataFlow framework by adapting synthesis strategies to task domain, difficulty, and output format through environmental feedback.The feedback loop enables continuous improvement of data-curation strategies without relying on rigid human-designed logic.
  • Human Involvement: The from-scratch agent approaches the human baseline in Instruction Difficulty but performs worse in Instruction Diversity and Response Diversity.This contrast indicates that LLM-driven engineering can produce challenging data while remaining weaker at generating diverse data.
  • Failure Analysis: LLMs often fail valid submission generation because they lack quantity assurance, mishandle complex formatting, or over-engineer verbose pipelines, while specialization failures arise from distribution shift and naive rule-based augmentation.Insufficient Valid Samples commonly causes failure of the 1,000-instance check; Claude-4-Sonnet truncation reaches 52.63% in Code and 59.31% in Finance.
  • Failure Analysis: Overall, LLMs can guide end-to-end curation but lack robust post-generation safeguards for stringent quality assurance and reliable quantity control.Observed failures include overfitting from narrow topic allocation and syntactically broken noise from indiscriminate numerical perturbation.

5 Related Work

Prior work develops LLM agents that use reasoning, tools, feedback, memory, and accumulated experience to complete tasks, while data-centric research explores synthetic training data. This work instead studies agents whose primary objective is to produce and iteratively optimize data using downstream student-model feedback.

  • LLM Agent: LLM agents combine foundation-model reasoning with external tools and environmental feedback to complete tasks across domains such as data analysis and scientific discovery.
  • LLM Agent: A related research direction enables agents to improve through accumulated experience by structuring knowledge from past successes and failures.
  • LLM Agent: Unlike agents that consume data to complete tasks, this work treats producing and optimizing data as the agent’s primary objective.
  • Data Centric AI: Data-centric AI shifted from human-annotated data toward LLM-synthesized training data amid emerging data scarcity.
  • Data Centric AI: The study fixes the training pipeline, parameter settings, teacher model, and student model, isolating data curation as the sole variable optimized through student-model feedback.

6 Conclusion

The paper presents Agentic Data Engineering for Model Specialization as a closed-loop, end-to-end capability for LLM agents. Across Science, Code, and Finance, iterative agents produce stronger and more stable specialization, with GPT-5.2 achieving a 57.29% average gain.

  • Conclusion: LLM agents conduct end-to-end data engineering in a closed loop for model specialization.The study systematically analyzes Agentic Data Engineering by requiring agents to perform the full data-engineering process.
  • Conclusion: Across Science, Code, and Finance, iterative agents consistently yield stronger and more stable specialization.Feedback-driven iteration improves both data strategy and alignment.
  • Conclusion: 57.29% average gain is achieved by GPT-5.2, demonstrating autonomous data-curriculum authoring by LLM agents.The result comes from the agents’ ability to autonomously author data curricula.

Limitations

The study acknowledges limitations in evaluation scope, computational demands, and result variability. Its QA-focused design restricts assessment of open-ended generation, while iterative optimization requires substantial resources and repeated runs to mitigate fluctuations.

  • Evaluation scope: QA-focused evaluation restricts assessment of open-ended generation tasks because automated evaluation is difficult.The design enables reliable environmental feedback for closed-loop optimization.
  • Computational demands: The Iterative Agent demands considerable computational resources for model inference and fine-tuning despite strict budget caps.This limitation concerns the computational cost of iterative optimization.
  • Result variability: Results are averaged across multiple runs to mitigate fluctuations.The passage identifies repeated-run averaging as a response to variability, though the supplied text is truncated afterward.

A Dataset Details

The study evaluates autonomous end-to-end data engineering across specialized Science, Code, and Finance domains. Each domain uses tailored source datasets and construction rules for seed and evaluation data, including deterministic scoring or disjointness constraints where specified.

  • Domain coverage: Datasets span three specialized domains: Science, Code, and Finance.The datasets are curated to systematically analyze LLM agents’ capability in end-to-end data engineering.
  • Science: Science combines SciBench for college-level scientific reasoning with SciInstruct for seed construction, filtering instances to retain definitive numeric answers for deterministic rule-based scoring.SciBench covers physics, chemistry, and mathematics.
  • Code: Code uses LCB-TOP for program-output prediction, with seed instances sampled from LCB releases v1–v6 and augmented by stratified TACO samples.LCB-TOP requires predicting program execution outputs and is designed to demand deep algorithmic understanding.
  • Code: Code evaluation sets contain only LCB-TOP instances and enforce zero overlap with the seed data.This preserves separation between programming-domain construction and evaluation.
  • Finance: Finance derives both seed and evaluation sets from FinanceReasoning using disjoint stratified splits, preventing contextual leakage between the seed pool and test sets.FinanceReasoning targets financial logic, numerical reasoning, and domain-specific text comprehension.

B Main Result Details … C.2 Iterative Agent Configuration

The study establishes baseline performance for a unified LLaMA-3.1-8B-Instruct student model and specifies configurations for one-shot and iterative data-synthesis agents. The iterative setup adds feedback-driven dataset refinement through repeated drafting, training, evaluation, and improvement cycles.

  • B Main Result Details: The unified LLaMA-3.1-8B-Instruct base model scores 16.74 on SciBench, 21.18 on LCB-TOP, and 39.93 on FinanceReasoning before instruction tuning or distillation.These private-test-set scores establish the pre-specialization baseline across science, code, and finance.
  • B Main Result Details: The main experiments report MATS and Accuracy (%) for Specialization from Scratch and Specialization with Seed across all three domains.At least two independent runs are reported for each agent configuration to assess stability.
  • C Experiment Configuration Details: The experimental configurations cover data-synthesis agents, LoRA-based model training, and teacher- and student-model inference parameters.This configuration breakdown spans both One-Shot and Iterative agents.
  • C.1 One-Shot Agent Configuration: The One-Shot Agent generates the synthetic dataset in a single pass, balancing generation speed with robustness against code-execution failures.It uses a 12-hour runtime limit, 2,000-sample dataset, 3-hour per-execution limit, and 50,000 teacher API calls per task, with ≤5,000 per attempt.
  • C.1 One-Shot Agent Configuration: The One-Shot Agent uses Qwen3-30B-A3B with Api Concurrency 80, permits 8 generation attempts, and validates locally with Vllm Max Num Seqs 128.Retries address code failures or invalid JSON, while the local student environment improves validation throughput.
  • C.2 Iterative Agent Configuration: The Iterative Agent analyzes trained-model performance and refines its dataset through a cycle of drafting, training, evaluating, and improving.This feedback loop requires a more sophisticated configuration than single-pass synthesis.
  • C.2 Iterative Agent Configuration: The Iterative Agent runs for at most 30 iterations with Max Debug Attempts 3, while observing the 12-hour global limit and 50,000-call Total Api Limit.Each complete iteration cycle takes approximately 1–2 hours, and the system uses resource availability to continue refining or finalize submission.
  • C.2 Iterative Agent Configuration: The iterative configuration uses Qwen3-30B-A3B as the teacher model with API_CONCURRENCY 80 and assigns an agent controller through AGENT_MODEL: your_agent_model.The configuration inherits common, teacher, and student settings.

C.3 Model Training Parameters … D Platform Design

The paper standardizes parameter-efficient student training and separates teacher data generation from student evaluation, while fixing models and budgets across settings. Its execution platform supports controlled agent runs, monitors resources, and verifies final gains through replayed training and private evaluation.

  • C.3 Model Training Parameters: Student models are fine-tuned with LoRA-based supervised fine-tuning on 2× H100 GPUs.LoRA targets all linear layers with rank 8 and alpha 16.
  • C.3 Model Training Parameters: Training uses AdamW with a learning rate of 1.0 × 10−4, cosine scheduling, 0.1 warmup, three epochs, and Bf16 precision.The per-device batch size is 1 with 8 gradient accumulation steps, and evaluation occurs every 500 steps.
  • C.4 Model Inference Parameters: Inference is split into teacher data generation and student model evaluation, with phase-specific parameters optimized for their objectives.The teacher uses Qwen3-30B-A3B on separate 2× H100 GPUs via vLLM, while the student runs locally with vLLM.
  • C.4.1 Teacher Model Inference: Teacher generation uses an asyncio.Semaphore concurrency limit of 80 and standard sampling parameters to increase throughput and synthetic-data diversity.The teacher deployment is fixed as Qwen3-30B-A3B with vLLM on 2× H100 GPUs.
  • C.4.2 Student Model Inference (Evaluation): Student evaluation reserves 0.85 GPU memory utilization, allows 128 concurrent sequences, automatically scales tensor parallelism, and dynamically generates sampling parameters.The student inference configuration also sets max_model_len to 8192.
  • C.5 Human Involvement Analysis Setup: Across settings, Qwen3-30B-A3B is fixed as the teacher, Iterative Agent receives a 5,000-call budget, DataFlow receives 6,000 calls, and each final set has 2,000 examples.Student models and training and inference parameters remain identical across settings.
  • D Platform Design: The execution platform enables agent-driven data synthesis and end-to-end training through an isolated environment with validation, training, evaluation, API tracking, and resource monitoring tools.Runs are terminated when they exceed prescribed budgets, and final submissions are replayed by retraining on submitted data and evaluating on a private test set.

E Generalization Across Different Teacher-Student Configurations · F Running Example of Iterative Agent · G Code Snippets for Failure Analysis

The paper shows that GPT-5.2’s iterative data-engineering approach generalizes across teacher–student configurations and improves data synthesis through self-reflection and environmental feedback. A running Science-task example illustrates increasingly robust generation, filtering, repair, and checkpointing procedures, while accompanying code implements failure-analysis cases.

  • E Generalization Across Different Teacher-Student Configurations: The iterative agent consistently outperforms the one-shot baseline across teacher–student model configurations on the Science task.Although Gain (%) varies with the models’ base capabilities, the iterative approach produces positive gains and a significant performance gap in every configuration.
  • E Generalization Across Different Teacher-Student Configurations: The fixed teacher–student setup provides a consistent comparison scale, while autonomous data engineering generalizes to other model combinations.Additional GPT-5.2 experiments were conducted with varying teacher and student configurations to test robustness against overfitting to one model pair.
  • F Running Example of Iterative Agent: The running example traces GPT-5.2’s Science-task optimization from an original data-synthesis plan to an improved solution guided by self-reflection and environmental feedback.The improvement plan targets solution quality, problem diversity, and prompt engineering before generating more complex and robust code.
  • F Running Example of Iterative Agent: The improved pipeline broadens scientific coverage and difficulty while enforcing concise, complete, correctly formatted solutions with explicit units and boxed numeric answers.Generation spans Mathematics, Physics, and Chemistry, including introductory, intermediate, and advanced problems and applied, theoretical, laboratory, and engineering scenarios.
  • F Running Example of Iterative Agent: The implementation repairs invalid examples while preserving questions, regenerates deficits in fixed batches, deduplicates valid items, and saves periodic checkpoints.Repair rewrites solutions to be correct and concise; generation and repair stop when deficits are filled or API-call and availability constraints are reached.
  • G Code Snippets for Failure Analysis: The paper provides code implementations generated by agents for the failure cases analyzed in Section 4.3.These snippets correspond the agents’ concrete implementations to the reported failure-analysis cases.

G.1 Science Task: Partial Distribution Shift.

The Science partial-distribution-shift task uses an agent-generated pipeline to create instruction-tuning data for college-level scientific problems. It combines strict JSON and answer-format constraints, validation and sanitization checks, targeted topic prompting, and iterative generation toward a 2,000-item dataset.

  • Data schema and constraints: The generated data schema requires exactly one JSON object with string input and output fields.The input must start with “Question:” and explicitly state the answer unit; the output must provide a step-by-step solution.
  • Prompt strategy: Targeted prompts focus on five scientific problem families, including Boltzmann ratios, photoelectric energy, Coriolis deflection, manometry, and orbital mechanics.The prompts also specify relevant physical constants and require original, solvable, multi-step quantitative problems.
  • Iterative generation: The pipeline alternates targeted and general modes while generating batches, validating each response, removing duplicate inputs, and saving accepted items.The main loop regenerates data until reaching the target or exhausting its call budget.

G.2 Code Task: Naive Rule-based Augmentation. … I Prompt Templates

The paper documents a naive rule-based augmentation pipeline, representative domain seed data, and prompt templates governing one-shot and iterative autonomous data synthesis. Together, these materials specify the implementation, input domains, constraints, and feedback-driven agent behaviors.

  • G.2 Code Task: Naive Rule-based Augmentation.: The Code task implementation loads valid instruction-input-output seeds, applies stochastic regex perturbation, labels augmented variants, removes duplicates, and expands the dataset toward a target size.The generated code identifies augmentation with "regex_perturbation_v2" and uses content hashes for deduplication.
  • G.2 Code Task: Naive Rule-based Augmentation.: The augmentation tool defaults to generating a target dataset size of 5000 entries.The command-line argument defines 5000 as the default target size.
  • H Seed Examples: Representative seed examples are drawn directly from seed.json files for the Science, Code, and Finance domains.The examples are presented as experimental seed data across the three domains.
  • H.1 Science Domain; H.2 Code Domain; H.3 Finance Domain: Science seeds contain complex physics and chemistry calculations, Code seeds contain algorithmic problems with examples and test inputs, and Finance seeds pair financial contexts with quantitative reasoning questions.The domain examples include charge interactions and epidemics, arithmetic-series and recurrence problems, and mortgage or stock-price calculations.
  • I Prompt Templates; I.1 System Description: The prompt framework asks agents to create synthetic instruction-tuning data whose quality is evaluated by the fine-tuned standard model’s performance improvement.Agents receive task resources, must save approximately {dataset_size} entries in the required JSON format, and operate under API and runtime constraints.
  • I.1 System Description: The system instructions require diverse, complex, task-aligned data generated through provided teacher models, while prohibiting direct synthetic-data enumeration and model-training code.The environment also requires direct use of helper functions and executable code.
  • I.2 One-Shot Agent: The One-Shot Agent receives one comprehensive prompt requesting a synthesis plan and execution code, with no execution feedback unless a crash triggers a retry.Its response separates analysis, plan, and implementation code.
  • I.3 Iterative Agent: The Iterative Agent loops across draft, debug, repair, and improve states, using execution errors, submission errors, current metrics, and failed cases to revise the dataset.Iteration prompts expose remaining time and API calls, and improvement prompts target weaknesses identified from failed cases.
Loading 2605.30407v2…