PaperlayerView exact revision on arXiv

Source-linked AI summary

DataChef: Cooking Up Optimal Data Recipes for LLM Adaptation via Reinforcement Learning

Yicheng Chen, Zerun Ma, Xinchen Xie, Yining Li, Kai Chen

arXiv:2602.11089v2cs.CLcs.AI

TL;DR

Data recipe design for LLM adaptation remains labor-intensive because existing LLM automation usually targets individual pipeline components rather than complete recipes. DataChef formulates end-to-end recipe generation and trains DataChef-32B with online reinforcement learning guided by a proxy reward, achieving performance comparable to strong baselines and expert-curated approaches across six held-out tasks.

  • Problem

    Data recipe construction remains dependent on human heuristics and iterative refinement, while LLMs typically automate only individual filtering or synthesis components.

  • Method

    DataChef-32B generates executable data pipelines from target benchmarks and available sources, using a Data Verifier as a low-latency proxy reward for online reinforcement learning.

  • Results

    Across six held-out tasks, DataChef-32B matches Gemini-3-Pro and produces recipes that surpass Qwen3-Next-80B ⊕ Kimi-K2 by +8.6% and +9.2% in DVSavg@32 on in-domain and out-of-domain tasks, respectively.

  • Takeaways & Limitations

    DataChef demonstrates the potential of automatically exploring data-processing pipelines, including selection, mixing, synthesis, and filtering, for LLM adaptation.

Abstract

from arXiv · show

In the current landscape of Large Language Models (LLMs), the curation of large-scale, high-quality training data is a primary driver of model performance. A key lever is the \emph{data recipe}, which comprises a data processing pipeline to transform raw sources into training corpora. Despite the growing use of LLMs to automate individual data processing steps, such as data synthesis and filtering, the overall design of data recipes remains largely manual and labor-intensive, requiring substantial human expertise and iteration. To bridge this gap, we formulate \emph{end-to-end data recipe generation} for LLM adaptation. Given a target benchmark and a pool of available data sources, a model is required to output a complete data recipe that adapts a base LLM to the target task. We present DataChef-32B, which performs online reinforcement learning using a proxy reward that predicts downstream performance for candidate recipes. Across six held-out tasks, DataChef-32B produces recipes that yield performance comparable to those curated by human experts. Notably, the recipe from DataChef-32B adapts Qwen3-1.7B-Base to the math domain, achieving 66.7 on AIME'25 and surpassing the official post-training checkpoint (Qwen3-1.7B). This work sheds new light on automating LLM training and developing self-evolving AI systems.

1 Introduction

DataChef frames automated end-to-end data recipe generation as a response to labor-intensive manual pipeline design. It introduces a task pool, proxy-reward online reinforcement learning, and DataChef-32B, whose recipes perform strongly across held-out tasks.

  • Training-data composition and quality are decisive factors in shaping LLM performance.
  • Data recipe design remains heavily dependent on expert heuristics and iterative empirical refinement, despite LLM automation of individual filtering and synthesis steps.
  • End-to-end data recipe generation asks models to produce executable processing pipelines from a target benchmark and available data sources.
  • DataChef-32B matches Gemini-3-Pro across six held-out tasks and surpasses the official Qwen3-1.7B checkpoint with 66.7 on AIME’25 and 46.3 on ClimaQA.

2 Related Work

Related work covers human-designed data-processing heuristics, general-purpose pipeline frameworks, and lightweight approaches for evaluating data quality or model performance.

  • Existing approaches rely on human experts to design data-mixing, data-sampling, and data-synthesis heuristics.
  • General-purpose data-processing frameworks provide standardized modules and scalable pipeline construction for large-scale training-data curation.
  • Lightweight proxies are motivated by the substantial computational resources required to train and evaluate LLMs.
  • Existing data-evaluation approaches include indicator-based metrics for properties such as diversity, complexity, and relevance.

3 Methodology

DataChef formalizes recipe generation as a policy that produces executable data pipelines for adapting language models, then optimizes that policy with verifier-based rewards and GRPO. The framework constructs diverse tasks and evaluates sampled data to balance data quality with executability.

  • 3.1 Problem Formulation: A task combines an instruction, available raw data sources, and an evaluation metric, while a recipe specifies a pipeline and its resulting training dataset.The pipeline is implemented as Python scripts in the experiments.
  • 3.3 End-to-end Data Recipe Generation: The Data Verifier evaluates sampled data instances and assigns quality categories ranging from invalid or incorrect samples to task-matched passing samples.The reported categories assign scores of 0 to invalid, format-error, and incorrect samples, 0.4 to task mismatch, and 1.0 to passing samples.
  • 3.2 Data Pool and Evaluation: The study organizes 19 domains, 31 benchmarks, and 257 data sources into 25 seed tasks, while evaluating against six held-out tasks.The benchmark table reports DVSavg@32 and DBS, with DBSnorm normalized relative to SINGLE-SOURCEbest (100.0).
  • 3.3 End-to-end Data Recipe Generation: The policy generates a natural-language orchestration plan and executable code that transforms raw sources into training data for downstream adaptation.During training, the resulting dataset is evaluated to guide reinforcement-learning updates; during inference, the recipe is used directly for adaptation.
  • 3.3 End-to-end Data Recipe Generation: Recipe rewards use the sampled subset’s average instance score when execution succeeds, with separate penalties for empty outputs and training-format violations.The empty-output and format penalties are controlled by positive coefficients λ∅ and λfmt.
  • 3.3 End-to-end Data Recipe Generation: GRPO optimizes the policy from groups of candidate recipes using importance-ratio clipping, group-relative advantages, a fixed reference policy, and KL regularization.For each training task, the method samples a group of candidate recipes before updating the policy parameters.

4 Experiments

Experiments evaluate DataChef-32B against baselines, assess its Data Verifier and reinforcement learning, and analyze generated recipes. DataChef-32B performs strongly across held-out tasks, while verifier-guided RL and end-to-end generation improve recipe quality and generalization.

  • Main Comparison: DataChef-32B surpasses Qwen3-Next-80B ⊕ Kimi-K2 by +8.6% and +9.2% in DVSavg@32 on in-domain and out-of-domain tasks, respectively.It achieves comparable performance to Gemini-3-Pro.
  • Data Verifier: The Data Verifier maintains strictly positive correlations across all evaluated tasks, unlike baselines that show negative correlations such as −0.48 for IFD on Math and −0.15 for DEITA on Code.Four of six tasks satisfy p < 0.1 for the verifier correlation.
  • RL Effectiveness: RL improves average DVSavg@32 by 3.6% for the 8B model and 3.7% for the 32B model, with gains concentrated on out-of-domain tasks while preserving in-domain performance.Training rewards trend upward while their standard deviation decreases.
  • Data Verifier: Replacing the continuous Data Verifier score with a constant success reward causes noticeable performance drops, indicating that reward granularity guides selection of high-utility recipes.The ablation compares fine-grained Rdense with constant Rsparse rewards for valid execution.
  • Recipe Analysis: DataChef-32B identifies high-utility datasets, augments or synthesizes task-specific data, and extracts relevant subsets using self-generated keywords.These behaviors are illustrated through generated recipes for financial and ClimaQA tasks.

5 Conclusion

The paper presents automated data recipe generation with a Data Verifier as a cost-effective reward for online RL. DataChef-32B generalizes strongly and matches human-level expertise on specific benchmarks, while specialized evaluators remain a future direction because LLM-as-a-Judge rewards may be imprecise for niche tasks.

  • Conclusion: DataChef-32B combines automated data recipe generation, a holistic training and evaluation dataset, and a Data Verifier for online reinforcement learning.The paper frames this paradigm as a way to streamline LLM adaptation.
  • Conclusion: DataChef-32B demonstrates strong generalization capabilities and matches human-level expertise on specific benchmarks.The conclusion connects automated recipe generation with model evolution.
  • Limitations: LLM-as-a-Judge proxy rewards may sacrifice precision on niche tasks, motivating specialized evaluators with higher-resolution reward signals.The paper identifies this as a limitation and future research direction.

A.1 Details of Task Pool

The task pool combines benchmark-linked dataset retrieval with explicit leakage checks and model-assisted data-source selection. Prompt and model choices support both planning and executable implementation of candidate recipes.

  • The benchmark pool is documented in Table 4, which lists the benchmarks, domains, and usage.
  • Candidate datasets are retrieved by synthesizing 3–5 task-specific keywords with an LLM.
  • The search procedure queries Hugging Face and Google Search APIs, then retains the top-4 datasets for each keyword after ranking by community popularity.
  • Retrieved candidates undergo strict verification to ensure they do not overlap with the benchmark and therefore avoid data leakage.
  • DataChef uses gpt-oss-120b as the Data Verifier, while Qwen3-Next-80B-A3B-Thinking handles planning and Kimi-K2-Instruct handles code implementation for cold-start supervision.

B.1 Data Evaluation Metrics Settings

The evaluation setup combines several complementary data-quality metrics, covering instruction-following difficulty, complexity, quality, response quality, and dataset diversity. These metrics are computed with specified backend models or official checkpoints.

  • OpenDataArena-Tool1 provides the data assessment framework and default configurations for the evaluation metrics.
  • IFD uses Qwen2.5-3B-Instruct to compute Instruction-Following Difficulty scores, treating scores above 1 as outliers and assigning those anomalies a score of 0.
  • DEITA defines the final data score as the product of Complexity Score and Quality Score computed from official DEITA checkpoints.
  • RewardModelScore uses Skywork-Reward-V2-Llama-3.1-8B-40M as a proxy for response quality.
  • VendiScore uses Qwen3-Embedding-0.6B embeddings and Euclidean distance to measure dataset diversity.

B.2 Evaluation Setup

Downstream evaluation uses OpenCompass with task-specific evaluators and reporting conventions, while oracle selection combines Data Verifier ranking with human review. Recipe outputs are also constrained by explicit data budgets.

  • All downstream task evaluations use the OpenCompass framework with benchmark-specific settings.
  • The other benchmarks use specified reporting metrics and evaluators, including average accuracy for PHYSICS, ClimaQA, and OpenFinData, pass@1 for LiveCodeBench v6, and development-plus-test accuracy for CHID.
  • AIME’25 evaluation covers both 2025 parts, generates 8 responses per question, and reports average accuracy using xVerify-9B-C.
  • Oracle selection first retains the top-8 of 32 generated recipes by Data Verifier Score, then has a human expert check format alignment, context integrity, and pipeline comprehensiveness.
  • DBS evaluation uses a 10K data budget, while training and DVS evaluation use 100 samples for rapid proxy reward calculation.
  • For ClimaQA, DataChef-32B generates valid training data by LLM-based task-specific augmentation and keyword-driven extraction of relevant subsets.

D Additional Results on Correlation Analysis

The paper reports correlation analyses across all six evaluation tasks to assess whether the Data Verifier remains robust across diverse domains. Figure 11 presents these cross-task results against baseline metrics.

  • Correlation results are provided across all six evaluation tasks in Figure 11.
  • The analyses validate the robustness of the Data Verifier compared with baseline metrics across diverse domains.

E Computational Cost Analysis

The analysis reports the computational cost of supervised fine-tuning and reinforcement learning, including GPU allocation and per-step runtime. It also documents the implementation components used to generate and parse training data recipes.

  • E Computational Cost Analysis: A single RL step processes 128 candidate recipes in an average of 20.2 minutes.The Data Verifier contributes less than 2 minutes of this duration.
  • E Computational Cost Analysis: RL allocates 8 H200 GPUs to the policy model and 2 H200 GPUs to the Data Verifier deployment.
  • E Computational Cost Analysis: SFT takes an average of 2.4 minutes per step with batch size 32 on 8 H200 GPUs.
  • E Computational Cost Analysis: The recipe-generation implementation loads StackExchange Earth Science data and converts open question-answer pairs into four-option multiple-choice questions.
  • E Computational Cost Analysis: Generated responses are parsed into structured JSON containing a question, four options, and an answer before being formatted into user-assistant training examples.
Loading 2602.11089v2…