PaperlayerView exact revision on arXiv

Source-linked AI summary

DataPrep-Bench: Benchmarking LLMs as Training Data Preparators

Hao Liang, Qifeng Cai, Yibo Lin, Jianzhuo Du, Qifeng Xia, Sizhe Qiu, Linzhuang Sun, Meiyi Qiang, Zhaoyang Han, Xiaochen Ma, Bohan Zeng, Ruichuan An, Conghui He, Wentao Zhang

arXiv:2607.20465v1cs.LGcs.CL

TL;DR

Training-data preparation lacks a unified end-to-end benchmark covering both data construction and quality evaluation. DataPrep-Bench evaluates both against downstream utility, finding domain-dependent construction strengths and DAS strongest overall across four of six domains.

  • Problem

    No unified benchmark measures how LLMs, agents, and data-centric workflows construct and evaluate training data end to end, despite training-data quality governing LLM performance.

  • Method

    DataPrep-Bench jointly evaluates raw-source-to-supervision construction and candidate-dataset scoring against downstream fine-tuning utility across shared domains and base models.

  • Results

    DAS has the strongest domain-averaged correlation in four of six domains and is the only metric with r > 0.70 simultaneously in Math, Science, and Medical, while construction methods remain domain-dependent.

  • Takeaways & Limitations

    Downstream-grounded evaluation is necessary because synthesized data can degrade Dolly-only performance in ways surface-level quality metrics do not anticipate.

  • Takeaways & Limitations

    The released candidate pool is an initial release whose sizes vary across domains and are expected to grow with additional in-domain candidates.

Abstract

from arXiv · show

The quality of training data fundamentally determines the capabilities of large language models (LLMs), yet no unified benchmark exists to measure how well LLMs, agents, and data-centric workflows actually prepare training data end to end. We view LLM-driven data preparation as comprising two complementary capabilities: data construction, which transforms raw sources into supervised training data, and data quality evaluation, which predicts the training value of candidate datasets before downstream training; throughout, "quality" refers to downstream training utility rather than surface-level textual properties. We introduce DataPrep-Bench, the first unified benchmark that jointly evaluates both capabilities under a shared downstream-grounded protocol over six domains and multiple base models. For data construction, methods consume identical raw sources and are scored by fine-tuning a base model on their outputs jointly with Dolly-15k; alongside this track we release Data-Construction-Skill, a skill-guided agent that lifts the Dolly-only baseline by nearly 20 points absolute on Llama-3.1-8B Finance and is competitive with the strongest agent- and DataFlow-based methods in knowledge-extraction-dense domains. For data quality evaluation, scoring functions are scored by Pearson correlation with downstream performance on a shared candidate pool; we release the Distributional Alignment Score (DAS), a distribution-based evaluator that uses MMD between a candidate dataset and a domain proxy. DAS attains the strongest cross-model correlation in four of six domains and is the only metric clearing r > 0.70 simultaneously in Math, Science, and Medical, outperforming existing quality-, diversity-, and heuristic-based evaluators. DataPrep-Bench provides a unified, downstream-grounded framework for measuring progress on both capabilities as co-equal targets of LLM-driven data preparation.

1 Introduction

DataPrep-Bench unifies downstream-grounded evaluation of LLM-driven data construction and data quality evaluation under shared sources, models, protocols, and benchmarks. Its experiments show that synthetic data can hurt, construction methods are domain-dependent, and DAS is the most reliable evaluator overall.

  • Track definitions: Data construction converts raw domain sources into supervised data, while quality evaluation predicts which candidate datasets improve downstream model performance before training.The construction track fine-tunes with Dolly-15k; the evaluation track correlates scalar scores with downstream performance.
  • Benchmark: DataPrep-Bench jointly evaluates data construction and data quality evaluation under the same raw sources, base models, training protocol, and downstream benchmarks.The benchmark is organized into two tracks and provides shared evaluation conditions.
  • Experimental takeaways: Adding synthesized domain data to Dolly-15k often hurts downstream performance across DataFlow-based, direct-LLM, and agent-based generators and both base models.The comparison attributes regressions directly to domain-specific synthetic data, which surface-level quality proxies may miss.
  • Experimental takeaways: Nearly 20 points absolute: Data-Construction-Skill lifts the Dolly-only baseline on Llama-3.1-8B Finance and matches the strongest agent- and DataFlow-based methods.DataFlow-Skill leads structured domains, agent-based methods lead reasoning-heavy domains, and Science and parts of Law remain open.
  • Experimental takeaways: DAS has the strongest domain-averaged correlation in four of six domains and is the only metric clearing r > 0.70 simultaneously in Math, Science, and Medical.Finance and Law remain difficult for every tested metric.

2 Background and Related Works

Prior work frames LLM-driven data construction as seed-based synthesis, domain-grounded extraction, or staged and agentic workflows, while data-quality evaluation ranges from per-example scoring to dataset-level analysis. Existing benchmarks and tooling do not jointly compare these approaches across construction and evaluation under downstream performance, motivating DataPrep-Bench.

  • Data construction: Seed-based methods synthesize instruction data from human-written examples, rewriting or scaling it with LLMs while remaining anchored in the seed distribution.Examples include Self-Instruct, WizardLM, UltraChat, Tülu 3, and Infinity-Instruct.
  • Data construction: Domain-grounded methods extract and reformulate supervision from authoritative books, papers, and technical documents, but differing sources, models, and recipes make comparisons anecdotal.Representative datasets include UltraMedical and MegaScience.
  • Data construction: Workflow and agentic methods decompose construction into stages such as chunking, filtering, scoring, rewriting, and verification, yet lack shared corpus-level specifications for valid supervision.Existing systems include DataFlow, Text2SQL-Flow, and step-level verification pipelines.
  • Data quality evaluation: Per-example quality evaluators aggregate individual scores from judges, classifiers, complexity measures, diversity signals, and reward-model-style scorers, whereas dataset-level methods assess variation or lexical diversity.Dataset-level examples include Vendi Score, MTLD, and HD-D.
  • Benchmark gap: DataComp-LM and related benchmarks evaluate curation through downstream performance, while Data-Juicer and DataFlow provide modular tooling without comparing approach effectiveness.DataComp, DataPerf, and DCBENCH cover related data selection, filtering, debugging, or cleaning tasks.
  • Benchmark gap: DataPrep-Bench is the first benchmark spanning data construction and data quality evaluation through end-to-end downstream performance across multiple domains and model architectures.It directly evaluates how well LLMs perform data preparation.

3 Benchmark Design

DataPrep-Bench jointly evaluates data construction and data quality evaluation under a shared principle: preparation decisions count only when they improve downstream models. Construction turns raw domain sources into SFT data, while quality evaluation predicts candidate datasets’ downstream training utility before fine-tuning.

  • Track Overview: The benchmark has two tracks: construction converts raw domain sources into SFT data, while quality evaluation predicts which candidate datasets improve downstream models.Both tracks use downstream model performance as the governing criterion, despite taking different inputs.
  • Data Construction Track: Construction performance is measured by the downstream benchmark score induced by each synthesized dataset, serving as a proxy for practical training utility rather than surface-level quality.The downstream benchmark is curated independently of the source corpus and captures the domain’s core capabilities.
  • Data Construction Track: Construction methods process common raw inputs consisting mainly of long domain-specific books and knowledge materials, extracting and reformulating supervision rather than consuming SFT-ready data.PDFs are converted to Markdown with preserved section structure, tables, and formulas so methods differ in processing rather than input format.
  • Data Quality Evaluation Track: Quality evaluation scores candidate datasets from public SFT corpora, including in-domain and out-of-domain distributions, against downstream performance obtained by fine-tuning a shared base model.Metrics may optionally use a domain proxy dataset, which should align with the target capability, be established and curated, and remain disjoint from candidates and downstream tests.
  • Data Quality Evaluation Track: Pearson correlation measures how well pre-training metric scores track empirical downstream performance, with reliability requiring statistical significance at p < 0.05.The protocol reports per-model correlations, domain averages across base models, and aggregate cross-domain summaries when score directions are aligned.
  • Data Quality Evaluation Track: The candidate pool’s ground truth is actual fine-tuning utility, not intrinsic dataset quality, and its size is not fixed across domains.This design reflects the use case of ranking or selecting datasets before incurring downstream training costs.

4 Methods

Section 4 presents Data-Construction-Skill, a skill-guided agentic framework for converting long-form domain sources into reusable QA supervision, and DAS, a distribution-based metric for estimating candidate training utility. Data-Construction-Skill regulates construction through reusable specifications and validation, while DAS measures candidate–proxy distributional alignment with MMD.

  • Data-Construction-Skill: Data-Construction-Skill transforms long-form domain documents into reusable QA-style supervised fine-tuning data through a skill-guided agentic procedure.The method targets supervision extraction and reformulation from hundreds of pages of expert-authored content.
  • Data-Construction-Skill: Skills package task instructions, output schemas, quality constraints, filtering rules, and auxiliary resources into a reusable control layer for agent execution.This framework guides and stabilizes agent behavior beyond one-shot prompting or a fixed pipeline.
  • Data-Construction-Skill: The construction pipeline partitions each corpus into semantically coherent chunks, discards uninformative material, and extracts source-grounded knowledge propositions for supervision.Chunking preserves local semantic completeness while providing manageable units for downstream construction.
  • Data-Construction-Skill: Up to three complementary QA forms—concept-oriented, reasoning-oriented, and case-based—are adaptively generated as source-grounded instruction-response pairs.They cover reusable atomic knowledge, concise reasoning patterns, and simple application scenarios.
  • Distributional Alignment Score: DAS scores a candidate dataset by measuring distribution-level similarity to a domain proxy using Maximum Mean Discrepancy (MMD).Text samples are mapped into a fixed feature space, and the induced feature distributions are compared with MMD.

5 Experiments

Experiments evaluate data construction and data quality evaluation under identical downstream-grounded conditions. Construction results favor agents and skill-guided methods in specialized domains, while DAS provides the strongest cross-domain quality prediction with a favorable accuracy–compute trade-off.

  • Experimental protocol: All construction methods consume the same raw corpus and are evaluated through downstream performance after fine-tuning on their synthesized SFT datasets.Quality metrics are likewise evaluated on identical candidate pools and downstream-score pairs.
  • Data Construction Track: Adding domain-specific synthetic data to Dolly-15k frequently reduces performance, with the Dolly-only baseline already strong in Math and Science on Llama-3.1-8B.The regression occurs across most generator families rather than being confined to one method type.
  • Data Construction Track: Agents claim most top slots in Math, Medical, and Law, while DataFlow-Skill leads Finance on Qwen2.5-7B and direct generators all fall below the Dolly-only baseline on Finance.DataFlow methods perform strongly in structured domains but fall below the Dolly-only baseline in Science on Llama-3.1-8B.
  • Data Quality Evaluation Track: 4 of 6 domains are led or tied by DAS, which is the only metric exceeding 0.70 simultaneously in Math, Science, and Medical.DAS reaches r > 0.93 on two of three base models in Math and is roughly 1.4–1.5× faster than BERTVendi and Deita-Quality.

6 Conclusion

DataPrep-Bench unifies downstream-grounded evaluation of LLM-driven training-data preparation across data construction and data quality evaluation. The conclusion emphasizes that synthesized domain data can hurt the Dolly-only baseline and outlines future benchmark expansion and skill-guided evaluation work.

  • Conclusion: DataPrep-Bench evaluates data construction and data quality evaluation through end-to-end downstream performance.The construction track assesses transformation of raw domain sources into supervised training data, while the quality track assesses prediction of candidate datasets’ downstream training value.
  • Conclusion: Synthesized domain data frequently degrades the Dolly-only baseline across construction methods and model architectures.This result would not have been anticipated by surface-level quality metrics, underscoring the necessity of downstream-grounded evaluation.
  • Conclusion: Future work will expand Finance, Law, and Medical candidate pools and investigate skill-guided designs for open-ended scientific reasoning.The paper also proposes pairing DAS with complementary evaluation signals.

Appendix · A Detailed Introduction to the Benchmarks

The appendix details DataPrep-Bench’s evaluation suite across general text, mathematical reasoning, science, medical, financial, and legal expertise. The benchmarks range from corrected multitask questions to domain-specific assessments of reasoning, knowledge, and problem solving.

  • A Detailed Introduction to the Benchmarks: The benchmark descriptions organize evaluation by capability area, pairing broad general-text testing with specialized assessments in mathematics, science, medicine, finance, and law.The appendix presents these benchmarks as the evaluation suite used in DataPrep-Bench.
  • Appendix: MMLU-Redux provides 5,700 manually re-annotated questions across all 57 MMLU subjects to improve multitask-understanding evaluation reliability.It is designed to correct ground truth errors and ambiguities in the original benchmark.
  • A Detailed Introduction to the Benchmarks: Science evaluation includes MMLU-STEM, MMLU-Pro, GPQA, SuperGPQA, ChemBench, PIQA, and SciBench.The suite spans broad scientific knowledge, graduate-level expert reasoning, chemistry, physical commonsense, and college-level scientific problem solving.
  • A Detailed Introduction to the Benchmarks: Medical expertise is assessed with MedR-Bench, MedMCQA, and MedCaseReasoning.MedR-Bench contains 1,453 structured clinical cases across 13 body systems and 10 specialties, while MedMCQA contains over 194,000 questions across 21 subjects and 2,400 healthcare topics.
  • A Detailed Introduction to the Benchmarks: Financial expertise uses XFinBench, FinEval-KR, and CPA-KQA to test knowledge-intensive finance, subfield-specific reasoning, and accounting concepts.FinEval-KR provides decoupled metrics across 22 financial subfields.
  • A Detailed Introduction to the Benchmarks: Legal evaluation uses LegalBench and LexGLUE for legal reasoning and legal natural language understanding.Their tasks include realistic expert-designed legal challenges, judgment prediction, and contract analysis.

B Implementation Details · B.1 Data Construction: Training Hyperparameters

Data construction experiments use standardized fine-tuning settings across base models and constructed datasets, with Dolly-15k included to isolate domain-data effects from instruction-following ability. Evaluation follows benchmark-specific protocols and aggregates domain performance by averaging benchmark scores.

  • B.1 Data Construction: Training Hyperparameters: All models are fine-tuned for 3 epochs with cosine scheduling, an initial learning rate of 5.0 × 10−6, and a warmup ratio of 0.1.These hyperparameters are held constant across data construction experiments to ensure comparability.
  • B.1 Data Construction: Training Hyperparameters: Training uses per_device_train_batch_size 1 and gradient_accumulation_steps 4, producing a total global batch size of 32 on an 8×H20 GPU cluster.
  • B.1 Data Construction: Training Hyperparameters: Math evaluation uses decoding temperature 0.6 and a 16,384 token context window.
  • B.1 Data Construction: Training Hyperparameters: General Text evaluation on MMLU-Redux uses a 5-shot prompting strategy.
  • B.1 Data Construction: Training Hyperparameters: Domain-level performance is computed by averaging scores across all benchmarks within each domain.
  • B.1 Data Construction: Training Hyperparameters: Qwen2.5-7B and Llama-3.1-8B serve as the base models for data construction experiments.
  • B.1 Data Construction: Training Hyperparameters: Dolly-15k is included in every fine-tuning run as a general instruction-following corpus because the base models lack instruction-following training.
  • B.1 Data Construction: Training Hyperparameters: Each constructed dataset is fine-tuned jointly with Dolly-15k, while the Dolly-15k-only baseline uses Dolly-15k exclusively without domain-specific synthetic data.This design isolates the downstream effect of domain data construction from basic instruction-following ability.

B.2 Dataset Size Design

Dataset construction methods are not constrained by a uniform sample cap; each method determines its output size according to its pipeline, context window, or completion judgment. This produces substantial yield differences across method families, with DataFlow pipelines generating the most samples and direct LLM generators far fewer.

  • Dataset Size Design: Methods autonomously determine how much data to emit from the shared raw corpus rather than following a uniform sample cap.DataFlow pipelines emit their produced output, direct generators are bounded by context windows, and agents continue until judging construction complete.
  • Dataset Size Design: 745K–849K samples is the highest overall yield achieved by DataFlow-based pipelines through multi-step extraction that processes every input page.
  • Dataset Size Design: 12K–56K total samples are produced by direct LLM-based generators, an order of magnitude fewer because each book is processed in one context-window-bounded pass.

B.3 Data Quality Evaluation: Implementation Details

Data quality evaluation tests whether distributional-alignment correlations generalize across three distinct base-model architectures. DAS uses a fixed embedding, sampling, and Gaussian-kernel configuration across domains and candidate datasets.

  • Cross-model evaluation: Three base models—Qwen2.5-7B, Llama-3.1-8B, and Mistral-7B-v0.3—evaluate whether correlations between distributional alignment and downstream performance depend on architecture.The models span distinct architectural families.
  • DAS computation: 4096-dimensional Qwen3-Embedding-8B representations encode dataset samples for DAS computation.The embedding model is fixed across all domains and candidate datasets.
  • DAS computation: 5,000 randomly sampled instances per dataset balance statistical stability and computational efficiency.The same sampling size is used across all domains and candidate datasets.
  • DAS computation: MMD uses a Gaussian RBF kernel with fixed bandwidth σ = 1.0 across all domains and candidate datasets.The kernel configuration remains constant alongside the sampling size and embedding model.

C Prompts

The prompts define how LLMs and agents construct data, while skill-enabled agents add procedural guidance for knowledge extraction, sample generation, and validation. The skill is automatically loaded before execution and can be installed through the Clawhub repository.

  • C.1 Prompt of LLM-based Generators: Figure 2 presents the prompt guiding the LLM during data construction.
  • C.2 Prompt of Agent-based Generators: Figure 3 presents the prompt guiding the agent during data construction.
  • C.2 Prompt of Agent-based Generators: The agent prompt specifies the task objective, expected input format, and required output format.
  • C.2 Prompt of Agent-based Generators: The agent is instructed to extract reusable knowledge from source documents and transform it into structured supervision data.
  • C.3 Prompt of Agent-based Generators with a Skill: Skill-enabled construction agents use the same Figure 3 prompt, supplemented by automatically loaded procedural instructions and quality constraints.
  • C.3 Prompt of Agent-based Generators with a Skill: The skill directs agents to identify knowledge-bearing chunks, generate supervision samples, and validate the generated data.It supports concept_qa, process_qa, and case_applicat sample types.
  • C.3 Prompt of Agent-based Generators with a Skill: The full skill is provided in the Clawhub project repository and can be downloaded through its CLI command.
  • C.3 Prompt of Agent-based Generators with a Skill: After a raw-document path is provided, the code agent automatically invokes the skill and initiates data construction.
Loading 2607.20465v1…