Programming with Data: Test-Driven Data Engineering for Self-Improving LLMs from Raw Corpora

• Domain fine-tuning remains open-loop: benchmark failures diagnose symptoms but cannot localize deficiencies in training data, so practitioners rely on indiscriminate augmentation.This is especially problematic because domain corpora are limited and structured, making undirected scaling expensive, poorly interpretable, and without guaranteed improvement.Source: §1 · p. 2
• Programming with Data derives training data and benchmarks from a shared knowledge representation, making failures traceable to identifiable data gaps and repairable through targeted patches.The paper frames this as the same structural prerequisite that lets shared software specifications connect failing tests to code defects.Source: §1 · p. 2
• The paradigm maps raw corpora to requirements specifications, synthesized training data to source code, training to compilation, and benchmarking to unit testing.This correspondence reconceptualizes how human-readable domain knowledge becomes executable model capabilities.Source: §1 · p. 2
• A three-level structure of concepts, relational triples, and reasoning chains links training and evaluation, while CORE requires contextualized, organized, rigorous, and evolving synthesis.The shared structure turns evaluation from terminal judgment into actionable diagnosis and governs adversarial robustness, non-overlap, and iterative refinement.Source: §1 · p. 4
• ProDa extracts the shared structure, compiles it into a pre-training benchmark, synthesizes CORE-compliant data, and repairs concept gaps or reasoning deficits through targeted patches.All phases operate on the same knowledge structure, so benchmark failures become structured diagnostics rather than prompts for indiscriminate data scaling.Source: §1 · p. 4
• After one debugging round, a 32B open-source model surpasses GPT-5.4, Gemini-3-flash, and DeepSeek-v3.2 on the 16-discipline average while preserving MMLU [13] and CEVAL [17].Across Llama and Qwen models from 3B to 32B, every tested model and scale improves consistently with no exceptions.Source: §1 · p. 4

2.1 A structural correspondence with software engineering

• A shared knowledge representation makes corpus-derived training data analogous to source code, training to compilation, models to binaries, and benchmarks to tests.Failures can be traced through the shared specification to targeted data repairs rather than treated as uninterpretable evaluation signals.Source: §2.1 · pp. 4–5
• The three-level structure separates L1 concepts, L2 typed relations, and L3 reasoning chains, enabling benchmark failures to decompose into concept gaps or reasoning deficits.Repairs target missing knowledge nodes or incorrect inferential composition, making test failures actionable engineering diagnostics.Source: §2.1 · p. 5

2.2 The ProDa pipeline

• ProDa implements the paradigm with Builder, Tester, and Debugger components coupled through the shared structure and governed by the CORE standards.CORE requires contextualized, organized, rigorous, and evolving data synthesis.Source: §2.2 · pp. 5, 7
• The Builder extracts L3 chains before L2 relations and L1 concepts, ensuring every knowledge entry connects to a testable and debuggable reasoning chain.Document-level context preservation and quality scoring protect against incomplete or misleading decontextualized units.Source: §2.2 · pp. 5–6
• The Tester constructs pre-training benchmarks from L3 chains with adversarial distractors and instance-level separation from training samples, testing novel reasoning composition rather than memorization.Training data draws from L1/L2 entries, whereas benchmark items require composition of those entries through L3 chains.Source: §2.2 · p. 6
• The Debugger classifies failures as concept gaps or reasoning deficits, applies contrastive reinforcement or scaffolded reasoning samples, and retrains on patches plus selected mastered data.This evolving loop repairs deficits while preserving previously acquired capabilities.Source: §2.2 · pp. 6–7

3.1 Specification: structured knowledge from raw corpora

• The extracted corpus contains 458,622 hierarchically connected knowledge nodes from 48,000 high-quality chunks, with exactly zero orphan nodes across 16 disciplines.Extraction yields 43,953 L3 reasoning chains, 186,784 L2 relations, and 227,869 L1 concepts; every lower-level node is reachable from a higher-level node, enabling benchmark traceability.Source: §3.1 · p. 7, Figure 3 · p. 8

3.2 Tester: validating the ProDa-16 benchmark

• ProDa-16 correlates with 11 established benchmarks at mean ρ = 0.847, reaches approximately 76% on frontier models, and discriminates across all disciplines without floor or ceiling effects.Its strongest reported correlations are with GPQA (ρ = 0.943) and MMLU-Pro (ρ = 0.905); every discipline’s median exceeds the 25% chance baseline and remains below 82%.Source: §3.2 · pp. 9–10, Figure 4 · p. 9

3.3 Implementation: first-pass compilation

• A single 160K-instance first-pass compilation produces domain competence competitive with official Instruct models, with Qwen-3-32B-V1 scoring 77.35% without human annotation or preference optimization.V1 exceeds its counterpart at selected scales but remains inferior at others, revealing static synthesis’s unresolved concept gaps and long-tail multi-step reasoning failures.Source: §3.3 · p. 10

3.4 Debug: traceable repair and the value of structure

• On ProDa-16, Qwen-2.5-32B-V2 and Qwen-3-32B-V2 reach 78.84% and 79.52%, respectively, exceeding every Instruct model in Table 1, while all nine models improve on average.Gains range from +0.77 points for Qwen3-14B to +32.67 points for Llama-3.1-8B, with larger improvements for models with lower starting performance.Source: §3.4 · p. 12
• Diagnostic repair preserves general capabilities while outperforming conventional synthesis: ProDa V2 averages 68.72% with 1K targeted samples and reaches 72.11% at 5K versus 56.18% for DataFlow Filter.On MMLU, seven of nine V2 models match or exceed Base scores, with a median Base-to-V2 change of +0.27 points after V1 caused a median −0.48-point drop.Source: §3.4 · pp. 12–13, Table 2 · p. 12, Figure 5 · p. 13

3.5 Case studies

• Three case studies show the same closed loop correcting physics concept gaps, an economics-and-law capability deficit, and a biomedicine concept gap through node-anchored patches.The system identifies incorrect responses, grounds diagnoses in L1/L2/L3 knowledge, generates targeted patches, and verifies correction at V2; ProDa Studio exposes these stages as one provenance-preserving workflow.Source: §3.5 · p. 14, §3.6 · p. 16

4.1 Knowledge structure extraction

• The corpus spans 16 disciplines, and top-down extraction builds reasoning chains, relations, and concepts while guaranteeing that every concept and relation participates in a testable chain.This reachability property prevents orphan entries that would be untestable and undebuggable.Source: §4.1 · p. 17

4.2 Benchmark and training data synthesis

• The shared structure K generates structurally separated benchmarks and training samples, with benchmark items requiring multi-step composition and adversarial discrimination rather than verbatim recall.Training samples are synthesized from contextualized L1 and L2 entries in open-ended, multiple-choice, and true-false formats, while benchmark items derive from L3 chains.Source: §4.2 · p. 18

4.3 Failure diagnosis and data repair

• Each benchmark failure produces a traceable repair: concept gaps trigger contrastive knowledge reinforcement, reasoning deficits trigger explicit chain-of-thought scaffolding, and replay preserves complementary prior knowledge during retraining.Repair batches are mixed with replay data, allocated according to each discipline’s error share, scaled to the initial corpus volume, and trained from the same base checkpoint.Source: §4.3 · pp. 18–19

5.1 Data Synthesis for LLMs

• Synthetic-data methods shifted LLM training from data quantity toward quality, using bootstrapped instructions, evolved tasks, teacher explanations, and curated textbook-like examples [14] [21] [32].Self-Instruct [40], Alpaca, WizardLM [41], Orca [32], LIMA [47], and the Phi series [12] [25] exemplify this progression.Source: §5.1 · pp. 19–20
• Specialized-domain synthesis remains unreliable because general NLP capability does not ensure high-quality generation, and most pipelines lack ground-truth anchoring and closed-loop debugging [10] [20].The proposed approach instead derives synthetic data as source code from authoritative corpora and uses debugging to enforce domain fidelity.Source: §5.1 · p. 20

5.2 LLM Benchmarking and Evaluation

• LLM evaluation spans broad static benchmarks, specialized reasoning tests, and LLM-as-a-Judge systems, but contamination and scalar scorecards limit their ability to diagnose failures [1] [6] [7] [9] [13] [17] [18] [27] [28] [36] [46] [48].This work reframes benchmarking as dynamically synthesized unit tests that activate a data-debugging loop [19] [30].Source: §5.2 · p. 20

5.3 Self-Improving LLM

• Self-play-inspired methods, including SPIN [5], Absolute Zero [45], and R-Zero [15], cannot reliably acquire domain-specific expertise without authoritative grounding.The framework reframes self-evolution as code-level data engineering, synthesizing executable data patches from authoritative corpora to address diagnosed failures beyond sample-level self-filtering.Source: §5.3 · p. 20

6 Discussion

• The framework maps knowledge structure to requirements, synthesized data to implementation, benchmarks to tests, and diagnosis to fault localization, making fine-tuning data and model behavior traceable and repairable.Unlike pre-training, fine-tuning targets defined competencies for which encoded knowledge and deficiencies can be identified. The framework addresses gaps left by synthetic-data, self-improvement, data-centric, and diagnostic approaches [40] [32] [12] [25] [43] [42] [39] [44] [30] [18].Source: §6 · pp. 20–21
• The proposed architecture is a general-purpose blueprint rather than a finished system, with future extensions including retrieval-augmented grounding and mechanistic interpretability for finer diagnosis.Each module remains an independent research problem open to domain-specific improvement.Source: §6 · p. 21

A Corpus Curation

• Corpus curation retains scientific or technical documents at undergraduate level or above when they contain non-descriptive reasoning, while recording domain, academic level, reasoning type, and confidence.The appendix supplies classification rules and a strict JSON output requirement for document triage.Source: §A.1 · p. 26, Academic Level Rules · p. 26, Reasoning Type Rules · p. 26

A.2 Chunk quality scoring rubrics

• The six-dimensional chunk-quality matrix evaluates reasoning depth, prerequisite density, scenario applicability, counter-intuitive content, knowledge synthesis, and breakpoint smoothness.These dimensions prioritize reasoning chains, control difficulty, support application-oriented and hard-negative examples, expose concept relationships, and require selected chunks to achieve a breakpoint-smoothness score of at least 4.Source: §A.2 · p. 27, §A.2.1 · p. 27, §A.2.2 · p. 27

B.1 L3 reasoning chain extraction prompt

• L3 extraction identifies one complete, coherent primary reasoning chain per text, preserving direct step connections, natural closure, topic cohesion, and appropriate granularity.Chains also record preconditions, negative constraints, and confusions for downstream benchmark construction.Source: §B.1 · p. 33

B.2/B.3 Knowledge structure extraction

• L2 converts each adjacent pair of reasoning-chain steps into one typed atomic statement with subject, object, predicate, and source quote, skipping unsupported links.L1 then deduplicates and canonicalizes statement terms, defines them contextually, assigns domain types, and preserves parent-statement and source-CID traceability.Source: Decomposition Procedure · p. 34, §B.3 · p. 35, Extraction Procedure · pp. 35–36

B.4 Per-discipline statistics — L3/L2/L1

• The per-discipline statistics report L3 chain steps, L2 statement decomposition, predicate types, L1 concept types, and concept participation across disciplines.These statistics characterize the extracted hierarchy rather than supplying additional performance comparisons.Source: Table 10 · p. 36, Table 11 · p. 37, Table 12 · p. 37

B.5 Cross-discipline case studies

• Case studies show the hierarchy generalizing from chromatin activation and electrolysis to visual representation, where framing mediates viewing through boundaries, immersion and withdrawal, and metarepresentational exposure.The sociology example extracts Parergon (Frame), demonstrating application to an interpretive social-science discipline.Source: §B.5.1 · p. 38, Figure 15 · p. 38, §B.5.2 · p. 39

C.1 SFT generation prompts (QA / Choice / TF)

• SFT prompts generate choice, QA, and true/false data from atomic L1 concepts and L2 factual statements, targeting precise, educational understanding rather than database-like text.Choice prompts produce single- and multiple-choice questions; QA prompts create classroom-style pairs; true/false prompts test statement verification.Source: §C.1 · p. 41, §3 · p. 42, §5 · p. 43

1. Coverage Requirement (CRITICAL)

• The SFT prompts enforce broad source coverage and sufficient volume, requiring at least 70–90% L2-statement coverage where possible and totals meeting or exceeding the requested maximum.Multiple questions or variations may be generated per statement when the requested dataset is large.Source: §1 · pp. 41, 42, 44, §3 · pp. 41, 44

2. Single-Choice Question Construction

• Choice construction uses subject–predicate–object facts for single-choice items and related L2 statements around an L1 concept for multi-select items, with correct facts and plausible distractors.Single-choice items use one L2 statement and three distractors; multiple-choice items combine 2–4 related statements and require selecting all correct options.Source: §2 · p. 41, §3 · p. 41

6. Prohibitions (Strictly Avoid)

• Across SFT and benchmark prompts, explanations and outputs must be strict, natural, educational, and free of internal metadata, markdown, translation, or other formatting violations.Explanations should justify answers through domain reasoning and explain why distractors are wrong.Source: §3 · pp. 41–42, §6 · p. 46

2. Atomic Focus

• QA and true/false generation preserve atomic factual focus while refining statements into fluent, unambiguous natural language with explicit nouns, educational explanations, and misconception-resistant truth values.QA pairs focus on one L2 statement; true statements preserve facts, whereas false statements alter relationships, add misconceptions, or invert facts without obvious giveaways.Source: §3 · pp. 42, 44, §4 · p. 43, §2 · p. 44

C.2 Benchmark item construction prompts

• Benchmark prompts transform reasoning chains into comprehensive MCQs that test causal relationships, logical sequences, mechanisms, synthesis, and application instead of trivial factual recall.The priority order is process reasoning, causal analysis, critical understanding, and application to related scenarios.Source: §C.2 · p. 45, Goal · p. 45, §1 · p. 45

4. Options and Distractor Quality

• Benchmark items must include specific, complete context and four substantial, parallel options with exactly one unambiguous answer, while retaining the input language and avoiding vague references or incomplete citations.Questions typically span 40–100 words, options 20–60 words, and all referenced steps or preconditions must be included in full.Source: §3 · p. 45, §4 · p. 45, §6 · pp. 45–46

C.3 Distractor generation strategy

• Distractors are designed as plausible logical traps representing reversed causality, missing preconditions, wrong step order, overgeneralization, or misattributed mechanisms.They must remain semantically related, substantial, similar in length and style, and parallel in structure rather than obviously wrong.Source: §C.3 · p. 46

C.4 SFT data statistics

• SFT_v1 is balanced across 16 disciplines, with open-ended questions comprising 60.0%, single-choice 23.2%, true/false 10.0%, and multiple-choice 6.8%.The dataset maintains varied question-type ratios while preserving equal disciplinary representation.Source: Figure 23 · p. 47, Table 13 · p. 48

D.1 Diagnostic prompt

• The diagnostic prompt instructs an evaluation expert to classify model failures as conceptual gaps or reasoning deficits and return strict JSON.This classification provides the basis for targeted downstream training-sample generation.Source: Figure 24 · p. 49

D.2 Concept gap prompt

• The concept-gap prompt generates publication-grade samples that directly correct diagnosed misconceptions through precise definitions, contrasts, context, and explicit knowledge boundaries.It prioritizes the failed behavior, addresses all supplied facts, uses diverse question types, and structures answers around definitions, attributes, and applications or counterexamples.Source: §D.2 · p. 49, Input · p. 50

Input

• Concept-repair inputs provide the target concept, definitions, facts, examples, and question limit, while generated samples must explicitly challenge the documented error and distinguish correct from incorrect interpretations.The requirements include at least two questions correcting the error, complete fact coverage, progressive difficulty, contrastive examples, exact terminology, and a validation checklist.Source: Input · p. 50

Format-Specific Requirements

• Format-specific concept-gap outputs require structured open-ended answers, scenario-based multi-select questions with explanations for every option, and true/false answers with subtle conditions and reasoning.These requirements keep all answer content in the designated answer field for multiple-choice and true/false formats.Source: Format-Specific Requirements · pp. 50–51, Figure 25 · p. 51

D.3 Capability deficit prompt

• The capability-deficit prompt converts diagnostic insights into teaching-grade Chain-of-Thought samples that make failed reasoning paths explicit and justify every intermediate step.It targets missing steps, broken logical chains, and failed cause-effect connections while requiring reasoning checkpoints, validation, counterfactuals, comparisons, and boundary tests.Source: §D.3 · p. 52, Input · pp. 52–53

Output Format

• The generation pipeline requires strict JSON arrays with format-dependent question, option, and answer fields, including embedded step-by-step explanations for reasoning tasks.Open-ended and true/false outputs use concise reasoning, while multi-select outputs include four options and explanations for correct and incorrect choices.Source: Format-Specific Requirements · p. 53, Output Format · p. 53

Advanced Reasoning Patterns to Apply

• Advanced reasoning samples apply causal-chain, elimination, construction, and comparison patterns to build answers from facts, constraints, first principles, or trade-offs.These patterns operationalize multi-hop reasoning and selection among alternative approaches.Source: Advanced Reasoning Patterns to Apply · p. 53

D.4 Data mixing and replay strategy

• The Round 2 corpus retains 160,000 samples but reallocates category quotas according to each category’s proportion of benchmark errors.Round 1 uses 10,000 samples per category across 16 categories before diagnostic evaluation.Source: §D.4 · p. 54
• Each diagnosed conceptual gap or capability deficit receives exactly 20 targeted repair samples distributed as open-ended QA, multiple-choice, and true/false examples in a 6:3:1 ratio.The format distribution is 60% open-ended QA, 30% multiple-choice, and 10% true/false.Source: §D.4 · p. 54
• L2-disjoint experience replay fills category-quota deficits with correctly answered Round 1 questions while preventing catastrophic forgetting of previously mastered concepts.Every replay sample is mapped to an L2 factual statement under the stated disjointness constraint.Source: §D.4 · p. 54, Figure 27 · p. 55
• The complete strategy progresses from uniform baseline initialization through error-proportional allocation, targeted repair generation, and quota-filling replay.Figure 27 summarizes the fixed-budget transition, 6:3:1 repair formats, and L2 ID-disjoint replay mechanism.Source: §D.4 · p. 54, Figure 27 · p. 55

D.5 Diagnostic report example

• Qwen2.5-7B-SFT achieves 65.86% overall accuracy, with 9,268 correct answers out of 14,072 and 4,804 error samples.The diagnostic report is timestamped 20260210_220124.Source: §D.5 · p. 55

Subject-wise Performance (Truncated)

• Subject-wise results vary across the reported examples, with accuracies of 64.6%, 65.0%, and 63.0% for Subject-001-en, Subject-002-en, and Subject-003-en, respectively.Each listed subject contains 1,000 examples; the corresponding error counts are 354, 350, and 370.Source: Subject-wise Performance (Truncated) · p. 55

• Capability deficits account for 3,093 samples, exceeding the 1,509 samples classified as concept gaps.Source: Error Pattern Analysis · p. 55
• Multiple-choice questions comprise 4,787 samples, compared with only 17 single-choice questions.Source: Error Pattern Analysis · p. 55
• The dataset therefore contains substantially more capability-deficit errors than concept-gap errors and overwhelmingly favors multiple-choice questions.Source: Error Pattern Analysis · p. 55

• The model predicted A,B,C,D instead of the ground-truth A,B,D for the Fresnel half-wave-strip analysis of single-slit Fraunhofer diffraction.Source: Representative Error Sample: Index 3… · p. 55
• The error reflects misunderstanding of interference in diffraction patterns and the conditions required for bright and dark fringes; reviewing diffraction and interference principles is recommended.Source: Representative Error Sample: Index 3… · p. 55

Representative Error Sample: Index 7 (Capability Deficit)

• The model predicted A,B,C,D instead of the ground-truth A,B,C for equilibrium equations governing a curved slender rod.Source: Representative Error Sample: Index 7… · p. 55
• The extra option D indicates that the model failed to accurately assess prerequisites for deriving the equilibrium equations.Source: Representative Error Sample: Index 7… · p. 55
• The recommended repair is to enhance reasoning capabilities through training on multi-step reasoning.Source: Representative Error Sample: Index 7… · p. 55

Other Samples

• Figure 28 showcases an automated diagnostic report for Qwen2.5-7B-SFT that quantifies error patterns and qualitatively diagnoses specific failure cases for subsequent refinement.The report is presented as an example of how model failures can be analyzed to guide refinement.Source: Figure 28 · p. 56

E.1 Training hyperparameters and infrastructure

• LoRA fine-tuning used LLaMA-Factory with bf16 mixed precision, while LoRA Rank, LoRA Alpha, and learning rate scaled empirically with base-model size.LoRA Dropout, Batch Size, and Warmup Ratio remained constant for controlled experiments.Source: §E.1 · p. 56, Table 14 · p. 56
• Training hyperparameters were documented across base models in Table 14, covering LR, BS, maximum sequence length, and mixed precision.Source: Table 14 · p. 56

E.2 Evaluation protocols

• Evaluations used the OpenCompass framework with strict configurations to ensure reproducibility.Source: §E.2 · p. 56

1. OpenCompass Setup

• OpenCompass ran local HuggingFace inference with GenInferencer, standardized zero-shot option-letter prompts, greedy decoding, and a 15-token generation cap.Greedy decoding used do_sample = False and temperature = 0 to eliminate sampling variance.Source: §1 · p. 56

2. Strict Scoring Rules & Post-processing

• Generated answers were parsed to remove reasoning traces, deduplicate valid options, and alphabetically format outputs such as A,B,C.Source: §2 · p. 57
• Exact-match scoring awarded 1 only for a normalized prediction that perfectly matched the ground truth, with no partial credit for missing, extra, or incorrect options.Source: §2 · p. 57
• To reduce computational overhead while testing cross-domain generalizability, evaluation used 12 discipline-mapped MMLU subsets and 6 discipline-mapped C-Eval subsets.The listed MMLU subsets span algebra, chemistry, computing, engineering, history, mathematics, logic, machine learning, marketing, and religion; C-Eval covers qualification, business, physics, architecture, and logic.Source: §2 · p. 57