PaperlayerView exact revision on arXiv

Source-linked AI summary

Programming with Data: Test-Driven Data Engineering for Self-Improving LLMs from Raw Corpora

Chenkai Pan, Xinglong Xu, Yuhang Xu, Yujun Wu, Siyuan Li, Jintao Chen, Conghui He, Jingxuan Wei, Cheng Tan

arXiv:2604.24819v1cs.SEcs.AI

TL;DR

Specialized knowledge from raw text is difficult to encode in language models because failures cannot be traced to specific training-data deficiencies. The paper introduces Programming with Data, linking shared knowledge structures, training, evaluation, and debugging, and reports consistent gains across tested models and scales.

  • Problem

    Existing workflows lack a principled mechanism for tracing domain-task failures back to specific deficiencies in training data.

  • Method

    Programming with Data uses a shared knowledge structure to derive training data and benchmarks, making failures diagnosable as concept gaps or reasoning-chain breaks.

  • Results

    After one data-debugging round, a 32B open-source model surpasses GPT-5.4, Gemini-3-flash, and DeepSeek-v3.2 on the 16-discipline average while general capabilities remain fully preserved.

  • Takeaways & Limitations

    The work presents a general-purpose blueprint for converting raw textual knowledge into verifiable model competence through transparent, diagnosable, and repairable data engineering.

  • Takeaways & Limitations

    Static one-off data synthesis leaves conceptual blind spots and cannot dynamically correct model-specific concept gaps or long-tail multi-step reasoning errors.

Abstract

from arXiv · show

Reliably transferring specialized human knowledge from text into large language models remains a fundamental challenge in artificial intelligence. Fine-tuning on domain corpora has enabled substantial capability gains, but the process operates without feedback: when a model fails on a domain task, there is no method to diagnose what is deficient in the training data, and the only recourse is to add more data indiscriminately. Here we show that when a structured knowledge representation extracted from the source corpus serves as the shared foundation for both training data and evaluation, the complete data-engineering lifecycle maps onto the software development lifecycle in a precise and operative way: training data becomes source code specifying what the model should learn, model training becomes compilation, benchmarking becomes unit testing, and failure-driven data repair becomes debugging. Under this correspondence, model failures decompose into concept-level gaps and reasoning-chain breaks that can be traced back to specific deficiencies in the data and repaired through targeted patches, with each repair cycle producing consistent improvements across model scales and architectures without degrading general capabilities. We formalize this principle as Programming with Data and instantiate it across sixteen disciplines spanning the natural sciences, engineering, biomedicine, and the social sciences, releasing a structured knowledge base, benchmark suite, and training corpus as open resources. By demonstrating that the relationship between training data and model behaviour is structurally traceable and systematically repairable, this work establishes a principled foundation for the reliable engineering of human expertise into language models.

1 Introduction

Programming with Data closes the open loop in domain-specific fine-tuning by deriving training data and evaluation from a shared knowledge structure. The ProDa framework operationalizes this lifecycle across 16 disciplines, with experiments showing consistent gains across tested models and scales.

  • Motivation: Specialized knowledge remains difficult to encode because it resides largely in unstructured corpora that must be transformed into verifiable model capabilities.The paper identifies this transformation as a central data-engineering challenge.
  • Motivation: Domain-specific fine-tuning inherits an open-loop workflow in which failures cannot be systematically connected to deficiencies in the training data.The introduction contrasts this with pre-training, where scale and broad post-hoc evaluation can provide statistical coverage.
  • Programming with Data: Programming with Data maps data engineering onto test-driven software development: corpora specify requirements, training data acts as source code, training is compilation, and benchmarking is testing.This correspondence enables failure-driven repair rather than indiscriminate data addition.
  • Programming with Data: A three-level knowledge structure of atomic concepts, relational triples, and reasoning chains is shared by training data and benchmarks to make failures traceable to data deficits.The shared structure turns evaluation into an actionable diagnostic process.
  • Evaluation and resources: 16 disciplines are covered by ProDa, whose released resources include 227k key concepts, 16k evaluation items, and 160k synthesized training samples.Experiments span Llama and Qwen models from 3B to 32B parameters, with every tested model and scale improving across debugging iterations.

2 Programming with Data

Programming with Data closes the data-engineering loop by using a shared three-level knowledge representation to connect training data, evaluation, diagnosis, and repair. It operationalizes this correspondence through ProDa’s Builder, Tester, and Debugger components.

  • 2.1 Formal correspondence: A shared structured knowledge representation makes the LLM data-engineering lifecycle parallel software development: corpus requirements become training data, training becomes compilation, and benchmarks become tests.The raw corpus specifies the knowledge scope; synthesized data encodes concepts and relationships; training produces model weights; and benchmarks verify implementation.
  • 2.1 Formal correspondence: The shared specification has three layers: L1 concepts, L2 typed relations, and L3 multi-step reasoning chains.L1 entries define atomic disciplinary vocabulary, L2 entries represent semantic relations, and L3 chains decompose inferential pathways into annotated steps.
  • 2.1 Formal correspondence: Benchmark failures on L3 chains decompose into concept gaps involving missing L1/L2 knowledge or reasoning deficits involving incorrect composition of the required inferential sequence.This decomposition makes failures traceable to specific knowledge components or reasoning steps rather than undifferentiated data insufficiency.
  • 2.2 Operational pipeline: ProDa operationalizes the framework as an automated Builder–Tester–Debugger pipeline governed by CORE standards for contextualized, organized, rigorous, and evolving data synthesis.The Builder extracts knowledge and synthesizes training data, the Tester constructs reasoning-based benchmarks, and the Debugger diagnoses failures for targeted repair.
  • 2.2 Operational pipeline: Structural coupling lets the Debugger trace a failed benchmark item backward through its L3 chain, L2 relations, and L1 concepts to identify under-represented or absent training knowledge and generate a targeted patch.The Builder, Tester, and Debugger remain coupled through the shared knowledge structure, enabling failure-driven repair.

3 Results

Across 16 disciplines, the study constructs a structured knowledge base and aligned ProDa-16 benchmark, showing that automatically synthesized training data yields competitive domain competence and that targeted debugging repairs model failures. Repair gains are systematic, sample-efficient, and traceable to concept-level omissions or broken reasoning chains through L1/L2/L3 grounding.

  • Knowledge representation: Quality filtering compresses 117,000 textbook-grade documents into 48,000 high-quality chunks totaling approximately 1.5 billion tokens.The retained corpus represents a 10:1 compression of the source documents.
  • Knowledge representation: The extracted knowledge organizes L3 reasoning chains over L2 relational statements and L1 atomic concepts under a strict reachability invariant.L2 statements encode causal mechanisms, definitional relationships, and prerequisite conditions between L1 concepts.
  • Benchmark validation: ProDa-16 covers all 16 target disciplines, with per-discipline node counts ranging from approximately 7,000 to over 45,000.The benchmark’s score distribution forms an adjacent-score ladder, while every discipline’s median accuracy exceeds the 25% four-choice chance baseline.
  • Initial training results: Qwen-2.5-7B-V1 scores 65.86% versus 62.31% for the official Instruct model, while Qwen-2.5-32B-V1 scores 76.54% versus 73.61%.A single fine-tuning round on automatically synthesized data produces domain competence competitive with industrial alignment pipelines across most model scales.
  • Debugging and repair: +0.77 points to +32.67 points are the V1-to-V2 gains across nine models, with larger improvements for models starting from lower V1 performance.Every model improves on average; Qwen3-14B gains +0.77 points and Llama-3.1-8B gains +32.67 points.
  • Diagnostic case studies: Targeted patches correct optics hallucinations, legal reasoning-chain errors, and biomedical concept omissions through L1/L2/L3-grounded diagnosis and verified V2 correction.The three cases share a closed loop of incorrect-option identification, diagnostic classification, targeted patch generation, and verification.

4 Methods

The methods construct a three-level knowledge structure from curated disciplinary text, then derive structurally linked benchmarks and training data from it. Model failures are classified and converted into targeted repair batches, completing an iterative debugging loop.

  • Corpus curation: The source collection spans 16 disciplines and is filtered, classified, segmented, and scored to select academically substantive, inferentially rich text chunks.Selection emphasizes academic authority, disciplinary diversity, contamination control, academic level, and reasoning type.
  • Knowledge representation: The shared knowledge structure K comprises atomic concepts K1, typed relations K2, and multi-step reasoning chains K3 with source provenance and inferential annotations.Relations connect concepts through types such as specialization, causality, prerequisite, and contrast; reasoning chains annotate each inferential transition.
  • Knowledge extraction: Top-down extraction begins with reasoning chains, decomposes them into relations and concepts, and guarantees every concept and relation participates in a testable chain.This reachability property prevents orphan entries that would be untestable and therefore undebuggable.
  • Artifact synthesis: Benchmarks are synthesized from L3 chains with structural metadata and adversarial distractors, while training samples are synthesized from L1 and L2 entries in multiple formats.Instance-level separation prevents benchmark answers from being obtained by verbatim recall of individual training samples.
  • Debugging and refinement: After training and evaluation, the Debugger classifies errors as concept gaps or reasoning deficits and generates contrastive reinforcement or chain-of-thought repair batches.The repaired corpus S′ is retrained, making each benchmark failure a traceable data repair in an evolving debugging loop.

5 Related Work

Prior work has advanced synthetic data generation, LLM evaluation, and autonomous self-improvement, but specialized-domain reliability remains limited by weak grounding and open-loop processes. This work addresses these gaps through closed-loop, code-level data engineering that uses authoritative corpora to diagnose and repair failures.

  • Synthetic Data Generation: Synthetic-data methods shifted LLM training emphasis from data quantity to quality, progressing from instruction bootstrapping to iterative instruction complication and textbook-quality data.Self-Instruct and Alpaca bootstrapped instruction data from off-the-shelf LLMs; WizardLM introduced Evol-Instruct, while LIMA and Phi showed that carefully curated or textbook-quality synthetic data can support strong performance.
  • Synthetic Data Generation: Existing synthesis pipelines often lack ground-truth anchoring, and general NLP capability does not guarantee high-quality specialized-domain data.AgoraBench highlighted differences among generators, motivating a closed-loop process that treats synthetic data as source code derived from corpora and enforces debugging for domain fidelity.
  • LLM Evaluation: LLM evaluation combines broad static benchmarks such as MMLU and Big-Bench with specialized reasoning datasets such as GSM8K and increasingly uses LLM-as-a-Judge for open-ended generation.These suites assess diverse domains, while GSM8K focuses on multi-step mathematical reasoning and MT-Bench exemplifies judge-based evaluation.
  • Autonomous Model Improvement: Autonomous-improvement research progressed from RLHF and RLAIF feedback loops to iterative generate–filter–finetune frameworks such as STaR and ReST.This trajectory shifts supervision from human-annotated signals toward self-generated signals.
  • Autonomous Model Improvement: Self-play cannot reliably acquire domain expertise without authoritative grounding, motivating code-level data engineering that synthesizes executable patches from authoritative corpora.LANCE and EVOLVE adopt data-centric perspectives, while this work targets diagnosed failures beyond sample-level self-filtering.

6 Discussion … B.5.3 Case Study: Sociology

Programming with Data aligns fine-tuning data engineering with software development by sharing a knowledge structure across training, evaluation, and diagnosis. The paper presents this as a general, traceable, and repairable blueprint, supported by structured extraction procedures and cross-disciplinary examples.

  • 6 Discussion: Fine-tuning targets defined competencies, enabling specific model behaviours to be traced through a shared structure linking training data, evaluation, and diagnosis.The framework distinguishes this from pre-training, where tracing behaviours to individual documents is neither feasible nor necessary.
  • 6 Discussion: The framework maps knowledge structure to requirements, synthesized data to implementation, benchmarks to tests, and diagnosis to fault localization.This correspondence provides the operative software-engineering analogy underlying Programming with Data.
  • 6 Discussion: Existing methods address isolated parts of the loop but do not jointly diagnose and repair deficiencies in knowledge-producing training data.Synthetic-data, self-improvement, data-centric, and diagnostic-benchmark approaches each leave structural gaps in the complete feedback cycle.
  • 6 Discussion: The work establishes Programming with Data’s macro-level architecture while leaving each module open to domain-specific improvement.The discussion identifies retrieval-augmented generation and mechanistic interpretability as particularly relevant intersections.
  • 6 Discussion: A shared knowledge specification makes the relationship between fine-tuning data and model behaviour transparent, diagnosable, and repairable.The paper characterizes the result as a discipline-agnostic blueprint rather than a finished system.
  • A.2 Chunk quality scoring rubrics: Six quality dimensions guide corpus processing, including reasoning depth, prerequisite density, scenario applicability, counter-intuitive content, knowledge synthesis, and breakpoint smoothness.These dimensions support reasoning-chain extraction, difficulty control, application-oriented samples, hard negatives, relational extraction, and boundary-quality gating.
  • B Knowledge Structure Extraction: L3 extraction identifies one complete primary reasoning chain, L2 converts adjacent chain steps into atomic relations, and L1 canonicalizes and defines linked concepts.The procedures enforce continuity, adjacency, textual support, contextual definitions, and full traceability across the hierarchy.
  • B.5 Cross-discipline examples — Biology, Chemistry, Sociology: The cross-discipline examples show the hierarchy capturing mechanisms in biology, quantitative electrolysis laws in chemistry, and visual framing in sociology.The sociology example extracts Parergon (Frame), demonstrating application to interpretive social-science material.

C Data Synthesis and Benchmark Construction … Prohibitions

The paper constructs fine-tuning and benchmark data from atomic knowledge points and reasoning chains, emphasizing broad coverage, natural language, deep causal understanding, and strict output constraints. Distractors are designed to expose specific logical errors while remaining plausible and structurally parallel.

  • C.1 SFT generation prompts (QA / Choice / TF); Variety: Fine-tuning prompts generate single-choice, multiple-choice, question-answer, and true/false data from atomic L1 concepts and L2 factual statements.The outputs are intended to test precise domain understanding in natural, educational language rather than reproduce database structures.
  • 2. Single-Choice Question Construction; 3. Multiple-Choice Question Construction: Single-choice items map one L2 Subject–Predicate–Object statement to a stem, one correct answer, and three plausible distractors, whereas multiple-choice items combine related facts around one L1 concept.Multiple-choice questions use 2–4 related L2 statements, typically provide 2–3 correct answers, and require detailed explanations for correct and incorrect options.
  • 2. Atomic Focus; 3. Natural Language Refinement (Critical); 4. Contextualization: QA pairs focus on one L2 statement, rephrase it naturally, and replace pronouns with specific nouns so each question is contextualized and unambiguous.The prompts explicitly reject mechanical JSON-to-sentence conversion and favor classroom-style wording.
  • 2. True Statement Construction; 3. False Statement Construction: True/false data rephrases accurate L2 statements as true items and creates educational false items by reversing relationships, introducing misconceptions, or changing facts.False statements must remain fluent and avoid obvious truth-value giveaways.
  • C.2 Benchmark item construction prompts; Goal; 1. Depth and Complexity; 2. Question Type (Priority Order); 5. Question Generation Strategy: Benchmark MCQs are generated from reasoning chains to test causal relationships, logical sequences, mechanisms, synthesis, and application rather than trivial factual recall.The construction process prioritizes process reasoning, causal analysis, critical understanding, and application in that order.
  • 3. Question Length and Detail; 4. Options and Distractor Quality; 6. Prohibitions (Strictly Avoid): Each benchmark question uses complete contextual detail, four substantial parallel options, exactly one unambiguous answer, and distractors based on misconceptions, missing preconditions, or incorrect reasoning sequences.Questions are typically 40–100 words, options 20–60 words, and all fields must remain in the exact input language without vague or incomplete citations.
  • C.3 Distractor generation strategy; Distractor Types (Logical Traps); Option Quality & Structure; Prohibitions: Distractors must be semantically related, substantial, similarly styled, and plausible, specifically instantiating reversed causality, missing preconditions, wrong step order, overgeneralization, or misattributed mechanisms.The strategy excludes obviously wrong options so questions test deep comprehension instead of recognition.

C.4 SFT data statistics … Output Format

The SFT_v1 dataset is balanced across 16 disciplines while using predominantly open-ended questions. Diagnostic classification and targeted patch-generation prompts distinguish conceptual gaps from reasoning deficits and produce structured corrective training samples.

  • C.4 SFT data statistics: SFT_v1 is perfectly balanced across 16 disciplines, with open-ended questions comprising 60.0%, single-choice 23.2%, true/false 10.0%, and multiple-choice 6.8%.The dataset includes varied ratios of multiple-choice, open-ended, single-choice, and true/false questions.
  • D.1 Diagnostic prompt: The diagnosis prompt instructs an evaluation expert to classify model failures as conceptual gaps or reasoning deficits and enforce a strict JSON schema.This classification supplies the basis for subsequent targeted patch generation.
  • D.2 Concept gap prompt: Concept-gap patching generates publication-grade samples with precise definitions, explicit contrasts, rich context, and enough detail for a strong student to learn independently.The prompt’s role is to eliminate conceptual misunderstandings identified in diagnostic reports.
  • Input: Concept-gap inputs require generated questions to address diagnosed error patterns, contrast correct and incorrect interpretations, and reinforce concept boundaries and conditions.The prompt uses a target concept, definitions, facts, related examples, and a question limit as inputs.
  • Format-Specific Requirements: Error-driven concept patching analyzes misconceptions, directly challenges them, includes at least 2 corrective questions, uses contrastive examples, and covers all provided L2 facts.Format-specific requirements further constrain open-ended, multiple-choice, and true/false answers.
  • D.3 Capability deficit prompt: Capability-deficit patching converts diagnostic insights into teaching-grade Chain-of-Thought samples that build multi-step reasoning with explicit, justified intermediate steps.Its quality bar is that a strong student could learn the reasoning pattern from one sample.
  • Input: Reasoning-deficit inputs target missing steps, broken logical chains, and failed cause-effect connections, requiring the exact failed path, necessary intermediate steps, and explicit justification at each hop.The input includes a target concept, knowledge snippet, diagnosis note, and maximum question count.
  • Advanced Reasoning Patterns to Apply: Capability patches require diagnostic alignment, at least 3 reasoning hops, intermediate checkpoints, self-validation, and patterns such as causal chains, elimination, construction, and comparison.Output formats specify step-based answers for open-ended, multiple-choice, and true/false questions, returned as strict JSON arrays with required fields.

D.4 Data mixing and replay strategy · D.5 Diagnostic report example · Subject-wise Performance (Truncated)

The protocol begins with a uniform 160,000-sample baseline, then reallocates data according to diagnostic errors, generates targeted repairs, and uses disjoint replay to preserve mastered concepts. A Qwen2.5-7B-SFT diagnostic report illustrates the evaluation outputs, including 65.86% overall accuracy and subject-level variation.

  • D.4 Data mixing and replay strategy: Round 1 uses 160,000 training samples uniformly distributed across 16 categories, with 10,000 samples per category.The baseline model is evaluated against the benchmark to produce category-level error counts.
  • D.4 Data mixing and replay strategy: Round 2 keeps the dataset at 160,000 samples but replaces uniform allocation with error-proportional category quotas.A category representing 30% of benchmark errors receives exactly 30% of the Round 2 samples.
  • D.4 Data mixing and replay strategy: Targeted repair generation creates samples for every diagnosed Conceptual Gap and Capability Deficit identified in the diagnostic report.The repairs explicitly address the corresponding failure rather than adding data indiscriminately.
  • D.4 Data mixing and replay strategy: Exactly 20 repair samples are generated per diagnosed error, using a 6:3:1 ratio across open-ended QA, multiple-choice, and true/false formats.The format distribution is 60% open-ended QA, 30% multiple-choice, and 10% true/false.
  • D.4 Data mixing and replay strategy: L2-disjoint experience replay fills quota deficits with correctly answered Round 1 questions while preventing catastrophic forgetting of previously mastered concepts.Each sample is mapped to an L2 factual statement under the L2 ID-disjoint constraint.
  • D.5 Diagnostic report example: The Qwen2.5-7B-SFT diagnostic report records 65.86% overall accuracy, with 9,268 correct answers out of 14,072 and 4,804 error samples.The report is timestamped 20260210_220124.
  • Subject-wise Performance (Truncated): Subject-wise results show accuracies of 64.6%, 65.0%, and 63.0% for Subject-001-en, Subject-002-en, and Subject-003-en, respectively.Their corresponding error counts are 354, 350, and 370, with 1,000 total samples per subject; 13 additional subjects are omitted.

Error Pattern Analysis

Error analysis found more capability-deficit than concept-gap samples, while the benchmark was overwhelmingly composed of multiple-choice questions rather than single-choice questions.

  • By Issue Type: 3,093 capability_deficit samples exceeded 1,509 concept_gap samples in the issue-type breakdown.The analysis categorized 4,602 samples by issue type.
  • By Question Type: 4,787 multiple-choice questions far outnumbered 17 single-choice questions in the question-type breakdown.The analysis categorized 4,804 questions by question type.

Representative Error Sample: Index 3 (Concept Gap)

Index 3 illustrates a concept gap in Fresnel half-wave-strip analysis for single-slit Fraunhofer diffraction. The model added an incorrect option because it misunderstood the conditions governing bright and dark fringes, motivating review of diffraction and interference principles.

  • Concept Gap: The ground-truth answer was A,B,D, whereas the model predicted A,B,C,D.The extra option C is the model’s specific error.
  • Concept Gap: The error reflects a misunderstanding of interference in diffraction patterns.The diagnosis identifies interference in diffraction patterns as the key concept.
  • Concept Gap: The model incorrectly included option C because it misunderstood the conditions required for bright and dark fringe formation.The recommended repair is to review the principles of diffraction and interference.

Representative Error Sample: Index 7 (Capability Deficit)

Index 7 illustrates a capability deficit in which the model adds an incorrect answer choice when deriving equilibrium equations for a curved slender rod. The failure reflects inadequate prerequisite assessment and motivates multi-step reasoning training.

  • Error diagnosis: The model predicted A,B,C,D instead of the ground-truth answer A,B,C for equilibrium equations of a curved slender rod.The incorrect addition of option D identifies the representative capability failure.
  • Error diagnosis: The key concept is the derivation mechanism of equilibrium equations for curved slender rods.The error concerns accurately applying this specialized concept.
  • Recommended repair: The diagnosis attributes the error to failure to assess the prerequisites for the equilibrium equations accurately.The model’s reasoning therefore requires targeted improvement rather than indiscriminate data expansion.
  • Recommended repair: The recommendation is to enhance reasoning capabilities by training on multi-step examples.This intervention is intended to address the reasoning deficit revealed by the incorrect option selection.

Other Samples · E Experimental Configuration

Figure 28 presents an automated diagnostic report for Qwen2.5-7B-SFT, combining quantitative error-pattern analysis with qualitative diagnoses of specific failures to guide refinement.

  • Other Samples: Figure 28 showcases an automated diagnostic report for Qwen2.5-7B-SFT.The report is presented as an example of automated diagnosis.
  • Other Samples: The report provides a quantitative breakdown of error patterns.Its quantitative analysis characterizes how failures are distributed.
  • Other Samples: It also gives qualitative diagnoses for specific failure cases.These diagnoses complement the quantitative error analysis.
  • Other Samples: The diagnostic report is designed to guide subsequent refinement.Its analyses support decisions about how to improve the system after failures are identified.
  • Other Samples: Specific failure cases are examined through both error-pattern measurements and qualitative diagnosis.The example links aggregate quantitative reporting with case-level interpretation.
  • Other Samples: Qwen2.5-7B-SFT serves as the evaluated model in the showcased diagnostic example.The figure identifies the model whose performance the report analyzes.

E.1 Training hyperparameters and infrastructure … 2. Strict Scoring Rules & Post-processing

The paper standardizes LoRA fine-tuning and OpenCompass evaluation to improve efficiency, reproducibility, and deterministic scoring. Evaluation uses constrained generation, strict option post-processing, exact-match scoring, and representative MMLU and C-Eval subsets.

  • E.1 Training hyperparameters and infrastructure: All fine-tuning used LLaMA-Factory with LoRA across base models and bf16 mixed precision for computational efficiency and numerical stability.These choices standardized the training infrastructure while balancing efficiency and model performance.
  • E.2 Evaluation protocols: Evaluations were conducted with strict configurations in OpenCompass to ensure reproducibility.OpenCompass provided the common evaluation framework for all reported results.
  • 1. OpenCompass Setup: Inference used local HuggingFace deployment through OpenCompass GenInferencer.This setup kept evaluation within a locally deployed inference environment.
  • 1. OpenCompass Setup: Standardized zero-shot prompts required models to output only option letters.Restricting outputs to option letters supported consistent downstream answer extraction.
  • 1. OpenCompass Setup: Greedy decoding with do_sample = False and temperature = 0 eliminated sampling variance.The decoding configuration made repeated evaluations deterministic.
  • 1. OpenCompass Setup: Generation was capped at 15 tokens to optimize concise option extraction.The token limit constrained outputs before post-processing.
  • 2. Strict Scoring Rules & Post-processing: parse_multi_choice_answer removed reasoning traces, deduplicated valid options, and alphabetized formatted outputs such as "A,B,C".This post-processing converted generative responses into standardized multiple-choice predictions.
  • 2. Strict Scoring Rules & Post-processing: Exact-match scoring awarded 1 only for a perfect normalized match, while any missing, extra, or incorrect option received 0.The protocol provided no partial credit and evaluated the complete option set.
Loading 2604.24819v1…