PaperlayerView exact revision on arXiv

Source-linked AI summary

Function-Level Execution Feedback for Code Preference Optimization

Idris Nechnech, Sehwan Kim, Jimin Seo, Yeongoon Kim, Minhae Oh, Sangwoo Hong, Jungwoo Lee

arXiv:2608.23632v1cs.AIcs.SE

TL;DR

Code process supervision lacks a standard step definition, making feedback for partially correct programs coarse. STEP-KTODER defines steps as module-level functions, labels them with generated unit tests, and combines that supervision with whole-program KTO; it improves code models over outcome-only KTO and DPO, while judge-based labels degrade optimization.

  • Problem

    Code lacks a standardized sequence of intermediate steps, so whole-program supervision provides coarse feedback for partially correct programs.

  • Method

    STEP-KTODER decomposes programs into module-level functions, assigns execution-based binary labels from generated unit tests, and combines function-level supervision with outcome-level KTO.

  • Results

    STEP-KTODER consistently improves over outcome-only KTO and DPO across code benchmarks, with gains up to +26.7% on BigCodeBench Hard and +27.0% on LiveCodeBench.

  • Takeaways & Limitations

    Function-level execution feedback offers a practical path toward process supervision for code generation, while execution-based labels are central to the reported gains.

  • Takeaways & Limitations

    The method is most natural for solutions decomposable into independently testable functions and was evaluated mainly on already post-trained instruction-tuned code models.

Abstract

from arXiv · show

Process supervision has improved mathematical reasoning, where intermediate steps are naturally expressed as chains of thought. In code generation, however, process supervision remains underexplored because there is no standard notion of a step. Supervision can target lines, reasoning traces, or program states, making it unclear what to label and optimize. We propose STEP-KTODER, a framework for code preference optimization that defines steps as module-level functions in decomposed multi-function programs and assigns binary correctness labels via automatically generated unit tests. Our method provides a code-specific instantiation of stepwise KTO, combining function-level process supervision with outcome-level feedback on the full program. We evaluate on HumanEval(+), MBPP(+), BigCodeBench, and LiveCodeBench, showing that STEP-KTODER improves over outcome-only KTO and DPO. Further analysis shows that execution-based labels are essential: LLM-as-a-judge annotations systematically over-predict function failures, corrupt positive step labels, and degrade downstream preference optimization. Code is available at: https://github.com/inechnech/STEP-KTODER.

1 Introduction

STEP-KTODER addresses the coarse and ambiguous nature of code process supervision by treating independently testable module-level functions as steps. It combines execution-based function feedback with whole-program outcome feedback for preference optimization.

  • Outcome-only code supervision labels an entire program undesirable even when only a narrow component fails, obscuring which functions are correct or responsible.
  • Unlike mathematical reasoning, code lacks a standardized sequence of intermediate steps, motivating a function-based definition of process supervision.
  • STEP-KTODER decomposes reference solutions into module-level functions, generates unit tests, and uses execution outcomes to assign binary local-correctness labels.
  • The framework jointly optimizes outcome-level KTO and function-level supervision, reinforcing locally correct functions while penalizing locally incorrect ones.
  • Across benchmarks, STEP-KTODER improves post-trained code models, whereas LLM-as-a-judge step labels weaken supervision and degrade performance.

2 Related Work

Prior work provides process supervision, localized preference updates, modular decomposition, and execution-guided inference at different granularities. STEP-KTODER instead uses independently testable functions as steps and incorporates execution labels directly into offline binary preference optimization.

  • Mathematical process supervision and Step-KTO optimize intermediate reasoning quality alongside final-answer correctness using stepwise feedback.
  • Code alignment methods have localized feedback to token spans, targeted regions, lines, or whole-program preference pairs, but use different supervision granularities.
  • STEP-KTODER’s distinctive contribution is defining steps as semantically meaningful, independently testable functions within a binary preference optimization objective.
  • Related code-generation work promotes modularity through fill-in-the-middle training, AST-based blocks, hierarchical prompting, and joint code-test generation.
  • Execution-grounded methods such as S∗ and ORPS guide inference-time search or selection, whereas STEP-KTODER uses function-level execution during offline data construction.

3 Method

STEP-KTODER decomposes reference solutions into independently testable functions, labels candidate outcomes and function correctness, and combines outcome-level KTO with masked step-level supervision. This localized signal reinforces correct functions while penalizing faulty ones, with conflict cases retained or masked during dataset construction.

  • 3.1 Problem Formulation: Reference solutions are rewritten into decomposed multi-function programs, with each function treated as a step.The decomposition preserves the solution structure needed for function-level supervision.
  • 3.1 Problem Formulation: Each candidate receives an outcome label for the full test suite and step labels indicating whether each function passes its unit tests.Unavailable tests produce a null label and do not contribute to step loss.
  • 3.3 Step-KTO Objective: STEP-KTODER combines outcome-level KTO with masked function-level KTO to reinforce locally correct functions and penalize locally incorrect ones.Samples without supervised steps fall back to outcome-only KTO.
  • 3.4 Data Construction Pipeline: The data pipeline decomposes reference solutions, generates and validates function-level tests, samples eight on-policy candidates per skeleton, and labels candidates by execution.Functions without valid tests receive null labels and are masked from step supervision.
  • 3.4 Data Construction Pipeline: A failing program can preserve locally correct functions while penalizing only the faulty function, unlike outcome-only KTO, which penalizes every function span.The pipeline also compares conflict-preserving and masked variants for mismatches between local and end-to-end correctness.
  • 3.4 Data Construction Pipeline: Validated-test availability reaches 83.2% across testable decomposed functions, while most problems have near-full availability.Per-problem availability is summarized in Figure 3.

4 Experimental Setup

The experiments fine-tune instruction-tuned code models with LoRA on training data constructed from TACO and APPS, comparing STEP-KTODER with several preference-optimization baselines across seven benchmarks. Training uses one epoch and distinct learning rates for KTO-based methods and DPO.

  • 4 Experimental Setup: Models are fine-tuned for 1 epoch with LoRA and global batch size 16; KTO methods use 1 × 10^-6, while DPO uses 5 × 10^-7.The default step-loss weight is λstep = 1.0.
  • 4 Experimental Setup: Training data comes from the TACO and APPS training splits and uses decomposed references plus on-policy candidate completions.Qwen2.5-Coder-32B-Instruct performs decomposition, while target models generate candidates.
  • 4 Experimental Setup: Step-supervised rows require at least two supervised function labels, while fallback KTO rows use outcome-only KTO.Average steps per active row is computed only over step-supervised rows.
  • 4 Experimental Setup: Evaluation covers HumanEval, HumanEval+, MBPP, MBPP+, BigCodeBench Full, BigCodeBench Hard, and LiveCodeBench release v4_v5.All evaluations use greedy decoding with vLLM.
  • 4 Experimental Setup: STEP-KTODER is compared with the original instruction-tuned model, DPO, KTO, and Target-DPO.Baseline construction details are provided in the paper’s appendix.

5 Results

STEP-KTODER improves over outcome-only KTO across model families, with the clearest gains on harder benchmarks and advantages over Target-DPO. Ablations show that preserving outcome–step conflicts and calibrating step supervision are important for performance.

  • Main results: +11.5% on BigCodeBench Hard and +7.7% on LiveCodeBench for Qwen2.5-Coder-1.5B-Instruct over KTO.For Qwen2.5-Coder-3B-Instruct, the corresponding gains are +11.3% and +9.5%.
  • Main results: +4.9% on BigCodeBench Hard and +3.2% on LiveCodeBench for DeepSeek-Coder-6.7B-Instruct over KTO.STEP-KTODER improves monotonically over DPO and KTO on BigCodeBench Hard.
  • Robustness: STEP-KTODER’s gains over KTO persist on BigCodeBench Hard and LiveCodeBench across three seeds.On easier benchmarks, it remains competitive, often achieving the best or tied-best result.
  • Comparison with Target-DPO: STEP-KTODER exceeds Target-DPO on both hard benchmarks for Qwen2.5-Coder-3B-Instruct and DeepSeek-Coder-6.7B-Instruct.The gains are +18.5% on BigCodeBench Hard for both models and +13.4% and +17.0% on LiveCodeBench, respectively.
  • Ablations: Masking outcome–step conflicts reduces STEP-KTODER to KTO-level performance on BigCodeBench Hard and decreases LiveCodeBench performance.The ablation also finds λstep = 1.0 and at least two supervised steps to be preferred settings.

6 Analysis

Function-level labels provide information that program-level outcomes cannot express, especially when local and global correctness disagree. Repair and label-quality analyses indicate that execution-based supervision localizes relevant faults more reliably than judge-based labels.

  • 6.1 Why Function-Level Supervision Helps: Program-level DPO and KTO cannot distinguish which functions caused a generated program’s success or failure.STEP-KTODER instead reinforces locally correct functions and penalizes locally incorrect ones within the same program.
  • 6.1 Why Function-Level Supervision Helps: Conflict preservation supplies training directions unavailable from outcome-level supervision when local and global labels disagree.A locally negative function can be penalized even when the full program passes.
  • 6.2 Function-Level Labels Localize Failures: 74.9% of failing candidates are repaired by replacing locally negative functions, outperforming random and locally positive replacements.The intervention supports execution-based labels as indicators of repair-relevant faults.
  • 6.3 Execution-Based Labels Versus LLM Judgments: Execution-based labels agree with GPT-5.4 mini judgments on 73.2% of function labels, but aggregate agreement conceals asymmetric errors.The analysis reports that false-negative bias corrupts passing step labels needed for stable function-level training.

7 Conclusion

STEP-KTODER defines code-generation process steps as module-level functions and combines execution-based function labels with outcome-level KTO. Experiments and ablations report gains over outcome-only methods while identifying local/global conflict preservation and execution-based labels as central.

  • Conclusion: STEP-KTODER defines process-supervision steps as module-level functions in decomposed programs.It combines outcome-level KTO with execution-based function labels to provide localized feedback.
  • Conclusion: Experiments on post-trained code models show consistent gains over outcome-only KTO and DPO.Ablations identify preserving local/global label conflicts and using execution-based labels as central to these gains.

8 Limitations

STEP-KTODER is most suitable for decomposable, independently testable solutions and currently targets already post-trained instruction-tuned code models. Its step labels depend on automatically generated tests that are validated by execution but not formally verified.

  • Model and data scope: STEP-KTODER is most natural for solutions decomposable into independently testable functions.The approach is less suitable when problems lack meaningful function-style structure.
  • Model and data scope: Experiments focus on instruction-tuned code models that have already undergone substantial post-training.Applying function-level supervision earlier during SFT or on larger base-model training runs remains future work.
  • Difficulty distribution: The filtering pipeline biases training data toward problems with clear function-style structure and underrepresents harder competition problems.Extending the pipeline to more complex I/O formats and harder regimes is identified as a next step.
  • Unit-test quality: Automatically generated unit tests are validated by execution but not formally verified.A mutation audit rejects 92.6% of semantically perturbed reference implementations, while repair results show negative labels can identify failure-causing functions.
  • Unit-test quality: Stronger automated test-generation or verification methods could further improve step-label quality.The paper reports execution-based labels as more reliable than LLM judgments, but leaves room for better automated validation.

Ethical Considerations

The paper uses publicly available research-use datasets and benchmarks, licensed research models, and candidate-level preference construction for KTO, DPO, and STEP-KTODER. Training uses LoRA, while seed and general-purpose-model experiments assess robustness.

  • Data and model use: The models and datasets used are licensed or distributed for academic research and research use.The models are Qwen2.5-Coder and DeepSeek-Coder; datasets include TACO and APPS, with public evaluation benchmarks.
  • Candidate construction: For each problem, eight target-model completions yield up to one positive and one negative training candidate.Selection uses outcome and function-level step labels.
  • Candidate construction: Positive candidates prioritize clean step-level signals, while negative candidates prioritize mixed passing and failing functions that localize failures.Fallbacks use candidates with locally failing functions or generated-context labels.
  • Label validation: Reference-context and generated-context labels agree on 89.4% of 43,458 comparable supervised steps.Agreement remains 89.2–89.6% across models.
  • Preference optimization: Unlike DPO, KTO and STEP-KTODER can train on unpaired positive or negative candidates.DPO requires paired preferences, whereas KTO-based methods use individually labeled candidate rows.
  • Robustness: Seed-matched STEP-KTODER exceeds KTO in all three runs on BigCodeBench Hard and LiveCodeBench.Mean gains are 0.015 and 0.014, respectively; differences on the other five benchmarks are at most 0.002.
  • Robustness: With Qwen2.5-7B-Instruct, STEP-KTODER exceeds KTO on six of seven benchmarks, with largest gains on BigCodeBench Hard and LiveCodeBench.The experiment uses on-policy candidates and validated ground-truth anchors.

G Dataset Statistics

The dataset pipeline filters for decomposable problems, validates decompositions and generated tests by execution, and produces training sets skewed toward easier tasks. Local labels expose both useful hidden defects and limits caused by incomplete tests, while the joint objective performs best overall.

  • Difficulty distribution: The pipeline retains function-style problems and excludes many high-difficulty or monolithic-interface examples.TACO contributes easy through medium_hard problems, while APPS retains introductory and interview problems but excludes competition problems.
  • Ground-truth validation: Ground-truth decomposition validation is 88.9% for APPS versus 43.9% for TACO.The TACO decline is concentrated in harder buckets and reflects difficulty producing semantically equivalent decompositions with complex global state.
  • Final training sets: Final training sets skew toward easier problems because decomposition-friendly distributions and difficulty-dependent validation rates shape retention.This skew is discussed as a limitation of the resulting training data.
  • Unit-test quality: Validated tests remain for 83.2% of testable decomposed functions, with 84.4% mean per-problem availability.Covered functions have around 5.5 valid tests on average, and first-pass parsing exceeds 98%.
  • Unit-test quality: Retained tests reject 92.6% of semantically perturbed reference implementations in the mutation-sensitivity audit.The audit covers 11,705 functions with at least one valid generated test.
  • Local/global supervision: An incomplete local test can assign positive local labels to an incorrect full program, preserving the need for outcome-level supervision.The Pair of Shoes example tests only empty input and misses the candidate’s defect.
  • Local/global supervision: A negative local label can expose a hidden defect even when the candidate passes dataset-provided tests.In Task Scheduler, calculate_min_intervals understates required intervals for some inputs.
  • Repair intervention: Replacing local-negative functions repairs 78.6% of wrong-answer failures on the combined APPS and TACO subset.This supports local-negative labels as indicators of semantic errors.

M Off-Policy Training Data in Cross-Family Training

Off-policy candidates reduce performance for both KTO and STEP-KTODER on the hardest benchmark, but STEP-KTODER is less affected on LiveCodeBench. The paper attributes this relative robustness to shorter-span function-level signals under policy mismatch.

  • Experimental design: The off-policy comparison keeps decomposed skeletons and unit tests fixed while changing only the model that generates candidate completions.Qwen-generated candidates are fully off-policy relative to the DeepSeek reference model.
  • Hard-benchmark effects: On LiveCodeBench, KTO falls from 0.127 to 0.112, while STEP-KTODER declines from 0.131 to 0.127.The smaller STEP-KTODER regression suggests greater robustness to policy mismatch.
  • Interpretation: The paper attributes the BigCodeBench Hard regression to poorly calibrated log-ratio rewards under candidate-distribution divergence.Function-level signals may partially compensate because they operate over shorter token spans.

N LLM-as-a-Judge vs. Execution-Based Step Labels

The comparison finds that execution-based function labels are more reliable than GPT-5.4 mini judgments for STEP-KTODER, whose judge errors are strongly biased toward predicting failure. Replacing execution labels substantially harms downstream performance, while the appendix also documents prompts for decomposition, testing, candidate generation, and judging.

  • Experimental setup: Execution-based labels are compared with GPT-5.4 mini annotations using agreement statistics and a downstream training experiment on Qwen2.5-Coder-3B-Instruct.The agreement analysis samples N=88,469 supervised functions, while the downstream experiment substitutes judge labels into the STEP-KTODER training set.
  • Agreement analysis: Overall agreement is 73.2%, with 94.1% agreement on execution-labeled failures but only 52.2% on execution-labeled passing functions.Agreement is high in both failing-function strata—92.4% for (o=1, z=0) and 94.8% for (o=0, z=0)—but approximately 52% for passing strata.
  • Agreement analysis: The judge exhibits a false-negative bias, frequently flagging true failures while hallucinating issues in correct implementations and corrupting clean positive step labels.This asymmetry preserves many negative conflict labels but damages the larger pool of positive function labels needed for function-level training.
  • Downstream impact: Relative performance drops are 25.8% on BigCodeBench Hard and 11.8% on LiveCodeBench when execution labels are replaced by GPT-5.4 mini judgments.BigCodeBench Hard falls below the outcome-only KTO baseline, showing that moderate aggregate agreement can conceal downstream-harmful systematic bias.
  • Prompt specifications: The data-construction appendix specifies prompts for decomposing reference solutions, generating function-level unit tests, sampling candidate completions, and judging function correctness.The testing prompt returns only test inputs, with expected outputs obtained by executing the reference implementation; candidate generation fills skeletons whose function bodies are pass.
Loading 2608.23632v1…