PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Execution Simulation Improves Coding Models

Gallil Maimon, Ori Yoran, Felix Kreuk, Michael Hassid, Gal Cohen, Pierre Chambon, Yossi Adi

arXiv:2604.03253v1cs.CLcs.LG

TL;DR

Code LLMs often struggle to faithfully estimate program execution, limiting the reliability of generated code. This paper trains models to simulate execution step by step and use those predictions for self-verification and iterative self-fixing, yielding consistent improvements on competitive programming benchmarks.

  • Problem

    Current code LLMs often fail to faithfully simulate runtime execution, and translating execution prediction into consistent programming-task gains remains unresolved.

  • Method

    The approach combines supervised fine-tuning on natural-language execution traces with verifiable-reward reinforcement learning for output prediction, self-verification, and iterative self-fixing.

  • Results

    Up to 43% improvement in CruxEval output prediction and up to 39% in competitive-programming solutions are reported versus evaluated baselines, with self-verification improving correctness by up to 5.5 percentage points.

  • Takeaways & Limitations

    Predicted execution signals can support code-solution selection and refinement without relying on external execution, with a relatively small gap from real execution.

  • Takeaways & Limitations

    The approach is currently limited to single-file competitive-programming questions, leaving generalization to full-repository software-engineering tasks for future work.

Abstract

from arXiv · show

A promising research direction in enabling LLMs to generate consistently correct code involves addressing their inability to properly estimate program execution, particularly for code they generate. In this work, we demonstrate that Code LLMs can be trained to simulate program execution in a step-by-step manner and that this capability can be leveraged to improve competitive programming performance. Our approach combines supervised fine-tuning on natural language execution traces, textual explanations grounded in true execution, with reinforcement learning using verifiable rewards. We introduce two complementary objectives: output prediction given code and inputs, and solving competitive programming tasks with either ground-truth or self-predicted execution feedback. These objectives enable models to perform self-verification over multiple candidate solutions, and iterative self-fixing by simulating test execution. Across multiple competitive programming benchmarks, our method yields consistent improvements over standard reasoning approaches. We further present ablations and analysis to elucidate the role of execution simulation and its limitations.

1. Introduction

The paper shows that Code LLMs can learn step-by-step self-execution simulation for external and self-generated code, then use predicted execution to self-verify and iteratively self-fix solutions. This training recipe improves output prediction and competitive programming performance while retaining relatively small degradation compared with ground-truth execution.

  • Contributions: The work trains Code LLMs to simulate program execution step by step, including for code they generated.It begins with natural-language execution traces grounded in real program executions.
  • Contributions: Predicted execution supports self-verification by selecting among parallel candidate solutions and iterative self-fixing through proposal, simulation, and refinement.The framework includes best@k verification and a multi-turn reinforcement-learning process for refinement.
  • Results: Up to 43% improvement in output prediction on CruxEval and up to 39% improvement in competitive programming solutions occur relative to the evaluated baseline.These gains apply to both external and self-generated code solutions.
  • Results: Up to 5.5% absolute points of code-correctness improvement arise when best@k uses output prediction to verify candidate solutions.The improvement is reported for competitive programming tasks.
  • Results: Multi-turn variants show consistent gains across evaluated configurations, while best@k and multi-turn methods degrade relatively little compared with ground-truth execution.The paper also analyzes the approach’s strengths and limitations.

2. Boosting Execution Simulation

The method boosts execution simulation by fine-tuning on natural-language execution traces and applying verifiable-reward training to output prediction and competitive programming. It uses traced programs and tests from both general Python code and more complex competitive programming solutions.

  • 2. Boosting Execution Simulation: The pipeline records line-by-line executions for Python programs paired with inputs and outputs, converts traces into natural-language explanations, and uses them for supervised fine-tuning.The approach then applies verifiable-reward training to output prediction.
  • 2. Boosting Execution Simulation: ∼30M functions come from basic code sources, while 35k come from competitive programming problems with substantially more complex programs.Inputs are synthesized using LLM prompting and lightweight fuzzing for repository functions, while provided tests are retained for CodeContests solutions.
  • 2. Boosting Execution Simulation: Free-form natural-language traces add semantic context to operations and align more closely with reasoning-style data used by language models than structured execution formats.The format can explain operations such as array updates in the scope of dynamic programming.
  • 2. Boosting Execution Simulation: ∼80 M execution descriptions are retained for general Python functions and 115 k for competitive programming solutions after discarding incorrect translated outputs.Qwen3-32B-FP8 performs the translation from raw structured traces to natural-language explanations.
  • 2. Boosting Execution Simulation: RLVR trains an output-prediction environment that maps a code-and-stdin pair to predicted stdout, awarding +1 for an exact match and −1 otherwise.Float comparisons allow 1e −5 tolerance.

3. Self-Execution For Verification

This section uses self-execution simulation for post-hoc verification: models simulate candidate programs on public tests and select candidates whose predicted outputs best match expected outputs. The same approach supports self-generated or externally generated solutions, with evaluation using best@k and rank_score_at_k.

  • Candidate solutions are simulated on public or generated tests and retained when predicted outputs align with expected outputs.This provides a straightforward post-hoc solution-filtering strategy.
  • In best@k, the model samples k candidates and selects the candidate predicted to pass the greatest number of public tests.For each candidate, the model simulates execution and compares predicted outputs with expected outputs.
  • The selection heuristic scores each solution by matching simulated outputs Msim(s, int) against the expected outputs in public input-output pairs.Selection is based on the indicator 1[Msim(s, int) = outt].
  • The evaluation uses rank_score_at_k with 20 generated solutions per task and 5 output-prediction attempts per test.This setup is intended to provide an unbiased accuracy estimate for generating and selecting k solutions.
  • Self-simulation uses the same LLM to generate candidate solutions and simulate their execution, but the method also applies to solutions from other models.The paper reports empirical evaluation of both setups in Section 6.

4. Self-Execution For Fixing

This section introduces multi-turn self-execution for sequential code fixing, using predicted execution outputs as feedback instead of actual program execution. The design separates solving, execution simulation, and submission or fixing, while training initially uses ground-truth execution feedback to mitigate noisy self-predictions.

  • Approach: The approach iteratively refines candidate solutions using predicted execution outputs as feedback rather than actual program execution.Unlike methods that verify multiple solutions through self-execution, this setup refines solutions sequentially from predicted feedback.
  • Multi-turn workflow: Explicit context switching represents each interaction as an independent single-turn prompt containing only relevant information.This isolates execution simulation from solution reasoning and access to correct outputs while mitigating long-context challenges.
  • Multi-turn workflow: The multi-turn workflow solves a question, independently simulates each public test, then submits the code or fixes it using expected and predicted outputs.Execution simulation receives a code snippet and test input, while the final turn decides whether to submit or produce a new solution.
  • Training feedback: Training initially provides ground-truth execution feedback because early self-predicted outcomes may be inaccurate and could otherwise be ignored as noisy.The transition from true execution signals to model-predicted outputs may occur during training or be deferred until inference.

5. Experimental Setup

The experiments train models with multiple reasoning datasets and reinforcement-learning objectives, then evaluate competitive-programming solving and output prediction across distinct benchmark sets. Evaluation uses public tests for basic checks and private tests for broader correctness, including edge cases and runtime compliance.

  • Evaluation protocol: Public tests provide basic correctness and formatting checks, while private tests assess broader correctness through edge-case coverage and runtime compliance.
  • Training data: Supervised fine-tuning combines NLEX with OpenMathReasoning and OpenCodeReasoning to bootstrap reasoning abilities.
  • Training data: During RL, models jointly optimize competitive-programming solving and solution-output prediction using approximately 12.2k filtered CodeContests training problems.
  • Evaluation datasets: LCB-IO contains 287 stdio-evaluated LiveCodeBench-v6 problems, reducing output prediction to determining stdout from a given stdin.
  • Evaluation datasets: DMC uses CodeContests validation and test splits, providing 282 problems from a different distribution.
  • Evaluation datasets: CruxEval-O evaluates whether models infer Python function return values from code and input–output examples.

6. Results

Results show that execution-trace training substantially improves output prediction, while self-verification and self-RLEF can further improve competitive-programming performance. However, joint training and inference-only scaffolding have important limitations, including weaker output prediction and limited exploration.

  • Competitive programming: On competitive-programming output prediction, results suggest that including NLEX data improves post-trained Qwen2.5 models on LCB-IO and DMC.The evaluation uses 3B and 7B models on test splits from both benchmarks.
  • Self-execution prediction: Jointly training solving and output prediction underperforms output-prediction-only training; CWM scores 80.2 and 86.5 versus 85.0 and 88.6 pass@1.The comparison concerns models evaluated on their own solutions.
  • Self-verification: Self-verification with simulated execution provides a 2−8 point best@k boost over standard selection, though it remains below oracle execution.The oracle comparison measures the simulation gap between executing and simulating public tests.
  • Self-RLEF: Self-RLEF evaluates official CWM, competitive-programming post-trained CWM, and jointly optimized CWM using real or simulated execution feedback.Performance is reported with pass@k for k ∈ {1, 5, 10} and public-test pass rates on LCB-IO and DMC.
  • Knowing when to submit or fix: When initial solutions fail, Self-RLEF fixes them on 17.0% of public tests and 10.4% of private tests, raising pass rate from 57.8% to 63.2%.The analysis compares initial generated solutions with final submitted solutions on DMC.

7. Related Work

Prior work studies code-output simulation, models’ difficulty simulating their own flawed code, and learning from execution or human feedback. Interpreters have improved performance across mathematics, code generation, competitive programming, and agentic coding.

  • Code Simulation & Verification: Research has examined whether LLMs can simulate or predict the outputs of given code snippets.Other studies use models to simulate tool execution for synthetic data generation.
  • Code Simulation & Verification: Models may struggle to simulate their own code because they are blind to its flaws.
  • Learning from Feedback: Models can learn to use feedback about the execution of their generated code.
  • Learning from Feedback: Providing interpreters has improved performance in mathematics, code generation, competitive programming, and agentic coding.
  • Learning from Feedback: Training with human-written feedback on code can improve performance.

8. Discussion

Execution simulation offers a useful inductive bias for reasoning about program behavior despite noise from imperfect estimation of complex operations. The approach remains limited to single-file competitive programming, while richer execution feedback could support future iterative code fixing and repository-scale software engineering tasks.

  • Limitations: Execution simulation can be noisy because models struggle to estimate complex operations such as multiplying large numbers and computing logarithms.The passage identifies complex computational operations as the main limitation of simulating program execution.
  • Limitations: Despite imperfect execution estimates, simulation provides a useful inductive bias for reasoning about program behavior when direct execution is expensive or infeasible.The authors qualify this benefit by noting that simulation can introduce noise.
  • Limitations: The current approach is limited to single-file competitive programming questions, leaving generalization to full-repository software engineering tasks as future work.The passage explicitly frames repository-scale SWE generalization as an interesting future research direction.
  • Future Work: Future iterative code fixing could use full execution simulations rather than only final outputs, providing richer feedback about what output occurs and why.Such explanations may expose tests that pass for incidental reasons.

9. Conclusion

The work shows that LLMs can be trained to predict execution outcomes for general programs and generated code by combining SFT on NLEX with RLVR. This capability supports self-verification and iterative self-fixing without external execution, improving competitive programming performance over standard approaches.

  • 9. Conclusion: Combining SFT on NLEX with RLVR enables models to predict execution outcomes for general programs and code they generate.The training approach targets execution prediction through supervised fine-tuning and reinforcement learning with verifiable rewards.
  • 9. Conclusion: Predicted execution signals enable self-verification and iterative self-fixing without relying on external execution.These strategies select or refine candidate solutions using the model’s own execution predictions.
  • 9. Conclusion: Empirical results on competitive programming tasks show consistent improvements over standard approaches.The improvements are attributed to leveraging predicted execution signals for candidate selection and refinement.

A. Appendix. … A.5. Data Samples

The appendix reports additional evaluations, implementation settings, prompts, and data samples supporting execution simulation, reinforcement learning, self-verification, and iterative self-fixing. It also illustrates how natural-language traces represent concrete program states and outputs.

  • A. Appendix.; A.1. Additional Results; A.1.1. SUPERVISED FINE-TUNING.: The appendix evaluates the NLEX data mix on standard coding and mathematics benchmarks to assess whether output-prediction gains harm general-task performance.The comparison uses supervised fine-tuning with the same training budget while changing only the data mix.
  • A.1.2. THE EFFECT OF RL ON OUTPUT PREDICTION: The additional RL phase significantly improves CWM output prediction on competitive-programming questions compared with training without RL and includes the official post-trained CWM as a reference.Qwen results without RL are omitted because their performance was significantly lower.
  • A.1.3. SELF-VERIFICATION: Self-verification is additionally evaluated when verification tests are absent from both training and inference, so the tests provide completely new information.Solutions are generated by a model jointly trained for output prediction and competitive-programming solving.
  • A.1.4. SELF-RLEF: With a maximum of 3 solve turns, Self-RLEF uses an average of 2.38 turns, extending solve-rate analysis under tighter compute constraints.The default maximum is 10 turns, while the unrestricted setup averages 3.33 turns.
  • A.1.5. BEYOND Self -VERIFICATION: Dedicated output-prediction verifiers consistently improve best@k solution ranking for Qwen3-4B and CWM Solve-RL, with only slight degradation relative to ground-truth execution.A smaller dedicated Qwen2.5-7B verifier is also effective and outperforms same-model solving and verification, especially for limited-capacity models.
  • A.2. Self-RLEF Example Inference: A Self-RLEF inference example shows simulation detecting that an attempted solution produced NO instead of the expected YES, identifying mishandled Backspace on an empty state, and submitting the corrected behavior.The example progresses through solving, simulation, fixing, resimulation, and submission turns.
  • A.3. Hyper-Parameters: The appendix specifies training configurations, including Qwen sequence length 65,536, CWM context length 131,072, 15.5k supervised-fine-tuning steps, and 65B total tokens.RL uses NVIDIA H100 GPUs, while evaluation selects sampling temperatures per model based on DMC pass@1 rates.
Loading 2604.03253v1…