Source-linked AI summary
Towards a Neural Debugger for Python
Maximilian Beck, Jonas Gehring, Jannik Kossen, Gabriel Synnaeve
TL;DR
Existing neural interpreters model sequential execution traces but lack the interactive control developers use when debugging. The paper introduces neural debuggers trained from execution traces and reports accurate conditional forward and inverse execution, including strong CruxEval input and output prediction. These models provide a basis for simulated debugging environments and future agentic coding systems.
Problem
Existing execution-trace models predict whole-program execution but do not model developers’ interactive, non-sequential use of debugger actions.
Method
The authors train neural debuggers by representing execution as debugger-action-conditioned state transitions and preparing trajectories through a state-tree data pipeline.
Results
Neural debuggers achieve accurate intermediate-state and overall execution prediction, with the 32 B-parameter model exceeding 90% forward next-state accuracy across key actions and reaching 66.5 input and 83.2 output pass@1 on CruxEval.
Takeaways & Limitations
Neural debuggers support simulated debugging and provide a potential execution-grounded world model for future agentic coding systems.
Takeaways & Limitations
The current work is limited to Python programs and uses random action policies for trajectory generation; future work proposes additional languages and more structured policies.
Abstract
from arXiv · showhide
Training large language models (LLMs) on Python execution traces grounds them in code execution and enables the line-by-line execution prediction of whole Python programs, effectively turning them into neural interpreters (FAIR CodeGen Team et al., 2025). However, developers rarely execute programs step by step; instead, they use debuggers to stop execution at certain breakpoints and step through relevant portions only while inspecting or modifying program variables. Existing neural interpreter approaches lack such interactive control. To address this limitation, we introduce neural debuggers: language models that emulate traditional debuggers, supporting operations such as stepping into, over, or out of functions, as well as setting breakpoints at specific source lines. We show that neural debuggers -- obtained via fine-tuning large LLMs or pre-training smaller models from scratch -- can reliably model both forward execution (predicting future states and outputs) and inverse execution (inferring prior states or inputs) conditioned on debugger actions. Evaluated on CruxEval, our models achieve strong performance on both output and input prediction tasks, demonstrating robust conditional execution modeling. Our work takes first steps towards future agentic coding systems in which neural debuggers serve as a world model for simulated debugging environments, providing execution feedback or enabling agents to interact with real debugging tools. This capability lays the foundation for more powerful code generation, program understanding, and automated debugging.
1 Introduction
Existing execution-trace models predict sequential program behavior but do not model developers’ interactive debugger actions. This work introduces neural debuggers, a data pipeline, and models that support conditional execution prediction with strong benchmark performance.
- Traditional debuggers let developers inspect and control execution through stepping, breakpoints, and function returns to localize faults and understand control flow.
- Execution-trace models enable line-by-line prediction of whole programs but overlook interactive, non-sequential debugging behavior.
- Neural debuggers predict Python execution conditioned on actions such as step_into, step_over, breakpoint, and step_return.
- Neural debuggers can simulate execution for non-executable or partially specified programs and approximate inverse execution from arbitrary states.
- The training pipeline builds state trees, samples debugger trajectories, and tokenizes them into a structured format for language models.
- The 32 B-parameter model exceeds 90% forward next-state accuracy across key actions, while finetuned and scratch-trained models achieve strong CruxEval input and output scores.The 1.8 B-parameter model scores 53.6 input and 57.7 output pass@1; the finetuned 32 B-parameter model scores 66.5 input and 83.2 output pass@1.
2 Related work
Prior work established that Transformer models can learn execution traces and code-state dynamics, while CWM scaled this approach to large Python-trace training. Neural debuggers extend execution modeling with explicit debugger actions and inverse capabilities.
- Learned-interpreter research trains neural networks to predict intermediate Python execution states and source lines, with scratchpad strategies improving execution modeling.
- Transformer-based models can model control flow and variable-state dynamics during execution, strengthening code understanding.
- CWM trains on large-scale Python execution traces formatted as observation-action pairs and supports line-by-line execution prediction without live execution environments.
- Neural debuggers condition next-state prediction on debugger actions, enabling direct jumps, reverse execution, function-input prediction, and termination prediction unavailable in CWM’s manually steered setup.
3 Python program execution traces
Python execution produces code objects and stack frames containing source and variable information. The authors capture these runtime events with sys.settrace to construct execution traces for neural debugger training.
- Python parses code into abstract syntax trees, compiles it into bytecode-containing code objects, and executes those objects in its evaluation loop.
- Stack frames contain local and global variables plus the active code object, including the source line and bytecode being executed.
- Using sys.settrace, a custom trace function records frame, event, and argument data across function executions and repository tests.
- The resulting traces contain sequences of program states that feed the neural debugger data pipeline.
4 Neural debugger
The paper models neural debugging as controlled traversal over forward and inverse program-state trees, representing execution states and debugger actions in structured traces. This framework supports conventional stepping and breakpoint operations, inverse execution, and stochastic trajectory generation for language-model training.
- A neural debugger predicts program execution while exposing debugger-like interactions without requiring an executable target program.
- The debugger is formalized as an MDP whose states contain program locations and variable values, while actions induce transitions through the reconstructed call-stack tree.The formulation omits a reward function because the work focuses on state prediction rather than learning action policies.
- Each state records runtime event type, local variables or arguments, and the source line being executed.
- Debugger actions traverse the state tree through step_into, step_over, step_return, breakpoint, and continue operations.step_into follows the immediate next node, step_over stays at the current level, step_return targets the current function's return event, and continue reaches the exit code.
- Inverse execution reverses the forward state tree so the debugger can infer plausible predecessor states, inputs, or function arguments.Inverse call nodes preserve choices between stepping into and stepping over a function call.
- The training pipeline serializes forward and inverse state-action trajectories in a structured language format and samples diverse trajectories with a stochastic action policy.The format extends CWM-style special tokens to represent debugger actions and inverse execution.
5 Experimental results
Experiments evaluate neural debuggers on forward and inverse next-state prediction, comparing finetuning with pre-training and testing downstream CruxEval input/output prediction. Performance varies by debugger action, state component, model size, and prediction horizon.
- 5 Experimental results: Models are evaluated on forward and inverse execution using function-level and repository-level validation traces, with exact-match next-state accuracy measured per debugger action.The evaluation samples trajectories, truncates traces after the relevant action, and compares parsed predictions with ground-truth program states.
- 5.1 Finetuning and pre-training neural debuggers: Step actions achieve higher exact-match accuracy than jump actions, while jump-action performance continues improving with additional training.Step transitions usually reach the next source line, whereas jump actions require predicting multi-line transitions.
- 5.1 Finetuning and pre-training neural debuggers: Finetuned CWM improves rapidly early in training, reflecting prior exposure to trace data despite its differing format and lack of debugger actions.The comparison concerns the finetuned CWM model in the function-level and repository-level evaluations.
- 5.1 Finetuning and pre-training neural debuggers: Inverse prediction improves consistently but remains less accurate than forward prediction, with exact-match performance constrained by ambiguity in prior states.Input prediction evaluated with pass@1 accounts for this ambiguity and yields accuracies comparable to output prediction.
- 5.2 Next program state prediction by state component: Source lines and state events are predicted reliably, whereas local variables and return or exception arguments account for most remaining errors.Repository-level source-line accuracy is lower than function-level accuracy, particularly for step_return, with drops of ∼5 % and ∼10 % points respectively.
- 5.3 Input and output prediction on CruxEval: On CruxEval, breakpoint output prediction reaches 83.2 %, while step_return reaches 77.9 %, a 19.8 %-point improvement over stock CWM at 58.1 %.The smaller 1.8 B Transformer reaches 57.7 % with breakpoint and 48.0 % with step_return after 150 B training tokens.
- 5.3 Input and output prediction on CruxEval: Input and output prediction accuracy decreases as the prediction horizon increases from single-step prediction toward full-function prediction.The analysis varies the number of skipped intermediate program states before the final jump action.
6 Limitations and future work
The paper identifies limitations in current neural debuggers and outlines future work spanning applications, data generation, inverse prediction, and Python object representations.
- Applications: Neural debuggers could support agentic coding systems that self-debug generated code or control debuggers in real debugging environments.
- Data generation: The current data pipeline is limited to Python programs and uses random action policies for trajectory generation.
- Data generation: Future data generation could add other languages and structured or goal-directed policies informed by syntactic constructs such as conditions and loops.
- Inverse debugging: Inverse debugging requires better modeling of ambiguity and feasible value sets, alongside metrics that accommodate multiple valid traces or outcomes.
- Python object representations: Text serialization of arbitrary Python objects becomes infeasible for very large or complex structures, motivating compact neural representations.
7 Conclusion
The paper introduces neural debuggers that predict program execution conditioned on debugger actions and formalizes them as execution-aware models. Experiments show accurate intermediate-state prediction and strong overall execution prediction performance.
- Neural debuggers predict line-by-line program execution conditioned on actions such as step_into, step_over, breakpoint, and step_return.
- The debugger is formalized as a Markov Decision Process whose states contain program variables and source lines, with transitions defined by call-stack-derived tree traversal rules.
- Fine-tuning large language models or pre-/mid-training on debugger traces yields accurate intermediate state predictions and strong overall execution prediction performance.
A.1 Extended debugger trace dataset
The extended dataset appendix presents distributions of debugger trajectory lengths and the action-probability policy used to generate traces. Repository-level trajectories contain substantially more tokens despite similar action counts.
- Figure A.1 reports debugger trajectory lengths in tokens and actions, alongside code lengths in characters.
- Repository-level trajectories have significantly higher average token counts than function-level trajectories despite similar numbers of actions.
- Table A.1 gives the action probabilities for the policy mix used to generate the debugger trace dataset.
A.2 Evaluating Inverse Execution Prediction
The appendix evaluates inverse execution prediction by comparing predicted states or inputs against reference traces and outputs using exact-match-based metrics. Because inverse execution can be ambiguous, exact match against one reference input may underestimate performance.
- CruxEval input prediction is evaluated by executing the predicted input and comparing its output with the reference output, while output prediction compares predicted output with the reference input.
- Table A.2 compares exact_match@1 and pass@1 for single-step inverse prediction, highlighting the effect of multiple valid input predictions.
- Predictions are parsed and their individual state components are compared with ground truth using exact match.
- For deterministic forward functions, exact_match@1 and pass@1 are identical.
- Inverse execution may be nondeterministic even when forward execution is deterministic, so single-reference exact match can underestimate true absolute performance.
B.1 Training recipe
The models use AdamW with weight decay, Llama-2 architecture, and distinct learning-rate schedules for pre-training and fine-tuning.
- The neural debuggers use AdamW with weight decay 0.1.
- Pre-training uses a 1.8 B-parameter Llama-2 architecture with 750 warmup steps and cosine decay from 1 × 10^-3 to zero.
- Fine-tuning uses 750 warmup steps followed by a constant peak learning rate of 1 × 10^-5.
B.2 Extended next state prediction results
The appendix extends next-state prediction analysis to repository-level validation data and illustrates CruxEval output and input prediction prompts alongside a forward/inverse trace example.
- Extended next state prediction results: Repository-level results include next-state prediction accuracy by debugger action during training.Figure B.1 reports forward and inverse execution prediction, with corresponding function-level results leading to similar conclusions.
- Extended next state prediction results: Repository-level results also break next-state prediction accuracy down by state component and debugger action.The corresponding function-level results leave the conclusions unchanged.
- CruxEval evaluation examples: CruxEval output prediction prompts illustrate the step_return and breakpoint actions.step_return produces a return-event frame with its argument, while breakpoint produces a line-event frame containing the local-variable dictionary.
- CruxEval evaluation examples: CruxEval input prediction is illustrated with a prompt for the inv_step_call action.The example uses a 1.8 B-parameter neural debugger trained on 150 B debugger trace tokens.
- Neural debugger trace example: The trace example uses step_into and inv_step_into actions to visit every frame in forward and inverse debugging.Forward execution is deterministic for fixed inputs, whereas an inverse trace with return argument n=2 admits infinitely many s and t input combinations.