Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMs

1 Introduction

• LLM-based APR relies on static code analysis, while debugging complex issues can require understanding runtime behavior such as variable and control-flow changes.Execution traces provide diagnostic information beyond static analysis.Source: §1 · p. 1
• Adding execution traces does not consistently outperform trace-free prompts, and longer traces with more variable assignments are less effective.The paper evaluates trace prompts against failing-test-only and trace-free chain-of-thought prompts.Source: §1 · p. 1
• Among three trace representations, LLM-optimized prompts provide the most consistent repair results.The alternatives are collated traces and a representation selected according to the LLM’s confidence.Source: §1 · p. 2
• Two follow-up studies compare trace-based prompting with fine-tuning and probe LLMs on trace-understanding tasks.The fine-tuning comparison is inspired by TraceFixer.Source: §1 · p. 2

2 Related Work

• Prior work used compiler and test diagnostics for self-supervised APR training, and execution traces for pretraining code models on execution-related tasks.Examples include SelfAPR, TRACED, and program-state prediction pretraining.Source: §2 · p. 2
• TraceFixer improved APR by 13% on synthetic bugs over a code-only baseline but struggled with real bugs.The cited work’s authors note potential generalization limitations.Source: §2 · p. 2
• Self-Debug uses LLM-generated code explanations in a chain-of-thought approach to program generation.The paper distinguishes this training-free approach from prompting a pretrained LLM with actual execution traces.Source: §2 · p. 2
• The authors state that, to their knowledge, prior work had not studied placing execution traces in prompts to pretrained LLMs.This is the paper’s stated distinction from the related work it discusses.Source: §2 · p. 2

3.1 Set Up

• The evaluation uses Refactory, RunBugRun, and HumanEval-Java, with GPT-3.5 Turbo and GPT-4; traces log runtime state changes tied to code lines.The datasets span student Python bugs, sampled Python submissions, and synthetic bugs in Java-adapted HumanEval problems.Source: §3.1 · p. 3
• Prompt comparisons measure Correct Fix Accuracy (CFA) and Correct Program Accuracy (CPA), with one prediction per test-case-specific prompt.CFA counts fixes passing all test cases; CPA counts programs with at least one correct fix.Source: §3.1 · p. 3, §3.2 · p. 4

3.2 RQ1. Are prompts with execution traces more effective at program repair than prompts without traces?

• Trace prompts do not consistently outperform error-only prompts: GPT-4 benefits on two datasets, while GPT-3.5 does not appear to benefit on RunBugRun.On Refactory, error-only prompts outperform trace prompts for both models on fix and program accuracy; HumanEval-Java results vary by model and metric.Source: §3.2 · p. 4

3.3 RQ2. How does trace complexity affect the effectiveness of trace-based prompts?

• For HumanEval-Java and RunBugRun, failing fixes have higher median trace lengths and variable modifications; Refactory has shorter median traces for failing fixes.Median variable modifications are one in Refactory versus five in RunBugRun, suggesting variable changes may matter to trace effectiveness.Source: §3.3 · p. 5, Figure 2 · p. 5
• Longer execution traces are associated with a lower chance of generating a correct fix.The analysis considers trace length and the number and type of variable state changes.Source: §3.3 · p. 5

4.1 Modified Traces

• Some traces exceed 10,000 entries, and trace prompts are truncated in 5% of Refactory cases and almost 10% of RunBugRun cases.The passage reports these extreme traces in all datasets and for both correct and failing fixes.Source: §4.1 · p. 5
• Collated traces place trace entries inline with matching code lines, while OPT uses GPT-4-32k to condense traces and confidence-based selection falls back to OPT for low confidence.Confidence is measured on a 1–5 Likert scale; scores below 3 trigger the fallback.Source: §4.1 · pp. 5–6

Trace-length Based Prompt Selection (TRL

• Length-based routing uses the ordinary trace prompt below threshold N and switches to an OPT or error prompt above N.The paper evaluates both trace-to-OPT and trace-to-error routing.Source: Trace-length Based Prompt Selection (TRL · p. 6

4.2 RQ3 Results.

• OPT ranks among the top three prompting techniques for CPA across all dataset/model pairs, leads CFA in three of six, and is second-best for CFA in the other three.Table 2 reports the RQ3 quantitative results for Correct Fix Accuracy (CFA) and Correct Program Accuracy (CPA).Source: §4.2 · p. 6, Table 2 · p. 7
• Confidence-based selection performs comparatively better for GPT-4 but worse for GPT-3.5, and remains unreliable for GPT-4 because it performs significantly worse than OPT on average.The authors infer that GPT-4 provides significantly better confidence scores than GPT-3.5.Source: §4.2 · p. 6
• Length routing yields the best CPA only for GPT-3.5 on HumanEvalJava, and changing N has limited impact on CFA and CPA.The authors find no strong evidence that routing by trace length is significantly beneficial.Source: §4.2 · p. 7
• Collated prompts do not improve over trace prompts; inline loop traces can stretch prompts, while LLMs struggle to track variable changes across iterations and truncation can exclude original problem content.The authors suggest unfamiliarity with inline state-change comments as one possible explanation for the lack of improvement.Source: §4.2 · p. 7

5.1 Trace-based prompting compared to finetuning a smaller model.

• The deepseek-coder-1.3b-instruct model was fine-tuned on buggy programs, failing-test results, and execution traces from HumanEval-Java and RunBugRun.For each dataset, 80% of problems were used for training and the rest for testing.Source: §5.1 · p. 7
• All prompting techniques outperformed the fine-tuned model on CPA and CFA in the reported test comparison.The comparison used the same test programs for prompting-based techniques; the authors suggest limited training data may be one reason for the fine-tuned model’s lower scores.Source: §5.1 · p. 7, Figure 4 · p. 8

5.2 Probing Studies of Trace Understanding

• The probing studies used GPT-4 on Refactory and Geeks-for-geeks programs to test trace collation and trace prediction, reporting exact-match rates.They distinguished traces for working and failing programs; outputs were compared with ground-truth traces or collated trace/programs after light post-processing.Source: §5.2 · p. 8
• Trace-collating accuracy was 88% on reference Refactory programs, fell by nearly ten percent on failing programs, and reached 45% on Geeks-for-geeks.The authors note that the latter dataset also eliminates the possibility of prompt leakage.Source: §5.2 · p. 8
• Zero-diff trace prediction did not exceed 50% on reference Refactory programs, was halved for failing programs, and was 15% on Geeks-for-geeks.The authors conclude that real traces can contribute information not yet easily inferred by strong LLMs such as GPT-4.Source: §5.2 · p. 8

• Trace-based prompts did not consistently outperform error-based prompts, and their effectiveness varied with dataset and LLM.LLM-optimized traces offered more consistent improvements without limiting trace complexity heuristically.Source: §6 · p. 8
• Longer traces and more variable assignments reduced trace-prompt effectiveness.Source: §6 · p. 8
• LLMs had limited capacity for trace generation, which the authors say explains the weaker Self-Debug baseline and highlights traces’ potential utility in code tasks.Source: §6 · p. 8

• The document was prepared for informational purposes and is not a product of JP Morgan’s Research Department.Source: §7 · p. 9
• It is not investment research or investment advice, or a recommendation, offer, or solicitation to buy or sell a security or financial product.Source: §7 · p. 9
• JP Morgan disclaims warranties and liability for the information’s completeness, accuracy, or reliability.Source: §7 · p. 9

• Error-based prompts always outperformed program-only prompts in the preliminary experiment, so they were used as the comparison point for trace prompts.Source: A Rationale behind prompt choice and… · p. 10
• Including all failing test cases did not improve results over one case and performed slightly worse overall.The authors hypothesize that a single failing test case is generally sufficient to guide repair, while the buggy function supplies much of the information needed.Source: A Rationale behind prompt choice and… · p. 10
• Few-shot prompts may improve error-based prompts, but can make trace prompts exceed the 8k context window.Source: A Rationale behind prompt choice and… · p. 10

A.1 Trace Understanding Probing Studies Full Results and Qualitative Findings

• Most discrepancies between ground-truth and LLM-collated or predicted traces involve added or deleted variable modifications.Source: §A.1 · p. 14
• Within loops, the LLM tends to miss or add variable modifications, potentially reflecting limits in reasoning depth and memory.Source: §A.1 · p. 14
• For trace prediction from scratch, function returns are the second most erratic behavior, with errors in value or placement in execution flow.Source: §A.1 · p. 14
• Other observed discrepancies include unreliable exception predictions, formatting changes, removal of untaken branches, extra commentary, and hallucinated literals.Source: §A.1 · p. 14