PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Code Semantics Help? A Comprehensive Study on Execution Trace-Based Information for Code Large Language Models

Jian Wang, Xiaofei Xie, Qiang Hu, Shangqing Liu, Yi Li

arXiv:2509.11686v3cs.SEcs.AI

TL;DR

Code LLMs have difficulty reasoning about runtime behavior, and existing work lacks unified tools and systematic evidence about semantic representations. This paper builds a framework to integrate trace-based semantics into fine-tuning and inference prompts, finding limited usefulness for fine-tuning and test-time scaling.

  • Problem

    Code LLMs struggle to reason about runtime behavior, while existing work lacks a unified study of semantic representations and their effects on code reasoning and generation.

    Source: §1 · pp. 1–2

  • Method

    The paper builds a framework and dataset for integrating multiple execution-trace representations into Code LLM fine-tuning and inference prompts.

    Source: §1 · p. 2

  • Results

    Existing trace-based semantic information has limited benefits for tuned Code LLM performance and does not enhance fine-tuning or test-time scaling.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    The study concludes that existing code semantic information does not enhance Code LLM fine-tuning and test-time scaling.

    Source: §7 · p. 8

    The framework currently supports only Python, and experiments cover models around 7B parameters because of computational-resource constraints.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Code Large Language Models (Code LLMs) have opened a new era in programming with their impressive capabilities. However, recent research has revealed critical limitations in their ability to reason about runtime behavior and understand the actual functionality of programs, which poses significant challenges for their post-training and practical deployment. Specifically, Code LLMs encounter two principal issues: (1) a lack of proficiency in reasoning about program execution behavior, as they struggle to interpret what programs actually do during runtime, and (2) the inconsistent and fragmented representation of semantic information, such as execution traces, across existing methods, which hinders their ability to generalize and reason effectively. These challenges underscore the necessity for more systematic approaches to enhance the reasoning capabilities of Code LLMs. To address these issues, we introduce a generic framework to support integrating semantic information~(e.g., execution trace) to code task-relevant prompts, and conduct a comprehensive study to explore the role of semantic information in enhancing the reasoning ability of Code LLMs accordingly. Specifically, we focus on investigating the usefulness of trace-based semantic information in boosting supervised fine-tuning~(SFT) and post-phase inference of Code LLMs. The experimental results surprisingly disagree with previous works and demonstrate that semantic information has limited usefulness for SFT and test time scaling of Code LLM.

1 Introduction

The introduction identifies limited runtime-semantic reasoning and fragmented semantic representations as gaps, then presents a framework and systematic study of semantic information in Code LLM fine-tuning and inference. The reported findings say semantic information has limited benefits for tuned-model performance and does not positively contribute to inference.

1 Introduction

  • Code LLMs primarily capture static code dimensions, while runtime behavior remains difficult to reason about despite its relevance to semantic understanding.The introduction cites evidence that even GPT4 struggles with runtime behavior.Source: §1 · p. 1
  • Existing approaches lack a unified study and tool for diverse semantic representations and systematic evidence on how training-data composition affects code reasoning and generation.Prior work uses representations such as natural-language descriptions and execution traces.Source: §1 · p. 2
  • Semantic information integrated into prompts has limited benefits for tuned Code LLMs and does not positively contribute to inference.The study examines semantic information in both fine-tuning and inference.Source: §1 · p. 2
  • The authors introduce a generic framework for multiple code-semantic representations and construct a dataset featuring bug-patch pairs, unit tests, and multiple semantic layers.The dataset and related implementations are described as publicly available.Source: §1 · p. 2
  • The study evaluates semantic information for both supervised fine-tuning and inference across code-related tasks.The introduction frames these as two approaches to improving Code LLM reasoning.Source: §1 · pp. 1–2

2 Problem Statement

The problem statement asks whether execution-trace semantics can improve Code LLM code generation through fine-tuning or inference. It defines a repair-based training setup and a budgeted verify-and-refine inference strategy.

2 Problem Statement

  • The paper frames two subproblems: fine-tuning with semantic information and inference with semantic information to improve Code LLM code generation.The motivation is that static text may miss nuanced semantics and existing models lack human-like reasoning and refinement processes.Source: §2 · p. 2
  • The fine-tuning setup uses buggy code b, execution-trace rationale r, and patched code a to train models through a repair-based input-output pair.The paper asks how to encode r so models can learn better code generation.Source: §2 · p. 2
  • The inference question is whether semantic representations in prompts improve performance under a fixed test-time compute budget N.The strategy θ proposes candidates, verifies or scores partial outputs, and refines them within N.Source: §2 · pp. 2–3
  • The compute-optimal test-time strategy θ* maximizes the expected probability of a correct answer, with refinement steps and verified candidate paths allocated under budget N.Verification signals can help discard incorrect paths or improve partially correct ones.Source: §2 · p. 3

3 Evaluation Framework

The framework constructs trace-based semantic data, fine-tunes models, and supports inference-time scaling. Its trace adapter converts execution traces into several representations for prompts and training.

3.1 Overall Design

  • The framework curates repair and downstream-task datasets, then uses a trace adapter to generate execution-trace rationales from buggy and correct code.The adapter runs code to collect raw traces and converts them into five representations; the resulting dataset includes code, tests, descriptions, and trace types.Source: §3.1 · p. 3
  • Fine-tuning proceeds in two stages: repair-based training uses trace differences between buggy and correct code, followed by downstream datasets for task-specific knowledge.The figure depicts prompt-tuning data construction followed by parameter-efficient or full-parameter fine-tuning.Source: Figure 1 · p. 3, §3.1 · p. 4
  • Inference supports sequential and parallel scaling; sequential scaling tests sampled programs, adds trace diagnostics for failures, and repeats revisions until success or the round budget is reached.Sequential scaling explores at most N × Rmax candidates; parallel scaling generates multiple solutions and selects one.Source: §3.1 · p. 4

3.2 Trace Representation Adapters

  • The trace adapter supports distinct execution representations, including NExT inline state comments, SemCoder natural-language explanations, and separate Code Executor traces.Concise omits variables whose values do not change on a line, unlike Code Executor.Source: §3.2 · pp. 4–5

4 Experiment Design

The experiments evaluate fine-tuned models on synthesis, repair, and reasoning tasks, and assess inference-time scaling on LiveCodeBench using adapted trace representations.

4 Experiment Design

  • The study fine-tunes models with semantic-information datasets and evaluates them on code synthesis, repair, and reasoning tasks, while test-time scaling is evaluated on LiveCodeBench.The synthesis datasets include HumanEval, MBPP, LiveCodeBench, and BigcodeBench; repair includes HE-R and MBPP-R, and reasoning includes CRUXEval-I and CRUXEval-O.Source: §4 · p. 5
  • Fine-tuning covers DeepSeek-Coder, LLaMA, and Gemma2, while inference also includes GPT-4o, Deepseek-Chat, phi-4, and Marco-o1.The latter two models are included for reasoning-ability evaluation.Source: §4 · p. 5
  • Prompts are generated automatically with DSPY, and code execution occurs in a sandbox; HumanEval and HE-R results are reported in appendices.Detailed configurations and prompt templates are also in the appendices.Source: §4 · p. 5

5 Result Analysis

Across fine-tuning and test-time scaling, trace-based semantic information generally does not improve Code LLM performance. Results also depend on the representation, model, and scaling hyperparameters.

5.1 Fine-Tuning with Semantic Information

  • In 7 of 9 code-synthesis cases, trace-free fine-tuning scored best; traces brought only limited repair gains and no significant synthesis improvement.For program repair, only SemCoder improved over w/o trace, with results reported from 0.3 to 1.4.Source: §5.1 · p. 5
  • No trace representation consistently outperformed the others; SemCoder was best for repair, while SemCoder (GPT4o) consistently enhanced reasoning.The authors recommend SemCoder for repair and SemCoder (GPT4o) for code reasoning.Source: §5.1 · p. 5

5.2 Parameter-Efficient Fine-Tuning

  • Fine-tuning strategy effects were model-dependent: full fine-tuning led for DeepSeek in 5 of 6 cases, while LoRA helped LLaMA and Gemma2 in 11 of 12.The reported performance gap between training methods reached 21.6 for LLaMA; Figure 3 depicts Full, LoRA64, and LoRA8.Source: §5.2 · p. 6, Figure 3 · p. 7
  • Gemma2-9B adapted better to traces, reaching 10.34% Pass@1 with Semcoder; LoRA64 without traces led general code generation, while repair-focused traces maximized repair.The section concludes that fine-tuning without semantic information remains the best choice overall when considering these training methods.Source: §5.2 · p. 6

5.3 Inference Test-Scaling Computation

  • Table 3 compares Pass@1 on LiveCodeBench easy private tests under equal compute budgets; the prose reports closed-source LLMs perform significantly better at test time.The table caption describes greedy, CoT, sequential scaling, and parallel scaling settings.Source: §5.3 · pp. 6–7, Table 3 · p. 7
  • In 36 of 56 cases, adding semantic information did not improve correctness; test scaling beat Greedy and CoT in 65 of 70 cases.Concise achieved Pass@1 no worse than w/o trace in 11 of 14 cases.Source: §5.3 · p. 7

5.4 Hyperparameter Study

  • At T=0.2, Pass@1 was negatively affected; higher temperatures performed relatively better.Scores fluctuated across temperatures.Source: §5.4 · p. 7
  • More generated samples and more sequence-scaling rounds improved code generation, but sample count traded off against performance; samples=8 was the default.The sample-count results are reported in Figure 5 and round-count results in Table 9a.Source: §5.4 · p. 7

6 Related works

Related work covers reasoning and tool-use approaches, execution-behavior signals for software tasks, and inference-time computing. This study builds on those lines by evaluating multiple execution-based representations.

Chain-of-Thought (CoT) and Tool-Integrated Reasoning (TIR)

  • Chain-of-Thought decomposes problems into intermediate reasoning steps, while Tool-Integrated Reasoning adds external tools for computations requiring precision (Wei et al., 2022; Gou et al., 2023).The passage notes TIR's use in symbolic computation and high-complexity algorithms.Source: §6 · pp. 7–8

LLMs for Software Engineering and Execution Behavior

  • Prior execution-behavior work includes Self-Debugging explanations, NExT inline execution comments, and natural-language runtime descriptions; this study examines multiple representations.The cited prior studies are Chen et al. (2023), Ni et al. (2024), and Ding et al. (2024).Source: §6 · p. 8
  • Prior work spans software-engineering tasks and benchmarks, while inference-time computing research uses reward models and solution sampling; this study evaluates semantic information across training and inference.The passage cites token-level reward models, process reward models, and an adapted framework for inference scaling.Source: §6 · p. 8

7 Conclusion

The paper presents a framework and systematic evaluation of trace-based semantic information, finding no general performance benefit in fine-tuning or test-time scaling. It proposes future work on representations and integration strategies.

7 Conclusion

  • The paper introduces a generic framework for generating trace-based code semantic information and evaluates its usefulness in Code LLM fine-tuning and inference.Source: §7 · p. 8
  • The evaluation finds that existing code semantic information does not enhance Code LLM fine-tuning or test-time scaling.Source: §7 · p. 8
  • Future directions include more aligned semantic representations and improved integration through architectural changes, specialized pretraining objectives, or adaptive prompting.The work is proposed as a baseline for studying semantic information in Code LLMs.Source: §7 · p. 8

8 Data and Source Code availability

The paper states that its source code, datasets, and intermediate reproduction data are available online.

8 Data and Source Code availability

  • Source code, datasets, and intermediate reproduction data are available at the project GitHub repository.Source: §8 · pp. 8–9

Limitations

The framework and experiments are limited to Python and models of around 7B parameters. The authors identify broader language support and larger-model experiments as future work.

Limitations

  • The framework currently supports Python only; Java and C++ are not supported.The authors plan to maintain the framework to cover more languages.Source: Limitations · p. 10
  • Experiments cover LLMs of around 7B parameters because of computational-resource constraints.Experiments with larger models are left for future work.Source: Limitations · p. 10

Ethical Considerations

The authors describe safeguards for research use, including privacy screening of traces, controls on potentially harmful outputs, and procedures for licensing and responding to reported harms.

Ethical Considerations

  • The released datasets, code, and models are intended for research and evaluation, not autonomous production deployment.The project’s stated aim is to deepen understanding of dynamic program semantics in Code-LLM reasoning toward safer, more reliable coding assistants.Source: Ethical Considerations · p. 10
  • Fine-tuning and evaluation code comes from OSI-approved permissive-licence repositories, and the authors commit to revising or removing resources within 30 days of a substantiated harm report.They publish the redaction and inspection pipeline and provide a security-contact email for vulnerability disclosures.Source: Ethical Considerations · p. 10
  • Runtime traces undergo pattern-based redaction, dynamic taint masking, and manual review of a random 2% sample; traces failing checks are discarded.The pipeline targets secrets and PII, including values originating from environment variables, network sockets, or file I/O.Source: Ethical Considerations · p. 10
  • The authors withhold weights fine-tuned on security-sensitive benchmarks and use server-side filters for exploit-related outputs.They also document residual unsafe generations in an appendix to support stronger community safeguards.Source: Ethical Considerations · p. 10
  • The authors focus the empirical study on mainstream languages and encourage audits of bias when extending the framework to new ecosystems or developer populations.They provide PySnooper for Python and Solidity to encourage broader community-driven extension.Source: Ethical Considerations · p. 10

A.1 Experiment Detail.

These experiment details describe the study’s tuning and evaluation setup, refinement-dataset construction, and analysis of prediction uncertainty. The reported uncertainty results distinguish increased confidence from improved correctness.

A.1 Experiment Detail.

  • Fine-tuning uses full-parameter training and LoRA configurations; sequential revisions append Python-interpreter feedback for up to 8 rounds, while parallel scaling samples 16 candidates for reward-model ranking.The experiments also include greedy sampling of one answer at temperature 0 and use Phi-4 as the reward LLM for parallel candidate scoring.Source: §A.1 · p. 14

B.1 Finetune Dataset Quality Filtering and Decontamination

  • The refined training dataset contains 248,934 Python functions after benchmark-related functions and functions with low-quality or misleading docstrings are removed.Benchmark filtering checks for prompt or solution substrings; StarCoder2-15B classifies docstring quality.Source: §B.1 · pp. 14–15

B.2 Evaluation dataset

  • Evaluation covers program repair, code synthesis, and code reasoning, including input, output, state, and coverage prediction tasks.Repair prompts include buggy code, a failed test case, and its execution traces; repair datasets use 164 samples each from HumanEval, MBPP, and CRUXEval. Synthesis uses HumanEval and MBPP with base and plus test sets.Source: §B.2 · p. 15

B.3 Fine-tune dataset(Refinement Dataset)

  • The refinement dataset combines buggy code, patches, test cases, and execution traces; code pairs are matched by embedding similarity, and retained test cases fail on buggy code but pass on patched code.The dataset also adds reasoning examples for input, output, state, and coverage prediction, and enriches contest-problem descriptions with implementation constraints and test cases.Source: §B.3 · pp. 15–16

C.1 Prediction uncertainty analysis.

  • For code generation, prediction uncertainty positively correlates with model performance, while fine-tuning with semantic information increases confidence but not prediction correctness.The authors leave how to use semantic information to improve fine-tuning as an open problem.Source: §C.1 · p. 16

D Experiment results of Test-scaling on MBPP-R

On MBPP-R, test-time trace information generally improves code-repair performance, although no trace representation is consistently best. The accompanying material also reports pass-rate gains across five self-debugging rounds.

D Experiment results of Test-scaling on MBPP-R

  • In 98 of 112 cases, adding trace information improves performance by up to 10.85, with the maximum in NeXT using GPT-4o; no trace representation consistently outperforms the others.Concise performs best among representations under instruction versions, and Sequential Revisions and Parallel are identified as optimal test-time scaling strategies.Source: D Experiment results of Test-scaling… · p. 16, Figure 5 · p. 17, Table 8 · p. 19

E.1 Prompt templates

  • Pass rate rises from 61.54% in Round 1 to 88.46% by Round 5, while syntax errors drop sharply and more examples reach the testcase-pass category.The figure reports percentages by round, with blank cells indicating zero occurrences.Source: §E.1 · p. 19

Related papers

Loading 2509.11686v3…