PaperlayerView exact revision on arXiv

Source-linked AI summary

SelfPiCo: Self-Guided Partial Code Execution with LLMs

Zhipeng Xue, Zhipeng Gao, Shaohua Wang, Xing Hu, Xin Xia, Shanping Li

arXiv:2407.16974v1cs.SE

TL;DR

Partial code often cannot be executed because definitions or third-party dependencies are missing. SelfPiCo uses a fine-tuned Code Llama in an interactive loop to guide execution; 86 functions and 125 snippets were fully executed by its approach but failed under Lexecutor.

  • Problem

    Many code snippets are not directly executable because they lack variable or function definitions or third-party dependencies.

    Source: §1 · p. 1

  • Method

    SelfPiCo fine-tunes Code Llama for an interactive loop that guides partial-code execution using execution hooks and inferred values.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    86 functions and 125 code snippets were fully executed by SelfPiCo but failed by Lexecutor.

    Source: §4.6 · p. 9

  • Takeaways & Limitations

    The study reports that SelfPiCo detected 51 type errors, illustrating its usefulness for dynamic analysis of arbitrary code.

    Source: §8 · p. 11

    The main external validity threat is whether the selected testing dataset is representative, although its selection followed the baseline method's strategy.

    Source: §6 · p. 11

Abstract

from arXiv · show

Code executability plays a vital role in software debugging and testing (e.g., detecting runtime exceptions or assertion violations). However, code execution, especially partial or arbitrary code execution, is a non-trivial task due to missing definitions and complex third-party dependencies. To make partial code (such as code snippets posted on the web or code fragments deep inside complex software projects) executable, the existing study has proposed a machine learning model to predict the undefined element types and inject the pre-defined dummy values into execution. However, the performance of their tool is limited due to its simply designed dummy values and the inability to continue learning. In this paper, we design and implement a novel framework, named SelfPiCo (Self Guided Partial Code Executor), to dynamically guide partial code execution by incorporating the open-source LLM (i.e., Code Llama) within an interactive loop. Particularly, SelfPiCo leverages few-shot in-context learning and chain-of-thought reasoning to elicit human knowledge and logical reasoning based on fine-tuning the Code Llama model. SelfPiCo continuously learns from code execution results and refines its predictions step after step. Our evaluations demonstrate that SelfPiCo can execute 72.7% and 83.3% of all lines in the open-source code and Stack Overflow snippets, outperforming the most recent state-of-the-art Lexecutor by 37.9% and 33.5%, respectively. Moreover, SelfPiCo successfully detected 18 and 33 runtime type error issues by executing the partial code from eight GitHub software projects and 43 Stack Overflow posts, demonstrating the practical usage and potential application of our framework in practice.

1 Introduction

SelfPiCo targets incomplete code that blocks reuse and dynamic analysis, replacing fixed-value prediction with an interactive LLM-guided execution framework. Its evaluation reports higher execution coverage than Lexecutor and successful runtime type-error detection.

1 Introduction

  • 75% of code snippets cannot be directly executed or reused because they lack definitions or third-party dependencies, while partial execution can support dynamic analysis of key code areas.This can avoid complex builds and dependencies in large projects.Source: §1 · p. 1
  • Lexecutor predicts missing-element types and injects fixed dummy values, but those values can be too inflexible and its predictions do not adapt to execution results.A correctly predicted Callable can still fail when its dummy returns one value where the code expects two.Source: §1 · p. 2
  • An interactive value predictor regenerates values using execution errors, while a complementary type predictor supplies a predefined dummy value when the interactive predictor reaches its iteration limit.The runtime engine instruments partial code with hooks that catch exceptions and inject predicted values.Source: §1 · p. 2
  • Unlike automated program repair, SelfPiCo injects missing values without changing original code, aiming to make partial code executable rather than to pass all program tests.The paper presents partial-code execution as a base for dynamic analysis and notes that it can expose runtime errors without making bug repair its final goal.Source: §1 · p. 2
  • SelfPiCo executed 72.7% of all lines in open-source code and 83.3% in Stack Overflow snippets, outperforming Lexecutor by 37.9% and 33.5%, respectively.It also detected 18 type errors in eight GitHub projects and 33 in Stack Overflow posts.Source: §1 · p. 2
  • SelfPiCo engages Code Llama in an LLM-in-the-loop framework for partial-code execution, with fine-tuning and richer complementary dummy types.The fine-tuned model performs similarly to GPT-3.5 for guiding partial-code execution.Source: §1 · p. 2

2 Motivation

The motivation illustrates how partial execution can help expose runtime type errors without building or running an entire project. The authors present type-error detection as a preliminary application, with potential extension to other dynamic analyses.

2 Motivation

  • In the Luigi example, a string is passed where replace expects bytes, and the runtime type error remained undiscovered for more than two years amid complex dependencies.Static checkers primarily rely on type annotations, which are often unavailable.Source: §2 · p. 3
  • SelfPiCo reproduced the runtime error without building or running the whole project, illustrating checks on newly updated code without extensive test cases.The authors describe checking updated partial snippets during development as a way to expose potential errors early.Source: §2 · p. 3
  • Runtime type-error detection is a preliminary validation of practical usage; the framework is also described as extendable to applications such as assertion checking and taint analysis.Combining it with advanced dynamic checking techniques is left as future work.Source: §2 · p. 3

3 Our Approach

SelfPiCo uses an instrumented runtime engine to execute partial code and route undefined elements to an interactive value predictor. A complementary type predictor supplies a fallback when repeated LLM interactions fail.

3 Our Approach

  • The runtime engine instruments a non-executable snippet with hooks, executes it, and catches exceptions triggered by undefined elements.The raised exception and contextual information are sent to the interactive value predictor.Source: §3 · p. 3
  • The interactive value predictor infers and injects values for undefined elements, then adaptively regenerates candidates and checks whether execution succeeds.The engine's hooks pass the inferred values back into code execution.Source: §3 · p. 3
  • When repeated interactions fail to produce valid values, the complementary type predictor predicts an element's type and returns a predefined dummy value.This fallback is activated after the LLM fails to generate valid values through multiple interactions.Source: §3 · p. 3

3.1 Runtime Engine

The runtime engine instruments selected AST nodes with hooks that attempt ordinary reads and calls, then query the interactive value predictor when execution raises an exception. The hooks provide contextual information for generating replacement values.

3.1 Runtime Engine

  • Execution hooks catch exceptions during partial-code execution, request values from the interactive predictor, and inject the replies to continue execution.For variable reads, the hook receives the variable name and a lambda that attempts to read its value.Source: §3.1 · pp. 3–4
  • The engine instruments variable reads, attribute reads, and function or method calls in the code's abstract syntax tree.The corresponding hooks are _n_, _a_, and _c_.Source: §3.1 · p. 4
  • The hooks perform the read or call and query the interactive value predictor when exceptions such as NameError or AttributeError occur.Each query includes the element name and kind, its source line, and the execution error message.Source: §3.1 · p. 4

3.2 Interactive Value Predictor

The interactive value predictor generates and checks candidate definitions or assignments, using execution errors to refine predictions until the partial code runs. Its prompts combine role instructions, developer-selected examples, and chain-of-thought reasoning, while the LITL loop hands off to a backup predictor after repeated failures.

3.2 Interactive Value Predictor

  • The predictor checks generated definitions or assignments by executing them, then feeds execution errors back to refine values until the partial code executes successfully.The checker loads valid values for the runtime engine and queries the generator again when validation fails.Source: §3.2 · p. 4, §3.2.2 · p. 5

3.2.1 Interactive Value Generator.

  • The value-generator prompt casts the LLM as a Python programmer and restricts its output format.The prompt combines role instructions with six examples of chain-of-thought reasoning and outputs, using separate initial-assign and interactive-fix prompts.Source: §3.2.1 · pp. 4–5
  • Three experienced developers selected six representative examples by proposing values for undefined elements in 30 snippets each and reaching consensus after discussion.The examples provide desired inputs and outputs for few-shot learning.Source: §3.2.1 · p. 4
  • The chain-of-thought prompt divides prediction into importing modules, defining missing context, and assigning or interactively fixing undefined values.The assignment step uses the undefined element, its type, and the runtime error message; the fix step revises a prior prediction using its failure message.Source: §3.2.1 · p. 4

3.2.3 LITL (LLMs-In-The-Loop) Algorithm.

  • The LITL algorithm alternates value generation and execution, returning the result code and loaded value on success or triggering the complementary type predictor after t failed iterations.The initial assign prompt starts the loop; an interactive fix prompt supplies the previous prediction and failure message for refinement.Source: §3.2.3 · p. 5

3.3 Complementary Type Predictor

The complementary type predictor backs up value prediction when interactions are exhausted, predicting an undefined element's type and supplying a corresponding predefined dummy value. It classifies elements from contextual code and uses prompts with few-shot examples.

3.3 Complementary Type Predictor

  • When value prediction exceeds its interaction limit, the complementary predictor infers an undefined element's type and returns a corresponding predefined dummy value.SelfPiCo fine-tunes Code Llama on Lexecutor's training set and extends the predefined types with prompt engineering.Source: §3.3 · p. 6
  • The predictor's abstraction classes include built-in types, Callable, Object, Resource, and the added Tensor, Array, and DataFrame categories.Elements outside these classes receive a dummy object value.Source: §3.3.1 · p. 6

3.3.2 Prompt Engineering.

  • The type-prediction prompt asks the LLM to classify a code word into one of the listed categories, using one randomly selected few-shot example per abstraction class.The prompt supplies the undefined element and its contextual information and requests only the category.Source: §3.3.2 · p. 6

3.4 Implementation

The implementation fine-tunes Code Llama for missing-type prediction and uses the resulting model for both interactive value prediction and complementary type prediction. It also specifies few-shot counts and an iteration threshold for the predictors.

3.4 Implementation

  • The implementation uses the 34-billion-parameter Code Llama instruct model, fine-tuned on Lexecutor's code-element type dataset with PEFT and LoRA.Training used four A800 GPUs, a learning rate of 3e-4, batch size 128, two epochs, and a warmup ratio of 100.Source: §3.4 · p. 6
  • The adaptive value predictor uses six few-shot examples, while the complementary type predictor uses 15 examples, one for each abstraction class.The adaptive predictor's six examples cover two examples for each kind of undefined code element.Source: §3.4 · p. 6
  • The adaptive value predictor's maximum iteration threshold is five, set according to a small-scale pilot study.Source: §3.4 · p. 6

4 Evaluation

Across open-source project functions and Stack Overflow snippets, SelfPiCo outperforms Lexecutor-based models, while component, sensitivity, timing, and failure analyses qualify its gains. The evaluation also shows where limited context and instrumentation constrain execution.

4.1 Experimental Setup

  • Evaluation reused Lexecutor-style datasets—1,000 project functions and 586 Stack Overflow snippets—and measured code coverage and fully executed rate across five runs.The datasets contain 7,225 and 4,540 non-empty, non-comment lines, respectively.Source: §4.1 · pp. 6–7, Table 4 · p. 7

4.2 RQ1: Effectiveness of SelfPiCo

  • SelfPiCo significantly outperforms Lexecutor-CodeT5 and Lexecutor-CodeLlama on all metrics across both datasets.The comparison covers open-source project functions and Stack Overflow code snippets.Source: §4.2 · p. 7, Table 5 · p. 7
  • SelfPiCo fully executes 228 project functions (40%) and 185 Stack Overflow snippets (39.8%) that Lexecutor misses, while Lexecutor-only counts are 14 (2.5%) and 23 (4.9%).Most snippets Lexecutor covers are also covered by SelfPiCo.Source: §4.2 · p. 7, Figure 3 · p. 8

4.3 RQ2: Component Analysis

  • Removing either predictor reduces performance; the combined system benefits from their complementary strengths on Stack Overflow snippets and project functions.The interactive value predictor performs better on Stack Overflow, while the complementary type predictor performs better on project functions.Source: §4.3 · p. 8, Table 6 · p. 8

4.4 RQ3: Sensitivity Analysis

  • Performance rises rapidly in the first two interactions, with slower improvement after three; five interactions were judged reasonable for optimal results.The authors suggest limited code context or unclear execution errors may explain the slower later gains.Source: §4.4 · p. 8, Figure 4 · p. 8
  • Without chain-of-thought reasoning, code coverage, branch coverage, and fully executed rate decrease by 24.7%, 61.4%, and 45.9% on project functions, and 15.3%, 31.1%, and 27.5% on Stack Overflow snippets.The analysis drops the complementary type predictor for this setting.Source: §4.4 · p. 8, Table 7 · p. 8

4.5 RQ4: Time Cost Analysis

  • SelfPiCo predictions take 6.25s versus Lexecutor's 0.18–0.48s, while extra Lexecutor time leaves code coverage at 0.527 on open-source data and 0.624 on Stack Overflow.The authors attribute much of SelfPiCo's prediction time to interactive learning.Source: §4.5 · p. 9

4.6 Result Discussion

  • SelfPiCo's advantages include valid value injection, accurate type prediction, and broader data types, illustrated by tuple-unpacking and pandas DataFrame cases.Lexecutor's predefined DummyObject conflicts with the expected tuple value; SelfPiCo supports DataFrame by importing pandas and defining it as an extended type.Source: §4.6 · p. 9

5 Practical Applications

SelfPiCo was evaluated on reported Python type errors in GitHub projects and Stack Overflow posts, where it detected more cases than Lexecutor-CodeLlama. Its performance also has false positives and remains constrained by code complexity and available context.

5.1 Runtime Type Error Detection

  • The evaluation collected 42 errors from eight GitHub projects and 47 from 200 randomly selected Stack Overflow posts, excluding cases with ungeneratable domain knowledge or invalid syntax.A detection counted only when execution stopped at the same fault localization and reported the issue’s error message.Source: §5.1 · p. 10
  • 18/42 GitHub type errors and 33/47 Stack Overflow type errors were detected by SelfPiCo, versus 8 and 21, respectively, by Lexecutor-CodeLlama.Detection required matching the issue’s fault localization and error message.Source: §5.1 · p. 10
  • 11% of 100 sampled partial codes triggered false positives for SelfPiCo, compared with 28% for Lexecutor-CodeLlama.Of SelfPiCo’s false-positive cases, 7 involved insufficient code context and 4 involved complex variable values.Source: §5.1 · p. 10
  • Failures mainly involved overly complicated partial code or insufficient context to infer the types that trigger the reported error.For example, an ambiguous variable could plausibly be assigned a DateTime object.Source: §5.1 · p. 10

5.2 Discussion

  • Unlike unit-test generation, which assumes a complete, compilable method and injects values at function interfaces, SelfPiCo handles partial code and injects values at arbitrary execution points.The authors note this can support discovery of state-dependent and boundary-condition bugs that unit-test generation cannot essentially check.Source: §5.2 · p. 10

6 THREATS TO VALIDITY

The authors identify threats from stochastic LLM outputs and from whether the testing dataset represents the target setting. They combine five runs to mitigate the former and follow the baseline’s dataset-selection strategy to address the latter.

6 THREATS TO VALIDITY

  • LLM randomness may produce different outputs for the same input, so metrics combine results from five independent executions.Source: §6 · p. 10
  • Dataset representativeness is the main external validity threat; the evaluation follows the baseline’s selection strategy, while practical tests use developer-confirmed issues as ground truth.The authors identify detecting more runtime type errors in the wild as future work.Source: §6 · p. 11

7 Related Work

Related work addresses incomplete-code execution, runtime behavior prediction, program repair, and dynamic analysis. SelfPiCo is distinguished by executing partial code after predicting and injecting values, rather than only predicting behavior or modifying programs.

Incomplete code execution

  • Earlier incomplete-code execution tools inject values or force execution paths, whereas SelfPiCo predicts definitions and assignments and injects realistic values.Source: §7 · p. 11

Execution behaviour analysis

  • Runtime behavior studies predict variable values, types, execution branches, or errors, while SelfPiCo also practically executes the program.Source: §7 · p. 11

Automated Program Repair

  • Automated program repair modifies a program to pass compilation or execution, whereas SelfPiCo targets partial-code execution without modifying the target program or its semantics.Source: §7 · p. 11

Dynamic analysis for Python and Unit Test Generation

  • Python dynamic-analysis and unit-test tools generally require executable code or a complete, compilable method, while SelfPiCo targets unexecutable partial code.Unit-test generation injects values at defined interfaces; SelfPiCo can inject values at arbitrary points during execution.Source: §7 · p. 11

8 Conclusion and Future Work

SelfPiCo interactively fills partial code to enable dynamic analysis and detected 51 type errors in an exploratory practical-use study. The authors plan to explore execution-time information beyond error messages.

8 Conclusion and Future Work

  • SelfPiCo interactively fills partial code to make it executable for dynamic analysis.The conclusion describes the approach as leveraging LLMs to guide execution.Source: §8 · p. 11
  • 51 type errors were detected in an exploratory study of SelfPiCo’s practical use.The authors present this as evidence of usefulness for arbitrary-code dynamic analysis.Source: §8 · p. 11
  • The study uses execution error messages for LLM guidance and proposes exploring partial ASTs and API sequences in future work.Source: §8 · p. 11
Loading 2407.16974v1…