Source-linked AI summary
Reasoning Runtime Behavior of a Program with LLM: How Far Are We?
Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, Xin Xia
TL;DR
Code-reasoning benchmarks often focus on program inputs and outputs, leaving intermediate runtime behavior and logical consistency across related predictions insufficiently assessed. REval adapts executable benchmarks to evaluate runtime behavior and Incremental Consistency; the study reports poor performance for most evaluated LLMs on both dimensions.
Problem
Existing code-reasoning evaluations often focus on predicting program inputs and outputs while omitting intermediate runtime behavior and logical consistency across sequentially related predictions.
Source: §I · pp. 1–2Method
REval adapts executable benchmarks to evaluate runtime behavior across four execution dimensions and consistency across sequentially related tasks.
Source: §I · p. 2Results
Most evaluated LLMs perform poorly on Runtime Behavior Reasoning and Incremental Consistency, and all evaluated open-source LLMs score below 20 on IC.
Source: §I · p. 2Takeaways & Limitations
The study highlights the importance of evaluating code LLM reasoning with runtime behavior and incremental consistency.
Source: §I · p. 2REval does not explore other dynamic program features such as memory allocation and exception handling.
Source: D. Threats to Validity · p. 11
Abstract
from arXiv · showhide
Large language models for code (i.e., code LLMs) have shown strong code understanding and generation capabilities. To evaluate the capabilities of code LLMs in various aspects, many benchmarks have been proposed (e.g., HumanEval and ClassEval). Code reasoning is one of the most essential abilities of code LLMs, but existing benchmarks for code reasoning are not sufficient. Typically, they focus on predicting the input and output of a program, ignoring the evaluation of the intermediate behavior during program execution, as well as the logical consistency (e.g., the model should not give the correct output if the prediction of execution path is wrong) when performing the reasoning. To address these problems, in this paper, we propose a framework, namely REval, for evaluating code reasoning abilities and consistency of code LLMs with program execution. We utilize existing code benchmarks and adapt them to new benchmarks within our framework. A large-scale empirical study is conducted and most LLMs show unsatisfactory performance on both Runtime Behavior Reasoning (i.e., an average accuracy of 44.4%) and Incremental Consistency Evaluation (i.e., an average IC score of 10.3). Evaluation results of current code LLMs reflect the urgent need for the community to strengthen the code reasoning capability of code LLMs. Our code, data, and \newname leaderboard are available at https://r-eval.github.io.
I. INTRODUCTION
Existing code-reasoning benchmarks often focus on input-output prediction, leaving intermediate runtime behavior and logical consistency across related predictions insufficiently evaluated. REval addresses these gaps with runtime-behavior tasks and Incremental Consistency (IC), and its study finds that most evaluated models perform poorly on both.
I. INTRODUCTION
- Existing code-reasoning evaluations often predict program inputs and outputs but ignore intermediate runtime information such as execution paths, limiting assessment of runtime behavior.The paper defines code reasoning as predicting execution behavior without directly running the code.Source: §I · p. 1
- REval evaluates four runtime behaviors: whether a statement executes, variable values and types, the next statement, and program output.These tasks are Code Coverage Prediction (CCP), Program State Prediction (PSP), Execution Path Prediction (EPP), and Output Prediction (OP).Source: §I · p. 2
- A model can predict the next execution statement correctly while failing to predict the resulting variable value, creating a logical inconsistency across sequential tasks.The paper notes that control flow depends on program state and distinguishes this issue from semantic consistency in prior work.Source: §I · p. 2
- Most evaluated LLMs perform poorly on both Runtime Behavior Reasoning and Incremental Consistency, with every evaluated open-source LLM scoring below 20 on IC.The study uses diverse general and code LLMs.Source: §I · p. 2
- REval adapts executable benchmarks including HumanEval and ClassEval, and provides its code, data, and leaderboard publicly.The framework extracts runtime behavior, constructs problems, and filters them.Source: §I · p. 2
II. BACKGROUND AND RELATED WORK
The background distinguishes information available before or after execution from intermediate runtime information and reviews how execution behavior is used in code research. Prior code-reasoning evaluations largely focus on pre- and post-execution information, while REval also targets runtime behavior.
1) Code Execution Behavior:
- Execution behavior includes pre- or post-execution information, such as inputs and outputs, and runtime information, such as program state and execution paths.Runtime information is available while code is running.Source: 1) Code Execution Behavior · p. 3
- Prior work uses execution behavior in code understanding, generation, testing, and vulnerability detection, including execution feedback and learned execution traces.Examples include using execution results to rerank generated code and feedback to guide code self-debugging.Source: 1) Code Execution Behavior · p. 3
2) Code Reasoning with Large Language Models:
- CRUXEval and related evaluations focus on pre- or post-execution information, whereas REval also measures intermediate runtime behavior.A separate line of work simulates execution with attention to algorithm complexity and structure; REval is described as applicable beyond algorithm problems to general programming scenarios.Source: 2) Code Reasoning with Large Language… · p. 3
B. Consistency for Large Language Models
Prior code-model consistency work emphasizes semantic consistency, while logical consistency across sequential code-reasoning tasks has received less attention. REval evaluates runtime behavior alongside consistency across incrementally harder tasks.
B. Consistency for Large Language Models
- Prior code-model consistency studies focus on semantic or functional consistency, while logical consistency concerns avoiding contradictory predictions and has received less attention for code LLMs.The paper identifies this gap as motivation for its Incremental Consistency metric.Source: B. Consistency for Large Language Models · p. 3
1) Code LLMs:
- Code LLMs include specialized models for generating and understanding programming languages, and code reasoning evaluates their predictions about program behavior without execution.The paper frames code reasoning as an important but comparatively under-evaluated capability.Source: 1) Code LLMs · pp. 3–4
2) Benchmarks:
- REval uses existing executable benchmarks and offers a runtime-behavior perspective distinct from code-understanding benchmarks such as CodeXGLUE.The paper characterizes its framework as applicable to different programming scenarios.Source: 2) Benchmarks · p. 4
A. Overview of Framework
- REval combines Runtime Behavior Reasoning across four execution dimensions with Incremental Consistency Evaluation across sequentially related tasks.Because later tasks require knowledge from earlier tasks, the task sequence increases progressively in difficulty and supports consistency assessment using existing predictions.Source: A. Overview of Framework · pp. 4–5
B. Runtime Behavior Reasoning
Runtime Behavior Reasoning covers code coverage, execution path, program state, and output; the supplied task descriptions detail coverage prediction and program-state prediction. Program-state predictions are evaluated jointly on variable value and type.
B. Runtime Behavior Reasoning
- Runtime Behavior Reasoning evaluates four dimensions of execution information: code coverage, execution path, program state, and output prediction.The paper presents these as representative dimensions of intermediate runtime information.Source: B. Runtime Behavior Reasoning · p. 5
1) Code Coverage Prediction (CCP):
- CCP asks whether a specified statement executes for a given program and input, with accuracy and F1 score used for evaluation.The task treats execution as a binary prediction for a statement index.Source: 1) Code Coverage Prediction (CCP) · p. 5
2) Program State Prediction (PSP):
- PSP predicts a variable’s value and type after a specified statement executes, and a prediction is correct only when both match the ground truth.The paper defines source-level program state as variables in the current runtime scope, each with a value and type.Source: 2) Program State Prediction (PSP) · p. 5