PaperlayerView exact revision on arXiv

Source-linked AI summary

InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback

John Yang, Akshara Prabhakar, Karthik Narasimhan, Shunyu Yao

arXiv:2306.14898v3cs.CLcs.LGcs.SE

TL;DR

Static coding benchmarks provide limited support for iterative error correction and comparison across heterogeneous interactive setups. InterCode standardizes interactive coding as a reinforcement-learning environment with code actions and execution observations, using safe Docker sandboxes and adaptable rewards. Across its Bash, SQL, and Python tasks, the framework demonstrates benefits from interaction, including GPT-4’s InterCode-SQL improvement from 9.1% to 73.7%.

  • Problem

    Static code generation can propagate errors, disconnect generated code from execution environments, and offers limited opportunity for intervention, while existing interactive studies are difficult to compare.

  • Method

    InterCode defines interactive coding as a standard reinforcement-learning environment where code actions produce execution observations, using Docker sandboxes and customizable rewards.

  • Results

    Across task difficulties and file systems, interactive performance was superior for all evaluated models, including GPT-4’s InterCode-SQL increase from 9.1% to 73.7%.

  • Takeaways & Limitations

    InterCode provides a challenging, extensible benchmark for studying interactive code understanding and generation across multiple languages and prompting strategies.

  • Takeaways & Limitations

    The current environments are limited to Bash, SQL, and Python, while the CTF dataset contains only four manually curated task instances.

Abstract

from arXiv · show

Humans write code in a fundamentally interactive manner and rely on constant execution feedback to correct errors, resolve ambiguities, and decompose tasks. While LLMs have recently exhibited promising coding capabilities, current coding benchmarks mostly consider a static instruction-to-code sequence transduction process, which has the potential for error propagation and a disconnect between the generated code and its final execution environment. To address this gap, we introduce InterCode, a lightweight, flexible, and easy-to-use framework of interactive coding as a standard reinforcement learning (RL) environment, with code as actions and execution feedback as observations. Our framework is language and platform agnostic, uses self-contained Docker environments to provide safe and reproducible execution, and is compatible out-of-the-box with traditional seq2seq coding methods, while enabling the development of new methods for interactive code generation. We use InterCode to create three interactive code environments with Bash, SQL, and Python as action spaces, leveraging data from the static NL2Bash, Spider, and MBPP datasets. We demonstrate InterCode's viability as a testbed by evaluating multiple state-of-the-art LLMs configured with different prompting strategies such as ReAct and Plan & Solve. Our results showcase the benefits of interactive code generation and demonstrate that InterCode can serve as a challenging benchmark for advancing code understanding and generation capabilities. InterCode is designed to be easily extensible and can even be used to create new tasks such as Capture the Flag, a popular coding puzzle that is inherently multi-step and involves multiple programming languages. Project site with code and data: https://intercode-benchmark.github.io

1 Introduction

InterCode addresses the limitations of static code generation by standardizing interactive coding with execution feedback. It provides a flexible benchmark and evaluates models across interactive tasks and prompting strategies.

  • Static code generation can propagate simple errors, disconnect generated code from downstream execution environments, and leave little room for human intervention.
  • InterCode is a language- and platform-agnostic interactive coding benchmark that treats generated code as actions and execution feedback as observations.
  • The framework uses self-contained Docker environments for safe execution while remaining compatible with traditional seq2seq generation methods.
  • InterCode supports Bash, SQL, and Python environments built from existing static datasets, and evaluates diverse models and prompting methods including ReAct and Plan & Solve.
  • The paper contributes a lightweight, extensible benchmark and empirical tools for converting static code datasets into interactive tasks.

2 Related Work

Prior coding benchmarks largely treat code generation as static instruction-to-code transduction, while newer execution-based and interactive approaches use heterogeneous settings. InterCode combines execution-based evaluation with flexible task construction to standardize comparisons.

  • Most coding benchmarks frame coding as sequence transduction rather than interactive decision-making with an execution environment.
  • Execution-based evaluation improves assessment of code functionality but often remains non-interactive, mapping context and code to execution and score without iterative feedback.
  • InterCode combines execution-based evaluation with flexible task construction, enabling diverse problem-solving paradigms within a unified coding formulation.

3 The InterCode Benchmark

InterCode formalizes interactive coding as a modular, sandboxed reinforcement-learning environment in which code actions change state and produce execution observations. The framework supports customizable rewards and implementations across Bash, SQL, and Python.

  • 3.1 Formulation: InterCode models coding as a partially observable Markov decision process with instructions, states, code actions, observations, transitions, and rewards.
  • 3.1 Formulation: Episodes continue through admissible code actions and execution feedback until submission, with Success Rate measuring episodes receiving reward 1 and Error % measuring non-admissible actions.
  • 3.2 Construction pipeline: Task construction is divided into environment construction, data collection, and reward design, enabling safe execution, flexible rewards, and adaptation of existing instructions.
  • 3.2 Construction pipeline: Docker virtual containers provide stateful execution sandboxes designed for safety and reproducibility across Docker-equipped machines.
  • 3.2 Construction pipeline: Datasets require natural-language queries and gold answers or code blocks, while reward functions compare execution outputs and state modifications with task-specific criteria.
  • 3.3 Implementations: The framework implements Bash and SQL environments from existing datasets, alongside a Python environment based on MBPP and its unit-test evaluation procedure.

4 Methods

InterCode evaluates interactive coding with Bash and SQL using several prompting strategies that vary from zero-shot generation to iterative feedback and structured reasoning workflows.

  • Experiments: Experiments assess current large language models on interactive coding tasks with Bash and SQL.The evaluation spans models from OpenAI and PaLM-2 families.
  • Prompting strategies: Figure 2 summarizes the four adjusted prompting strategies evaluated on InterCode.
  • Prompting strategies: Single Turn uses zero-shot prompting: the model generates code once, which is then evaluated in InterCode.The prompt provides a task description and specifies the programming language.
  • Prompting strategies: Try Again gives the agent multiple turns, feeding execution output back as observations until completion or n turns are exhausted.The approach is intended to mirror human software development.
  • Prompting strategies: ReAct and Plan & Solve adapt existing reasoning frameworks to interactive coding through prompts and task-specific workflows.Episodes terminate according to the agent’s own judgment in these approaches.

5 Experiments

InterCode experiments show that interactive execution feedback improves coding success across SQL and Bash, while prompting strategies and task structure shape how models use multiple turns. The benchmark also exposes late-turn limitations and supports more complex interactive tasks such as CTF.

  • Base models comparison: 73.7% versus 9.1%: GPT-4’s InterCode-SQL success rate increases substantially from single-turn to interactive evaluation.Interactive performance is superior across models, SQL difficulty levels, and Bash file systems.
  • Base models comparison: Interactive trajectories use later turns to discover context, correct errors through execution feedback, and iteratively construct or edit actions.Models also plan and modularize solutions for chained Bash commands and multi-subproblem SQL queries.
  • Failure cases: Success rate and error rate plateau as interaction turns accumulate, with late trajectories often repeating actions or failing to use relevant observations.The limitation is especially evident on hard and extra SQL tasks requiring context across several tables and clauses.
  • Prompting strategy comparison: ReAct and Plan & Solve generally outperform Try Again on SQL, raising success from 47.3% to 58.7% while using fewer turns and more admissible commands.Table 4 compares success rate, average turns, and error rate across prompting strategies.
  • Prompting strategy comparison: SQL emphasizes context discovery and conditional actions, whereas declarative, multi-step Bash instructions are more suited to planning and modular task decomposition.The differing task formulations create distinct code-understanding challenges.
  • Prompting strategy comparison: More flexible ReAct reasoning is generally superior to rigid Plan & Solve procedures, but structured prompts can still solve different subsets of tasks.For gpt-3.5-turbo, jointly solved tasks comprise 57% of successful SQL tasks and 27.6% of successful Bash tasks.
  • New tasks & datasets opportunities: InterCode-CTF contains 100 picoCTF objectives and requires multi-step reasoning across Bash, Python, assets, and execution feedback.Virtual containers provide a safe environment for potentially damaging actions such as rm -rf or sudo access.

6 Discussion

InterCode provides a lightweight, extensible interface for interactive coding in secure execution environments. Its API supports customization of task loading, execution, rewards, resetting, logging, and trajectory storage.

  • Discussion and limitations: InterCode’s conclusion emphasizes secure error recovery, dataset conversion through the InterCodeEnv API, and extensibility to new interactive coding tasks.The authors identify the current offering as limited to Bash, SQL, and Python and note that the manually curated CTF dataset contains four instances.
  • Framework interface: InterCodeEnv inherits the OpenAI Gym API and centralizes execution logic, logging, data management, sandboxed execution, and environment- or task-level customization.Creating an environment requires a subclass, a Dockerfile, and a dataset with query and gold fields; reward customization is optional.
  • Framework interface: The interface initializes a Docker container from a specified image, validates dataset formatting, configures logging, and supports preprocessing before each task episode.Preprocessing enables task-instance-specific customization of the execution environment.
  • Execution loop: Each interaction step processes an action, records the action-observation sequence, optionally computes reward on submission, and saves trajectory information.Saved metadata includes task information, interaction sequences, and reward data.
  • Customization: Environment-specific methods define action execution, reward calculation, and container resetting because invocation, scoring, and restoration differ across tasks.The default executor runs the action inside the container with a timeout, while reward functions compare actions with task-specific gold commands.

A.2 Bash Environment

The appendix defines task environments and their evaluation mechanisms for Bash, SQL, and Python. Bash rewards combine execution-output similarity with file-system changes, while SQL rewards assess record overlap and ordering.

  • Bash environment: InterCode-Bash uses Ubuntu-based Docker environments with multiple interchangeable file systems to evaluate commands across varied execution contexts.Three file systems were designed to support generalizability and include commands that produce meaningful state changes.
  • Bash environment: Bash task completion evaluates both the latest execution output and file-system modifications against the gold command.The reward uses lexical output similarity and penalizes missing or unnecessary file-system changes.
  • Bash environment: A Bash reward of 1 requires the correct file paths, correct modifications, and an execution output matching the gold output exactly.The function is designed to support complex rewards beyond exact-match success-rate evaluation.
  • SQL environment: InterCode-SQL adapts Spider development examples across 20 databases and initializes MySQL schemas and records inside Docker.The environment converts SQLite databases to MySQL format and prepares a unified database dump.
  • SQL environment: SQL correctness uses Intersection over Union for returned records, adjusts it with Kendall’s τ for ordering, and assigns task completion only to a correctly ordered exact record set.Non-record outputs such as error strings receive zero reward.

B Experiment Details

The experiments configure baseline models through inference API calls without performing model training.

  • The evaluations use inference requests to OpenAI, PaLM, and Hugging-Face API endpoints rather than model training.OpenAI and PaLM runs use deterministic sampling settings with one completion.

B.2 Additional Experiments & Analysis

Additional experiments examine schema information, admissible actions, robustness, and hybrid prompting. They show persistent interaction benefits but also reveal that longer horizons and fixed strategy sequences remain difficult for models.

  • SQL schema ablation: Providing the full relevant SQL schema improves Single Turn performance, but Try Again retains a non-trivial advantage.The ablation tests whether interaction benefits are merely caused by unequal access to schema information.
  • Admissible actions: 60.3% of second-turn SQL actions contain SHOW TABLES or DESC, indicating that models increasingly explore database context after initial errors.Prompting strategies, exploratory wording, and demonstrations reduce first-turn hallucinations of table and column names.
  • Admissible actions: Later-turn admissible-action rates do not improve consistently because small corrections often repeat mistakes or introduce new issues.The analysis identifies context discovery and modularized problem solving as more efficient alternatives to minor perturbations.
  • Robustness: Try Again consistently outperforms Single Turn across models in the robustness evaluation on comparable Bash file systems.The comparison excludes the intentionally harder first file system and reports standard errors for both modes.
  • Additional prompting strategy: Plan & Solve + Refine achieves a 45.5 average Success Rate on 44 SQL instances but does not significantly improve over the original Plan & Solve strategy.The fixed planning-then-refinement order does not provide enough adjustment based on execution output.

B.4 Task Trajectory Analyses

Trajectory analyses show that models often begin with a direct answer, while explicit reasoning strategies elicit broader use of execution feedback for interactive problem-solving.

  • Default model behaviors: Without a reasoning framework, models typically generate an answer immediately, then make incremental error corrections and occasional context-discovery commands.The Try Again trajectories in Bash and SQL illustrate this default behavior.
  • Prompting strategy effects: ReAct produces richer problem-solving behavior by prompting models to reason about execution outputs.The comparison highlights context discovery, error correction, and more extensive problem-solving than Try Again.
  • Prompting strategy effects: Better prompting strategies may substantially advance interactive coding by eliciting expressive and creative solutions to multiple task challenges.The paper presents this as a direction that InterCode can benchmark, while noting that the challenges cannot be overcome by one isolated reasoning technique.

B.5 Capture the Flag Analysis

Capture the Flag challenges require iterative interaction across exploration, debugging, and exploitation, and InterCode supports agents in carrying out these multi-step processes.

  • Task characteristics: Capture the Flag challenges typically require trial-and-error techniques, environment exploration, debugging, and dynamic exploitation across sequential steps.Their inherently interactive structure makes iterative agent approaches necessary.
  • Trajectory analysis: Without a hint about the sleuthkit library, the agent fails; with the hint, it installs the library and uses its functionality to analyze a disk image.The agent then employs the mmls command to inspect the disk image.
  • Trajectory analysis: InterCode enables agents to explore environments, decompose tasks, debug with traces and logs, and accumulate knowledge iteratively, resembling the approach of a human expert.The human expert used a similar approach after receiving the hint.

B.6 Human Performance Baseline

A small human baseline study found substantially more modularized and flexible reasoning than in agent trajectories, while the authors caution that the performance gap requires broader study.

  • Reasoning patterns: Human trajectories used modularized subproblems, whereas agents tended to generate a single self-contained command intended to solve the instruction in one go.The example separates identifying the most popular dog breed from computing the corresponding average weight.
  • Interpretation: The authors conclude from this small study that humans generally show more flexible and variegated reasoning capabilities than agents in interactive coding.They identify incorporating human reasoning and execution as guidance, feedback, or reward as a direction for improving models.
  • Limitations: The human-agent performance gap is not fully established because a more thorough and variegated study is required.The authors explicitly qualify their conclusion as based on a small study.
  • Evaluation setup: Prompting strategies are evaluated through templates that specify initial, instruction, and observation messages for agent-environment interaction.Observation messages may recur across an episode as the agent generates subsequent actions.

B.8 Supported Datasets

InterCode currently supports multiple datasets and languages, while ongoing work targets broader language, dataset, and task coverage, including approaches for compiled languages and non-code applications.

  • Current support: InterCode evaluates Bash and SQL with NL2Bash and Spider while supporting additional datasets across Bash, SQL, and Python.Table 10 summarizes the datasets supported by each environment, and repository guidance describes transformation and usage procedures.
  • Future coverage: The framework is being expanded to cover more languages, datasets, and tasks.This expansion is described as ongoing work.
  • Compiled language support: Compiled languages such as C, C++, Java, Go, and Rust are less malleable for the exact Bash or SQL interaction model.The paper proposes third-party interpreters or multi-language environments as viable support avenues.
  • Language selection: Bash, Python, and SQL were initially chosen because abundant datasets existed, while comparable datasets were relatively scarce for languages such as Java, C++, and JavaScript.The language-agnostic framework is intended to support task construction despite this dataset imbalance.
  • Beyond code generation: Interactive tasks beyond code generation can be converted into Python-based code interaction problems for activities including search, shopping, robotics, and multi-step mathematics.The paper presents code as a medium for communication in non-code interactive tasks.
  • Prompt templates: Prompt-template tables document Try Again, ReAct, Capture the Flag, and Plan & Solve evaluations.The Plan & Solve table distinguishes the optional Post-Plan Refinement Message used only for Plan & Solve + Refine.
Loading 2306.14898v3…