PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement

Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, Xiang Yue

arXiv:2402.14658v3cs.SEcs.AIcs.CL

TL;DR

Open-source code models have limited capacity to use feedback for refinement; OpenCodeInterpreter combines compiler diagnostics and human feedback in an iterative system. With synthesized GPT-4 feedback, it reaches 91.6 (84.6) and narrows the performance gap with proprietary systems.

  • Problem

    Current code models are constrained in their capacity to use execution and human feedback for refinement.

    Source: §1 · p. 2

  • Method

    OpenCodeInterpreter combines compiler diagnostics and human feedback for iterative code refinement, training on Code-Feedback’s 68K multi-turn interactions.

    Source: §1 · p. 2

  • Results

    91.6 (84.6) with synthesized GPT-4 feedback, OpenCodeInterpreter narrows the performance gap between open-source models and proprietary systems.

    Source: §1 · p. 2

  • Takeaways & Limitations

    OpenCodeInterpreter can dynamically refine code in response to evolving user intents and complex coding tasks.

    Source: §6 · p. 9

    The system may struggle with extremely complex or ambiguous user intents.

    Source: Limitations · p. 10

Abstract

from arXiv · show

The introduction of large language models has significantly advanced code generation. However, open-source models often lack the execution capabilities and iterative refinement of advanced systems like the GPT-4 Code Interpreter. To address this, we introduce OpenCodeInterpreter, a family of open-source code systems designed for generating, executing, and iteratively refining code. Supported by Code-Feedback, a dataset featuring 68K multi-turn interactions, OpenCodeInterpreter integrates execution and human feedback for dynamic code refinement. Our comprehensive evaluation of OpenCodeInterpreter across key benchmarks such as HumanEval, MBPP, and their enhanced versions from EvalPlus reveals its exceptional performance. Notably, OpenCodeInterpreter-33B achieves an accuracy of 83.2 (76.4) on the average (and plus versions) of HumanEval and MBPP, closely rivaling GPT-4's 84.2 (76.2) and further elevates to 91.6 (84.6) with synthesized human feedback from GPT-4. OpenCodeInterpreter brings the gap between open-source code generation models and proprietary systems like GPT-4 Code Interpreter.

1 Introduction

OpenCodeInterpreter combines code execution with execution and human feedback to iteratively refine generated code. Its 33B model approaches GPT-4’s benchmark performance, with higher reported scores when augmented by synthesized GPT-4 feedback.

1 Introduction

  • 83.2 (76.4) accuracy on the average (and plus versions) of HumanEval and MBPP puts OpenCodeInterpreter-33B near GPT-4’s 84.2 (76.2), rising to 91.6 (84.6) with synthesized GPT-4 human feedback.Figure 1 highlights comparable HumanEval pass@1 accuracy with appropriate feedback.Source: Figure 1 · p. 1, §1 · p. 2
  • Existing code models are limited in using execution outputs and diagnostics or human follow-up guidance to refine code and better meet user expectations.The introduction distinguishes execution feedback, which helps rectify syntactic and logical errors, from human feedback, which clarifies user instructions.Source: §1 · p. 2
  • OpenCodeInterpreter is trained on Code-Feedback’s 68K multi-turn user, code-model, and compiler interactions, integrating compiler diagnostics and human insights to refine code.Execution feedback helps correct syntactic and logical errors, while human feedback helps align solutions with user requirements.Source: §1 · p. 2

2 Code-Feedback

Code-Feedback is an instruction-tuning dataset for OpenCodeInterpreter, designed around challenging real-world coding queries, multi-turn dialogue with execution and human feedback, and responses interleaving explanations with code. It draws on open-source datasets and LeetCode challenges, using five construction methods to meet these goals.

2 Code-Feedback

  • Code-Feedback targets diverse, challenging real-world coding queries in multi-turn dialogues with execution and human feedback, blending natural-language explanations with code responses.The dataset draws queries from open-source datasets and LeetCode challenges and uses five construction methods.Source: §2 · p. 2

2.1 Coding Queries from Open-source Data

  • 156K high-quality single-turn code instructions were retained after Qwen-72B-Chat rated query difficulty and filtering kept only scores of 4 or 5.The filtering was repeated with two distinct prompts to strengthen the selection of challenging instructions.Source: §2.1 · p. 3
  • 16.6K multi-turn instances were created from 105K single-turn instances by grouping similar queries, selecting two or three of each query’s four nearest neighbors, and preventing neighbor reuse.Queries were embedded with BERT-base before nearest-neighbor selection.Source: §2.1 · p. 3
  • 51K examples were constructed by simulating code-interpreter interactions with GPT-3.5 and GPT-4, feeding execution outputs and compiler diagnostics into follow-up responses, then adding feedback from ten predefined categories.The execution-and-refinement cycle stopped when GPT-4 deemed the solution correct or after at most three iterations.Source: §2.1 · p. 3
  • 500 error-correction interactions used intentionally incorrect code and execution error messages as correction cues, then replaced the initial prompts with prompts for correct code generation.This stage was designed to teach both code generation and debugging.Source: §2.1 · p. 4

2.2 Coding Challenges from LeetCode

  • 303 multi-turn instances combined related LeetCode questions, while 200 multi-round instances presented alternative solutions to identical questions across complexity levels or programming languages.GPT-4 added natural-language explanations to solutions and standardized instances into a consistent format.Source: §2.2 · p. 4

3 Experimental Setup

The experimental setup fine-tunes CodeLlama and DeepSeekCoder variants using a blend of Code-Feedback and WizardCoder data, then evaluates single-turn generation and multi-turn refinement across four code benchmarks. Multi-turn tests examine responses to execution and synthetic human feedback under a two-round limit.

Training Setup

  • CodeLlama (Roziere et al., 2023) and DeepSeekCoder (Guo et al., 2024) models at 7B, 13B, 34B, and 70B are fine-tuned uniformly for three epochs.Training uses a 2e-5 learning rate, 0.05 warm-up ratio, cosine scheduler, and 4096-token cutoff.Source: §3 · p. 4
  • Fine-tuning blends Code-Feedback with high-quality WizardCoder 110k single-turn data at a 2:1 ratio.The authors state that this blend may further boost coding ability.Source: §3 · p. 4

Evaluation Setup

  • Evaluation covers HumanEval, MBPP, HumanEval+, and MBPP+ using EvalPlus to address edge cases in the original test suites.Solutions are generated with greedy decoding and post-processed by EvalPlus’s sanitizer; GPT-3.5 Turbo and GPT-4 Turbo comparisons use temperature 0.Source: §3 · p. 4
  • Single-turn code generation uses an instruction-wrapped original prompt and is assessed with pass@1.The exact prompts are provided in Appendix D.Source: §3 · p. 4
  • Multi-turn evaluation tests refinement using execution results and GPT-4 synthetic feedback in execution-feedback, synthetic-human-feedback, and oracle-feedback scenarios.In the oracle scenario, GPT-4 also sees the ground-truth solution.Source: §3 · pp. 4, 6
  • Refinement stops when a solution passes evaluation or after two rounds; failures feed the solution and test results back into the prompt.Feedback can report exceptions, mismatched expected and actual outputs with test inputs, or an execution timeout.Source: §3 · p. 6

4 Main Results

OpenCodeInterpreter performs strongly in single-turn and iterative code generation, with results highlighting the value of feedback and varied training data. The analyses also report low benchmark overlap, improved second-turn performance on MT-Bench, and both successful refinements and limitations in challenging user queries.

4.1 Results of Single-turn Code Generation

  • OpenCodeInterpreter configurations matched or surpassed leading benchmarks, and the DS 33B variant achieved the highest scores among open-source models.The authors note this result despite substantial low-quality or incorrect data in the initial training set.Source: §4.1 · p. 6

4.2 Results of Multi-turn Code Generation

  • With a two-round limit, all model scales surpassed state-of-the-art benchmarks with execution feedback, and 33B matched GPT-4 Turbo’s single-round score; human-feedback results also showed strong gains.The 6.7B model with Human Feedback exceeded GPT-4 Turbo’s single-round score, while the 33B model with Human Feedback (Oracle) averaged above the 90 benchmark on HumanEval/MBPP.Source: §4.2 · pp. 6–7

4.3 Ablations of Data Sources

  • Adding high-quality WizardCoder single-turn data significantly improved multi-turn performance by supplying syntactically accurate and logically coherent inputs for later refinement.The study varied the proportion of WizardCoder 110K data in the multi-turn dataset across six configurations.Source: §4.3 · p. 7
  • Single-turn Packing, Interaction Simulation, and Code Correction Data enhanced debugging and refinement, with Code Correction Data notably improving error-correction efficiency.The sources were tested individually and in combination.Source: §4.3 · p. 7

4.4 Analysis of Dataset Leakage

  • Code-Feedback had low duplicate-line ratios with HumanEval (+) and MBPP (+), and most duplicated lines were generic, indicating low dataset-leakage risk.The analysis compared consecutive code lines and concluded that overlap was minimal.Source: §4.4 · p. 8, Table 4 · p. 8

4.5 Evaluation of Multi-turn Settings

  • On MT-Bench coding questions, OpenCodeInterpreter-DS-33B and DS-6.7B tended to outperform other open-source models, and OpenCodeInterpreter improved in the second turn versus DeepSeek-Coder-33B-Instruct.The evaluation used 10 coding questions, each with a primary and follow-up question, and GPT-4 scoring from 1–10.Source: §4.5 · p. 8, Table 5 · p. 8

4.6 Case Study: Coding Queries in the Wild

  • Case studies showed successful iterative refinement for prime-number and IPv6 tasks, but the model failed to resolve multiple challenging errors in a list-intersection task within three attempts.The successful cases incorporated human feedback; the difficult case exposed limitations in addressing several errors simultaneously.Source: §4.6 · pp. 8–9

5 Related Work

Prior code LLM work includes code in general-model pretraining or specializes models for code, while this work fine-tunes general-purpose LLMs. Earlier refinement methods use feedback across tasks and code; this approach avoids reinforcement learning and uses intermediate generations.

LLMs for Code

  • Code LLMs are built by including code in general-model pretraining or pretraining specialized models, whereas this work fine-tunes general-purpose LLMs to leverage their existing code exposure.The authors argue that fine-tuning general-purpose models, already trained on extensive code, lets them explore ways to improve code generation.Source: §5 · p. 9

Iterative Code Generation and Refinement

  • Prior iterative-generation work spans script generation and summarization, while Self-Refine (Madaan et al., 2023) uses LLM-generated feedback to iteratively update initial outputs.This paper focuses that feedback-driven refinement paradigm on code generation.Source: §5 · p. 9
  • DebugBench (Tian et al., 2024) observes that incorporating runtime feedback improves code LLMs’ debugging performance, while StepCoder (Dou et al., 2024) follows reinforcement learning with compiler feedback.Source: §5 · p. 9
  • Unlike StepCoder (Dou et al., 2024), this approach avoids reinforcement learning and accesses intermediate generations, making training easier and more stable.The comparison concerns StepCoder's reinforcement-learning approach with compiler feedback.Source: §5 · p. 9

6 Conclusion

OpenCodeInterpreter bridges open-source code generation and proprietary systems by combining compiler diagnostics and human feedback for iterative refinement. Its Code-Feedback dataset supports adapting code to evolving user intents and complex tasks.

6 Conclusion

  • OpenCodeInterpreter bridges the gap between open-source models and proprietary systems such as GPT-4 Code Interpreter.The conclusion frames it as a significant advance in code generation.Source: §6 · p. 9
  • Integrating compiler diagnostics and human feedback enables iterative refinement beyond traditional one-off code generation.The paper describes this process as increasing adaptability and precision in open-source models.Source: §6 · p. 9
  • Code-Feedback’s extensive multi-turn interactions help OpenCodeInterpreter dynamically refine code for evolving user intents and complex coding tasks.Source: §6 · p. 9

Ethics Statement

The authors address responsible use through dataset diversity and safety checks, while emphasizing broader access through open release.

  • The dataset was designed to represent diverse coding styles, domains, and user scenarios to reduce biased or unfair outcomes.Source: Ethics Statement · p. 9
  • The authors checked the dataset for sensitive information and potential security vulnerabilities.Source: Ethics Statement · p. 9
  • OpenCodeInterpreter could lower barriers to coding, and the authors open-source its code, models, and datasets to maximize accessibility.Source: Ethics Statement · p. 9

Limitations

OpenCodeInterpreter’s performance may vary across languages and domains, and it may struggle with extremely complex or ambiguous user intents.

Limitations

  • Performance may vary across different languages and specific domains despite support for multilingual code generation and varied programming contexts.The limitation applies to both OpenCodeInterpreter and the Code-Feedback supporting it.Source: Limitations · p. 9
  • Extremely complex or ambiguous intents may exceed the model’s current understanding and the specificity of Code-Feedback.Source: Limitations · p. 10

A Source Data Filtering · Query Filtering Prompt 1 · Query Filtering Prompt 2

The supplied passages outline source-data filtering prompts and two query-filtering rubrics that rate query complexity and request a score before its explanation. They also reference figures summarizing filtering-stage data quantities and the filtered dataset’s language distribution.

A Source Data Filtering

Query Filtering Prompt 1

  • Query Filtering Prompt 1 rates complexity from 1 (easiest) to 5 (most difficult), with criteria spanning very basic operations to expert-level problem solving.The intermediate levels describe fundamental concepts, multi-step tasks, and advanced logic, algorithms, or data structures.Source: Query Filtering Prompt 1 · p. 13
  • Prompt 1 supplies a query placeholder and requires choosing only from a provided list.Source: Query Filtering Prompt 1 · p. 13
  • Prompt 1 asks for the score first, followed by an explanation.Source: Query Filtering Prompt 1 · p. 13

Query Filtering Prompt 2

B Simulating Interactions for Data Collection … Prompt for generating natural language explanations using GPT-4

The appendix specifies prompt templates for simulating multi-turn execution and human feedback, deliberately generating incorrect code, and asking GPT-4 to write natural-language code explanations.

B Simulating Interactions for Data Collection

  • The section illustrates prompts used for multi-turn execution-feedback and human-feedback interactions.The supplied execution-feedback passage is truncated before completing its description.Source: B Simulating Interactions for Data… · p. 15

System prompt for multi-turn execution feedback

  • The execution-feedback prompt asks the assistant to understand the request, explain its plan, execute Python, analyze results, fix errors, and respond in the user's language.These instructions define an iterative workflow from interpreting the request through execution and error correction.Source: System prompt for multi-turn execution… · p. 15

System prompt for multi-turn human feedback

  • The human-feedback prompt requires brief JSON feedback that identifies achieved and unmet code goals and bases its feedback on an unmet goal.It allows feedback to prompt code generation when no code is available and covers syntax, efficiency, functionality, clarity, bugs, security, compatibility, resource use, scalability, and best practices.Source: System prompt for multi-turn human… · p. 15

System prompt for deliberately generating incorrect code

  • The deliberately incorrect-code prompt requires at least one error while instructing the model to present the code as if it were correct.The listed error types are syntax, logical, type, name, and timeout errors; the model must not explain or disclose the mistakes.Source: System prompt for deliberately… · p. 16

C Natural Language Explanations Generation

  • The natural-language-explanation section uses GPT-4 to generate explanations for code.The following prompt specifies that the generated explanation should introduce the code and respond to the latest problem.Source: C Natural Language Explanations… · p. 16

Prompt for generating natural language explanations using GPT-4

  • The GPT-4 prompt requests only new pre-code text responding to the latest problem, potentially describing the language, algorithm, or implementation steps.It provides previous dialogues, the recent problem, and the correct code as context, and prohibits modifying the code or restating earlier dialogue.Source: Prompt for generating natural language… · p. 16

D Model Evaluation Prompts … Prompt for GPT models mimicking human feedback without canonical solution

The evaluation appendix specifies separate initial-generation prompts for HumanEval/HumanEval+ and MBPP/MBPP+, alongside GPT prompts that elicit human-like, code-free feedback with or without access to a canonical solution.

D Model Evaluation Prompts

  • The initial solution-generation prompts are shared within the HumanEval pair and within the MBPP pair, with a distinct prompt used for each benchmark pair.The passage describes the MBPP prompts as similar, rather than identical.Source: D Model Evaluation Prompts · p. 17

Prompt for HumanEval and HumanEval+

  • For HumanEval and HumanEval+, the prompt asks models to complete the supplied code without modifying it and return the completed code in one code block.The prompt supplies the original problem and specifies the programming language.Source: Prompt for HumanEval and HumanEval+ · p. 17

Prompt for MBPP and MBPP+

  • For MBPP and MBPP+, the prompt supplies the problem and test examples, asks for matching functions and input/output formats, and requires one code block.Source: Prompt for MBPP and MBPP+ · p. 17

Prompt for GPT models mimicking human feedback with canonical solution

Prompt for GPT models mimicking human feedback without canonical solution

  • Without a canonical solution, the GPT prompt bases feedback on execution results, asks for analysis of errors and edge cases, and requests code-free conceptual guidance in no more than three sentences.Source: Prompt for GPT models mimicking human… · p. 18

E Examples of Methods used in Data Collection · F Case Study

The appendix illustrates five data-collection methods and case studies showing both successful code refinement from execution and human feedback and a case where iterative correction failed within the attempt limit.

E Examples of Methods used in Data Collection

F Case Study

  • The prime-number case generalized code from the range 1–100 to arbitrary x–y and successfully corrected errors using human feedback.Figure A8 describes this as a good case of refinement.Source: Figure A8 · p. 24
  • For an IPv6-validation regex, execution errors helped the model identify a small mistake, explain the regex, and improve its readability with inline comments.Figure A9 describes this as a good case.Source: Figure A9 · p. 25
  • In the list-intersection case, the model corrected only one error per attempt and exceeded the three-attempt limit.Figure A10 identifies this as a bad case.Source: Figure A10 · p. 26

Related papers

Loading 2402.14658v3…