Source-linked AI summary
OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement
Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, Xiang Yue
TL;DR
Open-source code models often lack execution capabilities and iterative use of execution and human feedback for refinement. OpenCodeInterpreter addresses this gap with systems trained on 68K multi-turn Code-Feedback interactions, achieving competitive benchmark performance and further gains with synthesized human feedback from GPT-4.
Problem
Open-source code models often lack execution capabilities and the ability to use execution and human feedback for iterative refinement.
Method
OpenCodeInterpreter generates, executes, and iteratively refines code using compiler diagnostics and human feedback, trained on the 68K-interaction Code-Feedback dataset.
Results
OpenCodeInterpreter-33B achieves 83.2 (76.4) on the average (and plus versions) of HumanEval and MBPP, compared with GPT-4’s 84.2 (76.2), rising to 91.6 (84.6) with synthesized human feedback from GPT-4.
Takeaways & Limitations
OpenCodeInterpreter narrows the performance gap between open-source models and proprietary systems such as the GPT-4 Code Interpreter.
Takeaways & Limitations
Performance may vary across languages and domains, and the system may struggle with extremely complex or ambiguous user intents.
Abstract
from arXiv · showhide
The introduction of large language models has significantly advanced code generation. However, open-source models often lack the execution capabilities and iterative refinement of advanced systems like the GPT-4 Code Interpreter. To address this, we introduce OpenCodeInterpreter, a family of open-source code systems designed for generating, executing, and iteratively refining code. Supported by Code-Feedback, a dataset featuring 68K multi-turn interactions, OpenCodeInterpreter integrates execution and human feedback for dynamic code refinement. Our comprehensive evaluation of OpenCodeInterpreter across key benchmarks such as HumanEval, MBPP, and their enhanced versions from EvalPlus reveals its exceptional performance. Notably, OpenCodeInterpreter-33B achieves an accuracy of 83.2 (76.4) on the average (and plus versions) of HumanEval and MBPP, closely rivaling GPT-4's 84.2 (76.2) and further elevates to 91.6 (84.6) with synthesized human feedback from GPT-4. OpenCodeInterpreter brings the gap between open-source code generation models and proprietary systems like GPT-4 Code Interpreter.
1 Introduction
OpenCodeInterpreter addresses limitations in open-source code models by combining code execution with execution and human feedback for iterative refinement. Evaluations on HumanEval, MBPP, and EvalPlus show performance close to GPT-4 and higher with synthesized human feedback.
- Current code models have limited ability to use execution and human feedback for iterative refinement.Execution feedback helps correct syntactic and logical errors, while human feedback improves alignment with user requirements.
- OpenCodeInterpreter is an open-source family of systems for generating, executing, and iteratively refining code.It uses compiler diagnostics to correct errors and human insights to refine code generation.
- OpenCodeInterpreter is trained on Code-Feedback, a dataset containing 68K multi-turn interactions among users, code models, and compilers.
- 91.6 (84.6) is OpenCodeInterpreter’s performance with synthesized human feedback from GPT-4 on the average (and plus versions) of HumanEval and MBPP.
2 Code-Feedback
Code-Feedback is designed as a diverse, challenging multi-turn code instruction dataset incorporating execution and human feedback. Its construction combines curated sources, dialogue transformation, interaction simulation, and targeted code correction.
- Code-Feedback targets diverse and challenging real-world coding queries in a multi-turn dialogue structure.The dialogues incorporate execution feedback from compiler outputs and diagnostics, along with human feedback.
- The dataset draws queries from open-source datasets and LeetCode coding challenges.The construction uses five methods to satisfy the dataset’s diversity, complexity, and feedback requirements.
- Qwen-72B-Chat filters compiled instructions by complexity, retaining queries rated 4 or 5 for the seed set.
- Single-turn Packing: Single-turn Packing creates 16.6K multi-turn instances from 105K single-turn instances by grouping similar queries.BERT-base embeddings and nearest-neighbor selection identify related queries for multi-turn sequences.
- Interaction Simulation: Interaction Simulation executes preliminary code and feeds execution outcomes and compiler diagnostics to GPT-4 for follow-up responses.The process is repeated to model iterative interaction and solution refinement.
- Interaction Simulation: Simulated human feedback covers ten categories, including syntax, efficiency, functionality, bugs, security, compatibility, and scalability.
- Code Correction: Code Correction adds 500 error-correction interactions by using execution error messages as cues for revised code.
- LeetCode Similar Problem: GPT-4 enriches LeetCode solutions with natural-language explanations and code snippets in a consistent format.
3 Experimental Setup
The experiments fine-tune CodeLlama and DeepSeekCoder models on blended instruction data and evaluate single-turn and multi-turn code generation. Evaluation uses pass@1 and iterative scenarios based on execution and synthetic human feedback.
- Training Setup: Training uses CodeLlama and DeepSeekCoder models across 7B, 13B, 34B, and 70B scales.All models use uniform hyperparameters, three training epochs, a 2e-5 learning rate, and a 4096-token cutoff.
- Training Setup: The fine-tuning data combines WizardCoder 110k and Code-Feedback at a 2:1 ratio.
- Evaluation Setup: Evaluation applies greedy decoding for OpenCodeInterpreter and temperature 0 for GPT-3.5 Turbo and GPT-4 Turbo comparisons.EvalPlus’s sanitizer and evaluation tools process solutions across four benchmarks.
- Evaluation Setup: Single-turn performance is evaluated with pass@1 on HumanEval, MBPP, and their EvalPlus variants.Solutions use a simple instruction wrapping the original prompt.
- Evaluation Setup: Table 1 reports pass@1 accuracy for different models on HumanEval (+), MBPP (+), and their averages (+).The table distinguishes CodeLlama-based and DeepSeekCoder-based models and compares them with baselines.
- Multi-turn Evaluation: Multi-turn evaluation tests refinement using execution results and synthetic human feedback generated by GPT-4.The framework includes execution-feedback and synthetic-human-feedback scenarios.
- Multi-turn Evaluation: Each task allows at most two refinement rounds, reincorporating failed solutions and test results into the next prompt.
4 Main Results
OpenCodeInterpreter performs strongly in single- and multi-turn code generation, with iterative execution and human feedback supporting refinement. Ablations, leakage analysis, MT-Bench results, and case studies further examine its data sources, robustness, and practical behavior.
- 4.1 Results of Single-turn Code Generation: OpenCodeInterpreter-DS 33B achieves the highest scores among open-source models in the reported single-turn benchmark comparison.The comparison covers HumanEval, MBPP, HumanEval+, and MBPP+ across multiple model scales.
- 4.2 Results of Multi-turn Code Generation: OpenCodeInterpreter 33B matches GPT-4 Turbo’s single-round score under execution feedback and establishes a new SOTA benchmark among evaluated code models.The multi-turn evaluation limits iterations to two rounds for fairness and consistency.
- 4.2 Results of Multi-turn Code Generation: Human-feedback evaluations report that OpenCodeInterpreter 6.7B significantly outperforms GPT-4 Turbo’s single-round score, while the 33B Oracle variant exceeds 90 on HumanEval/MBPP.These assessments were restricted to the 6.7B and 33B models because of budget constraints.
- 4.3 Ablations of Data Sources: Adding high-quality single-turn data significantly improves multi-turn performance, while diverse feedback sources enhance debugging and refinement.Code Correction Data is specifically reported to improve error-correction efficiency.
- 4.4 Analysis of Dataset Leakage: Duplicate line ratios between Code-Feedback and HumanEval/MBPP benchmarks are notably low, indicating minimal overlap and mitigating dataset-leakage concerns.The duplicated lines are mostly generic, widely used snippets that contribute minimally to benchmark-specific improvements.
- 4.6 Case Study: Coding Queries in the Wild: Case studies show iterative correction and human-feedback-based enhancement on prime-number and IPv6 tasks, while a list-intersection task exceeded the three-attempt limit.The examples illustrate both dynamic algorithm adjustment and a concrete boundary of the refinement process.
5 Related Work
OpenCodeInterpreter builds on fine-tuning general-purpose LLMs and differs from reinforcement-learning approaches by using intermediate generations for easier, more stable training.
- 5 Related Work: OpenCodeInterpreter fine-tunes general-purpose LLMs rather than relying on specialized code-model pre-training.The paper motivates this choice by the extensive code exposure already present in general-purpose LLMs.
- 5 Related Work: Unlike reinforcement-learning approaches, the method accesses intermediate generations, making training easier and more stable.
6 Conclusion
OpenCodeInterpreter combines compiler diagnostics and human feedback with iterative refinement to improve open-source code generation and respond to evolving user intents and complex tasks.
- 6 Conclusion: OpenCodeInterpreter integrates compiler diagnostics and human feedback into iterative code refinement.The conclusion contrasts this process with traditional one-off generation approaches.
- 6 Conclusion: Code-Feedback provides extensive multi-turn interactions that support refinement in response to evolving user intents and complex coding tasks.
Ethics Statement
The paper describes dataset checks intended to support responsible use by addressing diversity, sensitive information, and security vulnerabilities.
- Ethics Statement: The dataset is designed to represent diverse coding styles, problem domains, and user scenarios to reduce biased or unfair outcomes.
- Ethics Statement: The authors check generated data for sensitive information and security vulnerabilities.
Limitations
OpenCodeInterpreter’s performance may vary across programming languages and domains, and the system may struggle with extremely complex or ambiguous user intents.
- Limitations: Performance may vary across different programming languages and specific domains.This limitation applies despite the system being designed for multi-language code generation and broad programming contexts.
- Limitations: OpenCodeInterpreter may struggle with extremely complex or ambiguous user intents.The limitation reflects constraints in the model’s current understanding and the specificity of Code-Feedback data.
A Source Data Filtering
The source data filtering process rates code queries by complexity and retains the most challenging instructions for the seed set. Figure A1 summarizes data quantities after each filtering stage, while Figure A2 shows the filtered dataset’s programming-language distribution.
- Query complexity scoring: Queries are rated on a 1-to-5 complexity scale using predefined scoring criteria.The prompts ask for a score followed by an explanation.
- Filtering overview: Figure A1 summarizes data quantities after each filtering stage.
- Dataset composition: Figure A2 illustrates the programming-language distribution after filtering.
B Simulating Interactions for Data Collection
The data-collection process simulates code-interpreter interactions through execution, human-style feedback, and deliberate error correction. Prompts structure both the assistant’s responses and the feedback exchanged during iterative refinement.
- Execution feedback: The code-interpreter prompt requires understanding the request, executing Python, analyzing results, and attempting fixes after errors.
- Human feedback: Human-feedback prompts ask for concise guidance based on the latest generated code and its execution results.Feedback is constrained to two short sentences and must address unmet points.
- Feedback categories: Feedback categories cover syntax, efficiency, functionality, clarity, bugs, security, compatibility, testing, and resource optimization.
- Code correction: The collection includes prompts that generate executable Python code containing intentional syntax or logical errors.These errors provide material for subsequent correction interactions.
- Code explanations: GPT-4 is prompted to generate explanatory text before code without modifying the code or restating prior dialogue.
D Model Evaluation Prompts
Evaluation prompts differ between HumanEval and MBPP, while simulated human-feedback prompts guide refinement using execution results and, in one variant, a canonical solution. The appendix specifies both task instructions and feedback-generation constraints.
- D Model Evaluation Prompts: HumanEval and HumanEval+ use identical prompts, while MBPP and MBPP+ use a similar shared prompt.
- Prompt for HumanEval and HumanEval+: HumanEval prompts require completing the given function without modifying the supplied code.The completed code must be returned in one code block.
- Prompt for MBPP and MBPP+: MBPP prompts provide the problem and test examples and require matching the tested functions’ input and output formats.Solutions must use the specified programming language and appear in one code block.
- Prompt for MBPP and MBPP+: GPT models are used to emulate human behavior in generating feedback.
- Prompt for GPT models mimicking human feedback with canonical solution: The canonical-solution feedback prompt compares existing code with the canonical solution and execution result to identify discrepancies and edge cases.It requests conceptual guidance rather than direct code and limits responses to three sentences.
- Prompt for GPT models mimicking human feedback without canonical solution: The feedback prompt without a canonical solution relies on execution results to identify errors, unexpected behavior, edge cases, and optimization opportunities.It likewise requests conceptual guidance without code and limits responses to three sentences.
E Examples of Methods used in Data Collection
The data-collection examples demonstrate query packing, simulated feedback, code correction, and follow-up interactions across open-source coding tasks and LeetCode problems. They include both successful refinements and a case where repeated corrections exceed the attempt limit.
- Collection methods: The examples cover similar-query packing, human-feedback simulation, code correction, similar-problem packing, and follow-up Q&A.
- Query examples: The collection includes coding queries involving file search, skyline construction, square placement, and space-fighter communication scenarios.
- Code correction: A GPA example shows execution feedback identifying incorrect lowest- and highest-grade logic and motivating a correction.The revised result reports the lowest and highest grades as letter grades derived from GPA values.
- Code correction: The corrected GPA code computes minimum and maximum GPA values directly before converting them to letter grades.
- Query examples: The space-fighter example uses Python to process positions, frequencies, velocities, and queries about blocked communications.
- Successful refinement: A prime-number task demonstrates successful correction and generalization from the range 1-100 to an arbitrary range.
- Successful refinement: An IPv6-validation task shows execution errors prompting recognition of a regular-expression mistake, explanation, and readability improvements.
- Unsuccessful refinement: A list-intersection task exceeds the maximum of three attempts after correcting only one error at a time.