PaperlayerView exact revision on arXiv

Source-linked AI summary

BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions

Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, Binyuan Hui, Niklas Muennighoff, David Lo, Daniel Fried, Xiaoning Du, Harm de Vries, Leandro Von Werra

arXiv:2406.15877v4cs.SEcs.AIcs.CL

TL;DR

Existing coding benchmarks mainly test short, self-contained tasks, leaving diverse tool use and complex instruction following insufficiently assessed. BigCodeBench evaluates these capabilities across multi-tool Python tasks, finding that GPT-4o solves merely 60% of BigCodeBench-Complete tasks.

  • Problem

    Existing benchmarks mainly evaluate short, self-contained, algorithm-focused tasks rather than diverse function calls and complex instructions required in real-world programming.

  • Method

    BigCodeBench uses human-LLM collaboration to construct 1,140 Python tasks covering 723 function calls from 139 libraries across 7 domains, plus an instruction-oriented variant.

  • Results

    GPT-4o solves merely 60% of BigCodeBench-Complete tasks and less than 50% of BigCodeBench-Instruct tasks, while models still struggle with multi-tool use and complex instructions.

  • Takeaways & Limitations

    The findings show that substantial progress remains necessary for LLMs to reliably use tools and follow complex programming instructions.

  • Takeaways & Limitations

    During benchmark construction, GPT-4 struggled with mocked test cases and could become stuck resolving runtime bugs, making continuous human feedback essential.

Abstract

from arXiv · show

Task automation has been greatly empowered by the recent advances in Large Language Models (LLMs) via Python code, where the tasks ranging from software engineering development to general-purpose reasoning. While current benchmarks have shown that LLMs can solve tasks using programs like human developers, the majority of their evaluations are limited to short and self-contained algorithmic tasks or standalone function calls. Solving challenging and practical tasks requires the capability of utilizing diverse function calls as tools to efficiently implement functionalities like data analysis and web development. In addition, using multiple tools to solve a task needs compositional reasoning by accurately understanding complex instructions. Fulfilling both of these characteristics can pose a great challenge for LLMs.To assess how well LLMs can solve challenging and practical tasks via programs, we introduce BigCodeBench, a benchmark that challenges LLMs to invoke multiple function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained tasks. To evaluate LLMs rigorously, each task encompasses 5.6 test cases with an average branch coverage of 99%. In addition, we propose a natural-language-oriented variant of BigCodeBench, BigCodeBench-Instruct, that automatically transforms the original docstrings into short instructions only with essential information. Our extensive evaluation of 60 LLMs shows that LLMs are not yet capable of following complex instructions to use function calls precisely, with scores up to 60%, significantly lower than the human performance of 97%. The results underscore the need for further advancements in this area.

1 INTRODUCTION

BigCodeBench is introduced to evaluate Python code generation for practical tasks requiring diverse tool calls and complex instructions. Across 60 models, even GPT-4o solves only 60% of BigCodeBench-Complete tasks and less than 50% of BigCodeBench-Instruct tasks.

  • Motivation: Real-world programming tasks require diverse function-call sequences as tools and the ability to follow complex instructions.Domain-specific libraries provide comprehensive functionalities, while task complexity makes accurate tool use and instruction following necessary.
  • Benchmark Challenges: Constructing a high-quality execution-based benchmark is difficult because realistic tasks with complex instructions are hard to source and often require cross-file information.These constraints differ from short, self-contained exercises such as those in HumanEval.
  • Benchmark Construction: BigCodeBench contains 1,140 rich-context, multi-tool-use Python programming tasks covering 723 function calls from 139 popular libraries across 7 domains.The benchmark is built through LLM sourcing, program refactoring, and test-case addition under constant human supervision.
  • Results: 60 models were evaluated, and GPT-4o solves 60% of BigCodeBench-Complete tasks and less than 50% of BigCodeBench-Instruct tasks.The results indicate difficulty invoking multiple function calls from cross-domain libraries and following more natural instructions.

2 BENCHMARK CONSTRUCTION

BigCodeBench is constructed through a human–LLM collaboration pipeline that synthesizes, refactors, tests, and curates function-level programming tasks. BigCodeBench-Instruct further transforms Complete prompts into natural-language-oriented instructions with author-based quality control.

  • BigCodeBench-Complete: The Complete benchmark follows three stages: data synthesis, semi-automatic program refactoring and test-case generation, and human curation.The construction process is presented as a human–LLM collaboration framework.
  • Data Synthesis: LLMs synthesize customized, self-contained programming tasks from API-usage snippets and brief human instructions while incorporating diverse libraries.This approach addresses the difficulty of using repository functions that require cross-file information or customized modules.
  • Program Refactoring and Testing: GPT-4 iteratively refines synthesized functions and writes unit tests, but human feedback is needed when mocking tests fail or runtime-bug resolution stalls.The refinement guidelines target realism, reduced ambiguity, and functional correctness.
  • Program Refactoring and Testing: 1,223 refactored programming tasks with paired test cases remain after incomplete test cases and invalid programs are removed.Incomplete files arose because GPT-4 often omitted partial content when writing long contexts.
  • Human Curation: Human curation examines tasks, pre-evaluates them with GPT-3.5-Turbo, and cross-checks docstrings and test cases with seven additional annotators.Curation adds tests, resolves flaky-test runtime issues, clarifies instructions based on model failures, and finalizes consistency across tasks.
  • BigCodeBench-Instruct: BigCodeBench-Instruct transforms Complete prompts into more natural instructions by omitting non-essential details, then has five new authors inspect sampled prompts and solutions for alignment.The variant targets natural-language-oriented programming-task instructions relevant to instruction-tuning and downstream applications.

3 BENCHMARK STATISTICS

BigCodeBench is designed as a more rigorous and complex function-level Python benchmark than existing alternatives. It also substantially broadens tool-use coverage through hundreds of function calls across standard and external libraries.

  • Overall Statistics: BigCodeBench provides a more rigorous execution-based evaluation of task solutions than mainstream function-level Python benchmarks.
  • Overall Statistics: BigCodeBench has complexity similar to HumanEval and substantially higher than DS-1000 and ODEX.Its high cyclomatic complexity indicates that solving the tasks requires non-trivial programming reasoning.
  • Tool Statistics: 281 function calls from 77 standard libraries and 442 from 62 external libraries make BigCodeBench’s tool coverage more diverse than other benchmarks.

4 EVALUATION

The evaluation measures functional correctness and tool use across BigCodeBench-Complete and BigCodeBench-Instruct with Pass@1, calibrated results, and library-usage analysis. Results show strong ranking alignment, benefits from instruction tuning, sensitivity to instruction verbosity, and frequent use of imported libraries.

  • Evaluation setup: 60 state-of-the-art LLMs were evaluated on BigCodeBench-Complete and 35 instruction-tuned LLMs on BigCodeBench-Instruct using unbiased Pass@K, with Pass@1 and greedy decoding for the main zero-shot experiments.Pass@5 uses random sampling with N=5, temperature 0.8, and top-p 0.95 in Appendix L.
  • Task-solving performance: Pearson’s r between model rankings on BigCodeBench-Complete and BigCodeBench-Instruct is 0.982, indicating strong alignment.The evaluation also reports calibrated Pass@1 after adding missing setup omitted by models.
  • Generation failures: Instruction-tuned LLMs can omit essential import statements in BigCodeBench-Complete, causing failures because required modules and constants are undefined.The omission is more likely when models must repeat long contexts, a behavior described as “model laziness” in long-context interactions.
  • Instruction tuning: Instruction-tuned LLMs achieve 40.7% mean calibrated Pass@1 on BigCodeBench-Complete versus 35.7% for paired base LLMs.The comparison indicates that instruction tuning improves adherence to complex programming constraints.
  • Instruction verbosity: BigCodeBench-Instruct yields an average Pass@1 decrease of 8.5% relative to BigCodeBench-Complete, although model rankings remain similar.The result suggests difficulty understanding condensed, less-verbose programming requirements.
  • Tool use: Models use imported libraries in more than 70% of tasks, while in the remaining 20% they tend to add libraries, most of which are standard.Tool-use analysis covers all 60 calibrated models on BigCodeBench-Complete and treats a library as correctly used when every test case passes.

5 RELATED WORK

Related work spans code-focused large language models and programming benchmarks. Existing benchmarks largely emphasize short, self-contained algorithmic tasks, while newer open-domain benchmarks target application-specific scenarios with selected tools and libraries.

  • Large Language Models for Code: Codex was the first base LLM pre-trained on code and became the backbone model for GitHub Copilot.
  • Large Language Models for Code: Subsequent base code models were developed to perform accurate code completion.
  • Programming Benchmarks: Most programming benchmarks evaluate short, self-contained, algorithm-specific tasks, exemplified by HumanEval and MBPP.
  • Programming Benchmarks: Recent open-domain coding benchmarks instead challenge code LLMs in application-specific scenarios using specific tool and library sets.

6 CONCLUSION · APPENDIX · A CONTRIBUTIONS

BigCodeBench is introduced as a human–LLM-constructed programming benchmark for tool use and complex instruction following. Evaluation of 60 LLMs shows substantial room for improvement and motivates continued work on advanced code models.

  • 6 CONCLUSION: BigCodeBench assesses tool use and complex instruction following in programming tasks.The benchmark was constructed through collaboration between human experts and LLMs.
  • 6 CONCLUSION: 60 LLMs were extensively evaluated, revealing substantial room for models to improve on the benchmark.The authors also share findings that may improve performance and urge development of more advanced code LLMs.
  • APPENDIX: The appendix includes a section on further analysis.The listed section appears as “L.2 Further Analysis” on page 45.
  • A CONTRIBUTIONS: Project leadership was provided by Terry Yue Zhuo, while benchmark construction involved Chien Vu, Jenny Chim, Han Hu, Haolan Zhan, Xiaoheng Hong, Wenhao Yu, and additional contributors.The benchmark-construction list contains 21 named contributors in total.
  • A CONTRIBUTIONS: Experiments were conducted by Terry Yue Zhuo and Binyuan Hui, and the evaluation framework was developed by Terry Yue Zhuo, Indraneil Paul, Jiawei Liu, and Zhoujun Cheng.These roles are listed separately in the contributions statement.
  • A CONTRIBUTIONS: Analysis was contributed by Terry Yue Zhuo, Binyuan Hui, Zhoujun Cheng, Alex Gu, and Naman Jain, while Terry Yue Zhuo handled paper writing and other contributors edited the presentation.Long-term advice and support were provided by Daniel Fried, Niklas Muennighoff, Qian Liu, Zijian Wang, Binyuan Hui, Xiaoning Du, David Lo, Jiawei Liu, Harm de Vries, and Leandro von Werra.

B DATACARD … C.4.3 IF OTHERS WANT TO EXTEND/AUGMENT/BUILD ON/CONTRIBUTE TO THE DATASET, IS THERE A MECHANISM FOR THEM TO DO SO?

BigCodeBench’s datacard and data sheet centralize benchmark information, document its purpose and limitations, and describe release, licensing, maintenance, and updating practices. The benchmark targets practical programming tasks requiring diverse function calls, complex instruction following, and compositional reasoning.

  • B DATACARD / C DATA SHEET: BigCodeBench’s datacard centralizes information relevant to benchmark analysis and follows established documentation frameworks.The datacard follows Bender and Friedman, while the data sheet follows Gebru et al.’s documentation frameworks.
  • B DATACARD: Annotator communications used mainstream English (en-US), but some annotators’ first language was not English, potentially causing inaccurate task-prompt expressions.Annotator nationalities were not provided because the benchmark was considered unrelated to regional or social dialects.
  • C.1.1 FOR WHAT PURPOSE WAS THE DATASET CREATED?: The dataset was created to assess programming-task capability, emphasizing diverse function calling and complex instruction following across challenging and practical tasks.It aims to help stakeholders understand abilities and limitations associated with deploying LLMs.
  • C.1.1 FOR WHAT PURPOSE WAS THE DATASET CREATED?: The benchmark was designed to be easy to use, practical across programming scenarios, and challenging enough to require compositional reasoning and instruction-following capabilities.The paper contrasts these expectations with repository-level, algorithm-specific, domain-specific, and simple-task benchmarks.
  • C.1.1 FOR WHAT PURPOSE WAS THE DATASET CREATED?: BigCodeBench is presented as an easy-to-use benchmark with practical and challenging programming tasks and an accompanying end-to-end evaluation framework.Its stated aim is to assess programming-task solving in an open-ended setting.
  • C.2 COMPOSITION/COLLECTION PROCESS/PREPROCESSING/CLEANING/LABELING AND USE: Answers about the dataset’s composition, collection, preprocessing, cleaning, labeling, and use are described in the paper and GitHub repository.The repository is identified as bigcode-project/bigcodebench-annotation.
  • C.3.1 WILL THE DATASET BE DISTRIBUTED TO THIRD PARTIES OUTSIDE OF THE ENTITY (E.G., COMPANY, INSTITUTION, ORGANIZATION) ON BEHALF OF WHICH THE DATASET WAS CREATED?: The dataset is managed and maintained by the BigCode community rather than distributed to third parties on behalf of the creating entity.The cited passage answers the third-party distribution question with “No.”
  • C.3.3 WHEN WILL THE DATASET BE DISTRIBUTED? / C.3.4 WILL THE DATASET BE DISTRIBUTED UNDER A COPYRIGHT OR OTHER INTELLECTUAL PROPERTY (IP) LICENSE, AND/OR UNDER APPLICABLE TERMS OF USE (TOU)? / C.4.2 WILL THE DATASET BE UPDATED (E.G., TO CORRECT LABELING ERRORS, ADD NEW INSTANCES, DELETE INSTANCES)?: The dataset has been released under the Apache-2.0 license and will be corrected or expanded on Hugging Face, with corresponding leaderboard updates posted on the website.The maintenance passage specifies adding tasks or correcting errors as update triggers.

D DATA CONTAMINATION

The authors test BigCodeBench for contamination against several potentially correlated code and instruction datasets, finding very limited overlap. They also describe release practices that reduce scraping-based contamination while acknowledging that complete privacy cannot be guaranteed.

  • Contamination evaluation: The authors conduct N-gram contamination experiments using ODEX intents, an anonymized Stack Overflow archive, and StarCoderData.These sources may be correlated with BigCodeBench instructions or used as code-training data for LLMs.
  • Contamination evaluation: No more than 2.5% of BigCodeBench tasks overlapped with tested data sources under the stricter 10-gram configuration.The authors characterize the overall likelihood of task-description contamination as extremely low.
  • Mitigation: The authors release BigCodeBench on Hugging Face rather than GitHub to reduce contamination from automated scraping of source code.They cite GitHub’s role in past contamination incidents involving HumanEval and MBPP, whereas Hugging Face lacks the same kind of automated scraping.
  • Limitations: Complete benchmark-data privacy is impossible because closed-source model APIs may collect and use high-quality inference data for training.Preventing this would require access to model weights and local execution, which is generally infeasible.

E EXTENDED RELATED WORK · G LONG-TERM ROADMAP AND CALL FOR CONTRIBUTIONS

BigCodeBench extends programming benchmarks toward real-world tasks involving open-endedness, multi-tool use, and instruction-following, while BigCodeBench-Hard provides a budget-conscious, user-centric proxy built from Stack Overflow queries. The paper also presents a long-term roadmap for developing open, reliable, and scalable programming evaluations with the community.

  • E EXTENDED RELATED WORK: Existing programming benchmarks mainly emphasize short, self-contained, algorithm-specific tasks, whereas BigCodeBench targets broader software tasks driven by real-world requirements.The benchmark uses collaborative LLM-human annotation and draws on requirements similar to StackOverflow queries.
  • E EXTENDED RELATED WORK: BigCodeBench emphasizes open-endedness, multi-tool use, and instruction-following as underexplored aspects of programming evaluation.These aspects complement its focus on broader software tasks and compositional reasoning.
  • E EXTENDED RELATED WORK: BigCodeBench-Hard is released as a minimal high-quality subset that serves as a proxy for the full benchmark when evaluating both BigCodeBench-Complete and BigCodeBench-Instruct is too burdensome.Its purpose is to reduce evaluation budgets for common users.
  • E EXTENDED RELATED WORK: 10.4 million questions and answers in the anonymized Stack Overflow archive provide a diverse source of user queries for constructing BigCodeBench-Hard.The archive covers diverse programming languages and topics.
  • E EXTENDED RELATED WORK: 0.7 is the similarity threshold used to select aligned tasks, yielding 6,895 queries and 626 BigCodeBench tasks after deduplication.The selection uses all-mpnet-base-v2 embeddings and ranks tasks by the dot product between normalized embeddings.
  • E EXTENDED RELATED WORK: Tasks with a solve rate below 50% are defined as hard tasks for further evaluation filtering.Solve rate is the number of models that solve a task divided by the total number of models.
  • E EXTENDED RELATED WORK: BigCodeBench-Hard is recommended for budget-limited, user-centric evaluation and can be periodically updated to remain challenging and aligned with user queries and evaluated models.The subset is dynamic by design.
  • G LONG-TERM ROADMAP AND CALL FOR CONTRIBUTIONS: The long-term roadmap aims to address BigCodeBench’s limitations and sustainably build open, reliable, and scalable programming evaluations with the community.The authors connect this goal to understanding LLM programming capabilities and program-aided language models for task completion and reasoning.

G.1 LIMITATIONS … I TOOL STATISTICS

The paper identifies limitations in language coverage, benchmark saturation, evaluation reliability, rigor, generalization, API evolution, interaction, and library diversity, while outlining extensions for OOD, interactive, and evolving settings.

  • G.1 LIMITATIONS: BigCodeBench is Python-only, limiting straightforward extension to other programming languages despite Python’s flexibility and popularity for task automation.Function calls are mostly language-specific, making equivalent packages or libraries difficult to identify outside Python.
  • G.1 LIMITATIONS: The benchmark faces saturation and evaluation limitations because some models perform reasonably well, test cases can be flaky, and high coverage does not guarantee correct assessment.Flakiness can arise from unstable connections, while existing test cases may still miss behaviors in generated code.
  • G.1 LIMITATIONS: BigCodeBench’s coverage is limited to common libraries and daily programming tasks, raising questions about generalization to emerging tools and the diversity of covered Python libraries.The paper notes that extending the benchmark with new high-quality tasks requires substantial effort and that annotation expertise was limited.
  • G.1 LIMITATIONS: Evolving libraries challenge tool-dependent benchmarks through obsolescence, version changes, and possible training-data contamination.Models may not memorize function calls from deprecated library versions, motivating periodic benchmark updates.
  • G.3 BIGCODEBENCH-INTERACT: The paper proposes adapting BigCodeBench to an interactive sandbox where models can program and repair iteratively using applications such as browsers and terminals.This agent-centric setting is intended to examine debugging and repair abilities in a less constrained environment.
  • G.2 BIGCODEBENCH-OOD: The paper plans BigCodeBench-OOD to evaluate generalization on completely synthetic features and real but long-tail libraries and programming tasks.The synthetic setting is intended to represent private libraries unknown to models.
  • G.4 BIGCODEBENCH-EVOLVED: For API evolution, the paper considers LLM-based API updates while noting that Python libraries exhibit many distinct evolution types.A study of 288 releases across six popular Python libraries identified 14 API-evolution types.
  • H ARTIFACTS: Table 4 lists artifacts for reproducibility.The supplied passage provides only the table caption and no further artifact details.

I.1 ANALYSIS … I.4 DOMAIN CLASSIFICATION

BigCodeBench exhibits long-tail diversity in both its libraries and function calls, while documenting comparisons with existing Python programming benchmarks, pinned library versions, and mappings from libraries to seven domains.

  • I.1 ANALYSIS: Library frequencies follow a long-tail distribution, highlighting broad diversity within BigCodeBench.Figure 12 sorts libraries by frequency count to show this distribution.
  • I.1 ANALYSIS: Function-call frequencies also follow a long-tail distribution, highlighting broad diversity within BigCodeBench.Figure 13 sorts function calls by frequency count to show this distribution.
  • I.2 COMPARISON TO EXISTING PROGRAMMING BENCHMARKS: BigCodeBench is compared with existing Python programming benchmarks using solution-character depth and function-call breadth.Table 5 defines depth as solution characters and breadth as function calls.
  • I.3 VERSION CONTROL: The benchmark specifies versions for libraries including pandas 2.0.3, scikit-learn 1.3.1, requests 2.31.0, and matplotlib 3.7.0.The implementation passage lists pinned versions for the benchmark’s library environment.
  • I.4 DOMAIN CLASSIFICATION: Domain classification maps libraries such as cryptography to Cryptography, PIL to Visualization, array to General, and bs4 to Network.These mappings illustrate the domain-labeling scheme.
  • I.4 DOMAIN CLASSIFICATION: The classification covers domains including Time, Network, Computation, General, System, and Visualization across libraries such as datetime, flask, cmath, collections, csv, and folium.The mappings include additional libraries such as urllib, uuid, yaml, zipfile, Levenshtein, cv2, and gensim.

J DETAILED BENCHMARK CONSTRUCTION … L.1 DETAILED RESULTS

The benchmark construction process synthesizes practical multi-library scenarios, refines functions and docstrings, and creates black-box tests emphasizing clear behavior, usable returns, and broad coverage. Human curation further checks library usage, implementation fidelity, deterministic testing, corner cases, and error handling.

  • J.1 DATA SYNTHESIS PROMPT: Scenario synthesis requires complex, practical Python functions that invoke more than three libraries and include requirements, examples, control flow, and return values.Imports, variables, constants, and sample data are placed outside or alongside the function as specified.
  • J.1 DATA SYNTHESIS PROMPT: Generated scenarios are separated by ‘GPT_ODEX_BREAK‘ and can combine libraries such as pandas, pytz, datetime, random, matplotlib, numpy, and dateutil.Examples include weather reporting across time zones and elapsed-seconds calculations from parsed dates.
  • J.2.1 PROGRAMMING TASK CLASSIFICATION PROMPT: Programming tasks are classified with labels covering formats, data structures, analysis, networking, processing, visualization, storage, encryption, and built-in functionality.The classifier outputs suitable labels in list format.
  • J.2.2 GUIDELINES: Function refactoring aligns imports, implementations, and docstrings while preserving functionality and making descriptions clear, concise, realistic, and testable.The guidelines require removing unused imports, importing libraries before the function, and ensuring library use is reasonable.
  • J.2.2 GUIDELINES: Refactored functions must expose outputs needed for testing, including returned properties, visualization objects, printed results, and specified value formats.The example changes a function from printing list properties to returning them for reuse elsewhere.
  • J.2.2 GUIDELINES: Black-box testing uses a run_tests() program with assertion-based cases, unique inputs, at least five different input cases, and checks of returned values and data attributes.Tests may generate or mock complex data with Faker or unittest.mock and should validate attributes such as plot titles, axes, and data points.
  • J.3 HUMAN CURATION GUIDELINES: Human curation standardizes anonymized function names, docstrings, imports, library-API usage, returned outputs, deterministic randomness, corner cases, and input-error handling.The curation process also requires tests to cover branches, use deterministic expected outputs, validate returned-object attributes and generated files, and avoid duplicated cases.

L.2 FURTHER ANALYSIS · L.3 PROMPTING TECHNIQUES

BigCodeBench performance generally improves with model scale, but capability varies by model openness, domain, and specialization. Zero-shot chain-of-thought prompting provides no significant advantage, underscoring the difficulty of reliable compositional reasoning.

  • L.2 FURTHER ANALYSIS: Larger models generally solve more BigCodeBench programming tasks, although Mistral-Large underperforms Mistral-Small on both benchmark variants.The scaling pattern holds for most instruction-tuned and base LLMs, while the Mistral result suggests under-fitting.
  • L.2 FURTHER ANALYSIS: Most strong BigCodeBench models are closed LLMs, led by OpenAI and Anthropic models.The strongest permissive models are relatively small and remain behind non-permissive models.
  • L.2 FURTHER ANALYSIS: Llama3-instruct-70B slightly outperforms Claude-3-Sonnet and ranks 5th on both BigCodeBench-Complete and BigCodeBench-Instruct.It is the best-performing open LLM in the reported comparison.
  • L.2 FURTHER ANALYSIS: Top base LLMs such as Dpsk-Coder-Base-6.7B excel across most domains but still fall short in some.The paper suggests domain-specific specialty may result from training data.
  • L.2 FURTHER ANALYSIS: Models are strongest on computation, cryptography, and general-domain tools, but frequently fail on tasks involving network tools.The authors encourage developing models specialized for these low-performing domains.
  • L.3 PROMPTING TECHNIQUES: Zero-shot chain-of-thought prompting shows no significant advantage for BigCodeBench tasks on GPT-4o and Gemini-1.5-Pro.The study appended “Let’s think step by step.” to the original prompt and found the technique insufficiently reliable for complex compositional reasoning.

M QUALITATIVE STUDIES · N UNIT TEST DESIGN

Qualitative studies show that models fail on subtle API semantics, edge cases, and function selection across diverse programming tasks. BigCodeBench unit tests therefore mock environmental scenarios and require substantially more human-designed coverage than simple input-output assertions.

  • M QUALITATIVE STUDIES: GPT-4o produces an infinite loop by using itertools.cycle incorrectly.The failure appears in a task that cycles and shuffles a list before counting frequencies.
  • M QUALITATIVE STUDIES: Models misapply Python semantics, including isinstance checks, os.path.dirname handling, and itertools.zip_longest argument requirements.These errors respectively concern type checking, nonexistent directories, and passing single values instead of iterables.
  • M QUALITATIVE STUDIES: Models also mishandle external-library behavior, including wikipedia exception types, URL regex extraction, and ipinfo.io hostname construction.The documented failures involve returning DisambiguationError rather than PageError, extracting a trailing comma, and incorrectly appending /json.
  • M QUALITATIVE STUDIES: Models choose incorrect functions when simpler or required alternatives exist, using a non-flattening approach and np.mean instead of np.nanmean for incomplete data.The latter causes a test failure because missing values require nan-aware averaging.
  • N UNIT TEST DESIGN: HumanEval tests mainly use input-output assertions, which are suitable only for simple programs without configuration and environment setup.This limitation motivates the broader test design used in BigCodeBench.
  • N UNIT TEST DESIGN: BigCodeBench unit tests mock network scenarios and validate program behavior across varied settings.The example uses unittest.mock.patch with HTTPS connections, alongside networking and SSL modules.
  • N UNIT TEST DESIGN: Compared with HumanEval and APPS, BigCodeBench unit tests require greater human effort to design and cover diverse settings.The tests explicitly assess behavior under mocked connection conditions rather than only checking fixed input-output pairs.

O COMPARISON TO EXISTING PROGRAMMING BENCHMARKS … R DEVELOPMENT TIMELINE

The paper compares BigCodeBench with established programming benchmarks, examines robustness to less-structured instructions, documents its evaluation infrastructure, and records development from initiation through release. These sections emphasize correlation analysis, ambiguity-related performance degradation, EvalPlus-based tooling, and the benchmark’s staged construction.

  • O COMPARISON TO EXISTING PROGRAMMING BENCHMARKS: BigCodeBench-Complete and BigCodeBench-Instruct model ranks are compared with HumanEval+, LiveCodeBench, MBPP+, and NaturalCodeBench using Pearson and Spearman correlations.The comparison is summarized in Table 9, which reports correlation coefficients against existing benchmarks.
  • P EVALUATION ON LESS-STRUCTURED INSTRUCTIONS: BigCodeBench-Instruct’s rule-based prompts may not fully reflect real-world requests, which can be less structured and more ambiguous.The passage illustrates this by contrasting a detailed programming description with a casual user-style request that omits the title name and return type.
  • P EVALUATION ON LESS-STRUCTURED INSTRUCTIONS: The less-structured evaluation rephrases a task into a casual request for a random forest classifier and a feature-importance bar chart.The requested output includes the trained classifier and chart, with features ordered by importance and labeled axes and title.
  • P EVALUATION ON LESS-STRUCTURED INSTRUCTIONS: Qwen2.5-Coder-32B-Instruct shows a more significant performance drop on rephrased instructions than other models despite stronger original BigCodeBench-Hard performance.The results suggest that rephrasing can introduce ambiguity and expose limited robustness to casual inputs.
  • Q B I G C O D E B E N C H: EVALUATION INFRASTRUCTURE: The bigcodebench evaluation infrastructure prototype is based on EvalPlus.The documented command-line interface supports complete or instruct splits, full or hard subsets, and multiple model backends.
  • R DEVELOPMENT TIMELINE: The project progressed from initiation in 04/2023–05/2023 through data synthesis, refactoring and test generation, human curation, and tool development.The construction and tooling phases continued through 04/2024.
  • R DEVELOPMENT TIMELINE: 04/2024–05/2024 covered benchmark finalization, experiments, analysis, drafting, and Code-Eval development, followed by the initial release in 06/2024.These entries mark the final preparation and public release stages.
Loading 2406.15877v4…