PaperlayerView exact revision on arXiv

Source-linked AI summary

BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions

Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, Binyuan Hui, Niklas Muennighoff, David Lo, Daniel Fried, Xiaoning Du, Harm de Vries, Leandro Von Werra

arXiv:2406.15877v4cs.SEcs.AIcs.CL

TL;DR

Existing code-generation benchmarks largely emphasize short, self-contained programming tasks, leaving evaluation of diverse tool use and complex instructions limited. BigCodeBench evaluates these capabilities, and across 60 models even GPT-4o solves only 60% of BigCodeBench-Complete tasks.

  • Problem

    Existing programming benchmarks largely contain short, self-contained, algorithm-focused tasks, while real-world programming often requires diverse function calls and compositional reasoning over complex instructions.

    Source: §1 · p. 2

  • Method

    The authors build BigCodeBench through human–LLM collaboration, creating 1,140 Python tasks spanning 723 function calls from 139 libraries across seven domains.

    Source: §1 · p. 3

  • Results

    Across 60 models, LLMs still struggle with cross-domain function calls and complex instructions; GPT-4o solves 60% of BigCodeBench-Complete tasks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors conclude that models remain far from perfect on BigCodeBench and call for more advanced code LLMs and continued benchmark development.

    Source: §6 · p. 10

    GPT-4's Code Interpreter struggled with mocking tests and runtime-bug resolution, making continuous human feedback essential to keep it on track.

    Source: §2.2 · pp. 4–5

Abstract

from arXiv · show

Task automation has been greatly empowered by the recent advances in Large Language Models (LLMs) via Python code, where the tasks ranging from software engineering development to general-purpose reasoning. While current benchmarks have shown that LLMs can solve tasks using programs like human developers, the majority of their evaluations are limited to short and self-contained algorithmic tasks or standalone function calls. Solving challenging and practical tasks requires the capability of utilizing diverse function calls as tools to efficiently implement functionalities like data analysis and web development. In addition, using multiple tools to solve a task needs compositional reasoning by accurately understanding complex instructions. Fulfilling both of these characteristics can pose a great challenge for LLMs.To assess how well LLMs can solve challenging and practical tasks via programs, we introduce BigCodeBench, a benchmark that challenges LLMs to invoke multiple function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained tasks. To evaluate LLMs rigorously, each task encompasses 5.6 test cases with an average branch coverage of 99%. In addition, we propose a natural-language-oriented variant of BigCodeBench, BigCodeBench-Instruct, that automatically transforms the original docstrings into short instructions only with essential information. Our extensive evaluation of 60 LLMs shows that LLMs are not yet capable of following complex instructions to use function calls precisely, with scores up to 60%, significantly lower than the human performance of 97%. The results underscore the need for further advancements in this area.

1 INTRODUCTION

BigCodeBench targets the gap between saturated short coding exercises and practical programming, where tasks require diverse function calls and compositional instruction-following. Its evaluation finds substantial remaining model limitations.

  • Practical programming tasks require diverse library function-call sequences and compositional reasoning to perform operations in the correct order and format outputs.Source: §1 · p. 2
  • Benchmark construction is difficult because repository functions need cross-file context, existing benchmarks skew toward data science, and underspecified specifications can undermine evaluation.Source: §1 · p. 2
  • BigCodeBench contains 1,140 Python tasks covering 723 function calls from 139 libraries across 7 domains, with docstring-based Complete and natural-language-oriented Instruct variants.The benchmark uses open-ended unit tests to examine behavior across different inputs.Source: §1 · p. 3
  • GPT-4o solves 60% of BigCodeBench-Complete tasks and less than 50% on BigCodeBench-Instruct; tested LLMs struggle with multi-library calls and complex instructions.Source: §1 · p. 3
  • Some instruction-tuned LLMs such as GPT-4 refuse long instructions that repeat essential context; performance also varies across domain-specific calls, while mainstream benchmarks correlate positively with BigCodeBench.Source: §1 · p. 3

2 BENCHMARK CONSTRUCTION

BigCodeBench is built through a human-LLM collaboration that synthesizes, refactors, tests, and curates programming tasks, with an Instruct variant created from the completed benchmark. Tasks are designed to use multiple libraries and include structured docstrings and at least five unittest cases.

2 BENCHMARK CONSTRUCTION

  • The construction involved 20 authors over one year and targeted PEP-257 docstrings, multiple-library tasks, and at least five unittest cases per task.The process comprised data synthesis, semi-automatic program refactoring and test generation, and human curation.Source: §2 · p. 3

2.1 DATA SYNTHESIS

  • LLMs synthesized self-contained tasks from ODEX seed examples using GPT-4 and a two-shot demonstration; obfuscation and docstring perturbation yielded 4,718 samples.The process addressed potential self-generation bias by replacing semantic-rich entry points with dummy names and back-translating docstrings.Source: §2.1 · p. 4

2.2 SEMI-AUTOMATIC PROGRAM REFACTORING AND TESTING CASE GENERATION

  • Human annotators guided GPT-4 through program refactoring and test generation, but mocking-test failures and runtime-bug loops required continuous human feedback.Annotators used GPT-4-assigned task categories to guide refinements; the model’s process included aligning implementations with docstrings and executing unit tests.Source: §2.2 · pp. 4–5
  • 1,223 refactored programming tasks with paired test cases remained after invalid programs were removed during post-processing.Many test cases were incomplete because GPT-4 omitted content when writing long contexts into files.Source: §2.2 · p. 5

2.3 HUMAN CURATION

  • Human curation examined tasks and tests, used a GPT-3.5 dry run to clarify prompts, then cross-checked docstrings, examples, and sandbox test results.The examination required at least two libraries and deterministic, pytest-compatible tests aligned with task instructions.Source: §2.3 · p. 5
  • 97% (32 out of 33) sampled tasks passed all test cases in the final quality check.Seven additional annotators cross-checked tasks, and the lead annotator evaluated solutions written for 33 randomly assigned prompts.Source: §2.3 · p. 5

2.4 BENCHMARKING NL-ORIENTED INSTRUCTIONS TO CODE GENERATION

  • BigCodeBench-Instruct converts Complete prompts into more natural instructions that omit non-essential details, with five authors checking prompt-solution alignment.The variant targets users’ high-level descriptions of desired functionality rather than verbose docstrings with redundant or overly specific details.Source: §2.4 · pp. 5–6, Figure 3 · p. 6

3 BENCHMARK STATISTICS

BigCodeBench combines demanding execution-based evaluation and complex tasks with broad, popular tool coverage and multi-library function-call sequences that require compositional reasoning.

Overall Statistics

  • BigCodeBench has more rigorous execution-based evaluation, longer prompts and solutions, and cyclomatic complexity similar to HumanEval but higher than DS-1000 and ODEX.The reported complexity indicates that solving its tasks requires non-trivial programming reasoning.Source: §3 · pp. 6–7, Table 1 · p. 7

Tool Statistics

  • BigCodeBench covers 281 function calls from 77 standard libraries and 442 from 62 external libraries, whose mean recent 30-day PyPI downloads exceed 56.9M.The passage presents these libraries as popular in real-world software development.Source: §3 · p. 7
  • Tasks frequently require sequences of calls from multiple libraries, and ground-truth solutions combine libraries, function calls, and domains more diversely than other benchmarks.The benchmark therefore tests compositional reasoning and comprehensive problem-solving.Source: §3 · p. 7

4 EVALUATION

BigCodeBench evaluation finds that performance depends on prompt form and model capabilities, while tool use varies across domains and function calls. Across the evaluated models, task-level results reveal gaps in following long and concise instructions, and tool-level results show both broad library use and failure-prone variation.

4 EVALUATION

  • Evaluation used zero-shot greedy Pass@1 for 60 state-of-the-art LLMs on BigCodeBench-Complete and 35 instruction-tuned LLMs on BigCodeBench-Instruct.Appendix results include random-sampling Pass@5 with N=5, temperature 0.8, and top-p 0.95.Source: §4 · pp. 7–8

4.1 MEASURING TASK-LEVEL PERFORMANCE

  • 0.982 was the Pearson’s r correlation between model rankings on BigCodeBench-Complete and BigCodeBench-Instruct; rankings also showed scaling-law signs, with a gap between top closed and open models.The main results report Pass@1, with calibrated results shown when they differ from original Pass@1 by at least 1%.Source: Figure 6 · p. 8, §4.1 · p. 8
  • Instruction-tuned models omitted essential setup more on long prompts: their average performance degradation was 0.8% on BigCodeBench-Complete versus less than 0.3% on BigCodeBench-Instruct.Calibration adds missing setup such as imports and global constants; GPT-4 tended to omit more context on BigCodeBench-Complete.Source: Figure 6 · p. 8, §4.1 · p. 9
  • Mean calibrated Pass@1 on BigCodeBench-Complete was 40.7% for instruction-tuned models and 35.7% for paired base models; CodeQwen1.5-7B showed no improvement from instruction tuning.The authors suggest the latter may reflect a lack of fine-grained training data.Source: §4.1 · p. 9
  • Pass@1 decreased by an average of 8.5% on BigCodeBench-Instruct versus BigCodeBench-Complete, although model rankings remained similar.The authors attribute the result to models’ difficulty understanding condensed requirements; lower-scoring models degraded less, limited by programming capability.Source: Figure 6 · p. 8, §4.1 · p. 9

4.2 MEASURING TOOL-LEVEL PERFORMANCE

  • Top instruction-tuned models such as GPT-4o excelled in most domains but fell short in some, with domain specialization suggested as a possible factor.Tool-use correctness was assessed using calibrated greedy results on BigCodeBench-Complete, counting an associated library as correct only when all tests passed.Source: §4.2 · p. 9, Figure 7 · p. 9
  • Across 60 calibrated models, imported libraries were used in more than 70% of tasks; for the remaining 20%, models tended to add libraries, mostly standard ones, and differing function calls were more failure-prone.Function-call variation is expected for open-ended tasks, but the authors found more diverse calls more likely to introduce failures.Source: §4.2 · p. 10, Table 2 · p. 10

5 RELATED WORK

Related work traces the development of code-focused LLMs and distinguishes BigCodeBench from benchmarks that target short algorithmic tasks, limited tool use, or code-agent frameworks.

Large Language Models for Code

  • Code LLMs progressed from Codex (Chen et al., 2021) and base models to instruction-tuned models, which generally outperform base models at programming (Muennighoff et al., 2023; Luo et al., 2023; Wei et al., 2023).Pre-trained base models were developed for accurate code completion, while instruction tuning enabled code generation aligned with natural-language instructions.Source: §5 · p. 10

Programming Benchmarks

  • Most benchmarks target short algorithmic tasks or simple intents with limited function calls, whereas BigCodeBench evaluates fundamental code generation across a broader range of software tasks.Unlike SWE-bench, which evaluates code-agent frameworks and uses repository test cases, BigCodeBench creates tasks through collaborative LLM-human annotation from seed queries (Wang et al., 2023c).Source: §5 · p. 10

6 CONCLUSION · APPENDIX

BigCodeBench is a human-expert-and-LLM collaboration that evaluates tool use and complex instruction following; evaluation of 60 LLMs found substantial room for improvement. The authors call for more advanced code LLMs and continued development of the benchmark and BigCodeBench-Hard.

6 CONCLUSION

  • The authors introduce BigCodeBench, a programming benchmark constructed through collaboration between human experts and LLMs.It assesses tool use and complex instruction following.Source: §6 · p. 10
  • BigCodeBench assesses models’ capability in tool use and complex instruction following.Source: §6 · p. 10
  • Evaluation of 60 LLMs found that models remain far from perfection on BigCodeBench.Source: §6 · p. 10
  • The authors share findings that could potentially improve model performance on the benchmark.Source: §6 · p. 10
  • The authors urge the community to develop more advanced LLMs for code.Source: §6 · p. 10
  • The authors call for continued development of BigCodeBench and its extended BigCodeBench-Hard benchmark.They discuss this work in their long-term roadmap.Source: §6 · p. 10

B DATACARD … C.4.3 IF OTHERS WANT TO EXTEND/AUGMENT/BUILD ON/CONTRIBUTE TO THE DATASET, IS THERE A MECHANISM FOR THEM TO DO SO?

The accompanying documentation describes BigCodeBench’s purpose, dataset documentation and distribution, and its planned maintenance. It also notes a language-related prompt limitation; the supplied passages do not specify a mechanism for outside contributions.

B DATACARD

  • Some annotators’ first language is not English, which may introduce inaccurate task-prompt expressions; communications were in mainstream English (en-US).The datacard follows Bender & Friedman (2018) and centralizes information relevant to benchmark analysis.Source: B DATACARD · p. 20

C DATA SHEET

  • The data sheet follows documentation frameworks provided by Gebru et al. (2021).Source: C DATA SHEET · p. 21

C.1.1 FOR WHAT PURPOSE WAS THE DATASET CREATED?

  • BigCodeBench is presented as the first easy-to-use benchmark combining practical, challenging programming tasks with an end-to-end framework for open-ended evaluation.Its focus is diverse function calling and complex instruction following.Source: §C.1.1 · p. 21

C.2 COMPOSITION/COLLECTION PROCESS/PREPROCESSING/CLEANING/LABELING AND USE

  • Answers about dataset composition, collection, preprocessing, cleaning, labeling, and use are described in the paper and GitHub repository.Source: §C.2 · p. 21

C.3.1 WILL THE DATASET BE DISTRIBUTED TO THIRD PARTIES OUTSIDE OF THE ENTITY (E.G., COMPANY, INSTITUTION, ORGANIZATION) ON BEHALF OF WHICH THE DATASET WAS CREATED?

  • No; the dataset is managed and maintained by the BigCode community.Source: §C.3.1 · p. 21

C.3.3 / C.3.4

C.4.2 WILL THE DATASET BE UPDATED (E.G., TO CORRECT LABELING ERRORS, ADD NEW INSTANCES, DELETE INSTANCES)?

  • If tasks are added or errors found, the dataset will be corrected on Hugging Face and leaderboard results updated; updates will appear on the website.Source: §C.4.2 · p. 22

D DATA CONTAMINATION · E EXTENDED RELATED WORK

Contamination checks found little overlap, though complete privacy cannot be guaranteed. The related-work review positions BigCodeBench as testing practical code generation through complex instructions and multiple tools, rather than customized agent systems.

D DATA CONTAMINATION

  • Contamination checks compared BigCodeBench instructions with ODEX intents, StarCoder2’s anonymized Stack Overflow archive, and StarCoderData using 10-gram and 13-gram setups.The 10-gram StarCoderData comparison timed out and was omitted because of computational resource requirements.Source: D DATA CONTAMINATION · p. 22
  • At most 2.5% of BigCodeBench tasks overlapped with tested data sources under the stricter 10-gram configuration.The authors characterize the likelihood of task-description contamination as extremely low.Source: Table 3 · p. 22, D DATA CONTAMINATION · p. 22
  • Releasing BigCodeBench on Hugging Face rather than GitHub is intended to reduce contamination from automated source-code scraping, but API providers may still train on submitted data.The authors say complete privacy is impossible to ensure; preventing API-provider use would require local access to model weights, which is usually infeasible.Source: D DATA CONTAMINATION · pp. 22–23

E EXTENDED RELATED WORK

  • Code LLMs progressed from encoder-only understanding and T5-based generation to decoder-only infilling, instruction-tuned models, and agentic programming systems.BigCodeBench assesses fundamental practical code-generation capability without customized prompting systems, whose backbone models are not its focus.Source: E EXTENDED RELATED WORK · p. 23
  • Existing programming benchmarks often target short, self-contained algorithms, while tool-use benchmarks typically use simple intents and few function calls per task.The passage contrasts these limits with BigCodeBench’s focus on more complex, realistic coding tasks.Source: E EXTENDED RELATED WORK · p. 23
  • Multilingual benchmarks often extend Python tasks through translation, while language-specific benchmarks remain limited in task quantity, complexity, or diversity.The passage names HumanEval-X, MultiPL-E, and CodeScope as translation-based examples and ClassEval and OOP as language-specific examples.Source: E EXTENDED RELATED WORK · p. 23
  • BigCodeBench uses collaborative LLM-human annotation to cover broader real-world software requirements and emphasizes open-endedness, multi-tool use, and instruction-following.Unlike code-agent benchmarks built around iterative prompting, environment interaction, and long-context exploration, it evaluates fundamental code-generation capability; the passage also notes SWE-bench’s reliance on GitHub repositories and existing tests.Source: E EXTENDED RELATED WORK · pp. 23–24

F BI GCO D EBE N C H-HA R D · G LONG-TERM ROADMAP AND CALL FOR CONTRIBUTIONS

BigCodeBench-Hard is a user-centric, budget-saving proxy for the full benchmark, and the authors call for community collaboration to build open, reliable, and scalable evaluations of LLM programming capabilities.

F BI GCO D EBE N C H-HA R D

  • BigCodeBench-Hard is a minimal high-quality proxy intended to reduce the evaluation burden when budgets cannot cover both full benchmark splits.The authors note the cost of running BigCodeBench-Complete and BigCodeBench-Instruct, particularly for large models.Source: F BI GCO D EBE N C H-HA R D · p. 24
  • From 10.4 million Stack Overflow questions and answers, embedding retrieval and a 0.7 similarity threshold yielded 6,895 queries and 626 deduplicated tasks.The authors used all-mpnet-base-v2 to rank query-task matches by normalized-embedding dot product and selected the threshold after manual inspection.Source: F BI GCO D EBE N C H-HA R D · p. 24
  • A user-query example and a benchmark prompt both target extracting hyperlinks from webpages and handling the data through CSV files.The implementations differ, but the passages describe their core objective and scope as closely aligned.Source: F BI GCO D EBE N C H-HA R D · p. 24
  • The final filters retain tasks requiring more than two libraries and over 426 solution tokens, with solve rates below 50%.The 426-token threshold is the average solution length in BigCodeBench; solve rate is the share of evaluated models that solve a task.Source: F BI GCO D EBE N C H-HA R D · pp. 24, 26
  • BigCodeBench-Hard rankings align with SEAL-Coding, while performance differs significantly from the full BigCodeBench set.SEAL-Coding ranks GPT-4-Turbo Preview, Claude 3.5 Sonnet, GPT-4o, and Gemini 1.5 Pro in that order; the authors attribute the difference from the full set to domain imbalance and many easy tasks.Source: F BI GCO D EBE N C H-HA R D · p. 26
  • The authors recommend BigCodeBench-Hard for budget-limited, user-centric evaluation and propose periodic updates based on user queries and evaluated models.Its dynamic design is intended to keep the subset challenging and user-centric.Source: F BI GCO D EBE N C H-HA R D · p. 26

G LONG-TERM ROADMAP AND CALL FOR CONTRIBUTIONS

  • The roadmap aims to build open, reliable, and scalable evaluations of LLM programming capabilities with the community, informed by program-aided language models for task completion and reasoning (Gao et al., 2023).The stated goal is to understand fundamental programming capabilities and identify ways to unleash LLMs’ potential.Source: G LONG-TERM ROADMAP AND CALL FOR… · p. 27

G.1 LIMITATIONS … I TOOL STATISTICS

The authors outline limitations in benchmark language coverage, test reliability and rigor, generalization, and API evolution, alongside plans for OOD, interactive, and evolution-aware benchmark variants. The supplied passages also identify a reproducibility-artifacts table.

G.1 LIMITATIONS

  • BigCodeBench is Python-only, and limited expert annotation and immature emerging libraries constrain language and library diversity.The authors note that Python’s popularity may still meet many community needs, while domain-specific libraries can introduce annotation errors, ambiguities, and unstable APIs.Source: §G.1 · pp. 27–28
  • BigCodeBench favors less ambiguous, author-validated tasks over SWE-bench’s under-specified instructions, while planned generalization tests include synthetic and real long-tail features.Proposed routes include synthesizing tasks from library and function-call details, or replacing known calls with synthetic names; the authors note uncertainty about synthesis quality and reduced naturalness for synthetic names. They also plan public and private benchmark updates as libraries evolve, to address deprecated APIs and contamination.Source: §G.1 · pp. 27–28, §G.2 · p. 28
  • Some execution tests remain flaky, and high ground-truth coverage does not ensure generated code is correctly assessed.The authors aim to keep uncontrollable Pass@1 changes under 0.6% and propose LLM-based test generation, including sandboxed self-refinement of solutions that pass existing tests.Source: §G.1 · p. 27
  • The authors plan to test BigCodeBench with LLM agents in interactive sandboxes that support application use, iterative debugging, and repair.Proposed evaluation ranges from showing test-failure backtraces to requiring models to debug independently by generating tests and inspecting execution outcomes.Source: §G.1 · p. 28, §G.3 · p. 28

G.2 BIGCODEBENCH-OOD

  • BigCodeBench-OOD will assess generalization using completely synthetic features and real but long-tail features.These categories are intended to represent respectively unknown private libraries and emerging features not yet prominent in training data.Source: §G.2 · p. 28

G.3 BIGCODEBENCH-INTERACT

  • BigCodeBench-Interact will evaluate iterative programming and repair in a sandbox, with either visible test-failure backtraces or independent debugging.The proposed setting expects capable models to generate tests and inspect execution results when backtraces are hidden.Source: §G.3 · p. 28

G.4 BIGCODEBENCH-EVOLVED

  • BigCodeBench-Evolved proposes combining static API-version analysis with multi-turn LLM interaction to update solutions and tests for library changes.For API removals without replacements, models may need additional turns; updated solutions and tests would be validated through sandboxed execution.Source: §G.4 · p. 29

H ARTIFACTS

I.1 ANALYSIS … I.4 DOMAIN CLASSIFICATION

The appendix characterizes BigCodeBench’s library and function-call distributions, compares benchmark scope, records dependency versions, and maps libraries to functional domains.

I.1 ANALYSIS

  • Libraries and function calls both follow long-tail frequency distributions, highlighting BigCodeBench’s broad diversity.The library comparison sorts libraries by frequency count, and the function-call comparison does the same for function calls.Source: Figure 12 · p. 31, Figure 13 · p. 31

I.2 COMPARISON TO EXISTING PROGRAMMING BENCHMARKS

  • Table 5 compares existing Python programming benchmarks by solution-character depth and function-call breadth.The caption identifies solution characters as the complexity measure and function calls as the diversity measure, but supplies no comparative values.Source: Table 5 · p. 31

I.3 VERSION CONTROL

  • The version-control section lists package dependencies alongside their version numbers.Examples include pandas==2.0.3, requests==2.31.0, and matplotlib==3.7.0.Source: §I.3 · p. 32

I.4 DOMAIN CLASSIFICATION

  • The mapping also includes General, Time, and Cryptography categories for libraries such as collections, datetime, and hashlib.Other listed examples include blake3 under Cryptography, pytz under Time, and uuid under General.Source: §I.4 · pp. 32–34
  • The domain mapping assigns libraries to categories including Network, Computation, System, and Visualization.Examples include requests as Network, numpy as Computation, openpyxl as System, and matplotlib as Visualization.Source: §I.4 · pp. 33–34

J DETAILED BENCHMARK CONSTRUCTION … L.1 DETAILED RESULTS

The appendices provide detailed documentation of BigCodeBench’s construction and refinement, including task and test design, inference and execution setup, and backend-specific prompts and evaluation results. They also note evaluation-specific caveats in result-table captions.

J.1 DATA SYNTHESIS PROMPT

  • Scenario prompts require practical, functionally testable tasks using multiple libraries, complex logic, docstring requirements and examples, and return values suitable for assertions.They also direct authors to define imports, variables, constants, and sample data outside the function where applicable.Source: §J.1 · p. 34

J.2.1 PROGRAMMING TASK CLASSIFICATION PROMPT

  • The classification prompt asks models to select suitable labels from categories including SQL, CSV, DataFrames, visualization, networking, and file storage.The selected labels should be returned as a list.Source: §J.2.1 · p. 36

J.2.2 GUIDELINES

  • Refinement guidelines require clarifying docstrings while preserving functionality, checking library use, and specifying behavior, inputs, outputs, library requirements, and examples.They also call for making functions accept inputs when needed and returning results rather than only printing them, so outputs can be tested and reused.Source: §J.2.2 · pp. 36–37
  • Testing guidelines require assertion-based black-box tests with at least five distinct inputs, complex representative data, and checks of returned-value attributes.They also say not to remove libraries or modules used in the original function.Source: §J.2.2 · pp. 37–38
  • Curators were instructed to test all return values and attributes, clarify any properties tested but omitted from the description, remove ambiguity, and verify that function.py passes all black-box tests.Descriptions and implementations should be made clear and concise when programmers could reasonably have questions.Source: §J.2.2 · p. 38

J.3 HUMAN CURATION GUIDELINES

  • Human curation required realistic, unambiguous docstrings aligned with solutions, at least five description-aligned branch-covering tests, APIs from two libraries, and seeded randomness when applicable.The guidelines also called for broad branch coverage and tests aligned with the docstring description.Source: §J.3 · p. 39

K.1 INFERENCE

  • Inference ran on A100 GPUs except for closed models, which were accessed through their official APIs.Source: §K.1 · p. 41

K.2 EXECUTION

  • Execution mainly used an Intel Xeon Gold 6150 CPU at 2.70 GHz, with two sockets and 18 cores per socket.Source: §K.2 · p. 41

K.3 PROMPT TEMPLATE

L.1 DETAILED RESULTS

  • Table 6 reports BigCodeBench-Complete evaluations and strips trailing newlines during inference for StarCoder2 and Granite-Code because of training flaws.Source: Table 6 · p. 43
  • Table 7 evaluates instruction-tuned models on BigCodeBench-Instruct with greedy decoding and calibrated results, computes ∆(NL2C −C2C), and omits Granite-Code-Instruct 8B and 3B for empty outputs.The delta is included to examine performance differences between BigCodeBench-Complete and BigCodeBench-Instruct.Source: Table 7 · p. 44

L.2 FURTHER ANALYSIS · L.3 PROMPTING TECHNIQUES

BigCodeBench performance varies with model scale, openness, and domain, while zero-shot CoT offers no significant advantage in preliminary tests. The analysis also finds that models can excel across most domains yet remain weak in particular ones.

L.2 FURTHER ANALYSIS

  • Models with more parameters generally solve more programming tasks, although Mistral-Large underperforms Mistral-Small on both BigCodeBench variants.The authors suggest Mistral-Large may be under-fitting; they note consistency with Yan et al. (2024).Source: §L.2 · p. 45
  • Most strong BigCodeBench models are non-permissive, but Llama3-instruct-70B slightly outperforms Claude-3-Sonnet and ranks fifth on both benchmarks.The most permissive models are relatively small and remain behind non-permissive models.Source: §L.2 · p. 45
  • Dpsk-Coder-Base-6.7B excels in most domains but falls short in some, which the authors suggest may reflect domain-specific training data.The passage says Figure 7 shows the top five instruction-tuned models ranked on BigCodeBench-Complete.Source: §L.2 · p. 45
  • Models tend to perform better on computation, cryptography, and general-domain tasks, but frequently fail on tasks involving network tools.The authors encourage developing models that specialize in low-performing domains; examples of network failures appear in Appendix M, Examples 6–8.Source: §L.2 · p. 45

L.3 PROMPTING TECHNIQUES

  • Appending “Let’s think step by step.” produced no significant zero-shot CoT advantage for GPT-4o and Gemini-1.5-Pro on BigCodeBench tasks.The comparison covers calibrated Pass@1 on BigCodeBench and BigCodeBench-Hard.Source: §L.3 · p. 45, Table 8 · p. 45
  • The authors consider prompt design tricky and do not recommend it for official evaluation, while following EvalPlus’s approach for consistency.They conclude zero-shot CoT is not sufficiently reliable for complex compositional reasoning.Source: §L.3 · p. 45

M QUALITATIVE STUDIES · N UNIT TEST DESIGN

This section presents qualitative examples of model failures on practical tasks, including misunderstandings of API behavior, semantics, endpoint construction, and function choice. It also describes a unit-test design using mocked scenarios and detailed checks of outputs and visualizations.

M QUALITATIVE STUDIES

  • GPT-4o misuses itertools.cycle, producing an infinite loop, and incorrectly interprets isinstance(j, (int, float)) as checking j’s type against int or float.Source: M QUALITATIVE STUDIES · p. 46
  • Examples also show incorrect handling of a missing directory with os.path.dirname and passing single values, rather than iterables, to itertools.zip_longest.Source: M QUALITATIVE STUDIES · p. 47
  • A password-authentication example uses different encodings for the supplied and expected hashes, causing matching passwords to fail authentication.Source: M QUALITATIVE STUDIES · p. 48
  • A Wikipedia example mishandles missing verbatim titles: wikipedia.page raises DisambiguationError rather than the PageError the model handles.Source: M QUALITATIVE STUDIES · p. 49
  • A URL-extraction example has a regex that captures a trailing comma in www.google.com,, causing errors downstream.Source: M QUALITATIVE STUDIES · p. 49
  • A missing `/json` segment in the ipinfo.io request URL caused a hostname-not-found error in the geolocation example.Source: M QUALITATIVE STUDIES · p. 50
  • The model used a function other than `flatten` to produce the flat list, resulting in an incorrect solution.Source: M QUALITATIVE STUDIES · p. 50
  • Using `np.mean` instead of `np.nanmean` caused a test failure when the data contained missing values.Source: M QUALITATIVE STUDIES · p. 51

N UNIT TEST DESIGN

  • BigCodeBench tests mock network scenarios and check program behavior, requiring more human effort and broader setting coverage than input-output assertions.Source: N UNIT TEST DESIGN · p. 52
  • Visualization tests validate DataFrame columns and values, chart title, bar heights, and x-axis labels across varied inputs.The function returns both a DataFrame and a bar-chart Axes object; the tests use a helper to check bar properties.Source: N UNIT TEST DESIGN · pp. 52–53

O COMPARISON TO EXISTING PROGRAMMING BENCHMARKS … R DEVELOPMENT TIMELINE

BigCodeBench model rankings correlate strongly with existing coding benchmarks, while results on rephrased instructions point to robustness gaps for some models. The sections also describe the evaluation infrastructure and development timeline.

O COMPARISON TO EXISTING PROGRAMMING BENCHMARKS

  • BigCodeBench model ranks correlate strongly with existing benchmarks, with MBPP+ showing the highest correlation; NaturalCodeBench correlates slightly less with BigCodeBench-Complete.The authors note that NaturalCodeBench prompts are more similar to BigCodeBench-Instruct.Source: Table 9 · p. 54, O COMPARISON TO EXISTING PROGRAMMING… · p. 54
  • BigCodeBench evaluates different aspects from existing benchmarks, so models can rank differently, and GPT-4o still has room to improve relative to human performance despite saturating HumanEval.The authors characterize BigCodeBench as aligned with mainstream evaluation trends.Source: O COMPARISON TO EXISTING PROGRAMMING… · p. 54

P EVALUATION ON LESS-STRUCTURED INSTRUCTIONS

  • The less-structured-instruction study rephrases BigCodeBench-Hard task descriptions with Llama-3.1-8B-Instruct, using temperature 0.2 and top_k 0.95.The setup addresses prompts that may omit specifics such as a title or return type, unlike the rule-based BigCodeBench-Instruct structure.Source: P EVALUATION ON LESS-STRUCTURED… · p. 54
  • Qwen2.5-Coder-32B-Instruct shows a more significant performance drop on rephrased instructions than other models, despite stronger original BigCodeBench-Hard performance.The authors suggest that added ambiguity may expose weaknesses in handling less-structured, casual inputs.Source: Table 10 · p. 55, P EVALUATION ON LESS-STRUCTURED… · p. 55

Q B I G C O D E B E N C H: EVALUATION INFRASTRUCTURE

  • The bigcodebench evaluation infrastructure is based on EvalPlus and supports evaluation across complete or instruct splits, full or hard subsets, and multiple backends.The documented backend options include vLLM, OpenAI, Anthropic, Google, Mistral, and Hugging Face.Source: Q B I G C O D E B E N C H: EVALUATION… · p. 55

R DEVELOPMENT TIMELINE

  • Development ran from project initiation in April–May 2023 through benchmark construction, curation, finalization, experiments, analysis, drafting, and the initial release in June 2024.Construction included data synthesis, semi-automatic program refactoring, and test-case generation.Source: R DEVELOPMENT TIMELINE · p. 55

Related papers

Loading 2406.15877v4…