NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Prompts

1 Introduction

• Unlike introductory algorithm, programming-basics, and data-science benchmarks, real user engineering queries can involve more complex and varied requirements.The passage contrasts a basic HumanEval function task with a real-query example involving XML parsing.Source: §1 · p. 2
• NCB contains 402 curated Python and Java problems across six domains, including complex inputs such as files.The problems were selected from real application-related user queries.Source: §1 · p. 2
• The semi-automated annotation pipeline has GPT-4 generate reference solutions and test cases, which humans then correct.Annotators correct errors rather than constructing the framework entirely manually.Source: §1 · p. 2, Figure 2 · p. 3
• More than 4× faster construction was achieved with the pipeline than with manual construction.The comparison involved programming experts working with or without the pipeline.Source: §1 · p. 3
• Models with similar HumanEval scores can differ substantially on NCB, and GPT-4 achieves only about a 53% pass rate.The authors interpret this pattern as possible inadequate focus on practical coding or over-specific optimization for HumanEval-style problems.Source: §1 · p. 3
• NCB's development set is public, while its test set is kept closed to avoid contamination.The development set contains 140 problems and the test set 262.Source: §1 · p. 3

2 Benchmark Construction

• The benchmark construction pipeline collects and filters problems, builds an evaluation framework, aligns model prompts, and translates problems into bilingual versions.Source: §2 · p. 3

2.1 Problem Selection

• The authors select 201 Python and 201 Java problems after GPT-3.5 pre-screening and manual review; over 80% of initial queries fail the stated testability criteria.Queries must pose a task with definitive input-output pairs and no randomness or uncertainty.Source: §2.1 · p. 4

2.2 Semi-automated Pipeline

• GPT-4 generates reference solutions and test cases from problem descriptions, and experts correct errors and supplement test cases.The prompts request ten test cases and use the reference solution to show names and formats; test generation targets both code coverage and corner values.Source: §2.2 · pp. 4–5

2.3 Alignment Between Different Models

• Because natural-language problems lack predetermined function or class names, prompts provide a representative test case specifying the required naming convention.The provided test case is not used in subsequent testing.Source: §2.3 · p. 5

2.4 Building Bilingual Benchmark

• The collected problems are translated into English and Chinese versions because most source queries are in Chinese.Source: §2.4 · p. 5

3 Dataset Statistics

• NCB contains 402 problems across six domains and nine input-data categories, including tensors, data frames, images, and specialized files.The dataset has 201 Python and 201 Java problems; the categories distinguish common basic types from more complex structured and file inputs.Source: §3 · p. 5, Table 2 · p. 6
• Each problem averages 9.3 test cases, with about 60% targeting statement and branch coverage and 40% targeting corner values.Problems average 78.3 words.Source: §3 · p. 6
• NCB's broader domain and data-type coverage supplements benchmarks concentrated on algorithmic and basic programming problems.HumanEval and MBPP comprise 96.9% and 89.5% algorithmic and basic programming problems, respectively; NCB also targets multiple file formats, though those problems have fewer test cases.Source: §3 · p. 6
• High problem-collection and evaluation-framework construction costs constrain NCB's size.Source: §3 · p. 6

4.1 Setup

• The setup evaluates 33 popular models with pass@k and compares five experts’ manual and pipeline construction on the same five problems, requiring at least 80% code coverage.The construction score is LineCov. + BranchCov.Source: §4.1 · pp. 6, 8

4.2 Results of LLMs

• GPT-4 reaches 52.8% pass@1 on NCB, while DeepSeek-Coder-33B-Instruct leads open-source models at 43.0%, 9.8 points below GPT-4.Most open-source models still do not match GPT-3.5’s 40.7%; GPT-4 approaches 90% on HumanEval.Source: Table 3 · p. 7, §4.2 · p. 8
• GPT-4 and GPT-4-Turbo score 1.1% and 3.9% higher in English than Chinese, while open-source pass@1 ranges vary across model-size bands.The reported ranges are 0.0%–23.9% below 10B, 3.9%–35.1% from 10B–30B, 21.8%–43.0% from 30B–60B, and 27.9%–33.2% above 60B; the authors also note smaller-model exceptions.Source: §4.2 · p. 8
• Under random sampling, Codellama-70B-Instruct outperforms GPT-3.5 on both pass@10 and pass@50, unlike its pass@1 result.The comparison reports the best-performing models in each LLM family for k ∈ {10, 50}.Source: §4.2 · p. 8

4.3 Performance mismatch on HumanEval and NCB

• Most models’ HumanEval and NCB scores grow linearly together, but Phi, Deepseek-Chat, WizardCoder, CodeGen, and Llama-3-Instruct tend to rank better on HumanEval.The tendency is stronger for the first three model series than for CodeGen and Llama-3-Instruct.Source: §4.2 · p. 8, §4.3 · p. 9
• The authors suggest NCB’s harder natural prompts may demand stronger generalization and instruction-following, and that HumanEval-style optimization or contamination may contribute to the mismatch.These are proposed explanations; the paper notes preliminary evidence that base models do not properly solve NCB through in-context learning alone.Source: §4.3 · p. 9

4.4 Results of Semi-automated Construction

• 40 minutes with the semi-automated pipeline yields almost identical test coverage to 175.9 minutes of manual construction, with a reported average score difference of 37.6.The authors conclude that construction is more than four times faster without substantial quality loss.Source: Table 4 · p. 8, §4.4 · p. 9

• Code-specific LLMs address generation, repair, testing, translation, and summarization; prominent generation models include Codex, CodeGen [44], Incoder [22], and PolyCoder [62].Source: §5 · p. 9
• Benchmarks include HumanEval’s 164 Python functions and MBPP’s 974 contributed programming challenges, alongside multilingual extensions such as HumanEval-X [70] and MultiPL-E.Source: §5 · pp. 9–10
• Other benchmark efforts evaluate competitive programming [25, 35] and code generation across languages, including CodeContests [35].Source: §5 · pp. 9–10
• More practical settings include AgentBench’s interactive Unix shell and MySQL tasks [38] and SWE-Bench’s GitHub issues, codebases, and tests [30].Source: §5 · p. 10

• NaturalCodeBench contains 402 problems selected from coding online services and supports automatic evaluation of LLM-generated code.Its semi-automated construction pipeline achieves an efficiency gain of more than 4x over manual construction.Source: §6 · p. 10

• Interface-creation and web-service problems are not tested, despite being common in real applications.Source: Limitations · p. 10
• The omitted problem types introduce evaluation bias that may affect accuracy for certain models.Source: Limitations · p. 10
• The pipeline still entails expensive OpenAI API access and does not eliminate human-resource use.Source: Limitations · p. 10

• LLM capabilities are used during benchmark construction to improve efficiency and reduce human labor.The instructions describe filtering low-quality queries, generating test cases, and aligning names.Source: A Instructions in NATURALCODEBENCH · p. 16
• Figure 3 presents instructions for quickly filtering out queries unsuitable for testing.Source: Figure 3 · p. 16, A Instructions in NATURALCODEBENCH · p. 16
• Figure 4 addresses mismatches between model-generated class or function names and the names used in test cases.Source: Figure 4 · p. 16, A Instructions in NATURALCODEBENCH · p. 16

B.1 Examples of Semi-Automated Pipeline

• In the Python example, GPT-4 drafts most of the test case, while experts correct erroneous expected outputs.The draft includes the execution procedure and test-case input.Source: §B.1 · p. 16
• For the Java example's complex file inputs, experts add missing procedures and create input files using GPT-4's commented reference content.Source: §B.1 · p. 16
• Table 5 reports pass@k results for the best-performing LLMs in each family, with k equal to 10 or 50.Source: Table 5 · p. 17

B.2 Example Problems

• The examples cover algorithm and data structure, software engineering, data science, front-end development, artificial intelligence, and system administration tasks.They respectively involve sequence transformations, Markdown tags, temperature CSVs, HTML replacements, tensor-point distances, and file renaming.Source: §B.2 · p. 17

• Development-set pass@1 results are essentially consistent with test-set results, although several model rankings change with domain distribution.Table 6 reports greedy-decoding results.Source: C Extra Results · p. 17, Table 6 · p. 18
• Pass@k evaluation covers each LLM family's best-performing models for k values of 10 and 50, but excludes five models because of API and resource limitations.Source: C Extra Results · p. 17
• Figures 5 and 6 provide Python and Java examples of the semi-automated pipeline.Source: Figure 5 · p. 19, Figure 6 · p. 19
• Figures 7–12 provide example problems across algorithm and data structure, software engineering, data science, front-end, artificial intelligence, and system administration.Source: Figure 7 · p. 20, Figure 8 · p. 20, Figure 9 · p. 21