Source-linked AI summary
CodeApex: A Bilingual Programming Evaluation Benchmark for Large Language Models
Lingyue Fu, Huacan Chai, Shuang Luo, Kounianhua Du, Weiming Zhang, Longteng Fan, Jiayi Lei, Renting Rui, Jianghao Lin, Yuchen Fang, Yifan Liu, Jingkuan Wang, Siyuan Qi, Kangning Zhang, Weinan Zhang, Yong Yu
TL;DR
Existing programming benchmarks often focus on a single task or language and may lack fine-grained categorization and human evaluation. CodeApex evaluates LLMs across bilingual programming comprehension, code generation, and code correction; GPT models perform at the top tier, but human comparisons indicate room for improvement.
Problem
Existing programming benchmarks often focus on one task in English and generally lack fine-grained categorization and human-expert evaluation, limiting the breadth of assessment and comparison.
Source: §1 · p. 2Method
CodeApex is a bilingual benchmark that evaluates 12 LLMs on programming comprehension, C++ code generation, and code correction, with comparisons across tasks and prompt scenarios.
Source: §1 · p. 3, §5 · p. 21Results
GPT models achieve top-tier programming performance, while human tests indicate that LLMs remain short of human performance and specialized models are competitive.
Source: §5 · p. 22Takeaways & Limitations
CodeApex provides a reference for studying LLM programming capabilities, with future updates planned to expand question types and programming languages.
Source: §5 · p. 22
Abstract
from arXiv · showhide
With the emergence of Large Language Models (LLMs), there has been a significant improvement in the programming capabilities of models, attracting growing attention from researchers. Evaluating the programming capabilities of LLMs is crucial as it reflects the multifaceted abilities of LLMs, and it has numerous downstream applications. In this paper, we propose CodeApex, a bilingual benchmark dataset focusing on the programming comprehension, code generation, and code correction abilities of LLMs. Programming comprehension task tests LLMs on multiple-choice exam questions covering conceptual understanding, commonsense reasoning, and multi-hop reasoning. The code generation task evaluates LLMs through completing C++ functions based on provided descriptions and prototypes. The code correction task asks LLMs to fix real-world erroneous code segments with different error messages. We evaluate 12 widely used LLMs, including both general-purpose and specialized models. GPT-4 exhibits the best programming capabilities, achieving approximate accuracy of 69%, 54%, and 66% on the three tasks, respectively. Compared to human performance, there is still significant room for improvement in LLM programming. We hope that CodeApex can serve as a reference for evaluating the coding capabilities of LLMs, further promoting their development and growth.
1 INTRODUCTION
CodeApex is proposed as a comprehensive bilingual benchmark spanning three programming tasks. Its evaluations find varying model performance across tasks, with GPT models especially competitive and substantial room for improvement relative to human performance.
1 INTRODUCTION
- Code and programming tasks reflect logical reasoning, instruction following, and structural data understanding, and support applications such as education and software engineering.The introduction also connects code evaluation to deployed coding tools and programming assistants.Source: §1 · p. 2
- Existing programming benchmarks often cover only one task in English and generally lack fine-grained categories and human-expert evaluation.The authors identify these limits as reasons existing evaluations can be incomplete and offer less insight into distinct programming abilities.Source: §1 · pp. 2–3
- CodeApex evaluates 12 LLMs across bilingual programming comprehension, code generation, and code correction, with fine-grained categories and human-expert evaluation.The paper compares model performance across tasks and prompt strategies, and reports that GPT models are highly competitive while LLM accuracy remains short of human performance.Source: §1 · p. 3
2 RELATED WORK
Prior work includes specialized code models and benchmarks for programming evaluation. CodeApex positions its evaluation across programming comprehension and other code-related tasks.
2.1 Code Foundation Models
- Code foundation models address tasks including code comprehension, generation, completion, translation, debugging, and documentation.The cited approaches include graph and AST representations, shared natural-language/code representations, and models such as CodeT5 and CodeBERT.Source: §2.1 · p. 3
- Code foundation models use approaches including code-specific fine-tuning, instruction tuning, and prompt engineering to support code generation and related tasks.Examples include WizardCoder and Code Llama-Instruct for instruction tuning, and DocPrompting for retrieving repository information into prompts.Source: §2.1 · pp. 3–4
2.2 Programming Evaluation
- Programming evaluation benchmarks use multiple-choice questions to assess capabilities such as reading comprehension, science reasoning, and subject knowledge.Examples include RACE, ARC, and C-Eval; CodeApex uses multiple-choice questions for programming comprehension.Source: §2.2 · p. 4
- Code-related benchmarks cover tasks from code search and generation to synthesis and industrial coding challenges, while CodeApex focuses on reasoning through algorithmic challenges and test cases.The passage describes benchmarks including FlashFill++, CodeSearchNet, CodeXGLUE, APPS, HumanEval, HumanEval-X, CoderEval, and DebugBench.Source: §2.2 · p. 4
3 EVALUATION PROTOCOL
CodeApex evaluates programming comprehension, code generation, and code correction. Its test data is available in aligned Chinese and English versions.
3 EVALUATION PROTOCOL
- The benchmark framework covers programming comprehension, code generation, and code correction.Figure 1 presents the overall experiment scenarios.Source: §3 · p. 4, Figure 1 · p. 5
- Programming comprehension uses 250 university-exam multiple-choice questions across conceptual understanding, commonsense reasoning, and multi-hop reasoning, tested with 0-, 2-, and 5-shot answer-only and CoT prompts.Questions are drawn from final exams in programming, data structures, and algorithms.Source: §3 · p. 5
- Code generation uses 476 C++ questions requiring completion of a function from its description and prototype, with function-only and function-with-context scenarios.The context scenario supplies the calling code in addition to the objective function description.Source: §3 · p. 5
- Code correction uses 1,330 erroneous code segments with Wrong Answer, Time Limit Exceeded, or Runtime Error messages and code-only, error-message, or standard-code prompts.The prompt scenarios provide progressively different information for correcting the code.Source: §3 · p. 5
- All three tasks have aligned Chinese and English versions to support comparisons across languages.The protocol describes task overviews, data processing, prompting strategies, and evaluation metrics.Source: §3 · pp. 5–6
3.1 Programming Comprehension Task
The programming comprehension task uses exam-style multiple-choice questions to assess programming knowledge and code understanding. It varies question type, prompting format, and language, then scores final-answer accuracy.
3.1 Programming Comprehension Task
- Programming comprehension assesses understanding of code syntax, execution flow, and structure through multiple-choice questions.The paper describes this capability as foundational to other code-related tasks.Source: §3.1.1 · p. 6
3.1.1 Overview.
- The 250 questions come from confidential real-world college course final exams, are manually cross-validated, and are edited into single-choice form when needed.The source and confidentiality are intended to reduce the possibility of overlap with LLM training data.Source: §3.1.2 · p. 6
3.1.2 Data Processing.
- Questions are categorized as conceptual understanding, commonsense reasoning, or multi-hop reasoning, corresponding to programming concepts, code understanding, or multi-step code reasoning.The categories support finer-grained analysis of programming comprehension.Source: §3.1.2 · p. 6
- Answer-only prompts require one selected option, while CoT prompts require analytical text followed by a final answer whose correctness is scored.The examples include English and Chinese versions of the prompts.Source: §3.1.3 · pp. 7–8, Figure 3 · p. 7
3.1.3 Prompting Strategy.
- The testcase illustration shows quadratic-complexity code passing at least six cases, while a linear prefix-sum solution passes all test points.The figure distinguishes testcase difficulty by which solutions pass.Source: Figure 4 · p. 8
- Accuracy is computed from the final choice extracted by regular expression; a choice that cannot be extracted is counted as incorrect.The metric applies to responses from either prompting setting.Source: §3.1.4 · p. 8
3.2 Code Generation Task
CodeApex evaluates C++ function generation from problem descriptions and prototypes, using test-case outcomes to assess generated code. The task design covers varied question forms, prompting conditions, and measures of partial and complete correctness.
3.2.1 Overview.
- The Online Judge source supplies natural-language exercises, code templates, and test cases at varying difficulty levels, with private data intended to reduce training-data overlap.Source: §3.2.2 · p. 8
3.2.2 Data Processing.
- GPT-4-assisted function decomposition addresses prompt-related output failures, while questions distinguish explicit requirements from scenario-embedded narrative requirements.Source: §3.2.2 · p. 9
3.2.3 Prompting Strategy.
- Generation prompts combine question descriptions, function descriptions, and data-range specifications, with function-only and function-with-context scenarios.Source: §3.2.3 · p. 9
- Each question has 5–10 test cases, and scores weight passed test cases equally regardless of difficulty.Source: §3.2.4 · p. 10
3.2.4 Evaluation Metric.
- Generated functions are extracted and concatenated into the dataset’s main function, then assessed by comparing their outputs against original test cases.Source: §3.2.4 · pp. 10–11
- AC@1 measures whether code passes at least one test, AC@all measures passing every test, and AC Rate captures overall code-generation performance.Source: §3.2.4 · p. 11
3.3 Code Correction Task
CodeApex evaluates code correction by asking models to revise erroneous code using task descriptions, error messages, and standard code. The paper treats correction as an essential programming stage with software-engineering relevance.
3.3.1 Overview.
- Correction examples come from real student submissions and include Wrong Answer, Time Limit Exceeded, and Runtime Error categories.Source: §3.3.2 · p. 11
3.3.2 Data Processing.
- Incorrect and correct submissions from the same user are paired, with edit distance capped at 50; question counts are balanced and Wrong Answer cases predominate.Source: §3.3.2 · p. 11
3.3.3 Prompting Strategy.
- The three prompt scenarios provide code alone, code with an error message, or code with an error message plus standard code.Source: §3.3.3 · pp. 11, 13
- Modified code is scored with AC@1, AC@all, and AC Rate; outputs with edit distance above 50 score zero, and exact copies of standard code are marked incorrect.Source: §3.3.4 · p. 13
4 EVALUATION RESULTS
Across programming comprehension, code generation, and code correction, GPT-4 achieved the strongest reported results, while findings also show differences by language, question type, prompting, and model capability.
4.1.1 Dataset Statistics.
- The benchmark includes 250 comprehension questions, 476 code-generation questions, and 1,330 erroneous code segments tied to 116 generation questions.The test data is downloadable, but standard answers are withheld; submitted predictions receive automated accuracy assessments.Source: §4.1.1 · p. 13
4.2 Programming Comprehension Results
- Chinese programming-comprehension results were higher than English results; most models did best on conceptual questions and worst on multi-hop reasoning.The authors suggest English translation noise and models’ training data may contribute to the language difference.Source: §4.2 · pp. 14, 17
Across evaluation tasks
- GPT-4 had the highest average programming-comprehension accuracy, the best code-generation accepted rate over 66%, and the best code-correction accuracy of 66%.Most models scored below 50% on programming comprehension, and the authors report substantial room for improvement in code correction.Source: §4.2 · p. 14, Table 3 · p. 15, §4.3 · p. 17
4.2 Programming Comprehension Results
- Most models matched or underperformed their answer-only accuracy with chain-of-thought prompting, though some improved on multi-hop questions.The authors report gains for ChatGLM2, educhat, and GPT3.5-turbo on multi-hop reasoning.Source: §4.2 · p. 17
- Novice programmers performed similarly to GPT-4 in closed-book tests after learning, while humans in open-book exams outperformed all LLMs.The paper characterizes the LLM comprehension task as semi-open-book because models have limited offline knowledge bases.Source: §4.2 · p. 17
4.3 Code Generation Results
- Providing function-call context helped code generation, and LLMs performed better on explicit than narrative questions, which require converting natural language into program logic.Most models produced more than 50% compilable code; the authors also report that human submissions outperformed LLMs, especially on narrative questions.Source: §4.3 · pp. 17–18
- Human submissions outperformed all LLMs in code generation and did better on narrative questions, which the authors attribute partly to easier translation between natural language and code.The comparison used student learning records and pass rates across one, three, and five submissions.Source: §4.3 · p. 18
4.4 Code Correction Results
- Generic error types did not significantly help code correction; wrong-answer errors were hardest, while time-limit and runtime errors were easier to locate.The authors note that the error messages lacked line numbers or other detailed information; standard code could serve as a reference for correction.Source: §4.4 · pp. 18–21
5 CONCLUSION
CodeApex evaluates programming comprehension, code generation, and code correction in a bilingual benchmark. The conclusion reports that LLMs remain behind human performance and identifies future expansion of the test set.
5 CONCLUSION
- The benchmark evaluates comprehension, generation, and correction, including 476 C++ coding questions and 1,332 real-world erroneous code segments.Generation is tested through acceptance rates; comprehension uses three multiple-choice question categories.Source: §5 · p. 21
- Table 12 compares English code-correction performance across wrong-answer, time-limit-exceeded, and runtime errors using AC@1, AC@all, and AC Rate.The caption identifies best, second-best, and third-best performances but supplies no values.Source: Table 12 · p. 22
- GPT achieves top-tier programming performance, but comparisons with humans show substantial room for improvement.Specialized LLMs are competitive and show potential for further enhancement.Source: §5 · p. 22
- Experiments cover 12 general-purpose and code-specialized LLMs, with comparisons across task-specific prompting and context scenarios.Correction is evaluated with code-only, error-message, and standard-code scenarios, plus analyses by error type.Source: §5 · pp. 21–22
- The authors plan to update CodeApex with more question types and programming languages and present it as a reference for future study.They hope it will foster development of programming capabilities.Source: §5 · p. 22
A BREAKDOWN PERFORMANCE IN PROGRAMMING COMPREHENSION
The appendix tables break down programming-comprehension accuracy across shot settings, languages, prompting formats, and question categories.
A BREAKDOWN PERFORMANCE IN PROGRAMMING COMPREHENSION
- Tables 1, 2, 3, and 4 present a breakdown of LLM performance on the programming-comprehension task across three prompting conditions: 5-shot, 2-shot, and 0-shot.Source: A BREAKDOWN PERFORMANCE IN PROGRAMMING… · p. 26
- Table 1 gives Chinese-version answer-only accuracy by shot count and question category.Categories are conceptual understanding, commonsense reasoning, and multi-hop reasoning.Source: Table 1 · p. 26
B BREAKDOWN PERFORMANCE IN CODE GENERATION
The supplied passages describe breakdown tables for code generation and a separate programming-comprehension table.
B BREAKDOWN PERFORMANCE IN CODE GENERATION
- Tables 5 and 6 break down LLM performance on code-generation tasks across code-only and code-with-context scenarios, also distinguishing explicit questions from narrative questions.Source: B BREAKDOWN PERFORMANCE IN CODE… · p. 26
- Table 2 reports Chinese-version programming-comprehension accuracy in the chain-of-thought setting across 5-shot, 2-shot, and 0-shot scenarios.It distinguishes conceptual understanding, commonsense reasoning, and multi-hop reasoning questions.Source: Table 2 · p. 27
C BREAKDOWN PERFORMANCE IN CODE CORRECTION
The supplied passages cover breakdown tables for programming comprehension, code generation, and code correction, including language, error-type, and input-scenario distinctions.
C BREAKDOWN PERFORMANCE IN CODE CORRECTION
- Tables 7, 9, 11, 8, 10, and 12 present LLM code-correction performance breakdowns for wrong-answer, time-limited, and runtime errors across code-only, code-with-error-message, and code-with-standard-code scenarios.Source: C BREAKDOWN PERFORMANCE IN CODE… · p. 27
- Tables 3 and 4 report English-version Programming Comprehension Task accuracy results for 5-shot, 2-shot, and 0-shot settings, with Table 3 using answer-only and Table 4 using CoT.Source: Table 3 · p. 28, Table 4 · p. 29
- Tables 5 and 6 compare code-generation results for explicit and narrative questions in Chinese and English versions.The reported metrics are AC@1, AC@all, and AC Rate.Source: Table 5 · p. 30, Table 6 · p. 30
- Tables 7 and 8 report wrong-answer-error correction performance for Chinese and English versions, respectively.Both tables use AC@1, AC@all, and AC Rate.Source: Table 7 · p. 30, Table 8 · p. 31
- Tables 9 and 10 report time-limit-exceeded-error correction performance for Chinese and English versions, respectively.Both tables use AC@1, AC@all, and AC Rate.Source: Table 9 · p. 31, Table 10 · p. 31
- Tables 11 and 12 report runtime-error correction performance for Chinese and English versions, respectively.Both tables use AC@1, AC@all, and AC Rate.Source: Table 11 · p. 32, Table 12 · p. 32