Source-linked AI summary
ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation
Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, Yiling Lou
TL;DR
Existing code-generation benchmarks mainly evaluate standalone functions or statements, leaving class-level generation of interdependent methods insufficiently studied. This paper introduces ClassEval and evaluates 11 LLMs with multiple generation strategies, finding substantially weaker class-level performance and model-dependent strategy advantages.
Problem
Existing benchmarks largely evaluate standalone function- or statement-level generation, providing limited evidence about generating classes containing interdependent methods.
Method
The authors manually construct the ClassEval benchmark and evaluate 11 state-of-the-art LLMs using holistic, incremental, and compositional generation strategies with Pass@k.
Results
LLMs perform much worse on class-level than method-level generation; GPT-4 and GPT-3.5 dominate, while most other models perform better with method-by-method generation.
Takeaways & Limitations
Method-level coding ability does not equivalently reflect class-level ability, and generation strategy should vary with model capabilities.
Takeaways & Limitations
The benchmark’s limited size and programming-language coverage cannot guarantee that the findings generalize broadly.
Abstract
from arXiv · showhide
In this work, we make the first attempt to evaluate LLMs in a more challenging code generation scenario, i.e. class-level code generation. We first manually construct the first class-level code generation benchmark ClassEval of 100 class-level Python code generation tasks with approximately 500 person-hours. Based on it, we then perform the first study of 11 state-of-the-art LLMs on class-level code generation. Based on our results, we have the following main findings. First, we find that all existing LLMs show much worse performance on class-level code generation compared to on standalone method-level code generation benchmarks like HumanEval; and the method-level coding ability cannot equivalently reflect the class-level coding ability among LLMs. Second, we find that GPT-4 and GPT-3.5 still exhibit dominate superior than other LLMs on class-level code generation, and the second-tier models includes Instruct-Starcoder, Instruct-Codegen, and Wizardcoder with very similar performance. Third, we find that generating the entire class all at once (i.e. holistic generation strategy) is the best generation strategy only for GPT-4 and GPT-3.5, while method-by-method generation (i.e. incremental and compositional) is better strategies for the other models with limited ability of understanding long instructions and utilizing the middle information. Lastly, we find the limited model ability of generating method-dependent code and discuss the frequent error types in generated classes. Our benchmark is available at https://github.com/FudanSELab/ClassEval.
1 INTRODUCTION
The paper introduces ClassEval to address the limits of standalone function-level evaluation and studies how 11 LLMs generate classes containing interdependent methods.
- Existing code-generation benchmarks mainly evaluate one standalone function or statement, rather than classes containing multiple interdependent methods.
- ClassEval is the first manually constructed class-level Python benchmark, comprising 100 tasks created with approximately 500 person-hours.
- The benchmark covers practical software-development topics and uses test suites with high branch- and statement-level coverage for correctness checking.
- The empirical study evaluates 11 state-of-the-art LLMs using holistic, incremental, and compositional generation strategies, with correctness measured by Pass@k.
- All evaluated LLMs perform much worse on class-level generation than on standalone method-level benchmarks, and method-level ability does not equivalently reflect class-level ability.
- GPT-4 and GPT-3.5 dominate class-level performance, while Instruct-StarCoder, Instruct-CodeGen, and WizardCoder form a similar second tier.
- Holistic generation is best only for GPT models; other models perform better with method-by-method generation and show limited ability to generate method-dependent code.
2 BACKGROUND
The background reviews LLM-based code generation and shows that existing benchmarks largely test short, standalone code units. ClassEval is motivated as an evaluation of longer classes with interdependent methods.
- 2.1 Large Language Models for Code Generation: LLMs for code generation include general and code-specialized models trained on massive corpora, with differing objectives such as next-token prediction and fill-in-the-middle.
- 2.2 Existing Benchmarks for Code Generation: Code-generation benchmarks provide natural-language inputs and ground-truth code, commonly assessing correctness with Pass@k.
- 2.2 Existing Benchmarks for Code Generation: The benchmark review covers 13 distinct datasets drawn from popular downloads, recent LLM papers, and enhanced benchmarks.
- 2.2 Existing Benchmarks for Code Generation: Existing benchmarks mainly evaluate one short, standalone function or statement and rarely provide additional code context.
- 2.2 Existing Benchmarks for Code Generation: Their task lengths typically range from 1 to 30 lines and 4.6 to 108.2 tokens, leaving longer code-generation capability unclear.
- 2.2 Existing Benchmarks for Code Generation: ClassEval addresses this gap by asking models to generate longer, compound classes containing multiple interdependent methods.
3 NEW BENCHMARK CLASSEVAL
ClassEval is a manually constructed benchmark for generating complete Python classes from structured class skeletons, with tests and canonical solutions. It targets larger, dependency-rich tasks across practical software topics.
- 3.1 Benchmark Format: Each task contains a target-class description, a test suite, and a canonical reference implementation.
- 3.1 Benchmark Format: Class skeletons specify class descriptions, names, method signatures, functional descriptions, and optional implementation-context elements.Optional elements include imports, constructors, parameter and return descriptions, and example inputs and outputs.
- 3.2 Benchmark Construction Procedure: The benchmark construction process selects tasks, builds contract-based skeletons, creates test suites, and writes canonical solutions.
- 3.2.1 Task Selection: ClassEval includes 100 diverse tasks spanning management systems, data formatting, mathematics, games, file handling, databases, and natural language processing.Tasks were selected from existing benchmarks, PyPI topics, and author brainstorming, while environment-dependent tasks were excluded.
- 3.3 Benchmark Characteristics: Methods are designed with diverse class-internal dependencies, and ClassEval combines larger snippets, diverse dependencies, sufficient tests, and practical topics.The benchmark distinguishes method-level and class-level tests to assess methods independently and in combination.
- 3.3 Benchmark Characteristics: ClassEval contains 100 classes and 412 methods, averaging 45.7 lines of code—4.0× HumanEval and 6.7× MBPP.
4 EMPIRICAL STUDY
The empirical study evaluates 11 LLMs on ClassEval using holistic, incremental, and compositional generation strategies. It measures correctness at class and method granularity and evaluates generated method and field dependencies.
- 4 EMPIRICAL STUDY: The study evaluates 11 recent state-of-the-art LLMs selected across model sizes, foundations, sources, and domains.The models include both closed-source and open-source systems, with reported filling-in-the-middle and instruction-following capabilities.
- 4.2 Generation Strategies: Holistic generation produces the entire class at once, whereas incremental and compositional generation complete methods sequentially.Incremental generation uses previously generated method bodies; compositional generation generates each method independently before assembly.
- 4.3 Prompting: Prompts differ by strategy and model capability, using instructions for instruction-following models and code context alone for models without that capability.
- 4.4 Metrics: Class-level Pass@k requires generated classes to pass all method-level and class-level tests, while method-level Pass@k requires passing method-level tests.
- 4.4 Metrics: DEP measures the percentage of canonical method and field dependencies reproduced in generated methods.The study separately considers method dependencies and field dependencies.
- 4.5 Experimental Settings: Each strategy uses five nucleus-sampled solutions to calculate Pass@k for k={1,3,5}, plus one greedy sample for Pass@1 and DEP.
5 RESULTS
ClassEval exposes substantially lower class-level correctness than standalone method-level performance, while revealing model-specific rankings, strategy preferences, dependency difficulties, and recurring execution errors.
- Overall Correctness: 85.4%/68.9% method-level Pass@1 for GPT-4/GPT-3.5 on HumanEval falls to 37.0%/27.0% class-level correctness on ClassEval.WizardCoder similarly drops from 59.8% on HumanEval to 11.0% correct classes on ClassEval.
- Model Comparison: GPT-4 and GPT-3.5 substantially outperform other models on class-level coding, exceeding third-ranked WizardCoder by 25.4% and 17.4% in nucleus-sampling Pass@1.SantaCoder is an exception among smaller or general models, matching larger code models with fewer parameters.
- Overall Correctness: WizardCoder, Instruct-StarCoder, and Instruct-CodeGen achieve similar class-level Pass@1 with greedy sampling, ranging from 10.0% - 11.1%.Their substantially different HumanEval performance therefore does not transfer directly to ClassEval.
- Generation Strategies: Holistic generation is best only for GPT-4 and GPT-3.5, improving class-level Pass@5 by 6% - 9% and 4% - 14%, respectively.Other models generally perform better with incremental or compositional method-by-method generation; CodeGeeX and SantaCoder gain 9% and 7% with incremental generation.
- Dependency Generation: All models generate field-accessing code more successfully than method-invoking code, while GPT-4 exceeds other models by at least 12.6%/6.3% in DEP(F)/DEP(M).Methods invoking one or two other methods have average correctness ratios of 27.7% and 27.6%, respectively.
- Bad Case Analysis: Most incorrect classes encounter AttributeError or TypeError, while some produce KeyError through erroneous dictionary access caused by misinterpreted field dependencies.The reported GPT-3.5 example accesses self.sex as a dictionary key where the dictionary contains “male”.
6 THREATS TO VALIDITY
The benchmark and empirical study face threats from limited benchmark scope, prompt sensitivity, possible data leakage, implementation issues, and response randomness.
- Threats in benchmark construction: ClassEval’s limited size and programming-language coverage may restrict the generalizability of the findings.The authors plan to extend the benchmark.
- Threats in benchmark construction: Benchmark construction may face data leakage, subjectivity, and manual mistakes, while experimental results may depend on prompt choice and model-response randomness.The authors use multiple participants, pilot prompt selection, official model versions, and greedy decoding to mitigate these threats.
7 RELATED WORK
Related work situates ClassEval within broader efforts to evaluate LLM capabilities across diverse tasks, including code generation and traditional NLP benchmarks.
- LLM evaluation: Multi-faceted LLM evaluation is important for understanding model capabilities given the black-box nature of LLMs.Existing evaluation spans traditional NLP tasks such as sentiment analysis, question answering, and reasoning, as well as specialized tasks.
8 CONCLUSION
The study establishes class-level code generation as a harder and distinct evaluation setting, with LLM performance differing substantially from method-level results. GPT models remain strongest, while most other models benefit from method-by-method generation and still struggle with dependent code.
- All evaluated LLMs perform much worse on class-level code generation than on method-level generation.
- Method-level coding performance does not equivalently predict model performance on class-level code generation.
- GPT models dominate other evaluated LLMs on class-level code generation, while method-level performance rankings do not hold.
- Most non-GPT models perform better when generating classes method by method rather than all at once.
- LLMs have limited ability to generate code containing dependencies between methods.