Source-linked AI summary
CoderUJB: An Executable and Unified Java Benchmark for Practical Programming Scenarios
Zhengran Zeng, Yidong Wang, Rui Xie, Wei Ye, Shikun Zhang
TL;DR
Existing programming benchmarks often omit either multiple practical tasks, executable evaluation, or real-world project contexts. CoderUJB evaluates LLMs on 2,239 questions from 17 open-source Java projects across five tasks, and its results show task-dependent performance and training effects. The study reports particular difficulty with non-functional code generation and advises care with specialized training strategies.
Problem
Existing evaluations often focus on simple programming puzzles or omit executable, multi-task assessment in realistic development scenarios.
Source: §1 · p. 2, §2.1 · p. 3Method
CoderUJB uses 2,239 questions from 17 open-source Java projects across five practical tasks, executing generated solutions in complete project contexts.
Source: §1 · p. 2Results
LLMs remain weak on non-functional code generation, especially defect detection, while specialized pre-training and instruction fine-tuning can reduce performance on some tasks.
Source: §1 · p. 2Takeaways & Limitations
The findings support comprehensive multi-task evaluation and caution with specialized training, whose effects vary by task and can hinder performance.
Source: §1 · p. 2The study’s external validity depends on its selected LLMs, which the authors consider representative and influential based on their literature review.
Source: §6 · p. 11
Abstract
from arXiv · showhide
In the evolving landscape of large language models (LLMs) tailored for software engineering, the need for benchmarks that accurately reflect real-world development scenarios is paramount. Current benchmarks are either too simplistic or fail to capture the multi-tasking nature of software development. To address this, we introduce CoderUJB, a new benchmark designed to evaluate LLMs across diverse Java programming tasks that are executable and reflective of actual development scenarios, acknowledging Java's prevalence in real-world software production. CoderUJB comprises 2,239 programming questions derived from 17 real open-source Java projects and spans five practical programming tasks. Our empirical study on this benchmark investigates the coding abilities of various open-source and closed-source LLMs, examining the effects of continued pre-training in specific programming languages code and instruction fine-tuning on their performance. The findings indicate that while LLMs exhibit strong potential, challenges remain, particularly in non-functional code generation (e.g., test generation and defect detection). Importantly, our results advise caution in the specific programming languages continued pre-training and instruction fine-tuning, as these techniques could hinder model performance on certain tasks, suggesting the need for more nuanced strategies. CoderUJB thus marks a significant step towards more realistic evaluations of programming capabilities in LLMs, and our study provides valuable insights for the future development of these models in software engineering.
1 INTRODUCTION
CoderUJB addresses the lack of benchmarks that combine multiple programming tasks, executable evaluation, and real-world project scenarios. Its study finds uneven LLM performance across tasks and cautions that specialized training can have task-dependent effects.
1 INTRODUCTION
- Existing benchmarks lack a comprehensive combination of multiple programming tasks, code execution, and real-world development scenarios.HumanEval evaluates Python function generation through unit tests; CoderEval adds project-context questions, but both focus on a single programming scenario.Source: §1 · pp. 1–2
- CoderUJB contains 2,239 questions from 17 open-source Java projects, spanning five practical tasks and providing a complete program context for execution-based evaluation.The tasks include functional code generation, code-based and issue-based test generation, automatic program repair, and defect detection.Source: §1 · p. 2
- Current LLMs remain weak on non-functional code generation, especially defect detection, while superior open models approach or surpass GPT-3.5-Turbo on functional generation and both test-generation tasks.The study also finds that specific-language continued pre-training can reduce performance in other languages, and instruction fine-tuning can diminish performance on functional generation and test-generation tasks.Source: §1 · p. 2
- The study advises caution with language-specific continued pre-training and instruction fine-tuning because their effects vary across tasks and can reduce performance.It also reports that program context is useful in code generation and that comprehensive evaluation matters because training strategies can yield disparate results across tasks.Source: §1 · p. 2
2 BACKGROUND AND RELATED WORK
Prior code LLM research includes continued pre-training for specific programming languages and instruction tuning, while evaluation has often focused on Python puzzles. Existing benchmarks address separate parts of practical programming evaluation, leaving room for a unified, executable, project-based benchmark.
2.1 Large Language Models for Software Engineering
- Code LLMs use large-scale code training, while continued pre-training and instruction tuning have been used to improve HumanEval performance; evaluations still often focus on Python puzzles.The section describes StarCoder-Python and WizardCoder as examples of these training approaches and notes that their effects across practical software engineering scenarios are not comprehensively assessed.Source: §2.1 · pp. 2–3
2.2 Coder Benchmark
- Existing benchmarks split between function-generation puzzles, project-derived code questions, and specialized repair or test-generation sets; CodeXGLUE lacks execution and XCodeEval lacks project-runnable realism.CoderUJB is presented as combining multiple programming tasks with executable evaluation in real-world development scenarios.Source: §2.2 · p. 3
3 CODERUJB BENCHMARK
CoderUJB is an executable Java benchmark spanning five programming tasks, built from real open-source projects and designed to evaluate programming skills in a uniform dataset. Its construction, prompts, and metrics are tailored to the distinct requirements of those tasks.
3.1 Tasks in CoderUJB
- CoderUJB covers functional code generation, code-based and issue-based test generation, defect detection, and automated program repair.The tasks respectively generate operational function code, tests from code context, tests that reproduce reported bugs, bug classifications, and repairs to defective functions.Source: §3.1 · pp. 3–4
3.2 Datasets in CoderUJB
- All five task datasets draw on the same 17 Defects4j Java projects to keep source quality and difficulty more comparable across tasks.The authors chose a common source to minimize data-quality impacts on their findings.Source: §3.2 · p. 4
3.3 Dataset Construction
- Construction yields 238 functional-code questions, 140 code-based test questions, 470 APR defects, 940 defect-detection samples, and 451 issue-based test questions.Functional-code questions were filtered using test coverage and manual quality checks; APR and defect detection focus on single-function defects, and issue-based questions match APR benchmark bugs.Source: §3.3 · pp. 4–5
3.4 Base Prompt Design
- Functional-code prompts provide code context and task requirements, while chat and completion variants preserve content and differ in formatting.For test generation, prompts include both tested-class and test-class context; issue-based test prompts instead omit abstract class context and function comments to assess deriving task information from issue reports.Source: §3.4 · pp. 5–6
3.5 Metrics in CoderUJB
- pass@k estimates the chance of finding a correct solution among k sampled from n candidates, with n set to 20 and dataset results averaged across problems.Correctness depends on the task: generated tests must run and cover tested code, while issue-based tests must reproduce the bug in the defective version and run in the fixed version.Source: §3.5 · p. 6
- count@n sums problems for which at least one of n generated solutions is correct, using a binary score for each problem.This metric is intended to present code-generation ability in terms of the number of problems solved.Source: §3.5 · p. 6
- coverage@n measures the percentage of tested-code lines covered by the combined set of n generated tests, averaged across problems.The calculation unions the lines covered by each solution before comparing them with all tested lines.Source: §3.5 · pp. 6–7
- Defect-detection performance is evaluated with accuracy.Source: §3.5 · p. 7
4 THE EXTENSIVE STUDY
The study evaluates LLMs across CoderUJB’s programming scenarios and examines prompting, language-specific continued pre-training, and instruction fine-tuning. Results vary by task: leading open-source models approach or exceed GPT-3.5-Turbo on generation and test generation, but lag on repair, while defect detection remains especially difficult.
4.1 Research Questions
- The study asks how LLMs perform across tasks and how continued language-specific pre-training and instruction tuning affect them, evaluating base, specific-language, instruction-tuned, and commercial models.The selected subjects include open-source code LLMs and three closed-source commercial LLMs.Source: §4.1 · p. 7, §4.2 · p. 7
4.3.1 RQ1: Does the Basic Program Context Prompt Improve the Per-
- Program context improved functional generation, code-based test generation, and repair, but CodeLlama-13B defect-detection accuracy was 48.60%, versus 49.68% and 51.51% with few-shot prompts.Invalid responses could lower accuracy, so the authors also reported error counts and retained the program-context prompt for defect detection.Source: §4.3.1 · p. 8
4.3.2 RQ2: How Do Open-Source and Closed-Source LLMs Perform Under CoderUJB.
- Advanced open-source LLMs matched or surpassed GPT-3.5-Turbo on functional code generation and both test-generation tasks, but trailed it on automated program repair; GPT-4 performed substantially better overall.The comparison covers five programming scenarios under CoderUJB.Source: Table 4 · p. 9, §4.3.2 · p. 9
- GPT-4 reached only 52.16% accuracy in defect detection, and the authors describe CoderUJB as more challenging than HumanEval and CoderEval.The authors point to complex logic errors and the need to understand the full project as challenges for defect detection.Source: §4.3.2 · pp. 8–9
- Specific-language continued pre-training generally helped corresponding-language functional generation and hindered other-language tasks, while its effects on test generation and repair were less predictable.The authors relate the variability to how closely a downstream task resembles the pre-training task.Source: §4.3.2 · p. 10
4.3.4 RQ4: How Does Instruction Fine-Tuning Influence the Perfor-
- CodeLlama-Instruct-13B’s functional-generation pass-all@k scores dropped 39.39% at k=1 and 31.05% at k=10 relative to CodeLlama-13B.Most open-source instruction-tuned models struggled to outperform their base models on functional and test generation; the authors contrast this pattern with HumanEval results.Source: §4.3.4 · p. 10
- Generation declines for most instruction-tuned models were attributed to lower solution quality, while a CodeLlama-Instruct-34B mode-collapse anomaly was excluded; repair gains were linked to task-format diversity.The authors suggest instruction tuning may disturb direct code generation but improve adaptability to diverse task formats for repair.Source: §4.3.4 · pp. 10–11
- Instruction tuning reduced performance on tasks resembling pre-training formats, such as functional generation and test generation, but improved performance on automated program repair.The authors encourage further exploration of more effective fine-tuning strategies.Source: §4.3.4 · pp. 10–11
5 IMPLICATIONS AND DISCUSSIONS
The authors recommend task-aware choices in training and evaluation: program context can help, while continued pre-training and instruction fine-tuning have different effects across tasks.
5 IMPLICATIONS AND DISCUSSIONS
- Program context can enhance performance across various programming tasks, motivating research into advanced prompt designs that better harness it.Source: §5 · p. 11
- Language-specific continued pre-training generally helps on similar tasks but may impede performance in other languages; effects become unpredictable as tasks diverge.Source: §5 · p. 11
- Instruction fine-tuning may reduce performance on tasks aligned with pre-training, while instruction-tuned models tend to perform better on tasks that differ substantially.Source: §5 · p. 11
- Because training strategies yield disparate results across programming tasks, the authors advocate evaluating models and techniques with comprehensive benchmarks such as CoderUJB.Source: §5 · p. 11
6 THREATS TO VALIDITY
The study identifies implementation bugs, model selection, and evaluation metrics as threats to validity, while describing steps taken to mitigate them.
6 THREATS TO VALIDITY
- Implementation bugs are an internal-validity threat; the authors reviewed code and scripts and released them with generated results for public scrutiny.Source: §6 · p. 11
- The selected LLMs are an external-validity threat, which the authors address through a literature review and judge sufficiently representative and influential.Source: §6 · p. 11
- Metric choice is a construct-validity threat; the study used pass@k with adequate test coverage and added accuracy, count@n, and coverage@n.Source: §6 · p. 11
7 CONCLUSIONS
CoderUJB evaluates LLMs on executable tasks drawn from real Java projects, and the study reports persistent challenges and training trade-offs.
7 CONCLUSIONS
- CoderUJB uses executable code from 17 open-source Java projects to simulate real-world software engineering tasks for LLM evaluation.Source: §7 · p. 11
- The study highlights difficulties in non-functional code generation and defect detection, alongside potential performance decreases from continued pre-training and instruction fine-tuning.Source: §7 · p. 11
- The findings point to the need for nuanced training approaches to support versatility and robustness across coding tasks.Source: §7 · p. 11