PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating and Aligning CodeLLMs on Human Preference

Jian Yang, Jiaxi Yang, Ke Jin, Yibo Miao, Lei Zhang, Liqun Yang, Zeyu Cui, Yichang Zhang, Binyuan Hui, Junyang Lin

arXiv:2412.05210v1cs.CL

TL;DR

Code-generation benchmarks mainly assess code correctness, leaving human preference in practical coding scenarios less directly evaluated. The paper introduces CodeArena, a human-curated benchmark, and SynCode-Instruct, a synthetic instruction corpus. Evaluations show differences from execution-based benchmarks and a performance gap between open and closed-source code LLMs.

  • Problem

    Code benchmarks mainly validate code correctness, while preference alignment in responses to practical coding queries remains under-evaluated.

    Source: §1 · pp. 1–2

  • Method

    The paper constructs CodeArena from real-world user queries, develops the web-scaled SynCode-Instruct corpus, and evaluates models using preference-based comparisons.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    CodeArena produces results that differ from execution-based benchmarks and shows a substantial performance gap between open-source code LLMs and closed-source models.

    Source: §1 · p. 2, §5.2 · p. 8

  • Takeaways & Limitations

    Qwen2.5-SynCoder, trained on SynCode-Instruct, performs strongly on CodeArena, supporting the authors’ proposed use of large-scale synthetic instructions.

    Source: §5.1 · p. 7

Abstract

from arXiv · show

Code large language models (codeLLMs) have made significant strides in code generation. Most previous code-related benchmarks, which consist of various programming exercises along with the corresponding test cases, are used as a common measure to evaluate the performance and capabilities of code LLMs. However, the current code LLMs focus on synthesizing the correct code snippet, ignoring the alignment with human preferences, where the query should be sampled from the practical application scenarios and the model-generated responses should satisfy the human preference. To bridge the gap between the model-generated response and human preference, we present a rigorous human-curated benchmark CodeArena to emulate the complexity and diversity of real-world coding tasks, where 397 high-quality samples spanning 40 categories and 44 programming languages, carefully curated from user queries. Further, we propose a diverse synthetic instruction corpus SynCode-Instruct (nearly 20B tokens) by scaling instructions from the website to verify the effectiveness of the large-scale synthetic instruction fine-tuning, where Qwen2.5-SynCoder totally trained on synthetic instruction data can achieve top-tier performance of open-source code LLMs. The results find performance differences between execution-based benchmarks and CodeArena. Our systematic experiments of CodeArena on 40+ LLMs reveal a notable performance gap between open SOTA code LLMs (e.g. Qwen2.5-Coder) and proprietary LLMs (e.g., OpenAI o1), underscoring the importance of the human preference alignment.\footnote{\url{https://codearenaeval.github.io/ }}

1 Introduction

Code-generation benchmarks largely test code correctness, while practical coding tasks also call for responses aligned with human preferences. The paper introduces CodeArena and SynCode-Instruct, then uses CodeArena to evaluate models and compare preference-alignment performance.

1 Introduction

  • Existing code benchmarks primarily validate generated snippets, while human preference and real-world response quality remain under-evaluated.The paper also identifies a scarcity of effective data for improving preference alignment during supervised fine-tuning.Source: §1 · pp. 1–2
  • CodeArena is a human-curated benchmark of 397 high-quality samples across 40 categories from real-world queries; SynCode-Instruct contains nearly 20 billion tokens of web-sourced instructions. Evaluation with CodeArena reveals significant performance differences between code-execution-based benchmarks and the human-curated benchmark.Source: §1 · p. 2
  • CodeArena targets response alignment with human preferences across seven major categories and 40 subcategories.The paper describes SynCode-Instruct as a large-scale synthetic code-instruction corpus and Qwen2.5-SynCoder as a strong CodeArena baseline.Source: §1 · p. 2
  • Evaluations of 40+ LLMs and a dynamically updated leaderboard are used to assess alignment between model responses and human preferences.The authors report a substantial performance gap between open-source code LLMs and closed-source models.Source: §1 · p. 2

2 CodeArena

CodeArena is a human-curated benchmark built from real-world coding queries, with a broad task taxonomy and programming-language coverage. Its annotation, quality checks, and preference-based judging support evaluation of prompts that are not well suited to unit tests.

2 CodeArena

  • CodeArena contains nearly 400 problems organized into seven main classes and 40 subclasses, with questions ranging from 5 to 6,736 tokens.The average question length is 291 tokens, measured with the Qwen2.5-Coder tokenizer.Source: §2 · p. 2

Multiple Programming Languages

  • CodeArena emphasizes commonly used programming languages and includes Google Apps Script and PowerShell for practical Q&A scenarios.The benchmark comparison describes coverage of 44 programming languages and 40 subtasks.Source: Multiple Programming Languages · pp. 2–3
  • Most CodeArena samples are classified as medium or hard, rather than easy.The benchmark classifies every sample as easy, medium, or hard.Source: Multiple Programming Languages · p. 3
  • Four full-time annotators created and labeled candidate queries, and four senior developers checked validity; 397 samples were retained after at least three checkers agreed.The process selected 822 suitable original samples from 2,000 uniformly sampled items.Source: Multiple Programming Languages · p. 3
  • GPT-4o-2024-08-06 judges responses in both A/B input orders to calculate each model’s win rate against the gpt-4-turbo-2024-04-09 baseline.The reversed order is used to avoid the relative positions of A and B affecting the results.Source: Multiple Programming Languages · p. 3
  • CodeArena includes realistic prompts that are not suitable for verification through unit testing, unlike benchmarks centered on code-execution evaluation.Its diverse user prompts are intended to evaluate alignment between generated responses and human preferences.Source: Multiple Programming Languages · p. 3

3 SynCode-Instruct

SynCode-Instruct is built by recalling and filtering web-sourced code material, then generating instruction-response pairs with quality selection tailored to query type. The resulting synthetic corpus is used in the first stage of a two-stage training process.

3 SynCode-Instruct

  • A trained fastText classifier identifies code-related text in Common Crawl, while retrieval and filtering cover both files and repositories.Weak model-based classifiers and scorers filter low-quality content.Source: §3 · p. 3
  • A CodeBERT-based classifier identifies programming languages in snippets, with high-resource language data downsampled to balance the data.The classifier is fine-tuned across nearly 100 programming languages, and samples without code snippets are removed.Source: §3 · p. 3
  • After rule-based cleaning, Qwen2.5-72B creates new questions from the recalled content rather than extracting existing question-answer pairs.Qwen2.5-Coder generates multiple responses for each question.Source: §3 · pp. 3–4
  • For non-algorithmic queries, the pipeline selects among four sampled responses using an LLM scorer; for algorithmic queries, generated tests and a multilingual sandbox check code correctness.The selected response is used to form SynCode-Instruct; synthetic Qwen2.5 data is used in the first training stage and GPT-4o data in the second.Source: §3 · p. 4

4 Experimental Setup

The setup describes CodeArena’s construction and the model suite, then specifies execution-based and judge-based evaluation and SynCoder training. It distinguishes benchmarks for code correctness from CodeArena’s pairwise preference comparisons.

4.1 Instruction Dataset

  • Website code Q&A and related text are clustered and classified with an LLM, then uniformly sampled across subtasks for manual annotation.The evaluation includes general and code-focused LLMs, including open- and closed-source models.Source: §4.1 · pp. 4–5, Figure 5 · p. 5

4.2 Evaluation Benchmark

  • EvalPlus reports HumanEval and MBPP scores with base and expanded tests, while MultiPL-E evaluates HumanEval and translated tests across several languages.CodeArena includes many non-algorithmic questions that are unsuitable for code-execution evaluation.Source: §4.2 · p. 5

4.3 Evaluation Metrics

  • Execution benchmarks use greedy Pass@1; CodeArena uses pairwise LLM judgments and reports win and tie rates against a baseline.The judge sees a question and two answers, and can declare one better or call a tie.Source: §4.3 · pp. 5–6, §4.2 · p. 5

4.4 Impletmentation Details

  • The authors fine-tune Qwen2.5-Coder-32B on nearly 20B; separately, Qwen2.5-SynCoder is fine-tuned on SynCode-Instruct using 256 NVIDIA A100-80GB GPUs.GPT-4o generates 1B tokens, and Qwen2.5-Coder-Instruct generates the remaining tokens. Training uses Adam, a global batch size of 2048 samples, and sentence truncation at 32K tokens.Source: §4.4 · p. 6

5 Results and Discussion

CodeArena comparisons show a gap between open code LLMs and leading closed-source models, while SynCoder performs strongly on both preference and execution benchmarks. The discussion contrasts benchmark distributions and response styles, and examines instruction-data scaling.

5.1 Main Results

  • Closed-source Claude and o1 models retain a notable CodeArena advantage over open code LLMs such as Qwen-Coder and DeepSeekCoder.The results are reported as win and tie rates for instruction LLMs on CodeArena.Source: §5.1 · p. 6, Table 4 · p. 7
  • Qwen2.5-SynCoder, trained entirely on SynCode-Instruct, achieves strong CodeArena performance.On EvalPlus and MultiPL-E, it significantly beats previous strong open-source baselines and closes the gap with GPT-4o and Claude.Source: §5.1 · p. 7

5.2 Discussion

  • Open-source models can score poorly on CodeArena when their answers omit explanations or other details, even when generated code is correct.The discussion describes closed-source o1 and Claude models as more balanced across CodeArena and execution benchmarks.Source: §5.2 · pp. 7–8, Figure 8 · p. 8
  • Increasing SynCode-Instruct data improves SynCoder's CodeArena performance, and two-stage SFT outperforms one-stage training in the reported comparison.The final addition of high-quality data brings a large improvement; Figure 9 shows results across data sizes.Source: Figure 9 · p. 8, §5.2 · p. 8
  • CodeArena queries better match real Q&A than fixed-format algorithmic benchmarks, and its query representations are more broadly distributed than MultiPL-E's.The language-distribution comparison is visualized in Figure 10.Source: §5.2 · pp. 7, 9, Figure 10 · p. 9

6 Related Work

The related work surveys code LLMs and benchmarks spanning several programming tasks, languages, and capabilities. It notes that many benchmarks assess a single aspect rather than broad real-world development contexts.

Code Large Language Model

  • Code LLMs have advanced in multilingual code generation and debugging, with models evaluated on benchmarks including MultiPL-E, McEval, and MdEval.The discussion also describes benchmarks for translation, retrieval, completion, debugging, and structured-data understanding.Source: §6 · p. 9

Code Benchmarks

  • Many benchmarks focus on one capability, motivating evaluation across practical code-development contexts.The passage names FullStack Bench as an effort to evaluate multiple such contexts.Source: §6 · p. 9

7 Conclusion

The paper presents CodeArena, a human-curated benchmark built from real-world coding queries, and SynCode-Instruct, a large synthetic instruction corpus. Evaluation with CodeArena finds differences from code-execution benchmarks and a performance gap between open-source and closed-source code LLMs.

  • CodeArena comprises 397 human-curated samples from real-world user queries, spanning 40 categories.Source: §7 · p. 9
  • SynCode-Instruct is a diverse synthetic instruction corpus of nearly 20 billion tokens, built by scaling web-sourced instructions.Source: §7 · p. 9
  • Evaluation of over 20 LLMs with CodeArena reveals performance discrepancies from code-execution-based benchmarks and between open-source and closed-source code LLMs.The paper cites DeepSeek-Coder as an open-source example and the o1 and Claude series as closed-source examples.Source: §7 · p. 9
Loading 2412.05210v1…