PaperlayerView exact revision on arXiv

Source-linked AI summary

CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences

Martin Weyssow, Aton Kamanda, Xin Zhou, Houari Sahraoui

arXiv:2403.09032v3cs.SEcs.CLcs.LG

TL;DR

The paper addresses limited methods for evaluating and tuning LLMs against nuanced coding preferences by introducing a preference dataset and an LLM-judged benchmark. It reports improved alignment and functional correctness after tuning CodeLlama-7B-Instruct.

  • Problem

    Existing evaluation and tuning methods do not adequately capture nuanced user coding preferences, and large-scale datasets for aligning LLMs to these preferences are lacking.

    Source: §1 · p. 2

  • Method

    The authors build CodeUltraFeedback with 10,000 instructions and rated responses, create a 500-instruction LLM-judged benchmark, and use the dataset for SFT and DPO preference tuning.

    Source: §1 · pp. 2–3

  • Results

    SFT- and DPO-tuned CodeLlama-7B-Instruct outperforms 34B LLMs on CodeUltraFeedback-Bench and improves functional correctness on HumanEval+.

    Source: §8 · p. 22

  • Takeaways & Limitations

    The authors conclude that CodeUltraFeedback and its benchmark can support further research on LLM alignment in automated software engineering.

    Source: §8 · p. 22

    CodeUltraFeedback-Bench grading requires reference responses, a dependency the authors identify for future work to alleviate.

    Source: §6.3 · p. 21

Abstract

from arXiv · show

Evaluating the alignment of large language models (LLMs) with user-defined coding preferences is a challenging endeavour that requires a deep assessment of LLMs' outputs. Existing methods and benchmarks rely primarily on automated metrics and static analysis tools, which often fail to capture the nuances of user instructions and LLM outputs. To address this gap, we propose using the LLM-as-a-Judge methodology to evaluate the alignment of LLMs with coding preferences. Based on this approach, we present CodeUltraFeedback, a comprehensive dataset designed to facilitate the evaluation and improvement of LLM alignment. CodeUltraFeedback consists of 10,000 coding instructions, each annotated with four responses generated from a diverse pool of 14 LLMs. These responses are ranked based on five distinct coding preferences using GPT-3.5 as a judge, providing both numerical scores and detailed textual feedback. Our analysis of CodeUltraFeedback reveals that responses from GPT-3.5 and GPT-4 are generally preferred over those from open-weight LLMs, highlighting significant differences in alignment between closed and open-weight models. In turn, we explore the usage of CodeUltraFeedback as feedback data to fine-tune and align CodeLlama-7B-Instruct using supervised fine-tuning (SFT) and reinforcement learning from AI feedback (RLAIF) with direct preference optimization (DPO). The resulting aligned CodeLlama-7B-Instruct model outperforms larger LLMs in terms of alignment with coding preferences and shows improved functional correctness on the HumanEval+ benchmark compared to the original instruct model. Therefore, our contributions bridge the gap in preference tuning of LLMs for code and set the stage for further advancements in model alignment and RLAIF in automated software engineering.

1 Introduction

The introduction identifies a gap in tuning and evaluating LLMs against nuanced coding preferences, then presents a preference dataset and benchmark and explores preference tuning.

1 Introduction

  • Existing benchmarks often miss complex user instructions, automated metrics and tools overlook language nuance, and large-scale tuning datasets for coding preferences are lacking.The introduction frames these as gaps in measuring and improving alignment with non-functional coding requirements.Source: §1 · p. 2
  • CodeUltraFeedback pairs 10,000 complex instructions with 40,000 responses generated by four randomly selected models from a 14-LLM pool and rated by GPT-3.5.The annotations include numerical ratings and textual rationales across instruction following, code explanation, complexity and efficiency, readability, and coding style.Source: §1 · pp. 2–3
  • CodeUltraFeedback-Bench uses 500 instructions and LLM-as-a-Judge single-answer grading to assess alignment with coding preferences.Its grading approach is designed to evaluate outputs more context-sensitively than automated metrics and external tools.Source: §1 · p. 3
  • SFT and DPO substantially improve CodeLlama-7B-Instruct alignment across coding preferences, surpassing larger LLMs under GPT-3.5-Turbo judging.On CodeUltraFeedback-Bench, gains are more mitigated under GPT-4-Turbo judging, although the tuned 7B model still outperforms CodeLlama-13B-Instruct and CodeLlama-34B-Instruct.Source: §1 · p. 3
  • Preference tuning substantially improves Pass@k on HumanEval and HumanEval+, and QLoRA enables SFT and DPO on a single 24GB RTX A5000 GPU.The paper reports that tuning does not hinder functional correctness.Source: §1 · p. 3

2 CodeUltraFeedback

CodeUltraFeedback is built as preference-conditioned feedback data: it samples balanced coding instructions, gathers diverse model responses, and has GPT-3.5-Turbo assess them against five coding preferences.

2.1 Coding Preferences and Principles

  • The five preferences are instruction-following, code explanation, code complexity and efficiency, code readability, and coding style.They complement functional correctness by assessing qualities tied to comprehension, maintainability, performance, and collaboration.Source: §2.1 · p. 4

2.2 Initial Dataset

  • The dataset starts with 10,000 randomly selected Magicoder Evol-Instruct samples, balanced at 2,000 instructions per preference, partly to reduce API costs.Source: §2.2 · p. 4

2.3 LLMs Selection

  • Four models are randomly drawn per instruction from a pool of 14 LLMs spanning eight families, capturing varied capabilities and styles.The pool includes both general-purpose chat models and models optimized for code-related tasks.Source: §2.3 · pp. 4–6

2.4 Principle-Driven Code Generation

  • Ten principles are created for each preference, and one matching principle is randomly appended to each model’s prompt to guide generation.Source: §2.4 · p. 6

2.5 LLM-as-a-Judge Annotation

  • GPT-3.5-Turbo rates all four responses together using preference-specific criteria, producing a 1–5 alignment score and a rationale for each.The authors use analogous assessment templates for the five preferences to support consistent evaluation.Source: §2.5 · pp. 6–7

3 CodeUltraFeedback: Dataset Analysis

GPT-3.5-Turbo and GPT-4-Turbo lead average preference scores, while the dataset also contains diverse top-rated responses and broad response lengths; pairwise matchups favor them over other LLMs.

3.1 LLMs Scores Exploration

  • GPT-3.5-Turbo and GPT-4-Turbo score highest across preferences and on average; most pairwise differences versus GPT-3.5-Turbo have p<0.001, and GPT-4-Turbo performs notably better.Source: §3.1 · p. 8

3.2 LLMs Responses Analysis

  • Another LLM supplies the best response for 73% of samples, and lower-average models win at least 3.1%; best responses trend slightly longer across broad length ranges.The authors describe this response diversity as useful for avoiding alignment to a single model’s style, and the length distribution as helping avoid verbosity bias.Source: §3.2 · p. 9, Figure 2 · p. 9, Figure 3 · p. 10

3.3 LLMs vs GPT-3.5-Turbo

  • In matchups against GPT-3.5-Turbo, Mistral-7B-Instruct wins 29.8% of the time, while GPT-4-Turbo’s win rate is 51.6%.The authors identify further alignment work as an opportunity to make LLMs more competitive with GPT-3.5 and GPT-4.Source: §3.3 · pp. 9–10

4 Aligning LLMs to Coding Preferences

The experiments test whether CodeUltraFeedback can improve coding-preference alignment by tuning CodeLlama-7B-Instruct with SFT and DPO. They compare models on a 500-instruction benchmark using reference-guided LLM judging.

4 Aligning LLMs to Coding Preferences

  • The study uses CodeLlama-7B-Instruct to test whether SFT and DPO can improve alignment with coding preferences.The authors select the 7B model because it can be tuned on modest computing infrastructure and trails stronger models in CodeUltraFeedback scores.Source: §4 · p. 10, §4.3 · p. 12

4.1 Supervised Fine-Tuning (SFT)

  • SFT uses teacher-generated instruction-response pairs as a supervised signal to tune a student model.The experiments use Magicoder Evol-Instruct, filtering out the 10,000 CodeUltraFeedback samples to reduce leakage between SFT and DPO.Source: §4.1 · p. 10, §4.3 · p. 12

4.2 Direct Preference Optimization (DPO)

  • DPO aligns a model to preference data by ranking a preferred response above a rejected response, using the SFT-tuned model as its reference policy.For DPO, the authors use the remaining 9,500 samples and select highest- and lowest-rated responses as preferred and rejected examples.Source: §4.2 · pp. 11–12, §4.3 · p. 12

4.3 Experimental Details

  • The 500-sample CodeUltraFeedback-Bench, balanced across preferences, is used to assess and compare LLM alignment.The benchmark is distinct from the training dataset and is intended to evaluate the impact of alignment methods.Source: §4.3 · p. 12
  • Each model is graded individually against a reference response, with a rationale and a score from 1 to 10.GPT-3.5-Turbo and GPT-4-Turbo responses serve as references, and both models serve as judges.Source: §4.3 · p. 12

5 Results

SFT and DPO improve CodeLlama-7B-Instruct’s alignment with coding preferences, with DPO-based variants outperforming larger models across preferences. The results also show that judge choice affects alignment scores and that functional-correctness gains are strongest with SFT.

Alignment results

  • 7.89 on code complexity and efficiency for CodeLlama-7B-Instruct+SFT+DPO, versus 6.16 for the baseline, 6.93 for Wizard-33B, and 6.42 for CodeLlama-34B-Instruct.GPT-4-Turbo scores above all other LLMs across every coding preference; instruction following scores are lower across models.Source: §5 · p. 13
  • p < 0.001 for DPO and SFT+DPO alignment gains across preferences, except instruction following; both variants outperform CodeLlama-34B-Instruct and WizardCoder-33B across preferences.SFT alone improves scores, but is not statistically different from baseline except for readability.Source: §5 · pp. 13–15

DPO comparison with GPT-4-Turbo

  • 4.8% and 6.2% win-rate increases, and 15.4% and 16.4% tie-score increases, for DPO and SFT+DPO, respectively, against GPT-4-Turbo.The win-tie-lose comparison covers all preferences combined.Source: §5 · p. 15, Figure 7 · p. 15

Judge and reference effects

  • 7.08 for CodeLlama-7B-Instruct+SFT+DPO against GPT-3.5-Turbo references, near GPT-3.5-Turbo’s 7.18 average; scores decrease against GPT-4-Turbo references.Under GPT-4-Turbo’s judgment, CodeLlama-13B-Instruct falls from 5.58 to 4.83, while WizardCoder-33B rises from 6.26 to 6.75.Source: §5 · p. 16, Table 4 · p. 16

Human agreement

  • GPT-3.5 shows stronger agreement with human evaluators than GPT-4 across coding preferences; for readability, its κ values are 0.83 and 0.91, versus 0.48 and 0.43 for GPT-4.For most preferences except instruction following, GPT-3.5’s judgments align more with individual human judges than the human judges align with each other; both judges would benefit from calibration.Source: §5 · pp. 16–17, Figure 8 · p. 17

Functional correctness

  • Pass@1 rises from 37.9 to 51.2 on HumanEval and from 33.2 to 45.6 on HumanEval+ with SFT; DPO variants remain above baseline but below SFT alone.The authors attribute most functional-correctness improvement to SFT and note that DPO favors preference-rated responses rather than functional correctness.Source: §5 · pp. 17–18, Table 5 · p. 18

Judge behavior on test-passing solutions

  • GPT-4 rates test-passing HumanEval+ solutions consistently higher, while GPT-3.5’s ratings distinguish little between passing and failing solutions.The comparison concerns solutions generated by CodeLlama-7B-Instruct+SFT+DPO.Source: §5 · pp. 18–19, Figure 9 · p. 18

6 Discussion

The discussion presents LLM-as-a-Judge as a way to assess user coding preferences that automated metrics and static analysis miss, while identifying open questions about alignment, judging, and benchmark design.

6.1 Limitations of Existing Evaluation Metrics

  • Automated metrics and static analysis capture correctness and rule compliance but miss user-centric preferences such as readability and instruction adherence.LLM judges can assess language nuances such as meaningful comments, readable variable names, and logically structured explanations.Source: §6.1 · p. 19
  • Combining automated metrics with LLM judgments could yield more holistic code evaluations, including weighted assessments of correctness, readability, and explanation quality.The discussion presents this as a future possibility, not an evaluated result.Source: §6.1 · p. 19

6.2 LLM Alignment and LLM-as-a-Judge

  • Balancing functional correctness with non-functional requirements remains an open research question, despite CodeLlama-7B-Instruct+SFT+DPO scoring higher on HumanEval/HumanEval+ than the base model.The authors call for further investigation into integrating these learning objectives across benchmarks.Source: §6.2 · p. 19
  • CodeUltraFeedback ratings and rationales could train a small critic LLM, potentially reducing CodeUltraFeedback-Bench’s reliance on costly models such as GPT-4.The authors connect this direction to prior work on Shepherd [51] and Prometheus [20].Source: §6.2 · p. 20
  • After five DPO epochs, overfitting correlated with improved performance while the chosen–rejected reward gap continued growing, so reward accuracy may not be useful for model selection.The authors suggest further DPO experimentation to improve model tuning and selection.Source: §6.2 · p. 20
  • Despite consistent LLM rankings with GPT-3.5-Turbo, judge bias remains possible, including a preference for lengthy responses, and warrants further study.The grading procedures use single-answer grading, reference responses, and chain-of-thought prompting to support consistent judgments.Source: §6.2 · p. 20

6.3 Limitations

  • The methodology assumes a highly capable model such as GPT-3.5 or GPT-4 can proxy for human evaluation; a 100-sample comparison found GPT-3.5 strongly correlated with human judgments.The authors also cite prior findings on alignment with human judgments [62] and ICE-Score [67].Source: §6.3 · p. 20
  • The current benchmark uses randomly selected samples, requires reference responses for grading, and may benefit from future filtering or human annotations.The authors also identify alleviating the reference-response dependency as future work.Source: §6.3 · pp. 20–21

7 Related Work

The related work spans execution-based, general code, and non-functional evaluation benchmarks. The paper distinguishes its preference-focused dataset and LLM-based judging from approaches centered on unit tests, ground-truth solutions, or static analysis.

7.1 Execution-based Benchmarks

  • Execution-based benchmarks evaluate functional correctness with unit tests, while CodeUltraFeedback focuses on coding preferences through LLM proxy evaluation without unit tests or automated metrics.The authors consider functional correctness complementary to their preference-alignment goal.Source: §7.1 · p. 21

7.2 General Datasets and Closed-Solutions Benchmarks

  • General code datasets and closed-solution benchmarks assess core capabilities using task-specific transfer learning and ground-truth solutions, which may overlook valid code variations.CodeUltraFeedback and CodeUltraFeedback-Bench instead focus on human coding preferences and nuanced language assessment over strict ground-truth adherence.Source: §7.2 · pp. 21–22

7.3 Non-Functional Evaluation of LLM-Generated Code

  • Non-functional code evaluation includes static-analysis approaches and narrower benchmarks, whereas this work uses LLM reasoning across several coding preferences and provides data for preference tuning.The paper contrasts its broader scope with work focused on security, efficiency, or static-analysis-based assessment.Source: §7.3 · p. 22

8 Conclusion and Future Work

The paper concludes that CodeUltraFeedback and CodeUltraFeedback-Bench support alignment evaluation and preference tuning, and reports improved benchmark alignment and functional correctness for tuned CodeLlama-7B-Instruct.

8 Conclusion and Future Work

  • CodeLlama-7B-Instruct tuned with SFT and DPO outperforms 34B LLMs on CodeUltraFeedback-Bench and improves functional correctness on HumanEval+.The authors plan to explore additional judges and fine-tune their own judge LLM to reduce dependence on closed-source models such as GPT-4.Source: §8 · p. 22

A Assessment Templates

The assessment-template appendix lists templates for code complexity and efficiency, readability, and coding style, alongside templates for single-answer grading and HumanEval samples.

A Assessment Templates

  • The assessment templates cover code complexity and efficiency, code readability, and coding style.The supplied template titles identify these assessment areas but do not specify their full criteria.Source: Table 8 · p. 27, Table 9 · p. 27, Table 10 · p. 28
  • The appendix defines readability as ease of reading and understanding, maintainability as ease of modification or extension, and efficiency as execution efficiency.These definitions appear in the assessment-template material.Source: A Assessment Templates · p. 28
  • The coding-style template uses a five-point adherence scale, ranging from non-adherent to exemplary, with readability, maintainability, and efficiency among its stated qualities.The intermediate scale descriptions include moderate and high adherence.Source: A Assessment Templates · p. 28
  • The appendix also lists single-answer grading templates for instruction following, code explanation, complexity and efficiency, readability, and coding style, plus a HumanEval evaluation template.The supplied table passages identify template scope by title.Source: Table 11 · p. 28, Table 12 · p. 29, Table 13 · p. 30
  • Separate principle tables are listed for instruction following, code explanation, complexity and efficiency, readability, and coding style.The supplied table passages identify these principle categories by title.Source: Table 17 · p. 34, Table 18 · p. 35, Table 19 · p. 36
Loading 2403.09032v3…