PaperlayerView exact revision on arXiv

Source-linked AI summary

PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedback

Bo Shen, Jiaxin Zhang, Taihong Chen, Daoguang Zan, Bing Geng, An Fu, Muhan Zeng, Ailun Yu, Jichuan Ji, Jingyang Zhao, Yuenan Guo, Qianxiang Wang

arXiv:2307.14936v1cs.CLcs.AIcs.LGcs.PLcs.SE

TL;DR

Code LLMs have been developed to improve code generation, but existing reinforcement-learning approaches face limited gains, complex training, and costly test execution. The paper introduces RRTF, which aligns Code LLMs using ranked responses, and reports that PanGu-Coder2 achieves new state-of-the-art performance across code-generation benchmarks. The authors identify high-quality training data as important and note a benchmark overlap caveat tied to StarCoder's training data.

  • Problem

    Existing reinforcement-learning approaches for Code LLMs offer limited improvement, require complex algorithms, and spend time running tests during training.

    Source: §1 · p. 2

  • Method

    RRTF fine-tunes Code LLMs by ranking candidate responses using test-case and heuristic feedback rather than absolute reward-model values.

    Source: §1 · p. 2, §5 · p. 12

  • Results

    PanGu-Coder2 improves nearly 30% over its base model and achieves new state-of-the-art performance on HumanEval, CoderEval, and LeetCode, surpassing previously published Code LLMs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors identify high-quality data as important for improving instruction following and code writing, and conjecture that Code LLMs can surpass natural-language models of the same or larger sizes on code generation.

    Source: §1 · p. 2

    HumanEval may overlap with StarCoder's training data, which consists only of code from before June 2022.

    Source: §4.1.2 · p. 6

Abstract

from arXiv · show

Large Language Models for Code (Code LLM) are flourishing. New and powerful models are released on a weekly basis, demonstrating remarkable performance on the code generation task. Various approaches have been proposed to boost the code generation performance of pre-trained Code LLMs, such as supervised fine-tuning, instruction tuning, reinforcement learning, etc. In this paper, we propose a novel RRTF (Rank Responses to align Test&Teacher Feedback) framework, which can effectively and efficiently boost pre-trained large language models for code generation. Under this framework, we present PanGu-Coder2, which achieves 62.20% pass@1 on the OpenAI HumanEval benchmark. Furthermore, through an extensive evaluation on CoderEval and LeetCode benchmarks, we show that PanGu-Coder2 consistently outperforms all previous Code LLMs.

1 Introduction

Existing reinforcement-learning approaches for code face limited gains, costly test-time feedback, and training complexity. The paper proposes RRTF, which uses ranked responses to align Code LLMs, and presents PanGu-Coder2 as a strong benchmark-performing model.

1 Introduction

  • Existing code-RL methods provide limited gains, use complex algorithms, and incur time-consuming test execution during training.The authors say earlier approaches consequently used modestly sized models and achieved rather limited improvement.Source: §1 · p. 2
  • RRTF applies natural-language LLM alignment techniques to Code LLMs, using response rankings rather than absolute reward-model values.The authors characterize this training approach as simpler and more efficient than prior approaches such as CodeRL and RLTF.Source: §1 · p. 2
  • Training-process analysis and manual inspection of generated code samples highlight high-quality data as important for instruction following and code writing.The paper also conjectures that Code LLMs can surpass natural-language models of the same or larger sizes on code generation.Source: §1 · p. 2
  • RRTF is presented as a data-efficient, easy-to-implement, model-agnostic framework for boosting pre-trained Code LLMs.The paper applies it to StarCoder 15B to produce PanGu-Coder2.Source: §1 · p. 2
  • Nearly 30% improvement over its base model, with new state-of-the-art results on HumanEval, CoderEval, and LeetCode, surpasses previously published Code LLMs.The paper reports PanGu-Coder2 as outperforming all previously published Code LLMs on these benchmarks.Source: §1 · p. 2

2 Related Work

Related work spans the growth of Code LLMs and their applications, preference-based and test-feedback reinforcement learning, and fine-tuning strategies. PanGu-Coder2 uses ranking feedback during fine-tuning.

2.1 Large Language Model for Code (Code LLMs)

  • Code LLM work expanded from open-source code generators to models for multilingual coding, interactive programming, API use, and generation at arbitrary code positions.Instruction tuning on curated high-quality datasets is another route to code-generation gains.Source: §2.1 · p. 2

2.2 Reinforcement Learning on LLM

  • RLHF aligns language-model behavior with human preferences, while code-RL methods use test feedback and include online, multi-granularity feedback.RLHF is resource-intensive and complex to tune; the related work also discusses ranked-output alternatives that use simpler training procedures.Source: §2.2 · p. 3

2.3 Fine-tuning Code LLM

  • Fine-tuning pre-trained models is established for downstream code tasks, and PanGu-Coder2 uses ranking feedback during fine-tuning.The section also describes instruction tuning as supervised fine-tuning on high-quality instruction corpora.Source: §2.3 · p. 3

3 Approach

RRTF samples responses, ranks them using unit tests and heuristic preferences, then trains the Code LLM on scored chosen/rejected responses. The approach uses an evolved instruction corpus and a StarCoder 15B base model.

3.4 RRTF framework

  • RRTF samples student and teacher responses, ranks them using test outcomes and heuristic preferences, then trains on scored chosen/rejected responses.Responses are sampled offline and in parallel; the ranking stage assigns scores by compilation and test results and prefers teacher responses when scores tie.Source: §3.1 · pp. 3–4, §3.4 · p. 5

3.1 Overview

  • The overview presents RRTF as a three-stage pipeline: sample responses, rank them using unit tests and heuristic preferences, and train on scored chosen/rejected responses.The sampling prompts are generated via Evol-Instruct.Source: §3.1 · pp. 3–4

3.2 Model Architecture

  • The model is based on a decoder-only Transformer with Multi-Query-Attention and learned absolute positional embeddings; Table 1 lists its hyperparameters.FlashAttention is used to reduce computation and memory, allowing a maximum model length of 8192.Source: §3.2 · p. 4, Table 1 · p. 4

3.3 Training Corpus

  • The corpus starts from Alpaca 20K, is evolved into 100K programming problems with answers, then filtered to 68K instruction-solution pairs.The authors report surveying the 68K corpus for overlap with HumanEval and finding no data leakage.Source: §3.3 · p. 4

3.4 RRTF framework

  • The framework replaces human-intent feedback with test-and-teacher feedback to rank responses, which can be obtained fully or semi-automatically.Training uses ranking loss alongside cross-entropy loss to learn the teacher response.Source: §3.4 · p. 5

3.5 Implementation Details

  • Training uses StarCoder 15B with a global batch size of 512 for 6 epochs, and samples use extracted code snippets with a triple-quoted prompt wrapper.The sample format is shown in Figure 3.Source: §3.5 · p. 6, Figure 3 · p. 6

4 Evaluation

The evaluation section describes its settings, experimental results, and findings.

  • The section reports evaluation settings, experimental results, and findings, but this passage does not specify the benchmarks or outcomes.The passage serves as an introduction to the evaluation section.Source: §4 · p. 6

4.1 Evaluation Setup

The evaluation compares PanGu-Coder2 with code models across three benchmarks, using pass@k to measure functional correctness. Decoding and prompt choices are specified to support comparison across models.

4.1.1 Main Evaluated Models

  • The evaluated models include CodeGen-mono, CodeGeeX, StarCoder, CodeT5+, and WizardCoder, with model sizes and distinguishing characteristics noted.CodeGen-mono is fine-tuned with additional Python code; StarCoder supports infilling, and WizardCoder uses Evol-Instruct.Source: §4.1.1 · p. 6

4.1.2 Benchmarks

  • The benchmarks cover HumanEval, realistic software-development functions in CoderEval, and LeetCode problems tested with editor hints and public tests.LeetCode includes 300 problems across easy, medium, and hard levels; problems were selected to avoid overlap with StarCoder’s pre-June 2022 training data.Source: §4.1.2 · pp. 6–7

4.1.3 Metric

  • Pass@k estimates functional correctness from generated samples, counting a sample as correct only when it passes all unit tests.The estimator uses the number of generations, correct samples, and sampling budget.Source: §4.1.3 · p. 7

4.1.4 Decoding Strategy

  • Pass@1 uses temperature 0.2, while pass@10 and pass@100 use temperature 1.2; model comparisons on three benchmarks use greedy decoding.Available models use 200 generated samples and top_p 0.95; closed-source results are retrieved from previous papers.Source: §4.1.4 · p. 7

4.1.5 Prompts

  • Each model is evaluated with the prompt reported in its corresponding paper, and Table 2 reports HumanEval pass@1/10/100 results under specified sampling settings.PanGu-Coder2, StarCoder, and WizardCoder use n=200 and top_p=0.95 for the reported scores.Source: §4.1.5 · p. 7, Table 2 · p. 8

4.2 Evaluation Results

PanGu-Coder2 achieves the best reported greedy pass@1 results across HumanEval, CoderEval, and LeetCode. Training analyses and case studies show how dataset size, training duration, and model-specific successes and failures qualify that performance.

4.2.1 Performance

  • 61.64% pass@1, 79.55% pass@10, and 91.76% pass@100 make PanGu-Coder2 the top open-source model on HumanEval across all three k values.It exceeds WizardCoder by 4.34% and improves pass@1 by 28% absolute over StarCoder; it ranks second among closed-source models, beats GPT-3.5, but trails GPT-4.Source: §4.2.1 · p. 8
  • 62.20% pass@1 on HumanEval, 38.26% on CoderEval, and 32/30/10 on LeetCode give PanGu-Coder2 the best results among the tested models across all three benchmarks.The comparison uses greedy decoding.Source: §4.2.1 · p. 8, Table 3 · p. 9

4.2.2 Findings

  • Larger 38k/68k datasets produce more stable training curves at roughly 2–3 epochs, while the 18k dataset still oscillates after 3 epochs.Overall greedy pass@1 rises with dataset size, and epochs needed to reach the best performance do not increase with corpus scale.Source: §4.2.2 · p. 9
  • After roughly 2 epochs, accuracy becomes more stable and rises as loss decreases; best performance arrives after 3 epochs, with greater stability after 4.The authors infer that about 3–4 epochs capture the dataset’s knowledge, with later steps offering little additional capability.Source: §4.2.2 · p. 9

4.2.3 Case Study

  • PanGu-Coder2 and WizardCoder solve complementary HumanEval problems, and some remain unsolved by all three models even after 200 samples.PanGu-Coder2 solves the most problems, while some are solved only by WizardCoder.Source: §4.2.3 · pp. 9–10, Figure 5 · p. 10
  • Case studies show PanGu-Coder2 succeeding where the other models miss logical aspects, WizardCoder winning some cases, and all three failing on a complex problem.The authors suggest step-by-step comment data may benefit training, while noting that these models still fall short on complex requirements.Source: Figure 6 · p. 10, §4.2.3 · p. 10, Figure 7 · p. 11

4.3 Inference Optimization

The inference optimization experiments compare quantization techniques on GPU memory consumption, speed, and HumanEval performance.

Inference optimization

  • The study evaluates quantization as a way to address GPU memory consumption and inference speed, comparing CTranslate2 and GPTQ.The techniques are evaluated on the same device.Source: §4.3 · p. 12, Table 4 · p. 12

Quantization results

  • Across the tested quantization techniques, memory usage decreases significantly and inference speed increases significantly.The inference setup uses 8-bit or 4-bit quantization, top_p=0.95, temperature=0.2, and max_new_tokens=512.Source: §4.3 · p. 12, Table 4 · p. 12
  • CTranslate2 quantization yields a slight improvement in PanGu-Coder2’s HumanEval performance, which the authors tentatively attribute to model robustness.They leave further study of this result to future work.Source: §4.3 · p. 12

5 Conclusion

The paper introduces RRTF and PanGu-Coder2, training on Evol-Instruct instruction/code pairs and ranking candidate solutions with test-case and heuristic feedback. Evaluations report state-of-the-art performance among billion-parameter-level Code LLMs.

5 Conclusion

  • PanGu-Coder2 achieves new state-of-the-art performance among billion-parameter-level Code LLMs, surpassing existing models by a large margin.Source: §5 · p. 12
  • Evol-Instruct provides substantial high-quality natural-language instruction and code-solution pairs for training.Source: §5 · p. 12
  • Candidate code solutions are ranked using test-case feedback and heuristic preferences.Source: §5 · p. 12
  • Future work will explore combining RRTF with instruction tuning to improve Code LLM performance.Source: §5 · p. 12
Loading 2307.14936v1…