PaperlayerView exact revision on arXiv

Source-linked AI summary

Chain-of-Thought in Neural Code Generation: From and For Lightweight Language Models

Guang Yang, Yu Zhou, Xiang Chen, Xiangyu Zhang, Terry Yue Zhuo, Taolue Chen

arXiv:2312.05562v2cs.SE

TL;DR

Manual CoT writing and reliance on large models make CoT generation costly, while whether lightweight models can generate or benefit from CoTs remains an open question. The paper introduces COTTON to generate code-generation CoTs with lightweight models. Its CoTs improve performance across benchmarks, including CodeT5+ 6B pass@1 rising from 26.83% to 43.90% on HumanEval-plus.

  • Problem

    Existing CoT generation relies heavily on manual writing or large models, and the paper investigates whether lightweight models can generate high-quality CoTs or benefit from them.

    Source: §1 · p. 2

  • Method

    COTTON uses lightweight models to automatically generate CoTs for code generation, with CodeLlama-7b instruction-tuned using LoRA.

    Source: §1 · p. 3

  • Results

    COTTON-generated CoTs improve code-generation performance across ℓLMs; CodeT5+ 6B pass@1 rises from 26.83% to 43.90% on HumanEval-plus.

    Source: §1 · p. 3, §8 · p. 18

  • Takeaways & Limitations

    Existing ℓLMs improve code-generation performance with COTTON-generated CoTs without additional model-parameter updates.

    Source: §8 · p. 18

    The automated CoT-quality evaluation includes term-overlap metrics BLEU, METEOR, and ROUGE-L, creating a construct-validity threat.

    Source: §6.7 · p. 16

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated remarkable potential in code generation. The integration of Chain of Thought (CoT) reasoning can further boost their performance. However, current CoT methods often require manual writing or LLMs with over 100 billion parameters to generate, impeding their applicability in resource-constrained scenarios. In this study, we investigate lightweight Language Models (lLMs), which are defined to have fewer than 10 billion parameters. Empirically, we find that most lLMs cannot generate high-quality CoTs when prompted by the few-shot method, but can take advantage of high-quality CoTs generated elsewhere to improve their performance in code generation. Based on these findings, we design a novel approach COTTON which can leverage lLMs to automatically generate CoTs for code generation. We synthesize new datasets and conduct extensive experiments on various benchmarks. The results show that the CoTs generated by COTTON outperform the baselines in terms of automated and human evaluation metrics. In particular, the CoTs generated by COTTON boost various lLMs to achieve higher performance gains than those generated by LLMs such as ChatGLM (130B), and are competitive with those generated by gpt-3.5-turbo (175B). Our study also showcases the potential of lLMs in software engineering applications.

1 INTRODUCTION

The introduction motivates code-generation methods that can work with lightweight language models (ℓLMs), whose deployment is more accessible than that of large models. It presents COTTON as a way to generate CoTs for code generation and reports gains across models and benchmarks.

1 INTRODUCTION

  • Most tested ℓLMs, ranging from 0.3 to 7 billion parameters, could not independently generate high-quality CoTs; here, “independently” means without training or parameter updates.The study examined 11 ℓLMs and two LLMs using zero-shot and few-shot approaches.Source: §1 · pp. 2–3
  • COTTON uses ChatGPT and multi-agent alignment to build the 9,264-pair CodeCoT-9k dataset; its contributions also include OpenEval, a reusable code-generation benchmark dataset.CodeLlama-7b is fine-tuned with instruction tuning and LoRA, enabling training on one consumer graphics card.Source: §1 · pp. 2–3
  • COTTON-generated CoTs outperform other approaches in automated and human evaluations, and yield ℓLM gains higher than ChatGLM 130B’s and competitive with Gemini and gpt-3.5-turbo’s.Source: §1 · p. 3
  • CodeT5+ 6B pass@1 rises from 26.22% to 42.68% on HumanEval, 26.83% to 43.90% on HumanEval-plus, and 20.22% to 35.39% on OpenEval.These are the reported results when incorporating COTTON-generated CoTs.Source: §1 · p. 3
  • With COTTON-generated CoTs, gpt-3.5-turbo improves code-generation performance and exceeds the GPT-4 zero-shot result on HumanEval.Source: §1 · p. 3
  • StarCoder-7B with COTTON exceeds zero-shot StarCoder-16B and achieves results comparable to fine-tuned StarCoder-16B.The authors note that this approach does not require fine-tuning each model individually.Source: §1 · p. 3

2 PRELIMINARIES

Section 2 formulates code generation as producing a code snippet from a functional description, then explains how generated CoT can augment that description. It also introduces CodeLlama-7B, COTTON’s base model, and outlines its main processing stages.

2.1 Code Generation Task Formulation

  • Code generation predicts a code snippet from a functional description, autoregressively conditioning each token on the description and previously generated tokens.Source: §2.1 · p. 3
  • A CoT model generates C_i from X_i; concatenating them creates an augmented input for the code-generation model.Source: §2.1 · p. 3

2.2 Code Language Model CodeLlama

  • COTTON uses CodeLlama-7B as its base model, whose described processing path applies RMSNorm, Group Query Attention, FFN, and vocabulary projection.Source: §2.2 · p. 4

3 OUR APPROACH

COTTON is organized into data collection, model training, and inference. It builds and cleans a CoT dataset, fine-tunes a lightweight model with LoRA, and generates CoTs for code-generation prompts.

3 OUR APPROACH

3.1 Data Collection

  • CodeCoT-9k draws on TheVault, MBPP, and LeetCode; cleaning filters invalid, mismatched, or leaking code and uses GPT-3.5 agents to check educational value and CoT consistency.Source: §3.1 · pp. 4–5

3.2 Model Training

  • COTTON uses discrete task instructions and LoRA, keeping pretrained weights fixed while training low-rank updates; A starts random and B starts at zero.Source: §3.2 · pp. 5–6

3.3 Model Inference

  • COTTON generates CoTs with greedy decoding on a single consumer graphics card; the authors recommend generating them only after code generation fails to avoid unnecessary overhead.Source: §3.3 · p. 6

4 EXPERIMENTAL SETUP

The experimental setup frames three questions about independent lightweight-model CoT generation, COTTON’s CoT quality, and whether lightweight models benefit from CoT. It evaluates code generation and constructs CoT datasets for these experiments.

4 EXPERIMENTAL SETUP

  • The experiments ask whether ℓLMs independently generate high-quality CoTs, whether COTTON improves CoT quality, and whether ℓLMs benefit from CoT.The first question concerns generation without model training or parameter updates.Source: §4 · p. 6

4.1.1 Code Generation

  • HumanEval and HumanEval-plus use the same 164 Python problems but different test cases, while OpenEval contains 178 AVATAR problems with five manually designed test cases each.Two software engineers with 2–3 years of experience designed OpenEval’s additional test cases.Source: §4.1.1 · p. 6

4.1.2 CoT Generation

  • The CoT datasets contain 9,264 samples, randomly split into 9,000 training and 264 validation samples; HumanEval-CoT and OpenEval-CoT are generated using Section 3.1’s methodology.Agent 2 is used as the Teacher Model; Table 1 reports dataset statistics.Source: §4.1.2 · p. 6, Table 1 · p. 7

4.2 Code Generation Models

The study evaluates several code-generation model families and uses separate metrics for code correctness with and without CoT guidance. It assesses generated CoTs through lexical, structural, and semantic-consistency measures.

4.2 Code Generation Models

  • Experiments use CodeGen models at 350M, 2B, and 6B; StarCoder at 1B, 3B, and 7B; and CodeT5+ at 220M, 770M, 2B, and 6B.The selected families are described as state-of-the-art ℓLMs for code generation.Source: §4.2 · p. 7

4.3.1 Code Generation

  • Pass@1 measures code that passes tests without CoT, while CoT-Pass@1 measures successful code generation after CoT guidance when the initial unguided code fails.Source: §4.3.1 · p. 7

4.3.2 CoT Generation

  • CoT quality is evaluated with BLEU-1 through BLEU-4, METEOR, ROUGE-L, and Agent 3’s consistency check between each CoT and its code snippet.The lexical-overlap measures range from n-gram overlap to longest-common-subsequence similarity; higher automatic-metric values indicate closer matches to ground-truth CoTs.Source: §4.3.2 · p. 7

4.4 Implementation Details and Running Platform

The implementation reports hyperparameter tuning, software libraries, hardware, and model-training time.

4.4 Implementation Details and Running Platform

  • COTTON hyperparameters are tuned according to actual performance, with values listed in Table 2.Source: §4.4 · p. 7, Table 2 · p. 8
  • Implementation uses PyTorch 1.8 and Transformers on an Intel Xeon Silver 4210 CPU with a GeForce RTX 3090 GPU.Source: §4.4 · p. 7
  • COTTON model training took about 6 hours.Source: §4.4 · p. 7

5.1 RQ1: Can ℓLMs generate high-quality CoTs independently?

Most ℓLMs struggle to produce high-quality CoTs independently, while LLMs generally score better on lexical and semantic measures. CoT-generation performance also varies by model family and scale.

5.1 RQ1: Can ℓLMs generate high-quality CoTs independently?

  • The evaluation compared ℓLMs from 0.35B to 7B with CodeLlama and representative LLMs, including InternLM 123B, ChatGLM 130B, Gemini, gpt-3.5-turbo, and gpt-4.Source: §5.1 · p. 8
  • Most ℓLMs scored below 0.3 on METEOR and below 0.6 on Consistency across both datasets, while gpt-4 scored around 0.35 and above 0.96 on HumanEval-CoT, respectively.On OpenEval-CoT, gpt-4 scored around 0.37 on METEOR and above 0.87 on Consistency; the majority of ℓLMs scored below 0.3 and 0.6, respectively.Source: §5.1 · p. 8
  • StarCoder and CodeLlama showed potential for high-quality CoT generation, whereas CodeT5+ performed lowest among the compared decoder-only CodeGen and StarCoder models.The authors suggest natural-language information in StarCoder and CodeLlama pretraining may contribute to better code understanding, and that architecture may explain CodeT5+’s lower performance.Source: §5.1 · p. 8
  • Only StarCoder showed a positive correlation between larger parameter scale and better performance; CodeGen and CodeT5+ did not strictly follow scale, with only their 2B versions improving.Source: §5.1 · p. 8

5.2 RQ2: Can COTTON generate higher-quality CoTs

COTTON generates CoTs that outperform the compared base models on automatic evaluations and human-rated quality. The section evaluates both similarity metrics and whether the CoTs are useful, natural, and educational.

5.2.1 Automatic Evaluation.

  • On HumanEval-CoT, COTTON led the base models with METEOR 0.38 and semantic similarity 0.93, and also performed better on OpenEval-CoT.It surpassed larger LLMs including InternLM 123B, ChatGLM 130B, and Gemini.Source: §5.2.1 · p. 9, Table 4 · p. 10

5.2.2 Human Evaluation.

  • The human study rated CoTs for similarity, naturalness, and educational value, using two assessors per group and scores averaged across assessors.Six postgraduate assessors evaluated samples from HumanEval-CoT and OpenEval-CoT.Source: §5.2.2 · pp. 10–11
  • COTTON outperformed Llama2 and CodeGeeX2 in similarity, naturalness, and educational value, with statistically significant differences and substantial assessor agreement.The overall Fleiss Kappa was 0.7.Source: Table 5 · p. 10, Table 6 · p. 10, §5.2.2 · p. 11

5.3 RQ3: Can ℓLMs benefit from the generated CoT?

Self-generated CoTs offer ℓLMs little or inconsistent benefit, while CoTs from stronger sources can improve their code generation. COTTON-generated CoTs outperform ChatGLM 130B and closely rival Gemini and GPT-3.5-turbo in CoT generation.

5.3.1 Effect of self-generated CoTs on ℓLMs.

  • Self-CoT produced very limited code-generation gains for ℓLMs, and performance sometimes decreased.This supports the finding that ℓLMs do not independently generate high-quality CoTs.Source: §5.3.1 · p. 12, Table 8 · p. 12

5.3.2 Effect of CoTs generated by different LMs on ℓLMs.

  • All tested ℓLMs could use CoTs to improve generated-code quality when the CoTs' quality was ensured.COTTON-generated CoTs outperformed ChatGLM 130B and closely rivaled Gemini and GPT-3.5-turbo in CoT generation.Source: §5.3.2 · p. 12

5.3.3 Effect of CoTs generated by different base models on ℓLMs.

  • Using CodeLlama as COTTON's base model produced greater code-generation improvements for most ℓLMs than the other tested base models.The comparison evaluates CoTs generated using different base models.Source: §5.3.3 · p. 12, Table 9 · p. 13

6 DISCUSSIONS

The discussion tests COTTON across model settings and design choices, including code-generation performance, data filtering, decoding, prompting, and LoRA parameters. It also identifies dataset and evaluation choices as threats to the empirical study's validity.

6.1 Evaluating COTTON on LLMs

  • Guiding gpt-3.5-turbo with CoTs improved Pass@1 over zero-shot and exceeded GPT4's zero-shot HumanEval result of 67.0.Self-CoT outperformed Self-planning and SCoT on all datasets.Source: §6.1 · p. 13, Table 10 · p. 13

6.2 Ablation study

  • Removing the Consistency Checker reduced COTTON's Consistency by 5.23% on HumanEval-CoT and 4.69% on OpenEval-CoT.The comparison used CodeCoT-9k and a 13k-example dataset without Consistency Checker filtering.Source: §6.2 · p. 14, Figure 5 · p. 14

6.3 Comparing COTTON with other tune methods

  • StarCoder-7B combined with COTTON exceeded zero-shot StarCoder-16B and reached performance comparable to fine-tuned or instruction-tuned StarCoder-16B.COTTON can also combine with fine-tuned or instruction-tuned models to further improve code generation.Source: §6.3 · p. 14, Table 11 · p. 14

6.4 The impact of different decoding strategies

  • COTTON's performance varied by no more than 0.01 in BLEU-4, METEOR, and Rouge-L across decoding strategies.The authors report that its effectiveness does not heavily rely on the specific decoding strategy.Source: §6.4 · p. 15, Figure 6 · p. 15

6.5 The impact of different prompt methods

  • The proposed prompt template achieved the highest scores on both datasets, while Prefix Tuning led the other prompt methods.Prompt Tuning and Alpaca Prompt also showed competitive performance.Source: §6.5 · p. 15, Table 13 · p. 16

6.6 The impact of different hyper-parameter setting

  • Higher LoRA Alpha tended to improve performance, while changing R had relatively little effect on BLEU-4 and METEOR.The comparison covered HumanEval-CoT and OpenEval-CoT.Source: §6.6 · p. 16, Table 14 · p. 16

6.7 Threats to Validity

  • The authors identify dataset selection and construction as an external-validity threat, and text-overlap metrics and Pass@1-only evaluation as construct-validity constraints.They also report using a human study to complement automated evaluation.Source: §6.7 · p. 16

7 RELATED WORK

This section reviews neural code-generation approaches, CoT-generation methods, and multi-agent collaboration relevant to COTTON.

7.1 Code Generation

  • Earlier code generation progressed from heuristic and statistical methods to neural networks and Transformers, which still require substantial labeled language-code training data.The reviewed approaches include probabilistic grammars, n-gram and Hidden Markov models, CNNs, RNNs, and LSTMs.Source: §7.1 · p. 16
  • Pretrained code-generation models shifted from sub-1B systems toward larger zero-shot LLMs, while fine-tuning LLMs remains computationally expensive and resource-intensive.The paper focuses on using ℓLMs for code generation without updating their parameters.Source: §7.1 · pp. 16–17

7.2 Chain of Thought Generation

  • CoT-generation research includes external-knowledge support, self-consistency, selection-inference, and least-to-most prompting.These approaches aim to improve CoT generation or solve complex problems through sequential subproblems.Source: §7.2 · p. 17
  • Prior CoT distillation methods rely on LLMs with more than 100 billion parameters, whereas this work targets a stand-alone, relatively small code-CoT generator.The stated objective is to generate high-quality code-generation CoTs at a manageable cost.Source: §7.2 · p. 17

7.3 Multi-agent Collaboration

  • Unlike code-generation agents organized around programming and test execution [115], COTTON's three agents filter code quality, generate CoTs, and check semantic consistency.The agents are the Quality Checker, CoT Generator, and Consistency Checker.Source: §7.3 · p. 17

8 CONCLUSION

The conclusion presents CodeCoT-9k and COTTON for generating code-generation CoTs with lightweight models, and identifies extensions to other programming languages and software-engineering tasks as future work.

8 CONCLUSION

  • CodeCoT-9k and COTTON use sub-10B lightweight models to generate code-generation CoTs, with the authors reporting effective and efficient generation.The paper reports significant code-generation performance improvements when existing ℓLMs use these CoTs.Source: §8 · p. 18
  • Using these CoTs improves existing ℓLMs' code-generation performance without parameter updates, making them more accessible to individual users.The conclusion also identifies lightweight models as promising for other software-engineering tasks.Source: §8 · p. 18
  • Future work includes extending COTTON to other programming languages and exploring retrieval-augmented generation, adversarial training, and contrastive learning.The authors also propose exploring lightweight language models for other software-engineering tasks.Source: §8 · p. 18

Related papers

Loading 2312.05562v2…