PaperlayerView exact revision on arXiv

Source-linked AI summary

Better patching using LLM prompting, via Self-Consistency

Toufique Ahmed, Premkumar Devanbu

arXiv:2306.00108v2cs.SEcs.LG

TL;DR

Software-engineering datasets often lack the reasoning paths needed for chain-of-thought and self-consistency prompting. This paper uses commit-log messages as reasoning paths in few-shot examples for program repair, finding that self-consistency improves over greedy decoding and that correct messages matter. The authors report higher accuracy than NatGen on both datasets, while noting that their results use a sample rather than the full test set.

  • Problem

    Software-engineering datasets often provide input-output pairs but lack the reasoning paths required for chain-of-thought prompting, which can be challenging to create afterward.

    Source: B. Chain-of-Thought (CoT) · p. 2

  • Method

    The paper uses commit-log messages as reasoning paths in few-shot examples and applies self-consistency by sampling candidate repairs and selecting the most frequent output.

    Source: §VI · p. 4

  • Results

    With BM25-selected few-shots, self-consistency improves over greedy decoding (p < 0.01), with relative gains of 9.65% and 13.08%.

    Source: A. RQ1: Performance of the proposed… · p. 4

  • Takeaways & Limitations

    Correct few-shot commit messages yield statistically significant improvement, while random messages do not provide significant improvement over prior approaches.

    Source: C. RQ3: Random commit message vs.… · p. 4

    MODIT commit messages are sometimes uninformative, such as “Bug Fixed,” and better-quality messages could potentially improve performance.

    Source: §V · p. 4

Abstract

from arXiv · show

Large Language models (LLMs) can be induced to solve non-trivial problems with "few-shot" prompts including illustrative problem-solution examples. Now if the few-shots also include "chain of thought" (CoT) explanations, which are of the form problem-explanation-solution, LLMs will generate a "explained" solution, and perform even better. Recently an exciting, substantially better technique, self-consistency [1] (S-C) has emerged, based on the intuition that there are many plausible explanations for the right solution; when the LLM is sampled repeatedly to generate a pool of explanation-solution pairs, for a given problem, the most frequently occurring solutions in the pool (ignoring the explanations) tend to be even more likely to be correct! Unfortunately, the use of this highly-performant S-C (or even CoT) approach in software engineering settings is hampered by the lack of explanations; most software datasets lack explanations. In this paper, we describe an application of the S-C approach to program repair, using the commit log on the fix as the explanation, only in the illustrative few-shots. We achieve state-of-the art results, beating previous approaches to prompting-based program repair, on the MODIT dataset; we also find evidence suggesting that the correct commit messages are helping the LLM learn to produce better patches.

I. INTRODUCTION

Language models generate prompt-conditioned text by repeatedly predicting tokens, while few-shot and chain-of-thought prompts guide their outputs with examples and explanations. The paper applies self-consistency to program repair using commit-log messages as explanations in few-shot examples, and reports evidence that correct messages help produce better patches.

I. INTRODUCTION

  • Language models generate prompt-conditioned token sequences by repeatedly applying a learned next-token distribution.The paper describes this process as autoregressive generation.Source: §I · p. 1
  • Few-shot prompts provide task examples, while chain-of-thought prompts add reasoning between each input and output.The paper treats few-shot learning and chain-of-thought as complementary approaches.Source: §I · p. 1
  • Self-consistency repeatedly samples explanations and answers, then selects the most frequent answer while ignoring the explanations.The paper introduces self-consistency as a recently proposed method [1].Source: §I · p. 1
  • The paper finds evidence that correct few-shot commit messages help self-consistency improve code-repair performance, whereas random messages do not.The messages serve as reasoning paths in the illustrative examples.Source: §I · p. 1

A. Few-shot Learning

Few-shot learning uses examples in prompts to elicit task outputs without weight updates, and has been applied to software-engineering tasks. Chain-of-thought prompting adds reasoning paths to those examples, but such paths are rarely available in software datasets; the paper proposes commit messages as a substitute.

A. Few-shot Learning

  • Few-shot prompting uses a small set of input-output examples to elicit a model response without updating model weights.The paper notes its use across software-engineering tasks, including code repair, summarization, and test generation.Source: A. Few-shot Learning · p. 2
  • BM25 retrieval selects buggy-fixed function examples for few-shot repair prompts, improving performance in code repair and assertion generation, with similar gains reported for summarization [9].The retrieved examples are concatenated with the target buggy function, and the model generates the fixed function.Source: A. Few-shot Learning · p. 2

B. Chain-of-Thought (CoT)

  • Chain-of-thought examples add an intermediate reasoning path to each input-output pair, and have improved performance on complex reasoning tasks [3].The model generates both the reasoning path and output after receiving several input-reasoning-output triplets.Source: B. Chain-of-Thought (CoT) · p. 2
  • Software-engineering datasets commonly contain input-output pairs but rarely reasoning paths, making after-the-fact creation of CoT examples challenging.Source: B. Chain-of-Thought (CoT) · p. 2
  • The paper proposes using summaries or commit messages as reasoning paths; unlike the described MODIT setup, S-C must generate the target's reasoning path before its output.MODIT inserted commit messages associated with the target input into its prompt.Source: B. Chain-of-Thought (CoT) · p. 2

C. Self-Consistency (S-C)

Self-consistency samples multiple reasoning paths and outputs, then selects the most frequent output regardless of its reasoning path. The paper applies this approach to program repair using commit messages as reasoning paths in few-shot examples.

C. Self-Consistency (S-C)

  • Self-consistency selects the most frequent output across sampled reasoning-path-and-output pairs, rather than relying on one greedy CoT generation.The approach follows the intuition that distinct reasoning paths may reach the same correct answer.Source: C. Self-Consistency (S-C) · pp. 2–3
  • The study generates up to 50 samples per input and uses majority voting to determine the final output.Source: C. Self-Consistency (S-C) · p. 3
  • For program repair, the paper uses commit messages as few-shot reasoning paths because software-engineering datasets rarely provide reasoning paths.The target's reasoning path must also be generated before the required output for S-C.Source: C. Self-Consistency (S-C) · p. 3

D. Research Questions (RQs)

The research questions examine whether self-consistency improves program repair, how performance changes with the number of generated reasoning paths, and whether original commit messages matter.

D. Research Questions (RQs)

  • RQ1 tests whether self-consistency combined with BM25 few-shot retrieval improves repair of complete buggy functions without bug localization.BM25-based few-shot retrieval is described as the high-water mark in this setting.Source: D. Research Questions (RQs) · p. 3
  • RQ2 examines performance as the number of generated reasoning paths varies, balancing possible consistency gains against linearly increasing computational costs.Source: D. Research Questions (RQs) · p. 3
  • RQ3 compares original fix-associated commit messages in few-shot examples with random messages to test whether message relevance matters to performance.The authors expect unrelated reasoning paths to confuse the model and diminish performance if commit messages are useful.Source: D. Research Questions (RQs) · p. 3

A. Dataset

The study uses MODIT bug-fix commits and developer commit messages to examine whether commit messages can serve as chain-of-thought for self-consistency in program repair. It focuses on Code-DaVinci-002 rather than a comprehensive comparison of models and datasets.

A. Dataset

  • MODIT contains GitHub bug-fix commits and commit logs; the experiments use BM25 to retrieve few-shot examples from training data and randomly sample 1,000 test examples.MODIT has two subsets, B2Fs with smaller sequences and B2Fm.Source: A. Dataset · p. 3
  • A comprehensive comparison across program-repair models and datasets is outside the paper’s scope; its focus is whether commit messages can serve as chain-of-thought for self-consistency.Source: A. Dataset · p. 3

B. Model

  • The experiments primarily use Code-DaVinci-002, a 175-billion-parameter model accessed through the OpenAI API.Source: B. Model · p. 3

C. Proposed Approach, Baseline & evaluation criteria

The evaluation compares greedy decoding, chain-of-thought, and self-consistency using BM25-retrieved few-shots. It measures exact-match repair performance and reports self-consistency results using 30 samples.

C. Proposed Approach, Baseline & evaluation criteria

A. RQ1: Performance of the proposed approached

Self-consistency improves exact-match repair performance over greedy decoding, including with BM25-retrieved few-shots. The study also examines sample counts and whether original commit messages outperform random ones.

A. RQ1: Performance of the proposed approached

  • With BM25-selected few-shots, self-consistency gains were 9.65% and 13.08% over greedy decoding; without BM25, gains were 42.10% and 38.39%, with all four improvements significant at p < 0.01.Chain-of-thought sometimes improved performance, but not significantly.Source: A. RQ1: Performance of the proposed… · p. 4

B. RQ2:How many samples are needed for S-C?

  • Accuracy improved through 10 self-consistency samples on both datasets; beyond 10, B2Fm showed no significant benefit while B2Fs continued improving with tapering gains.The study reports results using 30 samples; processing took less than five seconds per sample, excluding rate limitations.Source: B. RQ2:How many samples are needed for… · p. 4

C. RQ3: Random commit message vs. original ones

  • Random commit messages yielded 3.44% and 4.18% improvements over baseline without statistical significance, unlike the significant improvement with original messages.The authors suggest the original messages may help the LLM learn better reasoning paths for program repair.Source: C. RQ3: Random commit message vs.… · p. 4

V. DISCUSSION

The discussion compares the results with prior repair approaches and notes limits related to evaluation scope and commit-message quality. The conclusion reports improved performance using self-consistency with commit messages in few-shot examples.

V. DISCUSSION

  • On B2Fs and B2Fm, the paper reports 31.80% and 21.60% accuracy, versus NatGen’s 23.43% and 14.93%, respectively.NatGen included the test example’s commit message; this study generated and discarded it, and evaluated a random 1,000-example sample rather than the full test set.Source: §V · p. 4
  • Other studies report higher performance, but assume known bug locations and generate fix snippets; this study supplies whole buggy functions and outputs complete fixed functions.Source: §V · p. 4
  • MODIT commit messages are sometimes uninformative, such as “Bug Fixed”; better-quality messages could potentially improve chain-of-thought and self-consistency performance.The paper identifies generating informative messages with other models as a direction requiring further research.Source: §V · p. 4

VI. CONCLUSION

  • The conclusion reports that around 30 explanation-answer samples and majority voting work well, while random commit messages provide no significant improvement over prior approaches.Source: §VI · p. 4
Loading 2306.00108v2…