PaperlayerView exact revision on arXiv

Source-linked AI summary

Coder Reviewer Reranking for Code Generation

Tianyi Zhang, Tao Yu, Tatsunori B. Hashimoto, Mike Lewis, Wen-tau Yih, Daniel Fried, Sida I. Wang

arXiv:2211.16490v1cs.LGcs.CLcs.PLcs.SE

TL;DR

Coder-only likelihood reranking can prefer degenerate code solutions. The paper introduces Coder-Reviewer reranking, which combines coder and reviewer likelihoods, and finds consistent gains over Coder-only reranking, including up to 17% absolute accuracy improvement and frequent advantages over MBR-EXEC with executability filtering.

  • Problem

    Coder-only reranking can mistakenly prefer degenerate solutions such as extremely short or repetitive programs, with performance sometimes declining as candidate count increases.

    Source: §1 · p. 1

  • Method

    Coder-Reviewer reranking samples programs with a Coder model and selects them using the product p(x|y)p(y|x), where a prompted Reviewer evaluates the instruction given each program.

    Source: §1 · pp. 1–2

  • Results

    Coder-Reviewer reranking consistently improves over Coder-only reranking, with gains up to 17% absolute accuracy and frequent superiority over MBR-EXEC when combined with executability filtering.

    Source: §1 · p. 2, §7.1 · p. 7

  • Takeaways & Limitations

    Prompted Coder-Reviewer reranking is a broadly applicable alternative for code generation across datasets, programming languages, and model families.

    Source: §1 · p. 2, §7.1 · p. 7, §8 · p. 8

    Aggregating plots with different styles, such as colors and fonts, remains difficult even when they may all be correct.

    Source: §6.4 · p. 6

Abstract

from arXiv · show

Sampling diverse programs from a code language model and reranking with model likelihood is a popular method for code generation but it is prone to preferring degenerate solutions. Inspired by collaborative programming, we propose Coder-Reviewer reranking. We augment Coder language models from past work, which generate programs given language instructions, with Reviewer models, which evaluate the likelihood of the instruction given the generated programs. We perform an extensive study across six datasets with eight models from three model families. Experimental results show that Coder-Reviewer reranking leads to consistent and significant improvement (up to 17% absolute accuracy gain) over reranking with the Coder model only. When combined with executability filtering, Coder-Reviewer reranking can often outperform the minimum Bayes risk method. Coder-Reviewer reranking is easy to implement by prompting, can generalize to different programming languages, and works well with off-the-shelf hyperparameters.

1. Introduction

Coder-only likelihood reranking can favor degenerate programs, especially as candidate pools grow. Coder-Reviewer reranking addresses this by combining coder likelihood with a prompted Reviewer’s likelihood of the instruction given each program, improving results across diverse settings.

1. Introduction

  • Coder-only reranking can mistakenly select extremely short or repetitive programs, with performance often declining as the number of candidates increases.These failures motivate using a second model to check generated programs against the instruction.Source: §1 · p. 1
  • Coder-Reviewer reranking samples programs with p(y|x), evaluates instructions with p(x|y), and ranks candidates using their product p(x|y)p(y|x).The method is framed as collaborative code review between a generating Coder and a checking Reviewer.Source: §1 · p. 1, Figure 1 · p. 1
  • The Reviewer is implemented by reversing the instruction and generated-program order in a prompt, requiring no additional training and supporting different programming languages.The generated program is placed before the instruction when estimating p(x|y).Source: §1 · p. 2
  • Up to 17% absolute accuracy gain is reported for Coder-Reviewer reranking over Coder-only reranking across six datasets, three programming languages, and seven models from three families.With executability filtering, it can often outperform minimum Bayes risk decoding.Source: §1 · p. 2

2. Related Work

The paper builds on pretrained code-generation models, reranking methods, and MMI-based objectives. It differs by obtaining the Reviewer through prompting and applying the resulting objective to code generation without requiring execution.

2. Related Work

  • Pretrained code models support zero- and few-shot generation, while this work adds gains by using a Reviewer to evaluate the instruction probability given generated programs.The approach builds on established neural code-generation models and benchmarks.Source: §2 · p. 2
  • Coder-Reviewer reranking is a prompted, code-generation-specific instantiation of weighted Maximum Mutual Information rather than a separately trained Reviewer model.It uses p(x|y) together with p(y|x) and differs from prior MMI work through prompting and application to code generation.Source: §2 · p. 2
  • Unlike execution-based or supervised rerankers, Coder-Reviewer does not require execution, is not tied to a programming language or package, and can complement execution-based ranking.The paper reports empirical benefits from combining it with execution-based methods.Source: §2 · p. 2

3. Background

The background defines code generation and sampling, then shows why Coder-only likelihood is an inadequate reranking objective. Length effects and degenerate high-scoring programs cause accuracy to worsen as more candidates are sampled.

3. Background

  • The task generates programs y from natural-language instructions x with code context c, optionally using demonstration examples for few-shot generation.The Coder conditions on the instruction, context, and, in few-shot settings, demonstration inputs and outputs.Source: §3 · p. 2
  • Programs are sampled autoregressively from a temperature-scaled Coder distribution, where lower temperature increases the probability of well-formed programs.The method collects multiple candidate programs before reranking.Source: §3 · pp. 2–3
  • Coder-only reranking selects the highest-likelihood program, while length normalization instead uses average token-level log likelihood to counter its bias toward short programs.Prior work collects 25–100 samples and applies either unnormalized or normalized Coder likelihood.Source: §3 · p. 3
  • Coder likelihood prefers short solutions, whereas normalization can prefer long solutions, including trivial or repetitive programs.The two objectives therefore introduce different length-related biases.Source: §3 · p. 3
  • Accuracy degrades as more programs are sampled because degenerate but high-scoring solutions become increasingly likely to appear.This finding indicates that both ordinary and length-normalized Coder likelihood are insufficient reranking objectives.Source: §3 · p. 3, Figure 2 · p. 3

4. Coder-Reviewer Reranking

Coder-Reviewer reranking combines a Coder’s program likelihood with a Reviewer’s instruction likelihood, implemented by reversing prompt order without additional training. The method is a special case of MMI that seeks consensus and can reduce component-specific biases.

Prompting for the Reviewer Model

  • The Reviewer estimates p(x|y) by placing the generated program before the instruction and querying the pretrained model again.The prompting procedure uses the same underlying model with a different prompt and requires no additional training.Source: §4 · pp. 3–4

Combining Coder and Reviewer

  • Coder-Reviewer reranking selects programs using the product p(x|y)p(y|x), combining instruction fit with generation likelihood.The product makes the method sensitive to low probability under either model, favoring consensus between Coder and Reviewer.Source: §4 · p. 4

Relation to Maximum Mutual Information

  • Coder-Reviewer reranking is a special case of the MMI objective with α set to 0.5.This formulation interpolates likelihood with mutual information and favors programs that have high mutual information with the instruction.Source: Understanding the Relation between… · p. 4

Analysis of Degenerate Cases

  • Figure 4 reports Mean Reciprocal Rank for three degenerate cases and shows that Coder-Reviewer variants alleviate biases found in individual components.Coder, normalized Coder, and Reviewer methods each favor different degenerate patterns, whereas the combined method reduces these biases.Source: Figure 4 · p. 4

Relation to Maximum Mutual Information

  • The off-the-shelf hyperparameter α = 0.5 usually works well, while tuning α provides a small additional gain.The weighting parameter controls the mixing ratio between Coder and Reviewer scores.Source: Understanding the Relation between… · p. 4

5. Analysis of Degenerate Cases

The degenerate-case analysis tests how ranking methods handle artificially inserted empty, repetitive, and instruction-copying programs. Coder-Reviewer reduces the distinct biases exhibited by Coder, normalized Coder, and Reviewer ranking.

Experimental Design

  • The analysis inserts ReturnOnly, Repetitive, and CopyPrompt programs among sampled MBPP candidates to measure ranking bias.The experiment uses Codex002 on the zero-shot sanitized MBPP dataset and compares Coder, normalized Coder, Reviewer, and combined variants.Source: §5 · p. 4

Degenerate Cases

  • Coder-Reviewer reranking alleviates the component-specific biases observed across three artificially constructed degenerate program cases.Coder favors effectively empty programs, normalized Coder favors repetitions, and Reviewer can favor spurious surface overlap.Source: §5 · pp. 4–5

Degenerate Solutions Rejection

  • The study filters empty, trivial, repetitive, and spurious-surface-overlap programs before reranking to strengthen all compared methods.Filtering includes return or pass-only solutions, zlib-based repetition detection, and Python canonicalization with pyminifier.Source: §5 · p. 5
  • These procedures improve both baseline Coder methods and the proposed Coder-Reviewer methods.The paper analyzes their effects separately before presenting the broader empirical study.Source: §5 · p. 5

6. Experimental Setup

The evaluation spans six datasets, three programming languages, three model families, and zero-shot and few-shot settings. It uses fixed decoding choices, degenerate-solution rejection, executability filtering, and comparisons with Coder, Reviewer, Coder-Reviewer, MBR-EXEC, and CodeT.

6. Experimental Setup

  • The study covers six datasets, zero-shot and few-shot settings, task-specific and task-agnostic prompts, and eight models spanning two orders of magnitude in size.The datasets include Python function completion, plotting, MBPP, Spider, and NL2Bash settings.Source: §6 · p. 5, §6.1 · p. 5

6.1. Datasets

  • The datasets include three zero-shot Python benchmarks and three three-shot benchmarks covering programming languages beyond Python.The authors leave infilling-model applications to future work because much of DS-1000 is not easily handled by left-to-right generation.Source: §6.1 · p. 5

6.2. Models

  • Experiments use Codex, InCoder, and CodeGen families, including models from 1B to 175B parameters.InCoder is evaluated only with autoregressive generation, while CodeGen models include 2B, 6B, and 16B variants.Source: §6.2 · p. 5

6.3. Implementation Details

  • The main evaluation samples 125 programs per problem, reranks 25 samples, bootstraps 50 times, and uses temperature 0.4 with a 300-token limit.Because the benchmarks lack validation splits, the study avoids hyperparameter search and uses one decoding configuration.Source: §6.3 · p. 5
  • Figure 5 reports accuracy differences between the best Coder-Reviewer and Coder variants, with gains on 43 of 48 dataset-model pairs.The comparison includes variants with and without length normalization.Source: Figure 5 · p. 6
  • Executability filtering removes programs with runtime errors before applying the ranking methods.The paper applies this filtering to baseline and proposed methods for comparison with execution-heavy state-of-the-art approaches.Source: §6.3 · p. 6

6.4. Reranking Methods

  • The comparison includes Coder, normalized Coder, Reviewer, Coder-Reviewer, normalized Coder-Reviewer, MBR-EXEC, and CodeT.MBR-EXEC compares executed outputs, while CodeT generates assertions and uses graph-based program-test agreement; CodeT is reported separately because it is less portable across tasks and languages.Source: §6.4 · p. 6

7. Experimental Results

Across diverse datasets and models, Coder-Reviewer reranking generally improves over Coder-only ranking, remains robust across sample counts, and benefits from degenerate-solution rejection.

7.1. Primary Results

  • Coder-Reviewer yields significant gains with few exceptions, including more than 10% improvement for InCoder models on Plotting.Only 5 of 48 model-dataset pairs are worse than Coder, all involving CodeGen models.Source: §7.1 · pp. 6–7
  • Coder-Reviewer variants consistently improve over Coder variants and are often the best or second-best methods across benchmark settings.Across all data, Coder-Reviewer variants are best 62.5% of the time; Table 1 reports them as best or second best across all but one column.Source: §7.1 · p. 7, Table 1 · p. 7
  • With executability filtering, Coder-Reviewer can outperform MBR-EXEC, while MBR-EXEC depends more strongly on high-quality test inputs and easy output aggregation.MBR-EXEC is reported as more than 10% worse on HumanEval and 8% worse on Plotting for Codex002 than normalized Coder-Reviewer.Source: §7.1 · p. 7, Table 2 · p. 8

7.2. Understanding Hyperparameters

  • Coder-Reviewer remains more stable as the number of ranking samples increases, suggesting greater robustness to degenerate solutions than either component alone.The method fixes the Coder–Reviewer mixing ratio at α = 0.5; tuning α typically improves results by less than 1%.Source: §7.2 · p. 8, Figure 6 · p. 8

7.3. Understanding Degenerate Solution Rejection

  • Degenerate-solution rejection improves all evaluated ranking methods, has its largest effect on Coder, and still leaves an additional benefit for Coder-Reviewer.The rejection is especially important for normalized methods on NL2Bash, where egregious repetitions are more likely.Source: §7.3 · p. 8, Table 3 · p. 8

8. Conclusion

The paper proposes Coder-Reviewer reranking as a practical alternative to Coder-only ranking, combining prompting-based review with executability filtering and default hyperparameters.

8. Conclusion

  • Coder-Reviewer improves consistently over Coder-only reranking, can often outperform MBR-EXEC with executability filtering, and generalizes across programming languages.The method is implemented through prompting and works with off-the-shelf hyperparameters.Source: §8 · p. 8

B.1. Detailed Dataset Descriptions

The evaluated datasets cover Python function completion, SQL generation, Bash translation, and additional program-generation settings with distinct input formats and metrics.

B.1. Detailed Dataset Descriptions

  • HumanEval and MBPP-Sanitized evaluate Python programming questions presented as function headers with natural-language docstrings.HumanEval contains 164 questions, while MBPP-Sanitized contains 427 crowdsourced questions.Source: §B.1 · p. 10
  • MBPP uses 500 test questions from a 974-question corpus in a 3-shot format with an input-output assertion and model-defined function names.This differs from the 0-shot sanitized MBPP format.Source: §B.1 · p. 10
  • Spider evaluates natural-language-to-SQL generation using execution accuracy against database return values.Its prompt prepends the database schema as program context.Source: §B.1 · p. 10
  • NL2Bash evaluates translation from natural language to Bash commands with character-level BLEU-4 because executable environments are difficult to obtain.Source: §B.1 · p. 10
  • The appendix includes additional accuracy-versus-sample and mixing-ratio analyses, with α = 0.5 usually performing well and tuning sometimes improving results by 1–2%.The larger gains from tuning occur in a few settings, including CodeGen-16B on NL2Bash and Incoder-6B on Plotting.Source: Figure 8 · p. 11, Figure 7 · p. 11

C.1. Results from All Models

Across all eight evaluated models, Coder-Reviewer is usually the strongest ranking approach, and executability filtering generally improves methods without changing their relative comparison.

C.1. Results from All Models

  • Across all data, Coder-Reviewer variants are best 62.5% of the time, compared with 12.5% for Reviewer and 20.83% for MBR-EXEC.A Coder-Reviewer variant is almost always best for Codex models and is usually second best when it is not first.Source: §C.1 · p. 11

C.2. Analyzing the Effect of Number of Program Samples

Coder-Reviewer variants remain more stable as the number of reranked program samples grows, while tuning the mixing ratio usually adds little and an alternate mutual-information formulation generally underperforms.

C.2. Analyzing the Effect of Number of Program Samples

  • Coder-Reviewer variants remain more consistent across sample counts, whereas Coder performance degrades after five or ten samples.Normalized Coder is more stable on HumanEval and MBPP-S but also worsens with more samples on Plotting.Source: §C.2 · p. 11

C.3. Analyzing the Effect of Tuning Mixing Ratio α

  • Tuning α usually improves performance by less than 1%, with larger gains only for Incoder-6B on Plotting and CodeGen-16B on NL2Bash.Because most benchmarks lack validation splits, these tuned results are excluded from the main results; practitioners with validation data can grid-search α.Source: §C.3 · pp. 11–12

C.4. Alternate Formulation

  • The alternate maximum-mutual-information formulation generally underperforms Coder-Reviewer on 0-shot Python function-completion datasets.The alternate formulation estimates p(y) by removing the docstring and applies length normalization to its regularization term.Source: §C.4 · p. 12, Table 4 · p. 12
  • Across additional datasets, Coder-Reviewer is usually strongest on Spider, Codex HumanEval and MBPP-S, while MBR-EXEC often leads on CodeGen and Incoder variants.On NL2Bash, simulated executability filtering does not improve performance consistently.Source: Table 6 · p. 13, Table 7 · p. 14, Table 8 · p. 14
Loading 2211.16490v1…