Fault-Aware Neural Code Rankers

1 Introduction

• On APPS, Codex achieves 4% pass@1 versus 24% pass@100, motivating methods to rank samples toward the larger-sample performance.Source: §1 · p. 1
• Execution-based ranking relies on unit tests and installed dependencies, can miss failures on unknown tests, and may require isolation for unsafe code.Execution can also be infeasible for incomplete code under active development.Source: §1 · pp. 1–2
• Earlier neural rankers predict program correctness from the task and program, using executed samples for training rather than execution during inference.Source: §1 · p. 2
• CODERANKER trains classifiers on execution-derived multiclass labels to distinguish failure modes in general programming, where errors and output types vary.Source: §1 · p. 2
• CODERANKER raises Codex pass@1 from 26% to 39.6% on APPS validation and from 3.8% to 4.5% on APPS test, with gains also reported on HumanEval and MBPP.The paper also reports similar boosts for GPT-J and GPT-Neo.Source: §1 · p. 2

2.1 Code generation

• A code-generation task G is a token-sequence prompt; datasets also provide input-output pairs for testing generated solutions.Source: §2.1 · pp. 2–3
• The study samples programs from four models: Codex, GPT-J, GPT-Neo 1.3B, and GPT-Neo 125M.The models differ in scale and training setup.Source: §2.1 · p. 3
• pass@k measures the fraction of tasks solved by any of k samples, while exec@k measures tasks with at least one non-error execution.Source: §2.1 · p. 3

2.2 Code ranking

• CODERANKER scores each sampled program for a task and orders programs by descending score to define ranked pass@k.Source: §2.2 · p. 3

3 Fault-Aware Neural Code Ranker

• The ranker scores a task-program pair by the predicted probability that the program satisfies the task with respect to its unit tests.The score is extracted from the real-valued outputs of the layer before SoftMax.Source: §3 · p. 3

3.1 Code Ranker Dataset

• Training data comes from 100 sampled programs per training task, executed on corresponding unit tests; WRONG examples outnumber CORRECT ones by about 5X to 40X.Base generators are finetuned for at most two epochs to preserve sample diversity; Codex is used few-shot.Source: §3.1 · p. 4
• The dataset distinguishes execution errors from intent errors, then defines 10 frequent execution-error classes and 9 manually designed intent-error classes.Execution errors are identified from compiler messages, while intent-error classes are based on expected and generated outputs.Source: §3.1 · pp. 4–5

3.2 Code Ranker Tasks

• Five classification tasks range from binary CORRECT/WRONG labels to fine-grained intent or execution-error classes, with E+L also predicting the erroneous line.Ternary labels distinguish correct programs, intent errors, and execution errors; E+L includes a no-execution-error line label.Source: §3.2 · p. 5

3.3 Code Ranker Models

• The models fine-tune pretrained CodeBERT, adding a classification head and, for E+L, a head that scores newline tokens as potential error lines.The line head includes boundary newline tokens for no error and for an erroneous line beyond the encoded 512-token input.Source: §3.3 · pp. 5–6

4 Evaluation

• The evaluation investigates gains across code-generation models and datasets, the effects of ranker tasks, and the effects of mixing ranker datasets.Experiments use APPS for finetuning and evaluate on APPS, HumanEval, and MBPP.Source: §4 · pp. 6–7, §4.1 · p. 7

4.2 Main Results: CODERANKER improves code generation models

• On APPS validation (600 tasks), CODERANKER increases pass@1 by 5.1% to 13.6%; on APPS test, Codex pass@1 rises from 3.8% to 4.7%, and Codex gains 6% on HumanEval and MBPP.HumanEval and MBPP evaluations are zero-shot transfers using rankers trained on APPS.Source: Table 7 · p. 6, §4.2 · pp. 7–8, Table 8 · p. 8
• CODERANKER improves all metrics for all four models on APPS validation; GPT-Neo 125M with a 125M ranker also beats GPT-J, which has 50 times more parameters.The validation set contains 600 tasks; the reported pass@1 increase ranges from 5.1% to 13.6%.Source: §4.2 · p. 7
• On HumanEval and MBPP, gains occur across models and metrics except GPT-J pass@5 on MBPP; Codex pass@1 rises by 6% on each dataset.These results measure zero-shot transfer of rankers trained on APPS.Source: §4.2 · p. 8, Table 8 · p. 8, Table 9 · p. 8
• Improvements in exec@1 exceed those in pass@1, indicating stronger identification of execution errors than intent errors, as the authors describe it.The passage relates this pattern to the difficulty of executing code for language models [3] [34].Source: §4.2 · p. 8

4.3 Ablations

• With lower sampling temperatures, pass@100 decreases while pass@1 increases; CODERANKER further raises pass@1 in three of four setups, reaching 42.7% ranked pass@1 on HumanEval.On low-temperature MBPP, CODERANKER slightly decreases pass@1; the authors attribute this to the small pass@1–pass@100 gap, which makes beating random ranking difficult.Source: §4.3 · p. 8
• Ternary rankers perform best for Codex and GPT-J, while intent-aware and execution-aware-plus-error-line rankers perform best for GPT-Neo 1.3B and 125M, respectively.Binary rankers perform significantly worse, especially with smaller models; execution-aware and execution-aware-plus-error-line rankers lead ranked exec@1 across models.Source: §4.3 · p. 8
• Same-model ranker data performs best among individual datasets, while mixed-large data performs best overall; mixed-small is usually slightly worse than same-model data.The mixed-small set samples 25% of each model's ranker dataset; the authors suggest smaller models could augment the data to improve performance.Source: §4.3 · p. 9, Figure 5 · p. 9

Code generation tasks/models

• The evaluation covers four decoder-based code-generation models and three programming datasets, while the authors describe the approach as architecture-agnostic for models that generate by sampling.They frame ranking generated programs as a new code-understanding task, distinct from vulnerability detection in human-written code [9] [40].Source: §5 · pp. 9–10

Filtering/ranking for code generation models

• CODERANKER ranks sampled programs without inference-time execution, learning how code fails rather than only predicting a binary label; unlike prior neural rankers, it targets general programming rather than math problems.The paper reports benefits from an encoder-only ranker with 125M parameters, compared with at least 3B-parameter models used in [18].Source: §5 · p. 10
• Earlier execution-based filtering [11] [28] requires running potentially vulnerable completions; other work uses execution or static analysis to guide generation or augment model inputs [20] [34] [31].CODERANKER instead prunes wrong programs at inference time with a neural ranker.Source: §5 · p. 10

6 Conclusions and Future Directions

• CODERANKER ranks generated programs without explicitly executing them, using fine-grained failure-mode predictions to improve pass@k and exec@k across models and tasks.The authors identify the approach as fault-aware.Source: §6 · p. 10
• The approach is unsound, adds inference time by sampling n >> k programs, and currently ranks only after full programs are generated.Future directions include ranking partial programs, exploring other ranker architectures and tasks, and transfer to generic programming tasks.Source: §6 · p. 10
• The authors note a lack of generic-programming datasets and report preliminary observations that generated programs often have execution errors rather than intent errors.They connect that observation with CODERANKER's stronger identification of execution errors and express optimism about applying fault-aware ranking to generic programming.Source: §6 · p. 10

Checklist

• The checklist indicates that the paper discusses its contributions and scope, limitations, and potential negative societal impacts.It points to Section 6 for the limitations discussion.Source: Checklist · p. 13
• The authors report releasing code and data on GitHub and documenting training details and computational resources in the paper.They report using publicly available assets and citing their creators, but not stating asset licenses.Source: Checklist · p. 13

• Table 12 provides examples for each execution-error class.Source: §A.1 · p. 14
• Table 13 provides examples for each intent-error class.Source: §A.1 · p. 14
• Figure 6 plots execution-error class distributions on the left and intent-error class distributions on the right for Codex-generated ranker training data.Source: Figure 6 · p. 14, §A.1 · p. 14

• Tables 14 and 15 report ranked metrics for fault-aware ranker tasks using Codex and GPT-Neo 125M as code-generation models.Source: §A.2.1 · p. 14
• Table 14 covers rankers paired with Codex on the APPS validation dataset.Source: §A.2.1 · p. 14
• Figure 7 plots validation ranked exec@1 training curves for rankers across different code-generation models.Source: Figure 7 · p. 14

A.2.3 Alternate ranker architectures

• In an ablation on Codex completions, Dcodex achieved 36.8% ranked pass@1, while the CodeBERT-based ranker achieved 39.6%; evaluation used APPS validation and training on the ternary ranker dataset.Source: §A.2.3 · p. 16, Table 17 · p. 17

A.2.2 Ablations on different ranker datasets

• 2.7% additional improvement came from a mixed-large ranker dataset over the Codex dataset on Codex completions.Source: §A.2.2 · p. 15, Table 16 · p. 17

A.2.3 Alternate ranker architectures

• 38.6% ranked pass@1 for the GPT-Neo 1.3B ranker versus 39.6% for CodeBERT; training stopped at seven epochs after instability.GPT-Neo training took almost four days for seven epochs, while CodeBERT took 12 hours for 30 epochs.Source: §A.2.3 · p. 17, Table 17 · p. 17
• The authors view rankers as code-understanding tasks and consider pre-trained encoders more effective, while leaving a fuller architecture study to future work.Source: §A.2.3 · p. 17

A.2.4 Impact of example unit tests in the prompt

• Without example unit tests in prompts, the ranker performs slightly worse than with them but still significantly outperforms using no ranker.Source: Table 18 · p. 18, §A.2.4 · p. 18

A.2.5 Comparison to ranking using the code gen. model’s predicted probabilities

• For GPT-Neo 125M, a separate CodeRanker ranks best, while ranking by the generator’s probabilities performs slightly worse than no ranking.Source: Table 19 · p. 18, §A.2.5 · p. 18

A.2.6 Qualitative analysis of CODERANKER

• Table 20 pairs generated programs and compiler errors with ranker labels; its colors distinguish correct labels, incorrect labels, and wrong error types.Source: §A.2.6 · pp. 18–19, Table 20 · p. 19