PaperlayerView exact revision on arXiv

Source-linked AI summary

Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen

Om Nepal, Sushant Aryal, Oluseyi Olukola, Nick Rahimi

arXiv:2609.26749v1cs.SEcs.AIcs.CR

TL;DR

The paper asks whether compile rate and reference similarity reliably measure single-function vulnerability repair, where per-example execution tests are often unavailable. It tests these proxies in controlled experiments and examines a change-aware screen. Compile rate exhibits multiple failure modes, while whole-function CodeBLEU can favor a no-op over every tested model.

  • Problem

    Single-function vulnerability-repair evaluations often lack per-example executable tests or exploit oracles, so compile rate and reference similarity are used as proxies whose validity has received little scrutiny.

    Source: §1 · p. 2

  • Method

    The authors run five controlled experiments on 203 Big-Vul functions using three open-source code LLMs and three prompting strategies, and examine diff_F1 as a change-aware screen.

    Source: §1 · p. 2

  • Results

    Compile rate exhibits five distinct failure modes, and an unchanged vulnerable input outscores every tested model on whole-function CodeBLEU.

    Source: §1 · p. 2, §5 · p. 18

  • Takeaways & Limitations

    The authors frame diff_F1 as a change-aware screen before deeper analysis, not as a repair-quality metric.

    Source: §1 · p. 2, §4.7 · p. 18

    diff_F1 is an edit-overlap proxy, not an execution oracle; where per-example execution oracles exist, they are preferable.

    Source: §4.7 · p. 18

Abstract

from arXiv · show

Large language models (LLMs) are increasingly applied to the automated repair of C/C++ security vulnerabilities, and compile rate is a commonly reported proxy for progress: whether the generated patch compiles. We argue that compile rate is a scientifically unreliable metric for single-function vulnerability repair, and we support this with five controlled experiments over 203 vulnerable functions from Big-Vul, three open-source code LLMs (350M to 6.7B parameters), and three prompting strategies. Compile rate (i) barely responds to an intervention that substantially improves the generated code; (ii) is dominated by evaluation-harness and dataset artifacts rather than model quality, with about 64% of compile failures not attributable to the model, a share that is nearly invariant across models; (iii) shifts by 1.8 to 2.7 times on identical patches under a single compiler-standard flag, with zero regressions; (iv) ranks the three models in the opposite order to reference-similarity metrics; and (v) rewards non-repairs when used as an optimization target, since a compiler-feedback loop raises compile rate while similarity to the human fix falls, with manual inspection finding deletion- and placeholder-style non-repairs among the newly compiling outputs. The natural fallback, whole-function CodeBLEU, also fails: an unchanged copy of the vulnerable input outscores every model. We also examine diff_F1, a change-aware screen that scores only the edited region. It gives exactly zero credit to a no-op and near-zero credit to some, though not all, of the deletion-based gaming patches we observed, while still crediting genuine partial edits, so it may serve as a cheap screen before deeper, execution-based analysis. It is not a repair-quality metric, and we report where it falls short. Our findings argue for change-aware, execution-grounded evaluation of LLM-based vulnerability repair.

1 Introduction

LLM vulnerability-repair progress is often measured with compile rate or similarity to a developer’s fix, but these proxies have received little scrutiny for isolated functions. The paper tests their reliability and examines a change-aware screen.

1 Introduction

  • Because function-level datasets often lack executable tests or exploit oracles, compile rate and reference similarity are used as automatic proxies for repair quality.Repair quality requires removing the vulnerability without breaking intended program behavior.Source: §1 · p. 2

Contributions

  • The paper tests compile rate through five controlled experiments on 203 Big-Vul functions, three open-source code LLMs, and three prompting strategies.The experiments target distinct failure modes in single-function vulnerability repair.Source: §1 · p. 2
  • Compile rate and whole-function CodeBLEU can both reward outputs that do not repair the vulnerability, including an unchanged vulnerable input.The paper reports that the unchanged input outscores every tested model on whole-function CodeBLEU.Source: §1 · p. 2, Contributions · p. 3
  • The paper examines diff_F1 as an exploratory change-aware screen: it gives no credit to no-ops and near-zero credit to some deletion patches, but is not a repair-quality metric.It still credits genuine partial repairs, and does not give near-zero credit to every deletion-style patch.Source: §1 · p. 2, Contributions · p. 3

2 Related Work

Prior repair evaluations use different oracles, but generally score the generated output as a whole. This paper focuses on whether those measures suit vulnerability fixes, which often consist of small edits.

Vulnerability datasets and their quality

  • Big-Vul is a widely used Git-mined dataset, but label noise and duplication concerns mean results derived from it should be read cautiously.PrimeVul reports sharply lower apparent vulnerability-detection ability on cleaner, de-duplicated data.Source: §2 · p. 3

LLM-based program and vulnerability repair

  • Prior LLM vulnerability-repair work emphasizes improving repairs, while this study examines whether the metrics used to judge them are trustworthy.The study uses base models and prompting strategies also used in prior work.Source: §2 · p. 4

Evaluation metrics for code generation

  • Code-generation evaluation commonly uses pass@k and CodeBLEU, but neither was designed to isolate the small edit typical of a vulnerability fix.When executable tests are absent, pass@1 corresponds to compile rate under uniform sampling; CodeBLEU compares generated code with a reference fix.Source: §2 · p. 4

Critiques of secure-code and repair evaluation

  • Prior secure-code evaluation critiques include concerns about conflating security with functionality and achieving security through code deletion; this study examines deletion under compile-rate optimization.Dai et al. [5] propose an execution-grounded combined metric, whereas this study examines a lightweight change-aware score requiring no tests.Source: §2 · p. 5

Summary of the gap

  • Prior evaluation methods score whole outputs using tests, exact match, reference similarity, static analysis with execution, or an LLM judge; the paper explores scoring the edit instead.The cited examples include pass@k and SWE-bench [4, 14], VulRepair [9], CodeBLEU [17], Dai et al. [5], and an LLM judge [19].Source: §2 · p. 5, Table 1 · p. 5

3 Methodology

The methodology evaluates single-function C/C++ vulnerability repair on Big-Vul using three base LLMs, three prompting strategies, an isolated-function compile harness, and five controlled experiments. It combines compile rate/pass@k, whole-function CodeBLEU, and change-aware diff_F1 to examine metric reliability.

3.1 Dataset

  • The test set contains 203 functions across seven frequent CWE categories, with 29 functions per category and a disjoint few-shot pool of three examples per category.The evaluation is focused on these seven classes, not every category in Big-Vul.Source: §3.1 · p. 6, Table 2 · p. 7

3.2 A Big-Vul Preprocessing Artifact

  • 96–100% of missing-return-type cases stemmed from Big-Vul preprocessing, accounting for roughly 15–19% of compile failures.The stored input had already lost the return type, so the model reproduced a malformed signature.Source: §3.2 · p. 7

3.3 Models, Prompting, and Generation

  • Three base models spanning 350M–6.7B parameters use zero-shot, few-shot, and chain-of-thought prompts, with five independently sampled patches per model–strategy–function cell.The instruction-tuned DeepSeek-Coder-1.3B appears only in a robustness experiment.Source: §3.3 · pp. 7–8, Table 3 · p. 8

3.4 Isolated-Function Compile Harness

  • An iterative stub-injection harness adds declarations for missing identifiers before syntax-checking isolated functions with GCC or G++; it runs for up to six iterations.The default compiler standard is C23, and the harness uses variadic function stubs and reconciles declarations to avoid spurious errors.Source: §3.4 · pp. 8–9

3.5 Evaluation Metrics

  • Compile rate equals aggregate pass@1 under the binary syntax-check criterion; CodeBLEU compares whole functions, while diff_F1 scores only developer-changed sides.diff_F1 gives no-ops zero and exact developer edits one, but is a screen for no-ops and some deletion-style patches, not repair quality.Source: §3.5 · pp. 10–11
  • Semgrep returned no findings for all 9,131 non-degenerate patches, including 372 compiling patches, so the paper draws no claims from its static-analysis output.The authors attribute this to inability to parse most patches and limited default-rule coverage for the target CWEs.Source: §3.5 · p. 11

3.6 Experimental Design

  • Five controlled experiments isolate distinct metric failure modes; headline estimates use function-level bootstrap 95% confidence intervals, and paired experiments vary one factor on fixed functions.The generation-budget and compiler-standard comparisons use a fixed, stratified 70-function subset where repeated runs are needed.Source: §3.1 · pp. 6–7, §3.6 · p. 11
  • On the full 203-function test set, compile rate ranks CodeGen above DeepSeek-1.3B and DeepSeek-6.7B, while CodeBLEU ranks the models in the opposite order.The two DeepSeek models are statistically indistinguishable on compile rate and only weakly separated on CodeBLEU.Source: Table 5 · p. 12

4 Results

Across these experiments, compile rate often fails to track repair quality: it responds weakly to a quality improvement, shifts with compiler settings, and can reward non-repairs. Neither compile rate nor whole-function CodeBLEU reliably measures repair, while diff_F1 screens some—but not all—degenerate edits.

4.1 Three-Model, Three-Strategy Comparison

  • Compile rate and CodeBLEU rank the models in reverse order: CodeGen-350M leads compile rate at 5.45% but has the lowest CodeBLEU (0.463), while DeepSeek-6.7B has the lowest compile rate (3.32%) and highest CodeBLEU (0.675).Compile-rate intervals separate none of the three models on unpaired intervals; CodeBLEU separates CodeGen from both DeepSeek models, but neither metric cleanly separates the DeepSeek models.Source: §4.1 · p. 12
  • Prompting had a small and inconsistent effect: within each model, compile-rate spreads across the three strategies were at most 2.0 percentage points, and zero-shot had the highest rate for all three models.The comparison used the full 203-function test set under the default toolchain.Source: §4.1 · p. 12, §4 · p. 12, Table 6 · p. 13

4.2 Compile Rate Does Not Respond to Genuine Improvement

  • Raising the generation budget cut truncation by 17 to 30 percentage points and raised CodeBLEU by 0.06 to 0.12, while compile rate moved only +0.6 to +1.3 percentage points.The 256-to-512-token intervention used the 70-function subset with models and prompts held fixed.Source: §4.2 · p. 13, Table 7 · p. 13

4.3 Most Compile Failures Reflect the Evaluation Setup, Not the Model

  • About 64.1 to 64.9% of compile failures were not attributable to the model, chiefly because of missing project context and a Big-Vul return-type artifact.This share varied by only 0.8 percentage points across models spanning 350M to 6.7B parameters.Source: §4.3 · pp. 13–14, Table 8 · p. 14

4.4 A Single Compiler Flag Nearly Doubles Compile Rate

  • Changing only the C-standard flag moved compile rate by a factor of 1.8 to 2.7 on identical patches, with zero regressions.The gnu89 setting lifted rates for all three models, while gnu17 left them unchanged from the default.Source: §4.4 · pp. 14–15, Table 9 · p. 15

4.5 Compile Rate and CodeBLEU Rank Models in Opposite Order

  • The copy-input no-op scores 0.775 whole-function CodeBLEU, higher than every model, while compile rate and CodeBLEU rank models oppositely.Because whole-function similarity is dominated by unchanged context, it rewards a patch that changes nothing; the ranking disagreement alone does not establish which model repairs better.Source: §4.5 · p. 15

4.6 Optimizing Compile Rate Rewards Non-Repairs

  • Under compiler feedback, compile rate rose while CodeBLEU fell; manual inspection found newly compiling outputs that deleted or replaced the original function rather than repairing it.For the instruction-tuned model, compile rate more than doubled from 3.9% to 9.4%; examples included bare declarations and placeholders.Source: §4.6 · pp. 15–16, Figure 3 · p. 16

4.7 A Change-Aware Screen: diff_F1

  • diff_F1 gives the copy-input no-op exactly 0, but its credit for degenerate patches is mixed, ranging from 0.027 for whole-function #if 0 wraps to 0.22 for bodies that only return zero.The two Figure 4 patches score 0.078 and 0.087, while empty bodies with placeholder comments average 0.18; diff_F1 is a change-aware screen, not a repair-quality measure.Source: §4.7 · pp. 16–18, Table 10 · p. 17, Table 11 · p. 18

5 Discussion

Across five experiments, compile rate fails in several ways and can reflect benchmark and toolchain properties rather than repair quality. Whole-function CodeBLEU also favors a no-op, while change-aware scoring and explicit baselines offer qualified diagnostics.

5 Discussion

  • Compile rate exhibits five failure modes and can measure benchmark and toolchain properties rather than repair quality.The failures include sensitivity to compiler configuration, disagreement with reference similarity, and gameability through code deletion.Source: §5 · p. 18
  • A no-op copy of the vulnerable input outscores every model under whole-function CodeBLEU, while diff_F1 gives no credit to preserved context.In examined cases, diff_F1 also gives near-zero credit to some deletion-style patches, but not all.Source: §5 · p. 18
  • No-op and degenerate-patch baselines can expose construct-validity problems in similarity metrics, and compile results should include the exact harness and toolchain.The authors also recommend reporting known failure modes when using diff_F1 as a supplementary diagnostic.Source: §5 · p. 19
  • The observed deletion behavior also resembles reports of functionality removal when models optimize against static-analysis outcomes [5].The authors suggest this may reflect optimization against an easily satisfied proxy rather than a particular tool.Source: §5 · p. 19

6 Limitations and Future Work

The paper bounds its claims by the lack of execution-based validation, limitations in identifying gaming patches and classifying failures, and the study’s model, dataset, and task scope. It proposes further metric validation and extensions to stronger models and repository-level repair.

6 Limitations and Future Work

  • Neither compile rate nor diff_F1 verifies vulnerability removal or behavior preservation, and diff_F1 has not been validated against human repair-quality judgments.The study conducted no execution- or exploit-based validation, so its repair-quality claims remain bounded by that gap.Source: §6 · p. 19
  • Of 14 identified compiling degenerate patches, 12 came from the instruction-tuned model, and the heuristic search cannot establish that the set is exhaustive.The proposed explanation that deletion earns credit by overlapping developer-removed lines was not isolated by ablation.Source: §6 · p. 19
  • Truncation percentages are approximate, per-category failure splits depend on taxonomy ordering, and compiler-feedback similarity gaps are suggestive because per-round counts are small.Group-level failure totals are described as robust; qualitative deletion evidence is directly visible in the patches.Source: §6 · p. 20
  • The study covers three open-source base models, one dataset, single-function C/C++, and seven CWE categories, without a leakage or de-duplication audit.The authors avoid per-CWE difficulty claims because estimates are under-powered.Source: §6 · p. 20
  • The critique is established for the small-to-mid-size open-model regime studied, and substantially stronger or instruction-tuned models may exercise the metrics differently.The three base models span 350M to 6.7B parameters and compile at 3 to 5.5%.Source: §6 · p. 20
  • Future work could modify diff_F1 to reduce deletion credit, test variants on held-out gaming patches, and compare scores with human repair-quality ratings.The paper also proposes checking how much original function survives and replicating the experiments on larger and instruction-tuned models.Source: §6 · p. 20
  • Extending diff-level and execution-based evaluation to repository-level repair could use available project context and sometimes permit an execution oracle.Source: §6 · p. 21

7 Conclusion

Across five experiments, the paper finds compile rate unreliable for single-function vulnerability repair and whole-function CodeBLEU vulnerable to no-op advantage. It positions diff_F1 as a change-aware screen, not a repair-quality measure, and recommends execution-grounded verification where possible.

7 Conclusion

  • Compile rate is unreliable across five experiments, and a no-op copy of the vulnerable input outscores every model under whole-function CodeBLEU.The paper presents diff_F1 as a change-aware screen rather than a measure of repair quality.Source: §7 · p. 21
  • The stub-injection compile harness, diff_F1 implementation, prompt templates, test-set identifiers, and reproduction scripts are publicly available.Source: §7 · p. 21

Related papers

Loading 2609.26749v1…