APPATCH: Automated Adaptive Prompting Large Language Models for Real-World Software Vulnerability Patching

1 Introduction

• Automated patching matters because manual approaches are costly and struggle to keep pace with new vulnerabilities amid growing cyberattacks.Source: §1 · p. 1
• Prompt-based vulnerability patching faces four challenges: automating prompts, selecting exemplars, accommodating code context, and producing stable, valid, correct patches.Source: §1 · p. 2
• APPATCH scopes code to vulnerability-relevant behavior, reasons about root causes, selects exemplars dynamically, and uses an LLM ensemble to cross-validate patch quality.Source: §1 · p. 2
• APPATCH achieves up to 36.46% and 73.86% F1 on the two datasets, versus 28.41% and 68.54% with baseline prompting, respectively.Source: §1 · p. 2
• APPATCH achieves 49.48% and 90.00% recall on the two datasets, versus up to 17.53% and 85.00% for state-of-the-art traditional approaches, respectively.Source: §1 · p. 2
• APPATCH's evaluation suggests LLM-based patching is practically feasible, while its failure-case analysis examines where and why models may still fall short.Source: §1 · p. 2

2 Background and Motivation

• Both standard prompting and zero-shot completion produce a strncpy edit with a len+1 limit, but fail to analyze the actual size of p.Source: §2 · p. 3
• Semantics-aware scoping gives LLMs only the code entities that capture a program's core vulnerable behavior.Source: §2 · p. 3
• Vulnerability-semantics reasoning uses chain-of-thought prompting with exemplars to guide LLMs in analyzing and patching vulnerabilities.Source: §2 · p. 3
• Automated exemplar mining generates reasoning-step exemplars from existing real-world samples, avoiding time-consuming manual authoring.Source: §2 · p. 3
• Dynamic adaptive prompting selects exemplars automatically according to the testing sample's vulnerability root causes.Source: §2 · p. 3

3.1 Overview

• APPATCH takes vulnerable statements, CWE IDs, and existing vulnerable samples with known patches, then mines exemplars and uses them to guide patch generation and validation.Exemplars contain root-cause analyses, fixing strategies, and ground-truth patches; the system returns validated patches.Source: §3.1 · pp. 3–4, Figure 4 · p. 4

3.2 Vulnerability Semantics

• Vulnerability semantics comprise vulnerable statements and their control- and data-dependency context back to external inputs, guiding analysis while narrowing prompts and reducing distractions.APPATCH defines the semantics as the union of interprocedural static backward slices from external inputs to vulnerability locations.Source: §3.2 · p. 4

3.3.1 Semantics-Aware Scoping (Step 1.1)

• APPATCH builds an interprocedural system dependence graph, identifies external inputs such as input-variable definitions and selected function calls, then unions backward slices to vulnerable statements.This captures dependencies across functions between external inputs and vulnerability locations.Source: §3.2 · p. 4, §3.3.1 · p. 5

3.3.2 Exemplar Generation (Step 1.2)

• LLMs use vulnerability-semantics-guided reasoning and known patches to automatically generate exemplar root-cause analyses and fixing strategies; GPT-4’s generated reasoning was judged 92.98% correct.The exemplars also include ground-truth patches, and guided reasoning helps produce more comprehensive root-cause analyses than unguided reasoning.Source: §3.3.2 · p. 5

3.4.1 Semantics-Aware Scoping (Step 2.1)

• For test samples, APPATCH recomputes vulnerability semantics using the same system-dependence-graph and scoping procedure as in exemplar mining.Source: §3.4.1 · pp. 5–6

3.4.2 Dynamic Adaptive Prompting (Step 2.2)

• Progressive prompts generate a root cause, expand code context when requested, select exemplars with similar root causes, and ask the LLM to produce five candidate patches.APPATCH selects up to eight exemplars and provides their reasoning, fixing strategies, and known patches alongside the test sample to guide generation.Source: §3.4.2 · pp. 6–7

3.4.3 Multi-Faceted Patch Validation (Step 2.3)

• An ensemble of LLMs checks whether each candidate fixes the vulnerability while preserving functionality, retaining it if at least one validator approves.Candidates not approved by any validator are removed.Source: §3.4.3 · pp. 7–8
• APPATCH prioritizes improving recall while keeping precision over majority voting, which may improve precision but reduce recall and be less helpful for real-world patching.The passage describes recall with maintained precision as more practically valuable for real-world developers.Source: §3.4.3 · p. 8

4 Implementation

• Joern builds a system dependence graph from per-function program-dependence graphs and the call graph, enabling interprocedural dependence analysis without requiring compilable C samples.The implementation uses Joern because it supports scalable code analysis without compilability.Source: §4 · p. 8
• APPATCH extracts interprocedural backward slices from vulnerable statements, reusing SySeVR’s script to query control-flow nodes and find in-degree edges and nodes.Source: §4 · p. 8
• The extracted vulnerability-semantics slices are stored as sliced source code text for the LLMs.Source: §4 · p. 8

5.1 RQ1: Effectiveness

• On Zero-Day and ExtractFix, APPATCH achieves the best prompting-based correctness F1 scores, reaching 36.46% and 73.86%, respectively.The evaluation uses four general-purpose LLMs and manually checks generated patches; up to five patches are generated per sample.Source: §5 · pp. 8–9, §5.1 · p. 9, Table 2 · p. 10

5.2 RQ2: Contributions of Components

• Without multi-faceted validation, F1 falls to 20.41%–35.77% on Zero-Day and 40.00%–73.25% on ExtractFix; without semantics-aware slicing, it falls to 19.08%–30.08% and 37.21%–65.50%.The authors attribute the slicing ablation’s decline to irrelevant code distracting root-cause analysis; reasoning accuracy is higher with scoping.Source: §5.2 · p. 10
• Direct root-cause prompting, standard prompting, and zero-shot completion underperform APPATCH; standard prompting reaches at most 23.01% F1 on Zero-Day and 60.42% on ExtractFix, while zero-shot reaches 12.48% and 24.66%, respectively.The standard-prompting results are reported even with state-of-the-art LLMs; the zero-shot comparison uses its best template, s2.Source: §5.2 · p. 11

5.3 RQ3: Comparison to Existing Techniques

• APPATCH reaches 90% recall on ExtractFix and 49.48% on the Zero-Day dataset, outperforming the compared non-LLM tools on both evaluations.On Zero-Day, its recall exceeds VulRepair’s 17.53% and Getafix’s 6.19%; on ExtractFix, ExtractFix reaches 85% using test cases and constraints, while other baselines reach 25%–45%.Source: Table 4 · p. 11, §5.3 · pp. 11–12

5.2 RQ2: Contributions of Components

• For interprocedural vulnerabilities, APPATCH’s GPT-4 F1 drops from 33.30% overall to 31.37%, while baselines decline more sharply despite receiving favorable inputs.The authors attribute the baseline deterioration to their lack of interprocedural analysis.Source: §5.2 · p. 11

5.3 RQ3: Comparison to Existing Techniques

• On ExtractFix, APPATCH with GPT-4 reaches 90% recall; ExtractFix reaches 85% using test cases and constraints, while other baselines reach 25%–45%.The comparison uses recall because ExtractFix and VulnFix generate many patches.Source: §5.3 · pp. 11–12
• On Zero-Day, APPATCH reaches 49.48% recall, compared with 17.53% for VulRepair and 6.19% for Getafix.ExtractFix and VulnFix are excluded from this comparison because the samples are not compilable.Source: §5.3 · p. 12

5.4 RQ4: Efficiency

• Patch generation takes 37.148–50.209 seconds per sample, uses 5,684–6,802 input/context tokens, and costs 0.2–8 cents per sample.Generated tokens range from 584 to 886; Gemini-1.5 and Llama-3.1 may be free under the described usage or deployment options.Source: §5.4 · p. 12

6.1 Why APPATCH Works

• Semantics-aware scoping reduces distracting code and clarifies data/control flow, helping LLMs trace dependencies; dynamic exemplars guide deeper root-cause analysis, while vulnerability-semantics reasoning also serves as that analysis.In one example, scoping brought a variable’s last definition from six lines away to two; functionality semantics did not help models understand the vulnerability root cause as much.Source: §6.1 · pp. 12–13

6.2 Usability of APPATCH

• In both end-to-end scenarios, APPATCH worked reasonably well and outperformed baselines, but performance fell below perfect-input results; better upstream tools and more expert users could improve it.The scenarios use automated upstream outputs or outputs inspected and calibrated before patching.Source: Table 7 · p. 13, §6.2 · pp. 13–14

6.1 Why APPATCH Works

• Multi-faceted patch validation improves precision without reducing recall, raising F1 and reducing developers’ work selecting usable patches.Source: §6.1 · p. 13

6.3 How and Why APPATCH Fails

• Among failure symptoms, incorrect vulnerability identification accounts for 35.69% and insufficient or incorrect code modification for 31.23%; root causes include misunderstood vulnerabilities, inadequate context analysis, and missed edge cases.The three root causes account for 41.33%, 25.21%, and 10.21%, respectively.Source: §6.3 · p. 14

6.4 Limitations

• Root-cause errors can misdirect patches, and the 306+76+20 evaluation samples lack guaranteed compilation or test cases because reproducible vulnerability datasets are difficult to build.The authors attribute these limitations to the datasets used and note that existing compilable datasets with tests are relatively small.Source: §6.4 · p. 14

6.5 Extensibility

• APPATCH’s design is language- and CWE-agnostic; extending it requires adding exemplars for other CWEs and a code analyzer that constructs SDGs for other languages.The current implementation and evaluation use C and six common CWEs.Source: §6.5 · p. 14

6.6 LLMs for Automated Patching

• Simple patch requests remain weak even with the latest models; token limits motivate code slicing and test-adaptive exemplars, while using pretrained LLMs avoids costly training or fine-tuning.Source: §6.6 · pp. 14–15

• VuRLE uses machine learning to detect and repair vulnerabilities from examples, while Zhang et al. developed a similar example-based approach for Java [54].Source: §7 · p. 15
• Neural transfer learning for C vulnerability repair [14] and the T5-based VulRepair tool apply deep learning to automated vulnerability repair.Source: §7 · p. 15
• Gao et al. guide program vulnerability repair with crash constraints, while Pearce et al. study zero-shot LLM repair without extensive task-specific training.Source: §7 · p. 15

• APPATCH is an automated vulnerability-patching framework that uses dynamic adaptive prompting to elicit LLM reasoning for root-cause analysis and patch generation.Source: §8 · p. 15
• The framework is evaluated on four recent LLMs, and the authors report that it substantially outperforms existing prompting approaches and state-of-the-art non-LLM patching techniques.Source: §8 · p. 15
• The paper’s stated goal is to elicit effective LLM reasoning for vulnerability root-cause analysis as a basis for quality patch generation.Source: §8 · p. 15

• The authors state that their research complies with the ethical guidelines of USENIX Security’25 and responsible security-research principles.Source: Ethics Considerations · p. 15
• The vulnerabilities had already been publicly disclosed as CVEs by the time of writing, and the data were anonymized.Source: Ethics Considerations · p. 15
• The study reports that it did not experiment on production systems, avoiding potential disruption or harm to real-world users.Source: Ethics Considerations · p. 15

• The authors commit to sharing the codebase, datasets, and experimental results through a publicly accessible repository.Source: Open Science · p. 16
• APPATCH source code, documentation, and experimental results are available at a Zenodo repository.Source: Open Science · p. 16
• The authors state that their work aligns with USENIX Security’25’s ethical guidelines and open-science policy.Source: Open Science · p. 16

A Compostion and Statistics of the Datasets

• The Zero-Day dataset spans 18 projects and six CWEs, averaging 7.87 patched lines; ExtractFix spans seven projects with the same CWEs and averages 3.05 patched lines.Source: A Compostion and Statistics of the… · p. 18

B Code LLMs against the Zero-Day Dataset

• 10.12% F1 is the highest score among the evaluated code LLMs on the Zero-Day dataset, substantially below the general-purpose LLMs.The evaluated models were CodeLlama-70b, CodeQwen-1.5-7b, and DeepSeek-Coder-V2-236b.Source: B Code LLMs against the Zero-Day Dataset · p. 18

C Correct Reasoning Rate Comparison

• Across all four models, Vulnerability Semantics Reasoning achieves the highest correct reasoning rates, while Direct Reasoning and No Slicing generally score lower.Source: C Correct Reasoning Rate Comparison · p. 18

D More Results on Interprocedural Samples

• 33.57% and 31.37% F1 are the best scores, achieved by Claude-3.5 and GPT-4 with APPATCH on interprocedural samples.Source: D More Results on Interprocedural… · p. 18
• Standard Prompting and Zero-shot Completion perform worst on interprocedural samples, with F1 ranges of 1.79%-9.81% and 0.00%-8.39%, respectively.Source: D More Results on Interprocedural… · p. 18
• The reported APPATCH F1 scores for Claude-3.5 and GPT-4 outperform the ablated prompting versions on these samples.Source: D More Results on Interprocedural… · p. 18

E Data Leakage Assessment

• The ExtractFix data-leakage assessment found that LLMs usually did not know answers or answered incorrectly, but the authors cannot ensure there was no leakage.Source: E Data Leakage Assessment · p. 18, Table 11 · p. 19
• Three queries progressively provide CVE, project, and CWE information to test whether LLMs know the remaining vulnerability details, including the patch.The authors caution that they cannot claim these queries are the best general assessment of vulnerability-patch leakage.Source: E Data Leakage Assessment · p. 18
• The performance evaluation is primarily based on the Zero-Day dataset; ExtractFix enables comparison with the ExtractFix and VulnFix baselines.Source: E Data Leakage Assessment · p. 19

G More Details on Case Study Examples

• In one failure example, APPATCH misdiagnoses a packet-type logic error as an out-of-bounds write involving source_id and proposes fixes focused on that variable.The described correct fix adds a condition checking whether the context ID matches a specific packet type.Source: G More Details on Case Study Examples · pp. 19–20, Figure 10 · p. 19

F More End-to-end Experiment Results

• 23.85% F1 with the fully automated method and 27.29% with the realistic method are reported for APPATCH integrated with CodeQL on the Zero-Day dataset.Source: F More End-to-end Experiment Results · p. 19, Table 13 · p. 20

G More Details on Case Study Examples

• In another failure, proposed changes to build_label miss insufficient skb bounds checks, which permit out-of-bounds reads of edl data.The ground-truth patch adds multiple checks that validate skb->len against expected data sizes.Source: G More Details on Case Study Examples · p. 20, Figure 11 · p. 20