Source-linked AI summary
Benchmarking Automated Security Patch Backporting: How Far Are We?
Jincheng Yang, Yulong Fu, Chengwei Liu, Lyuye Zhang, Fangyuan Zhang, Bingyang Ren, Yang Liu, Hui Li
TL;DR
Current evidence does not establish how well automated security patch backporting tools generalize under aligned assumptions and common metrics. This paper introduces Porting Benchmark to evaluate five tools across diverse scenarios, finding that structural complexity sharply reduces success and changes the apparent performance landscape.
Problem
It remains unclear how far current patch backporting tools generalize under aligned assumptions, diverse scenarios, and common evaluation metrics.
Method
The paper constructs Porting Benchmark and a common evaluation framework spanning diverse scenarios, controlled localization assumptions, and five tools.
Results
Aligned evaluation changes the performance landscape, while the best commit-level success falls from 85.2% on Type-I patches to 24.0% on Type-IV patches.
Takeaways & Limitations
Automated backporting remains most challenged by structurally hard target adaptations, with static agreement potentially diverging from executable remediation.
Takeaways & Limitations
The results may not generalize beyond C/C++ security patches drawn from over 50 projects, and executable findings are limited to 45 cases.
Abstract
from arXiv · showhide
Automated security patch backporting is critical for mitigating N-day vulnerabilities. Recent tools report success rates above 80% on their respective datasets. However, these evaluations are often confined to homogeneous environments, such as one repository or specific project versions. Consequently, it remains unclear how well these tools generalize beyond their originally targeted scenarios. We present Porting Benchmark, a curated dataset of 1,234 security patch backporting cases spanning cross-version, cross-branch, and cross-repository scenarios, paired with a common evaluation framework. Using this benchmark, we evaluate five tools spanning program analysis, LLM prompting, and LLM agents under aligned settings. Our results show that aligned evaluation changes the apparent performance landscape: PortGPT and TSBPort remain comparatively strong on the Replication Dataset, while FixMorph and Mystique degrade substantially under the common protocol. Performance degrades sharply on structurally complex patches: the best commit-level success rate falls from 85.2% on Type-I patches to 24.0% on Type-IV. We identify four root-cause categories (missing target API awareness, cross-version semantic mismatch, non-local dependency propagation failure, and patch construction or localization failure) and derive concrete directions for next-generation tool design. On a 45-case dynamically validated subset with verified test cases and constructed POCs, we further observe that reference-based benchmark scores do not fully capture real-world remediation: exact match sharply under-credits harder target adaptations, while executable validation reveals residual integration failures in the target that static reference agreement misses. Executable-feedback refinement provides limited but measurable recovery on the hardest executable cases.
1 Introduction
Automated security patch backporting is essential for propagating vulnerability fixes across maintained versions, branches, and downstream forks, but existing evaluations are difficult to compare and may not generalize. Porting Benchmark addresses these gaps with 1,234 cross-scenario cases and aligned evaluation, revealing sharp performance declines on complex patches and recurring capability deficits.
- Benchmark and evaluation: Porting Benchmark contains 1,234 cases spanning cross-version, cross-branch, and cross-repository backporting scenarios under a unified, reproducible evaluation methodology.The benchmark includes 600 replication cases and 634 common-scope evaluation cases.
- Benchmark and evaluation: Aligned evaluation leaves PortGPT and TSBPort comparatively strong on the Replication Dataset, while FixMorph and Mystique degrade substantially under the common protocol.Existing tools report success rates from 42% to 95% on tool-specific datasets, but their metrics, inputs, and criteria are not directly comparable.
- Results and failure analysis: 85.2% on Type-I patches falls to 24.0% on Type-IV patches for the best commit-level success rate, making structural complexity the dominant bottleneck.The benchmark evaluates dimensions including cross scenario, porting type, patch scale, and information level.
- Executable validation: On 45 cases with verified tests and constructed POCs, executable validation exposes residual integration failures that reference-based scores miss, while executable-feedback refinement provides limited measurable recovery.PortGPT reaches 80.0% for both S-Succ and Full validation on this subset; refinement targets 29 hard Type-III/IV or cross-version cases.
- Results and failure analysis: The analysis identifies four recurring deficits: missing target API awareness, cross-version semantic mismatch, non-local dependency propagation failure, and patch construction or localization failure.These deficits explain why tools fail and define directions for next-generation backporting systems.
2 Background
Security patch backporting adapts fixes across divergent code contexts, often requiring relocation, rewriting, and revalidation rather than direct cherry-picking. Existing tools and evaluations differ in scenarios, pipeline stages, assumptions, datasets, and success criteria, leaving their generalization unclear.
- Backporting Challenges: Backporting fixes across syntactic, structural, and interface drift may require relocation, rewriting, or revalidation in the target codebase.Unlike direct cherry-picking, the target implementation can diverge substantially in names, control flow, and APIs.
- Prior Tools: Existing tools span program analysis, LLM prompting, and LLM agents, but their results are difficult to compare because scenarios, granularity, location assumptions, and validation criteria differ.Representative tools include FixMorph and TSBPort, Mystique, PPatHF, and MigGPT, and PortGPT.
- Pipeline Decomposition: Tools differ across localization, transformation, and validation, with validation remaining tool-specific and thereby driving a major comparability gap.Traditional tools emphasize program-analysis-based localization and transformation, whereas LLM-based tools rely more on context selection and generation.
- Evaluation Gaps: Most tools are evaluated in a single scenario, while public datasets concentrate on ecosystems such as the Linux kernel or narrow fork settings, potentially masking failures on diverse structural adaptations.A cross-repository case requires restructuring a four-function call chain because the target lacks a source struct entirely.
- Study Motivation: These gaps prevent reliable assessment of how current tools generalize under aligned assumptions and common metrics, motivating a benchmark with diverse scenarios and controlled localization assumptions.The proposed study design addresses scenario coverage, dataset composition, localization assumptions, and evaluation consistency.
3 Study Design · 3.1 Benchmark Dimensions
The study uses a common evaluation methodology and a unified taxonomy for security patch backporting. The taxonomy separates source–target relationships from patch complexity and treats path changes as an independent evaluation dimension.
- 3 Study Design: The study establishes a common methodology covering benchmark dimensions, tool selection, data construction, evaluation metrics, and research questions.
- 3.1 Benchmark Dimensions: The benchmark taxonomy defines two orthogonal dimensions: Cross Scenario for source–target relationships and Porting Type for patch complexity.
- 3.1.1 Cross Scenario.: Cross-version backporting occurs between different version lines of the same software and is primarily targeted by FixMorph and TSBPort.
- 3.1.1 Cross Scenario.: Cross-branch backporting spans branches within one repository, whereas cross-repository backporting spans distinct repositories such as forks or distributions.
- 3.1.1 Cross Scenario.: Mainline-to-stable/LTS Linux backports are classified as cross-version because stable/LTS branches represent distinct release lines with strict backporting rules.
- 3.1.1 Cross Scenario.: Backporting complexity is characterized along two orthogonal dimensions: code adaptation complexity and path change.
- 3.1.2 Porting Type and Path Change.: Code adaptation complexity comprises four types: Identical, Location Change, Syntactic Adaptation, and Structural Change.Type-I requires no code modifications; Type-II requires location adjustments; Type-III requires syntactic modifications; Type-IV requires structural change.
- 3.1.2 Porting Type and Path Change.: Path change is tracked separately because file renaming or directory restructuring can require localization even when code adaptation is Type-I.This dimension addresses the common assumption that patches apply to the same file path.
3.2 Tool Selection
The study selects five publicly available tools spanning three paradigms for evaluation. Selection requires C/C++ support and general patch-backporting capability rather than specialization to particular vulnerability types.
- Tool Selection: Five tools were selected from eight reviewed tools, covering program analysis, LLM prompting, and LLM agent paradigms.Table 2 summarizes the selected tools and their original evaluation setups.
- Tool Selection: Selection required public availability, support for C/C++ programs, and general patch backporting beyond specific vulnerability types.These criteria were applied uniformly to the reviewed tools.
- Tool Selection: Three tools were excluded from the evaluation, including PatchWeave because it requires test exploits unavailable for most vulnerabilities.The excluded tools are marked with † in Table 1.
3.3 Benchmark Construction
Porting Benchmark is designed for cross-tool comparability while covering diverse real-world backporting scenarios. Its two-tier datasets support reproducible aligned evaluation and broader generalization assessment through normalized records, leakage controls, and hierarchical patch representations.
- Benchmark goals: Porting Benchmark targets cross-tool comparability across a representative range of real-world scenarios.
- Dataset design: The two-tier design separates a Replication Dataset for reproducibility and an Evaluation Dataset for broader capability and generalization assessment.RQ1 uses aligned metrics, while RQ2–RQ4 examine newer, scenario-diverse cases.
- Replication Dataset: 917 file-level pairs and 2,020 functions form the shared replication pool, with tool-specific RQ1 subsets ranging from 224 to 600 cases.The subsets contain 367 PPatHF, 224 Mystique, 285 FixMorph, and 600 cases each for TSBPort and PortGPT.
- Evaluation Dataset: The Evaluation Dataset uses 2024+ cases and complementary collection routes to assess generalization beyond original tool-paper data while reducing LLM leakage.
- Representation and annotation: Each record preserves changed hunks in a hierarchical commit→file→function representation, mapping overlapping hunks to functions while retaining file-level hunks outside function bodies.A stratified sample of 100 cases achieved Cohen’s κ=1.000 for cross-scenario labels and 0.972 for porting-type labels; cross-version cases contained more Type-III/IV patches and larger edits.
3.4 Evaluation Framework
The evaluation framework aligns heterogeneous tools to common reference-based and executable assessments while preserving their native operation modes. It separates localization from transformation, evaluates end-to-end failures against a ground-truth denominator, and varies the information supplied to tools from repository-level to function-level.
- Common Evaluation Protocol: Common denominators, aggregation levels, and validation metrics improve comparability across tools with incompatible native metrics.Outputs are aligned as closely as available evidence permits.
- Validation Metrics: The framework reports EM for exact normalized identity and S-Succ for non-exact outputs passing L2–L5 structural checks followed by ManualVerify.S-Succ measures reference consistency rather than executable correctness; PPR, TPR, and Full provide executable evidence where available.
- Validation Metrics: ManualVerify reviews only non-exact function-level items that pass L2–L5, checking syntax, safety and cleanup, and missing or weakened security fixes.Items failing any L2–L5 check are not manually promoted.
- End-to-End Evaluation: End-to-End evaluation jointly measures localization and transformation against a ground-truth denominator, counting missing, malformed, unapplied, or mislocalized outputs as 0.The protocol reports EM and S-Succ against GT.
- Information Levels: Three information levels test localization sensitivity: I0 supplies repository-level context, I1 supplies the target file, and I2 supplies the target function.Tools run in native modes, with outputs mapped to a common ground-truth coordinate space; primary stratified analyses use each tool’s highest supported level.
3.5 Experimental Setup
Experiments used a fixed high-performance server and each tool’s recommended model or inference configuration. FixMorph was evaluated only on Linux repositories in RQ1 because validated build configurations were unavailable for other projects, so it was omitted from RQ2.
- Hardware and tool configuration: Experiments ran on 2× Intel Xeon Gold 6226R CPUs and 4× NVIDIA A100 80GB GPUs.The hardware configuration was shared across all experiments.
- Hardware and tool configuration: Each LLM-based tool used its originally recommended model and inference settings, including CodeLlama-13B-Instruct, StarCoder, or GPT-4o.Mystique and PPatHF used local LoRA fine-tuning; PortGPT invoked GPT-4o through the OpenAI API at temperature 0.5, while PPatHF used temperature 0.1.
- FixMorph evaluation scope: FixMorph was evaluated on Linux repositories in RQ1 but omitted from RQ2 because validated compilable configurations were available only for the original Linux kernel experiments.The standalone release required compilable Pa/Pb/Pc versions plus project-specific build and configuration commands to construct complete ASTs.
3.6 Research Questions
The paper is organized around five research questions covering tool performance, scenario-specific behavior, real-world security effectiveness, root causes, and executable-feedback improvement.
- Research Questions: Five research questions span tool performance, scenario-specific behavior, benchmark-to-real-world security effectiveness, root-cause explanation, and executable-feedback improvement.The questions address both evaluation outcomes and explanations for tool behavior.
- Research Questions: RQ1 asks how existing tools perform under a common evaluation protocol compared with their original reported results.This question targets performance stability across evaluation settings.
- Research Questions: RQ2 examines tool performance across different backporting scenarios and levels of complexity.This question targets scenario-specific behavior rather than aggregate performance alone.
4 Evaluation Results
Under aligned evaluation, PortGPT and TSBPort remain comparatively strong, but performance declines sharply with structural, cross-scenario, localization, and patch-scale complexity. Static agreement also imperfectly reflects executable remediation, with failures concentrated in patch construction and dependency propagation and limited recovery from refinement.
- Aligned evaluation: 80.5% and 75.8% S-Succ: PortGPT and TSBPort remain comparatively high at commit level, while every tool falls below its original report.Mystique drops from 92.4% to 14.7%, and FixMorph from 75.1% to 28.1%.
- Porting difficulty: 24.0%: PortGPT’s commit-level S-Succ falls from 85.2% on Type-I to 24.0% on Type-IV patches.Type-IV also reaches 10.4% for TSBPort, 4.2% for Mystique, and 2.1% for PPatHF.
- Cross-scenario generalization: 70.7%, 63.1%, and 62.7%: PortGPT leads cross-branch, cross-version, and cross-repository cases, while cross-version remains especially difficult for other tools.The other tools’ best scenario-level commit S-Succ ranges from 19.5% to 27.2%.
- Localization and scale: 60.6%: PortGPT’s I0 localization outputs hit the target function, and target-function hits reach 79.3% S-Succ.Although 86.3% of cases produce analyzable localization outputs, substantial end-to-end loss occurs before function alignment.
- Failure analysis: 45.0% and 39.2%: patch construction or localization failure and non-local dependency propagation failure are the most frequent root-cause categories in the 120-attempt coding sample.Missing target API awareness accounts for 8.3%, while cross-version semantic mismatch accounts for 6.7%.
- Executable validation: 36/45 to 37/45 (+2.2 pp): executable-feedback refinement raises PortGPT’s Full validation on the eligible cases.Reference-based S-Succ and Full rates both aggregate to 36/45 before refinement, but two cases differ individually.
5 Discussion
The discussion emphasizes threats to validity in reference-based evaluation, limited executable evidence, and uncertain generalizability. It also notes dataset, execution, and temporal limitations affecting interpretation of the results.
- Threats to validity: Reference-based EM and S-Succ may under-credit structurally divergent correct fixes, while ManualVerify covers only non-exact functions already passing L2–L5.PPR/TPR are reported as complementary executable evidence rather than replacements for reference agreement.
- Threats to validity: Executable findings and feedback gains are limited to 45 cases with reproducible POCs.This scope constrains how broadly executable-validation conclusions can be interpreted.
- Limitations: The results may not generalize beyond C/C++ security patches drawn from over 50 projects.The benchmark’s project and language coverage therefore limits external validity.
- Limitations: One oracleblocked and one incomplete case count as failures among the 45 cases but cannot be refined, and FixMorph is excluded from executable comparison.FixMorph did not run on newer kernels in the study setup; a temporal cutoff also cannot rule out LLM training-data contamination.
- RQ5 evaluation: Base Full uses corrected RQ3, while runner and materialization corrections are not RQ5 gains; eligible counts and rows may overlap.Stage+ may overlap with Δ Full, so RQ5 gain accounting requires these qualifications.
6 Related Work
Prior work spans program analysis, LLM prompting, and LLM agents for automated patch backporting, but heterogeneous datasets and scenarios hinder direct comparison. This benchmark distinguishes security patch backporting from related software engineering benchmarks and automated program repair.
- Automated Patch Backporting: Program-analysis, LLM-prompting, and LLM-agent tools study automated patch backporting, while PatchWeave, SKYPORT, and MigGPT target narrower transplantation or migration scenarios.These tools use different datasets and scenarios, making direct comparison difficult; LLMPort was not evaluated because its implementation was not public.
- Software Engineering Benchmarks: Defects4J and ManyBugs provide reproducible defects and test suites, whereas Big-Vul and CVEfixes organize vulnerability-fix data for mining tasks.BackportBench uses execution outcomes but studies multilingual package-level backporting rather than security patch backporting for C/C++.
- Software Engineering Benchmarks: BackportBench does not report results by backporting scenario or study refinement with execution feedback, both of which this benchmark evaluates.This distinguishes the benchmark’s evaluation scope from prior execution-based multilingual package-level backporting studies.
- Automated Program Repair: Unlike automated program repair, patch backporting starts from a known fix and preserves its repair intent across target-side syntactic, structural, and dependency changes.APR creates a fix from tests or other fault evidence, whereas this task adapts a supplied source patch to a changed target.
7 Conclusion
Porting Benchmark shows that automated security patch backporting fails mainly on structurally difficult target adaptations, especially Type-III/IV and cross-version cases. Static agreement can diverge from executable remediation, while executable feedback provides limited gains and stage-level progress.
- Conclusion: Automated backporting performance drops sharply on structurally hard Type-III/IV and cross-version target adaptations.These scenarios are identified as the main sources of failure in Porting Benchmark.
- Conclusion: Static agreement does not always reflect whether a backported patch achieves executable remediation.Reference-based evaluation can therefore diverge from practical remediation outcomes.
- Conclusion: Executable feedback yields a small Full-validation gain while also producing progress at intermediate stages.The improvement is limited but measurable in the benchmark’s executable evaluation.