PaperlayerView exact revision on arXiv

Source-linked AI summary

COMPASS: Predicting the Relationship of Multiple Patches for Vulnerabilities with LLMs

Yi Song, Dongchen Xie, Xiaoyuan Xie, He Zhang, Lin Xu, Chunying Zhou, Zhi Jin

arXiv:2609.39783v1cs.SE

TL;DR

Multiple patches for a single vulnerability complicate downstream adoption because different patch relationships call for different strategies. COMPASS uses an LLM-based pipeline and a taxonomy of six recurring relationships to predict patch relationships. On a 300-CVE benchmark, it outperforms compact learning-based and LLM-only baselines by an average of 85.04%.

  • Problem

    Multiple patches may fix one vulnerability, and their differing relationships complicate downstream adoption because maintainers must determine how to apply them.

    Source: §I · pp. 1–2

  • Method

    COMPASS derives six recurring patch-relationship types from observations and developer validation, then uses an LLM pipeline to predict relationships among a vulnerability’s patches.

    Source: §I · p. 2, D. Phase-3: Patch Relationship… · p. 6

  • Results

    85.04% average improvement over compact learning-based and LLM-only baselines on a benchmark of 300 multi-patch CVEs.

    Source: §IX · p. 11

  • Takeaways & Limitations

    The released benchmark and online querying website support community reuse of patch-relationship knowledge.

    Source: §IX · p. 11

    Effectiveness may vary across ecosystems or projects; larger-scale and industrial validation is still needed.

    Source: §VII · p. 10

Abstract

from arXiv · show

Modern software heavily relies on code reuse, so upstream vulnerability fixes do not automatically propagate to downstream codebases. Downstream maintainers must manually adopt patches to eliminate known risks. In practice, a single vulnerability often corresponds to multiple patches, which greatly complicates downstream patch adoption because different patch relationships imply different adoption strategies. To address this challenge, we first manually inspect large-scale multi-patch vulnerabilities (about 1K) in the real world and interview experienced developers, summarizing six typical types of patch relationships, i.e., Merge, Mirror, Better Solution, Fixing-of-Fixing, Collaboration, and Separation. Based on these observations, we propose COMPASS, an automated approach that predicts the relationships of multiple vulnerability patches with large language models. Given a CVE as input, COMPASS follows a four-phase pipeline that (i) identifies the patch group and pre-scans explicit relationships, (ii) performs individual patch analysis, (iii) infers relationship instances via a hierarchy-guided prompt, and (iv) validates completeness and consistency of the inferred results. As output, COMPASS reports the predicted relationships within the patch group and visualizes them as a relationship graph. We evaluate COMPASS on a benchmark of 300 multi-patch CVEs and compare it against mainstream learning-based and LLM baselines. Results show that our method achieves strong and consistent prediction effectiveness and outperforms SOTA by 85.04% on average. We publicly release an online querying website to support community reuse of patch relationships knowledge: https://patch-relation.com.

I. INTRODUCTION

Multiple patches for one vulnerability complicate downstream adoption because their relationships imply different strategies, while manual analysis is costly and difficult to scale. COMPASS uses an LLM-based pipeline to predict those relationships, supported by a six-type taxonomy and a 300-entry benchmark.

I. INTRODUCTION

  • Different patch relationships require different adoption strategies, yet downstream maintainers must analyze patches manually before deciding how to adopt them.Code copying, independent evolution, and project-specific customization can make upstream patches inapplicable or ambiguous.Source: §I · pp. 1–2
  • COMPASS combines six recurring relationship types with a four-phase LLM pipeline to predict and visualize relationships within a vulnerability’s patch group.The phases identify patches and explicit relationships, analyze patches individually, infer relationships, and check completeness and consistency.Source: §I · p. 2
  • The taxonomy was derived from inspection of 1,050 multi-patch vulnerability entries and interviews with 20 experienced developers.The six types are Merge, Mirror, Better Solution, Fixing-of-Fixing, Collaboration, and Separation.Source: §I · p. 2
  • The evaluation benchmark contains 300 real-world multi-patch vulnerabilities and is disjoint from the 1,050 entries used to construct the taxonomy.The benchmark’s patch relationships were manually analyzed and labeled.Source: §I · p. 2
  • The authors provide a public querying website and a tool that predicts and visualizes patch relationships for disclosed vulnerabilities.The benchmark is also publicly available for future research.Source: §I · p. 2

II. TYPICAL RELATIONSHIPS OF PATCHES

The paper derives a practical taxonomy of patch relationships from real-world cases and developer feedback. Its Merge type captures branch-level fixes consolidated into a unified fix.

A. Taxonomy Construction

  • The authors derived candidate relationship patterns by examining 1,050 multi-patch vulnerability entries from CVE, NVD, and Snyk, then refined them with developer feedback.The 20 interviewees had at least seven years of relevant open-source, security-maintenance, or vulnerability-fixing experience.Source: A. Taxonomy Construction · pp. 2–3
  • The resulting taxonomy contains six recurring, actionable types intended to cover dominant practical patterns, not every possible relationship.The types are Merge, Mirror, Better Solution, Fixing-of-Fixing, Collaboration, and Separation.Source: A. Taxonomy Construction · p. 3

B. “Merge” Relationship

  • Merge describes related fixes from separate branches being integrated into one branch, where the merge patch consolidates their effects into a unified vulnerability fix.Recognizing an applicable merge patch can spare downstream maintainers from separately reasoning about each merged patch.Source: B. “Merge” Relationship · p. 3

C. “Mirror” Relationship

The paper distinguishes equivalent fixes on separate branches from refinements to an earlier fix and later patches that correct flaws introduced by that fix. These distinctions imply different choices about which patches to adopt and when.

C. “Mirror” Relationship

  • Mirror describes functionally equivalent fixes independently applied to different versions or branches, often through cherry-picking, so maintainers can select the patch matching their target branch.Minor code differences may reflect branch context while the fixing logic remains equivalent.Source: C. “Mirror” Relationship · p. 3

D. “Better Solution” Relationship

  • Better Solution describes a later patch that improves readability, maintainability, or performance without changing the earlier patch’s vulnerability-remediation mechanism.The later patch typically depends on the earlier one; the initial fix is usually sufficient, while the refinement may be optional.Source: D. “Better Solution” Relationship · pp. 3–4

E. “Fixing-of-Fixing” Relationship

  • Fixing-of-Fixing describes a later patch correcting a flaw introduced by the initial vulnerability fix, with both patches interdependent.They should be applied in order and typically included in the same release to avoid an unsafe intermediate state.Source: E. “Fixing-of-Fixing” Relationship · p. 4

F. “Collaboration” Relationship

Collaboration and Separation differ in whether patches jointly complete a fix or independently address distinct aspects. COMPASS organizes prediction as a phased process that includes relationship reasoning and a validation check.

F. “Collaboration” Relationship

  • Collaboration describes complementary patches that jointly fix a vulnerability, with later patches building on an earlier foundation; all are needed for a complete fix.Unlike Fixing-of-Fixing, intermediate collaboration patches are incomplete but do not introduce new flaws, so they may be released separately when necessary.Source: F. “Collaboration” Relationship · p. 4

G. “Separation” Relationship

  • Separation describes independent patches for distinct vulnerability aspects or replicated vulnerable patterns, with no code-level interdependence and no required application order.The six relationship types capture recurring patterns with downstream adoption implications, rather than an exhaustive taxonomy.Source: G. “Separation” Relationship · p. 4

A. Overview

  • COMPASS extracts a vulnerability’s patch group, analyzes patches individually, predicts relationships using a hierarchy-guided prompt, and checks predictions for completeness and contradictions.The overview describes a four-phase pipeline, with three hierarchies used to organize the six relationship types during prediction.Source: A. Overview · pp. 4–5

B. Phase-1: Patch Group Extraction & Pre-Scanning

Phase-1 collects a vulnerability’s patch group and resolves explicit relationship cues when available; Phase-2 structures individual patch inspection to support later relationship reasoning.

B. Phase-1: Patch Group Extraction & Pre-Scanning

  • Phase-1 obtains linked patches from public vulnerability entries and pre-scans commit messages for explicit relationship cues, which are rare in practice.Terms such as “merge” or “improve performance” can indicate relationships; most cases proceed to the LLM-based phases.Source: B. Phase-1: Patch Group Extraction &… · p. 5

C. Phase-2: Individual Patch Analysis

  • Phase-2 supplies vulnerability details and each patch’s identifiers, messages, branch/version information, and code changes to the LLM for individual analysis.It examines vulnerability alignment, key changes, and implicit message or version signals, producing internal observations rather than relationship predictions.Source: C. Phase-2: Individual Patch Analysis · pp. 5–6
  • Patch-vulnerability alignment, changed components, and implicit signals provide evidence for subsequent relationship reasoning.Alignment can help identify dependency-oriented relationships, while changed locations support reasoning about overlaps, dependencies, or patch individuality.Source: C. Phase-2: Individual Patch Analysis · pp. 5–6

D. Phase-3: Patch Relationship Prediction

Phase-3 prompts the LLM to infer unresolved relationship instances using a hierarchy guided by the six observed relationship patterns. The hierarchy moves from structural similarity to dependency reasoning and then to cases without identified dependency.

D. Phase-3: Patch Relationship Prediction

  • Phase-3 represents unresolved relationships as instances, with patch roles encoded for five relationship types and each Separation patch represented independently.The instance representation records subject and object patches where applicable; Separation has no subject patch.Source: D. Phase-3: Patch Relationship… · p. 6
  • The three hierarchies address structural composition or high code similarity, dependency-based reasoning, and patch sets with no identified dependency.This orders prediction from clearer structural evidence through dependency analysis to non-dependent cases.Source: D. Phase-3: Patch Relationship… · p. 6

1) Hierarchy-1 (Merge and Mirror):

  • Hierarchy-1 checks whether one patch consolidates related changes or whether patches implement similar fixes across versions or branches.Merge analysis distinguishes the consolidated patch from merged patches; Mirror analysis considers code and commit-message similarity.Source: 1) Hierarchy-1 (Merge and Mirror) · p. 6

2) Hierarchy-2 (Better Solution, Fixing-of-Fixing, Collaboration):

  • Hierarchy-2 uses code dependencies and semantic intent to distinguish refinement, correction of an earlier fix, and complementary contributions to a complete fix.The prompt asks whether later patches build on earlier code or whether application order affects effectiveness.Source: 2) Hierarchy-2 (Better Solution,… · p. 6

3) Hierarchy-3 (Separation):

  • When no dependency is identified, Hierarchy-3 checks whether patches independently address vulnerability aspects or replicated vulnerable patterns, classifying supported cases as Separation.The Phase-3 prompt also includes one-shot examples for all six relationship types, drawn from CVEs outside the benchmark.Source: 3) Hierarchy-3 (Separation) · p. 6

E. Phase-4: Validation Check

Phase-4 checks that inferred relationship instances cover the relevant patches and obey hierarchy-specific labeling constraints. The running example shows COMPASS validating and visualizing its inferred relationships for CVE-2013-5029.

E. Phase-4: Validation Check

  • Phase-4 checks patch coverage and consistency with the hierarchy-specific constraints on relationship labels.Patches identified as not contributing to the fix are exempt from participation; Hierarchy-2 instances may carry multiple labels, unlike instances in the other hierarchies.Source: E. Phase-4: Validation Check · p. 6
  • The validated relationship instances are visualized as a graph to support downstream understanding and patch adoption.Figure 2 is identified as showing the patch relationships for CVE-2013-5029.Source: E. Phase-4: Validation Check · p. 7, Figure 2 · p. 7

F. Running Example

  • For CVE-2013-5029, COMPASS excludes a ChangeLog-only commit from remediation and infers Collaboration and Fixing-of-Fixing instances among the remaining patches.The example describes an initial protection patch, a subsequent hiding strategy, and a later patch conditioning that strategy to avoid unintended blocking; validation checks coverage and contradictory relationships before graph visualization.Source: F. Running Example · p. 7

IV. EXPERIMENTAL SETUP

The experimental setup compares four LLMs running COMPASS with learning-based and LLM-only baselines, and examines selected design choices through ablations. The learning-based baselines cast relationship prediction as supervised multi-label classification over patch pairs.

1) RQ1: How effective are different LLMs when running COMPASS:

2) RQ2: How does COMPASS perform compared with learning-based and LLM-only baselines:

  • RQ2 compares COMPASS with fine-tuned pre-trained-model baselines and an LLM-only prompt baseline without COMPASS’s structured pipeline.The paper states that no existing technique specifically targets multiple vulnerability-patch relationship prediction.Source: 2) RQ2: How does COMPASS perform… · p. 7

3) RQ3: How do key designs affect COMPASS’s effectiveness:

  • The study ablates Phase-3 one-shot prompting as Czero-shot to quantify its gain, and removes preparatory Phase-2 and post-hoc validation Phase-4 as Cpred-only to test their benefits beyond Phase-3.Source: 3) RQ3: How do key designs affect… · p. 7

1) Learning-based baselines:

  • Learning-based baselines fine-tune CodeReviewer, CodeT5, and BERT as multi-label classifiers over patch pairs, using vulnerability descriptions, commit information, and code changes.The setup uses 10-fold cross-validation with disjoint test folds and an 8:1:1 train-validation-test split in each fold.Source: 1) Learning-based baselines · pp. 7–8

2) LLM-only baseline:

The DS-only baseline tests whether COMPASS’s gains come from its structured pipeline rather than the LLM alone. The evaluation uses a disjoint 300-CVE benchmark, expert annotations, and type-aware precision, recall, and F1-score.

2) LLM-only baseline:

  • DS-only gives DeepSeek-v3 the vulnerability, patch, and relationship-type information while removing COMPASS’s structured analysis, hierarchy-guided reasoning, and validation.It is designed to isolate whether effectiveness gains come from the underlying LLM or COMPASS’s structured pipeline.Source: 2) LLM-only baseline · p. 8

C. Datasets

  • The benchmark contains 300 randomly selected multi-patch CVEs from 2009–2024, spans 213 repositories, and is disjoint from the 1,050 taxonomy cases.The entries come from CVE, NVD, and Snyk.Source: C. Datasets · p. 8
  • Two experts independently labeled 473 relationship instances across 694 patches; Cohen’s Kappa was 87.98%, with disagreements resolved by a third expert.Each CVE has 2–7 patches, and the labels follow the relationship guidelines.Source: C. Datasets · p. 8

D. Metrics

  • Precision, recall, and F1-score are computed per relationship type by matching predicted instances against expert annotations.Matching is type-aware: some relations require identical key-value pairs, Mirror can match by patch set, and Separation by patch ID.Source: D. Metrics · p. 8

V. RESULT AND ANALYSIS

COMPASS performs best overall against the learning-based and LLM-only baselines, with DeepSeek-v3 yielding the strongest results among the tested backbones. Its advantage is especially apparent for dependency-driven relationships, while soft-prompt variants remain weak on those types.

A. RQ1: Effectiveness of different LLMs

  • DeepSeek-v3 achieves F1-scores of 80.49% Merge, 87.36% Mirror, 62.50% Better Solution, 69.23% Fixing-of-Fixing, 63.64% Collaboration, and 82.86% Separation.It attains the highest Recall and F1-Score across all six relationship types under the same COMPASS pipeline.Source: A. RQ1: Effectiveness of different LLMs · p. 8
  • Gemini-2.5 has the highest Merge Precision (88.89%), while Claude-3.7 leads Fixing-of-Fixing Precision (78.95%); DeepSeek-v3 leads Recall and F1 across types.The paper notes that precision advantages do not necessarily yield the best overall effectiveness when Recall is considered jointly.Source: A. RQ1: Effectiveness of different LLMs · p. 8
  • The structured pipeline supports relationship inference across different LLM backbones, and DeepSeek-v3 is selected as COMPASS’s default model for the remaining research questions.The paper attributes the gains not merely to the LLM, but also to COMPASS’s structured pipeline.Source: A. RQ1: Effectiveness of different LLMs · p. 8, B. RQ2: Comparison with baseline… · p. 9

B. RQ2: Comparison with baseline techniques

  • COMPASS F1-scores are 62.50% Better Solution, 69.23% Fixing-of-Fixing, and 63.64% Collaboration; best-baseline F1-scores are 59.17% Merge, 76.12% Mirror, and 61.22% Separation.For Merge, Mirror, and Separation, COMPASS scores 80.49%, 87.36%, and 82.86%, respectively.Source: B. RQ2: Comparison with baseline… · p. 9
  • 85.04% average relative F1-score improvement over the best baseline per relationship type; COMPASS also achieves the best Recall, Precision, and F1-Score.The comparison includes learning-based and LLM-only baselines.Source: B. RQ2: Comparison with baseline… · pp. 8–9, Table II · p. 9
  • Soft-prompt variants show occasional gains but remain weak on Hierarchy-2: their best F1-scores are 5.30% Better Solution, 18.60% Fixing-of-Fixing, and 13.56% Collaboration.Examples of mixed results include BERT-s improving Fixing-of-Fixing Recall from 28.00% to 52.00%, while C-T5-s Fixing-of-Fixing Precision drops from 17.02% to 13.11%.Source: B. RQ2: Comparison with baseline… · p. 9

C. RQ3: Ablation of key designs

Ablations show that one-shot demonstrations and COMPASS’s supporting phases contribute to relationship inference, especially for dependency-driven types. COMPASS also recovers correct instances missed by selected baselines, although some baseline-only cases remain.

C. RQ3: Ablation of key designs

  • Removing supporting phases lowers Collaboration F1-Score from 63.64% to 36.11% and Separation F1-Score from 82.86% to 51.08%.The authors attribute these losses to the preparatory patch understanding and post-hoc validation provided by Phase-2 and Phase-4.Source: C. RQ3: Ablation of key designs · p. 9, Table III · p. 10
  • Removing one-shot demonstrations lowers F1-Score for Better Solution from 62.50% to 52.94%, Fixing-of-Fixing from 69.23% to 59.64%, and Collaboration from 63.64% to 32.14%.The ablation compares COMPASS with Czero-shot, which removes Phase-3 one-shot demonstrations.Source: C. RQ3: Ablation of key designs · p. 9, Table III · p. 10
  • Czero-shot slightly raises RecallMirror from 87.02% to 87.79% and PrecisionSeparation from 90.95% to 91.22%, while F1-Scores decrease.The authors describe these exceptions as reflecting a more aggressive or conservative bias rather than better overall instance inference.Source: C. RQ3: Ablation of key designs · p. 10

VI. DISCUSSION

  • COMPASS recovers 380 annotated instances versus 235∼277 for selected baselines, including 40 instances correctly recovered only by COMPASS.The UpSet plot also contains non-empty baseline-only intersections, indicating that baselines recover some instances COMPASS misses.Source: §VI · p. 10

VII. THREATS TO VALIDITY

The authors identify subjective judgment, possible data leakage, and limited ecosystem coverage as threats, while reporting steps to mitigate or bound them. They position COMPASS as instance-level relationship inference beyond prior patch identification and characterization, and call for broader patterns and larger-scale industrial evaluation.

VII. THREATS TO VALIDITY

  • Taxonomy and manual labels may involve subjective judgment; mitigation included practitioner feedback, a disjoint benchmark, independent expert annotation, and Cohen’s Kappa of 87.98%.The authors also describe data leakage as a potential risk, but say it is extremely slim because task-critical relationship labels were newly created and are not publicly available.Source: §VII · p. 10
  • The benchmark samples 300 CVEs from CVE, NVD, and Snyk across 213 repositories and 2009∼2024, but larger-scale and industrial validation is still needed.The authors note that effectiveness may vary across ecosystems or projects.Source: §VII · p. 10

VIII. RELATED WORK

  • Prior work identified or characterized multiple patches, whereas COMPASS infers instance-level relationships within a patch group to guide downstream adoption.The related-work discussion cites Li et al., Tan et al., Hommersom et al., Woo et al., and Xu et al..Source: §VIII · p. 10

IX. CONCLUSION

  • COMPASS predicts instance-level relationships under six recurring types and outperforms compact learning-based and LLM-only baselines on 300 multi-patch CVEs, with an average improvement of 85.04%.The authors also release the benchmark and an online querying website for community reuse.Source: §IX · pp. 10–11
  • Future work will explore broader relationship patterns and evaluate COMPASS in larger-scale and industrial downstream patch adoption scenarios.Source: §IX · p. 11
Loading 2609.39783v1…