PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMxCPG: Context-Aware Vulnerability Detection Through Code Property Graph-Guided Large Language Models

Ahmed Lekssays, Hamza Mouhcine, Khang Tran, Ting Yu, Issa Khalil

arXiv:2507.16585v1cs.CR

TL;DR

Detecting vulnerabilities in complex codebases remains challenging. LLMxCPG combines Code Property Graphs with large language models, reporting F1-score increases of up to 40% and consistent performance on rigorously verified datasets.

  • Problem

    Identifying vulnerabilities in complex codebases remains challenging despite the persistent security risks they pose.

    Source: §1 · p. 1

  • Method

    LLMxCPG combines Code Property Graphs and large language models, using CPGQL queries to extract vulnerability-focused slices before classifying them.

    Source: §3.1 · p. 3, §3.2 · p. 4

  • Results

    Up to 40% F1-score increases over existing approaches accompany consistent performance on rigorously verified datasets.

    Source: §7 · p. 14

  • Takeaways & Limitations

    LLMxCPG is a promising foundation for automated vulnerability detection across code transformations and complex, multi-function codebases.

    Source: §7 · p. 14

    Samples with nesting depth above 7 levels show performance degradation, with accuracy dropping to 0.33.

    Source: §4.4.2 · p. 11

Abstract

from arXiv · show

Software vulnerabilities present a persistent security challenge, with over 25,000 new vulnerabilities reported in the Common Vulnerabilities and Exposures (CVE) database in 2024 alone. While deep learning based approaches show promise for vulnerability detection, recent studies reveal critical limitations in terms of accuracy and robustness: accuracy drops by up to 45% on rigorously verified datasets, and performance degrades significantly under simple code modifications. This paper presents LLMxCPG, a novel framework integrating Code Property Graphs (CPG) with Large Language Models (LLM) for robust vulnerability detection. Our CPG-based slice construction technique reduces code size by 67.84 to 90.93% while preserving vulnerability-relevant context. Our approach's ability to provide a more concise and accurate representation of code snippets enables the analysis of larger code segments, including entire projects. This concise representation is a key factor behind the improved detection capabilities of our method, as it can now identify vulnerabilities that span multiple functions. Empirical evaluation demonstrates LLMxCPG's effectiveness across verified datasets, achieving 15-40% improvements in F1-score over state-of-the-art baselines. Moreover, LLMxCPG maintains high performance across function-level and multi-function codebases while exhibiting robust detection efficacy under various syntactic code modifications.

1 Introduction

LLMxCPG combines Code Property Graphs with large language models to address gaps in vulnerability detection, including limited inter-procedural context and robustness. It reports improved results across verified datasets, complex codebases, and code transformations.

Motivation

  • Prior deep learning approaches often miss inter-procedural dependencies, degrade on rigorously verified data and simple code changes, and face limited context windows.Accuracy dropped by up to 45% on a rigorously verified dataset compared with reported results on traditional datasets.Source: §1 · p. 1

Approach

  • LLMxCPG combines Code Property Graph analysis with large language models, using vulnerability-focused slices to address codebase context constraints.The approach extracts potential vulnerable paths with Joern and CPGQL, traverses the graph to find interacting code elements, then applies backward slicing.Source: §1 · pp. 1–2
  • The focused slices are used to fine-tune a large language model to recognize vulnerability patterns in concise, relevant code contexts.This targets learning beyond superficial patterns.Source: §1 · p. 2

Results

  • 15%-40% F1-score increases and 9-27% Accuracy increases over state-of-the-art baselines on function-level real-world vulnerability detection tasks accompany consistent verified-dataset performance, generalization, and transformation robustness.LLMxCPG also performs effectively on function-level and multi-function codebases, including scenarios where existing approaches do not exceed random-guess performance.Source: §1 · p. 2

Reproducibility

  • The study makes its source code and datasets openly available to support reproducibility, transparency, and further security research.Source: §1 · p. 2

2 Background

Vulnerability detection uses static and dynamic analysis, with machine learning approaches seeking to improve detection amid challenges in complex codebases. Code Property Graphs unify code structure, control flow, and dependencies for graph-based vulnerability analysis.

2.1 Vulnerability Detection

  • Static analysis examines code without execution and can offer broad coverage, but may produce false positives because it cannot verify runtime behavior.Dynamic analysis observes execution but faces challenges with comprehensive coverage and the growth of execution paths.Source: §2.1 · p. 2

2.2 Code Property Graphs

  • Code Property Graphs combine syntax, control flow, and program dependencies, enabling graph traversals to express complex vulnerability patterns.The paper uses Joern to generate CPGs and query them with CPGQL.Source: §2.2 · pp. 2–3

3 Methodology

LLMxCPG combines Code Property Graph analysis with large language models in a two-stage process: extracting vulnerability-focused code slices and classifying them as vulnerable or safe.

3.1 System Overview

  • LLMxCPG uses separate models for slice construction and classification: LLMxCPG-Q generates CPGQL queries, and LLMxCPG-D classifies extracted slices.The two models are fine-tuned from Qwen2.5-Coder-32B-Instruct and QwQ-32B-Preview, respectively.Source: §3.1 · p. 3

3.2 Slice Construction

  • Joern and CPGQL identify potentially vulnerable execution paths; the final slice adds interacting variables and code influencing either the path or those variables.This focuses on essential components rather than analyzing entire multi-function or multi-file codebases.Source: §3.2 · p. 4

3.2.1 Taint Path Extraction

  • LLMxCPG-Q generates CPGQL queries to identify vulnerability-relevant sources, sinks, and execution paths between them.The query-generation model is fine-tuned using generated queries that are tested on Joern and iteratively refined when they fail or contain syntax errors.Source: §3.2.1 · p. 5

3.2.2 Interacters: Finding Variables that Interact with the Execution Path

  • Interacters are identifiers sharing a line with an extracted path element; finding them adds variables needed to interpret the path.For example, the execution path alone can lack a constant definition needed to assess security implications.Source: §3.2.2 · p. 5

3.2.3 Backward Slicing for Focused Code Snippet Construction

  • Backward slicing gathers code influencing the execution path or its interacters, including relevant definitions and control-flow relationships.The process is automated with a CPGQL query; Joern uses the Program Dependency Graph to construct the backward slice.Source: §3.2.3 · pp. 5–6

3.3 Vulnerability Detection

  • The 85-line code function shown in Figure 2 can be compressed to an 18-line snippet using CPGQL queries, isolating security-relevant code patterns and eliminating non-essential code.LLMxCPG-D is fine-tuned to classify code slices as vulnerable or safe. The same approach reduces noise and highlights security-relevant modifications in patched code.Source: §3.3 · p. 6

4 Evaluation

The evaluation assesses LLMxCPG across function- and project-level datasets, including unseen and post-knowledge-cutoff cases, and examines query quality, code reduction, threshold calibration, errors, and robustness to code transformations. Results characterize its performance and generalization while identifying sensitivities to deeply nested code, transformation type, and precision–recall shifts.

4.1 Datasets

  • Evaluation uses FormAI-v2 and PrimeVul for training and SVEN and ReposVul for generalization, focusing on memory-related CWEs that static CPG analysis can model.The datasets include verified synthetic code, real-world function snippets, and project-level CVE samples; dynamic, runtime-dependent vulnerabilities such as race conditions are excluded.Source: §4.1 · pp. 6–7

4.2 Implementation

  • Thresholds calibrated on 20 labeled validation examples varied by dataset, from γ = 0.594 for PrimeVul to γ = 0.193 for ReposVul.The authors recommend calibrating thresholds with a small labeled sample from a new target domain, and report higher thresholds for training datasets than unseen datasets.Source: §4.2 · p. 8

4.3.1 Query Generation

  • Experts found that 76% of audited true positive/negative queries matched vulnerability patterns semantically, with Fleiss’ Kappa of 0.6429.Among false positive/negative cases, 28% of queries were semantically correct despite incorrect labels, 40% targeted a different CWE, and 32% missed critical context; most classification errors arose when the query model missed the target pattern.Source: §4.3.1 · p. 9
  • Slice construction reduced code size by 78.70% on FormAI, 67.84% on PrimeVul, 70.22% on SVEN, and 90.93% on ReposVul.The paper evaluates performance on these reduced slices in the subsequent analysis.Source: §4.3.1 · p. 9

4.3.2 Function-level Vulnerability Detection

  • On FormAI, LLMxCPG reached 0.8146 Accuracy and 0.8075 F1-score, with 0.941 accuracy and 0.97 F1-scores for CWE-119.The reported function-level results also showed strong performance on other CWEs, without specifying their values in the supplied passages.Source: Table 3 · p. 10, §4.3.2 · p. 10, Table 4 · p. 10

4.4.1 Function-level Vulnerability Detection

  • LLMxCPG improved accuracy by 20% over competing approaches on the unseen SVEN dataset.The benchmark used real-world code snippets that were not part of the training data of LLMxCPG or the considered baselines.Source: Table 5 · p. 10, §4.4.1 · p. 10

4.4.2 Project-level Vulnerability Detection

  • On ReposVul, LLMxCPG achieved 0.634 average Accuracy and 0.610 F1-score, and on post-cutoff CVEs it achieved 0.617 F1-score and 0.600 Accuracy.The post-cutoff evaluation used 57 CVEs paired with patches, yielding 114 balanced vulnerable and safe samples; accuracy fell to 0.33 on some samples with nesting depth above seven levels.Source: Table 7 · p. 11, §4.4.2 · p. 11

4.5 Misclassification Analysis

  • The misclassification analysis examines errors across FormAI, PrimeVul, ReposVul, and SVEN at each dataset’s optimal threshold, with performance reported by CWE category.The supplied passages describe the analysis scope and identify Table 8, but do not provide its findings.Source: §4.5 · pp. 11–12, Table 8 · p. 12
  • On trained FormAI and PrimeVul data, LLMxCPG achieves high accuracy and F1-scores for CWE-119, CWE-190, CWE-415, and CWE-416.The paper attributes this performance to balanced training representation and the model’s capture of distinct vulnerability code patterns.Source: §4.5 · p. 12
  • LLMxCPG struggles with CWE-120 and CWE-125, which are underrepresented in training and involve subtle variations and complex memory-access patterns.The paper links reduced effectiveness to both limited examples and the difficulty of capturing these patterns with limited training data.Source: §4.5 · p. 12
  • LLMxCPG achieves accuracy above 0.6 on both unseen ReposVul and SVEN datasets, indicating generalization to diverse real-world codebases.The paper notes room for improvement and proposes a high-quality dataset spanning multiple files and functions, with verified synthetic augmentation.Source: §4.5 · p. 12

4.6 Robustness to Code Augmentation

  • Comment removal (T4) does not affect LLMxCPG because its slice construction ignores comments and focuses on execution paths.The constructed slices are comment-free by design.Source: §4.6 · p. 13
  • On FormAI, LLMxCPG’s F1-score increases by 2.10% under transformations, its highest robustness across the evaluated datasets.The robustness experiments apply four semantics-preserving transformations to code slices from FormAI, PrimeVul, and SVEN.Source: §4.6 · pp. 12–13
  • Function extraction (T3) has the greatest transformation impact, particularly on recall, indicating sensitivity to changes in function boundaries.Source: §4.6 · p. 13
  • For PrimeVul, transformations decrease precision but improve recall, producing a precision–recall trade-off in vulnerability predictions.The paper describes the model as becoming more conservative under transformed code.Source: §4.6 · p. 13

5 Discussion

The discussion identifies boundaries in CPG coverage, dataset quality, reasoning supervision, and query-generation context, while noting early project-level results and the need for better project-level data.

Limitations in Vulnerability Type Coverage

  • CPG traversals remain limited for race conditions and design errors, which can depend on runtime behavior or system intent beyond static graph analysis.The discussion attributes this boundary to CPGs’ static nature.Source: §5 · p. 13

Dataset Quality and Availability

  • Only 38%-64% of functions labeled vulnerable in popular datasets contained security flaws, while unrelated changes in fixing commits can introduce noisy labels.The discussion cites PrimeVul’s analysis and notes that ReposVul highlights noise from automated collection.Source: §5 · p. 13

Project-level Vulnerability Detection

  • 0.634 accuracy on ReposVul indicates project-level promise, but performance varies with complexity and the authors call for curated project-level vulnerability datasets.LLMxCPG was primarily trained on function-level data; the discussion frames improved project-wide analysis as a future research direction.Source: §5 · p. 13

Limitations of Binary Classification

  • Reasoning traces did not improve accuracy over binary classification, which the authors attribute likely to trace quality and quantity; effective transfer cited 800k datapoints [13].Enhancing vulnerability reasoning with high-quality traces remains future work.Source: §5 · p. 13

Context Length Limitation of the Base LLM for Query Generation

  • LLMxCPG-Q was fine-tuned with a 32K-token context because of compute constraints, making query generation most effective for codebases or commit files fitting that limit.The limitation is specific to the query model; CPG slicing kept detection inputs manageable, including with an 8K context for LLMxCPG-D.Source: §5 · p. 14

6 Related Work

Related work progresses from deep-learning and graph-based vulnerability detectors to LLM methods and hybrid architectures. LLMxCPG distinguishes itself by using LLMs to generate CPGQL queries that guide graph traversal.

Deep Learning Approaches

  • Early deep-learning work moved from LSTM code-gadget modeling [22] and semantic feature extraction toward transformers and graph-based approaches [23] [25].The discussion notes concerns that models can rely on superficial patterns [2].Source: §6 · p. 14

LLMs for Vulnerability Detection

  • LLM research spans domain-specific pretraining [14], multidimensional embeddings, fine-tuning [6] [41], and graph-attention hybrids.The discussion also identifies foundational code models incorporating structural information [11] [7].Source: §6 · p. 14
  • Unlike approaches learning directly from code or combining language models with graph neural networks, LLMxCPG uses LLMs to generate explicit CPGQL queries for graph traversal.The paper presents explicit query generation as preserving graph-analysis benefits while making the integration more interpretable and controlled.Source: §6 · p. 14

7 Conclusion

LLMxCPG combines Code Property Graphs with Large Language Models for vulnerability detection. The paper reports improved F1-scores, verified-dataset performance, robustness to code transformations, and generalization to multi-function codebases.

7 Conclusion

  • F1-score increases of up to 40% and consistent performance on rigorously verified datasets are reported for LLMxCPG.The conclusion describes substantial improvements over existing approaches.Source: §7 · p. 14
  • LLMxCPG combines Code Property Graphs with Large Language Models to address limitations in deep-learning-based vulnerability detection.The conclusion calls the approach a promising foundation for future software security analysis research.Source: §7 · p. 14
  • The conclusion reports robust detection under code transformations and generalization to complex, multi-function codebases.These outcomes are presented as an advancement in automated vulnerability detection.Source: §7 · p. 14

Ethics Considerations

The authors describe responsible disclosure practices and safeguards for security research. They report that discovered vulnerabilities are routed through appropriate channels while confidentiality and system integrity are maintained.

Ethics Considerations

  • The research follows responsible disclosure protocols and established security research guidelines.The authors describe balancing identification of weaknesses against potential risks.Source: Ethics Considerations · p. 15
  • Discovered vulnerabilities are reported through appropriate channels with time allowed for patches before public disclosure.The passage describes this as the disclosure process.Source: Ethics Considerations · p. 15
  • The authors state that they maintain confidentiality and ensure their methods do not compromise system integrity or user privacy.These are stated safeguards for the research.Source: Ethics Considerations · p. 15

Open Science

The authors make their source code, fine-tuned models, and testing datasets publicly available to foster research.

Open Science

  • The source code is publicly available.The paper gives GitHub and Zenodo links.Source: Open Science · p. 15
  • Fine-tuned models are publicly available.The paper gives GitHub and Zenodo links.Source: Open Science · p. 15
  • Testing datasets are publicly available to foster research.The paper gives GitHub and Zenodo links.Source: Open Science · p. 15

A Code Metrics

This section defines software metrics for assessing code size, branching, nesting, flow complexity, and modularity.

A Code Metrics

  • Lines of Code counts executable statements, declarations, and definitions, excluding blank lines and comments.The passage describes LOC as an indicator of implementation scale and maintenance burden.Source: A Code Metrics · p. 17
  • Cyclomatic complexity measures the number of linearly independent paths through source code using control-flow analysis.The passage identifies graph edges, nodes, and connected components as variables in its calculation.Source: A Code Metrics · p. 17
  • The section also describes a function-level expression for complexity based on decision points and their predicates.The passage names D as the decision-point set and p(d) as predicates at each decision point.Source: A Code Metrics · p. 17
  • Number of Functions counts declarations and definitions, including methods, procedures, and subroutines, as a modularity metric.The passage links this measure to code compartmentalization and potential maintenance complexity.Source: A Code Metrics · p. 17
  • Branch count B sums the program’s individual branching constructs, including conditionals, switch cases, and loop conditions.The passage denotes each branching construct as bi.Source: A Code Metrics · p. 17
  • Nesting depth is the maximum control-structure embedding level among statements in a code block.The passage defines S as the block’s statements and d(s) as each statement’s nesting level.Source: A Code Metrics · p. 17

B Comparison with Other Base Models

The authors fine-tuned four models and compared them on PrimeVul to choose the model used for final detection.

B Comparison with Other Base Models

C Impact of Slice Construction and Query Generation Model

The section evaluates the effect of omitting CPG slicing, describes threshold selection, and presents prompts for query generation and vulnerability detection.

C Impact of Slice Construction and Query Generation Model

  • Without slicing, accuracy was 0.6762 on FormAI (down from 0.8146), 0.4875 on PrimeVul (from 0.7250), and 0.5078 on SVEN (from 0.6020).The passage reports declines across all three test datasets when the model uses full code instead of sliced code.Source: C Impact of Slice Construction and… · p. 18
  • Joern-scan detected none of the vulnerable samples among the 50 semantic-correctness cases because it relies on fixed sensitive C function calls.The passage notes that custom wrappers around those functions can make detection more difficult for static query-based approaches.Source: C Impact of Slice Construction and… · p. 18

D Choosing a Threshold

  • Threshold selection uses a few labeled validation examples, then searches [0,1] for the value maximizing LLMxCPG-D accuracy.The authors used 20 labeled examples and show accuracy against threshold for different datasets in Figure 7.Source: D Choosing a Threshold · p. 18, Figure 7 · p. 18

E Prompt Templates

Related papers

Loading 2507.16585v1…