PaperlayerView exact revision on arXiv

Source-linked AI summary

GRAML: Graph-Grounded Reasoning and Multi-Task Learning for LLM-Based Software Vulnerability Detection

Ruohan Li, Miaoqing Tian, Haipeng Qu, Ruobing Jiang

arXiv:2610.09605v1cs.SE

TL;DR

LLM-based vulnerability detection often makes limited use of control-flow and data-flow information, leaving structural context underused in reasoning. GRAML extracts graph evidence, uses it to guide vulnerability-description generation and four-task training, and reports average F1 scores of 66.67% to 68.70%, outperforming state-of-the-art baselines by up to 30.92%.

  • Problem

    Existing LLM approaches make limited use of control-flow and data-flow information, and organizing structural relations into effective context for vulnerability reasoning remains an open problem.

    Source: §I · p. 1

  • Method

    GRAML extracts critical lines and typed graph relations, uses them to guide ToT-VR vulnerability-description generation, and combines descriptions with Detection, Localization, and Assessment for multi-task training.

    Source: §I · pp. 1–2, D. Multi-Task Fine-Tuning · p. 6

  • Results

    Average F1 scores range from 66.67% to 68.70%, outperforming state-of-the-art baselines by up to 30.92% across the evaluated models.

    Source: §I · p. 2

  • Takeaways & Limitations

    The results support integrating static program analysis, structured reasoning, and multi-task supervision for LLM-based vulnerability detection.

    Source: §VI · p. 15

Abstract

from arXiv · show

Large Language Models (LLMs) have been widely applied to software vulnerability detection. However, their performance is often limited by insufficient use of control-flow and data-flow information. In this paper, we propose GRAML, a framework that combines graph evidence, vulnerability description generation, and multi-task training. GRAML first performs static analysis on C/C++ programs to extract critical source lines and typed line relations as structural evidence. It then uses this evidence to guide GPT-5 through the Tree-of-Thought-guided Vulnerability Reasoning (ToT-VR) process and generate vulnerability descriptions. These descriptions are further combined with Detection, Localization, and Assessment samples to build a unified four-task training dataset. We evaluate GRAML on an in-distribution (ID) test set and six out-of-distribution (OOD) datasets. The results show that GRAML achieves average F1 scores ranging from 66.67% to 68.70%, outperforming state-of-the-art baselines by up to 30.92%. Ablation experiments further show that ToT-VR and graph-guided vulnerability descriptions improve detection performance compared with standard Chain-of-Thought (CoT) reasoning and raw Code Property Graph (CPG) serializations. These findings provide practical guidance for building more reliable and secure software engineering systems with large language models.

I. INTRODUCTION

GRAML addresses the limited use of control-flow and data-flow information in LLM vulnerability detection by extracting graph evidence, generating vulnerability descriptions, and training across four tasks. The paper reports strong average F1 performance across ID and OOD evaluation, while its overview presents the framework as a sequence from context extraction through inference.

I. INTRODUCTION

  • Existing LLM approaches make limited use of control-flow and data-flow information, leaving structural relations for downstream vulnerability reasoning an open problem.This can make it harder to trace a vulnerability from source to sink and identify its root cause.Source: §I · p. 1
  • GRAML uses vulnerable-line information and reliability-aware propagation to select critical source lines and typed data-dependence, control-dependence, and control-flow relations.The resulting graph evidence supports vulnerability-description generation.Source: §I · pp. 1–2
  • The training dataset combines Detection, Description, Localization, and Assessment tasks, using ToT-VR-generated descriptions for Description and joint training across all four tasks.The data combines BigVul, DiverseVul, and newly collected CVE samples reported between 2019 and 2026.Source: §I · pp. 1–2
  • Average F1 scores range from 66.67% to 68.70%, outperforming state-of-the-art baselines by up to 30.92% across the evaluated models.The evaluation covers an ID test set and six OOD datasets.Source: §I · p. 2
  • ToT-VR uses GPT-5 to generate vulnerability descriptions by constructing candidate analyses, pruning and refining them, and checking them against available evidence.The final description covers vulnerability cause, affected code regions, and potential security consequences.Source: §I · pp. 1–2

A. Overview

  • The framework overview sequences graph-structured context extraction, structured vulnerability-description generation, multi-task fine-tuning, and vulnerability inference.The context-extraction step builds graph-structured contexts from critical lines and their structural dependencies.Source: A. Overview · p. 2, Figure 2 · p. 3

B. Graph-Structured Context Extraction

GRAML builds a compact, line-level graph context around known vulnerable lines, then uses structural relevance and redundancy control to select contextual lines and retain their typed relations.

B. Graph-Structured Context Extraction

  • Patch changes or CVE records identify vulnerable lines, which are aligned with physical source lines for subsequent graph-context extraction.The line annotations are carried through the mask, CPG, relation matrices, and critical-line selection.Source: B. Graph-Structured Context Extraction · p. 3
  • The line-aligned CPG supplies directed data-dependence, control-dependence, and control-flow relations, while a mask excludes empty or comment-only lines during relation fusion.The three relation types are represented separately before being combined into a weighted directed matrix.Source: B. Graph-Structured Context Extraction · pp. 3–4
  • Relation fusion weights DDG, CDG, and CFG edges at 0.50, 0.30, and 0.20, then symmetrization and normalization support selection across incoming and outgoing connections while reducing degree effects.The directed weighted matrix is retained for relation records; its normalized counterpart is used for critical-line selection.Source: B. Graph-Structured Context Extraction · p. 4
  • Critical-line scoring combines one- and two-hop relevance with relation-type diversity, while penalizing candidates whose structural neighborhoods overlap those already selected.A candidate's diversity increases with the number of relation types linking it to selected lines; the redundancy penalty is zero when no line has yet been selected.Source: B. Graph-Structured Context Extraction · p. 4
  • Selection retains vulnerable lines alongside chosen contextual lines and their directed DDG, CDG, and CFG relations; when no usable CPG relation exists, only vulnerable lines remain.The resulting records preserve each selected line's number and code, and each relation's source, destination, direction, and type.Source: B. Graph-Structured Context Extraction · p. 5

C. Structured Vulnerability Description Generation

ToT-VR uses GPT-5 to develop and check multiple vulnerability interpretations with graph evidence and CVE descriptions, then synthesizes the retained branch into training text.

C. Structured Vulnerability Description Generation

  • ToT-VR starts with source code and up to 20 critical lines, then introduces vulnerable lines and typed relations during branch refinement and the official CVE description during verification.The intermediate graph records and reasoning branches are not serialized into the downstream fine-tuning input.Source: C. Structured Vulnerability… · pp. 5–6
  • GPT-5 initializes multiple candidate branches from the code and critical lines, with branches considering different trigger points or causal explanations.The candidates can represent distinct interpretations of the same function.Source: C. Structured Vulnerability… · p. 5
  • Branch reflection checks consistency with code semantics and removes weak or unsupported interpretations before structural evidence is introduced.In the running example, reflection retains Branch1 and Branch3 while pruning Branch2.Source: C. Structured Vulnerability… · p. 5
  • Selected vulnerable lines and typed relations refine retained branches toward precise trigger points and dependency paths without requiring raw graph serialization.In the example, structural refinement retains Branch1 and prunes Branch3.Source: C. Structured Vulnerability… · p. 6
  • The official CVE description verifies whether retained branches match known vulnerability semantics, after which GPT-5 synthesizes the retained branch into a description of cause, affected code, and potential security consequences.The synthesized description is the supervision target for the Description task; in the example, Branch1 passes verification.Source: C. Structured Vulnerability… · p. 6

D. Multi-Task Fine-Tuning

GRAML trains four tasks using a shared instruction format, with vulnerability detection as the primary task and description, localization, and assessment as additional supervision.

D. Multi-Task Fine-Tuning

  • Detection predicts vulnerability, Description explains its cause, Localization identifies affected lines, and Assessment estimates severity; Detection is the primary task.Nonvulnerable samples use N/A for Localization and Assessment, and a response indicating no vulnerability exists for Description.Source: D. Multi-Task Fine-Tuning · p. 6
  • Each training sample contains a task instruction, source-code input, and target response, with the instruction specifying the objective and expected output format.The model-specific chat structure and special tokens change, while the task instruction, source code, and target response remain unchanged.Source: D. Multi-Task Fine-Tuning · pp. 6–7
  • Fine-tuning uses a causal language-modeling objective with completion-only masking, so only target-response tokens contribute to the loss.The task instruction and source-code tokens are excluded from loss computation.Source: D. Multi-Task Fine-Tuning · p. 7

E. Vulnerability Inference

GRAML converts Yes/No logits into vulnerability probabilities and fixes thresholds selected on ID validation for evaluation. The evaluation spans one ID benchmark and six OOD datasets.

E. Vulnerability Inference

  • The model derives vulnerability probability by normalizing the logits for the “Yes” and “No” tokens, then classifies using the selected threshold.A sample is vulnerable when its probability is at least the selected threshold; otherwise it is non-vulnerable.Source: E. Vulnerability Inference · p. 7
  • Each model’s highest-F1 threshold is selected on ID validation and fixed for the ID test set and all six OOD benchmarks, without test-set threshold adjustment.Each baseline likewise selects its own threshold on the same ID validation set and keeps it fixed for evaluation.Source: E. Vulnerability Inference · p. 7

III. EVALUATION

  • Evaluation includes baseline comparisons, semantics-preserving code transformations, multi-task and graph-context ablations, ToT-VR, threshold robustness, and parameter sensitivity.Source: §III · p. 7

A. Datasets

  • The ID benchmark combines BigVul, DiverseVul, and CVE-report functions from 2019–2026; its training, validation, and test sets contain 21,600, 2,310, and 2,311 instances.Each split is approximately balanced between vulnerable and non-vulnerable instances.Source: A. Datasets · pp. 7–8
  • All four tasks are used in training, while validation and ID test contain only vulnerability-detection instances.Of 21,600 training instances, 19,137 are detection, 823 description, 820 localization, and 820 assessment samples.Source: A. Datasets · p. 8
  • CVEfixes, Devign, DiverseVul, PrimeVul [8], Juliet (C/C++), and ReVeal are OOD detection-only benchmarks, filtered to remove ID overlaps.The OOD benchmarks are used only to evaluate cross-dataset generalization.Source: A. Datasets · p. 8

B. Evaluation Metrics

F1 is the main detection metric because it accounts for both false alarms and missed vulnerabilities; Precision–Recall curves and AP add threshold-wide views. Across the seven test sets, GRAML outperforms the reported encoder, few-shot LLM, and VulLLM comparisons in average F1.

B. Evaluation Metrics

  • F1 is the main metric because it considers both false alarms and missed vulnerabilities.Precision reflects correct predictions among samples predicted vulnerable, while Recall reflects detected samples among all vulnerable samples; a high F1 requires both to be high.Source: B. Evaluation Metrics · p. 8
  • Precision–Recall curves show how Precision and Recall vary across thresholds, while AP summarizes the curve and reflects ranking quality.Unlike F1 at one selected threshold, AP evaluates performance across all decision thresholds.Source: B. Evaluation Metrics · p. 8

C. Performance Comparison

  • 66.67%–68.70% average F1 for GRAML models exceeds CodeBERTa’s 59.26%, with CodeLlama-7B highest at 68.70%.GRAML also outperforms all encoder baselines on every test set; the reported gains range from 1.33% on ID to 18.22% on Juliet.Source: C. Performance Comparison · p. 8, Table V · p. 9
  • 66.67%–68.70% average F1 for GRAML models exceeds DeepSeek-V4-Flash’s 65.06%; CodeLlama-7B reaches 68.70%.GRAML achieves the best result on six of seven test sets, and its CodeLlama-7B model exceeds GPT-5’s few-shot average F1 of 53.73% by 14.97%.Source: C. Performance Comparison · p. 8
  • 64.87%–67.43% average F1 for GRAML models exceeds the strongest VulLLM variant’s 64.22%.The reported improvements range from 0.65% to 3.21%; the comparison uses GPT-4 for vulnerability explanation generation in both frameworks.Source: C. Performance Comparison · pp. 8–9, Table VI · p. 9

D. Robustness Evaluation

GRAML is evaluated on CodeLlama and Mistral under semantics-preserving transformations in three categories: semantic obfuscation, long-context noise, and structural restructuring. Both backbones outperform the strongest encoder baseline in all three categories.

Overall Results

  • 60.98%, 66.54%, and 66.40% are CodeLlama’s average F1 scores under semantic obfuscation, long-context noise, and structural restructuring, respectively.Mistral scores 59.25%, 63.79%, and 62.04% in the same categories; both backbones outperform the strongest encoder baseline in all three.Source: D. Robustness Evaluation · p. 9, Figure 5 · p. 10

Semantic Obfuscation

  • 7.35% and 5.62% are CodeLlama’s and Mistral’s average-F1 improvements over CodeBERTa’s 53.63% under semantic obfuscation.The transformation renames identifiers or removes comments without changing program behavior.Source: D. Robustness Evaluation · p. 9

Long-Context Noise

  • 11.34% and 8.59% are CodeLlama’s and Mistral’s average-F1 improvements over CodeBERTa’s 55.20% under long-context noise.CodeLlama scores 69.41% on PrimeVul versus 60.41% for the strongest encoder result; the noise category inserts irrelevant content.Source: D. Robustness Evaluation · pp. 9–10

Structural Restructuring

  • 10.46% and 6.10% are CodeLlama’s and Mistral’s average-F1 improvements over CodeBERTa’s 55.94% under structural restructuring.The transformations alter code organization while preserving semantics; CodeLlama scores 68.90% on PrimeVul versus 60.19% for the strongest encoder result.Source: D. Robustness Evaluation · p. 10

E. Ablation Study

The ablations find that all three auxiliary tasks contribute to detection performance, with Description having the largest individual effect. They also show that extracted graph evidence and its natural-language representation outperform source code alone and raw CPG serializations in the tested settings.

1) Effectiveness of Multi-Task Supervision:

  • 5.83% for CodeLlama and 13.25% for Mistral are the average-F1 decreases when training uses Detection Only instead of Full GRAML.The decline appears across both backbones, supporting the contribution of the three auxiliary tasks jointly.Source: 1) Effectiveness of Multi-Task… · pp. 10–11, Table IX · p. 11
  • 4.53% for CodeLlama and 10.70% for Mistral are the average-F1 decreases when Description is removed.These reductions exceed those from removing Localization or Assessment individually; the paper identifies Description as useful semantic supervision beyond binary labels.Source: 1) Effectiveness of Multi-Task… · p. 11
  • 2.58% and 10.08% are the average-F1 decreases for CodeLlama and Mistral without Localization; removing Assessment decreases them by 0.70% and 1.70%.The paper relates Localization to line-level guidance on vulnerability-related regions and Assessment to severity-related information.Source: 1) Effectiveness of Multi-Task… · p. 11

2) Effectiveness of Graph Evidence:

  • 68.39% for CodeLlama and 66.44% for DeepSeek are average F1 scores with natural-language descriptions, versus 66.44% and 65.84% with raw CPG serializations.The same advantage for natural-language descriptions is reported for Mistral and Qwen.Source: 2) Effectiveness of Graph Evidence · p. 11
  • 13.72% is Mistral’s average-F1 decrease without graph evidence, from 65.75% to 52.03%.CodeLlama also decreases, from 68.39% with graph evidence to 65.23% without it; the same direction is reported for DeepSeek and Qwen.Source: 2) Effectiveness of Graph Evidence · p. 11
  • 18.03% is Mistral’s Juliet F1 difference with versus without graph evidence, 65.02% versus 46.99%.CodeLlama shows the same direction on Juliet, scoring 77.78% with graph evidence and 65.47% without it.Source: 2) Effectiveness of Graph Evidence · p. 12
  • The ablations support using extracted structural information and converting it into natural-language vulnerability descriptions rather than relying on raw CPG serializations or source code alone.The study characterizes the descriptions as more effective supervision than either alternative.Source: 2) Effectiveness of Graph Evidence · pp. 11–12

F. Effectiveness of ToT-VR

ToT-VR outperforms CoT across the evaluated backbones and datasets. The threshold analyses and LoRA sensitivity study examine the robustness of GRAML’s detection performance and configuration.

F. Effectiveness of ToT-VR

  • 67.22% average F1 with ToT-VR versus 64.08% with CoT, a 3.14% improvement across four backbones; ToT-VR wins all 24 model–dataset comparisons.The gains range from 2.17% for Mistral to 3.66% for CodeLlama.Source: F. Effectiveness of ToT-VR · p. 12, Table XI · p. 13
  • ToT-VR’s clearest dataset-specific gain is on Juliet, where average F1 across the four backbones rises from 67.23% with CoT to 71.66% with ToT-VR.The passage also reports improvements on CVEfixes and the other external datasets.Source: F. Effectiveness of ToT-VR · p. 12

G. Threshold Robustness Analysis

  • 0.7537 AP gives Mistral the highest PR-curve score, while all four GRAML models score close to or above 0.70 AP.AP summarizes performance across thresholds rather than at one selected decision threshold.Source: G. Threshold Robustness Analysis · p. 12, Figure 6 · p. 12
  • Small changes around the ID-validation-selected thresholds produce limited F1 variation; Mistral reaches 72.19% F1 at its selected threshold.The selected thresholds are 0.41 for CodeLlama, 0.44 for Mistral, 0.53 for DeepSeek, and 0.54 for Qwen.Source: G. Threshold Robustness Analysis · p. 12, Figure 7 · p. 13

H. Parameter Sensitivity Analysis

  • All four backbones achieve their highest average F1 across the ID test set and six external datasets at LoRA rank r = 32.The authors use r = 32 in the main experiments.Source: H. Parameter Sensitivity Analysis · p. 13

A. Deep Learning for Vulnerability Detection

Prior work spans sequence and graph-based vulnerability detection, structure-aware LLM analysis, and reasoning with multi-task supervision. GRAML combines these directions, with reported gains on ID and OOD evaluation and a stated C/C++ scope.

A. Deep Learning for Vulnerability Detection

  • Earlier vulnerability detectors modeled code as sequences with RNNs [33] and LSTMs, while later work used CPG-based GNNs to capture structural dependencies,,,.Related approaches also explored causal reasoning, dataflow analysis, and graph denoising,.Source: A. Deep Learning for Vulnerability… · p. 13

B. Structure-Aware Program Analysis

  • Structure-aware methods use CPGs to expose control- and data-flow dependencies,, but long-range context and noisy graph relations remain challenges,.GRAML uses graph evidence as a training-time constraint and distills it into natural-language supervision, avoiding graph evidence at inference.Source: B. Structure-Aware Program Analysis · p. 14

C. Reasoning and Multi-Task Learning

  • Prior work has explored CoT, Self-Consistency [53], ToT, [54], and joint objectives; GRAML combines graph-guided ToT reasoning with multi-task supervision.The framework generates supervision for detection, localization, and assessment tasks.Source: C. Reasoning and Multi-Task Learning · p. 14

V. DISCUSSION

  • The discussion states that structurally guided descriptions and auxiliary labels reduce reliance on shallow cues, improve ID detection, and generalize better to OOD benchmarks.These descriptions are generated using CPG-based structural evidence and guide the four-task dataset.Source: §V · p. 14
  • The study focuses on C/C++ vulnerabilities, with extension to other languages and more diverse datasets identified as future work.The authors also propose incorporating additional static-analysis tools and exploring automated program repair.Source: §V · p. 14

VI. CONCLUSION

  • The best GRAML model achieves 68.70% average F1 on the ID test set and six external datasets, outperforming state-of-the-art baselines by up to 30.92%.The conclusion also reports consistent performance under the evaluated semantics-preserving code transformations.Source: §VI · pp. 14–15

APPENDIX A CPG EXTRACTION AND CONTEXT CONSTRUCTION

The appendices describe how GRAML constructs line-level graph context, generates and selects ToT-VR descriptions, and formats instructions for its four fine-tuning tasks.

APPENDIX A CPG EXTRACTION AND CONTEXT CONSTRUCTION

  • To balance irrelevant whole-function context against the loss of semantic context from vulnerable-line-only inputs, GRAML uses Joern to extract structural information for description generation.The appendix addresses isolated C++ snippets by replacing :: with __ before parsing and reduces graph redundancy by retaining valid line-associated nodes and inter-line relations.Source: Appendix A · p. 15
  • A multiprocessing wrapper isolates Joern parsing failures and removes residual files so batch processing can continue after individual sample failures.Joern errors on malformed or complex snippets can leave files that consume storage and interfere with batch processing.Source: Appendix A · p. 15

APPENDIX B PRACTICAL IMPLEMENTATION OF TOT-VR

  • ToT-VR generates multiple candidate descriptions through five sequential GPT-5 stages, adding processing time because each sample requires five API calls.Generating descriptions for 410 vulnerable samples in one iteration takes approximately three hours.Source: Appendix B · p. 15
  • The authors manually select among five descriptions per sample using root-cause accuracy, contextual completeness, impact clarity, and factual consistency criteria.They state that this adds manual effort but helps exclude unsupported or inaccurate descriptions from Description-task training samples.Source: Appendix B · p. 15

A. Task Instruction Prompts

  • Each fine-tuning instance combines a task-specific instruction, source-code input, and target response, with loss computed only on target-response tokens.The instructions specify the expected task and output format, and all tasks share a system prompt.Source: A. Task Instruction Prompts · pp. 15–16, Table XII · p. 16

B. Prompt Templates for Target LLMs

The appendices specify model-specific chat formatting, sequence-length choices, ToT-VR prompt stages, and semantics-preserving transformations used in robustness evaluation.

B. Prompt Templates for Target LLMs

  • The training script uses each target model’s official conversational template to separate system prompts, instructions, code, and assistant responses.The completion-only collator masks preceding tokens so training loss is computed on assistant responses rather than prompts or input code.Source: B. Prompt Templates for Target LLMs · p. 16, Table XIII · p. 16

C. Sequence Length Analysis

  • Training samples include instructions, code, graph-aware context, and responses; 91.04% are within 2,048 tokens, while 2.97% exceed 4,096 tokens.The distribution also reports that 0.93% exceed 8,192 tokens and some samples exceed 16,384 tokens.Source: C. Sequence Length Analysis · p. 16, Table XIV · p. 16
  • The authors use maximum sequence lengths of 8,192 for Mistral and 16,384 for DeepSeek, CodeLlama, and Qwen to reduce information loss while keeping training feasible.A small number of samples longer than 16,384 tokens are still truncated.Source: C. Sequence Length Analysis · pp. 16–17

APPENDIX D PROMPT TEMPLATES FOR TOT-VR

  • The five ToT-VR stages use a shared system prompt but separate user prompts and output formats.The appendix provides the prompt templates for each stage.Source: Appendix D · p. 17, Table XV · p. 17

APPENDIX E ROBUSTNESS TRANSFORMATIONS

  • Robustness evaluation applies seven semantics-preserving transformations grouped into semantic obfuscation, long-context noise, and structural restructuring.The evaluation follows the adversarial framework of Risse et al..Source: Appendix E · p. 17

A. Semantic Obfuscation

  • Semantic obfuscation tests reliance on readable names and comments by renaming parameters and functions and removing comments.These changes aim to make models rely on executable code rather than lexical hints.Source: A. Semantic Obfuscation · pp. 17–18

B. Long-Context Noise

The appendix probes robustness to long-context noise and structural reorganization, and attributes encoder near-zero Juliet results under perturbation to sequence truncation and confidence collapse.

B. Long-Context Noise

  • Long-Context Noise tests whether models retain vulnerability information amid irrelevant content, using dead-code and repetitive-comment insertions that preserve behavior.The dead-code transform appends an uncalled helper with 300 consecutive break statements; the comment transform repeats break 50 times.Source: B. Long-Context Noise · p. 18

C. Structural Restructuring

  • Function Extraction moves the original function body into helper_func and invokes it with preserved parameters, relocating vulnerability logic without changing behavior.The transformation identifies the body boundaries through brace matching.Source: C. Structural Restructuring · p. 18
  • Void-Function Call Insertion adds an empty helper with 150 return statements and calls it before the target function's original logic.The additional call is benign and interprocedural.Source: C. Structural Restructuring · pp. 18–19

APPENDIX F BASELINE FAILURE ANALYSIS

  • Perturbed Juliet inputs averaged over 860 tokens against 512-token limits, truncating vulnerable lines and contributing to near-zero encoder F1 scores.True-vulnerable predictions mostly fell below the 0.5 threshold, producing zero true positives; the analysis attributes this to sequence inflation rather than evaluation errors.Source: Appendix F · p. 19

Related papers

Loading 2610.09605v1…