PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking LLMs and LLM-based Agents in Practical Vulnerability Detection for Code Repositories

Alperen Yildiz, Sin G. Teo, Yiling Lou, Yebo Feng, Chong Wang, Dinil M. Divakaran

arXiv:2503.03586v2cs.CR

TL;DR

Real-world vulnerability detection needs scalable interprocedural and pairwise evaluation beyond isolated-function benchmarks. The paper introduces JITVUL for this setting and finds that ReAct Agents better distinguish vulnerable from benign versions, although both agents and LLMs remain inconsistent and require refinement.

  • Problem

    Existing vulnerability benchmarks often omit repository context, while repository-level approaches can be costly, lack pairwise evaluation, or use limited interprocedural context.

  • Method

    The paper constructs JITVUL, a pairwise JIT vulnerability-detection benchmark linking modified functions to vulnerability-introducing and fixing commits, and evaluates ReAct Agents with on-demand interprocedural tools.

  • Results

    ReAct Agents better differentiate vulnerable and benign versions using thought-action-observation and interprocedural context, while prompting strategies improve LLM performance and agents require further refinement.

  • Takeaways & Limitations

    Pairwise evaluation exposes analysis inconsistencies that aggregate performance measures may miss, motivating more robust evaluation and better-designed agent prompting.

  • Takeaways & Limitations

    JITVUL may not perfectly trace vulnerability-introducing commits because of complex code evolution and function interactions.

Abstract

from arXiv · show

Large Language Models (LLMs) have shown promise in software vulnerability detection, particularly on function-level benchmarks like Devign and BigVul. However, real-world detection requires interprocedural analysis, as vulnerabilities often emerge through multi-hop function calls rather than isolated functions. While repository-level benchmarks like ReposVul and VulEval introduce interprocedural context, they remain computationally expensive, lack pairwise evaluation of vulnerability fixes, and explore limited context retrieval, limiting their practicality. We introduce JitVul, a JIT vulnerability detection benchmark linking each function to its vulnerability-introducing and fixing commits. Built from 879 CVEs spanning 91 vulnerability types, JitVul enables comprehensive evaluation of detection capabilities. Our results show that ReAct Agents, leveraging thought-action-observation and interprocedural context, perform better than LLMs in distinguishing vulnerable from benign code. While prompting strategies like Chain-of-Thought help LLMs, ReAct Agents require further refinement. Both methods show inconsistencies, either misidentifying vulnerabilities or over-analyzing security guards, indicating significant room for improvement.

1 Introduction

Existing vulnerability benchmarks often abstract away repository context, while real-world vulnerabilities can span multi-hop function calls. JITVUL addresses this gap with pairwise JIT evaluation and ReAct-based interprocedural analysis, though both LLMs and agents remain inconsistent.

  • Motivation: Function-level benchmarks such as Devign and BigVul show promise but omit broader repository context needed for many real-world vulnerabilities.Real-world cases can arise through multi-hop function call chains and require analyzing callers, callees, and branch conditions.
  • Limitations of Existing Benchmarks: Repository-level benchmarks add interprocedural context but remain costly, lack effective vulnerable-versus-patched evaluation, and rely mainly on retrieval-based context integration.These limitations motivate more practical and comprehensive benchmarking approaches.
  • JITVUL: JITVUL links each target function to vulnerability-introducing and vulnerability-fixing commits, enabling pairwise JIT vulnerability detection.The benchmark is constructed from 879 CVEs and contains 1,758 pairwise commits spanning 91 vulnerability types.
  • Findings: ReAct Agents better distinguish vulnerable from benign versions, whereas CoT and few-shot prompting improve LLMs; both approaches still show inconsistent analyses.The results also motivate improved evaluation guidelines, prompting strategies, and reasoning models tailored to vulnerability analysis.
  • Approach: The study implements ReAct agents with varied prompting strategies and foundation models to evaluate on-demand use of interprocedural context.The agents are assessed alongside LLMs for JIT vulnerability detection.

2 Related Work

Related work spans function-level vulnerability datasets, repository-level benchmarks, context-enhanced LLM methods, prompting strategies, and agentic architectures. These approaches progressively add data quality, context, reasoning, or tool interaction for vulnerability detection.

  • Function-Level Benchmarks: Function-level datasets such as BigVul, MegaVul, DiverseVul, and PrimeVul use CVE-linked fixes and data-cleaning strategies to label or improve vulnerability examples.Their methods include repository filtering, accurate function extraction, deduplication, and vulnerability-introducing-commit filtering.
  • Repository-Level Benchmarks: ReposVul and VulEval extend vulnerability detection toward repository-level context by incorporating information such as callers and callees.They differ in data sources, filtering procedures, and contextual features.
  • LLM-Based Detection: LLM4SA combines language models with static analysis warnings, while LLM4Vuln adds context through retrieval-augmented generation.These methods target improved vulnerability identification and explanation.
  • Prompting Strategies: Chain-of-Thought and few-shot prompting enrich LLM inputs by encouraging stepwise reasoning or providing example traces for in-context learning.Both strategies are used in LLM-based vulnerability detection.
  • Agentic Architectures: ReAct agents interleave reasoning and actions to interact with an environment and analyze observations, while agentic architectures remain underexplored in vulnerability detection.Related approaches include Reflexion agents.

3 JITVUL: Just-in-Time Vulnerability Detection for Code Repositories

Practical repository-level vulnerability detection requires interprocedural context, scalable candidate selection, and pairwise comparison of vulnerable and patched code. JITVUL operationalizes these requirements through commit-linked paired samples derived from real-world CVEs.

  • Interprocedural Context: Interprocedural context is necessary because vulnerabilities can arise from interactions across functions, making isolated-function analysis insufficient.The relevant dependencies may include callers, callees, and execution-path conditions.
  • Scalability: Scanning every repository function is computationally infeasible for LLMs and agents, so JIT detection focuses on functions newly introduced or modified in commits.This limits analysis to a practical candidate set.
  • Pairwise Comparison: Pairwise benchmarking tests whether models distinguish vulnerable code from its patched benign version rather than relying on superficial patterns.This comparison is presented as necessary for reliable vulnerability-detection evaluation.
  • Benchmark Requirements: Existing benchmarks do not fully satisfy interprocedural-context, scalability, and pairwise-evaluation requirements simultaneously.PrimeVul provides pairwise evaluation without interprocedural context, whereas VulEval provides repository-level detection without pairwise evaluation and assumes same-commit callers and callees are relevant.
  • Task Definition: JIT vulnerability detection classifies a commit-modified target function as vulnerable or benign using interprocedural context from its repository.The task supports pairwise evaluation of LLMs and LLM-based agents.
  • Benchmark Construction: JITVUL is built through vulnerability-entry selection, target-function extraction, and pairwise-commit identification.The process derives vulnerable and benign function versions and identifies vulnerability-introducing and vulnerability-fixing commits.
  • Resulting Benchmark: JITVUL contains 1,758 pairwise samples from 879 CVEs, balanced between 879 vulnerable and 879 benign samples, spanning 91 CWEs.The repositories average 2,955.94 code files, and target functions average 696.40 lines of code.

4 Experimental Setup

The evaluation compares Plain LLMs, Dependency-Augmented LLMs, and ReAct Agents across prompting strategies, foundation models, and vulnerability-detection metrics. The setup includes pairwise evaluation of vulnerable and patched functions, with interprocedural context retrieved either deterministically or on demand.

  • Detection Methods: The study compares Plain LLM, Dependency-Augmented LLM, and ReAct Agent detection methods.Each method is evaluated with GPT-4o-mini and GPT-4o.
  • Detection Methods: Plain LLMs analyze only the target function, while Dep-Aug LLMs add the Top-5 similar callers and callees through lexical retrieval.Dep-Aug provides deterministic interprocedural context based on the VulEval approach.
  • Detection Methods: ReAct Agents iteratively acquire interprocedural context through tools that retrieve callers, callees, and complete function definitions.The workflow follows thought-action-observation and supports on-demand analysis.
  • Prompting Strategies: Each method includes vanilla, chain-of-thought, few-shot, and combined chain-of-thought-plus-few-shot prompting variants.Few-shot prompts use vulnerable and patched benign code examples with explanations of vulnerabilities and safeguards.
  • Metrics: Pairwise accuracy measures the proportion of commit pairs whose vulnerable and benign functions are both correctly labeled, alongside F1.The evaluation applies methods to vulnerability-introducing and vulnerability-fixing commit repositories and target functions.
  • Implementation: Ten vulnerable-benign example pairs are manually created from the 2024 CWE Top 25, while CFlow and CTags extract caller, callee, and function-body context.GPT-4o-mini and GPT-4o are used at temperature 0, with LangChain-0.3.14 for pipeline construction.

5 Results and Analyses

Results compare detection methods using both F1 and pairwise accuracy, showing that ReAct agents better distinguish vulnerable from patched code while conventional LLMs often achieve higher F1. Prompting, foundation-model choice, and pairwise analysis substantially affect outcomes, with persistent inconsistencies in code-version analysis.

  • Detection Method Comparison: ReAct agents achieve higher pairwise accuracy than other LLM-based methods across prompting strategies, improving by 0.1%–16.61%.The largest gain occurs with GPT-4o and vanilla prompting.
  • Detection Method Comparison: Higher F1 does not necessarily indicate better vulnerability characterization because LLM-based methods predict over 90% vulnerable labels in some settings while precision remains around 50%.This label-distribution effect produces higher F1 despite weaker pairwise discrimination, motivating pairwise evaluation.
  • Detection Method Comparison: ReAct agents improve pairwise accuracy by 9.46% over dependency-augmented LLMs with GPT-4o-mini and by 8.42% with GPT-4o.Their iterative retrieval of callers and callees contrasts with dependency augmentation’s fixed, similarity-based retrieval of top-five functions, which may introduce noise.
  • Prompting Strategy Comparison: CoT and few-shot prompting improve pairwise accuracy by 1.26%–17.76%, but these strategies do not consistently improve F1.The gains are more evident for LLMs than ReAct agents because few-shot examples use singleton snippets that do not require interprocedural analysis.
  • Foundation Model Comparison: GPT-4o outperforms GPT-4o-mini on average, whereas Llama-3.1-8B frequently fails to complete ReAct analysis and defaults to the benign label.Different models respond differently to CoT and few-shot prompting, so prompting should be customized to the foundation model.
  • Pairwise Comparison: Pairwise vulnerable errors dominate, ranging from approximately 40%–95% for LLMs and 35%–50% for ReAct agents, excluding Llama-3.1-8B.Pairwise reversed errors typically increase as pairwise accuracy improves.
  • Pairwise Comparison: Both LLMs and ReAct agents may miss vulnerable-version causes, over-analyze patched guards, and apply inconsistent effort across vulnerable and benign versions.For GPT-4o-mini with vanilla prompting, ReAct tool invocations average 5.85 for vulnerable versions versus 1.79 for benign versions.

6 Conclusion

JITVUL enables pairwise evaluation of LLMs and LLM-based agents for JIT vulnerability detection. ReAct Agents show better reasoning but need refinement, while both approaches exhibit recurring analysis errors.

  • JITVUL enables comprehensive pairwise evaluation of LLMs and LLM-based agents for just-in-time vulnerability detection.
  • ReAct Agents leverage thought-action-observation and interprocedural context to demonstrate better reasoning for vulnerability analysis.
  • Both LLMs and ReAct Agents can overlook critical flaws or over-analyze benign fixes.
  • The findings motivate improvements in agentic architectures, prompting techniques, dynamic interprocedural analysis, and vulnerability-focused reasoning models.

7 Limitations

The authors identify limitations in JITVUL’s commit tracing and in the scope of its pairwise evaluation.

  • JITVUL may not perfectly trace vulnerability-introducing commits because of complex code evolution and function interactions.The authors mitigate this through existing methodologies and manual inspection of selected instances.
  • The benchmark’s pairwise evaluation remains subject to a limitation concerning its labeling approach.

A Studied Methods

The studied methods use prompt variants for Plain LLMs and a ReAct workflow that retrieves interprocedural context. Few-shot examples contrast vulnerable and benign code versions.

  • Plain LLM: Plain LLM prompts use a vanilla template with optional few-shot examples and chain-of-thought instructions.The target function is inserted through a placeholder, and the output format includes vulnerability and CWE labels.
  • ReAct Agent: ReAct Agent prompts use an agent scratchpad for tool descriptions, prior observations, reasoning traces, and an input prompt.The input can be enhanced with prompt augmentation techniques.
  • ReAct Agent: ReAct Agent workflow retrieves interprocedural context through tools for callers, callees, and complete function definitions.
  • Few-shot Example: The few-shot example presents vulnerable and benign versions of a CWE-787 out-of-bounds write to highlight their key differences.
  • Tool Invocation Distribution: With GPT-4o and vanilla prompting, ReAct Agent usually invokes tools one to three times to retrieve callers or callees.

C Llama-3.1 Results

On JITVUL with Llama3.1-8B, ReAct Agents show lower performance because execution often fails from formatting and parsing issues.

  • ReAct Agents using Llama3.1-8B show significantly lower performance on JITVUL.
  • Formatting and parsing failures frequently cause the agents to default to the benign label.

D.1 CVE-2019-15164

The CVE-2019-15164 case study shows ReAct correctly distinguishing vulnerable and benign versions by retrieving caller context, while Plain LLM misclassifies the benign version and both methods struggle with precise, consistent analysis.

  • Case Study: The vulnerable function lacks validation of the network-supplied source before opening the device, while the benign version validates that source as a URL.This added if-condition distinguishes the patched version from the vulnerable one.
  • Label Predictions: ReAct correctly predicts both vulnerable and benign labels, whereas Plain LLM misclassifies the benign version as vulnerable.ReAct retrieves the caller daemon serviceloop and surrounding function bodies to analyze broader code context.
  • CWE Predictions: Plain LLM predicts CWE-120 for both versions, while ReAct predicts the related but incorrect CWE-20 for the vulnerable version.The case concerns SSRF caused by insufficient validation, so neither fine-grained prediction matches the ground-truth category.
  • Analysis Patterns: ReAct analyzes the vulnerable and benign versions inconsistently, applying a more comprehensive check to the benign version despite the need for similar analysis patterns.This inconsistency indicates insufficient robustness in capturing the actual vulnerability.

D.2 CVE-2019-3877

The CVE-2019-3877 case study shows Plain LLM failing on both versions: it reverses labels, misses insufficient URL validation in the vulnerable code, and over-analyzes the benign version.

  • Case Study: The vulnerable am_check_url function involves insufficient validation for url, while the benign version addresses the URL-redirection vulnerability.
  • Label Predictions: Plain LLM with GPT-4o and vanilla prompting produces incorrect predictions, often reversing the vulnerable and benign labels.
  • Analysis Patterns: Plain LLM misses insufficient URL validation in the vulnerable version and over-analyzes validation requirements in the benign version.These divergent analysis patterns lead to incorrect predictions for both versions.
Loading 2503.03586v2…