PaperlayerView exact revision on arXiv

Source-linked AI summary

Favia: Forensic Agent for Vulnerability-fix Identification and Analysis

André Storhaug, Jiamou Sun, Jingyue Li

arXiv:2602.12500v1cs.SEcs.AIcs.CR

TL;DR

Identifying vulnerability-fixing commits at scale is difficult because repositories contain millions of commits and realistic candidates include highly similar security-relevant alternatives. Favia combines scalable ranking with iterative, evidence-grounded agent reasoning and achieves the strongest precision–recall trade-offs and F1-scores under realistic candidate selection.

  • Problem

    Vulnerability-fix detection must search millions of commits, while realistic hard negatives resemble true fixes and make random-sampling evaluations overly optimistic.

  • Method

    Favia ranks all commits efficiently, then uses an LLM agent in the pre-commit repository environment for iterative, evidence-grounded semantic analysis.

  • Results

    Favia achieved the highest F1-scores across tested models, preserving recall up to 0.98 and improving precision over prior LLM-based approaches by up to 92%.

  • Takeaways & Limitations

    Randomly sampled commits substantially overestimate performance, whereas realistic candidate sets reveal meaningful differences between vulnerability-fix identification methods.

  • Takeaways & Limitations

    Evaluation may not generalize beyond GitHub-hosted repositories, publicly disclosed CVEs, or ecosystems less supported by existing tooling and CVE reporting.

Abstract

from arXiv · show

Identifying vulnerability-fixing commits corresponding to disclosed CVEs is essential for secure software maintenance but remains challenging at scale, as large repositories contain millions of commits of which only a small fraction address security issues. Existing automated approaches, including traditional machine learning techniques and recent large language model (LLM)-based methods, often suffer from poor precision-recall trade-offs. Frequently evaluated on randomly sampled commits, we uncover that they are substantially underestimating real-world difficulty, where candidate commits are already security-relevant and highly similar. We propose Favia, a forensic, agent-based framework for vulnerability-fix identification that combines scalable candidate ranking with deep and iterative semantic reasoning. Favia first employs an efficient ranking stage to narrow the search space of commits. Each commit is then rigorously evaluated using a ReAct-based LLM agent. By providing the agent with a pre-commit repository as environment, along with specialized tools, the agent tries to localize vulnerable components, navigates the codebase, and establishes causal alignment between code changes and vulnerability root causes. This evidence-driven process enables robust identification of indirect, multi-file, and non-trivial fixes that elude single-pass or similarity-based methods. We evaluate Favia on CVEVC, a large-scale dataset we made that comprises over 8 million commits from 3,708 real-world repositories, and show that it consistently outperforms state-of-the-art traditional and LLM-based baselines under realistic candidate selection, achieving the strongest precision-recall trade-offs and highest F1-scores.

1 Introduction

Identifying vulnerability-fixing commits is essential but difficult because repositories contain millions of commits and existing automated methods have poor precision–recall trade-offs. Favia addresses this challenge by combining scalable candidate ranking with iterative, evidence-grounded agent reasoning, evaluated on the realistic CVEVC dataset.

  • Motivation: Disclosed vulnerabilities require rapid identification of corresponding patch commits, but millions of repository commits make manual inspection and exhaustive analysis infeasible.Only a tiny fraction of commits correspond to security patches.
  • Limitations of Prior Methods: Existing rule-based, machine-learning, deep-learning, and LLM-based methods struggle with poor precision–recall trade-offs and shallow, single-pass reasoning.Conservative models miss true patches, whereas aggressive models produce many false positives; recent LLM systems typically reason over shallow context.
  • Favia: Favia combines lightweight candidate ranking with deep, iterative, evidence-grounded agent reasoning to balance scalability with fine-grained analysis.Ranking narrows the commit search space, while the agent analyzes each candidate using code navigation and evidence.
  • Evaluation: CVEVC contains over 8 million commits from 3,708 real-world repositories and supports evaluation under both random and all-commit candidate settings.The study compares Favia with state-of-the-art traditional and LLM-based baselines.
  • Results: On realistic datasets, Favia achieves the highest F1-scores across tested models, with recall up to 0.98 and precision improvements of up to 92% over prior LLM-based approaches.Iterative evidence-grounded code navigation enables causal alignment between CVE root causes and commit-level changes; 85% of agent errors arise from superficial associations or root-cause misinterpretation.

2 Preliminaries

Vulnerability-fix detection identifies repository commits that implement security patches, including direct or indirect changes addressing known vulnerabilities. The task is difficult because repositories contain vast candidate spaces and fixes may require contextual reasoning across files, control paths, and subtle behavior.

  • Definitions: Patch commits are repository changes that address and resolve known vulnerabilities associated with security advisories such as CVEs.They may directly modify vulnerable code or indirectly affect related components, configurations, or control flows.
  • Definitions: Vulnerability-fix detection identifies which version-controlled commits correspond to security patches.The task supports automated vulnerability tracking and remediation in large-scale software ecosystems.
  • Challenges: All repository commits are potential candidates, making the search space prohibitively large.The Linux kernel example illustrates the scale of this challenge as of January 2006, although the supplied passage is truncated before giving the repository count.
  • Challenges: Some fixes require broader contextual reasoning than localized checks or type constraints, particularly for logic flaws, race conditions, and privilege-escalation bugs.These vulnerabilities can demand understanding beyond the changed lines themselves.
  • Challenges: Vulnerability fixes may span multiple files, indirect control paths, or subtle behavioral changes that are not immediately evident in the diff.Distinguishing genuine patches from unrelated or cosmetic changes therefore requires understanding the vulnerability and surrounding codebase.
  • Illustrative Example: The POMM SQL-injection patch illustrates an easier case because proper string escaping directly addresses the root cause.The vulnerability allowed remote attackers to execute arbitrary SQL commands through improper handling of user-controlled input.

3 Related Work

Vulnerability-fix detection has evolved from heuristic and manually engineered methods to machine learning, deep structural models, ranking-based systems, and LLM-enabled semantic reasoning. Despite improved capabilities, LLM approaches remain challenged by computational cost, scalability, and limited real-world evaluation settings.

  • Heuristic and classical methods: Early vulnerability-fix detection used rule-based systems, manually crafted features, commit metadata, code-change characteristics, and keywords in commit messages.These approaches included searching log messages for terms such as “fix” and “bug,” later combining commit messages with code changes.
  • Heuristic and classical methods: Classical machine learning and natural language processing broadened detection across software ecosystems by analyzing commit messages, bug reports, and developer-written artifacts.Studies used issue-tracking data from platforms such as JIRA and Bugzilla, while later systems combined independent classifiers to reduce analyst burden.
  • Deep and structural models: Deep learning and syntax-aware approaches learned semantic or structural representations from code diffs, including hierarchical, AST-based, graph-based, and RNN-based models.PatchNet modeled sequential and hierarchical diff structure; Commit2Vec encoded AST symmetric differences, and PatchRNN extracted syntax-level features from diffs and commit messages.
  • Link recovery and ranking: Link-recovery and ranking-based methods addressed missing connections between commits and external vulnerability data by correlating identifiers, file locations, and textual similarity.HERMES enriched commit context through link recovery, while ranking methods reframed detection as vulnerability–commit correlation.
  • LLM-based approaches: LLM-based frameworks introduced higher-level semantic reasoning, with PatchFinder combining hybrid retrieval and semantic reranking and reporting recall@10 of 80.63%.Other frameworks, including CompVPD, LLM4VFD, and CommitShield, also leveraged pretrained LLMs for vulnerability-fix reasoning.
  • LLM-based approaches: LLM-based approaches face complexity, computational-cost, and scalability challenges when applied to repositories containing millions of commits.Evaluations are consequently often limited to curated or constrained settings, leaving robustness and real-world effectiveness open questions.

4 Favia Approach

Favia identifies vulnerability-fixing commits through a scalable two-stage pipeline: machine-learning ranking narrows all repository commits to top-k candidates, which a ReAct-based LLM agent evaluates through iterative, code-grounded semantic reasoning. The agent examines vulnerability context, commit changes, and pre-commit repository state to determine whether each candidate truly mitigates the CVE.

  • Stage 1: Candidate Ranking: Favia first ranks every repository commit with a machine-learning classifier using CVE descriptions, commit messages, and code changes, retaining only the top-k candidates.The ranking stage addresses repositories such as the Linux kernel, which contain over 1.4 million commits, making exhaustive LLM evaluation computationally infeasible.
  • Stage 2: Agent Classification: A ReAct-based reasoning agent then evaluates each top-k candidate by decomposing the task into iterative reasoning and tool-based interactions.This hybrid design combines scalable filtering with detailed semantic analysis of candidate patches.
  • Stage 2: Agent Classification: The agent analyzes the CVE and CWE reports, inspects the commit diff, and correlates the code changes with whether they mitigate the described vulnerability.The expected output includes a reasoning explanation, a 5-point ordinal confidence score, and a final verdict.
  • Code Environment: Favia places the agent in a simulated environment representing the repository immediately before the commit, enabling inspection of original logic and assessment of indirect fixes.Python-function tools support targeted inspection, contextual retrieval, and dynamically invoked codebase exploration.

5 Experimental Design · 5.1 Research Questions · 5.2 Design to answer RQ1

The experimental design evaluates Favia against representative machine-learning and LLM-based vulnerability-fix detection methods under both random and realistic candidate-selection settings. It uses diverse instruction-tuned LLMs, challenging similar-commit candidates, commit-level precision, recall, and F1-score, and a large multi-repository dataset to answer effectiveness, failure-analysis, and efficiency questions.

  • 5.1 Research Questions: The study asks how effective Favia is against existing approaches, why agent-based methods make incorrect predictions, and how efficient Favia is relative to alternatives.
  • 5.2 Design to answer RQ1: RQ1 evaluates diverse open-source LLMs that differ in scale, architecture, and design goals.All selected models are instruction-tuned for structured prompts, tool use, and multi-step agentic execution.
  • 5.2.1 Models: The model set includes gemma-3-27b-it, Llama-3.3-70B-Instruct, and Qwen3-235B-A22B-Instruct-2507, spanning dense transformers and a 235-billion-parameter MoE model.Qwen3-235B-A22B-Instruct-2507 has 22 billion active parameters per forward pass.
  • 5.2.2 Baselines: Favia is compared with VulFixMiner, PatchFinder, LLM4VFD, and CommitShield, representing commonly used and state-of-the-art machine-learning and LLM-based baselines.LLM4VFD’s development-artifact component is excluded because those artifacts are unavailable to Favia’s agent-based approach.
  • 5.2.3 Datasets: The evaluation contrasts a random setting with uniformly sampled repository commits against a realistic setting in which all commits are potential candidates.Random sampling can omit highly similar hard negatives and therefore overestimate performance.
  • 5.2.3 Datasets: To remain scalable, Favia uses machine-learning ranking to remove clearly unrelated commits before comparing methods on remaining candidates similar to the true fix.The realistic dataset selects the top 10 candidates ranked by PatchFinder, while the random_10 dataset selects up to 10 commits per CVE through random sampling.
  • 5.2.3 Datasets: The source data contains 17,293 CVEs, 4,682 corresponding repositories, and 23,303 patching commits spanning over 200 file types.Of 3,820 identified repositories, 3,708 were successfully downloaded, producing approximately 2 terabytes of data.
  • 5.2.4 Metrics: Vulnerability-fix detection is evaluated as commit-level binary classification using precision, recall, and F1-score, aggregated across all CVEs; accuracy is omitted because class imbalance makes it uninformative.F1-score balances false positives and false negatives as the harmonic mean of precision and recall.

5.3 Design to answer RQ2

RQ2 investigates why Favia’s agent-based reasoning fails by analyzing incorrect predictions, agent trajectories, tool usage, and reasoning traces. The evaluation uses normalized tool-call counts and an independent LLM to classify failures into eight predefined modes.

  • Failure analysis: RQ2 analyzes Favia’s incorrect predictions through agent trajectories, tool usage patterns, and reasoning traces.The analysis distinguishes insufficient exploration, misunderstood vulnerability semantics, overconfidence, and reliance on surface-level cues.
  • Tool usage: Tool invocations are counted once per tool type per step to avoid misleading totals from arbitrary repeated calls.This prevents loops such as repeatedly opening and scrolling through a file from dominating usage measurements.
  • Failure classification: An LLM classifies incorrect agent runs into eight failure modes defined in Table 1.The classification uses traces collected from Favia’s RQ1 results after correctly classified traces are removed.
  • Evaluation setup: The analysis uses gpt-oss-120b from an independent model family to provide a fair and unbiased evaluation.The model is configured with medium reasoning effort and is separate from the models used in Section 5.2.1.
  • Evaluation setup: The classification prompt supplies the CVE description and agent task, reasoning steps, and tool calls while excluding tool outputs.This focuses the analysis on agent behavior rather than the contents of tool results.

5.4 Design to answer RQ3

Section 5.4 evaluates efficiency across approaches by measuring token consumption and examining how computational cost changes with model size and reasoning depth. This characterizes the trade-off between detection performance and computational overhead and assesses practicality.

  • Efficiency measurement: Efficiency is measured using input, output, and embedding token consumption across all approaches.The analysis reports mean token usage per commit.
  • Scaling analysis: The study analyzes how token cost scales with model size and reasoning depth.This accounts for the additional cost introduced by multi-step interaction and context accumulation.
  • Practicality: The evaluation characterizes the trade-off between improved detection performance and increased computational overhead.It also assesses the practicality of agent-based reasoning.

6 Experimental Results

This section presents detailed experimental results for each research question.

  • The section reports detailed results for each research question.

6.1 Results of RQ1: Effectiveness

Favia consistently achieves the strongest precision-recall trade-offs across evaluated models, preserving very high recall while reducing false positives relative to other LLM-based methods. Results also show that random candidate sampling makes vulnerability-fix identification substantially easier than realistic candidate selection, understating deployment difficulty.

  • Random-dataset results: VulFixMiner remains recall-limited, reaching precision of 0.83 but recall of 0.01, whereas PatchFinder reaches precision of 0.86, recall of 0.23, and F1-score of 0.36 on random sampling.Random sampling primarily benefits these methods in precision, while both remain limited in recall.
  • Performance across approaches: LLM4VFD reaches recall of 0.89–0.94 but precision of 0.18–0.30, while CommitShield reaches recall of 0.85–0.99 but precision of 0.12–0.22.Both methods identify most vulnerability-fixing commits but over-predict, producing many false positives.
  • Performance across approaches: Favia achieves precision of 0.23–0.39, recall of 0.94–0.98, and F1-scores of 0.37–0.56, the strongest overall range across approaches.It maintains recall comparable to LLM4VFD and CommitShield while reducing false positives, especially with larger models.
  • Realistic versus random evaluation: All approaches perform substantially worse on realistic candidates than on random candidates, indicating that random evaluation reduces ambiguity and understates real-world difficulty.Randomly selected sets contain many trivially non-security-related changes, inflating precision and F1-scores; realistic deployment treats all commits as candidates.
  • Forensic reasoning: Favia correctly rejects a non-patch by aligning the CVE’s integer truncation in GetUpdateFile with the changed code, unlike baselines that relied on generic safety language or speculative analogies.The commit modified XML string encoding in vlc_xml_encode and did not address the CVE’s component, function, or vulnerability type.

6.2 Results of RQ2: Failure analysis results

Favia’s agents generally follow evidence-seeking trajectories, but failures primarily reflect insufficient semantic grounding rather than limited exploration or tool access. Random negatives make CVE–patch linking appear easier than semantically plausible hard negatives, which require causal alignment between vulnerability roots and code changes.

  • Trajectory patterns: Successful trajectories typically retrieve the CVE description first, localize relevant files next, and progressively narrow focus through code navigation.File search grounds the vulnerability description in repository locations, while repeated open–read–navigate actions align the commit diff with vulnerable logic.
  • Failure modes: Memorized cases remain uncommon, with rates of 0%, 24%, and 0.8% for Qwen, Llama, and Gemma on the realistic dataset, and 0%, 21%, and 0.5% on the random dataset.These cases omit prior CVE retrieval and may involve documentation-only or irrelevant code changes.
  • Failure modes: 58.7% of failures on the realistic dataset and 55.9% on the random dataset are Superficial Associations based on surface cues without causal alignment.The agent may rely on keyword overlap, file names, or coarse semantic similarity instead of linking code changes to the CVE.
  • Dataset effects: Agent trajectories are similar across datasets, indicating stable reasoning largely driven by CVE characteristics and insensitive to dataset distribution.The reported difference instead arises from structural properties of the candidate sets.
  • Dataset effects: Random negatives permit coarse semantic filtering, whereas PatchFinder-style hard negatives share surface similarities and require interpreting the CVE root cause, affected component, exploit mechanism, and code changes.Thus, random negative construction underestimates task difficulty, while hard-negative candidate sets provide a more faithful evaluation of real-world capability.

6.3 Results of RQ3: Efficiency

Favia incurs a clear efficiency cost because its multi-turn agent repeatedly reprocesses prior context, but this expenditure enables iterative, evidence-driven code inspection. Its token usage remains stable across dataset compositions and is economically feasible given the additional patches it identifies.

  • Token consumption: Multi-turn retrieval, file localization, and iterative code navigation accumulate context and inflate Favia’s input-token cost relative to fixed-context single-pass methods.Each previous turn and prior output is counted as new input data.
  • Token consumption: Favia’s realistic-dataset mean input usage is 66,159 tokens, versus 6,456 for LLM4VFD and 9,330 for CommitShield.The difference is driven almost entirely by input tokens; Favia’s mean output is 1,043 tokens, compared with 676 for LLM4VFD and 1,035 for CommitShield.
  • Usage stability: Favia’s token-consumption pattern remains stable between realistic and random datasets, indicating that CVE complexity rather than dataset composition governs usage.The comparison is consistent with the framework being largely insensitive to data-distribution differences.
  • Efficiency trade-off: Favia misses 33 true patches on average on the realistic dataset, compared with 75 for LLM4VFD and 97 for CommitShield, identifying 42 additional patches over the strongest baseline.Because Favia evaluates the top-10 candidates per CVE, the cited per-CVE cost is 10 · 0.13$ = 1.30$; the additional expenditure is described as economically feasible.
  • Token consumption: Favia’s realistic-dataset mean total usage is 67,202 tokens because every prior turn and output is appended as new input.Mean output tokens remain modest at 1,043, while embedding overhead is limited to LLM4VFD at 362.

7 Discussion

Favia advances vulnerability-fix detection through iterative, evidence-grounded semantic reasoning and realistic ranked-candidate evaluation. The discussion also emphasizes deployment practicality while identifying computational, dataset, language-coverage, and LLM-behavior limitations.

  • Contributions: Favia correlates CVE semantics with code changes through iterative, evidence-grounded reasoning rather than surface-level similarity.This supports identification of indirect fixes, multi-file patches, and subtle cases that similarity-based approaches can miss.
  • Evaluation methodology: Randomly sampled commits can substantially overestimate vulnerability-fix detection effectiveness because many negatives are trivially unrelated to security.Favia therefore evaluates both heterogeneous random commits and homogeneous, security-relevant candidates.
  • Efficiency and deployment: Favia remains practical for large repositories by applying costly agent-based reasoning only to top-k ranked commits.Its ranking stage can use classical machine learning, lightweight heuristics, or future retrieval systems.
  • External validity: Favia is language-agnostic, unlike CommitShield’s reliance on language-specific program-analysis pipelines and heavyweight static-analysis infrastructure.The dataset nevertheless skews toward programming languages well supported by existing tooling, limiting external validity.
  • Threats and limitations: Absolute performance may vary with prompt design, tool availability, step budgets, and latent LLM knowledge of well-known CVEs.These internal threats are only partially mitigated through fixed experimental parameters and analysis of tool usage and memorization-related failure modes.

8 Conclusion and Future Work

Favia combines scalable candidate ranking with deep, iterative semantic reasoning for vulnerability-fixing commit identification. Large-scale CVEVC evaluation shows agent-based reasoning outperforms traditional and existing LLM-based approaches under realistic candidate selection, while future work targets causal reasoning, efficiency, and broader evaluation.

  • Conclusion: Favia combines scalable candidate ranking with deep, iterative semantic reasoning to identify vulnerability-fixing commits.The framework is described as forensic and agent-based.
  • Conclusion: Agent-based reasoning consistently outperforms traditional and existing LLM-based approaches under realistic candidate selection.The comparison was conducted through large-scale evaluation on the CVEVC dataset.
  • Conclusion: The evaluation achieves the strongest precision–recall trade-offs and highest F1-scores across models.These results are reported for large-scale evaluation on CVEVC under realistic candidate selection.
  • Future Work: Future work should improve causal reasoning and confidence calibration to reduce false positives in ambiguous candidate sets.The passage identifies these improvements as directions motivated by the findings.
  • Future Work: Adaptive control of reasoning depth could reduce the computational cost of agent-based analysis.This is presented as a future direction alongside improvements in causal reasoning and confidence calibration.
  • Future Work: Extending CVEVC and similar datasets to additional ranking strategies, languages, and vulnerability classes would enable more realistic, scalable evaluation.The passage frames dataset expansion as a way to support broader evaluation without the prohibitive cost of full analysis.

A Prompts Used

The prompts specify a tool-using, iterative Thought–Code–Observation workflow that ends with a final answer. For vulnerability analysis, the agent examines the pre-commit codebase, diagnoses the CVE, analyzes the diff, and rigorously correlates changes with the vulnerability.

  • Prompt Template: The prompt instructs the assistant to solve tasks with code blobs and tools through repeated Thought, Code, and Observation sequences.Reasoning precedes tool use; Python code and printed outputs support subsequent steps.
  • Prompt Template: The template requires Python tool calls to be enclosed in <code> tags and uses print() outputs as inputs for later steps.The final response is returned through a final-answer mechanism.
  • Prompt Template: Worked examples demonstrate multi-step tool use for document question answering, image generation, translation, image analysis, web research, comparison, and computation.Examples include retrying a restrictive search, reading retrieved webpages, comparing city populations, and calculating a power.
  • Prompt Template: The vulnerability task asks whether commit ‘{{ commit_id }}‘ in repository ‘{{ repository }}‘ patches CVE ‘{{ cve_id }}‘ using the commit diff and parent-commit codebase.The agent is explicitly encouraged to search files, read contents, and navigate the repository with available tools.
  • Prompt Template: The analysis prompt directs the agent to assess vulnerability impact, triggering entry points, and the underlying root cause before evaluating the commit.Examples of impact include privilege escalation, denial of service, authentication bypass, memory corruption, and information disclosure.
  • Prompt Template: The agent must identify the scope of changed files and code elements, then rigorously determine whether the changes mitigate or eliminate the CVE’s vulnerable behavior.The prompt cautions against assuming relevance when changes cannot be tied to the CVE.
Loading 2602.12500v1…