PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing LLM-Based Vulnerability Detection via Representation-Aware Logical Ensemble

Francesco Crescenzo Grasso, Pietro Liguori, Roberto Natella, Domenico Cotroneo

arXiv:2610.33756v1cs.CR

TL;DR

Vulnerability detection research often relies on a single code representation, although raw code, ASTs, and CFGs capture different vulnerability-related signals. The paper evaluates these views and combines representation-specific LLM predictions with logical rules; RALE improves F1-score over individual models and achieves up to 78% accuracy and F1-score on a real-world dataset.

  • Problem

    Many vulnerability-detection approaches use a single representation, which does not fully exploit complementary code views that may specialize across vulnerability types.

    Source: §1 · p. 2

  • Method

    The study evaluates LLM classifiers on raw code, ASTs, and CFGs, then combines independently trained detectors using explicit logical rules.

    Source: §1 · p. 2

  • Results

    RALE improves F1-score by 5% over single-representation models and achieves up to 78% accuracy and F1-score on a real-world dataset.

    Source: §1 · p. 2, §8 · p. 14

  • Takeaways & Limitations

    The study finds that representation-specific strengths are complementary and that a hybrid rule prioritizing raw-code recall can minimize critical false negatives without a typical false-positive explosion.

    Source: §8 · p. 14

    Final evaluation used balanced test sets rather than naturally imbalanced vulnerability data, where actual false-positive rates could be higher.

    Source: §7 · p. 14

Abstract

from arXiv · show

Automated vulnerability detection in C source code is a long-standing challenge, as traditional static analysis tools often trade precision for scalability and struggle with complex vulnerability patterns. Recently, large language models (LLMs) have shown promising results; yet, most approaches rely on a single code representation, implicitly assuming uniform suitability across vulnerability types. In this work, we systematically study LLM-based vulnerability detection under multiple code representations, including raw source code, Abstract Syntax Trees (AST), and Control Flow Graphs (CFG). Through extensive empirical analysis, we show that different representations exhibit complementary strengths, with specific views being more effective for particular classes of vulnerabilities. Building on these findings, we propose a Representation-Aware Logical Ensemble (RALE) that integrates LLMs trained on different code representations through explicit logical rules and majority voting. While single-representation models achieve up to 73% F1-score, our integrated approach reaches 78% F1-score, with commensurate gains in accuracy, precision, and recall, consistently outperforming individual models. These results highlight the importance of explicitly leveraging complementary code representations when combining LLMs for effective vulnerability detection.

1 Introduction

Vulnerability detection needs automated, reliable techniques that scale, but existing analysis tools and single code representations have important limitations. The paper evaluates raw code, ASTs, and CFGs and proposes RALE, which combines representation-specific predictions using logical rules.

1 Introduction

  • Static tools often produce false positives, while dynamic tools face execution-coverage and computational-cost limits; neither natively operates at function-level granularity.Source: §1 · p. 2
  • Most approaches emphasize one representation: token-based methods weakly capture syntax and control flow, while graph-based studies do not fully exploit complementary program views.Source: §1 · p. 2
  • The study systematically evaluates raw source code, ASTs, and CFGs at global and vulnerability-type levels, finding distinct specializations across vulnerability types.Source: §1 · p. 2
  • RALE outperforms single-representation models and standard voting schemes, achieving up to 78% accuracy and F1-score on a real-world dataset.The approach combines predictions across raw code, AST, and CFG representations using explicit logical rules.Source: §1 · p. 2

2 Background

The background frames vulnerability detection as identifying exploitable flaws and reviews traditional analysis, learning-based methods, code representations, and ensemble learning.

2 Background

  • Vulnerability detection identifies software flaws that malicious actors can exploit and is often formulated as binary classification of code fragments.Source: §2 · p. 3
  • Static analysis examines code without execution but can generate false positives, whereas dynamic analysis observes runtime behavior but is limited by coverage and computational cost.Source: §2 · p. 3
  • LLM methods often process code as text, while graph-based approaches represent structural or execution information; both capture different aspects of program behavior.Source: §2 · p. 3
  • ASTs encode syntactic hierarchy, whereas CFGs represent possible control transfers between instruction blocks, suggesting complementary views for analysis.Source: §2 · p. 3
  • Ensembles aggregate diverse model predictions through methods such as majority voting or probability averaging, potentially leveraging heterogeneous input representations.Source: §2 · p. 4

3 Methodology

The methodology builds representation-specific vulnerability detectors from real-world C/C++ functions, evaluates alternative representations, and combines their predictions with a logical ensemble. Its analysis finds that representation performance varies by vulnerability category, motivating integration of complementary views.

3 Methodology

  • The methodology uses data generation, representation-specific fine-tuning, and logical aggregation to maximize vulnerability-detection coverage.Source: §3 · p. 4

3.1 Data Pipeline

  • MegaVul provides 17,380 vulnerable and 322,168 non-vulnerable C/C++ functions from 992 repositories, covering 169 CWE types and multiple code representations.Source: §3.1 · p. 4
  • ASTs and CFGs are generated or extracted as directed graphs whose edges encode syntactic hierarchy or execution flow; representations include complete and structural forms.ASTs were also generated with Clang, while CFG extraction relied on Joern for isolated function snippets; structural forms strip identifiers and literals.Source: §3.1 · pp. 4–5

3.2 Vulnerability Detection

  • The study tests 21 model-representation experiments across three architectures and seven functional representations, using accuracy, precision, recall, and F1-score.Clang CFG was excluded; a traditional static-analysis tool served as a baseline.Source: §3.2 · p. 6

3.3 Representation-Aware Logical Ensemble (RALE)

  • Different representations show heterogeneous performance across vulnerability categories, with no single representation uniformly optimal and their predictive behaviors complementary.Source: §3.3 · p. 6
  • The logical ensemble aggregates independently trained representation-specific predictions through explicit rules, preserving modularity and interpretability.Source: §3.3 · p. 6

4 Experimental Evaluation

The evaluation compares fine-tuned models across code representations, examines category-level strengths and prediction overlap, and tests ensemble strategies. RALE yields the largest reported gains over the raw-code baseline.

4.1 Fine-Tuning Setup

  • Fine-tuned models use raw code, AST, and CFG representations, with an 88%/6%/6% training, validation, and test split.The study evaluates accuracy, precision, recall, and F1-score; CodeBERT, CodeT5, and CodeGPT are included.Source: §4.1 · p. 7, §4.2 · p. 8

4.2 Evaluation of Fine-Tuned LLMs

  • Across models, raw code performs best overall, while Joern Complete AST and CFG are the strongest structured variants and closely match it.Representations omitting semantic or execution-related details show noticeable performance drops; CodeBERT mostly achieves the best overall results.Source: §4.2 · p. 8, Table 2 · p. 8

4.3 Performance Across Vulnerability Macro-Categories

  • Raw code performs best on Buffer Overflow, Input Validation and Injection, and Memory Management; CFGs excel on Resource Management and Concurrency Issues.ASTs improve performance on Error Handling and Logic Issues; the differing strengths reflect the representations’ semantic, execution, and structural emphasis.Source: §4.3 · p. 9

4.4 File-Level Complementarity Analysis

  • Models make substantially overlapping but non-identical errors, and raw code has more false positives while often detecting vulnerable samples missed by AST and CFG.AST has the lowest false-positive rate overall; the error complementarity motivates combining predictions rather than selecting one model.Source: §4.4 · p. 10, Figure 2 · p. 10

4.5 Logical Ensemble Evaluation

  • RALE accepts a raw-code vulnerability prediction directly; otherwise, it flags vulnerability when both AST and CFG agree.This rule preserves raw code’s high recall while using AST and CFG for false-positive control.Source: §4.5 · pp. 11–12, Figure 3 · p. 11
  • Approximately 5% gains in accuracy, precision, recall, and F1-score over the raw-code baseline are reported for RALE.Majority voting improves evaluation metrics by approximately 2% over the raw-code baseline.Source: §4.5 · p. 11, Table 3 · p. 11

5 Lesson Learned

The paper draws three lessons: code representations capture different vulnerability patterns, a logical ensemble can achieve high performance without prohibitive complexity, and RALE is positioned as a human assistant.

5 Lesson Learned

  • Different representations emphasize distinct structural and semantic properties, so models trained on them capture different vulnerability patterns.The paper cautions that a one-size-fits-all representation limits LLM detection capability.Source: §5 · p. 12
  • RALE achieves up to 78% F1-score while avoiding the prohibitive computational complexity associated with more opaque or deeply nested hybrid architectures.The authors describe this as a simplicity-effectiveness trade-off.Source: §5 · p. 12
  • For real-world use, the authors position RALE as a semiautomated early-warning assistant that provides initial suggestions rather than replacing manual review.This positioning reflects precision-recall trade-offs.Source: §5 · p. 12

6 Related Work

Related work spans traditional analyses, data-driven vulnerability detection, structural code representations, and ensembles. The paper distinguishes its work by studying multiple LLM code representations and combining them with RALE.

6 Related Work

  • Static analysis can produce many false positives, while dynamic analysis is constrained by execution coverage and computational cost; traditional methods also lack native function-level granularity.These limitations are discussed with markers,, and [15].Source: §6 · pp. 12–13
  • Prior LLM methods typically treat code as plain text [9], whereas some approaches use Code Property Graphs (CPGs), combining control-flow and data-flow information and capturing vulnerability patterns.Prior work reports that synthetic samples often fail to reflect real-world vulnerability patterns, limiting their effectiveness in practical settings.Source: §6 · p. 13
  • Unlike prior single-representation or ensemble approaches, this work studies Raw Code, AST, and CFG together and integrates them with RALE.The authors report that this integration exploits complementary strengths and improves detection performance, particularly by reducing false negatives.Source: §6 · p. 13

7 Threats To Validity

The authors identify limits in empirical scope, model selection, and test-set representativeness. Their evaluation uses balanced test sets, unlike the naturally imbalanced real-world setting.

7 Threats To Validity

  • Findings rely on the selected datasets and experimental settings, and relative representation performance may vary across languages, proprietary codebases, or vulnerability types.Different model backbones or hyperparameter optimization may also produce different performance baselines.Source: §7 · p. 13
  • Evaluation on balanced test sets does not fully represent naturally imbalanced vulnerability detection, where the actual false-positive rate could be higher.The authors chose balanced test sets to enable controlled comparison of code representations.Source: §7 · p. 14

8 Conclusion

RALE combines models trained on source code, AST, and CFG representations, achieving a 5% F1-score improvement over single-representation models. Its hybrid decision rule prioritizes raw-code recall and uses 2-out-of-3 voting for consensus.

  • 5% higher F1-score than single-representation models is reported for the RALE framework.Source: §8 · p. 14
  • RALE prioritizes raw-code models' recall and uses 2-out-of-3 majority voting for consensus.The authors report that this hybrid rule minimizes critical false negatives without the typical explosion of false positives.Source: §8 · p. 14
  • The authors conclude that carefully designed logical ensembles can improve the effectiveness and reliability of automated vulnerability detection.Source: §8 · p. 14

A Balancing Test

Preliminary Random Forest tests on Megavul found that a 50/50 class distribution yielded the most robust results across performance metrics. The authors explain that balancing gives both classes equal importance during optimization.

  • A 50/50 distribution yielded the most robust results across all performance metrics in preliminary Random Forest tests trained on Megavul.Source: A Balancing Test · p. 14
  • The balancing strategy assigns equal importance to the vulnerable and non-vulnerable classes during optimization.Source: A Balancing Test · p. 14
  • The authors state that balancing prevents optimization from being dominated by the majority non-vulnerable class and enhances sensitivity to security flaws.Source: A Balancing Test · p. 14

B CWE to Macro-categories Mapping

The appendix groups individual CWEs into eight macro-categories for evaluating model performance across functionally similar vulnerability types. Table 5 presents the category mapping.

Related papers

Loading 2610.33756v1…