Source-linked AI summary
Towards Scalable and Cost-Efficient Vulnerability Detection: A Study on Automatic Query Generation
Ivana Clairine Irsan, Ratnadira Widyasari, Huihui Huang, Ting Zhang, Yue Liu, Ouh Eng Lieh, Shar Lwin Khin, Kang Hong Jin, David Lo
TL;DR
The paper studies how to reduce the manual effort and limited coverage of CodeQL query authoring by using LLMs to synthesize executable queries from vulnerability data. It evaluates this framework across real-world vulnerabilities and reports improved detection performance, while identifying reasoning, false-positive, and operational-cost boundaries.
Problem
Manual CodeQL query development limits vulnerability coverage, while the practical utility of LLMs for generating structured, executable security queries remains underexplored.
Method
The study extracts vulnerability patterns from fixes and natural-language descriptions, then uses structured prompting, documentation retrieval, iterative synthesis, and compilation refinement to generate CodeQL queries.
Results
263% increased detection performance is reported for patterns mined via Kimi K2.5 and translated into CodeQL queries compared with baseline methods.
Takeaways & Limitations
LLM-driven query synthesis provides a scalable pathway for combining language-model reasoning with static analysis while avoiding the need to submit project-specific source code to third-party providers.
Takeaways & Limitations
Inactive code paths can produce false positives, motivating integration with reachability analysis to reduce manual verification.
Abstract
from arXiv · showhide
Static analysis remains a cornerstone of software security, yet the effectiveness of tools such as CodeQL is often limited by the substantial manual effort required to develop high-coverage query suites. While large language models (LLMs) have emerged as a potential solution for automated code reasoning, their practical utility in generating structured, executable security queries remains underexplored. In this paper, we conduct an empirical study to evaluate the ability of LLMs to synthesize CodeQL queries using vulnerability data from the National Vulnerability Database. Through this investigation, we explore the potential of using LLMs as an automatic CodeQL query generator. Subsequently, we systematically evaluate the performance of various LLM architectures across a diverse set of real-world vulnerabilities, measuring their ability to improve detection coverage and precision. Our findings reveal that LLM-generated queries significantly enhance the baseline CodeQL queries, yielding 82% improvement in average F1-score. Furthermore, we provide a detailed cost- benefit analysis showing that while direct LLM-based scanning of entire repositories is often computationally and financially prohibitive, leveraging LLMs to synthesize CodeQL queries offers a scalable and cost-effective alternative for large-scale vulnerability detection. Our results suggest that LLMs can effectively bridge the gap between unstructured vulnerability reports and formal static analysis specifications, offering a scalable path toward comprehensive automated vulnerability detection.
1 Introduction
The paper addresses limited vulnerability coverage and the expertise burden of manually authoring CodeQL queries by studying LLM-generated queries from vulnerability data. It reports broader detection coverage and evaluates the approach’s practical costs and deployment relevance.
- Motivation: Static analysis supports early vulnerability detection, but its effectiveness depends on manually written rules that encode complex vulnerability patterns.The paper identifies manual rule development as a central bottleneck for high-coverage detection.
- Motivation: CodeQL’s semantic analysis and queryable code database enable declarative vulnerability detection, yet writing effective queries requires program-analysis expertise, CodeQL proficiency, and vulnerability knowledge.These requirements restrict query authoring and contribute to insufficient detection coverage.
- Study scope: The study investigates an automated framework that generates CodeQL queries from National Vulnerability Database data and evaluates multiple proprietary and open-source LLMs.The evaluation targets high-impact CWE classes and compares the approach with established SAST tools and function-level deep-learning models.
- Results: 263% increased true-positive detections are reported for LLM-generated queries versus default CodeQL queries, without a significant increase in False Detection Rate (FDR).The result is reported for Java vulnerabilities selected from the MITRE Top 25 Most Dangerous Software Weaknesses.
- Motivation: Standard CodeQL query suites detect less than 20% of vulnerabilities in the evaluated recent subset, leaving comprehensive coverage dependent on custom queries.The paper also describes gaps for recent vulnerability patterns and novel sources and sinks in Java applications.
3 Methodology
The methodology decomposes vulnerability-specific CodeQL query generation into assessment, documentation retrieval, predicate and query synthesis, compilation checking, and iterative refinement. It evaluates compilability, real-world detection, performance conditions, and effectiveness-cost trade-offs.
- Research Questions: The study evaluates whether LLMs generate compilable CodeQL queries, detect real-world vulnerabilities, perform differently across vulnerability conditions, and balance effectiveness with cost.The research questions separate syntactic validity from downstream detection utility and operational efficiency.
- Evaluation: Detection performance is evaluated at file-level granularity against CodeQL and PDBERT baselines on real-world, high-impact software vulnerabilities.The methodology also compares localized flaws represented by short fixes with inter-procedural vulnerabilities.
- Pipeline Overview: The pipeline analyzes vulnerability-fixing commits, extracts generalizable semantic patterns, retrieves relevant CodeQL APIs, and guides synthesis with grounded documentation.A Vulnerability Assessor and Reasoner filters unsuitable fixes before synthesis, while the Relevant Documentation Retriever narrows the available API search space.
- Pipeline Overview: Predicate and Query Generator agents translate mined vulnerability logic into reusable CodeQL predicates and compose those predicates into complete queries.Separating predicate generation from query composition lets agents focus respectively on semantic logic and CodeQL structural requirements.
- Pipeline Overview: Each generated predicate and query is checked for compilation, and a Syntax Refiner iteratively corrects errors for up to three attempts.Compilation validation is used to ensure that synthesized artifacts are syntactically valid and executable as CodeQL queries.
4 Experimental Setup
The study evaluates LLM-synthesized CodeQL queries on real-world Java vulnerabilities using controlled experiments against CodeQL and PDBERT baselines. It selects recent CVEs from MoreFixes across ten CWE categories and measures detection with project-level metrics, while assessing operational cost and data-confidentiality considerations.
- Experimental design: The evaluation benchmarks automatically generated CodeQL queries against default CodeQL queries and PDBERT on known CVEs from real-world Java repositories.The study uses controlled experiments to assess vulnerability detection performance.
- Data: The dataset uses NVD-reported Java CVEs and MoreFixes vulnerability patches for pattern mining and evaluation.The study selects recent CVEs and security patches as the primary data sources.
- Data: The final selection covers ten CWE categories, generally requiring at least 20 CVE-code segment pairs for pattern extraction, with CWE-434 containing 19 pairs.The test set samples up to 20 recent CVEs per category and excludes vulnerabilities used during pattern mining.
- Cost and practicality: Direct repository-wide LLM scanning is omitted because its operational costs make large-scale deployment impractical.The study reports costs ranging from 17 USD for one Kimi K2.5 repository scan to approximately 18,000 USD for 100 GPT-4o scans.
- Cost and practicality: Avoiding direct LLM scanning also avoids transmitting proprietary source code to third-party providers, while local hosting requires substantial infrastructure investment.The paper cites four NVIDIA H200 GPUs and 150,000–300,000 USD in total local-hosting capital expenditure for Kimi K2.5.
- Metrics: The study evaluates Total Detected Vulnerabilities, Average False Discovery Rate, and Average F1-score.Average False Discovery Rate excludes projects with no retrieved paths, whereas Average F1-score assigns zero when no files are detected.
5 Results
The results show that single-shot LLM generation often fails to produce compilable CodeQL queries, while iterative syntax refinement substantially improves compilability. Generated queries then provide competitive vulnerability detection, with model strengths differing across detection rate, false discovery rate, and F1-score.
- Compilability: Nearly all models struggled to translate vulnerability patterns into compilable CodeQL code without syntax refinement.The result suggests insufficient familiarity with CodeQL-specific syntactical requirements.
- Compilability: Up to 120 queries incorrectly composed by Sonnet 4.5 were repaired through cross-model syntax refinement.Gemini 3 Flash Preview provided the specialized syntax-refinement capability in the hybrid approach.
- Compilability: Iterative syntax refinement effectively resolves lexical errors and improves the compilability of generated CodeQL queries.The Syntax Refiner Agent allows up to three repair attempts, and a hybrid strategy uses Gemini 3 Flash Preview as a syntax specialist.
RQ2: Effectiveness of Generated Queries in Detecting Real-World Vulnerabilities
Automatically generated queries improve detection performance over the default CodeQL suite, but model quality depends on semantic reasoning rather than compilability alone. Kimi K2.5 provides the strongest precision-recall balance, whereas Gemini 3 Flash Preview struggles to translate vulnerability logic into meaningful queries.
- 24.04% average F1-score was achieved by Kimi K2.5, an 82% improvement over the CodeQL baseline.
- 45.54% vulnerability detection was achieved by Claude Sonnet 4.5, but its higher false discovery rate reduced its overall balance.
- Kimi K2.5 trailed Sonnet 4.5 by one CVE while maintaining a false discovery rate more than three percentage points lower.
- Gemini 3 Flash Preview generated compilable queries effectively but failed to translate vulnerability logic into semantically meaningful CodeQL queries.
- The results position LLM-synthesized queries as outperforming the standard CodeQL suite in F1-score through a stronger precision-recall balance.
RQ3: Where Automated Generation Performs Better
Performance depends on how much vulnerability context is supplied and which vulnerabilities are evaluated. Broad patch information generally helps query generation, while short-fix contexts benefit only some models and generated queries are especially effective for localized intra-procedural bugs.
- Short fixes were defined as changes involving no more than 10 lines within a single file.
- 63% improvement in Avg F1-score occurred for Sonnet 4.5 when short CVEs were used as pattern seeds, rising from 11.94% to 19.49%.
- Codex and Kimi K2.5 degraded significantly when restricted to short CVEs, indicating that they use broader patch context to craft effective queries.
- Using all available vulnerability-report information was more effective for query generation than restricting inputs to short fixes.
RQ 4: Cost-Effectiveness and Practicality of Automatic Query Generation
Automatic CodeQL query generation offers a cost-efficient alternative for vulnerability detection by combining strong detection performance with substantially lower operational expense. Kimi K2.5 provides the clearest balance of cost, query production, and benchmark detections.
- 96 CodeQL queries generated for less than USD 13 detected 50 CVEs with Kimi K2.5, increasing detections 263% over the baseline’s 19.
- Kimi K2.5 achieved the highest Average F1-score at an efficient cost point despite not producing the absolute highest number of detections.
- Kimi K2.5’s cost and performance balance supports practical deployment for maintaining and updating CodeQL query packages.
- Kimi K2.5 was 65% more affordable than Claude Sonnet 4.5 and 25% cheaper than GPT 5.2 Codex while consistently outperforming the CodeQL baseline.
- The cost-efficiency visualization compares computational cost with detection performance, with marker area proportional to #Detected.
6 Discussion
LLM-generated queries improved detection but produced substantial false alarms, including alerts on unreachable code. The discussion therefore points toward combining query generation with reachability analysis and stronger false-positive filtering.
- LLM-generated queries outperformed CodeQL in detection rates while maintaining comparable FDR, but generated a notable volume of false alarms.
- The analyzed query targeted recursive method calls lacking both depth parameters and guard conditions.
- Over 500 files were flagged by the CVE-2022-40152 query, and manual review identified a representative alert on unreachable dead code.
- The flagged code was technically risky if executed but inactive in the application, making the alert a false positive without real-world exploit risk.
Developers cannot rely solely on the prepackaged queries in CodeQL.
Standard CodeQL query suites provide limited coverage, while LLM-generated queries generally perform better across vulnerability categories. Model strengths vary by CWE class, so no single evaluated model dominates universally.
- Less than 20% detection was achieved by standard CodeQL query suites on a diverse subset of vulnerabilities from the last five years.
- Evaluated LLMs consistently outperformed the standard CodeQL baseline, although no single model dominated all vulnerability categories.
- Sonnet 4.5 was most effective for XSS, whereas Kimi K2.5 performed best for CSRF, OS command injection, deserialization, and code injection.
- All LLM-generated queries detected every vulnerability in the CWE-862 test set despite limited historical training data.
- PDBERT performed worse than CodeQL even when evaluated on selected functions and a relatively balanced MoreFixes sample.
Function-level analysis may not fully capture repository-level vulnerabilities.
Function-level analysis can miss vulnerabilities whose causes lie in annotations or cross-functional context. The discussion therefore emphasizes improving precision through advanced false-positive filtering while addressing broader vulnerability context.
- CVE-2022-46688 was fixed by adding a @RequirePOST annotation, which function-only analysis could overlook because the function’s internal logic was unchanged.
- Function-centric evaluation can fail when vulnerability causes are represented outside the analyzed function.
- LLM-generated queries significantly improved detection rates over standard CodeQL queries, but future work should prioritize advanced false-positive filtering to improve precision and average F1-score.
Real challenge is in filtering false positive.
Prior work increasingly uses LLMs and repository-level context for structured security analysis, but existing approaches retain scope limitations. IRIS relies on predefined taint-analysis templates, while VulEval focuses on existing vulnerabilities and cross-functional context.
- Text-to-SQL research progressed from rule-based and template-driven systems toward LLM methods using prompting, in-context learning, and chain-of-thought reasoning.
- IRIS uses LLMs to infer CWE-specific source and sink labels for library APIs, then integrates those labels into CodeQL to identify vulnerability paths.
- IRIS still relies on predefined query templates that support only taint-analysis vulnerability detection.
- VulEval shows that capturing cross-functional dependencies improves detection precision and recall compared with traditional intra-procedural methods.
8 Conclusion and Future Work
The study finds that LLMs can translate vulnerability patterns from NVD entries into CodeQL queries that improve detection without compromising precision. It also identifies cost and data-coverage considerations for applying this approach at scale.
- 263% improvement in detection performance was achieved by patterns mined via Kimi K2.5 and translated into CodeQL queries compared to baseline methods.
- Repeated LLM inference can be cost-prohibitive, while Kimi K2.5 balances CodeQL syntax proficiency and reasoning capability.
- High-reasoning models paired with Gemini 3 Flash Preview can support deeper vulnerability-pattern mining with cost-effective syntax correction.
- Cross-language transfer remains future work because CVE datasets predominantly contain C/C++ entries, while Java and Python have fewer reported vulnerabilities.
- The study provides publicly available replication materials, including associated code, scripts, and raw datasets.