Source-linked AI summary
ARQ: Agentic CodeQL Query Refinement for C/C++ Vulnerability Detection
Chunyi Wang, Yunfei Ke, Junfeng Yang, Yun-Yun Tsai, Penghui Li
TL;DR
Existing C/C++ CodeQL queries can both miss vulnerabilities and flag safe code, while refining them typically depends on expertise or auxiliary artifacts. ARQ synthesizes and executes witness programs to ground an LLM refinement loop, substantially improving detection across benchmarks and real-world cases while preserving high precision.
Problem
Existing CodeQL queries still produce false positives and false negatives, while query-type-labeled benchmarks are unavailable for establishing well-defined negative and actual-positive totals.
Method
ARQ iteratively synthesizes C/C++ witness programs, compares Analyze and Oracle verdicts to identify confirmed failures, and uses those disagreements to refine queries.
Results
ARQ improved true-positive detection across 12 CodeQL queries on Juliet and FormAI while keeping Precision above 98.0%, fixed 3 CodeQL issues, and found 2 previously undiscovered library bugs.
Takeaways & Limitations
Execution-guided, example-driven refinement is presented as a practical strategy for improving static analysis without manual specifications or labeled vulnerability datasets.
Takeaways & Limitations
Because existing datasets lack query-type labels, the evaluation reports Precision rather than well-defined total-positive or total-negative-based measures.
Abstract
from arXiv · showhide
Static analyzers have been widely adopted for vulnerability detection in C/C++ programs. Query-based static analyzers (e.g., CodeQL) encode vulnerable code patterns in detection queries and match them against source code. However, existing queries still suffer from false positives (FPs, incorrectly flagging benign code as vulnerable) and false negatives (FNs, missing real vulnerabilities). We present ARQ, an agentic framework that automatically refines C/C++ CodeQL queries using execution-grounded evidence from synthesized C/C++ programs. Our key insight is that a synthesized program exposes a query's weakness whenever its execution disagrees with the query's verdict. If the program is genuinely vulnerable but the query stays silent, the query has an FN weakness; if the program is safe but the query fires anyway, it has an FP weakness. ARQ then runs an LLM-based refinement loop that repairs the query using these disagreements as ground truth. Unlike previous query refining methods, ARQ requires no labeled datasets, no commit history, and no vulnerability-specific templates. We demonstrate the effectiveness of ARQ by refining 12 official CodeQL queries using three commercial LLMs (GPT-5.4, Claude-Sonnet-4.6, and Gemini-3.5-flash). We compare both ARQ-refined and original CodeQL queries on the Juliet v1.3 and FormAI v2 datasets and show that ARQ-refined queries detect substantially more true positives, by up to 119.8\%, with a Precision of at least 98.0\% throughout. ARQ successfully fixed three unresolved GitHub issues raised in the official CodeQL query repository that had remained open for as long as \textit{27 months}. The refined queries also exposed two previously undiscovered bugs in the real-world libraries libpng and zlib.
1 Introduction
ARQ addresses persistent false positives and false negatives in manually written C/C++ CodeQL queries by refining them with execution-validated synthesized programs. Across benchmark datasets and real-world cases, it improves vulnerability detection while maintaining high precision.
- 33,000 reported CVEs from 2020–2025 underscore the security importance of detecting vulnerabilities in low-level software.
- CodeQL and related static analyzers encode vulnerable patterns in domain-specific queries used to scan source code, including large-scale projects.
- Existing CodeQL queries require substantial expertise and still produce false positives or miss real vulnerabilities, with reported bugs remaining open for 6–27 months.
- ARQ synthesizes C/C++ witness programs, executes them to validate whether query disagreements are genuine failures, and iteratively refines the query using that evidence.
- ARQ uses paired programs to guide LLM-based query refinement and suppress false positive explosion.
- Up to +119.8% more detected vulnerabilities were achieved on Juliet v1.3 and FormAI v2 while keeping Precision at or above 98.0%.
- ARQ fixed 3 unresolved CodeQL repository issues and uncovered 2 previously undiscovered bugs in zlib and libpng.
2 Preliminaries
ARQ frames query refinement as finding disagreements between static query verdicts and execution-grounded vulnerability behavior. Its illustrative DoubleFree.ql workflow generates witnesses, validates them, applies a targeted repair, and checks the repair against prior cases.
- Illustrative Example: DoubleFree.ql tracks allocations to deallocation calls but incorrectly requires both sink calls to lie within the same if branch.
- Illustrative Example: Two frees in separate independent if statements can both execute, yet the query misses this double-free because its branch-relatedness rule does not connect them.
- Illustrative Example: ARQ generates near-identical true-positive and false-negative witnesses whose differing control flow is intended to produce different query verdicts.
- Illustrative Example: Execution with a sanitizer confirms the vulnerability while the silent query establishes a genuine false negative rather than a hallucinated failure.
- Illustrative Example: The refinement relaxes sourceSinkIsRelated to allow two frees within the same enclosing function, then revalidates all confirmed witnesses before acceptance.
- Challenges of Refining Queries: Broadening query conditions can create a false positive explosion, so refinement requires independently confirmed runtime ground truth rather than query-only reasoning.
3 ARQ Framework
ARQ iteratively finds query weaknesses with execution-validated witness programs, refines the query, and regression-tests each update. Its design combines contrastive generation, ground-truth validation, targeted improvement, and rollback to preserve previously validated behavior.
- Witness generation and validation: A witness records a program, the query’s verdict, and the execution-derived ground truth, identifying false positives or false negatives when the two disagree.Confirmed witnesses are retained in a regression set and rechecked after later refinements.
- Witness generation and validation: ARQ asks an LLM to generate similar program pairs that form matched TN-FP or TP-FN cases, encouraging reasoning about vulnerability-relevant flow rather than exact names.The generator alternates pair types because improving Recall can reduce Precision, and vice versa.
- Witness generation and validation: 30% of generated witnesses remain beliefs until Analyze and Oracle confirm both the query verdict and runtime ground truth.Sanitizers such as AddressSanitizer and UndefinedBehaviorSanitizer provide a practical runtime Oracle for vulnerable behavior.
- Query refinement: ARQ’s two-phase iteration samples witnesses, applies targeted query edits, and accepts a candidate only after it passes accumulated regression checks.If misclassifications remain, the candidate is refined again; otherwise, the update is accepted.
- Query refinement: Failed refinements roll back, while the outer loop repeats witness search and regression-safe updates for up to T iterations.This preserves previously validated behavior while addressing multiple weaknesses incrementally.
- Implementation and control: ARQ delegates generation, improvement, analysis, and oracle functions to LLM-based coding agents equipped with task-specific tools, with budgets optionally expressed as time limits.CodeQL language-server support helps agents handle query syntax during improvement and analysis.
4 Evaluation
ARQ is evaluated on 12 official C/C++ CodeQL queries across Juliet v1.3 and FormAI v2, using three commercial LLMs and execution-grounded refinement. It improves true-positive detection, addresses unresolved real-world issues, and exposes previously undiscovered bugs, while evaluation remains constrained by coarse dataset labels and model-dependent outcomes.
- RQ1: Query Performance: 119.8% more true positives were detected on FormAI by Gemini-3.5-flash, while all reported Precision decreases remained within 2.0%.FormAI gains were +119.8% for Gemini-3.5-flash, +82.7% for Claude-Sonnet-4.6, and +77.7% for GPT-5.4; Juliet gains were +55.7% and +7.4% for the first two models, with GPT-5.4 decreasing by 3.2%.
- RQ2: Real-World Impact: All three ARQ-backed models produced working fixes for three unresolved UseAfterFree.ql GitHub issues, including one open for 27 months.The fixes correctly classified the supplied examples after manually verified refinement.
- RQ2: Real-World Impact: Refined queries found two previously undiscovered bugs in seven open-source C/C++ projects spanning around 985K lines of code, including libpng and zlib.The bugs were found by InconsistentNullnessTesting.ql in zlib and libpng.
- RQ3: Ablation Study: ARQ outperformed the pure-LLM baseline, which regressed below vanilla CodeQL in 5 of 6 model/dataset configurations versus 1 of 6 for ARQ.The comparison supports the role of Analyze and Oracle as execution-grounded evidence in refinement.
- RQ4: Efficiency: Agents ended with Success or No-op in 83.3% of iterations, while a full five-iteration refinement completed within an hour.Iterations averaged 433–690 seconds, produced correct witnesses 92–100% of the time, and spent nearly all time waiting on the LLM.
5 Related Work
Prior work combines LLMs with vulnerability detection and static-analysis tools, but commonly focuses on limited code scope or taint-based reasoning. These approaches do not capture the memory-lifetime and pointer-aliasing reasoning needed for many C/C++ vulnerabilities.
- Pure LLM vulnerability-detection approaches typically handle only a single function or a few files.
- LLM-assisted static-analysis systems such as IRIS and Artemis label taint sources and sinks during taint analysis.
- Taint analysis applies to limited CWE types and does not capture memory-lifetime or pointer-aliasing reasoning.
- LLMs are also used as agents interacting with software-engineering tools, including sanitizers, fuzzers, and language servers.
6 Conclusions
ARQ refines static-analysis queries through execution-validated program synthesis and iterative LLM-guided reasoning. Across CodeQL benchmarks and real-world projects, it improves true-positive detection while maintaining high precision and reveals previously unresolved issues and bugs; the approach may extend beyond CodeQL and C/C++.
- ARQ refines static-analysis queries through execution-validated program synthesis and iterative LLM-guided reasoning.
- Across 12 CodeQL queries on Juliet and FormAI, ARQ substantially improves true-positive detection while keeping Precision above 98.0%.
- ARQ fixed 3 previously unresolved CodeQL repository issues and found 2 real, previously undiscovered bugs in 7 widely-used C/C++ projects.
- Execution-guided, example-driven query refinement improves static analysis without requiring manual specifications or labeled vulnerability datasets.
- ARQ’s execution-grounding idea may apply to other static-analysis tools and languages with runtime signals or sanitizer-like tooling.