PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Analysis of CodeQL False Positives and Query Refinements for Java Vulnerabilities

Amirali Sajadi, Saikat Dutta, Preetha Chatterjee

arXiv:2609.04535v2cs.SEcs.CRcs.PL

TL;DR

False positives in CodeQL can consume developer triage effort, and the study asks whether source-level recurring causes can be modeled in queries. It reviews Java cases, builds query refinements, and tests agent adaptation. The refinements remove false positives while retaining known true positives, and refinement templates improve agents’ project-specific adaptation success.

  • Problem

    CodeQL may report benign flows when project-specific protections are absent from its specifications, and prior work had not used source-level false-positive evidence to refine SAST analyses.

    Source: §I · p. 1

  • Method

    The study manually reviews false positives to derive source-level cause patterns, encodes recurring patterns as CodeQL query refinements, and tests agent adaptation using project-specific tasks.

    Source: §I · p. 2, D. RQ3: Query Refinement Using LLM… · p. 4

  • Results

    15.79% of false positives in the full selected-query dataset were removed while 87.5% of known true positives were retained; guided agents succeeded on 28/50 and 31/50 tasks.

    Source: §VI · p. 11

  • Takeaways & Limitations

    The results support modeling recurring false-positive patterns as query refinements and adapting them to project-specific contexts to reduce repeated manual triage.

    Source: §VI · p. 11

    The study covers CodeQL, Java projects, and the ten queries with the highest false-positive rates in its dataset, so results may not generalize to other languages, tools, or query sets.

    Source: §V · p. 10

Abstract

from arXiv · show

Static application security testing (SAST) tools help developers find vulnerabilities before deployment, but false positives create substantial triage effort. We study whether CodeQL false positives in Java security analysis form recurring, explainable patterns that can be reduced by refining the analysis. We run CodeQL's Java security query suite on 167 CVE instances from 110 projects, focusing on the ten queries with the highest false positive rates. We manually review 500 sampled false positive paths and locations and construct a source-level taxonomy. The five categories are Missed Path Constraint or Sanitization (36.6%), Benign Execution Context (29.4%), Missing Trust Boundary Modeling (27.6%), Imprecise Concurrency Modeling (5%), and Imprecise Sink Modeling (1.4%). Guided by these findings, we implement CodeQL refinements that detect and filter recurring false positive patterns at the query level. The refinements remove 81.8% of reviewed false positives. Across the full selected-query dataset, they remove 15.8% of reported paths and locations while retaining 7 of 8 true positives. This shows that many false positives can be reduced in the analysis, although fixed refinements often depend on project-specific context. To address this generalization gap, we evaluate whether agentic coding tools can adapt refinement patterns to new projects. Given our patterns as templates, the two tools succeed on 56% and 62% of tasks, with query compile-pass rates above 90%. Without this guidance, both succeed on only 28%, while compile rates fall to 30-36%. These results support a refinement-oriented SAST workflow in which recurring false positives are modeled in CodeQL queries and automatically adapted to different project contexts, reducing repeated triage.

I. INTRODUCTION

False positives make SAST alerts costly to triage, while CodeQL’s explicit query logic provides a place to model recurring causes. The study examines query-level refinements and whether agents can adapt them to project-specific contexts.

I. INTRODUCTION

  • False-positive alerts consume triage time and can discourage SAST use, while CodeQL may flag benign flows when project-specific protections are absent from its specifications.CodeQL’s query-driven design makes its analysis logic inspectable and modifiable.Source: §I · p. 1
  • Prior work characterized warnings or filtered alerts, but did not use source-level false-positive evidence to refine SAST analyses or study adaptation to project contexts.The paper also notes that post-run LLM filtering can be costly to repeat in CI and difficult to reproduce.Source: §I · p. 1
  • The study runs CodeQL’s security-extended suite on 167 CVEs, then manually reviews 500 confirmed false positives to identify five source-level cause families.The families include missed path constraints or sanitization, benign execution context, missing trust-boundary modeling, imprecise concurrency modeling, and imprecise sink modeling.Source: §I · p. 2
  • The query refinements remove 409/500 reviewed false positives while retaining 7/8 true positives across the full dataset.Unlike one-off location suppressions, the refinements encode recurring patterns in query logic.Source: §I · p. 2
  • Because many refinements are project-specific, the study tests agent adaptation; guided agents succeed in up to 56% of cases versus 28% without templates.The RQ3 workflow gives an agent a false positive, project context, and RQ2 refinement queries as a template.Source: §I · p. 2

II. METHODOLOGY

The methodology uses CWE-Bench-Java projects to run CodeQL, selects queries with the greatest false-positive burden, and samples paths or locations for review.

II. METHODOLOGY

  • The study uses CWE-Bench-Java, which contains 213 high-severity Java vulnerabilities across 45 CWE classes, with vulnerable and fixed versions and build scripts.The dataset supplies project context for examining CodeQL alerts, paths, and locations.Source: A. Dataset and Sampling · p. 3

A. Dataset and Sampling

  • CodeQL v2.25.4 produced reports for 167 vulnerable project versions across 110 projects; the researchers selected ten queries with the most false-positive alerts.The security-extended suite resolved to 124 queries, and alerts were classified using the benchmark’s CVE-related path criterion.Source: A. Dataset and Sampling · p. 3
  • The researchers randomly sampled 50 eligible paths or locations per selected query, totaling 500; this exceeded the 382 instances required for the stated 95±5% confidence sample.The selected queries contained 19,456 eligible false-positive alerts, expanding to 52,024 paths and locations.Source: A. Dataset and Sampling · p. 3

B. RQ1: False Positive Characterization

The researchers review source context around sampled CodeQL paths and locations, use inductive coding to label false-positive causes, and group recurring labels into five families.

B. RQ1: False Positive Characterization

  • Review artifacts map reported locations to enclosing Java declarations and include ordered source-to-sink paths when the alert produces a path.Reviewers inspect the full vulnerable project when a reported path does not provide enough context.Source: B. RQ1: False Positive Characterization · p. 3
  • Inductive coding derives labels from observed cases, with reviewers assessing whether each sampled path or location could correspond to the reported vulnerability.Reviewers follow inline hop annotations and inspect surrounding code as needed.Source: B. RQ1: False Positive Characterization · p. 3
  • For each case, reviewers record execution context, practical source control, value type, data-versus-control flow, sink relevance, and protections before the sink.Protections include authorization, sanitization, allowlisting, parameterization, and existence checks.Source: B. RQ1: False Positive Characterization · pp. 3–4
  • Each case receives one primary root-cause label and may receive secondary labels; recurring explanations are grouped into fine-grained labels and then five higher-level families.The primary label identifies the cause that most directly explains why the reported result is not an exploitable vulnerability.Source: B. RQ1: False Positive Characterization · p. 4

C. RQ2: Query Refinement for False Positive Reduction

RQ2 translates recurring false-positive mechanisms into conservative predicates added to existing CodeQL queries, then evaluates their reductions and impact on true positives.

C. RQ2: Query Refinement for False Positive Reduction

  • Each fine-grained false-positive mechanism is translated into a predicate encoding program evidence such as validation, trusted origins, authorization contexts, or lifecycle code.The goal is to test targeted refinement, not to replace the original queries universally.Source: C. RQ2: Query Refinement for False… · p. 4
  • A refinement reuses an existing query and adds suppression predicates, reporting a result only when the original query finds it and the predicates do not match.The suppression point may be the source, sink, enclosing callable, primary expression, or reported method.Source: C. RQ2: Query Refinement for False… · p. 4
  • Predicates suppress findings only when evidence supports a false-positive pattern, with stricter conditions when a pattern might hide a real vulnerability.Evaluation covers the 500 reviewed cases and applicable false positives outside the sample, and checks whether true positives are removed.Source: C. RQ2: Query Refinement for False… · p. 4

D. RQ3: Query Refinement Using LLM Agents

The study tests whether agentic coding tools can adapt recurring CodeQL refinement patterns to project-specific false positives. It compares agents given a relevant refinement template with agents working without one.

D. RQ3: Query Refinement Using LLM Agents

  • The evaluation uses 50 project-specific tasks built from false positives outside the 500 instances reviewed for RQ1.Each task contains a target false positive, project source code, and the CodeQL query that reported it.Source: D. RQ3: Query Refinement Using LLM… · p. 4
  • Guided agents receive the relevant RQ2 refinement pattern as a template, while unguided agents receive no such pattern.The guided agents must adapt the general query logic to the local project context, not receive a task-specific refined query.Source: D. RQ3: Query Refinement Using LLM… · p. 4
  • Both settings require semantic query-level refinements, and submissions that suppress alerts by matching file paths, line numbers, result indexes, or project names are rejected.The target false positive serves as developer feedback.Source: D. RQ3: Query Refinement Using LLM… · pp. 4–5
  • Evaluation checks whether each query removes the target false positive and whether it removes any known true positives across the full selected-query dataset.The second check assesses whether suppressing the target sacrifices recall.Source: D. RQ3: Query Refinement Using LLM… · p. 5
  • The study evaluates Codex with GPT 5.5 and Claude Code with Claude Opus 4.5 as agentic coding systems.The tasks involve editing repository queries, interpreting project context, and running commands.Source: D. RQ3: Query Refinement Using LLM… · p. 5

A. RQ1: False Positive Reason Taxonomy

The review identifies recurring source-level explanations for CodeQL false positives, while the dominant patterns vary across queries. The taxonomy distinguishes five broad groups and documents how code constraints, execution context, trust boundaries, and analysis precision shape the alerts.

A. RQ1: False Positive Reason Taxonomy

  • The 500 confirmed false positives receive a primary fine-grained category, with secondary categories recorded when additional factors contribute to the judgment.Fine-grained breakdowns include both primary and secondary labels, so their totals can exceed a group's primary count.Source: A. RQ1: False Positive Reason Taxonomy · p. 5
  • Missed Path Constraint or Sanitization accounts for 36.6%, Benign Execution Context 29.4%, Missing Trust Boundary Modeling 27.6%, Imprecise Concurrency Modeling 5.0%, and Imprecise Sink Modeling 1.4%.The group-level counts use each false positive's primary category.Source: A. RQ1: False Positive Reason Taxonomy · p. 5
  • Path-constraint false positives include checks, transformations, program invariants, or control-flow conditions that make the reported behavior infeasible or non-exploitable.In the Jena path-injection example, registry validation, configured-root resolution, and existence and symbolic-link checks constrain the path before deletion.Source: A. RQ1: False Positive Reason Taxonomy · p. 5
  • Benign Execution Context covers alerts in tests, examples, evaluation utilities, deployment or bootstrap code, and lifecycle code outside the relevant production attack surface.The Tika SQL-concatenation example occurs in offline evaluation tooling rather than code exposed to untrusted production input.Source: A. RQ1: False Positive Reason Taxonomy · p. 5
  • Missing Trust Boundary Modeling covers values constrained by authorization, privileged control, trusted origin, or limited disclosure.In the Kylin example, an administrator-only workflow places a request-influenced file operation outside the untrusted-user attack surface.Source: A. RQ1: False Positive Reason Taxonomy · p. 6
  • Imprecise Concurrency Modeling includes lock alerts where manual review finds that a finally block releases a lock before control leaves the protected region.The query's conservative lifecycle abstraction reports a path with a positive lock count at method exit despite the cleanup.Source: A. RQ1: False Positive Reason Taxonomy · p. 6
  • Dominant false-positive reasons vary by query, from control-flow dependencies to value constraints, implicit invariants, actor capability, trusted origins, and restricted disclosure.The query-level variation motivates tailoring refinements to specific false-positive categories rather than applying a generic filter.Source: A. RQ1: False Positive Reason Taxonomy · p. 7, Table III · p. 7

B. RQ2: Effect of Query-Level False Positive Refinements

RQ2 tests whether evidence from reviewed false positives can be encoded as CodeQL predicates to reduce recurring patterns while preserving known true positives. The refinements substantially reduce reviewed and full-dataset false positives, but project-specific context limits their generalization.

B. RQ2: Effect of Query-Level False Positive Refinements

  • Refinements encode reviewed evidence as local structural predicates, value and control-flow checks, or project-context predicates.The review evidence includes enclosing callables, local control flow, value invariants, authorization checks, and framework-specific contexts.Source: B. RQ2: Effect of Query-Level False… · pp. 7–9
  • Table V maps each refinement to its targeted RQ1 false-positive reason, while the refinements cover the taxonomy’s fine-grained mechanisms.Structural examples include same-lock cleanup in a covering finally block; context predicates suppress findings only in reviewed callable contexts.Source: Table V · p. 8, B. RQ2: Effect of Query-Level False… · pp. 8–9
  • 409/500 (81.8%) reviewed false positives and 3158/20000 (15.8%) full-dataset paths/locations were removed; 49/56 (87.5%) baseline true positives were retained.Counts are after de-duplication; the seven removed true positives came from non-production code and overlapping trusted-origin and validation/sanitization predicates.Source: B. RQ2: Effect of Query-Level False… · p. 8
  • Fixed refinements generalize poorly when a false-positive pattern depends on project-specific context, and remaining cases often involve runtime or configuration assumptions.The authors report remaining cases involving deployment modes, framework configurations, or threat-model assumptions.Source: B. RQ2: Effect of Query-Level False… · pp. 8–9

C. RQ3: Query Refinement Using LLM Agents

The study evaluates whether coding agents can adapt existing CodeQL refinements to project-specific false positives. Guided agents succeed more often and compile more queries than unguided agents, though compiled queries do not always remove the target false positive.

C. RQ3: Query Refinement Using LLM Agents

  • With refinement templates, Codex succeeds on 28/50 (56.0%) tasks and Claude Code on 31/50 (62.0%), versus 14/50 (28.0%) for each agent unguided.A task is successful when the submitted query compiles and removes the target false positive.Source: C. RQ3: Query Refinement Using LLM… · p. 9
  • Guided compile-pass rates were 45/50 (90.0%) for Codex and 46/50 (92.0%) for Claude Code, compared with 15/50 (30.0%) and 18/50 (36.0%) unguided.Across agents, compile-pass rates rose from 33/100 (33.0%) unguided to 91/100 (91.0%) guided.Source: C. RQ3: Query Refinement Using LLM… · p. 9
  • The refinement patterns serve as adaptation templates, but some guided queries compile without removing the target, especially for log-injection and path-injection.The authors note that these false positives can require project context that is difficult for agents to capture in a local query adjustment.Source: C. RQ3: Query Refinement Using LLM… · p. 9

IV. RELATED WORK

The paper situates its work among studies of developer responses to SAST warnings, empirical evaluations of static analysis, and alert validation or reduction. Its focus is source-level explanations of CodeQL Java paths and query-level refinement rather than post-report filtering or standalone query synthesis.

IV. RELATED WORK

  • Prior work links false positives and hard-to-act-on warnings with developer distrust or avoidance, and examines practical factors such as warning phrasing, workflow integration, and suppression support,,.This paper instead analyzes why specific CodeQL security paths are false positives using source code and trust-boundary evidence.Source: §IV · p. 10
  • Empirical SAST and taint-analysis studies evaluate warnings and analyzer limitations across tools, languages, and settings, [11]–, –.This study focuses on Java CodeQL security paths in CVE projects and inspects the full path and surrounding project context.Source: §IV · p. 10
  • Alert review, ranking, exploit generation, and LLM filtering address validation or post-processing, while QLCoder and Knighter synthesize queries or checkers –[49], [19],.This work complements these approaches by refining existing CodeQL rules and testing whether agents adapt those patterns to project-specific contexts.Source: §IV · p. 10

V. THREATS TO VALIDITY

The authors identify reviewer judgment, study scope, and benchmark alignment as threats to validity. They describe mitigation steps but limit generalization and true-positive preservation claims to the evaluated setting and benchmark oracle.

V. THREATS TO VALIDITY

  • False-positive labels depend on reviewer judgment, particularly for project-specific details and threat models; reviewers inspected full paths and context, recorded evidence, and excluded plausible true positives.The RQ2 refinements may still overfit reviewed findings, although they use predicates over recurring program facts rather than result-specific suppressions and are evaluated on the full dataset.Source: §V · p. 10
  • The study covers CodeQL, Java, and the ten queries with the highest false-positive rates in its dataset, so results may not generalize to other tools, languages, query sets, or agents.The authors describe the setting as practical and representative but explicitly limit claims to the evaluated setting.Source: §V · p. 10
  • Benchmark CVE alignment may label a non-aligned finding as a false positive even though it could be a real vulnerability; sampled findings were manually inspected.True-positive preservation is measured against known benchmark positives, so unknown vulnerabilities outside the benchmark oracle could still be suppressed.Source: §V · p. 10

VI. CONCLUSIONS

From 500 manually reviewed false positives, the study identifies recurring source-level causes and uses them to design executable CodeQL refinements. The refinements reduce false positives, while agentic coding systems adapt the patterns to project-specific cases.

  • The authors reviewed 500 false positives, identified five groups of source-level causes, and used them to design executable refinements.Source: §VI · p. 10
  • Over 80% of reviewed false positives and 15.79% of false positives in the full selected-query dataset were removed while retaining 87.5% of known true positives.Source: §VI · p. 11
  • With refinement patterns, Codex and Claude Code succeeded on 28/50 and 31/50 tasks, respectively, versus 14/50 each without guidance, with compile-pass rates above 90%.Source: §VI · p. 11
  • The results support a workflow that models recurring false-positive patterns as query refinements and adapts them to project-specific contexts to reduce repeated manual triage.Source: §VI · p. 11
Loading 2609.04535v2…