Source-linked AI summary
Reducing False Positives in Static Bug Detection with LLMs: An Empirical Study in Industry
Xueying Du, Jiayi Feng, Yi Zou, Wei Xu, Jie Ma, Wei Zhang, Sisi Liu, Xin Peng, Yiling Lou
TL;DR
Static-analysis false positives create a substantial industrial review burden, while LLM effectiveness in proprietary enterprise settings remained unclear. This paper studies diverse LLM-based reduction techniques using Tencent data and finds high false-positive reduction with low per-alarm costs, while identifying context and reasoning limitations.
Problem
Static-analysis tools produce high false-positive rates in large-scale enterprise systems, and LLM effectiveness for reducing these alarms in proprietary software remained insufficiently evaluated.
Method
The study interviews Tencent developers and evaluates diverse LLM, prompting, hybrid static-analysis, and traditional learning-based techniques on real-world Tencent alarm data.
Results
The best techniques eliminated 94%–98% of false positives, with per-alarm costs of 2.1–109.5 seconds and $0.0011–$0.12.
Takeaways & Limitations
LLM-based techniques show strong potential for industrial false-alarm reduction while substantially lowering the manual inspection burden.
Takeaways & Limitations
LLM-based techniques remain less effective for long code contexts, complex cascaded constraints, and insufficient semantic understanding.
Abstract
from arXiv · showhide
Static analysis tools (SATs) are widely adopted in both academia and industry for improving software quality, yet their practical use is often hindered by high false positive rates, especially in large-scale enterprise systems. These false alarms demand substantial manual inspection, creating severe inefficiencies in industrial code review. While recent work has demonstrated the potential of large language models (LLMs) for false alarm reduction on open-source benchmarks, their effectiveness in real-world enterprise settings remains unclear. To bridge this gap, we conduct the first comprehensive empirical study of diverse LLM-based false alarm reduction techniques in an industrial context at Tencent, one of the largest IT companies in China. Using data from Tencent's enterprise-customized SAT on its large-scale Advertising and Marketing Services software, we construct a dataset of 433 alarms (328 false positives, 105 true positives) covering three common bug types. Through interviewing developers and analyzing the data, our results highlight the prevalence of false positives, which wastes substantial manual effort (e.g., 10-20 minutes of manual inspection per alarm). Meanwhile, our results show the huge potential of LLMs for reducing false alarms in industrial settings (e.g., hybrid techniques of LLM and static analysis eliminate 94-98% of false positives with high recall). Furthermore, LLM-based techniques are cost-effective, with per-alarm costs as low as 2.1-109.5 seconds and $0.0011-$0.12, representing orders-of-magnitude savings compared to manual review. Finally, our case analysis further identifies key limitations of LLM-based false alarm reduction in industrial settings.
1 INTRODUCTION
Static analysis tools are widely used but generate costly false alarms in large-scale enterprise software. This study evaluates LLM-based reduction techniques at Tencent and finds strong effectiveness alongside practical cost savings and identifiable limitations.
- Motivation: Over 95% false alarm rates have been reported for NPD detection by CodeQL and Infer in large-scale projects such as the Linux Kernel.Over-approximation and abstraction of features such as pointer aliasing can produce warnings along infeasible paths.
- Study scope: The study constructs a Tencent dataset from an enterprise-customized static analyzer covering real-world Advertising and Marketing Services software.The dataset was collected from historical alarms and analyzed alongside developer interviews.
- Industrial problem: Over 76% of Tencent’s static-analysis alarms were false alarms, requiring 10–20 minutes of manual inspection per alarm.False alarms could undergo a second validation round, increasing the burden on developers.
- Findings: LLMPFA eliminated 94%–98% of false positives across different backbone LLMs while maintaining high recall.The evaluation compares basic LLMs, prompting strategies, hybrid static-analysis methods, and traditional learning-based techniques.
- Findings: Average per-alarm costs ranged from 2.1 to 109.5 seconds and $0.0011–$0.12, compared with 10–20 minutes of manual inspection.These costs support the practical value of LLM-based false-alarm reduction in industrial practice.
- Limitations: LLM-based techniques performed best on DBZ bugs and worst on NPD bugs, with limitations involving long contexts, cascaded constraints, and semantic understanding.These challenges are identified as directions for future research.
2 RELATED WORK
Prior false-positive reduction research uses static analysis, supervised learning, and LLM-based techniques, but evidence remains limited for proprietary enterprise software. Existing LLM work increasingly combines prompting, code context, and static-analysis information.
- LLM-based approaches: LLM-based approaches use prompt engineering, fine-tuning, retrieval-augmented generation, and combinations of LLMs with static analysis.Examples include chain-of-thought, few-shot learning, historical vulnerability retrieval, and static-analysis context integration.
- Research gap: Existing LLM bug-detection benchmarks rely mainly on synthetic datasets and open-source projects that may differ from proprietary enterprise codebases.Prior work also had not systematically evaluated LLMs for reducing static-analysis false positives.
- Traditional approaches: Model-checking and symbolic-execution approaches aim to determine whether reported paths are feasible through formal or path-based analysis.These approaches represent established categories of static false-positive reduction methods.
- Learning-based approaches: Supervised learning approaches distinguish real bugs from false positives using source-code or warning features but require large labeled datasets for each bug type.This data requirement limits generalizability across bug types and enterprises without training data.
- Research gap: Prior surveys categorize false-positive elimination into statistical probability, dynamic program testing, machine learning, and clustering, without exploring LLMs.The cited survey also focuses on open-source software rather than proprietary enterprise settings.
3 RESEARCH SETTING
The study uses Tencent’s enterprise static-analysis workflow to collect and validate alarms, then evaluates diverse LLM, prompting, hybrid, and learning-based techniques. Its research setting covers industrial prevalence, effectiveness, costs, and bug-type variation.
- Data Collection: The data-collection pipeline consists of automated scanning, manual review, and secondary validation of verified outcomes.Static analyzers generated warnings, reviewers classified them, and the research team collected validated records.
- Data Collection: Tencent’s proprietary BkCheck scans large-scale projects without compilation and supports detection rules including NPD, logic defects, DBZ, and memory leaks.It is integrated into Tencent’s industrial code-review workflow.
- Data Collection: The study selects NPD, OOB, and DBZ as the three most frequent developer-validated bug categories, together accounting for 86% of warnings.Their approximate shares are 52%, 24%, and 10%, respectively.
- Research Questions: RQ1 examines industrial false-positive prevalence, integration into code review, and the manual effort required to validate alarms.The researchers use Tencent developer interviews and collected alarm data.
- Research Questions: RQ2 evaluates direct LLM use, advanced prompting, LLM–static-analysis hybrids, and comparisons with traditional learning-based techniques.The study explicitly separates these questions into four subquestions.
- Evaluation Design: The evaluation compares GPT-4o, Claude-Opus-4, Qwen-3-Coder, and DeepSeek-R1 with basic, CoT, few-shot, and bug-type-augmentation prompting.Hybrid methods integrate LLMs with static-analysis information for false-alarm reduction.
- Research Questions: RQ3 measures response-time and economic costs per case to assess the feasibility and sustainability of LLM deployment.The cost analysis covers average response time and API expense across models and methods.
4 RESULT AND ANALYSIS
In Tencent’s industrial setting, false alarms impose substantial review costs, while LLM-based methods—especially hybrids with static analysis—substantially reduce false positives at low processing and economic cost. Performance still varies by bug category, and case analysis identifies persistent weaknesses in long-context reasoning, complex cascaded constraints, and semantic understanding.
- RQ2: Effectiveness of LLM-based Methods: Hybrid LLM–static-analysis methods achieved 0.93–0.94 accuracy and eliminated 94–98% of false positives while maintaining high recall.These methods also provide natural-language explanations that support subsequent human validation.
- RQ2: Comparison with Traditional Learning-based Methods: Traditional learning-based methods performed poorly on enterprise code, with highest accuracy reaching only 0.36–0.44 except for LineVul.ReGVD achieved recall of 0.87 but false-positive recall of only 0.19, limiting practical alarm reduction.
- RQ3: Cost Analysis: 2.1–109.5 seconds and $0.0011–$0.12 per alarm make LLM-based reduction substantially cheaper than 10–20 minutes of manual inspection.The reported costs vary by method and backbone model, but remain within the authors’ acceptable enterprise range.
- RQ4: Breakdown Analysis: LLM-based methods perform best on Divide-by-Zero bugs and worst on Null Pointer Dereference bugs.The paper attributes this variation to simpler, more consistent DBZ patterns and more complex, diverse NPD patterns.
- RQ4: Case Analysis: LLMs remain limited by long code contexts, complex cascaded constraints, and insufficient semantic understanding.All-method failures involved functions averaging 95.6 lines longer than the dataset average, while misclassified cases had 21 more conditional statements on average.
5 DISCUSSIONS
LLM-based false alarm reduction methods show strong promise in enterprise code review, but their effectiveness varies by context and limitations remain for long-context and cascaded reasoning. Combining LLMs with static analysis offers a practical way to balance effectiveness, interpretability, and efficiency.
- Implications for Practitioners: LLM4PFA achieved 0.93–0.94 accuracy across different state-of-the-art backbone models, but recall remained below the enterprise-expected threshold of 90%.The authors conclude that some manual review remains necessary, although LLM-based methods could replace at least one round in multi-stage validation workflows.
- Implications for Practitioners: LLM-based methods can assist enterprise code reviewers by offering a more balanced trade-off between effectiveness, interpretability, and efficiency than manual rules or traditional deep learning.The comparison is framed against manually crafted rules with poor scalability and traditional deep learning approaches with limited generalization to proprietary data.
- Limitations: Effectiveness degrades when warnings require deep or cross-module context because the evaluated static analyzers mostly report short code fragments.The dataset’s strong performance partly reflects this limited context sensitivity.
- Limitations: LLMs still struggle with cascading constraints and incomplete bug-type coverage, although LLM4PFA alleviates complex constraint analysis to some extent.The authors identify complex program-behavior simulation and the focus on mainly memory-safety bugs as unresolved boundaries.
- Implications for Researchers: Knowledge-enhanced prompting, task decomposition, and hybrid pipelines are proposed as directions for improving reasoning over complex industrial code.Bug-type descriptions and illustrative examples improved reasoning accuracy, while future work could automate targeted knowledge acquisition and decompose intricate reasoning tasks.
- Implications for Researchers: Static analysis can provide reliable program facts and intermediate results that reduce hallucinations and help LLMs break down complex reasoning tasks.The authors present LLMs and static analysis as complementary: LLMs generalize across patterns, while static analysis supplies structured program information.
6 THREATS TO VALIDITY
The study addresses validity threats through repeated reviews, multiple LLM trials, and rigorous filtering, but its dataset remains limited by enterprise access constraints.
- Internal Validity: 433 real cases spanning 10 months formed the final dataset after enterprise constraints limited collection to available data and rigorous filtering was applied.The authors regard the dataset as useful for identifying common false-positive patterns and evaluating LLM-based reduction.
- Construct Validity: Multiple review rounds by experienced developers and professional code reviewers were used to mitigate manual-labeling reliability concerns.The research team performed additional validation and reported no correctness issues in the reviewed labels.
- Internal Validity: Multiple trials and majority outcomes were used to mitigate reliability and consistency risks caused by LLM randomness.Results for methods using different LLMs were also included in a replication package for transparency and reproducibility.
7 CONCLUSION
This study provides an industrial evaluation of LLM-based static false alarm reduction at Tencent. It finds substantial false-positive prevalence, strong reduction potential, and low time and monetary costs, while also identifying practical research directions.
- Conclusion: 94–98% of false positives were eliminated by the best techniques in the Tencent industrial setting.The conclusion presents this result as evidence of the potential of LLMs for reducing false alarms in practice.
- Conclusion: The study combines developer interviews with analysis of real industrial static-analysis data to evaluate diverse LLM-based false alarm reduction techniques.It is described as the first comprehensive empirical study of these techniques in an industrial context at Tencent.
- Conclusion: LLM-based techniques are reported as cost-effective because they require low time and monetary costs.The conclusion frames cost effectiveness as a practical implication for industrial false alarm reduction.