PaperlayerView exact revision on arXiv

Source-linked AI summary

RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing

Jinyao Guo, Chengpeng Wang, Xiangzhe Xu, Zian Su, Xiangyu Zhang

arXiv:2501.18160v3cs.SEcs.PL

TL;DR

Repository-level LLM auditing must handle non-local, path-sensitive code relationships while limiting hallucinations and resource costs. RepoAudit uses demand-driven function exploration, agent memory, and validation of data-flow facts and path feasibility. It detects 40 true bugs with 78.43% precision across 15 benchmark projects and finds 185 new bugs in high-profile repositories, 174 of which were confirmed or fixed.

  • Problem

    Existing LLM auditing methods support compilation-free customization but are largely limited to small codebases, while repository bugs require reasoning across complex data-flow and path relationships.

  • Method

    RepoAudit performs path-sensitive, demand-driven repository exploration with agent memory and validates data-flow facts and inter-procedural path-condition satisfiability.

  • Results

    40 true bugs were detected across 15 benchmark projects with 78.43% precision, while 185 new bugs were found in high-profile projects, 174 confirmed or fixed.

  • Takeaways & Limitations

    RepoAudit provides precise and efficient repository-level code auditing while reproducing bugs found by existing techniques and identifying previously unknown bugs.

  • Takeaways & Limitations

    Analysis overhead rises with repository source-element count, and limiting flow analysis to four functions can miss bugs involving longer call chains.

Abstract

from arXiv · show

Code auditing is the process of reviewing code with the aim of identifying bugs. Large Language Models (LLMs) have demonstrated promising capabilities for this task without requiring compilation, while also supporting user-friendly customization. However, auditing a code repository with LLMs poses significant challenges: limited context windows and hallucinations can degrade the quality of bug reports, and analyzing large-scale repositories incurs substantial time and token costs, hindering efficiency and scalability. This work introduces an LLM-based agent, RepoAudit, designed to perform autonomous repository-level code auditing. Equipped with agent memory, RepoAudit explores the codebase on demand by analyzing data-flow facts along feasible program paths within individual functions. It further incorporates a validator module to mitigate hallucinations by verifying data-flow facts and checking the satisfiability of path conditions associated with potential bugs, thereby reducing false positives. RepoAudit detects 40 true bugs across 15 real-world benchmark projects with a precision of 78.43%, requiring on average only 0.44 hours and $2.54 per project. Also, it detects 185 new bugs in high-profile projects, among which 174 have been confirmed or fixed. We have open-sourced RepoAudit at https://github.com/PurCL/RepoAudit.

1. Introduction

Repository-level auditing is difficult because non-local, path-sensitive program relationships exceed direct prompting and conventional analysis assumptions. REPOAUDIT addresses this with demand-driven exploration, agent memory, and validation while achieving strong benchmark accuracy and efficiency.

  • LLM auditing directly analyzes source code without compilation or execution, but existing techniques largely remain limited to small-scale codebases.Natural-language prompts and few-shot examples simplify customization, yet complex real-world repository auditing remains unsupported.
  • Non-local bugs require reasoning across interconnected snippets spanning multiple functions, classes, and files, making direct repository prompting ineffective.Repository-level control-flow, data-flow, and interdependency graphs can be substantially more complex than structures represented in model pretraining data.
  • Path-sensitive bug detection requires analyzing feasible execution paths, but path counts grow exponentially with program statements.This path explosion makes presenting an entire project to an LLM impractical for comprehensive analysis.
  • Human auditors instead follow execution-order paths and selectively explore those most relevant to the targeted property, excluding irrelevant paths.This practice motivates REPOAUDIT’s path-sensitive, demand-driven graph traversal.
  • REPOAUDIT combines an initiator, explorer, and validator to identify bug-relevant sources, analyze functions on demand, retain findings in memory, and validate outputs.The explorer follows relevant inter-function propagation, while validation checks well-formed properties to reduce hallucinations.
  • 40 true bugs were detected across 15 real-world projects at 78.43% precision, averaging 0.44 hours and $2.54 per project.The evaluation also reports 185 new bugs in actively maintained repositories, with 95 and 79 confirmed and fixed, respectively.

2. Preliminaries

Repository auditing requires path-sensitive reasoning over large program graphs, while LLMs show useful localized abilities but also hallucinate under broad prompting. REPOAUDIT uses abstraction, points-to reasoning, and validation to focus analysis on feasible data-flow paths.

  • 2.1. Auditing Entails Path-Sensitive Reasoning on Complex Graphs: Many critical bug types require modeling whole-project graphs and reasoning about properties along and across individual execution paths.NPD detection, for example, uses data dependence graphs whose nodes represent statements and whose edges represent data-flow facts.
  • 2.1. Auditing Entails Path-Sensitive Reasoning on Complex Graphs: A feasible path contains execution-order statements whose conditional checks can be jointly satisfied by some input.Data-flow facts are analyzed along such paths to determine whether source values can reach bug-relevant sinks.
  • 2.1. Auditing Entails Path-Sensitive Reasoning on Complex Graphs: NPD analysis follows a chain from a null source through data-flow facts to a dereferenced pointer sink across function boundaries.In the illustrated example, field2json returns NULL when repeated is false, and parse_msg later dereferences the returned value.
  • 2.1. Auditing Entails Path-Sensitive Reasoning on Complex Graphs: 19 of the 25 2024 CWE Top Dangerous Software Weakness categories require global, path-sensitive source-sink reachability reasoning.The investigation reports that these categories involve call graphs, data dependence graphs, and control-flow graphs.
  • 2.2. LLMs’ Inadequacy: Foundation models’ extended context performance on Needle in a Haystack does not align well with the path-sensitive program understanding required here.The passage notes that many models were initially pretrained on relatively short text or code snippets.
  • 2.3.2. POINTER HANDLING: Claude 3.5 Sonnet identifies two points-to outcomes for json: allocated memory when repeated is true and NULL when repeated is false.This illustrates localized pointer reasoning within a single function.
  • 2.3. LLMs’ Intrinsic Strengths: LLMs can perform basic analyses effectively within limited scopes, including program abstraction, pointer handling, and feasible-path exploration.Abstraction retains property-relevant statements and reduces the number of paths requiring analysis.

3. REPOAUDIT

REPOAUDIT navigates repositories on demand, analyzing one function at a time while preserving path-sensitive data-flow facts in agent memory. Its validator filters invalid flows and infeasible inter-procedural bug paths.

  • 3. REPOAUDIT: REPOAUDIT uses demand-driven graph traversal to align repository exploration with LLMs’ sequential reasoning.The agent navigates externally through repository graphs and prompts the model with one function at a time.
  • 3.1. Initiator: The initiator identifies bug-specific source values, and each source triggers a repository scanning procedure.Tree-sitter pattern matchers identify sources such as null values for null-pointer dereference detection.
  • 3.2. Explorer: The explorer analyzes selected functions on demand, stores results in agent memory, and uses those results to guide further exploration.Its actions include function analysis, function selection, and bug-report candidate generation.
  • 3.2.1. Analyzing Individual Functions: Agent memory maps each function and program value to feasible paths and their associated data-flow facts.This representation supports sharing analysis results across functions and caching previously obtained facts.
  • 3.2.2. Selecting Functions for Exploration: The explorer follows values across function boundaries, avoids further exploration when values do not escape, and reuses cached facts.Bug candidates are assembled from complete cross-function data-flow trails when targeted values reach relevant sinks.
  • 3.3. Validator: The validator checks control-flow ordering of data-flow facts and discards bug candidates whose inter-procedural path conditions are contradictory.A candidate is valid only when the conjunction of path conditions across functions is satisfiable.

4. Evaluation

REPOAUDIT was evaluated on memory-corruption bugs across benchmark and additional real-world projects, with comparisons covering LLM-driven and industrial tools. It achieved substantial bug-detection results with low reported time and cost, while remaining limited by source volume and bounded call context.

  • 4.1. Experimental Setup: REPOAUDIT evaluates three bug types—NPD, MLK, and UAF—across fifteen real-world projects averaging 251 KLoC.The evaluation targets weaknesses among the CWE Top 25 and includes reproducing prior reports plus finding new bugs.
  • 4.2. Evaluation Results: 40 true positives and 11 false positives yield 78.43% precision, including 21 inter-procedural bugs.REPOAUDIT reproduces previously reported bugs and reports 19 new bugs in historic versions, 14 fixed in latest commits.
  • 4.2. Evaluation Results: 0.44 hours and $2.54 per project are the average auditing time and cost, respectively.Audits complete within an average of 100.67 prompting rounds; each true-bug detection costs $0.95 on average.
  • 4.2. Evaluation Results: CoT prompting detects one true bug at single-function level and 10 at multiple-function level, while LLMDFA uses 165.23 times the prompting rounds and 123.18 times the token costs.These comparisons use functions relating to true bugs detected by REPOAUDIT.
  • 4.2. Evaluation Results: Alternative models achieve precisions of 88.46%, 86.79%, and 82.35%, while temperature variants retain precision ≥72.92% and recall ≥85.71%.The model results are for Deepseek R1, Claude 3.7 Sonnet, and OpenAI o3-mini, respectively.
  • 4.2. Evaluation Results: 185 true bugs are detected across nine additional projects, with 95 confirmed and 79 fixed by developers.The projects range from 14K to 1.7M LoC and have an overall precision of 85.71%.
  • 4.3. Limitations and Future Works: REPOAUDIT’s overhead grows with the number of source elements, and its four-function call-context limit can miss bugs involving longer call chains.The authors identify these as limitations of the current system.

5. Related Work

Prior work applies LLMs to code auditing and has introduced several vulnerability benchmarks, but repository-level auditing remains divided between LLM assistance and conventional symbolic analysis. Existing benchmarks may omit calling context for buggy functions.

  • 5. Related Work: LLM code-auditing research commonly uses prompts to analyze source code without compilation or execution and simplify customization through natural-language requirements.The cited work includes prompt engineering and few-shot examples.
  • 5. Related Work: BigVul, PrimeVul, and DiverseVul are established code-auditing benchmarks, but they lack calling context for buggy functions.The missing context is described as degrading the validity of function-level auditing techniques.
  • 5. Related Work: Repository-level techniques generally either use LLMs to support symbolic analyzers or scan repositories with LLM-centered approaches.The first group uses LLMs for specialized prior knowledge or initial bug-report examination while conventional symbolic analyzers scan the main codebase.

6. Conclusion

RepoAudit is an autonomous LLM agent for precise, efficient repository-level auditing that mimics manual auditing and performs path-sensitive reasoning. It detects true bugs across benchmark and high-profile projects.

  • RepoAudit mimics manual code auditing while leveraging program abstraction and path-sensitive reasoning.
  • 40 true bugs were detected in 15 real-world benchmark projects with 78.43% precision, reproducing all bugs found by existing techniques.
  • 185 previously unknown bugs were detected in nine high-profile open-source projects, with 174 confirmed or fixed by developers.

Impact Statement

The paper argues that LLM-powered auditing can address several constraints of classical code auditing by operating directly on source code and reducing dependence on manually designed abstractions. It also describes applicability across programming languages and incomplete programs with little customization.

  • Source-code auditing avoids dependence on compiler-specific intermediate representations that require maintenance across compiler versions.The paper contrasts this with classical auditors, which rely on evolving compiler-generated IR formats.
  • LLM-powered auditing can avoid manually crafting abstractions or implementing analysis algorithms for particular precision levels.Classical approaches require choices such as Andersen-style or Steensgaard’s pointer analysis, with precision and scalability trade-offs.
  • Similar prompting strategies can extend RepoAudit to C/C++, Python, and JavaScript instead of reimplementing the same algorithm for each language.

A. Comparison with LLM-driven Detectors

The comparison evaluates RepoAudit against single-function, multiple-function, and agent-centric LLM detectors, as well as industrial static analyzers. RepoAudit addresses limitations in cross-function context, control-flow reasoning, and computational cost.

  • Single-function level detection: Single-function detectors miss interprocedural null-pointer bugs because they lack calling context and may overlook essential intra-function control flows.
  • Agent-centric comparison: LLMDFA-PATHSCAN requires 165 times as many prompting rounds and 123 times as many input tokens as RepoAudit on average.
  • Agent-centric comparison: LLMDFA-SRCSCAN requires 1,737 times as many prompting rounds and 1,193 times as many input tokens as RepoAudit on average.
  • Industrial tools: Meta INFER crashes on five projects even after testing multiple versions, while compilation and analysis failures limit coverage.
  • Industrial tools: Amazon CODEGURU detects no true positives on the targeted 10 projects while generating 18 false positives.

C. Ablation Study

The ablation study shows that program abstraction, validation, and caching each materially support RepoAudit’s accuracy or efficiency. Removing abstraction or validation increases false positives, while removing caching greatly increases computational costs.

  • Program abstraction: Removing program abstraction decreases true positives by 47.50% and increases false positives by 181.82%, reducing precision to 40.38%.The paper attributes this decline to complex conditional control flows and execution paths that increase hallucinations.
  • Validation: Disabling validators increases false positives to 31, a 245.45% increase caused mainly by hallucinated infeasible data-flow facts.Branch conditions, jumps, and early exits can be missed without validation.
  • Caching: Without caching, icu exceeds 72 hours and 20,000 prompting rounds, while other projects average 3.55 times more rounds, 3.48 times higher cost, and 3.75 times longer analysis.Caching is particularly valuable when dense call graphs and data-dependence graphs cause functions to recur across propagation paths.

D. Evaluation with More Reasoning Models

REPOAUDIT benefits from stronger reasoning models, improving bug detection and precision while exposing trade-offs in cost and analysis time.

  • Reasoning-model integrations identify all known bugs, discover additional unreported bugs, and achieve higher precision than the Claude 3.5 Sonnet version.The stronger performance is largely attributed to reasoning models’ ability to autonomously construct logical reasoning pathways.
  • 46 true positives and 88.46% precision were achieved with DeepSeek R1, the highest precision among evaluated models.DeepSeek R1 averaged $0.57 per project but required 4,571 seconds per project, making it the slowest model.
  • 46 true positives and 86.79% precision were achieved with Claude 3.7 Sonnet.Its average cost was $1.59 per project, the highest among the evaluated models, partly because its output token count was 1.9 times higher than DeepSeek R1’s.
  • 42 true positives and 82.35% precision were achieved with OpenAI o3-mini, which showed slightly lower code-analysis performance.The passage attributes this potentially to weaker capture of complex code semantics, particularly data-flow modeling.

E. Evaluation with Different Temperatures

REPOAUDIT remains relatively stable across temperature settings, although performance declines at the highest temperature as output randomness increases.

  • At temperature 1.0, both precision and recall decline.The authors suggest that greater randomness can produce incorrect reasoning steps, increasing false positives and false negatives.

F. Examples of False Positive/Negative

The examples expose two failure modes: hallucinated null dereferences caused by missing semantic constraints and missed leaks caused by incomplete error-path tracking.

  • The examples identify hallucination during complex control-flow reasoning and insufficient understanding of implicit semantic constraints as limitations.
  • A reported null-pointer dereference is false because YANG schema validation guarantees that vrfname is non-NULL.Although vrf_get can return NULL when both name=NULL and vrf_id=VRF_UNKNOWN, that condition cannot hold when vrfname is non-NULL.
  • REPOAUDIT’s false positive arose because the LLMs lacked the implicit semantic constraint that yang_dnode_get_string never returns NULL.The apparent bug was the dereference of vrfp without a null check.
  • Claude 3.5 Sonnet missed the memory leak because it failed to track the error-handling execution path accurately.Reasoning-oriented models such as DeepSeek R1 recognized execution paths more precisely and detected the memory-management issue.
  • A memory leak occurs when sass_prepare_context returns after calloc fails without freeing or otherwise transferring cpp_ctx.The allocated object originates in sass_make_data_compiler and is passed into sass_prepare_context.
Loading 2501.18160v3…