RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing

Motivation

• Traditional analysis may require compilable code and substantial expertise, while prior LLM auditing largely targeted small codebases; human auditors instead follow relevant execution paths.RepoAudit draws on this sequential reasoning practice to avoid exploring irrelevant paths.Source: §1 · pp. 1–2
• Directly prompting on repository fragments struggles with non-local bugs because repositories contain complex cross-file data-flow relationships and path-sensitive bugs can cause path explosion.The shadowsocks-libev project has more than one million data-flow edges.Source: §1 · pp. 1–2

Approach

• RepoAudit uses an initiator to identify property-relevant starting points, an explorer to traverse functions on demand, and a validator to check facts and path conditions.The explorer uses LLMs to distinguish relevant from irrelevant paths, extending into callers or callees when needed; validation checks outputs before storing them in agent memory and examines candidate bug paths.Source: §1 · p. 2

Results

• 78.43% precision: RepoAudit reproduced 21 previously reported bugs and found 19 new bugs across 15 real-world projects.The projects averaged 251 KLoC, and 14 of the 19 new bugs had been fixed in the latest commit. The implementation used Claude 3.5 Sonnet and averaged 0.44 hours and $2.54 per project.Source: §1 · p. 2
• 185 new bugs were found in ten actively maintained repositories over two months; 95 were confirmed and 79 fixed by developers.RepoAudit supports development-time auditing, unlike Meta INFER and other compilation-dependent bug detectors.Source: §1 · p. 2
• Amazon CODEGURU reported 18 false positives and no true positives, while Meta INFER reported seven true positives and two false positives.With other LLMs, RepoAudit achieved precision of 88.46% with Deepseek R1, 86.79% with Claude 3.7 Sonnet, and 82.35% with OpenAI o3-mini.Source: §1 · p. 2

2.1. Auditing Entails Path-Sensitive Reasoning on Complex Graphs

• 19 of 25 CWE Top 25 weakness categories (76%) require global, path-sensitive source–sink reachability reasoning, while six can be identified through localized AST analysis.For NPD detection, this means tracing data-flow facts from a null value to a dereferenced pointer along a feasible program path; such facts may span distant statements or directories.Source: §2.1 · p. 3

2.2. LLMs’ Inadequacy

• LLMs’ extended context windows and strong Needle in a Haystack results do not establish the path-sensitive program understanding repository auditing requires.In a controlled Claude 3.5 Sonnet test on five functions, the model hallucinated null values for almost all dereferenced pointers, even after few-shot prompting.Source: §2.2 · p. 3, Figure 2 · p. 4

2.3. Intrinsic Strengths of LLMs

• LLMs can perform program abstraction, pointer handling, and feasible-path exploration within limited scopes, potentially avoiding compilation-dependent methods that struggle to scale efficiently.The paper presents these as primitive abilities that can be leveraged for repository-level auditing.Source: §2.3 · p. 4

2.3.1. PROGRAM ABSTRACTION

• Program abstraction retains statements relevant to a property within a defined scope, reducing the program and the number of paths to analyze.For NPD, relevant statements include null assignments, propagations, guarding checks, and cross-function propagation; Claude’s abstraction of field2json omitted an irrelevant switch statement.Source: §2.3.1 · p. 4

2.3.2. POINTER HANDLING

• Within a function, LLMs can identify possible points-to facts and their path constraints, whereas SVF computes points-to facts without path conditions by default.The paper contrasts this scoped semantic reasoning with conservative, relational traditional analyses that can over-approximate possible memory objects.Source: §2.3.2 · p. 4, Figure 3 · p. 5

2.3.3. FEASIBLE PROGRAM PATH EXPLORATION

• Within a limited scope, LLMs can assess path feasibility by skipping irrelevant statements and detecting contradictory branch conditions, avoiding exploration of infeasible paths.In the field2json example, Claude identifies three feasible paths; traditional constraint-solving approaches can be computationally expensive and struggle to model behaviors such as loops, aliasing, and pointer arithmetic.Source: §2.3.3 · p. 5

3.1. Initiator

• The initiator identifies source values for analysis using tree-sitter pattern matchers tailored to supported bug types.The matchers require a one-time implementation effort; LLM synthesis is noted as an alternative avenue.Source: §3.1 · p. 5

3.2. Explorer

• RepoAudit explores only relevant functions on demand, stores analysis results in agent memory, and uses them to guide further exploration.For each source value, the explorer begins at the identified source and performs a repository scanning round; its actions include analyzing functions, selecting further functions, and generating bug-report candidates.Source: §3.2 · pp. 5–6

3.2.1. ANALYZING INDIVIDUAL FUNCTIONS

• The explorer prompts the LLM to abstract code, handle pointers, and identify data-flow facts along feasible paths, storing per-path results in agent memory.General prompts can address graph types such as DDGs and CFGs without bug-specific instructions; the memory associates a function and program value with paths and their data-flow facts.Source: §3.2.1 · p. 6

3.2.2. SELECTING FUNCTIONS FOR EXPLORATION

• The explorer follows a value into other functions only when it escapes the current function, and checks memory first to avoid repeating prior analyses.When a value propagates across a function boundary, the call graph identifies relevant functions for follow-up exploration.Source: §3.2.2 · p. 6

3.2.3. GENERATING BUG REPORT CANDIDATES

• The explorer creates bug-report candidates when data-flow facts reach a sink, assembling the cross-function facts and inter-procedural path.For some bug types, such as MLK, a candidate is reported when no sink is reached along feasible paths.Source: §3.2.3 · pp. 6–7

3.3. Validator

• The validator adds two mechanisms to improve the quality of bug reports.Source: §3.3 · p. 7

Alignment Validation of Data-flow Facts and Control

• A parsing-based check rejects data-flow facts that violate control-flow order, while an LLM check discards reports whose cross-function path conditions contradict.The latter check tests whether the conjunction of conditions along the inter-procedural path is satisfiable.Source: Alignment Validation of Data-flow… · p. 7

4.1. Bug Types and Dataset

• The evaluation targets NPD, MLK, and UAF across five projects per bug type, averaging 251 KLoC and thousands of functions.The projects were selected from prior bug reports and are described as well-maintained, with most having thousands of GitHub stars.Source: §4.1 · p. 7

4.2. Evaluation Results

• 40 true positives and 11 false positives yield 78.43% precision, with 21 true positives involving inter-procedural bugs.REPOAUDIT detects all previously reported bugs and reports 19 new bugs in historic versions, 14 of which were fixed in the latest commit; it averages 0.44 hours per project and costs $2.54 per project audit, or $0.95 per true bug detected.Source: §4.2 · p. 7, Table 2 · p. 8
• CoT prompting detects one true bug in single-function detection and 10 in multiple-level detection, while LLMDFA uses 165.23 times the prompting rounds and 123.18 times the token costs of REPOAUDIT.The CoT comparisons concern functions relating to REPOAUDIT-detected true bugs; LLMDFA is compared on relevant functions relating to buggy program paths (Chen et al., 2023; Ding et al., 2024; Wang et al., 2024a; Li et al., 2024a;b).Source: §4.2 · pp. 7–8
• Deepseek R1, Claude 3.7 Sonnet, and OpenAI o3-mini achieve precisions of 88.46%, 86.79%, and 82.35%, respectively; Claude 3.5 Sonnet retains precision of at least 72.92% and recall of at least 85.71% across tested temperatures.The Claude 3.5 Sonnet temperature settings are 0.25, 0.5, 0.75, and 1.0.Source: §4.2 · p. 8
• Across nine additional open-source projects, REPOAUDIT detects 185 true bugs at 85.71% precision, with 95 confirmed and 79 fixed by developers.The projects range from 14K to 1.7M LoC and average 420K LoC and 8.8K GitHub stars.Source: §4.2 · p. 8, Table 4 · p. 9

4.3. Limitations and Future Works

• REPOAUDIT’s analysis overhead can rise substantially with repository source-element counts, and its four-function flow limit can miss bugs involving longer call chains.The paper also notes false positives and negatives caused by LLM hallucinations.Source: §4.3 · p. 9
• Proposed improvements include stronger or task-fine-tuned models and programming-focused training data, plus broader retrieval tools and compilation-free control-flow analysis.The authors suggest that better branch and loop representation in an improved retrieval-augmented generation design could reduce false positives and false negatives.Source: §4.3 · p. 9

Related Work

• BigVul, PrimeVul (Ding et al., 2024), and DiverseVul (Chen et al., 2023) lack buggy-function calling context, degrading the validity of function-level auditing.The literature also includes work directly applying LLMs to code auditing (Zheng et al., 2025; Zheng et al.; Li et al., 2024b;a; Wang et al., 2024a).Source: §5 · p. 9
• Repository-level approaches either use LLMs to assist conventional symbolic analyzers or use LLMs as code interpreters to extract semantic properties.The symbolic-analysis group includes IRIS, which identifies sensitive values to aid CODEQL taint-style detection, but depends on compilation and lacks IDE-time analysis.Source: §5 · p. 9
• RepoAudit uses demand-driven exploration rather than LLMDFA’s exhaustive data-flow summaries, improving scalability, and individual-function analysis rather than LLMSAN’s repository-level prompting to improve recall and mitigate hallucinations.LLMDFA and LLMSAN use few-shot chain-of-thought prompting to discover data-flow paths for bug detection.Source: §5 · p. 9

• REPOAUDIT mimics manual code auditing, leveraging LLM program abstraction to conduct path-sensitive reasoning.Source: §6 · p. 9
• Powered by Claude-3.5-Sonnet, REPOAUDIT detected 40 true bugs in 15 real-world benchmark projects at 78.43% precision and reproduced all bugs discovered by existing techniques.Source: §6 · p. 9
• REPOAUDIT detected 185 previously unknown bugs in nine high-profile open-source projects, 174 of which developers confirmed or fixed.Source: §6 · p. 9

Impact Statement

• The paper sees LLM-powered auditing as potentially transformative for software engineering but notes risks of private source-code leakage and high token costs.It states that it does not expect the work to have a negative broader impact and acknowledges that the work has limitations.Source: Impact Statement · p. 10
• REPOAUDIT’s source-code analysis avoids dependence on compiler-specific IRs and supports multiple language standards, reducing the need to adapt implementations as compiler IRs change.The paper notes that Clang had ten major version updates over the past decade, with variations in generated IR requiring manual migration and maintenance.Source: Impact Statement · p. 10
• LLMs can interpret program behavior directly, avoiding manually designed abstractions and analysis algorithms for different pointer-analysis precision and scalability trade-offs.The paper contrasts this with classical auditing’s reliance on pointer-analysis choices such as Andersen-style and Steensgaard’s, which require implementation effort and domain expertise.Source: Impact Statement · p. 10
• Similar prompting strategies could extend REPOAUDIT to C/C++, Python, and JavaScript without reimplementing IR-semantics reasoning for each language.The paper describes LLMs as general code interpreters with strong performance on short code snippets across programming languages.Source: Impact Statement · p. 10

A. Comparison with LLM-driven Detectors

• The LLM comparison covers single-function and multiple-function prompting, while the agent-centric baseline is LLMDFA, compared with RepoAudit only on NPD and UAF because LLMDFA does not support MLK.Single-function evaluation supplies sink-containing functions and few-shot examples; multiple-function evaluation supplies relevant functions on buggy paths to avoid exceeding context limits. LLMDFA-PATHSCAN summarizes functions on buggy paths, whereas LLMDFA-SRCSCAN summarizes functions reachable from each source.Source: A. Comparison with LLM-driven Detectors · p. 13
• The single-function detector found one intra-procedural NPD bug, while the multiple-function detector found 10 bugs; additional calling context helped, but hallucinated data-flow facts still caused false negatives.RepoAudit’s path-sensitive reasoning uses program abstraction to discover data-flow facts more precisely.Source: Figure 6 · p. 13, A. Comparison with LLM-driven Detectors · pp. 13–14

B. Comparison with Industrial Bug Detectors

• Meta INFER supports all three evaluated bug types but requires successful compilation; CODEGURU analyzes source without compilation but does not support MLK detection.The study manually labels detector reports as true or false positives and compares industrial-tool results with RepoAudit’s results in Table 2.Source: B. Comparison with Industrial Bug… · p. 14
• Meta INFER yielded 7 TPs and 2 FPs across eight projects, while CODEGURU found 0 TPs and 18 FPs on 10 projects; RepoAudit identified 40 TPs.Meta INFER detected no MLK or UAF bugs found by RepoAudit. Its five TPs on libfreenect relied on an assumption that realloc and malloc can return null, which does not always hold; RepoAudit does not rely on that assumption.Source: B. Comparison with Industrial Bug… · p. 15
• Meta INFER analyzed eight projects after five of 13 compilable projects crashed, while CODEGURU’s 18 false positives reflect limited robustness to different forms of similar buggy code.The INFER crashes persisted across multiple tested versions; the authors attribute CODEGURU’s limitations to its formal-reasoning techniques and machine-learning models detecting only certain bug patterns.Source: B. Comparison with Industrial Bug… · p. 15

C. Ablation Study

• The ablations separately remove program abstraction, validation, or agent-memory caching during per-function exploration.REPOAUDIT-NOVAL removes validation of discovered data-flow facts and skips examining bug reports; REPOAUDIT-NOCACHE disables caching while the explorer analyzes individual functions.Source: C. Ablation Study · p. 15
• 47.50% fewer TPs and 181.82% more FPs leave REPOAUDIT-NOABS at 40.38% precision without program abstraction.The authors attribute this to complex conditional and loop structures and nested execution paths that increase hallucinations, causing missed or invented buggy propagation paths.Source: C. Ablation Study · p. 15, Table 6 · p. 15
• 31 FPs, a 245.45% increase, followed disabling validators in REPOAUDIT-NOVAL.Without validation, branches, jump statements, and early exits can lead to hallucinated data-flow facts that do not satisfy feasibility requirements; NOVAL also skips bug-report examination.Source: C. Ablation Study · p. 15
• Without caching, REPOAUDIT-NOCACHE exceeds 72 hours and 20,000 prompting rounds on icu; other projects average 3.55x rounds, 3.48x cost, and 3.75x analysis time.Parsing-related functions in icu are cache-hit 623 times; caching helps when functions recur across faulty-value paths in intricate call graphs and dense DDGs.Source: Table 7 · p. 15, C. Ablation Study · p. 16

D. Evaluation with More Reasoning Models

• Reasoning models found all known bugs reported by prior work and additional unreported bugs with higher precision than RepoAudit powered by Claude 3.5 Sonnet.The paper attributes the improvement largely to reasoning models’ ability to construct logical pathways for more precise control-flow and data-flow reasoning.Source: D. Evaluation with More Reasoning Models · p. 16
• DeepSeek R1 detected 46 true positives at 88.46% precision, Claude 3.7 Sonnet detected 46 at 86.79%, and OpenAI o3-mini detected 42 at 82.35%.DeepSeek R1 had the highest precision among evaluated models.Source: D. Evaluation with More Reasoning Models · pp. 16–17, Table 10 · p. 17
• DeepSeek R1 cost an average of $0.57 per project and took 4,571 seconds per project, the slowest analysis time among the evaluated models.The paper notes network latency and limited throughput as possible contributors to its analysis time.Source: D. Evaluation with More Reasoning Models · p. 16
• Claude 3.7 Sonnet had similar prompting rounds and input-token use per project to DeepSeek R1, but 1.9 times its output-token count and the highest average cost, $1.59 per project.Given randomness, the authors regard its program-analysis capabilities as comparable to DeepSeek R1’s.Source: D. Evaluation with More Reasoning Models · p. 17
• OpenAI o3-mini cost $0.30 and took 624.05 seconds per project, while issuing significantly fewer queries per project than the other two reasoning models.The authors suggest its weaker handling of complex code semantics, especially data flow, and narrower function coverage may contribute to false positives and missed bugs.Source: D. Evaluation with More Reasoning Models · p. 17, Table 10 · p. 17
• RepoAudit can benefit from advances in LLM capabilities through improved precision and recall, while more efficient LLM inference can further improve auditing efficiency.Source: D. Evaluation with More Reasoning Models · p. 17

E. Evaluation with Different Temperatures

• The evaluation uses temperature 0.0 by default and tests 0.25, 0.5, 0.75, and 1.0, recording true positives, false positives, precision, and recall.Recall is measured by investigating the proportion of reproduced bugs.Source: E. Evaluation with Different… · p. 17
• Precision stays above 72% and recall remains consistently high across temperature settings, though both decline at 1.0.The reported statistics cover true positives, false positives, precision, and recall measured through reproduced bugs.Source: E. Evaluation with Different… · p. 17, Table 11 · p. 17
• At temperature 1.0, greater output randomness can cause incorrect reasoning steps, increasing false positives and false negatives.The passage associates the increase in errors with declines in both precision and recall.Source: E. Evaluation with Different… · p. 17

F. Examples of False Positive/Negative

• These cases illustrate limitations including hallucination during complex control-flow reasoning and insufficient understanding of implicit semantic constraints.Source: F. Examples of False Positive/Negative · p. 17
• YANG schema validation guarantees vrfname is non-NULL, so vrf_get cannot return NULL in this call and the reported dereference is safe; REPOAUDIT’s warning was a false positive.The LLMs did not know that yang_dnode_get_string never returns NULL.Source: F. Examples of False Positive/Negative · pp. 17–18
• When calloc fails, sass_prepare_context returns without freeing cpp_ctx, causing a memory leak detected by Deepseek R1 but missed by Claude 3.5 Sonnet.The miss resulted from Claude overlooking the error-handling path; reasoning-oriented Deepseek R1 tracked execution paths more precisely.Source: F. Examples of False Positive/Negative · p. 18