PaperlayerView exact revision on arXiv

Source-linked AI summary

Repairing Broken Access Control in Web Applications with LLMs: An Empirical Benchmark Study

Mingrui Liu, Liang He, Xinchu Wang, Yeting Li, Dongxu Han, Yunpeng Li, Yuling Liu

arXiv:2610.34290v1cs.CRcs.AIcs.SE

TL;DR

Broken access control is difficult to repair automatically because its intended policy is embedded in application-specific business logic, and no standard benchmark existed to assess repair. The paper builds BAC-Bench and a framework for characterizing patch strategies, then evaluates LLMs and repair scaffolds. Its findings identify Content as the dominant failure mode and show that some generated patches surpass official fixes, including one that closed a long-undiscovered vulnerability.

  • Problem

    BAC repair remains underexplored and lacks a standardized benchmark, while its runtime-dependent behavior and business semantics complicate reproducible assessment.

    Source: §1 · p. 2, applications? · p. 3

  • Method

    The authors construct BAC-Bench and a four-dimensional patch-strategy framework, then study LLM repair, failure mitigations, and generated-versus-official patches.

    Source: §3 · p. 6, §9 · p. 33

  • Results

    Content remains the dominant failure mode, and some LLM-generated patches surpass official fixes, including one that closed a dormant Apache InLong vulnerability.

    Source: §9 · p. 34

  • Takeaways & Limitations

    The findings delineate the current capability boundary of LLM-based BAC repair and identify inferring application-specific intended authorization policy as a next step.

    Source: §9 · p. 34

    The effectiveness assessment depends on PoC and functional tests, since exploit blocking alone cannot establish that legitimate functionality is preserved.

    Source: §7 · p. 31

Abstract

from arXiv · show

Broken access control (BAC) ranks among the most severe web-application vulnerability classes, yet its automated repair remains largely unexplored. Unlike memory-safety vulnerabilities, BAC is a logic vulnerability whose repair demands a deep understanding of an application's access-control model and business semantics, which traditional automated vulnerability repair techniques handle poorly. Large language models (LLMs) open a promising avenue for this problem, but no standard benchmark exists for studying how well they repair BAC vulnerabilities. We construct BAC-Bench, the first benchmark for BAC repair, consisting of 46 real-world BAC vulnerabilities drawn from six widely used web applications spanning four programming languages. Each vulnerability is equipped with a sandboxed runtime environment, a PoC validation program and a functional test program. We further propose a BAC repair-strategy framework that characterizes any BAC patch along four dimensions, Position, Coverage, Mechanism, and Content. Evaluating eight base LLMs and two commercial LLM agents on BAC-Bench, we find that single-turn base LLMs repair 37.68% of the vulnerabilities on average, while two minimal scaffolds, Apply-Feedback and Navigate-and-Locate, raise this average to 66.03%, with the best base LLM reaching 75.36%, close to the 78.99% of the best agent. Notably, some LLM-generated patches surpass the official patches, and one incidentally closed a zero-day vulnerability that had lain dormant for three years. Our findings delineate the current capability boundary of LLM-based BAC repair and offer guidance toward its practical adoption.

1 Introduction

The paper asks whether LLMs can repair BAC despite its business-logic complexity and the lack of a standard benchmark. It argues that LLMs may infer authorization intent from code context, motivating an empirical evaluation.

1 Introduction

  • BAC is a severe and prevalent risk: OWASP ranks it first among web-application and API risks, and an access-control defect exposed 29 million users’ profiles.The paper also cites increases in reported access-control weaknesses and rising vulnerability disclosures.Source: §1 · pp. 1–2
  • Prior BAC repair tools remain scarce, rely on templates or execution-flow patterns, target missing checks, and have limited fit for modern web applications.Existing approaches include hand-annotated templates for PHP, path profiling, and standalone guard files.Source: §1 · p. 2
  • BAC repair is difficult because the intended policy is often undocumented, application-specific, and spread across heterogeneous code and access-control mechanisms.A mistaken repair can introduce new logic errors or over-restrict legitimate access.Source: §1 · p. 2
  • LLMs may address these challenges by reasoning over business logic, inferring implicit policy from code context, and adapting across access-control models and implementations.The paper describes these as potential strengths motivating study, not established repair results.Source: §1 · pp. 2–3
  • The central empirical question is how effectively LLMs repair BAC vulnerabilities in web applications.The supplied passage ends mid-sentence after introducing this question.Source: §1 · p. 3

applications?

The paper establishes BAC-Bench to support reproducible BAC-repair research and introduces a framework for analyzing patch strategies. It evaluates base LLMs and agents, finding that minimal scaffolds substantially improve base-model repair rates.

applications?

  • The benchmark addresses a gap: BAC’s runtime-dependent behavior makes standardized reproduction and testing difficult, hindering repair-method development and fair comparison.BAC-Bench includes reproducible runtime environments and validation programs for its cases.Source: applications? · p. 3
  • Single-turn base LLMs repair 37.68% of vulnerabilities on average; Apply-Feedback and Navigate-and-Locate raise this to 66.03%, with Claude Opus 4.8 at 75.36% versus Claude Code at 78.99%.The comparison covers eight base LLMs and two commercial closed-source agents.Source: applications? · p. 3
  • BAC-Bench contains 46 real-world vulnerabilities from six widely used applications across Java, PHP, Go, and Python, with sandboxed environments and PoC and functional tests.Each case also includes a vulnerability description, textual PoC description, and official patch.Source: applications? · pp. 3–4
  • The repair-strategy framework characterizes patches by Position, Coverage, Mechanism, and Content to analyze how patches are constructed and why they fail.The supplied contribution text describes these as the framework’s four dimensions.Source: applications? · pp. 3–4
  • Some LLM-generated patches surpass official fixes; one closed a latent vulnerability in Apache InLong that had remained undiscovered for over three years.The issue was responsibly disclosed and assigned CVE-2026-63040.Source: applications? · p. 3
  • The authors present their findings as guidance for project communities pursuing automated BAC repair.The supplied passage states this as an intended contribution.Source: applications? · p. 4

2 Background and Preliminary

BAC is a divergence between an application’s intended access policy and the decisions its code implements, arising in varied forms and contexts. The paper defines repair as blocking the reported exploit while preserving legitimate behavior.

2.1 Broken Access Control in Web Applications

  • BAC includes function-, object-, and property-level authorization failures, horizontal or vertical privilege violations, and read- or write-based operations.These perspectives are complementary; a vulnerability can be described across all three.Source: §2.1 · p. 4
  • BAC is a logic vulnerability whose correctness depends on business semantics; policies are rarely documented, and applications vary in access-control models and implementation styles.The source code is often the only practical specification, making automated generalization difficult.Source: §2.1 · pp. 4–5

2.2 Definition of BAC Vulnerability Repair

  • The intended policy is set by application designers or business requirements, whereas the implemented policy is determined by source code and is observable and testable.The paper uses this distinction to define under- and over-restriction.Source: §2.2 · p. 5
  • A BAC vulnerability exists when at least one request is denied by the intended policy but admitted by the application’s code.The paper calls the set of such requests the under-restriction set.Source: §2.2 · p. 5
  • Over-restriction captures functionality or availability defects in which legitimate requests are wrongly denied.The repair definition therefore requires both exploit closure and preservation of legitimate behavior.Source: §2.2 · pp. 5–6
  • BAC repair requires a patch to deny every request exploiting the reported vulnerability while preserving legitimate requests the original implementation allowed.A patch that blocks the exploit but breaks legitimate functionality is not effective.Source: §2.2 · pp. 5–6

3 Overview

The study uses BAC-Bench to evaluate LLM repair effectiveness, diagnose failures and mitigations, characterize patch strategies, and compare generated patches with official fixes. Its design is organized around four research questions.

3 Overview

  • The study asks how effectively LLMs repair BAC, what limitations and mitigations arise, which strategy choices underlie failures, and whether official patches are always better.These are the four research questions for the study.Source: §3 · p. 6
  • Official and LLM-generated patches are evaluated side by side under the same dynamic validation protocol.This comparison addresses the question of whether official patches are always better.Source: §3 · p. 6
  • BAC-Bench covers real-world vulnerabilities across Java, PHP, Go, and Python, reflecting diverse access-control implementation patterns.Table 1 lists the web applications included in the benchmark.Source: §3 · p. 6, Table 1 · p. 7
  • The study analyzes failing patches as application failures or semantic failures, then designs Apply-Feedback and Navigate-and-Locate to mitigate them.The overview attributes these patterns largely to hallucination and limited context windows, respectively.Source: §3 · p. 6
  • The authors analyze real-world patches to develop a four-dimension repair-strategy framework and apply it to semantically failing generated patches.The framework addresses Position, Coverage, Mechanism, and Content.Source: §3 · p. 6

4 Study Design

The study builds a benchmark and evaluation process for BAC repair, combining reproducible vulnerability instances, semantic validation oracles, and controlled LLM evaluations. It also introduces two scaffolds that give base models feedback on patch application and repository files on demand.

4.1.1 BAC vulnerbility collection.

  • 46 BAC vulnerabilities from six open-source projects and four programming languages form the dataset, collected from prior research and screened CVE records.The projects are OpenEMR, Apache InLong, BookWyrm, Nextcloud, Usememos, and XWiki; the languages are PHP, Java, Go, and Python.Source: §4.1.1 · p. 7

4.1.2 Supporting Artifacts.

  • Each vulnerability includes six artifacts, and the PoC and functional-test programs are validated by execution against the unpatched application.The artifacts include a textual PoC description, official patch and source files, sandboxed runtime, PoC validation program, and functional test program.Source: §4.1.2 · pp. 7–8

4.2 Oracle Design

  • The validation design requires repairs to block exploits, preserve legitimate functionality, and produce replayable test results.BAC lacks the universal sanitizer oracle available for some memory-safety bugs, so the study uses explicit decision criteria.Source: §4.2 · p. 8

4.2.1 Vulnerability-Blocking Oracle.

  • Unauthorized reads are detected with unique seeded markers, while writes are checked through persisted database state and victim-side rereads.For writes, database inspection checks whether the attacker’s marker persisted; a reread through the application checks that the change is visible.Source: §4.2.1 · p. 8

4.2.2 Functionality-Preservation Oracle.

  • A targeted functional test checks that legitimate users retain the affected operation, while destructive PoCs reset their state to make runs repeatable and independent.The functional test targets the vulnerable functionality; destructive PoCs create their own prerequisite data and restore the pre-run state.Source: §4.2.2 · p. 9, §4.2.3 · p. 9

4.3.1 Model Selection.

  • The evaluation selects eight base LLMs and two commercial agents, isolates runs from network access, and uses different interaction setups for models and agents.Base models receive files modified by the official patch in a single turn; agents navigate repositories over multiple turns, and the study also gives base models multi-turn scaffolds.Source: §4.3.1 · p. 9, §4.3.2 · pp. 9–10

4.3.3 Two Scaffolds for Base LLMs.

  • Apply-Feedback retries a patch after application diagnostics, returning the first patch that applies or the most recent candidate after the attempt budget.The experiments allow up to five self-healing attempts after initial generation.Source: §4.3.3 · p. 10
  • Navigate-and-Locate lets models request repository files as needed, while the two decoupled scaffolds address patch-application failures and limited source context.The scaffolds can also be composed into a single pipeline released as a reproducible BAC-Bench baseline.Source: §4.3.3 · pp. 11–12

4.4 BAC Repair-Strategy Framework

The framework describes BAC patches by where authorization is enforced, the protection scope, the implementation mechanism, and the decision’s policy content. It operationalizes these dimensions with fine-grained labels and binary mappings for comparisons.

4.4.1 Conceptual Dimensions.

  • The framework characterizes authorization patches along Position, Coverage, Mechanism, and Content, turning a pass/fail repair verdict into a description of patch strategy.The dimensions ask where authorization is enforced, how many request paths it covers, how it is implemented, and what decision it makes.Source: §4.4.1 · pp. 12–13
  • Position describes the check’s architectural location and placement on the vulnerable execution path, balancing broader reach against operation-specific precision.A centralized upstream check may govern multiple vulnerability units, while a business-logic check may provide more focused protection.Source: §4.4.1 · p. 13
  • Coverage measures protection across vulnerability units sharing a root cause and the attack paths reaching those units.Units can include read, update, and delete operations on the same object; paths can include different routes, sequences, or input conditions.Source: §4.4.1 · p. 13
  • Mechanism distinguishes reuse of existing authorization helpers, bespoke checks, and SQL filtering predicates, which carry different consistency and implementation trade-offs.Helper reuse is less error-prone, hand-built checks can invite omissions, and query-level predicates require precise handling.Source: §4.4.1 · p. 13
  • Content captures the business-specific policy decision, from role-level discrimination to per-object ownership, under models such as DAC, RBAC, ABAC, or ReBAC.The intended decision depends on the application’s access-control model and scenario.Source: §4.4.1 · p. 13

4.4.2 Operationalization and Binary Mapping.

  • The coding scheme assigns fine-grained labels before mapping them to binary outcomes; Position requires a check aligned with the vulnerable request’s execution path and timing.For multi-point patches, any path-level or temporal misalignment makes the Position result a failure.Source: §4.4.2 · pp. 13–14, Table 3 · p. 14
  • Coverage is complete when k=n among units frozen before patch inspection, but scoring considers only the documented attack path and does not judge authorization-condition correctness.Here, k is the number of units receiving a structural security intervention and n is the enumerated number of units.Source: §4.4.2 · p. 14
  • Only correct Content labels count as success; partially correct labels remain distinct from incorrect ones in detailed analysis but both map to binary failure.The study also uses an LLM for patch labeling and has cybersecurity PhD students independently relabel a stratified sample to assess agreement.Source: §4.4.2 · p. 15

5 Study Result

This section examines LLM-based BAC repair effectiveness, remaining challenges, and patterns in repair failures and audit reliability. It also presents cases where LLM-generated repairs outperform official patches, alongside defects in some official patches.

5.1 RQ1: How effective are LLM at repairing BAC vulnerabilities in web application?

  • 77.90% mean effectiveness for LLM agents exceeds the single-turn base LLM average of 37.68%, with Claude Code highest at 78.99%.Codex reaches 76.81%; the agents’ advantage over base models is especially pronounced in the matched Opus comparison: 78.99% versus 30.43%.Source: §5.1 · pp. 15–16, Table 4 · p. 16

5.2.1 Patch-Application Failures and Apply-Feedback Mitigation.

  • 96.11% macro-average applicability under Apply-Feedback, up from 62.50%, contrasts with a smaller final-effectiveness gain from 37.68% to 51.18%.The scaffold feeds git apply diagnostics back to the model, primarily mitigating malformed or otherwise inapplicable patches rather than semantic BAC failures.Source: §5.2.1 · pp. 18–19, Figure 4 · p. 19

5.2.2 Semantic Failures and Navigate-and-Locate Mitigation.

  • 66.03% mean final effectiveness under combined scaffolds exceeds Apply-Feedback’s 51.18%, while Claude Opus 4.8 reaches 75.36%.The combined condition improves 28.35 percentage points over the Base condition’s 37.68%; its PoC pass rate rises to 72.37% from 56.88%, while functional-test pass rate changes only to 78.35% from 77.45%.Source: Table 6 · p. 20, §5.2.2 · p. 21

5.3.1 Failure-Dimension Analysis.

  • Content failures are most prevalent at 23.44%, followed by Mechanism at 18.21%, Coverage at 10.60%, and Position at 6.00%.Adding Navigate-and-Locate to Apply-Feedback reduces failure prevalence across all four dimensions, with Mechanism falling from 27.16% to 11.49% and Content from 31.57% to 18.30%.Source: Table 7 · p. 22, §5.3.1 · pp. 22–23

5.3.3 Annotation Reliability.

  • 96.3% of audited patch-dimension labels agree with human consensus, with Cohen’s κ=0.926 and 16 disagreements among 432 cells.Three blinded cybersecurity PhD annotators independently labeled a stratified sample of 108 patches, and majority vote determined the consensus labels.Source: Table 9 · p. 24, §5.3.3 · p. 25

5.3.2 Dynamic Failure Patterns.

  • 72.19% Content failure prevalence in patches that pass the PoC but fail functional tests indicates that blocking an exploit can reject legitimate behavior.PoC-failure/functional-pass patches have the highest Coverage failure proportion at 45.92%, while patches failing both tests have the highest Mechanism failure proportion at 84.91%.Source: §5.3.2 · p. 24

5.4 RQ4: Are Official Patches Always Better than LLM-Generated Patches?

  • The study compares official and LLM-generated patches under the same dynamic validation protocol, identifies defects in some official patches, and introduces three representative cases.For OpenEMR’s CVE-2022-4505, the described official patch uses the session patient identifier to retrieve appointments through the existing fetchAppointments() facility.Source: §5.4 · p. 25
  • For OpenEMR CVE-2022-4505, the official patch rejected legitimate appointments at index 0, while Gemini 3.5 Flash used an ownership SQL query and passed both dynamic tests.The official repair misread array_search()’s index as a Boolean; a follow-up patch corrected this by comparing strictly with false.Source: §5.4 · pp. 26–27, Figure 8 · p. 26, Figure 7 · p. 26
  • For OpenEMR CVE-2023-2946, the official ACL check used prescriptions instead of configured value rx and denied legitimate operations, while a Kimi K2.7-Code patch used the correct pair and passed both dynamic tests.OpenEMR corrected the official check the following day by replacing prescriptions with rx.Source: §5.4 · pp. 27–28, Figure 11 · p. 27, Figure 10 · p. 27
  • While repairing a BAC vulnerability disclosed in 2023, some LLM patches incidentally protected sibling forceDelete; that vulnerability remained undiscovered for more than three years across multiple InLong releases despite remediation of related paths.Source: §5.4 · p. 29
  • For CVE-2022-4684, CVE-2022-4802, and CVE-2022-4803, defective official patches failed PoC or functional tests, while effective LLM patches passed both.The defects involved inaccurate repair position, a buggy security helper, and incomplete coverage.Source: §5.4 · p. 29
  • Some LLM repairs outperform official counterparts by using project-native security mechanisms more accurately or covering sibling paths more comprehensively.The examples include avoiding errors in authorization checks and protecting a previously unknown sibling path sharing the same root cause.Source: §5.4 · p. 29

6 Discussion

The discussion finds that lightweight scaffolds bring base LLM repair closer to agent performance, while framing ideal BAC patches as choices across four dimensions. It identifies policy reconstruction as a common challenge and notes a trade-off between scalable binary labels and richer patch analysis.

6.1 Reason for Choosing Base LLMs

  • With two lightweight scaffolds, Opus 4.8 improves from 30.43% in the single-turn setting to 75.36%, 3.63 percentage points below Claude Code.Across eight base LLMs, average repair rises from 37.68% to 66.03%.Source: §6.1 · p. 30

6.2 About an Ideal Patch

  • An ideal BAC patch balances Position and Coverage—location and protection scope—with Mechanism and Content, which assess security facilities and policy correctness.A business-layer check may fix one vulnerability unit across its attack paths, while an endpoint check may protect several units only along paths through that endpoint.Source: §6.2 · p. 30

6.3 Future Work

  • Content failures remain the most common, motivating future policy-inference work using sibling endpoints, data models, and project conventions.The proposed aim is to recover intended access-control policy from source code so generated checks implement it more faithfully.Source: §6.3 · p. 30
  • Binary correctness labels make large-scale assessment feasible but trade away analytical detail; the framework can support richer analysis of individual patches.The authors plan to refine the framework to evaluate repair tools more comprehensively and guide their improvement.Source: §6.3 · p. 31

7 Threats to Validity

The validity discussion identifies limits in benchmark scope, comparison and labeling reliability, and the operationalization of repair effectiveness. The authors describe measures intended to address these threats, including broad model coverage, repeated runs, human audit, and paired behavioral tests.

External Threats

  • BAC-Bench covers 46 vulnerabilities from six applications in four languages, so findings may not generalize to other languages or access-control architectures.The sampled cases span RBAC, ACL, and hybrid models, as well as BFLA and BOLA, horizontal and vertical violations, and read and write operations.Source: §7 · p. 31

Internal Threats

  • Three repeated runs address LLM stochasticity, while a blinded audit of 108 patches agreed with human consensus at 96.3% (κ=0.926).Most vulnerabilities were disclosed around 2022 and may have appeared in pretraining corpora, but sampled patches rarely exactly matched official fixes; one repaired a zero-day unknown to the models.Source: §7 · p. 31

Construct Threats

  • A PoC alone cannot establish repair effectiveness because a patch may block an attack yet break legitimate functionality; each vulnerability therefore receives a targeted functional test.The authors also run project test suites when available, use marker and database-state oracles, and manually review and execute oracle programs in the sandbox.Source: §7 · pp. 31–32

8 Related Work

Related work spans relatively mature BAC detection, limited and constrained automated BAC repair, and broader progress in LLM-based program repair. The paper positions its focus against both template-based access-control repairs and repository-level repair agents.

8.1 Detecting Broken Access Control

  • BAC detection includes white-box source analysis and black-box behavioral testing; the latter can identify unauthorized reads and writes, and detection is described as relatively mature in both paradigms.White-box examples infer intended permissions or authorization models, while black-box tools use observable behavior, differential responses, or SQL-layer signals.Source: §8.1 · p. 32

8.2 Repairing Broken Access Control

  • Automated BAC repair remains scarce: prior methods largely insert a single missing-check pattern, limiting coverage of object-level, property-level, and more complex violations.Existing approaches include template-based insertion, path-profiled repair, standalone guard files, and smart-contract RBAC practices that do not transfer directly to diverse web applications.Source: §8.2 · pp. 32–33

8.3 LLM-Based Automated Program Repair

  • LLM program repair has progressed from function-level generation to conversational repair and repository-level agents that navigate code and run tests.The surveyed systems include structured search, fixed localization-repair-validation pipelines, and security agents that integrate localization, patch generation, and validation.Source: §8.3 · p. 33

9 Conclusion

The study benchmarks LLM repair of real-world BAC vulnerabilities and finds that lightweight scaffolds substantially improve base-model repair rates. It also identifies intended-policy inference as the dominant failure mode and reports a generated patch that closed a long-dormant zero-day.

9 Conclusion

  • Base LLMs repaired 37.68% of vulnerabilities on average in single-turn; two minimal scaffolds raised this to 66.03%, with Claude Opus 4.8 reaching 75.36%, 3.63 percentage points below Claude Code’s 78.99%.Source: §9 · p. 33
  • Intended authorization-policy inference remains the dominant failure mode, while one generated patch incidentally closed a three-year-old Apache InLong zero-day.The vulnerability was assigned CVE-2026-63040 after responsible disclosure.Source: §9 · pp. 33–34
  • The authors identify inferring each application's precise business-specific authorization policy as a next step toward practical automated BAC repair.All benchmark artifacts are publicly available.Source: §9 · p. 34

A Examples of Patch-Application Failures

The examples show that patch-application failures can arise from source-context mismatches or file paths incompatible with the experiment's patch-application procedure.

A Examples of Patch-Application Failures

  • A blank-line mismatch near user_group caused git apply to reject the BookWyrm patch because it strictly matches hunk context.Similar failures can follow added or omitted comments, decorators, blank lines, or nearby statements.Source: A Examples of Patch-Application Failures · p. 38
  • The Apache InLong patch omitted the a/ and b/ file-header prefixes, so git apply -p1 could not resolve the target file.The human-readable file identification was incompatible with the application procedure.Source: A Examples of Patch-Application Failures · p. 38

B Examples of Semantic Failures

The semantic-failure examples show that a locally plausible authorization check can leave the underlying policy incomplete or fail to run on the vulnerable request path.

B Examples of Semantic Failures

  • The OpenEMR patch added a patients/pat_rep entry check but omitted ACL configuration changes removing that permission from Front Office and Accounting groups.The complete repair required updates to installation or upgrade paths; restricted source context prevented full exploration of OpenEMR's ACL mechanism.Source: B Examples of Semantic Failures · p. 38, Figure 16 · p. 39
  • The Usememos patch added MemoPatch.Validate, but the vulnerable PATCH handler never invoked it, leaving the ownership check outside the request's execution path.The proposed check therefore could not protect the operation.Source: B Examples of Semantic Failures · p. 39, Figure 17 · p. 39

C Frozen Vulnerability Unit Counts in BAC-Bench

Table 11 reports the frozen vulnerability-unit count for each BAC-Bench vulnerability.

C Frozen Vulnerability Unit Counts in BAC-Bench

Related papers

Loading 2610.34290v1…