PaperlayerView exact revision on arXiv

Source-linked AI summary

From Verdict to Diagnosis: Attributable Security Review of Pull Requests

Zhuo Chen, Boyang Wang, Xiyue Zhang, Xiaoyun Xu, Ahmad-Reza Sadeghi, Stjepan Picek, Lichao Wu

arXiv:2608.25730v1cs.CR

TL;DR

Automated PR security review is commonly evaluated by whether it blocks malicious changes, but a block can miss the vulnerability that makes the PR unsafe. This paper introduces MALPR-BENCH and PRGUARD to separate verdicts from target diagnosis and evidence validation, finding that verdict totals can mask substantial differences in attributable review quality.

  • Problem

    Existing PR-security evaluations emphasize approval or rejection but do not necessarily establish that reviewers identified the vulnerability making a PR unsafe.

  • Method

    The paper introduces MALPR-BENCH with case-specific vulnerability and evidence rubrics, and PRGUARD, which validates candidate vulnerability premises against repository evidence.

  • Results

    Across held-out and production cases, similar blocking totals concealed higher target-identification and attributable-block counts for PRGUARD than CodeRabbit.

  • Takeaways & Limitations

    Automated security review should be judged by whether its block identifies and substantiates the vulnerability that justifies it, not only by verdict correctness.

  • Takeaways & Limitations

    The reported configuration gains do not isolate the effects of evidence scope or knowledge-base availability because configurations change several inputs simultaneously.

Abstract

from arXiv · show

Automated code reviewers are increasingly used as gates on pull requests (PRs), yet evaluations measure whether they block a malicious change. A block may be triggered by an unrelated issue rather than the vulnerability that makes the PR unsafe; fixing the reported issue can leave the target defect exploitable. We call this discrepancy the Verdict-Diagnosis (VD) gap. We present MalPR-Bench, a mechanism-grounded benchmark of 89 malicious PRs and 50 paired benign controls across 44 repositories and eight language families. Each malicious case has a pre-committed rubric specifying the target vulnerability, accepted mechanism descriptions, required repository evidence, and off-target findings receiving no credit. Reviews are scored separately for verdict correctness, target-vulnerability identification, and evidence validation; an attributable block requires all three. We introduce PRGuard, an attributable PR security reviewer that constructs candidate vulnerabilities and validates their premises against repository evidence using deterministic, non-executing tools and bounded retrieval. Across 31 common-coverage held-out malicious PRs, PRGuard and CodeRabbit produce similar blocking totals (22/31 vs. 24/31), but PRGuard identifies 22 target vulnerabilities versus 16 for CodeRabbit, a 1.38x difference. On 14 absence-type cases, both block 9, while PRGuard identifies 9 targets versus 3. CodeRabbit identifies 16/24 targets when required evidence lies within touched files and 0/7 when validation requires evidence outside them. Finally, PRGuard uncovers twelve previously undisclosed, proof-of-concept-backed vulnerabilities across five projects. PRGuard/DeepSeek and CodeRabbit both block 10/12 discovery PRs, but produce 10/12 and 4/12 attributable blocks, respectively. Thus, verdict-only evaluation can substantially overstate the security value of automated review.

1 Introduction

Automated PR security review can produce correct blocking verdicts without diagnosing the vulnerability that makes a change unsafe. The paper defines this Verdict–Diagnosis gap, introduces mechanism-grounded evaluation and PRGuard, and reports that attribution reveals meaningful differences hidden by verdict totals.

  • The Verdict–Diagnosis gap is the discrepancy between a reviewer’s verdict and diagnosis of the actual vulnerability.
  • A correct review must distinguish verdict correctness, target-vulnerability identification, and repository-evidence validation.
  • PRGUARD formulates candidate vulnerabilities and tests their critical premises against repository evidence, including context beyond changed lines when needed.
  • On twelve production vulnerabilities, PRGUARD and CodeRabbit both block 10/12, but produce 10/12 and 4/12 attributable blocks, respectively.
  • MALPR-BENCH contains 89 malicious PRs and 50 benign controls across 44 repositories and eight languages, with rubrics specifying targets and required evidence.
  • On 31 common-coverage cases, PRGUARD/DeepSeek identifies 1.38× as many target vulnerabilities as CodeRabbit despite similar blocking totals.

2 Problem Definition

PR security review must answer both whether a change is safe to merge and what vulnerability justifies the verdict. The framework separates identification from evidence validation because the target defect may depend on unchanged repository code and a broad or off-target diagnosis may leave it exploitable.

  • A PR security review returns a merge-safety verdict and a diagnosis of any vulnerability introduced or left exploitable by the change.
  • The target vulnerability may be absent from the diff, requiring inspection of unchanged security-relevant repository code.
  • The threat model treats the proposed diff and all PR-controlled additions, including comments, documentation, and tests, as untrusted.
  • Vulnerability identification requires stating the affected behavior, violated security condition, and resulting consequence rather than only a broad category.
  • Evidence validation grounds security-critical premises in concrete, auditable repository facts and code locations.
  • An attributable block requires a diagnosis that both identifies and substantiates the target vulnerability; divergence between verdict and diagnosis constitutes the VD gap.

3 MALPR-BENCH

MALPR-BENCH turns attributable security review into case-level ground truth and separates verdict correctness from target identification and evidence validation. Its tiered construction preserves distinct selection effects and repository-evidence requirements.

  • 3.1 Measurement Requirements: Each malicious PR has a frozen rubric specifying the target vulnerability, accepted descriptions, required security-critical premises, and repository evidence.The rubric is applied unchanged across reviewers; unrelated findings do not substitute for the target.
  • 3.2 Operational Scoring and Grading: Reviews receive separate binary scores for blocking verdict V, target-vulnerability identification I, and evidence validation E.E requires correct identification plus grounding the required premises in repository facts and code locations.
  • 3.2 Operational Scoring and Grading: An attributable block requires the PR to be blocked and the delivered diagnosis to identify and substantiate the target vulnerability.The component scores expose where verdict and diagnosis diverge, without requiring access to internal reviewer reasoning.
  • 3.3 Benchmark Construction: MALPR-BENCH combines mined project histories, public security advisories, and validated production discoveries into separate experimental tiers.The 14 development cases construct PRGUARD’s mechanism knowledge, while held-out tiers use repositories contributing no knowledge-base entries.
  • 3.3 Benchmark Construction: The discovery tier is selection-biased because the originating PRGUARD configuration selected its cases, so it does not provide an unbiased estimate of that configuration’s performance.These independently run reviews are reported for context and examine whether the VD gap appears on validated production vulnerabilities.
  • 3.4 Diagnostic Case Properties: Cases annotate defect class and evidence location separately, distinguishing validation obligations from the repository scope needed to establish them.Evidence locations range from changed lines alone (L0) to unchanged touched files (L1), structurally connected external code (L2a), or semantically corresponding external code (L2b).

4 PRGUARD

PRGUARD separates evidence collection, vulnerability identification, evidence validation, and verdict generation so repository context is gathered before hypotheses determine the review. Its deterministic, non-executing design produces attributable reviews from validated candidates.

  • 4.1 Design Overview: PRGUARD decomposes review into structural collection, knowledge-directed retrieval, candidate construction, evidence validation, deterministic verdict policy, and review synthesis.The pipeline distinguishes a plausible candidate from an established vulnerability before deriving the verdict.
  • 4.2 Evidence-First Collection: Stage 0 deterministically collects callers, callees, definitions, imports, contracts, cross-location data uses, and security-coverage comparisons before model reasoning.This evidence-first collection prevents the first hypothesis from determining which repository code later stages can see.
  • 4.2 Change Characterization: Stage 1 characterizes affected components, before-and-after behavior, guards or state, sensitive sinks, and security effects, recording unsupported elements as unknown.Its mechanism summary drives knowledge-directed retrieval, while touched components and unresolved questions drive code-directed analysis.
  • 4.3 Mechanism Knowledge: The frozen mechanism KB contains 21 mechanisms represented by 25 entries and is derived only from 14 development cases through regression testing.Held-out evaluation is used for effectiveness because development-set improvement is treated as motivation.
  • 4.3 Where Is the Required Evidence?: Two-stage mechanisms require a matched retrieval anchor and grounded relation, and are ineligible when any requested relation cannot be grounded.Fixed relations include CALLER-SOURCE, CONSUMER-SINK, GUARD-DEF, and SIBLING-ENDPOINT.
  • 4.4 Candidate Construction and Validation: Stage 3 formulates concrete exploit explanations, while Stage 4 tests the same candidates against repository evidence and may use bounded named-definition retrieval.A deterministic policy maps VALIDATED to BLOCK, DOWNGRADED to COMMENT when no candidate is validated, and otherwise to APPROVE.
  • 4.5 Safety Constraints: PRGUARD operates on a sealed repository snapshot and frozen read-only knowledge store without executing repository code or permitting PR-directed network retrieval.Model-directed actions are limited to four retrieval relations and bounded named-definition requests.

5 Evaluation

The evaluation separates verdicts from diagnoses and evidence validation, showing that similar blocking rates can conceal substantial differences in target identification. PRGUARD’s evidence-oriented configurations improve validation, while the configuration ladder does not isolate individual component effects.

  • Overall Performance: PRGUARD/DeepSeek identifies 22 target vulnerabilities versus CodeRabbit’s 16 across 31 common-coverage cases, despite blocking 22 versus 24.This is a 1.38× identification difference.
  • Overall Performance: On 14 absence-type cases, both systems block 9, while PRGUARD/DeepSeek identifies 9 targets versus CodeRabbit’s 3.Absence-type defects require recognizing missing enforcement.
  • Output-Level Scoring: Attribution is computed case by case as A = V ∧ I ∧ E rather than inferred from marginal totals.The evaluation scores verdict, identification, and evidence validation separately.
  • Overall Performance: CodeRabbit identifies 16/24 targets when evidence is within touched files but 0/7 when validation requires evidence outside them.The study reports a strong association with outside-file evidence requirements without claiming that location alone determines performance.
  • Evidence-Configuration Ladder: From E0 to E3, evidence validation rises from 6 to 12 cases for GPT-5.5 and from 6 to 10 for DeepSeek, while identification rises only from 13 to 14 and 10 to 12.The observed gains concentrate in validation, but multiple inputs change across configurations.
  • Evidence-Configuration Ladder: The evidence-configuration ladder cannot identify which component causes the observed difference because it simultaneously changes evidence scope, stage separation, retrieval, closure, and knowledge-base guidance.The authors interpret the comparison at the configuration level.

6 Real-World Vulnerability Discovery

PRGUARD is applied to filtered historical PR candidates and uncovers twelve previously undisclosed, PoC-backed production vulnerabilities. On these discovery cases, identical blocking totals produce sharply different attributable-review outcomes.

  • Discovery Findings: PRGUARD uncovers twelve previously undisclosed, PoC-backed vulnerabilities across five widely used projects and five programming languages.The findings include nine absence-type and three present-type vulnerabilities.
  • Discovery Methodology: The discovery funnel focuses manual validation on a manageable shortlist rather than exhaustively scanning every repository revision.The study therefore evaluates discovery capability rather than recall or expected yield on arbitrary PRs.
  • Discovery Methodology: All admitted findings are frozen and verified using pinned repository revisions, exhaustive source searches, and PoC reproduction before cross-system comparison.The cases include authorization failures, SSRF, missing enforcement, incorrect matching logic, and state-consistency errors.
  • The VD Gap on Real-World Vulnerabilities: PRGUARD/DeepSeek and CodeRabbit both block 10/12 discovery PRs, but produce 10/12 and 4/12 attributable blocks, respectively.CodeRabbit’s verdict and attribution totals coincide for its target-identifying reviews.
  • The VD Gap on Real-World Vulnerabilities: Six of CodeRabbit’s ten blocks are triggered by findings that do not identify the vulnerability making the PR unsafe.Repairing those reported issues does not necessarily remove the validated target vulnerability.
  • Evidence Mechanisms: The discovery cases exercise caller, sibling, consumer, and guard relationships targeted by PRGUARD’s typed evidence access patterns.Examples include following an unchanged checkpoint store and determining which search paths never consume an allowed-domain guard.

7 Discussion

The results show that verdicts and diagnoses diverge, especially for absence-type defects and cases requiring non-local evidence. Comparable blocking totals can therefore conceal differences in target identification and attributable review quality.

  • Verdict and diagnosis: 22 versus 16 target vulnerabilities were identified by PRGUARD/DeepSeek and CodeRabbit, respectively, despite blocking 22 and 24 cases.PRGUARD/DeepSeek produced 19 attributable blocks versus CodeRabbit’s 16.
  • Absence-type defects: 9/14 absence-type cases were blocked by both systems, but PRGUARD/DeepSeek identified 9 targets versus CodeRabbit’s 3.The discrepancy is concentrated in defects requiring inference of missing enforcement or non-local security context.
  • Evidence location: Target diagnosis is harder when validating a vulnerability requires evidence outside the touched files, including implicit guards or repository context absent from the diff.These cases include absence-type defects and L2a/L2b cases.
  • Evidence selection: Effective review requires selecting callers, consumers, guards, or peers relevant to the concrete security question, not merely accessing repository context.The configuration-level retrieval results motivate role-relevant evidence selection.
  • Real-world significance: The twelve production findings were previously undisclosed, proof-of-concept-backed defects across five projects, including cross-project data access, authorization bypass, and host-account takeover.The findings were reported through coordinated disclosure, and several were vendor-confirmed.

8 Related Work

Prior vulnerability benchmarks commonly use binary labels, while related review benchmarks emphasize detection and rejection. This work instead focuses on whether a reviewer can locate and explain the missing security mechanism using repository evidence.

  • Vulnerability-review benchmarks: Most vulnerability datasets assign binary vulnerable-or-benign labels to functions or commits, despite concerns about noisy labels and unrealistic class balance.The cited prior work shows that these properties can inflate apparent performance.
  • Vulnerability-review benchmarks: The closest review benchmark evaluates LLMs and tool-using agents on paired vulnerable and benign commits, measuring detection and off-target speculation.It does not make the mechanism-location question central.
  • This work’s distinction: This setting asks whether a reviewer can locate the relevant reference and explain a missing guard, binding, or state update in a PR.The evidence may be in the touched file, dependency-reachable code, or a role-related peer without a dependency edge.
  • Agents and retrieval: MALPR-BENCH represents evidence-location variation through absence-type defects, while PRGUARD uses bounded, role-typed peer queries during review.The approach treats repository knowledge as evidence selected for a concrete security question.

9 Conclusion

The paper argues that automated security review should evaluate whether a block identifies the vulnerability that justifies it, not only whether the PR is blocked. MALPR-BENCH separates these outcomes, and PRGUARD improves diagnosis relative to similar verdict totals.

  • Conclusion: PRGUARD and CodeRabbit produced similar blocking totals on 31 common-coverage cases, yet PRGUARD identified more target vulnerabilities.The conclusion states that verdict-level performance can substantially overstate review quality.
  • Conclusion: PRGUARD identified three times as many targets as CodeRabbit on absence-type defects.This is the largest reported difference in the common-coverage comparison.
  • Conclusion: On twelve previously undisclosed, proof-of-concept-backed production vulnerabilities, both systems produced identical blocking totals, but PRGUARD yielded 2.5× as many attributable blocks.The attributable outcome requires the review to identify the vulnerability and supporting evidence in addition to blocking the PR.
  • Conclusion: The findings motivate review systems that tell maintainers what must be fixed and where the supporting evidence lies.This follows the paper’s conclusion that verdict-only evaluation is insufficient.

A Ethical Considerations

The study used public source code and researcher-controlled local or test instances, with coordinated disclosure procedures for previously undisclosed findings. Hosted-model use was limited to public repository content, excluding confidential repositories.

  • Data and disclosure: The study analyzed public source code and researcher-controlled local or test instances, without accessing user data or probing production services.Previously undisclosed findings were handled through coordinated disclosure.
  • Data and disclosure: Undisclosed findings were reported with mechanisms, affected revisions, and safe proof-of-concept details, while unfixed or unacknowledged findings were anonymized.Reproduction detail was withheld until disclosure conditions permitted restoration of project identities and credit.
  • Model confidentiality: Hosted models received public repository content only, so repositories containing confidential code were outside the experiment.An on-premises open-weights backend is described as a deployment option.
  • Model confidentiality: The claimed security boundary concerns reviewed-input execution and store writes, not provider confidentiality.The authors weigh the defensive benefit against residual risk while limiting exploit detail until fixes are available.

B Reproducibility Records

The benchmark preserves auditable construction and scoring records while separating mined histories, public advisories, and production discoveries into distinct tiers. External Pool A pairs systematically omit enforcement mechanisms and classify the non-local evidence needed to identify them.

  • Reproducibility Records: The artifact links aggregate results to frozen configurations, repository revisions, manifests, rubrics, patch stacks, and execution metadata.It also records environment toggles, sealed-worktree metadata, and stage-level cost and output logs.
  • External Pool Construction: External Pool A admits advisory-backed pairs through fixed criteria excluding duplicates, unreachable residuals, oversized diffs, pool overlap, and defense-in-depth changes.Candidate admission is fixed before model execution and follows each advisory’s fix commit and parent revision.
  • External Pool Construction: Seven Pool A pairs retain production hunks but omit the listed enforcement hunk or file, while benign twins contain the complete fix.Identification requires the omitted enforcement mechanism rather than a generic or neighboring authorization issue.
  • Evidence Requirements: All seven Pool A cases are non-diff-local: five require unchanged code in touched files, and two require evidence outside the touched files.The latter two specifically test cross-file evidence acquisition.
  • Case Annotation: MALPR-BENCH distinguishes present- and absence-type defects from four evidence locations, assigning the most non-local applicable label when validation spans multiple locations.Deleting an existing guard is present-type, whereas failing to add a required guard is absence-type.

I Implementation Parameters

PRGuard uses a logged, bounded pipeline with a frozen knowledge base and constrained retrieval. Its implementation records stage outputs and limits context, repository scans, evidence closure, and recovery operations.

  • Pipeline Logging: The isolated runner records model and provider settings, repository revision, selected mechanisms, structural counts, stage outputs, timing, token counts, and verdicts.Stage 0 enumerates changed paths, while context packing caps hunk excerpts, callees, contract references, and importer excerpts.
  • Knowledge Base: The frozen knowledge base contains 21 mechanisms and 25 entries, with separate guidance allocations for Stages 3 and 4.Sixteen entries use single retrieval and nine use two_stage retrieval.
  • Retrieval: Retrieval keeps top-5 coarse results above 0.35 and top-3 fine results above 0.50 using a fixed BGE-small encoder revision.The fine score combines coarse and structural components with weights 0.8 and 0.2.
  • Records: The implementation and reproducibility records include case-level sibling evidence, disclosure status, and measured cost tables.These records cover sibling-shaped cases, coordinated disclosure, and PRGuard cost per run.
  • Bounded Retrieval: Repository loopback scans at most 5,000 files, returns 20 hits, and injects up to 12 compact evidence items.Evidence closure permits at most three named symbols per round, three rounds, and eight fetched definitions total.

J Commercial-Agent Probe

The commercial-agent probe evaluates delivered behavior under blinded conditions and treats it as diagnostic rather than part of the main performance scores. It separately examines run stability, retrieval variation, costs, and an uncovered Lua slice.

  • Blind Probe: Two fresh agents receive only a neutral diff and the reviewed repository revision, without paper-specific hints, network access, or future history.Repository search and autonomous iteration remain unrestricted.
  • Interpretation: Blind repetitions are diagnostic and are excluded from the main performance scores.Only four of six benign inputs are genuine paired twins.
  • Run Variability: Across 26 runs, tool invocations range from 12 to 47 and files read from 5 to 23, with up to 2.2× invocation-count variation for the same input.Wall time spans 1m22s–10m08s across 23 runs, while exact product token consumption is unavailable.
  • Stability Measurement: The stability analysis defines verdict wobble from the deterministic final-verdict field, recomputes attribution per run, and does not replace it with divergent prose.COMMENT-to-APPROVE changes therefore do not count as verdict wobble.
  • Retrieval Stability: Stage 1 query hashes changed in all 28 paired Pool A reruns, while Stage 0 evidence packs and file sets matched in all 14 pairs for each backend.Table 16 measures how much this upstream variation persists in the selected knowledge-base mechanisms.
  • Coverage Boundary: Lua is absent from Stage 0’s configured extension set, and the four APISIX cases therefore produce mixed outcomes in the retained uncovered slice.The real-IP case forms the right hypothesis with repository context but never validates Admin-API reachability.
Loading 2608.25730v1…