PaperlayerView exact revision on arXiv

Source-linked AI summary

CIVER: Causal Intervention Verification for Reliable Code Review

Xinyuan Song, Hasibul Haque, Jay Somasundaram, Ryan Turner, Bowen Zhu, Liang Zhao

arXiv:2610.36010v1cs.SEcs.AI

TL;DR

LLM reviewers can produce unsupported findings, making it difficult to distinguish genuine defects from false positives. CIVER evaluates critiques through targeted repairs and unrelated changes. It reports 77% accuracy on matched pairs, generalizes to unseen repositories, and improves downstream coding-agent repair success.

  • Problem

    LLM-generated reviews can contain false positives or unsupported claims, so genuine defects must be distinguished from findings that should not guide code changes.

    Source: §1 · pp. 1–2

  • Method

    CIVER jointly optimizes review, intervention, and verifier modules to assess critique responses to targeted repairs and unrelated modifications.

    Source: §1 · pp. 2–3

  • Results

    CIVER achieves 77% accuracy distinguishing defective code from repaired counterparts in matched pairs, generalizes to unseen repositories, and improves coding-agent repair success.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Causal verification has practical value for intervention retrieval and downstream code repair.

    Source: §7 · p. 9

    CIVER’s reliability depends on how well interventions isolate the claimed defect.

    Source: §6 · p. 8

Abstract

from arXiv · show

Code review is essential for maintaining software repositories by identifying defects, improving code quality, and supporting reliable software evolution. With the growing adoption of LLM-based coding agents, an increasing number of code contributions and repository changes are generated by these agents. Consequently, LLM-based code review agents are becoming increasingly prevalent to evaluate these contributions and maintain software quality. However, LLM reviewers often generate lengthy reviews containing numerous potential issues, many of which are false positives or lack sufficient evidence. These unreliable reviews can mislead developers and downstream coding agents, motivating the need to distinguish actionable findings from unsupported claims. We propose CIVER, a causal intervention-based framework that identifies the core issues underlying code-review findings and distinguishes genuine defects from false positives. CIVER integrates three modules: a review module assesses findings, an intervention module generates repair-oriented perturbations, and a verifier module scores review reliability. Through joint optimization via expert iteration, the three modules identify genuine defects by responding to targeted repairs while remaining stable under unrelated changes. Extensive experiments on SWE-Review-Bench demonstrate that CIVER correctly identifies 77% of genuine defects through targeted repairs. Cross-project and downstream repair evaluations further demonstrate that CIVER generalizes across repositories and improves coding agents' task success rates, highlighting its effectiveness for reliable automated code review.

1 Introduction

LLM-generated reviews can contain unsupported findings, making it important to verify which critiques identify genuine defects. CIVER uses targeted repairs and unrelated changes to assess critique reliability, with reported gains on benchmark, transfer, and downstream repair evaluations.

1 Introduction

  • False positives and unsupported findings burden review, and when passed to coding agents may trigger unnecessary modifications, new defects, or disruption of correct functionality.Source: §1 · pp. 1–2
  • Prior work explores contextual review, reflection, adversarial verification, feasibility checks, and repair-guided filtering, but does not explicitly establish a critique’s causal connection to its claimed defect.Source: §1 · p. 2
  • CIVER combines review, intervention, and verifier modules, jointly optimized through expert iteration to assess critiques against targeted repairs and unrelated perturbations.Source: §1 · p. 2
  • 77% accuracy on SWE-Review-Bench distinguishes defective code from its repaired counterpart in matched pairs; transfer AUROC is 0.760 on unseen repositories, and accurate critiques improve coding-agent repair success.Source: §1 · p. 2
  • Its causal approach tests whether critiques respond to repairs of their claimed defect while remaining stable under unrelated modifications.Source: §1 · p. 2

2 Related Work

Prior work addresses code-review reliability through varied benchmarks and verification approaches. CIVER distinguishes its approach by learning from critique responses to targeted repairs and unrelated changes.

2 Related Work

  • CodeFuse [15], AACR-Bench [42], SWE-PRBench [20], CodeReviewBench, c-CRAB [45], and SWE-Review [38] evaluate complementary review settings and downstream utility.Source: §2 · p. 3
  • OpenCodeReview [22], Refute-or-Promote [2], Adversarial Review [32], and SGCR [37] use reflection, adversarial verification, or specification grounding; CIVER adds intervention-response learning.Source: §2 · p. 3
  • Fix-guided Verification Filter [17] executes original and repaired implementations, while CIVER learns a critique-conditioned verifier and intervention ranker using targeted repairs and unrelated modifications.Source: §2 · p. 3
  • VISION [12], ComPass [43], and CRScore [26] apply counterfactual, semantic-invariance, or code-grounded techniques, whereas CIVER jointly models review reliability and intervention relevance.Source: §2 · p. 3
  • Table 1 compares methodological components, highlighting CIVER’s explicit distinction between targeted repairs and unrelated modifications.Source: §2 · p. 3

3 Problem Formulation

The formulation treats a critique as valid when its claimed defect exists in the current program state. It tests validity under targeted defect removal and unrelated changes while holding the critique fixed.

3 Problem Formulation

  • The verifier estimates the conditional probability that a candidate critique’s claimed defect exists in a repository state or pull request.Source: §3 · p. 3
  • A targeted intervention removes the defect claimed by the critique, while an unrelated intervention preserves that defect; both modify the program while the critique is held fixed.Source: §3 · p. 3
  • For a valid critique, targeted defect removal should lower validity, whereas an unrelated intervention should leave validity unchanged; the verifier approximates these probabilities with its scores.Source: §3 · p. 3

4 CIVER: Causal Intervention Verification

CIVER verifies review findings by testing whether they respond to repairs of the claimed defect while remaining stable under unrelated or behavior-preserving changes. Its coupled review, intervention, and verifier modules are jointly trained to score valid findings and suppress redundant ones.

4 CIVER: Causal Intervention Verification

  • CIVER jointly optimizes review, intervention, and verifier modules while keeping pretrained models frozen.The objectives target accurate verification, defect sensitivity, robustness to irrelevant changes, consistent predictions, relevant compact repairs, and nonredundant findings.Source: §4 · p. 4, §4.4 · p. 6

4.1 Intervention Module: Program Intervention Construction

  • Targeted repairs remove the claimed defect, unrelated modifications leave it intact, and semantics-preserving transformations change representation without changing behavior.The intervention module ranks candidate edits and selects the one most relevant to the critique; evaluation focuses on identifier renaming and line shifts among transformations.Source: §4.1 · p. 4

4.2 Review Module: Review Generation

  • The review module evaluates critiques using code, text, and paired representations, and predicts validity scores, locations, and supporting code across program states.Its pretrained reviewer and encoders remain frozen; the intervention module uses its outputs to rank modifications, and the verifier aggregates responses into a validity probability.Source: §4.2 · pp. 4–5

4.3 Verifier Module: Review Verification and Selection

  • The verifier aggregates original and intervention-response scores, favoring critiques whose scores fall after targeted repairs but stay stable under unrelated or semantic changes.It selects valid, nonredundant findings by balancing validity scores against pairwise semantic similarity; redundancy can suppress a finding without making it invalid.Source: §4.3 · p. 5

4.4 Training Objectives

  • Training combines validity supervision with objectives for defect removal, unrelated-change stability, semantic consistency, intervention ranking and minimality, and review selection.The ranking objective favors targeted repairs over unrelated modifications, while selection favors high-validity critiques and discourages duplicates.Source: §4.4 · pp. 5–6

5 Experiments

The experiments evaluate review validity and quality, causal intervention behavior, generalization, edit retrieval, and downstream repair. Results cover filtering performance, intervention sensitivity, transfer, and repair outcomes.

5.1 Experimental Setup

  • The evaluation spans six benchmarks, with measures covering validity, review quality, intervention effects, consistency, generalization, retrieval, and downstream repair.Baselines include a frozen reviewer, reflection and verification methods, and text-similarity evaluators; reviewer-based baselines are controlled implementations rather than exact reproductions.Source: §5.1 · pp. 6–7

5.2 Review Evaluation.

  • On AACR-Bench, CIVER reaches AUROC 0.60 and precision 0.77, while reducing false positives per PR from 2.14 to 1.65 versus the frozen reviewer.CIVER also has the lowest review volume, 6.0 comments per PR.Source: §5.2 · p. 7, Table 8 · p. 8
  • On CRScore, CIVER achieves the highest precision, recall, F1, AUROC, and AP among compared evaluators, including AUROC 0.72 and F1 0.43.The always-negative baseline benefits from class imbalance in accuracy, but CIVER performs better on the other reported classification measures.Source: §5.2 · p. 7
  • CIVER achieves the highest AUROC, recall, and F1 on both subsets, reaching AUROC 0.69 on the second; across ten languages, it raises average non-Python AUROC from 0.55 to 0.60.CIVER achieves AUROC 0.75 on PHP.Source: §5.2 · p. 7

5.3 Causal Intervention Evaluation

  • Across four code-review benchmarks, CIVER reaches PairAcc 0.89 on SWE-Review-Bench; under transformations it attains verdict agreement 1.00 and FP reduction 0.16.After excluding potentially misaligned pairs, it reports AUROC 0.68 for validity, recall 0.79 for targeted repair, and PairAcc 0.86 for transformation consistency.Source: §5.2 · p. 7, §5.3 · p. 7, Table 11 · p. 8
  • For targeted repairs, CIVER achieves ranking accuracy 0.88, AUROC 0.67, and defect-removal success 0.86, outperforming the reported ablations on defect removal.These results assess whether CIVER distinguishes original code from modifications intended to address reported issues.Source: §5.3 · p. 7
  • CIVER outperforms retrieval baselines across metrics, reaching Top-1 accuracy 0.43 with five candidates.The retrieval evaluation ranks critique-associated edits among candidate pools containing distractors.Source: §5.3 · p. 7, Table 9 · p. 8

5.4 Downstream Utility

  • Accurate review guidance achieves a 0.93 resolution rate, improving on the no-review baseline by 0.32; false reviews reduce resolution by 0.08.The downstream evaluation uses 121 SWE-Review-Bench issues and a fixed base mini agent.Source: §5.4 · p. 8

6 Discussion

CIVER measures whether critiques respond to targeted repairs, using negative controls and semantics-preserving transformations to assess specificity and invariance. Its reliability depends on interventions isolating the claimed defect.

6 Discussion

  • CIVER’s reliability depends on interventions isolating the claimed defect.Negative controls and semantics-preserving transformations assess causal specificity and invariance.Source: §6 · p. 8
  • Table 12 evaluates code changes on SWE-Review-Bench using rejection, ranking accuracy, flip rate, score reduction, and pair AUROC.The caption defines the metrics but does not provide their values.Source: Table 12 · p. 9
  • Table 13 examines critique-alignment sensitivity through finding validity, targeted-repair response, and transformation consistency after excluding missing critique targets.Source: Table 13 · p. 9

7 Conclusion

CIVER uses causal interventions to verify code-review findings, evaluating critique responses to repairs, unrelated edits, and semantics-preserving transformations. Experiments report improved review reliability and generalization across repositories and programming languages.

7 Conclusion

  • CIVER evaluates critique responses to targeted repairs, unrelated edits, and semantics-preserving transformations rather than relying only on correlation-based assessment.Source: §7 · p. 9
  • Experiments demonstrate improved review reliability, stronger causal sensitivity, and robust generalization across repositories and programming languages.Source: §7 · p. 9
  • The results highlight the practical value of causal verification for intervention retrieval and downstream code repair.Source: §7 · p. 9

Related papers

Loading 2610.36010v1…