PaperlayerView exact revision on arXiv

Source-linked AI summary

CRJudgeBench: Can AI Detect Plausible but Invalid Code Reviews?

Yue Pan, Jiawei Li, Ziyuan Zhang, Xiangxin Zhao, He Ye

arXiv:2609.37216v1cs.SEcs.AI

TL;DR

Existing code-review evaluations do not directly test whether an individual comment is technically correct in its repository context. CRJudgeBench and Sentinel address this through a repository-grounded benchmark and an agentic judge that gathers code evidence; Sentinel outperforms evaluated baselines on the test set, though untrustworthy comments remain difficult to detect.

  • Problem

    As coding-agent reviews grow, determining whether an individual generated comment is technically trustworthy remains crucial because plausible comments can still be false alarms.

    Source: §1 · p. 1

  • Method

    The paper introduces CRJudgeBench, a repository-grounded benchmark, and Sentinel, an agentic judge that searches code context and patches to verify review-comment claims.

    Source: §1 · p. 2

  • Results

    76.60% accuracy on the 359-instance test set exceeded GLM-5.3 by 6.13 percentage points and Sentinel’s base model by 19.78 points.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Detecting false-alarm comments remains challenging, while repository-grounded evidence seeking and specialized training substantially improve the agent’s capability.

    Source: §1 · p. 3

    The authors leave further analysis of repository performance variation for future work, noting that repository scale and development history may affect results.

    Source: §4 · p. 9

Abstract

from arXiv · show

Large language models can generate plausible code-review comments, but such comments may contain technically incorrect claims that mislead developers. We study technical trustworthiness judgment: determining whether a review comment's core technical claims are correct and applicable to the reviewed code in its repository context. Existing code-review benchmarks primarily evaluate review generation, issue discovery, or general comment quality, but do not directly assess whether an agent can determine the technical trustworthy of an individual review comment. To fill this gap, we introduce CRJudgeBench, a benchmark of 1199 instances constructed from real pull requests and expert-verified perturbations, covering both trustworthy and plausible but untrustworthy comments. We further present Sentinel, a repository-grounded agentic judge that actively gathers code evidence to verify review comments before making judgments. Starting from Qwen3-Coder-30B-A3B-Instruct, Sentinel is trained on the CRJudgeBench training split through iterative action-level learning from a privileged teacher. On the 359-instance CRJudgeBench test set, Sentinel achieves 76.60\% accuracy, outperforming GLM-5.3 by 6.13 percentage points and its base model by 19.78 points. These results show that even state-of-the-art general-purpose LLMs struggle to identify untrustworthy comments, while iterative action-level learning substantially improves the accuracy of repository-grounded trustworthiness judgments. Our dataset is available at https://huggingface.co/datasets/dcloud347/CRJudgeBenchmark

1 INTRODUCTION

The paper frames technical trustworthiness judgment as a gap in code-review evaluation: fluent, plausible comments can still be technically wrong in repository context. It introduces CRJudgeBench and Sentinel, which gathers repository evidence, and reports improved accuracy over evaluated baselines.

1 INTRODUCTION

  • Coding agents increase development efficiency, but their generated code still needs review, and growing review volume can overwhelm human reviewers.The paper positions scalable review mechanisms as a need alongside stronger coding agents.Source: §1 · p. 1
  • Existing code-review evaluations do not directly establish whether an individual comment’s technical claims are correct in its patch and repository context.Plausible comments may still be false alarms, and correctness can depend on callers, guards, definitions, or tests outside the patch.Source: §1 · pp. 1–2
  • CRJudgeBench contains 1,199 repository-grounded instances, including 764 trustworthy comments and 435 untrustworthy comments from incorrect reviews and expert-verified perturbations.The perturbations make comments factually incorrect or ungrounded in their associated code context.Source: §1 · p. 2
  • Sentinel is a repository-grounded agentic judge that searches relevant code and inspects patches and context before judging a review comment’s technical claims.Its student learns evidence-gathering actions from a privileged teacher’s intermediate supervision, without receiving ground-truth labels as input.Source: §1 · pp. 2–3
  • 76.60% accuracy on the 359-instance test set exceeded GLM-5.3’s 70.47%, while both correctly identified 98.69% of trustworthy comments.Sentinel detected 37.69% of untrustworthy comments, compared with GLM-5.3’s 20.77%.Source: §1 · p. 3

2 CRJUDGEBENCH

CRJudgeBench is built from code-review comments in active, popular Python repositories, with filtering and expert review to retain assessable instances. Expert-verified comment perturbations add untrustworthy cases, and PR-disjoint splits support training and evaluation.

2 CRJUDGEBENCH

  • The authors target popular, actively maintained Python projects, filtering top-starred repositories for more than 1,500 PRs and recent activity.A project needed at least one commit or PR recorded between August 15, 2024 and August 15, 2025.Source: §2 · p. 3
  • Table 1 characterizes task instances through average and maximum values for PR text, patch, review comment, and PR context.Figure 3 presents the instance distribution across repositories.Source: Table 1 · p. 4, Figure 3 · p. 4
  • LLM-assisted screening and expert review narrowed 106,241 PRs to 7,086 whose descriptions had substantive, interpretable technical content.Screening excluded descriptions under 40 words or dependent on unavailable external information; experts manually checked a random sample of 383 PRs.Source: §2 · p. 4
  • Instances combine a target review comment with the PR description, linked issues, patch, review timeline, and repository code at the corresponding version.The benchmark labels each instance as trustworthy or untrustworthy.Source: §2 · p. 4
  • Comment filtering reduced 76,190 initial instances to 48,027, then manual review focused on instances from the ten most-starred repositories.The manually reviewed set contained 764 technically correct and 65 technically incorrect comments before perturbation.Source: §2 · p. 4
  • Three expert-verified perturbations—negation, symbol replacement, and comment relocation—produced 370 negatives, yielding 1,199 instances: 764 trustworthy and 435 untrustworthy.The resulting PR-disjoint split has 714 training, 126 validation, and 359 test instances.Source: §2 · pp. 4–5

3 SENTINEL

Sentinel learns to assess review comments by gathering repository evidence through iterative student rollouts and teacher-guided action supervision. The teacher’s labels guide supervision but are withheld from the student, and supervised actions are checked before being retained.

3 SENTINEL

  • Sentinel uses iterative LEAP-style training: a student explores the repository while a privileged teacher uses gold labels to propose improved reasoning and actions.The student is trained to inspect context, verify claims, and submit a judgment.Source: §3 · p. 5
  • Student trajectories start from the review comment, patch, PR, and a repository reset to the pre-patch commit, with bash tool calls and observations forming each ReAct step.The trajectories are collected for later teacher supervision.Source: §3 · p. 5
  • Teacher feedback is selectively requested at code search, relevant change inspection, and final reasoning steps.These steps are chosen as important to assessment, while selective supervision reduces the cost of feedback at every step.Source: §3 · p. 5
  • The teacher proposes executable next actions using student-visible observations, while keeping the gold label hidden from the student.For an inconclusive search in a pre-PR checkout, the teacher guides inspection of the patch, which reveals a newly added import.Source: §3 · p. 6
  • A separate teacher verification step retains actions judged executable and useful for checking the claim or code location, while screening for label leakage and unsupported observations.Actions that return errors are not automatically removed because they may still provide useful information for subsequent steps.Source: §3 · p. 6
  • Training units pair a student trajectory prefix with a teacher-verified action; the prefix provides context but does not contribute to the completion-only loss.Distillation uses teacher-verified actions rather than teacher token probabilities, without requiring teacher logits or aligned vocabularies.Source: §3 · p. 7
  • The policy is iteratively updated with LoRA adapters while base-model parameters remain frozen, and earlier teacher-verified units are retained alongside newly collected ones.The updated student generates subsequent trajectories for further teacher supervision.Source: §3 · p. 7

4 EXPERIMENT AND RESULTS

Sentinel outperforms the strongest baseline and its untrained base model on CRJudgeBench, but untrustworthy comments remain much harder to detect than trustworthy ones. Results also vary across repositories, and all models show a bias toward predicting comments as trustworthy.

Experimental Setup

  • The evaluation uses mini-swe-agent with a diverse set of state-of-the-art LLMs; Sentinel uses Qwen3-Coder-30B-A3B-Instruct and a privileged GPT-5.6 Sol teacher.Sentinel is trained on the CRJudgeBench training split with LoRA adapters while its base parameters remain frozen, and all models are evaluated on the 359-instance test split.Source: §4 · p. 7

Scoring Protocol and Main Results

  • 76.60% accuracy: Sentinel correctly classifies 275 of 359 instances, versus 70.47% for GLM-5.3 and 56.82% for its untrained base model.The scoring protocol also credits a model that correctly repairs a technical error during its trajectory, even if its final judgment is based on the corrected comment.Source: §4 · p. 7

Untrustworthy Comments Are Harder to Judge

  • 37.69% negative-class recall and 53.85% F1: Sentinel improves detection of untrustworthy comments, but identifying them remains the task’s central challenge.Baselines obtain negative-class recall of only 2.31%–20.77% and F1 scores of 3.73%–33.75%; trustworthy-class recall across models is 87.77%–98.69%.Source: §4 · p. 8

Models Show Strong Bias towards Predicting Comments as Trustworthy

  • Every model predicts comments as trustworthy for at least 85.52% of instances, although trustworthy comments make up 63.79% of the test set.Sentinel has the lowest true-prediction rate, but it still exceeds the ground-truth trustworthy proportion by 21.73 percentage points.Source: §4 · p. 8, Table 3 · p. 8

Detection Variance across Repositories

  • Repository-level accuracy varies substantially; all four models reach 80.00%–82.50% on AutoGPT, while Sentinel and Claude-Opus-5 reach 83.33% and 100.00% on ComfyUI, respectively.Figure 5 reports repository-level accuracy on the test split.Source: §4 · p. 8, Figure 5 · p. 8
  • 59.46%–67.57% accuracy: PostHog, with the most commits and lines of code, yields the lowest repository results.The authors suggest repository scale and development history affect performance, and leave further analysis of repository variation to future work.Source: Table 4 · p. 8, §4 · p. 9

5 RELATED WORK

Prior work studies repository-level software engineering agents and automated code review, but does not directly assess whether an individual review comment’s technical claim is correct in repository context. CRJudgeBench targets that evaluation question, and Sentinel gathers repository evidence for the judgment.

5.1 REPOSITORY-LEVEL SOFTWARE ENGINEERING AGENTS

  • Repository-level agents use tools for code navigation, editing, and testing; this work instead applies an agent framework to verify review-comment trustworthiness and trains evidence-seeking behavior from privileged expert feedback.SWE-agent, OpenHands, and AutoCodeRover address coding tasks, general repository operation, or fault localization and repair.Source: §5.1 · p. 9

5.2 AUTOMATED CODE REVIEW AND EVALUATION

  • Prior automated-review work generates comments, refines patches, or scores comment quality, but does not assess technical trustworthiness using repository context.Earlier studies used similarity-based metrics, while newer reference-free metrics still rely mainly on patches and comments without repository context.Source: §5.2 · p. 9
  • c-CRAB and CR-Bench evaluate end-to-end utility, issue discovery, or spurious findings; CRJudgeBench instead checks an individual comment’s technical claim in its code and repository context.Developer feedback studies also examine false positives, redundancy, and misalignment with developer intent.Source: §5.2 · p. 9

6 CONCLUSION

The paper introduces CRJudgeBench and Sentinel for repository-grounded technical trustworthiness judgments, reporting improved test accuracy over GLM-5.3 and the untrained base model. Its appendices document AI use, ethics, dataset availability, and the agent’s task prompt.

6 CONCLUSION

  • CRJudgeBench contains 1,199 expert-verified instances from real pull requests, and Sentinel achieves 76.60% accuracy on the 359-instance test set.Sentinel exceeds GLM-5.3 by 6.13 percentage points and its base model by 19.78 points; the authors say general-purpose LLMs struggle to detect plausible but incorrect comments.Source: §6 · p. 9

AI USE STATEMENT

  • Generative AI assisted with research and methodology, experimental design, implementation, results interpretation, translation, literature review, manuscript editing, and figure preparation.The authors report manually checking cited literature, reviewing AI-assisted code, verifying results, and retaining final responsibility for the work.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT / REPRODUCIBILITY STATEMENT

  • The benchmark uses public open-source projects under their licenses, excludes personally identifiable information, and is publicly available with fixed train, validation, and test splits.The paper says its benchmark construction procedure and evaluation and training details are described in the main text.Source: ETHICS STATEMENT · p. 10, REPRODUCIBILITY STATEMENT · p. 10

A.1.1 SYSTEM PROMPT (VERBATIM) / A.1.2 USER PROMPT (VERBATIM)

  • The student prompt requires a Boolean trustworthiness prediction based on technical correctness and file location, and directs the agent to inspect the task’s comment.The prompt says missing evidence alone is not proof of an error.Source: §A.1.1 · p. 13, §A.1.2 · p. 13

A.2 TASK INPUTS, REPOSITORY STATE, AND INFORMATION ISOLATION

The task exposes only information needed to assess a target review comment and separates the student from privileged labels and metadata. The repository remains at the PR base commit, with the patch supplied separately for inspection when needed.

A.2.1 STUDENT-VISIBLE INPUTS

  • The student receives whitelisted task information, while gold labels, source metadata, perturbation type, synthetic-instance indicators, and other privileged fields are excluded.The public task also excludes subsequent replies, author identities, and review state, preventing label inference from privileged metadata.Source: §A.2.1 · pp. 13–14, Table 5 · p. 14

A.2.2 EXECUTION ENVIRONMENT AND VERSION SEPARATION

  • Each task runs in an isolated Docker environment with network access disabled; the repository is checked out at the PR base commit and the review patch is separate and unapplied.When patched-version evidence is needed, the student can copy the repository to /tmp and apply the patch there, preserving the original task state.Source: §A.2.2 · p. 14

A.3 MINI-SWE-AGENT CONFIGURATION

This section describes the mini-swe-agent trajectory setup and a teacher that guides repository-grounded verification through constrained actions. A paired example illustrates how base-file context and patch evidence inform verification, with a null prediction requesting further checks rather than a final judgment.

A.3 MINI-SWE-AGENT CONFIGURATION

  • The implementation instantiates mini-swe-agent’s DefaultAgent with the CRJudge prompts, a 30-step limit, a 900-second wall-time limit, and up to three consecutive format errors.The recorded trajectory configuration is presented for reproducibility, not as a separate configuration file that must be loaded.Source: §A.3 · p. 15, Table 6 · p. 15

B.1 TEACHER PROMPT

  • The teacher proposes one next action from a student-visited state, using the private human label to guide checks without inventing evidence or assuming unobserved results.It must finish only with observed evidence, or return needs_review if evidence contradicts the label and no reasonable check resolves the conflict.Source: §B.1 · p. 16
  • The student protocol judges a comment true only when it is technically correct and attached to the correct file, while missing evidence alone does not prove an error.A general PR comment without a file is judged for technical correctness; technical errors are false even if the comment identifies a useful key point.Source: §B.1 · p. 16

B.2 TEACHER INPUT FORMAT

  • The teacher input is a JSON-serialized payload with three top-level fields, including student history and the candidate action, and is passed to the Responses API as an input string.History messages can include roles, content, tool calls, and tool-call IDs; tool calls invoke bash with a string-valued command.Source: §B.2 · pp. 16–17, Table 7 · p. 17

B.3 TEACHER OUTPUT FORMAT

  • The teacher must return a schema-conforming proposal with six required fields, no additional fields, and explicit nulls for unused nullable fields.Native proposals are converted into student bash calls; private review notes, explanations, and evidence are not appended to the student’s submission command.Source: §B.3 · pp. 17–18, Table 8 · p. 17

B.5 OUTPUT EXAMPLE: INSPECTING THE TARGET-FILE PATCH

  • In the unused-import example, the student proposes false after finding no matching import in the base-version file, while the teacher instead requests inspection of the target-file patch.The student’s history shows that the patch was not applied and the base file lacked the import; the teacher notes that this does not establish whether it is unused in the reviewed version.Source: §B.4 · pp. 18, 20, §B.5 · p. 21
  • Message index 3 identifies the task JSON and base-commit declaration, while index 15 is a search result with no match.The search result should not be treated as evidence from the patched version; the focused patch hunk should establish whether the added router namespace is used elsewhere in the reviewed file.Source: §B.5 · pp. 21–22
  • A null prediction requests additional verification rather than submitting a final judgment.Source: §B.5 · p. 22

B.6 TEACHER CONFIGURATION

The appendix specifies teacher settings and a separate reviewer check for supervised actions. It also describes the reviewer’s inputs, decision format, and an approved example.

B.6 TEACHER CONFIGURATION

  • The teacher configuration lists its model and request settings, including high reasoning effort, a 4,096-token output limit, and three follow-up steps.The configuration is identified as the teacher API and orchestration setup.Source: §B.6 · p. 22, Table 10 · p. 22

B.7 REVIEWER VERIFICATION

  • An independent reviewer checks each executed teacher action for suitability as an action-level training target and decides whether to retain the example.Unlike the teacher, the reviewer sees the action’s actual execution result.Source: §B.7 · p. 22

B.7.1 REVIEWER PROMPT

  • The reviewer receives the student’s pre-action history, proposed action, execution result, and private human label, and must assess verification value without relying on label agreement.The reviewer also checks code-version choice, answer-key leakage, factual support, and whether a final judgment has evidence.Source: §B.7.1 · p. 22

B.7.2 INPUT AND OUTPUT FORMATS

  • The reviewer returns strict JSON with a decision and reason; the unused-import example approves inspecting the relevant patch section.The reviewer configuration reuses the teacher’s model and request-level settings; no separate reviewer model configuration is specified.Source: §B.7.2 · p. 23

C.1 TRAINING RECORD FORMAT

The training record pairs a student trajectory prefix with one reviewer-approved teacher action. The prompt supplies context, while the completion supplies the action targeted by the training loss.

C.1 TRAINING RECORD FORMAT

  • Each action-level training record has three fields: prompt, completion, and tools.The prompt contains student instructions, prior actions, and actual tool observations; the completion is one reviewer-approved teacher action, and tools supplies the bash schema.Source: §C.1 · p. 24
  • In the unused-import example, the target action inspects the focused patch because absence from the base file does not establish whether the import is unused in the reviewed version.The abbreviated record retains the completion while replacing lengthy task, patch, and file observations with omission notices.Source: §C.1 · pp. 24–25
  • Only the single teacher action in completion receives loss; the prompt’s student-history tokens and later tool observations are conditioning context.The action is represented as an assistant message with one bash tool call and an optional concise explanation.Source: §C.1 · pp. 24–25

C.2 TRAINING-DATA SCALE

The appendix reports the action-level data used across two training rounds and describes the LoRA optimization settings used in both rounds.

C.2 TRAINING-DATA SCALE

  • The two-round action-level datasets deduplicate exported records by task id, history, and action.Table 13 summarizes the training data used in the rounds.Source: §C.2 · p. 25, Table 13 · p. 25

C.3 OPTIMIZATION AND HARDWARE CONFIGURATION

  • Training updates only LoRA adapters, with rank 16 and alpha 32 selected on validation and held fixed for test evaluation.The base-model weights remain frozen, and the same training configuration is used in both rounds.Source: §C.3 · p. 25, Table 14 · p. 26
Loading 2609.37216v1…