PaperlayerView exact revision on arXiv

Source-linked AI summary

OneCodeReviewBench: Typed Decisions for Code Review and Search

Mohamad Mohebifar, Alex Bit

arXiv:2610.33389v1cs.SE

TL;DR

Bounded code-review and search decisions require explicit contracts, and their reliability across systems is the central question. OneCodeReviewBench pairs such cases with reference-review evidence; the leading system varies by decision type.

  • Problem

    Review and search decisions need file scope, runtime assumptions, and answer criteria because finding an API call alone does not establish a bug.

    Source: §I · p. 1

  • Method

    OneCodeReviewBench pairs bounded review and search cases with explicit contracts, reference-review evidence, and comparisons across model systems and checking procedures.

    Source: §I · p. 1, §VI · p. 5

  • Results

    The leading system varies by decision type; excluding both incomplete-contract families preserves the leaders, with Kev-27B at 76.5% Choice accuracy and Score MAE 0.366.

    Source: A. Ten systems on identical cases · p. 3, D. Mechanical checking and contract… · p. 5

  • Takeaways & Limitations

    The artifact supports paired evaluation of typed review and search decisions, while repository-grounded use and efficiency still require assessment.

    Source: §VI · p. 5

    Synthetic fixtures and related constructions reduce effective diversity; repository-grounded cases are still needed to assess transfer.

    Source: §V · p. 5

Abstract

from arXiv · show

Code review requires finding functional bugs, memory-safety errors, security weaknesses, and performance problems across languages and software stacks. Tools can express bounded parts of this work as decisions about supplied code. We present OneCodeReviewBench, 1,968 synthetic cases across 12 programming languages and YAML, with Boolean, categorical, and ordered answers. Explicit contracts specify runtime assumptions, source scope, and answer criteria; separate evidence records support reference review. One author reviewed 150 cases; two additional engineers reviewed 100 of those cases. We compare ten model--adapter systems on the same cases and contracts: OpenAI Decisions reaches 81.3% Boolean balanced accuracy; Kev-27B reaches 76.8% Choice accuracy and Score mean absolute error of 0.367. Excluding two families with incomplete contracts preserves these leaders. Additional comparisons cover 586 cases with raw symbol search and 40 with a bounded Python import resolver; the resolver matches all 40 references within its supported profile. A versioned artifact reproduces these analyses and retains the 85 cases with matching engineer answers; the reported review subset excludes two incomplete-contract cases. Remaining reviewer disagreements and repaired inputs require resolution; repository-scale transfer and matched latency/cost evaluation remain future work.

I. INTRODUCTION

Code-review and search decisions need explicit scope, runtime assumptions, and answer criteria; benchmark references also require reviewable evidence. OneCodeReviewBench evaluates bounded decision interfaces through shared cases and paired review records.

I. INTRODUCTION

  • Decision model means a bounded-answer interface, not a particular architecture; measuring end-to-end speed and cost requires separate experiments.Such a model could filter candidate files or answer a narrow question directly.Source: §I · p. 1
  • Review questions span correctness, safety, security, concurrency, and performance; finding an API call alone does not establish a bug.Contracts therefore specify file scope, runtime assumptions, and answer criteria.Source: §I · p. 1
  • Label errors can affect benchmark comparisons, and vulnerability datasets have documented labeling and duplication issues.The paper records how references were produced, which cases engineers reviewed, what mechanical checks establish, and which reviews remain unresolved.Source: §I · p. 1
  • Table I frames comparison scope through task units and reference evidence.The contribution is a dataset and paired evaluation protocol for comparing interfaces on shared cases and tracing ambiguous answers to contract rules.Source: Table I · p. 1, §I · p. 1
  • The benchmark pairs bounded review and search questions with supplied files, assumptions, answer rubrics, and reference-review evidence.Its contribution is the dataset and review records, not the established answer types or input/answer-key separation.Source: §I · p. 1

A. Related work and task scope

Prior benchmarks cover distinct units such as changes, program behavior, and retrieval, while OneCodeReviewBench emphasizes review decisions and closed-file inventory contracts. Its case contracts specify evaluator inputs and answer criteria, with references stored separately.

A. Related work and task scope

  • OneCodeReviewBench complements change-level review, program execution, and open-corpus retrieval benchmarks with review-specific decisions and closed-file inventory contracts.Related benchmarks include real-change review, input/output prediction, categorical code questions, execution-trace references, and natural-language retrieval [10].Source: A. Related work and task scope · pp. 1–2

II. CASE FORMAT AND DECISION CONTRACTS

  • Each case separates evaluator-visible files, context, question, and criteria from the stored historical reference and linked review records.This separation supports scoring and reference review without exposing the answer key in a model request.Source: §II · p. 2
  • Boolean, Choice, and Score contracts define complementary truth conditions, mutually exclusive categories, and four ordered levels, respectively.Search contracts specify file universe and counting rules; behavior contracts specify entry point, inputs, runtime, and observation boundary.Source: §II · p. 2

A. Worked source-inventory case

The worked inventory case shows how explicit file and import rules determine a source-count answer without making a behavioral safety judgment. A separate bounded behavior example illustrates why runtime and scheduling conditions matter.

A. Worked source-inventory case

  • The shield inventory counts a call only when its imported identity and source location satisfy the stated production-file scope; its reference and checker both return level 1.A string, same-named local function, or excluded test-file call would not count; the result guides inspection rather than judging cancellation safety.Source: A. Worked source-inventory case · p. 2
  • In the Tokio example, a synchronous mutex held across an await can interact with a second task blocking the single-thread executor before sending a wake-up signal.The answer depends on scheduling and lock semantics; the cases state conditions for assessing potential defects but do not establish whole-repository coverage.Source: A. Worked source-inventory case · p. 2
  • Table II identifies AST counts as shield-checker coverage and notes that the 40-case import comparison overlaps nine cases.Engineer and static coverage can overlap, while 24 adapter controls sit outside the collection.Source: Table II · p. 2

B. Review, abstention, and repair

Reviewers and mechanical checkers provide distinct evidence, while contract repairs require fresh review and inference. The collection combines generated fixtures with a range of task types and remains small-scale relative to repository context.

B. Review, abstention, and repair

  • When a contract is repaired, the historical input is retained, but the new version requires fresh review and inference rather than reuse of old predictions.Mechanical checkers abstain on unsupported semantics, and reviewer uncertainty differs from a missing decisive case rule.Source: B. Review, abstention, and repair · p. 2

III. COLLECTION AND REFERENCE REVIEW

  • The collection began with 500 draft templates across Boolean, Choice, and Score tasks, then expanded behavior, configuration, and search cases across 12 languages and YAML.Forty templates were individually authored, and agent-assisted review addressed answer overlap and scope.Source: §III · p. 2
  • Sixteen synthesis jobs exhausted, leaving 1,968 cases from 161 templates; engineer review and AST checking provide complementary, potentially overlapping coverage.GPT-6-Sol generated fixtures, context, and initial reference answers.Source: §III · p. 2
  • The median fixture has two files and 27 source lines, while repository-scale context retrieval and coordination remain outside the evaluation.Source inventory accounts for 840 cases; the remaining 1,128 cover behavior, configuration, and structural decisions.Source: §III · p. 2

A. Engineer review of 150 cases

The review packet combines engineer answers with source audits, while unresolved responses and a quota-based selection limit what it can establish. Reviewer agreement is evidence for case review, not a population correctness estimate or a final adjudicated key.

A. Engineer review of 150 cases

  • One author answered all 150 cases, and two additional engineers reviewed disjoint 50-case batches.The packet excluded reference answers and system predictions.Source: A. Engineer review of 150 cases · pp. 2–3
  • Of 100 cases with two responses, 85 agreed, seven differed, and eight second reviewers abstained; unresolved items remain unadjudicated.Fifty cases still need a second response, and disagreements, abstentions, and known contract repairs remain in the follow-up queue.Source: A. Engineer review of 150 cases · p. 3
  • The full-packet reviewer matched 149 of 150 stored references, but the quota sample and unresolved cases do not support a population correctness estimate.The sole mismatch concerned a nested-body exclusion missing from the packet; a source audit found no visible answer contradiction in the other 148 cases, without executing all pinned frameworks.Source: A. Engineer review of 150 cases · p. 3
  • The packet balances answer types and labels, supporting case review rather than a pooled estimate of label error across the run.A population estimate would require sampling weights and completed resolution.Source: A. Engineer review of 150 cases · p. 3

IV. MODEL COMPARISON AND CASE-QUALITY CHECKS

The section compares ten model–adapter systems on shared cases and contracts, while distinguishing valid typed outputs from task agreement. Results vary by decision type, and the review subset and adapter comparisons retain important scope limits.

A. Ten systems on identical cases

  • Table III lists system access modes, but its four Kev checkpoints appear as separate result rows and hosted backend architectures are not isolated.Adapters also serialize files, context, and criteria differently, so these are model–adapter comparisons rather than controlled architecture comparisons.Source: Table III · p. 3, A. Ten systems on identical cases · p. 3
  • OpenAI Decisions leads observed Boolean BA, while Kev-27B leads Choice accuracy and Score MAE; the leader depends on decision type.Table IV reports paired results on 728 Boolean, 680 Choice, and 560 Score cases.Source: Table IV · p. 3, A. Ten systems on identical cases · p. 3
  • All systems returned valid predictions for the same 1,968 identifiers, but the comparison measures agreement with stored references using task-specific metrics.Boolean uses a 0.5 threshold and balanced accuracy, Choice exact category matching, and Score MAE on each stated 0–3 rubric.Source: A. Ten systems on identical cases · p. 3, §IV · p. 3
  • Kev-27B exceeds Jev on Choice by 5.15 points (95% interval: 1.91–8.53), while tested Boolean and Score intervals include zero.The exploratory paired template-cluster bootstrap used 4,000 draws; intervals condition on stored references and are unadjusted for multiple comparisons or shared template lineage.Source: A. Ten systems on identical cases · p. 3

IV. MODEL COMPARISON AND CASE-QUALITY CHECKS

  • The runs do not measure a matched latency/cost advantage, and the retained historical references are being resolved separately from this evaluation.The easy-control table separately reports eight cases per type and notes that SemIf was not evaluated on that set.Source: A. Ten systems on identical cases · p. 4, Table V · p. 4

A. Ten systems on identical cases

  • Of 85 cases with matching engineer answers, excluding two incomplete-contract families leaves 83 cases for descriptive comparisons.Kev-27B reaches 89.0% Boolean BA, 81.5% Choice accuracy, and Score MAE 0.228; this agreement-selected subset is not a population sample or final adjudicated key.Source: A. Ten systems on identical cases · p. 4

B. Easy controls and adapter diagnostics

Easy controls diagnose basic typed-decision behavior, while device and option-order checks reveal a sensitivity in the evaluated model–adapter setup.

B. Easy controls and adapter diagnostics

  • Von and Kev-0.8B answered only four of eight Boolean controls correctly; these small controls neither prove an adapter defect nor bound benchmark performance.Laya and Von also had substantial Score error on these easy cases; SemIf lacks this control run because the available rerun environment did not meet its MPS requirement.Source: B. Easy controls and adapter diagnostics · p. 4
  • Shuffling Choice criteria insertion order changed 336–357 of 680 generated Von answers across three seeds (49.4–52.5%), although all eight Choice controls stayed correct.The checks preserved labels and descriptions; the shuffled seeds are not independent benchmark samples.Source: B. Easy controls and adapter diagnostics · p. 4

C. Source-search baselines

The source-search baselines cover different task subsets: raw symbol search handles inventory questions across languages, while a bounded Python resolver handles a narrower set of import-aware cases.

C. Source-search baselines

  • Raw symbol search answers 586 cases across 34 templates and 12 languages plus YAML by counting case-sensitive terminal-identifier occurrences.It includes imports, declarations, comments, and strings, but does not resolve bindings or types.Source: C. Source-search baselines · p. 4
  • Table VI compares systems on each baseline’s answered subset, with Choice accuracy and Score MAE reported for different case populations.These are development subsets, not a shared case population.Source: Table VI · p. 4, C. Source-search baselines · p. 4
  • All 40 cases answered by the bounded Python import resolver match the references within its supported profile.It excludes instance/type and cross-file resolution and abstains on ambiguous bindings, escapes, or mutation.Source: C. Source-search baselines · p. 4

D. Mechanical checking and contract audit

A source audit found omitted contract rules in two template families; excluding those families preserves the leading system results, while repaired inputs still require fresh review and inference.

D. Mechanical checking and contract audit

  • The bounded checker matched references on 15 of 16 asyncio.shield cases and abstained on one unsupported imported-callable case.It used Python 3.14.4 with the 3.12 grammar setting, not execution under the declared CPython 3.12.7 runtime.Source: D. Mechanical checking and contract… · p. 4
  • The audit found omitted rules for nested Python function/class bodies and category priority for co-occurring Go violations, affecting two template families.The omissions establish incomplete contracts, not that all 36 archived repaired inputs had wrong answers.Source: D. Mechanical checking and contract… · p. 4
  • Reviewing individual cases prompted repairs to both template families, and repaired inputs need fresh responses and inference.Historical predictions do not stand in for results on repaired inputs.Source: D. Mechanical checking and contract… · pp. 4–5
  • Excluding both incomplete-contract families leaves 1,932 cases and preserves the leaders, with Boolean results unchanged.Kev-27B reaches 76.5% Choice accuracy and Score MAE 0.366; this sensitivity analysis does not evaluate repaired versions.Source: D. Mechanical checking and contract… · p. 5

E. Reproduction

The release supports offline reproduction, but reviewer coverage, synthetic fixtures, and unresolved contract repairs limit what the current evaluation establishes.

E. Reproduction

  • The public reproduction release includes saved predictions and an offline verifier that reproduces analyses without model calls.Preserved predictions support reproduction despite endpoint changes; tokenizer/truncation traces and some historical raw control runs remain unavailable.Source: E. Reproduction · p. 5

V. SCOPE AND NEXT EVALUATION STEPS

  • The 150-case engineer sample and bounded static procedures provide complementary, potentially overlapping evidence, not a full-run label-error rate.A 65-case follow-up queue remains, alongside plans to expand review and re-review repaired contracts.Source: §V · p. 5
  • Synthetic fixtures and related constructions limit effective diversity, so repository-grounded cases are needed to assess transfer.The data do not establish the direction or size of any provider-overlap effect.Source: §V · p. 5
  • Matched generative/decision-model comparisons must record hardware, batching, latency, and cost at stated correctness levels to test tooling benefit.The present source-tool baselines address inventory tasks; behavioral review needs task-appropriate analyzers or execution checks.Source: §V · p. 5
  • Further studies should freeze checked inputs, group templates by lineage, and record effective model inputs.The release provides aggregate review counts and subset case identifiers, but excludes individual responses, reviewer identities, and private comments.Source: §V · p. 5

VI. ARTIFACT AVAILABILITY AND CONCLUSION

The paper releases a versioned dataset and reproduction artifact, while its conclusion identifies outstanding review, transfer, and efficiency evaluations.

VI. ARTIFACT AVAILABILITY AND CONCLUSION

  • The archived artifact has verified offline analyses and licenses the dataset under CC BY 4.0 and reproduction software under Apache-2.0.A separate label-free packet supports new blinded review; the public archive contains references and predictions and should remain separate from annotation inputs.Source: §VI · p. 5
  • OneCodeReviewBench supports paired evaluation of typed code-review and search decisions across ten systems.The release includes 1,968 cases, engineer review records for 150 cases, source-tool comparisons, and audits that identified missing contract rules.Source: §VI · p. 5

AI ASSISTANCE DISCLOSURE

  • OpenAI Codex assisted with implementation and manuscript work, and Anthropic Claude Fable 5.1 provided pre-submission critique.The human authors retain responsibility for reported results; validation items remain open.Source: AI ASSISTANCE DISCLOSURE · p. 5

Related papers

Loading 2610.33389v1…