CRScore: Grounding Automated Evaluation of Code Review Comments in Code Claims and Smells

• CRScore measures conciseness, comprehensiveness, and relevance using pseudo-references spanning claims, issues, and implications of a code change.Source: §1 · p. 1
• Its neuro-symbolic pseudo-reference generation combines LLMs with code-analysis tools that detect issues such as formatting errors and code smells.Source: §1 · p. 2
• 82.6% of CRScore’s pseudo-references were judged correct, and its human-judgment alignment was greatest among reference-based metrics (Spearman correlation 0.5431).Source: §1 · p. 2
• The authors collect human annotations of pseudo-reference quality and review dimensions, plan to release the scores, and benchmark code-review LLMs on CodeReviewer.Source: §1 · p. 2
• A diff can support multiple valid review topics, so a reference-based metric assigned an alternative valid review a BLEU score of only 0.0458.The example’s human and model-generated reviews focus on different aspects of the change.Source: §1 · p. 1, Figure 1 · p. 2

2.1 Reference Based Evaluation Metrics:

• Prior studies report low BLEU validity and reliability (Evtikhiev et al., 2023), while BERTScore can fail on erroneous candidates resembling references lexically and stylistically.Source: §2.1 · p. 2

2.2 Code Review Evaluation

• Prior code-review datasets from Tufano et al. (2021, 2022) and Li et al. (2022) use reference-based metrics; CRScore instead targets content-focused, reference-free evaluation.Source: §2.2 · pp. 2–3

2.3 Reference Free Evaluation

• Software-documentation and review studies identify dimensions including conciseness, completeness, understandability, relevance, and evidence (Piorkowski et al., 2020; Turzo and Bosu, 2024); CRScore operationalizes three automatically.Source: §2.3 · p. 3

2.4 Code Smell Detection

• Code smells are design flaws and bad practices; detection spans static analysis, learning, and LLM methods, whose coverage and language limits motivate using code-analysis tools.Source: §2.4 · p. 3

3.1 Quality Dimensions

• Relevance balances conciseness and comprehensiveness, analogous to an F-score balancing precision and recall.Conciseness captures how much of a review is on topic; comprehensiveness captures how much information it covers.Source: §3.1 · p. 3

3.2 Pseudo Reference Generation

• CRScore generates pseudo-references from low- and high-level code claims, then combines them with code smells and static-analyzer findings.The LLM and analysis tools complement one another by targeting aspects that either approach might miss.Source: §3.2 · pp. 3–4

3.3 Computing Similarity with Pseudo References:

• A sentence transformer computes pairwise semantic similarity between pseudo-references and review sentences.The paper uses mxbai-embed-large-v15 for STS, selected for its English STS performance among models under 1B parameters as of July 2024.Source: §3.3 · p. 4
• Conciseness is the fraction of review sentences matching any pseudo-reference above threshold τ; comprehensiveness is the fraction of pseudo-references matched by any review sentence.The two measures resemble precision and recall, respectively.Source: §3.3 · p. 4
• Relevance is the harmonic mean of conciseness and comprehensiveness, and the similarity threshold is set to τbest = 0.7314.The threshold is the average similarity between review sentences and their most similar corresponding pseudo-references across 9,869 instances.Source: §3.3 · p. 5

4.1 Rating Quality of Pseudo-References

• For threshold calibration, a threshold derived from ground-truth reviews, τGT = 0.6576, yields comparably good correlation with human annotations.Two trained annotators rated pseudo-references for 300 code changes across Python, Java, and JavaScript, marking claims correct, incorrect, or unverifiable and adding missing claims.Source: §4.1 · p. 5

4.2 Rating Review Quality Dimensions

• Annotators rated conciseness, comprehensiveness, and relevance on a five-point scale using updated pseudo-references; inter-rater alpha values were 0.8868, 0.8505, and 0.8806, respectively.Incorrect and unverifiable references were removed, missing claims added, and code-analysis issues included.Source: §4.2 · p. 6

4.3 Annotating Similarity between Pseudo-References and Reviews

• Human annotations produced 5.7k pseudo-reference–review pairs, used to assess whether STS similarity predicts if a review addresses a pseudo-reference.The study reports precision, recall, and F1 across similarity thresholds.Source: §4.3 · p. 6

4.2 Rating Review Quality Dimensions

• Only CRScore and the LaaJ variants achieve high, statistically significant agreement with human relevance rankings; CRScore is comparable to LaaJ-GPT and better than LaaJ-Magic.LaaJ-GPT uses a more powerful closed-source model, while LaaJ-Magic uses the same base model as CRScore.Source: Table 2 · p. 6

4.4 Review Generation Systems

• The evaluated systems span BM-25 and LSTM baselines, CodeReviewer, open-source LLMs, and closed-source LLMs.The open-source models range from 3B to 13B parameters and are prompted with three examples.Source: §4.4 · pp. 6–7

4.5 Reference-based Metrics

• Reference-based comparisons include BLEU, normalized edit distance, ROUGE-L, chrF, chrF++, BERTScore, and two LLM-as-a-judge variants.The judge variants use GPT-4o and Magicoder-S-DS-6.7B.Source: §4.5 · p. 7

5.1 Validity of Pseudo-References

• 82.6% of CRScore’s pseudo-references are correct across languages, with JavaScript performing best and Java worst.Errors most often involve code comprehension or over-generalization; unverifiable claims are the largest issue observed for JavaScript.Source: §5.1 · p. 7

5.2 Validity of Review Quality Dimension Scores (Con, Comp and Rel)

• CRScore’s Rel achieves the second-best correlation behind LaaJ-GPT and outperforms LaaJ-Magic; its system rankings mostly match human rankings.Only LaaJ metrics and CRScore achieve strong, statistically significant system-ranking correlations; CRScore also shows greater sensitivity to human preferences.Source: Table 3 · p. 7, §5.2 · p. 8

5.3 CodeReviewer Dataset Reference Quality

• Human annotators rate CodeReviewer references only slightly above the nine evaluated systems in relevance, but find them more concise.For conciseness, comprehensiveness, and relevance, respectively, the references average 3.05, 1.88, and 2.13; the nine systems average 2.57, 1.84, and 1.99. GPT-3.5 scores 3.63, 2.65, and 2.9.Source: §5.3 · p. 8

5.4 Failure Cases

• CRScore can underestimate comprehensiveness when its pipeline generates fewer pseudo-references than the dataset average, and low STS precision also contributes to errors.Underestimation cases average 2.44 pseudo-references versus 4.76 across the whole data; brief comments with stopwords and inline code are additional difficult cases.Source: §5.4 · p. 8
• Inline code appears in 45% of overestimation cases, versus 28% of all reviews and 12% of underestimation cases.The comparison identifies an association in the observed cases, not a cause of overestimation.Source: §5.4 · p. 8

6 Discussion

• CRScore addresses one-to-many reviews and noisy references with conciseness, comprehensiveness, and relevance dimensions grounded in LLM- and code-smell-generated pseudo-references.The discussion describes human validation of pseudo-references and annotations for nine review-generation systems and CodeReviewer reference reviews across Python, Java, and JavaScript.Source: §6 · p. 9
• 0.4577 τ and 0.5425 rs are CRScore’s review-quality correlations; its system-ranking correlations are 0.95 rs and 0.8889 τ, the strongest among open-source metrics.Overall, CRScore has the second-greatest alignment with human preferences, behind LaaJ-GPT, and is more efficient than rerunning an LLM judge for each evaluated review model.Source: §6 · p. 9
• CRScore still systematically under- or overestimates review quality in some cases and falls short of powerful closed-source models.The discussion identifies scope to improve pseudo-reference generation and semantic textual similarity matching.Source: §6 · p. 9

7 Conclusion

• CRScore is an automated, efficient, open-source, and reproducible metric for capturing code-review quality dimensions.The paper proposes conciseness, comprehensiveness, and relevance as those dimensions.Source: §7 · p. 9
• CRScore achieves the best alignment with human preferences among open-source metrics and is the most sensitive metric in the reported comparison.The comparison covers seven reference-based metrics and nine review-generation systems using collected human annotations.Source: §7 · p. 9
• The conclusion identifies better pseudo-reference generation and STS matching as routes toward the performance of closed-source models such as GPT-4o.Source: §7 · p. 9

8 Future Work

• CRScore systematically under- and overestimates review quality in some cases, falling short of powerful closed-source models.The authors attribute these limitations to pseudo-reference coverage and STS matching of code-and-text data.Source: §8 · p. 9
• Future work could add pseudo-references for code security and efficiency issues and improve embeddings for matching review sentences to them.Source: §8 · p. 9
• Code-smell detection currently covers Python, Java, and JavaScript; the authors identify C/C++, Ruby, and Go as expansion targets.Source: §8 · p. 9

Limitations

• Human quality raters saw pseudo-references overlapping with CRScore’s, although they could remove or add claims before rating reviews.The final pseudo-reference set differed from CRScore’s automatically generated set and provided annotators a shared basis for ratings.Source: Limitations · pp. 9–10
• STS can misalign relevant pseudo-references and review sentences, especially for reviews with few claims or inline code snippets that inflate similarity scores.These conditions can distort automated review-quality scores.Source: Limitations · p. 10
• Incomplete pseudo-reference generation can underestimate review quality, and the pipeline could add detectors for more smell types and languages.The authors name security and efficiency smells and Go, C/C++, and Ruby as possible extensions.Source: Limitations · p. 10
• The metric’s correlation with human review-quality annotations is moderate, despite outperforming reference-based metrics in alignment and sensitivity.The authors frame CRScore as a first step toward better code-review metrics.Source: Limitations · p. 10

Ethics Statement

• The authors report no ethical-guideline or copyright violations, while acknowledging a slight risk of harmful or toxic text in LLM-generated pseudo-references.They do not consider the risk major based on their pseudo-reference quality annotations.Source: Ethics Statement · p. 10

A.1 Benefits of Neuro Symbolic Pseudo-Reference Generation

• Combining LLMs with code analysis tools can reduce LLM self-selection bias while supplementing the tools’ narrower coverage of issues.The authors report that CRScore, despite using Magicoder as its evaluation LLM, did not rank Magicoder above any model other than LLaMA-3 relative to human rankings.Source: §A.1 · p. 13

A.2 Release of Code and Data

• The authors planned to release the human-preference annotation dataset and evaluation code publicly if the paper was accepted.They said a GitHub link would be added to the camera-ready version.Source: §A.2 · p. 13

B.1 Code Specific Reference Based Metrics

• Code-specific reference-based metrics adapt text metrics with code structure or embeddings, but prior work reports low validity for BLEU and related metrics.CodeBLEU adds dataflow and syntax matching, CrystalBLEU filters shared n-grams, RUBY uses syntax-tree and dependency-graph distance, and CodeBERTScore uses CodeBERT embeddings.Source: §B.1 · pp. 13–14

B.2 Code Review Automation

• Prior code-review work includes datasets and automated generation, refactoring, and issue-detection approaches, while the underlying task remains difficult and resource-intensive.The cited work includes LLM fine-tuning and prompting, review-based refactoring, best-practice detection, and code-smell detection.Source: §B.2 · p. 14

B.3 Comparing with Review Usefulness/Presentation Style Methods

• Earlier review-quality methods emphasize style, presentation, or comment categorization rather than grounding usefulness in claims and issues in the changed code.The passage also notes that conceptual similarity based directly on code lines and comments could reward incorrect references to code snippets.Source: §B.3 · p. 15

B.4 Code Smell Detection

• Because learning-based smell detectors have limits in target coverage, data requirements, evaluation, and cross-language transfer, the authors use traditional language-specific analysis tools.The passage notes that code smells differ across languages.Source: §B.4 · p. 15

C Method Details

• CRScore aligns review sentences with pseudo-references using STS; its comprehensiveness score measures covered pseudo-references, while its conciseness score measures review sentences matched above a threshold.A sample of 100,000 sentence-pair similarities was roughly normal, supporting a 0.85 high-similarity threshold, although the Q-Q plot deviated at very high values.Source: §C.1 · p. 15, Figure 4 · p. 16, Figure 5 · p. 16

C.2 Code Smell Detection Details

• The appendix describes class cohesion and cyclomatic complexity as code-smell dimensions, with Radon flagging Python code blocks at complexity grade C or higher.Cyclomatic complexity is described as the number of linearly independent paths through a program.Source: §C.2.1 · p. 15, §C.2.2 · pp. 16–17, Table 6 · p. 17

C.3 Hyperparameters for Training Pseudo-Reference Generation Pipeline

• The appendix specifies Magicoder training settings and describes BM-25 and LSTM review-generation baselines alongside an LLM judge prompt for relevance.The judge prompt defines relevance in terms of concise and comprehensive reviews.Source: Figure 3 · p. 14, §C.3 · p. 17, §D.3 · pp. 18–19

D.2 Rating Review Quality using Pseudo-References

• For review-quality annotation, raters assess pseudo-reference necessity, link references to reviews, and score conciseness, comprehensiveness, and relevance on a 1–5 Likert scale.The references include claims and implications, plus issues and smells from static analysis; raters also receive source files for context.Source: §D.2 · p. 18, §D.1 · p. 18

D.5 Dataset Statistics

• 300 CodeReviewer test samples were initially selected across Python, Java, and JavaScript, yielding 99, 98, and 96 code changes and roughly 2.6k machine-generated reviews for correlation analyses.The annotations covered 416, 416, and 399 claims, respectively, and about 2.9k reviews per code change; ground-truth reviews were excluded from correlations.Source: §D.5 · p. 20

D.6 Computational Budget

• LLM systems ran on an 80 GB A100, while Magicoder pseudo-reference training used GPT-4 data for 1,000 validation instances and took about 8–10 hours.The paper also reports using around 10,000 GPT-3.5 API calls.Source: §D.6 · p. 20

D.7 Annotator Information

• The two annotators had Java, Python, and JavaScript experience; training and iterative guideline improvements took several weeks.They received research credit rather than monetary compensation and were co-authors.Source: §D.7 · p. 20

D.9 Does Including the Pseudo-References in the Review Quality Dimension Rating Lead to Biased Annotations?

• The authors argue that annotations were not biased toward CRScore because raters edited the pseudo-references and LaaJ-GPT, which did not use them, correlated more highly with human ratings.They say pseudo-references also gave annotators a concrete basis for judging comprehensiveness.Source: §D.9 · pp. 20–21

E.1 Finding Failure Cases of CRScore

• 16 cases (0.61%) underestimated relevance and 98 (3.74%) overestimated it; the authors report that these small subsets nevertheless influence correlation results.Underestimation cases tended to involve fewer claims and brief reviews, while overestimation cases tended to contain more inline code snippets.Source: §E.1 · p. 21, Table 16 · p. 27, Table 17 · p. 27

E.2 Impact of Varying STS Threshold (τ)

• CRScore’s relevance rankings matched human rankings except for LLaMA-3-8B-Instruct, and its relevance metric varied more across human-rated review quality than the other metrics.The threshold analysis describes the method as robust to slight threshold variations.Source: Figure 8 · p. 24, Table 15 · p. 26, Table 18 · p. 28