Source-linked AI summary
Beyond Lexical Metrics: Sentence-Embedding Detection of Reviewer Habituation in AI Code Review
Haoran Yu, Lifei Liu, Danping Zhang
TL;DR
The paper asks whether repeated exposure to AI-agent code changes reviewer scrutiny and whether review comments reveal that change. It analyzes reviewer behavior and comment language using hand-crafted metrics, sentence embeddings, and temporal tests; approval rises, while embedding structure detects drift that lexical metrics miss.
Problem
It is unclear whether reviewers reduce scrutiny with repeated exposure to AI-generated code and whether inline review language reveals that change.
Source: §1 · p. 2Method
The study analyzes AIDev reviews and inline comments using four hand-crafted linguistic indicators, sentence-embedding analyses, Granger tests, and reviewer-stratified phase classification.
Source: §1 · p. 2, §3.3 · p. 4, §3.4 · p. 5Results
Approval rises by +6.1 pp early-to-late, four hand-crafted metrics show no significant monotonic decline, and embedding geometry yields classifier F1 = 0.74.
Source: §1 · p. 2Takeaways & Limitations
In this corpus, reviewer drift is detectable in embedding structure rather than classical lexical metrics, and approval changes predominantly precede language shifts.
Source: §1 · p. 2, §6 · p. 13Improving AI-agent code over time could contribute to rising approval independently of reviewer change, and the available human-PR control addresses only global leniency.
Source: §5.4 · p. 12
Abstract
from arXiv · showhide
Code review is a key quality checkpoint between AI-generated code and production. As AI coding agents submit pull requests at scale, it is unclear whether reviewers reduce scrutiny with repeated exposure and whether review comments reveal this change. We study 11,429 reviews from 400 repeat reviewers over 207 days, paired with 10,104 human-authored inline comments from AIDev. Approval rates rise from 30.5% in reviewers' early periods to 36.6% in late periods (Wilcoxon p = 8.6 x 10^-8; Cohen's d = 0.25). However, four hand-crafted linguistic features - lexical diversity, Shannon entropy, technical specificity, and constructive actionability - show no monotonic decline across exposure deciles (all Spearman absolute rho <= 0.53, p >= 0.11; Bonferroni-corrected Mann-Whitney p >= 0.36). A logistic-regression classifier based on these features reaches F1 = 0.485, below the majority-class baseline. Sentence-embedding structure does carry signal: reviewers' late-period comment centroids shift farther from their early-period centroids than under within-reviewer random permutations (Wilcoxon p < 0.001), and a small MLP using three embedding statistics reaches F1 = 0.74 under 5-fold reviewer-stratified cross-validation. Granger analysis shows that approval-rate changes predict later shifts in technical specificity at all tested lags (p < 0.001), while the reverse direction is significant at only one of four lags. Reviewer adaptation is therefore detectable in latent distributional structure rather than in classical lexical metrics, and language shifts follow rather than precede changes in approval behavior.
1 Introduction
The paper tests whether repeated exposure to AI-agent code changes reviewer behavior and whether that change appears in review language. Approval rises over time, hand-crafted linguistic metrics remain flat, and embedding structure captures reviewer drift.
1 Introduction
- Approval rates rise by +6.1 pp from early to late reviewer periods.The study reports behavioral drift alongside a +14.9 pp gradient across experience deciles.Source: §1 · p. 2
- Four hand-crafted indicators show no statistically significant monotonic decline across reviewer experience, including in length-controlled subsets.The indicators are lexical diversity, Shannon entropy, technical specificity, and constructive actionability.Source: §1 · p. 2
- Approval-rate changes predict later technical-specificity shifts at all tested lags, while the reverse direction is significant at only one lag.The reported p-values are p < 10^-3 for approval-to-specificity and p = 0.036 at lag 2 in the reverse direction.Source: §1 · p. 2
- Embedding geometry detects per-reviewer drift, and an MLP using three embedding statistics reaches F1 = 0.74 versus 0.59 with canonical linguistic features.Late-period centroids also lie farther from early-period centroids than under a within-reviewer permutation null (paired Wilcoxon p < 10^-3).Source: §1 · p. 2
2 Related Work
Prior work studies useful review language, automation in software workflows, and habituation in human-automation settings. This paper applies those lines of inquiry to autonomous AI code authors and uses change-point and Granger methods as probes.
2.1 Code Review Quality and Language
- Prior code-review research studies comment quality and language, including specificity, actionability, and intent classification.This paper asks whether those linguistic markers change when the code author is an AI agent.Source: §2.1 · p. 3
2.2 Bots and Automation in Software Engineering
- Earlier automation studies examine bots, CI adoption, and LLM-assisted understanding, whereas this work studies autonomous agents and reviewers’ linguistic and representation dynamics over months.The distinction is the AI agent’s autonomous code authorship.Source: §2.2 · p. 3
2.3 Automation Complacency and Habituation
- Prior work documents automation complacency, vigilance decrement, cry-wolf effects, and judgment shifts after extended interaction with opinionated language models.The paper hypothesizes an analogous code-review mechanism and finds a behavioral component, but not the predicted linguistic-degradation component.Source: §2.3 · p. 3
2.4 Change-Point Detection and Granger Causality
- Change-point detection and Granger causality are used as evidence-driven probes: PELT tests for approval-rate regime shifts, while Granger tests which signal leads.The approval-rate signal is described as well fit by a linear trend rather than regime shifts.Source: §2.4 · p. 3
3 Method
The study uses public AIDev reviews and human-authored inline comments to examine behavioral, linguistic, and embedding changes across reviewer experience. It pairs longitudinal analyses with a reviewer-stratified classifier.
3.1 Dataset
- The AIDev corpus covers public GitHub PRs from five AI coding agents between January and July 2025; the analysis includes 400 repeat reviewers, 11,429 reviews, and 10,104 human inline comments.Reviewer identifiers are consistently hashed, and only aggregate patterns are reported.Source: §3.1 · p. 4
3.2 Behavioural Analysis (RQ1)
- Approval-rate shifts are measured by splitting each reviewer’s chronological reviews at the midpoint; decile analysis orders events by within-reviewer experience for 173 reviewers with at least 20 reviews.The decile analysis covers 8,481 reviews and reports an overall +14.9 pp gradient.Source: §3.2 · p. 4, Table 1 · p. 6
3.3 Hand-Crafted Linguistic Features (RQ2)
- Four comment-level indicators measure lexical diversity, Shannon entropy, code-grounded technical specificity, and constructive actionability, with tests repeated on length-controlled subsets.MTLD measures lexical diversity; specificity counts code-related surface features normalized by comment length, while actionability is a binary indicator.Source: §3.3 · pp. 4–5
3.4 Embedding-Based Representation Analysis
- MiniLM sentence embeddings are tested for drift using centroid distance, dispersion, PCA-projected drift, cluster shifts, and paired early-versus-late reviewer centroid distances against permutation nulls.Each of 10,028 non-empty comments is encoded as a 384-dimensional vector.Source: §3.4 · p. 5
3.5 Granger Causality and Cross-Correlation (RQ3)
- First-differenced approval, linguistic, and embedding signals are tested bidirectionally with bivariate Granger tests at lags 1–4, alongside lag ±5 cross-correlations.The review events are grouped into 50 within-reviewer experience percentile bins.Source: §3.5 · p. 5
3.6 Habituation Phase Classifier (RQ4)
- The classifier predicts whether each reviewer’s five-comment window belongs to the late sequence half, using linguistic, embedding, or combined features under reviewer-stratified five-fold GroupKFold.The sequence-position label avoids reusing approval-rate or embedding signals; models include class-balanced logistic regression and a small MLP.Source: §3.6 · p. 5
4 Results
Across this corpus, approval rose with reviewer experience, but the four hand-crafted language metrics remained flat. Embedding-space drift and classification detected reviewer-level change, while temporal tests placed approval-rate changes before technical-specificity shifts.
4.1 RQ1: Behavioural Drift Replicates
- Across 400 repeat reviewers, approval rose from 30.5% early to 36.6% late (∆ = +6.1 pp; p = 8.6 × 10^-8; Cohen’s d = 0.25).The trajectory was gradual rather than an abrupt change-point pattern.Source: §4.1 · p. 6
4.2 RQ2: Hand-Crafted Linguistic Decline Does Not Replicate
- None of the four hand-crafted metrics showed a monotonic decline across experience deciles, and all corrected Mann–Whitney p-values were ≥0.36.The null pattern also held in paired reviewer tests and length-controlled subsets.Source: §4.2 · pp. 6–7, Table 2 · p. 7
4.3 RQ2b: Embedding Geometry Does Carry Drift Signal
- Across 342 reviewers, late-to-early centroid drift averaged 0.444 versus 0.330 under within-reviewer permutations (paired Wilcoxon p < 10^-3), without a population-level monotonic shift.The result indicates idiosyncratic drift that does not aggregate into a monotonic population trend.Source: Figure 3 · p. 8, §4.3 · p. 8
4.4 RQ3: Temporal Ordering Reverses
- Approval-rate changes predicted later technical-specificity shifts at all four lags (p < 0.002 after correction), whereas reverse prediction did not survive correction.The uncorrected reverse-direction result appeared at lag 2 only (p = 0.036).Source: §4.4 · pp. 8–9, Figure 4 · p. 9
4.5 RQ4: Embedding Classifier Outperforms Hand-Crafted
- On reviewer-stratified 5-fold cross-validation, the embedding-statistics MLP reached F1 = 0.74 versus the majority-class baseline’s F1 = 0.71.Embedding statistics measure displacement from each reviewer’s earlier comments, unlike fixed hand-crafted dimensions; the model’s high recall (0.88) comes with specificity of about 0.38.Source: §4.5 · pp. 9–10, Table 3 · p. 10
4.6 Cross-Agent and Calendar Controls
- After calendar-date and PR-size adjustment, experience remained positively associated with approval; agent-PR approval rose while human-PR approval showed no directional change in overlapping months.The calendar coefficient was negligible (−0.007).Source: §4.6 · p. 10, Figure 6 · p. 11
5 Discussion
The discussion finds that hand-crafted linguistic metrics miss reviewer change while embedding geometry detects individual style drift, and that approval changes predominantly precede language shifts. It presents embedding distance as a monitoring signal while noting limits in corpus scope, model choice, ground-truth labels, and causal interpretation.
5.1 What We Got Right and What We Got Wrong
- Embedding geometry detects per-reviewer style drift missed by hand-crafted metrics, while the behavioral-to-language ordering reverses the predicted framing.The authors offer short comments, idiosyncratic review styles, and a small behavioral shift as plausible reasons fixed lexical dimensions fail.Source: §5.1 · p. 11
5.2 Implications for Habituation Monitoring
- Per-reviewer embedding centroid distance is presented as a usable monitoring signal; MTLD or technical-specificity dashboards should not be expected to flag habituation in this corpus.The signal is unsupervised, uses a single 384-d encoder and about 1 GB RAM, and grows beyond permutation expectations in about 60% of individual cases.Source: §5.2 · pp. 11–12
5.3 Implications for Causal Framing
- Greater sensitivity of continuous linguistic signals can make PELT breakpoints appear earlier, although Granger results predominantly favor approval-rate-to-language prediction.The relationship is not perfectly unidirectional: technical specificity shows weak bidirectional signal at one lag.Source: §5.3 · p. 12
5.4 Limitations
- The evidence covers seven months of public-repository activity; enterprise contexts and longer reviewer trajectories may differ, and downstream defect rates were not measured.The behavioral shift is small (d = 0.25), so its practical importance remains dependent on those defect rates.Source: §5.4 · p. 12
- The classifier used off-the-shelf all-MiniLM-L6-v2 without code-aware fine-tuning, so its F1 = 0.74 is framed as a lower bound for representation-based monitoring.Code-domain encoders may carry different signal.Source: §5.4 · p. 12
- Late phase is labeled by sequence position rather than an externally validated habituation event, which may contaminate both halves and reduce achievable F1.The authors suggest self-reported reviewer phase markers for future work.Source: §5.4 · p. 12
- Improving AI-agent code quality over time remains an alternative explanation for rising approvals; post-merge defect rates, the cleanest test, were outside the dataset.The human-PR control addresses only the global-leniency confound.Source: §5.4 · p. 12
6 Conclusion
The paper tests whether reviewer habituation appears in inline comments as well as approval behavior. It finds that embedding geometry detects per-reviewer drift missed by canonical metrics, while the dominant temporal direction runs from approval changes to later language shifts.
6 Conclusion
- Approval behavior shifts, canonical hand-crafted indicators miss linguistic decline, and embedding geometry detects per-reviewer style drift with F1 = 0.74.The dominant temporal pattern runs from approval changes to subsequent language shifts.Source: §6 · p. 13
- The authors conclude that distributional representation, rather than hand-crafted lexical metrics, is the more suitable signal for slow reviewer change in AI-agent contexts.They frame the paper as a corrective and constructive proposal for researchers and tool-builders.Source: §6 · p. 13