PaperlayerView exact revision on arXiv

Source-linked AI summary

When AI Reviews Train AI Reviewers: Scientific-Judgment Collapse and Mitigation

Sy-Tuyen Ho, Minghui Liu, Furong Huang

arXiv:2609.20942v1cs.LG

TL;DR

As model-generated reviews can enter later training corpora, the paper asks whether this recursive feedback narrows scientific judgments. In a controlled one-step study, synthetic exposure compresses ratings and reduces semantic diversity; the authors also introduce TrustReviewer’s curation and steering interventions.

  • Problem

    The paper asks whether reviewers trained on generated reviews express a narrower range of scientific judgments as AI-generated reviews enter later training corpora.

    Source: §1 · p. 1

  • Method

    The authors vary official-versus-generated review mixtures in a controlled one-step training setup and introduce TrustReviewer with curated-corpus training and paired activation steering.

    Source: §1 · p. 2

  • Results

    At 100% versus 0% synthetic exposure, same-paper and corpus-level semantic diversity decline approximately 11% and 5%, respectively, while rating distributions compress.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings characterize a concrete risk of recursive reviewer training and offer mechanisms for more diverse and better-aligned AI-assisted scientific evaluation.

    Source: §5 · p. 11

    The exposure proportion counts reviews explicitly generated by the model, while released official reviews may include unobserved AI assistance.

    Source: §2.1 · pp. 3–4

Abstract

from arXiv · show

Large language models (LLMs) increasingly participate in scientific evaluation, both as automated reviewers and as assistants to human reviewers. As model-generated reviews enter public data and future training corpora, AI peer review can become recursive: later reviewers learn from judgments produced by earlier models. We study one step of this feedback loop in a controlled setting. Starting from Llama 3.1 8B, we first fine-tune a reviewer on official ICLR reviews from 2018--2023 and then train four successor models on ICLR 2024 data with systematically varied mixtures of official and model-generated reviews. Our study shows that introducing synthetic reviews compresses rating distributions and reduces both same-paper and corpus-level semantic diversity. We call this pattern $\textbf{scientific-judgment collapse}$. To mitigate this failure mode, we introduce $\textbf{TrustReviewer}$, an open-source LLM-based system for generating peer reviews of AI and machine learning papers. TrustReviewer intervenes at two complementary stages. For training-time prevention, we train the core reviewer in a single stage on a curated corpus designed to reduce low-quality and semantically degenerate supervision. For test-time correction, paired activation steering aims to further mitigate residual tendencies toward collapsed judgments without further training or additional expert annotation. Together, these results characterize a concrete risk of recursive reviewer training and provide practical interventions for preserving judgment diversity and improving recommendation alignment in AI-assisted scientific evaluation.

1. Introduction

The introduction frames AI-assisted review as a recursive training loop and asks whether generated reviews narrow scientific judgments. It reports collapse under synthetic exposure and presents TrustReviewer’s training- and inference-time mitigations.

1. Introduction

  • LLM-written or edited reviews may enter future training corpora, prompting the question of whether generated supervision narrows reviewers’ scientific judgments.Source: §1 · p. 1
  • The controlled experiment varies official and generated review mixtures while holding initialization, filtering, and optimization settings fixed.The setup isolates one recursive training step rather than measuring naturally occurring AI assistance in peer review.Source: §1 · p. 2
  • At 100% versus 0% synthetic exposure, same-paper and corpus-level semantic diversity decline approximately 11% and 5%, alongside compressed rating distributions.The pattern is homogenization, not a systematic move toward leniency or harshness.Source: §1 · p. 2
  • TrustReviewer pairs single-stage training on a curated corpus with activation steering to mitigate residual collapsed judgments without retraining or added expert annotation.The curation targets low-quality and semantically degenerate supervision; steering contrasts official and model-generated reviews of the same papers.Source: §1 · p. 2

2. Recursive AI Review Training Analysis

The study varies synthetic-review exposure in a controlled recursive training setup and finds that it compresses rating distributions and reduces semantic diversity both within papers and across the corpus.

2.1. Controlled Recursive AI Review Training Framework

  • After training M1 on earlier official ICLR reviews, successor reviewers are trained on 2024 mixtures, varying synthetic exposure while holding initialization and training setup fixed.Each paper contributes three reviews, with zero, one, two, or three synthetic reviews across the exposure settings.Source: §2.1 · pp. 2–4

2.2. LLM Training Setup

  • Shared filtering and fine-tuning settings are applied across models, which are compared using reviews generated for a held-out 2,000-paper evaluation set.The filtering removes malformed, structurally invalid, excessively short or repetitive reviews, and examples exceeding the context limit.Source: §2.2 · p. 4

2.3.1. Recursive Synthetic Exposure Compresses Rating Diversity

  • 1.44 standard deviation and 2.14 entropy at 33% synthetic exposure, with both below the official-only baseline at 66% and 100%.The rating results indicate compression of rating diversity rather than a systematic shift toward leniency or harshness.Source: §2.3.1 · pp. 4–5, Figure 2 · p. 5

2.3.2. Recursive Exposure Reduces Same-Paper Semantic Diversity

  • Approximately 11%: same-paper semantic distance falls from 0.159 at 0% to 0.142 at 100% synthetic exposure.The distance decreases monotonically across the 33% and 66% settings as well.Source: §2.3.2 · p. 5, Figure 3 · p. 6

2.3.3. Collapse of Corpus-Level Semantic Diversity

  • Approximately 5%: corpus semantic spread falls from 0.609 at 0% to 0.579 at 100% synthetic exposure.The spread continues to decrease across the exposure settings.Source: Figure 3 · p. 6, §2.3.3 · pp. 6–7

3. TrustReviewer: Mitigating Scientific-Judgment Collapse

TrustReviewer pairs curated training-time supervision with paired activation steering to address scientific-judgment collapse. On held-out evaluations, it combines stronger recommendation agreement than the evaluated baselines with higher rating and semantic diversity, while steering adds gains without improving every diversity measure.

3. TrustReviewer: Mitigating Scientific-Judgment Collapse

  • TrustReviewer combines single-stage training on a curated official-review corpus with paired activation steering to mitigate residual collapsed judgments.The steering direction contrasts official and model-generated reviews of the same papers; the intervention requires neither further training nor additional expert annotation.Source: §3 · p. 7, §3.2 · pp. 7–8

3.1. Training-Time Prevention via Corpus Curation

  • The curated corpus filters malformed, duplicated, excessively short, repetitive, and follow-up reviews to reduce low-quality and semantically degenerate supervision.The paper calls these reviews “official,” not necessarily human-written, because recent reviews may contain unobserved AI assistance.Source: §3.1 · p. 7

3.2. Test-Time Correction via Paired Activation Steering

  • Steering vectors are estimated from 5,000 paired paper–review examples, keeping paper content fixed when comparing official and generated reviews.The held-out evaluation papers are excluded from calibration; steering strength and layer selection are chosen using 100 separate validation pairs.Source: §3.2 · p. 8

3.3.1. Evaluation setup

  • Evaluation uses held-out papers, three generation runs per model, recommendation agreement metrics, and two measures of semantic diversity.Baselines include the TrustReviewer initialization, OpenReviewer, and Qwen3.6-35B-A3B; exact match and lower MAD indicate closer agreement with official recommendations.Source: §3.3.1 · p. 9

3.3.2. Evaluation results

  • 75.40% exact match and 1.079 MAD make TrustReviewer the strongest overall recommendation-agreement model among the evaluated reviewers.Its exact-match rate exceeds OpenReviewer (73.10%), Qwen3.6-35B-A3B (61.85%), and its Llama initialization (33.93%); its MAD is lower than every baseline.Source: §3.3.2 · p. 9, Table 1 · p. 10
  • TrustReviewer’s rating entropy is 2.18, above Llama (1.53), Qwen (1.94), and OpenReviewer (2.10), and its semantic diversity exceeds those baselines.Figure 4 reports corpus-level semantic spread and mean same-paper pairwise semantic distance; official reviews are shown as a reference.Source: §3.3.2 · p. 9, Figure 4 · p. 10
  • Steering raises exact match from 73.85% to 75.40% and rating entropy from 2.13 to 2.18, while MAD is essentially unchanged.It expands corpus-level semantic spread, but same-paper semantic distance decreases slightly; the authors therefore report gains rather than uniform improvement across diversity metrics.Source: §3.3.2 · p. 9, Figure 4 · p. 10

4. Related Work

Related work covers recursive synthetic-data training, automated scholarly review, and interventions on training data or model representations. This paper distinguishes training on generated reviews from other forms of LLM assistance and studies how such supervision changes a subsequent reviewer.

Recursive training and synthetic-data feedback

  • Recursive-training studies report that outcomes depend on data conditions: generated samples can erase distributional tails, while accumulating real data can avoid collapse in studied settings.Related work also reports distributional shift and n-gram over-concentration as synthetic-data fractions rise, with shift magnitude depending on the underlying training data.Source: §4 · p. 10

LLMs for scholarly peer review

  • Prior scholarly-review work evaluates score prediction, review generation, review–revision questions, and review-process datasets, while this study asks how generated reviews affect the next model’s training.Existing studies describe generated reviews as potentially fluent yet generic, overly positive, weak at identifying substantive weaknesses, or unevenly grounded.Source: §4 · pp. 10–11
  • LLM assistance, autonomous reviewing, and training on resulting review text are distinct interventions; released reviews are therefore called “official,” not verified human-only reviews.Prior work includes corpus-level estimates of substantial LLM modification and a randomized study of guarded LLM feedback’s effects on review specificity and actionability.Source: §4 · p. 11

Data quality and test-time steering

  • TrustReviewer applies data curation and inference-time steering to review generation, adapting activation contrasts between official and model-generated reviews of the same manuscript.Related work frames curation as improving training-corpus quality and diversity, and activation steering as changing model behavior without retraining the full model.Source: §4 · p. 11

5. Conclusion

The conclusion identifies synthetic-review exposure as a source of compressed ratings and reduced semantic diversity, and presents TrustReviewer as a two-stage mitigation system.

5. Conclusion

  • Increasing synthetic-review exposure compresses rating distributions and reduces same-paper and corpus-level semantic diversity, revealing scientific-judgment collapse.Same-paper diversity decreases monotonically, while corpus-level diversity contracts modestly.Source: §5 · p. 11
  • TrustReviewer combines training-time corpus curation with test-time activation steering to mitigate this failure mode.Source: §5 · p. 11
  • TrustReviewer achieves stronger recommendation agreement and greater judgment diversity than evaluated external baselines, with steering adding gains without retraining or expert annotation.Official reviews are a practical reference, not a verified human-only or uniquely correct standard.Source: §5 · p. 11

Appendix

The disclosure distinguishes generative AI used to produce research outputs from its use in editing and presentation, and states that authors verified the revisions.

Appendix

  • Generative AI was used to generate synthetic reviews and model outputs as part of the research methodology.Source: AI Use Disclosure · p. 15
  • Separately, generative AI assisted with language editing, manuscript organization, title development, and presentation of existing results.Source: AI Use Disclosure · p. 15
  • The authors state that generative AI did not derive results or create citations, and that they verified revisions and take responsibility for the text, claims, and artifacts.Source: AI Use Disclosure · p. 15

A. Detailed Data Curation

The curation appendix describes how papers and reviews are prepared and filtered for training, while the semantic-analysis appendix specifies review embedding and distance procedures.

A. Detailed Data Curation

  • The paper collection covers ICLR papers and reviews from 2018 to 2025; PDFs are converted to Markdown, retaining main-paper text and discarding appendices.Source: A. Detailed Data Curation · p. 15
  • The reviewer prompt follows Idahl and Ahmadi (2025) and uses year-specific ICLR forms, so requested fields and rating scales match the original supervision.The minimal user prompt asks for a review according to ICLR guidelines.Source: A. Detailed Data Curation · p. 15
  • Reviews must pass structural and quality checks, including a parseable rating, year-specific structure, length and repetition criteria, before pairs exceeding 64,000 tokens are excluded.The filters remove follow-up markers, reviews under 600 characters, those with at least 8 duplicated lines or 4 repeated 30-grams, and overlength training sequences.Source: A. Detailed Data Curation · pp. 15–16

B. Semantic Embedding Details

  • Semantic analyses use Jina embeddings capped at 8,192 tokens; all 37,681 held-out reviews fit without truncation, and cosine similarity is the dot product of normalized embeddings.Corpus-level analysis averages and renormalizes official review embeddings per paper, whereas same-paper analysis uses pairwise distances between individual reviews.Source: B. Semantic Embedding Details · p. 16

C. Steering Hyperparameter Selection

Steering layer and strength are selected on validation pairs separate from calibration and held-out evaluation data, using exact recommendation match as the criterion.

C. Steering Hyperparameter Selection

D. Review Generation and Recommendation Parsing

Reviews were generated under specified decoding settings and prompts, then parsed with strict rules for valid numerical ratings. The analysis also used a relaxed parser to identify ratings and excluded cases without a numerical recommendation.

D. Review Generation and Recommendation Parsing

  • Three reviews per paper were generated in three independent runs using temperature 0.6, top_p 0.95, repetition penalty 1.0, and a 4,096-token limit.Generation used no fixed decoding seeds; variation therefore reflects independent runs rather than deterministic, seed-labeled replications.Source: D. Review Generation and… · p. 16
  • Prompts combined a system prompt, year-specific ICLR form, and paper text converted to Markdown and truncated after references; local baselines used the same held-out papers and prompt structure unless overridden.The supplied passages do not specify the override settings.Source: D. Review Generation and… · pp. 16–17
  • The strict parser accepted an integer or NA immediately after the structured Rating heading, retried malformed outputs up to 10 times, and wrote reviews only for valid numerical ratings.Downstream relaxed parsing also searched rating blocks for rating-like expressions; missing, unparseable, or NA ratings were excluded from score-based metrics rather than assigned defaults.Source: D. Review Generation and… · p. 17

E. Recommendation Agreement Metrics

Recommendation alignment is evaluated on papers with both a parsed numerical recommendation and at least one valid official rating. The section reports exact matching and distance from the average official rating across three generation runs.

E. Recommendation Agreement Metrics

  • The valid evaluation set includes papers with a parsed numerical recommendation and at least one valid official rating.N is the set size, and m_i is the number of valid official ratings for paper i.Source: E. Recommendation Agreement Metrics · p. 17
  • Recommendation matching measures exact recommendation matching between a generated recommendation and the paper’s official reviewer ratings.The supplied passage introduces the measure but does not provide its displayed formula.Source: E. Recommendation Agreement Metrics · p. 17
  • Mean absolute distance measures disagreement from the average official rating; higher matching and lower MAD indicate closer recommendation alignment, not review-text correctness or completeness.Each metric is computed separately for three generation runs, with the mean and standard deviation reported across runs.Source: E. Recommendation Agreement Metrics · p. 17

G. Limitations

The evaluation is limited in scope, and its measures do not establish review quality or scientific correctness. Figure 5 displays rating distributions across reviewers with official reviews as a reference.

G. Limitations

  • The experiment covers one recursive-training step, one base-model family, and one scientific-review domain; effects may differ across generations, scales, families, fields, conferences, and review formats.These boundaries limit how broadly the observed pattern can be generalized.Source: G. Limitations · p. 17
  • Figure 5’s heatmap reports the percentage of reviews at each overall rating for TrustReviewer with and without steering and baseline models, alongside official reviews as a reference.The caption specifies what to compare but does not state an outcome.Source: Figure 5 · p. 18
  • Greater judgment diversity does not necessarily mean higher review quality, and exact match or MAD measures agreement with official ICLR ratings rather than correctness.Official reviews are an institutional reference, not ground truth, and may contain errors, disagreement, or unobserved AI assistance.Source: G. Limitations · p. 18
  • Embedding-based distances are proxies for substantive diversity and do not directly establish distinct valid concerns, factual critiques, or useful evidence.Human assessment of critique correctness, specificity, and substantive coverage would complement these measures.Source: G. Limitations · p. 18

Related papers

Loading 2609.20942v1…