Source-linked AI summary
Do large language models scrutinise what they review? A multimodal audit of scoring calibration, error detection, and author-identity effects
Emad Alharbi
TL;DR
As LLMs enter peer review, evidence is needed about whether their evaluations are genuinely critical rather than merely helpful or biased. This study tests two multimodal LLMs on version-matched, post-training-cutoff ICLR 2026 manuscripts under controlled identities, modalities, prompts, and verified errors. The models assigned inflated scores and detected few errors, while figures raised scores but reduced error detection and author identity had minimal effect.
Problem
Prior evaluations provide limited evidence about LLM review quality because contamination, manuscript-version mismatch, unverified error benchmarks, and modality blindness can confound assessment.
Method
Two multimodal LLMs reviewed 165 ICLR 2026 manuscripts with controlled author identities, input modalities, manuscript integrity, prompts, and 145 verified injected errors.
Results
LLMs assigned inflated scores, detected few errors, and showed reduced error detection with figures, while author identity had minimal effect on scores and error detection.
Takeaways & Limitations
Until LLMs demonstrate reliable critical scrutiny, they should complement rather than replace human reviewers.
Takeaways & Limitations
The evaluation covered two LLMs and manuscripts from a single venue, research field, and review format.
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly used to generate peer reviews, prompting examination of their capacity for critical evaluation. This study evaluates two multimodal LLMs, Qwen2.5-VL-72B and Pixtral-Large-124B, as reviewers across 165 submissions to the 2026 International Conference on Learning Representations, a venue that postdates both models' training cutoffs. Manuscripts were presented to both models with author identities blinded, replaced with high-prestige affiliations, or replaced with low-prestige affiliations, and in either text-only or text-with-figure format. Additionally, 145 verifiably detectable errors were inserted into 55 manuscripts to assess error identification under natural and verification-oriented prompts. Across all manuscript groups, including rejected submissions, LLM scores ranged from 7.0 to 8.1, whereas human mean scores ranged from 3.4 to 6.8. The models detected 12.1\% of the verified errors under natural prompting, and a one-sentence verification instruction increased detection to 22.2\%; however, 78\% of the errors remained undetected. Providing figures reduced error detection while increasing review scores. No visual error was reliably verified against its corresponding figure, and half of the text-only reviews described figures that were not provided. Author identity did not influence either review scores or error detection. LLM editorial decisions exactly matched those produced by simple score averaging.
1 Background
The paper motivates LLM-assisted peer review by distinguishing helpfulness from review quality and identifying contamination, version mismatch, unverified errors, and modality blindness as evaluation weaknesses.
- Motivation: LLMs may accelerate peer review, but helpfulness to authors does not establish review quality.The motivation is especially salient as ICLR 2026 required approximately 80,000 review reports.
- Methodological weaknesses: Existing LLM-review evaluations can permit retrieval from training data when evaluated papers may have appeared in model training.This study instead uses ICLR 2026 submissions that postdate both models’ training cutoffs.
- Methodological weaknesses: Accepted-paper evaluations may compare LLM reviews with human reviews of different manuscript versions because public PDFs are often camera-ready versions.The prevalence and consequences of this mismatch had not been quantified according to the paper.
- Methodological weaknesses: Error-injection studies often lack verified detectability, preserved contradictory evidence, or answer keys, making detection rates difficult to interpret.The paper constructs a verified 145-error benchmark while retaining and labelling contradictory evidence.
- Study response: The study addresses these weaknesses through version-matched data, verified errors, controlled author identities and modalities, and 9,900 structured reviews.The design spans 165 stratified manuscripts and two frontier open-weight models.
2.1 Study design
The study evaluates multimodal LLM reviewing under controlled changes to manuscript inputs, author identities, error content, prompts, and reviewer roles.
- Study design: Each manuscript was reviewed with text only or text plus figures, under blinded, high-prestige, and low-prestige author identities.The design also varied natural versus error-detection prompts and used two LLMs in reviewer, editor, and judge roles.
2.2 Data collection
The dataset combines ICLR 2026 submissions, official reviews, and decisions from a venue whose submissions were unavailable during both models’ training.
- Data collection: 19,814 ICLR 2026 submissions were collected, including official reviews and decisions; 14,175 had final decisions.The submission deadline postdated both LLMs’ training-data cutoffs.
2.3 Data version and recovery
Because OpenReview hides revision histories, the study recovered reviewed versions of accepted manuscripts from Internet Archive snapshots and recovered 16 manuscripts.
- Data version and recovery: Accepted-manuscript PDFs are camera-ready versions rather than human-reviewed versions, while rejected-manuscript versions are publicly available as reviewed.Internet Archive snapshots from the review period yielded 16 accepted manuscripts.
2.4 Data sampling and anonymization
The study sampled 165 ICLR 2026 manuscripts across clear accepts, borderline papers, and clear rejects, then removed identifying and attribution-related material for anonymized review.
- Data sampling: 165 manuscripts were selected, including 50 clear accepts, 50 borderline manuscripts, and 50 clear rejects.An additional 15 Internet Archive manuscripts expanded the initial stratified sample.
- Anonymization: Author blocks, acknowledgments, and reference lists were removed to prevent identity inference through institutions or self-citations.Page headers were also cropped to remove the publication venue from camera-ready versions.
2.5 Experimental conditions
The experiments varied author identity, figure access, prompting, and injected manuscript errors to test review quality and integrity under controlled comparisons.
- Experimental design: 18 experiments compared otherwise identical manuscript groups that differed only in the factor under investigation.Each applicable experiment used both reviewer models with three replications.
- Author identity: Author identities were removed, replaced with highly ranked institutions, or replaced with fictitious small-institution-like affiliations.These conditions tested blinded review and high- versus low-prestige affiliation effects.
- Error benchmark: Injected errors comprised numeric contradictions, logical or trend contradictions, and figure–text mismatches.Some errors appeared nowhere else in the text and required checking the corresponding figure or table.
- Prompting: Two prompts tested review behavior: a natural prompt and a verification prompt requiring cross-checks against numbers, tables, figures, and references.The verification prompt explicitly required comparing figure claims with images and checking cited content.
- Review pipeline: Reviews included overall scores, located strengths and weaknesses, and a figure-or-table critique, with each review generated three times before averaging scores.An LLM editor adjudicated review pairs, while a separate LLM judge counted an error only when location and nature were correctly identified with quoted evidence.
- Statistical analysis: Binomial generalized estimating equations clustered by 145 errors estimated factor effects, while manuscript-level LLM scores were correlated with mean human scores.The analysis used the verified-error benchmark and compared score patterns across reviewers.
3 Results
The LLMs assigned consistently high scores, detected few injected errors, and showed different responses to figures and verification prompts, while author identity had no measurable effect and editorial decisions matched score averaging.
- Score calibration: LLM scores ranged from 7.0 to 8.1 across manuscript groups, versus human mean scores ranging from 3.4 to 6.8.Under blinded text-only review, both models scored groups narrowly at 7.0–7.2; figures increased scores by 0.5–0.9 for Qwen2.5-VL and 0.1–0.2 for Pixtral-Large.
- Error detection: Error detection was significantly lower with figures, with trend-reversal errors undetected by either model under either input condition.Pixtral-Large most often detected invalid statistical and p-value errors, while Qwen2.5-VL detected almost no numeric contradiction errors.
- Error detection: 12.1% of verified errors were detected under natural prompting, increasing to 22.2% with verification-oriented prompting.The verification prompt increased detection odds by 2.05 (p = 1.3×10^-12), but most errors remained undetected.
- Author identity: Neither model’s review scores or error detection differed across blinded, high-prestige, and low-prestige author identities.Qwen2.5-VL scores were 7.43, 7.44, and 7.38 across these conditions; Pixtral-Large scores were 7.11, 7.13, and 7.08.
- Editorial decisions: The LLM editor agreed with venue decisions for 60.6% of review pairs, exactly matching a naive threshold baseline.The baseline accepted manuscripts with a mean reviewer score of at least 6.
4 Discussion
The two LLMs produced inflated scores, weakly aligned with human judgment, and detected few injected errors. Verification prompting improved detection, but figures reduced scrutiny, while author identity had minimal effect.
- The two LLMs assigned consistently higher scores than human reviewers, with only weak correlations to human judgment.
- Under natural prompting, both LLMs detected only a small fraction of injected errors, while a one-sentence verification instruction approximately doubled detection.
- Providing figures reduced error detection rather than improving it, despite increasing review scores.
- Author identity had minimal effect on review scores, and no affiliation bias was observed in scoring or scrutiny.
- The authors recommend using current LLMs as comment generators under human oversight rather than as scorers, screeners, or figure checkers.Verification-oriented prompting is presented as an inexpensive default for assisted-review pilots, but not a complete solution.
- The evaluation was limited to two model families and manuscripts from one venue, research field, and review format.One model also judged error detection, and one reviewer shared the judge's model family; human validation and symmetric authorship-blind evaluation were used as mitigation.
5 Conclusions
Controlled tests found that multimodal LLM reviewers failed to detect errors, inflated review scores, and sometimes described unavailable figures. The authors therefore position them as complements to human reviewers until critical scrutiny becomes reliable.
- Multimodal LLM reviewers failed to detect errors, assigned inflated review scores, and described figures that were not provided.
- Providing figures slightly increased review scores while reducing error detection, whereas author identity had minimal effect on reviews.
- The authors conclude that LLMs should complement rather than replace human reviewers until they demonstrate reliable critical scrutiny.