ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing

Motivation

• The study is motivated by strained peer review, millions of researcher-hours spent reviewing, and evidence that flawed papers can pass human review.These pressures motivate examining whether LLMs can relieve parts of the scientific review process.Source: §1 · p. 1

Pilot study

• A pilot found GPT-4 was the only tested model successful at finding errors, while targeted questions produced more useful responses than asking for a review.The comparison included Bard, Vicuna, Koala, Alpaca, LLaMa, Dolly, OpenAssistant, and StableLM.Source: §1 · p. 1

Identifying errors

• GPT-4 identified deliberately inserted errors in 7 of 13 short papers, spanning mathematical and logical errors.The result was comparable to human peer reviewers in the cited controlled experiments.Source: §1 · p. 2

Verifying checklists

• Across 119 checklist question–paper pairs, GPT-4 achieved 86.6% accuracy using the majority answer over three responses.Half of the LLM’s errors involved questions undeterminable from the paper text, such as information requiring figures.Source: §1 · p. 2

Choosing the better paper

• GPT-4 erred in 6 of 10 abstract pairs when choosing the superior paper, despite deliberately straightforward quality differences.Failures included bias toward positive results, misreading bounds or parameter ranges, prompt injection, bombastic language, and algorithm names.Source: §1 · p. 2

Overall findings

• GPT-4 shows promise for specific parts of review but is not yet capable of functioning as a standalone reviewer.The study evaluates targeted reviewing tasks rather than complete paper or proposal evaluations.Source: §1 · pp. 1–2

Large language models

• LLMs have advanced factuality and reasoning research and expanded into simulation, data generation, computational social science, human-computer interaction, health, biomedical, and legal applications.These works include embodied and human-agent simulation, automated question and label generation, theory-of-mind and moral-judgment studies, believable HCI accounts, and health and legal applications.Source: §2 · pp. 2–3

Automation in peer review

• Peer-review automation covers reviewer expertise and assignment, randomization against collusion rings, subjectivity and calibration, guideline checking, plagiarism detection, conflict checking, and review-related machine learning tools.Comprehensive review-generation attempts had limited success, while related algorithms proved useful for tasks such as paper summarization.Source: §2 · p. 3

Language models for peer review

• Emerging language-model research addresses peer-review agent roles and safe use, paper fluency and review sentiment, and validation of peer-grading scores.The cited approaches use RoBERTa for fluency and sentiment analysis and fine-tuned distilBERT for peer-grading validation.Source: §2 · p. 3

Peer-review datasets

• This paper contributes a small-scale dataset comprising 13 flawed papers, a pilot paper, 119 checklist question–paper pairs, and 10 abstract pairs for evaluating reviewing models.The constructions support error detection, checklist verification, direct comparison, distraction resistance, and few-shot learning or prompting, assuming models were not trained on the paper.Source: §2 · p. 3
• Existing peer-review datasets cover reviewer expertise, collusion strategies, strategic reviewer behavior, review ratings, author feedback, public reviews, review discussions, and argument- or sentiment-based annotations.These datasets originate from conferences, peer-review platforms, mock review exercises, and public review repositories inside and outside computer science.Source: §2 · pp. 3–4

3 Can LLMs detect errors in computer science papers?

• The experiment tests whether LLMs can identify flaws in short computer science papers and uses GPT-4 based on its pilot performance.GPT-4 was the only model in the pilot to detect the error.Source: §3 · p. 4

Error-detection setup

• The researchers constructed 13 short papers, each containing a deliberately inserted mathematical or conceptual error, and asked the LLM to identify incorrect claims.The papers were designed to test whether the model could detect errors ranging from mathematical mistakes to conceptual fallacies.Source: §3.1 · p. 4

Prompting strategies

• They compared three error-checking prompts: Prompt-Direct, Prompt-OneShot with an example, and Prompt-Parts that supplied the paper incrementally.Prompt-Parts presented a few sentences at a time, with each main-claim sentence provided separately.Source: §3.1 · pp. 4–5

Implementation

• GPT-4 was accessed through ChatGPT using the May 3 and May 12 builds for these experiments.The prompts instructed the model to reason step by step when checking claims.Source: §3.1 · p. 5

Evaluation procedure

• The evaluation queried three responses per prompt and marked the overall result positive when any response to any prompt detected an error.This aggregation reflects a practical strategy of flagging a paper if any prompted response identifies an error.Source: §3.2 · p. 5

Error detection

• 7 of 13 short papers had their deliberately inserted errors detected by GPT-4, while the model also occasionally produced false positives.In all six failures, the papers lacked complete proofs, requiring the model to infer flaws using extraneous knowledge.Source: §3.2 · p. 5

Model comparison

• None of the other tested models—Bard, Vicuna, Koala, Alpaca, LLaMa, Dolly, OpenAssistant, and StableLM—identified an error in any paper.Some also raised unwarranted or nonsensical critiques, including demanding experiments in papers presenting theoretical results.Source: §3.2 · p. 5

3.3.1 Bias/fairness

• The LLM correctly identified a logical fallacy in the FairAIrecruiter paper’s inference from equal interview rates to a false claim about applicant-quality distributions.Per-company interview rates were equal across groups, while aggregate counts differed because applicants were distributed differently among companies.Source: §3.3.1 · pp. 7–8

3.3.2 Non-parametric regression

• For the non-parametric regression example, Prompt-Direct missed the invalid implication from mean-square convergence to exact equality, whereas Prompt-Parts correctly detected it.Two of three Prompt-OneShot attempts hinted at the issue; convergence of the integrated squared error to zero does not imply Probability(f_hat_n = f) converges to 1.Source: §3.3.2 · pp. 8–9

3.3.3 Sorting (harder)

• For the harder sorting example, the LLM missed that the restrictions leave too few admissible Fibonacci non-primes for any R greater than 2.The constructed algorithm claims sqrt(R)*log(R) comparisons instead of R*log(R), but the paper’s conditions make lists larger than two impossible; the LLM also raised a false alarm about the Fibonacci condition.Source: §3.3.3 · pp. 9–11

3.3.4 Sorting (easier)

• For the easier sorting example, the LLM accepted a sqrt(R)*log(R) comparison claim under the sole restriction that all values exceed 9, despite this restriction not changing the R*log(R) lower bound.The LLM noted that the result was purely theoretical and lacked a detailed proof and empirical validation, but did not recognize the central invalidity.Source: §3.3.4 · pp. 11–12

3.3.5 Noisy pairwise comparison model (harder)

• The section also introduces a noisy pairwise-comparison model generalizing the Bradley-Terry-Luce model, but supplies no evaluation result for this example.The model is described as making claims about estimation under noisy pairwise comparisons.Source: §3.3.5 · p. 12
• For the noisy-comparison papers, the LLM regarded the statistical setups and risk trends as plausible but could not verify the algorithms or theoretical guarantees from the short papers alone.The harder model assumes f is twice differentiable, 1-Lipschitz, and bijective; both papers report no matching lower bounds and conjecture upper-bound tightness.Source: §3.3.5 · p. 13, §3.3.6 · p. 15
• In the harder noisy-comparison model, the claimed risk bound of 2d/sqrt{n} is invalid because scaling all nonzero parameters leaves comparison probabilities unchanged, making accurate estimation impossible.The LLM missed this identifiability error and could not definitively assess the claim without the full proof; matching lower bounds are also absent.Source: §3.3.5 · p. 13

3.3.6 Noisy pairwise comparison model (easier)

• In the easier noisy-comparison model, the same risk bound of 2d/sqrt{n} remains uncheckable because additive shifts of all parameters leave the comparison distributions unchanged.The LLM did not recognize this issue and reported that verifying the bound, its tightness conjecture, and the implementation would require the full paper, proofs, and source code.Source: §3.3.6 · pp. 14–15

3.3.7 Classification

• The classification paper’s claim that its ensemble remains linear is false, because majority voting applies a nonlinear sign operation after combining linear classifiers.The LLM identified this error in some responses, while the reported 12% test error versus at least 19% for prior linear classifiers still requires validation.Source: §3.3.7 · pp. 16–17

3.3.8 Game theory

• The game-theory paper incorrectly moves from a 2/3 guessing probability to an implicit probability of 1 when claiming the Passenger always wins.The paper models the game as Stackelberg and argues that optimal play guarantees a Passenger win, but the LLM recognized the gap between these arguments.Source: §3.3.8 · p. 18
• 3.3.8 Game theory: The Stackelberg and Monty Hall modeling is valid, but information advantage does not guarantee that the Passenger wins.After switching, the Passenger has a 2/3 chance of guessing the Driver’s move, followed by a Rock Paper Scissors outcome with 1/3 win, 1/3 loss, and 1/3 draw probabilities.Source: §3.3.8 · p. 19

3.3.9 Error correcting codes

• 3.3.9 Error correcting codes: The claimed certain recovery from any arbitrary 2420 symbols is impossible, although the fountain-code design is standard.Independent random linear combinations retain a non-zero failure probability, so complete-certainty recovery cannot be guaranteed; the 13% latency improvement also lacks enough comparison detail.Source: §3.3.9 · pp. 20–21

3.3.10 Optimization

• 3.3.10 Optimization: The reviewer-assignment formulation is valid, but the paper’s claim that its constraints violate total unimodularity is false.The problem has binary assignment variables, load and conflict constraints, and an objective maximizing similarity; the cited literature establishes total unimodularity.Source: §3.3.10 · pp. 21–22

3.3.11 Clustering

• 3.3.11 Clustering: The paper’s claim that one-dimensional K-means always gives the same clusters regardless of initialization is false, though initialization is less problematic in one dimension.K-means can converge to different local optima; the reported clusters nevertheless had near-equal sizes and perfectly agreed with the human evaluator.Source: §3.3.11 · pp. 23–24

3.3.12 Distinguishing styles: 80% accuracy

• 3.3.12 Distinguishing styles: 80% accuracy: 80% accuracy supports detectable separation, but failure to classify above chance would not prove that writing styles are identical.Model or feature limitations, subtle or multidimensional differences, and alternative metrics can all explain classification failure; the LLM detected this fallacy only with sentence-by-sentence prompting.Source: §3.3.12 · p. 25

3.3.13 Distinguishing styles: 50% accuracy

• 3.3.13 Distinguishing styles: 50% accuracy: This paper repeats the 10,000-per-group writing-style experiment but reports 50% test accuracy and concludes there is no style difference.The supplied passage identifies the changed accuracy and conclusion but provides no further evaluation of that inference.Source: §3.3.13 · pp. 25–26
• The study controlled for potential confounders and evaluated a binary classifier trained on one sample split and tested on another.Researchers matched writing topics and checked demographic differences; the classifier used training and validation sets.Source: §3.3.13 · p. 26
• Prompt-Direct and Prompt-OneShot responses also identified the interpretive problem in the 50% accuracy case.The issue may have been easier to detect because the fallacious conclusion appeared at the paper’s end.Source: §3.3.13 · p. 26
• 50% accuracy led the authors to conclude no writing-style difference, but that inference is not necessarily valid.The accuracy could reflect a weak classifier, inadequate features, or other model-development problems rather than identical writing styles.Source: §3.3.13 · pp. 26–27
• Further testing with alternative classifiers, refined features, or other metrics was recommended before concluding that the writing styles do not differ.The observed 50% accuracy may fail to capture genuine stylistic differences.Source: §3.3.13 · p. 27

4.1 Methods

• The evaluation uses 15 accepted or rejected NeurIPS 2022 papers with author-submitted checklists and considers 16 checklist questions with available ground-truth labels.Only questions authors marked “Yes” are evaluated, using papers published after GPT-4’s training-data cutoff.Source: §4.1 · p. 27
• A graduate-student labeler constructs ground truth through keyword searches, full-paper scans, author-answer comparisons, and a second calibration pass.When textual evidence is absent, labels may rely on author knowledge reported in the checklist.Source: §4.1 · p. 27
• GPT-4 receives question-relevant paper sections rather than entire papers, and answers rephrased reviewer-perspective questions with yes, no, or n/a plus brief justifications.The 8k-token context limit prevents supplying complete papers; the prompt also includes the paper title and reviewer role.Source: §4.1 · p. 28

4.2 Summary of results

• 86.6% accuracy: GPT-4 correctly verifies 119 NeurIPS checklist question–paper pairs against hand-labeled ground truth.Each answer uses a majority vote across three GPT-4 responses; Table 2 reports correctness by checklist item and overall.Source: §4.2 · p. 28, Table 2 · p. 29
• GPT-4 and author checklists each match the ground truth 86.6% of the time, but their errors overlap little.GPT-4 disagrees with 12 of 16 mismatched author responses, while 9 of its 16 incorrect answers have correct author responses.Source: §4.2 · p. 28
• Excluding eight questions without textual evidence raises GPT-4’s accuracy to 92.8%, but 86.6% better represents checklist-verification deployment conditions.These cases could require access to figures or other information beyond GPT-4’s current capabilities, whereas authors answered all eight correctly.Source: §4.2 · p. 29

4.3 Paper and response details

• In an example, GPT-4 correctly joins citation-index sequences in all three responses without hallucinating incorrect numbers.The task asks whether authors cite creators of existing assets such as code, data, or models.Source: §4.3 · p. 29

Experimental setup

• BASQ is tested in three proposal-distribution variants—IVR, IGB, and UB—against Monte Carlo samplers, batch WSABI, and three nested-sampling methods, including MLFriends.Synthetic experiments average results over 10 repeats using multicore CPUs without GPUs, with nested-sampling posteriors estimated by weighted-sample KDE.Source: §5 · p. 30

Synthetic likelihoods

• The synthetic evaluation uses the two-dimensional Branin-Hoo, Ackley, and Oscillatory likelihood functions, each designed with multimodal structure.Branin-Hoo has eight modes; Ackley has point-symmetric periodic peaks; the Oscillatory function has reflection-symmetric periodic peaks over highly correlated ellipsoids.Source: §5 · p. 30

Battery simulator

• BASQ is also evaluated for estimating three parameters of the SPMe lithium-ion battery simulator from a time-series voltage-current signal.The parameters are anode and cathode lithium diffusivities and experimental noise variance; each query takes 1.2 seconds on average.Source: §5 · p. 30

Asset attribution

• The experiments cite creators of existing assets, including code, models, datasets, likelihood functions, nested-sampling methods, and the SPMe simulator.The section provides references for the methods and assets used, including [71, 29, 46, 14, 15, 72, 17], [49, 73, 32], and SPMe [61].Source: §5 · p. 30

Conclusion

• The neural harmonizer aligns DNN visual strategies with human vision and can improve performance, addressing the growing trade-off between benchmark performance and human perceptual predictions.The authors connect this finding to concurrent work showing similar trade-offs across tasks.Source: §5 · p. 31

Broader impacts

• The authors argue that encouraging DNNs to use human-like visual strategies could provide a scalable path for correcting dataset biases in artificial intelligence.They identify dataset-bias exploitation as a persistent issue and emphasize perceptual facets that DNNs fail to capture.Source: §5 · p. 31
• The authors do not explicitly discuss potential negative societal impacts, instead focusing on aligning DNNs with human perception and reducing dataset biases.Their broader-impact discussion centers on positive effects rather than negative societal consequences.Source: §5 · p. 31

A.1 Experimental design

• The experiments recruited anonymous Amazon Mechanical Turk users, prioritizing Master-qualified workers or highly experienced users with over 10,000 completed HITs and acceptance rates above 98%.Because participants remained anonymous, the researchers did not collect sensitive personal information.Source: §A.1 · p. 32
• The experimental design is summarized in Figure 1 and uses online participants selected through Amazon Mechanical Turk.The platform-based recruitment procedure emphasized participant quality through AMT qualifications.Source: §A.1 · p. 32
• Before the experiment, participants reviewed and agreed to a consent form describing the study’s objective, procedure, expected 5–8-minute duration, $1.4 reward, risks, benefits, and confidentiality.The study anticipated no risks and offered no direct benefits to participants.Source: §A.1 · p. 32

3-stage screening process

• Participants completed practice, instruction-comprehension, and catch-trial screens before inclusion in further analysis.The catch trial reused a training image whose answer remained visible, so participants were expected to answer correctly.Source: §A.2 · p. 32
• Participants who failed any screening stage were excluded from further analysis.Source: §A.2 · p. 32

5.1 Methods

• The experiment used 13 deliberately constructed abstract pairs, sometimes adding bombastic language to inferior abstracts, and asked GPT-4 to select the paper with stronger results.GPT-4 was accessed through ChatGPT builds from May 3 and May 12, with accuracy measured as the proportion of correctly identified superior abstracts.Source: §5.1 · p. 33

5.2 Summary of results

• At least one incorrect choice occurred in 6 of 10 abstract pairs, as GPT-4 missed key differences or was influenced by extraneous factors.The evaluation used three responses per prompt and marked an overall case incorrect if any response was wrong.Source: §5.2 · p. 33, Table 3 · p. 34

5.3.1 Interpreting a null result

• The LLM preferred Abstract 1, which cautiously interpreted p=0.35 as failing to find evidence for an effect, over Abstract 2’s claim that coffee’s effect was definitely disproved.Abstract 1 acknowledged the need for further research, whereas Abstract 2 overinterpreted the result by claiming coffee had no effect on walking speed.Source: §5.3.1 · p. 35

5.3.2 Positive result bias

• An LLM favored the abstract reporting a statistically significant result at p=0.049999 over the larger study reporting p=0.050001, despite otherwise identical abstracts and marginally different p-values.The larger study had 342 participants versus 212, and the close p-values might not imply a substantial difference in effect size or practical significance.Source: §5.3.2 · pp. 36–37

5.3.3 Parameter ranges

• GPT-4 chose the narrower parameter range d≥2k−1 over the broader d≥2k−2 range and hallucinated differences to justify its decision.The first abstract’s broader range also implied applicability for all n, which the model overlooked when the second abstract explicitly highlighted that property.Source: §5.3.3 · pp. 37–38

5.3.4 Lower bounds

• GPT-4 misjudged a lower bound of twice the evaluator count as stronger than thrice, reversing the mathematically preferable comparison.The abstracts differed only in the lower bound on the same ratio; the supplied analysis states that the larger lower bound is stronger.Source: §5.3.4 · p. 39

5.3.5 Upper bounds

• GPT-4 correctly identified that the smaller upper bound, twice rather than thrice the evaluator count, is stronger.The two abstracts were otherwise virtually identical, differing only in the derived upper bound.Source: §5.3.5 · p. 40
• The LLM accepted Abstract 1 because its upper bound was tighter, indicating a better guarantee of expertise under strategyproofness.A smaller bound represents a less severe compromise on evaluator expertise.Source: §5.3.5 · p. 41

5.3.6 Prompt injection attack

• In one of three responses, the LLM said it was swayed by a prompt injection embedded in Abstract 2.The abstracts were not an obvious choice, and the authors caution that explanations may not faithfully reflect the LLM’s process.Source: §5.3.6 · pp. 41–42

5.3.7 Bombastic language

• In two of three responses, the LLM favored Abstract 1 because of bombastic claims despite Abstract 2 containing an additional Bitcoin experiment.The abstracts were otherwise nearly identical, with Abstract 1 adding promotional language and Abstract 2 adding experimental evidence.Source: §5.3.7 · pp. 42–43

5.3.8 Algorithm name

• The LLM correctly selected Abstract 2 in two of three responses for its additional result, but once hallucinated a computational advantage for Abstract 1.When both abstracts used the same algorithm name, the LLM consistently identified Abstract 2 as superior.Source: §5.3.8 · p. 45

5.3.9 Buzzwords

• Buzzwords did not help Abstract 2, but in two of three responses the LLM failed to recognize Abstract 1’s additional analysis.The added artificial-intelligence, machine-learning, and data-science language was identified as distracting.Source: §5.3.9 · p. 46

5.3.10 Author identities

• The author-identity experiment directly provides author names to test whether identities bias the LLM’s choice between otherwise similar abstracts.Abstract 2 adds a generalization to quantized worker confidence, whereas Abstract 1 reports the same core mechanism and experiments.Source: §5.3.10 · p. 47
• The LLM refused to favor the Nobel laureate’s abstract and based its decision on Abstract 2’s additional result.The abstracts concerned incentive mechanisms for crowdsourcing and differed in author attribution and one added result.Source: §5.3.10 · p. 48
• Both abstracts proposed simple payment mechanisms addressing low-quality crowdsourcing data and validated them with experiments involving over 900 worker-task pairs.The abstracts reported significant error-rate reductions and mechanisms designed to minimize payments to spammers while satisfying a ’no-free-lunch’ requirement.Source: §5.3.10 · p. 48
• Abstract 2 was judged stronger because it retained Abstract 1’s mechanism while extending it to workers’ quantized confidence reports.This broader setting was described as potentially enabling more nuanced confidence assessment and improved data quality.Source: §5.3.10 · p. 48
• The evaluation explicitly treated research quality and contribution—not author prestige—as the appropriate basis for acceptance.Although Vernon L. Smith was identified as a Nobel laureate, Sabiou Inoua’s abstract was considered the stronger contribution in this comparison.Source: §5.3.10 · p. 48

Implications

• LLMs may improve review efficiency by verifying checklists and flagging flaws for human validation, but their mistakes make sole reliance unsuitable.The authors therefore frame current peer review and LLM capabilities as a space for human-AI collaboration.Source: §6 · p. 48

LLM limitations

• Token limits prevented passing complete papers, while missing figure information caused GPT-4 difficulty with figure-related questions.Larger-context GPT-4 32k and multimodal GPT-4 models [Ope23b] may address these limitations.Source: §6 · pp. 48–49

Future model development

• Current open-source models perform worse than proprietary GPT-4, complicating efforts to fine-tune open models for specialized reviewing.The authors identify fine-tuning for specific reviewing purposes as a future direction.Source: §6 · p. 49

Data limitations

• Handcrafted papers were few because experiments required clear ground truth and papers absent from LLM training data, motivating larger peer-review datasets.The NeurIPS 2022 checklist sample also excluded most rejected papers because their authors did not opt in to public release.Source: §6 · p. 49

Future applications

• Future work will test LLMs on CONSORT checklists, reviewer expertise assignment, and reviewer-paper similarity against established models [Coh+20] using a gold-standard dataset [Ste+23].The authors also note that checking whether authors incorrectly select “N/A” instead of “No” remains unevaluated.Source: §6 · p. 49

A Pilot experiments

• The appendix describes the study’s pilot experiments.Source: A Pilot experiments · p. 53

Model selection

• GPT-4 [Ope23b] was the only evaluated LLM to detect the deliberately inserted error in the fictitious paper.The comparison included Bard, Vicuna, Koala, Alpaca, LLaMa, Dolly, OpenAssistant, and StableLM.Source: §A.1 · p. 53

Prompting setup

• The experiments used Prompt-Direct (Section 3.1) and reused the fictitious paper subsequently for Prompt-OneShot.The paper’s footnote identifies the same pilot paper as the later Prompt-OneShot example.Source: §A.1 · p. 53

Model selection

• Several responses incorrectly judged the paper’s claims valid, including its sqrt(m) comparison bound, tight lower bound, and simulation confirmation.These responses accepted the paper overall and endorsed its theoretical and experimental claims.Source: §A.1 · pp. 54–55
• Other responses raised substantive concerns, including that the claimed sqrt(m) bound conflicts with Binary Search’s O(log m) complexity.One response concluded that claims 1 and 2 were highly likely false, while the remaining claims required additional information.Source: §A.1 · p. 57

• Asking the LLM to check a paper’s correctness succeeded where prompting it to write a general review failed to identify errors.The pilot therefore shifted from open-ended reviewing to targeted questioning.Source: §A.2 · pp. 57–58
• The general-review prompt instructed the LLM to act as an expert conference reviewer and write a review of a short paper containing its setting and main claims.Source: §A.2 · p. 58
• The generated review evaluated the paper’s claimed sqrt(m) comparisons and lower bound as potentially impressive, while noting that the algorithm’s technical details were insufficient.It specifically requested information about the AI model, training, and techniques producing the sqrt(m) result.Source: §A.2 · p. 58
• The review also called for a more extensive comparison with existing solutions to substantiate the paper’s novelty and practical importance.Its conclusion characterized the missing methodological details and comparative analysis as major shortcomings.Source: §A.2 · p. 58

B NeurIPS checklist and prompts

• The original first-person checklist wording was rephrased into third-person questions about the authors and their work for individual LLM inputs.For example, “Did you describe the limitations of your work?” became “Do the authors describe the limitations of their work?”Source: B NeurIPS checklist and prompts · p. 58, §B.2 · p. 59

B.1 Checklist item selection

• Two of 18 checklist questions were removed because their answers could not be reliably labeled from submitted paper information.Item 1(a) required substantial domain expertise, while Item 1(d) asked about whether authors had read ethics guidelines, an unverifiable action.Source: §B.1 · p. 59
• Item 1(a) was excluded because evaluating whether abstract and introduction claims were accurate required significant field-specific expertise.Most papers were randomly selected across the conference, limiting the available expertise for ground-truth labeling.Source: §B.1 · p. 59
• Item 1(d) was excluded because authors’ reported information could not verify whether they had read and followed the ethics review guidelines.Consequently, neither manual labelers nor the LLM could retrieve a reliable answer.Source: §B.1 · p. 59

B.2 Full checklist questions

• The remaining 16 NeurIPS 2023 checklist items were paired with third-person prompt questions and listed in Table 4.These questions formed the final component of the user prompt provided to the model.Source: §B.2 · p. 59, Table 4 · p. 60
• Each prompt question combines its checklist item with any prerequisite condition, such as whether theoretical results or experiments were included.Examples address limitations, potential negative societal impacts, assumptions and proofs, reproducibility materials, training details, and error bars.Source: §B.2 · p. 59, Table 4 · p. 60