Usefulness of LLMs as an Author Checklist Assistant for Scientific Papers: NeurIPS'24 Experiment

• The NeurIPS 2024 experiment evaluated an LLM assistant that scrutinized authors’ checklist responses and proposed enhancements for meeting conference requirements.The assistant was available only to authors, so its outputs were not used by reviewers, meta reviewers, or program chairs.Source: §1 · pp. 1–2
• Over 70% of surveyed authors found the assistant useful, and over 70% said they would modify their papers or checklist responses based on its feedback.Authors generally reported a positive experience with the LLM-assisted checklist review.Source: §1 · p. 2
• Authors most often reported inaccuracy (20/52) and excessive strictness (14/52), while AI-assisted rewriting could make the assistant significantly more lenient.These findings expose reliability and gaming vulnerabilities in automated checklist review.Source: §1 · p. 2
• Qualitative evidence indicated that checklist review meaningfully helped some authors improve their submissions, although changes could not be causally attributed to the assistant.The study assessed this through author feedback and analysis of resubmissions.Source: §1 · p. 2
• 35 survey respondents described specific submission modifications they would make in response to the Checklist Assistant.Source: §1 · p. 2
• Among 40 authors who submitted papers twice to the verifier, checklist justifications generally became significantly longer between submissions, suggesting additions made in response to LLM feedback.These pairs accounted for 80 total paper submissions.Source: §1 · p. 2

2 Background

• The background section frames the checklist as the basis for evaluating LLM assistance with submission-standard compliance.Source: §2 · p. 3, §2.1 · p. 3

2.1 The NeurIPS 2024 Author Checklist

• The NeurIPS 2024 Author Checklist requires authors to answer standardized questions about their papers.Authors provide answers such as “Yes,” “No,” or “NA” with justifications.Source: §2.1 · p. 3
• The checklist is designed to help authors assess whether their work meets NeurIPS research and manuscript preparation standards.Source: §2.1 · p. 3

2. Limitations: Does the paper discuss the limitations of the work performed by the authors?

• The checklist asks whether papers disclose enough training, testing, and experimental-setting detail to understand and reproduce their results.Source: §2 · p. 3
• It asks authors to provide open-access data and code with sufficient instructions for faithful reproduction when applicable.Source: §2 · p. 3
• It requires reporting suitable error bars or other information supporting the statistical significance of experiments.Source: §2 · p. 3
• The checklist also covers conformity with the NeurIPS Code of Ethics and discussion of positive and negative societal impacts.Source: §2 · p. 3

11. Safeguards:

• The safeguards checklist asks authors to document responsible release practices, asset licensing, new-asset documentation, human-subject procedures, and IRB approvals.Source: §11 · pp. 3–4

2.2 Related work

• Prior work has used LLMs to generate full peer reviews, but evaluation ratings do not clearly establish whether those reviews identify errors, select better papers, or provide useful feedback.Source: §2.2 · p. 4
• This study instead evaluates LLMs on the narrower task of validating author-checklist compliance during an actual peer-review conference.Source: §2.2 · p. 4
• Earlier studies found mixed performance on identifying deliberately inserted scientific errors and chance-level performance when comparing objectively stronger abstracts.Source: §2.2 · p. 4
• The paper notes that checklist-compliance studies were retrospective, whereas this system was deployed live to help authors improve compliance before submission.Source: §2.2 · p. 4

3 Design and deployment

• The assistant evaluates each checklist answer and justification against the complete paper, appendices, and relevant guidelines, returning targeted suggestions and a score.Scores are 1 for no issues, 0.5 for minor improvements, and 0 for critical issues.Source: §3 · p. 5, §3.2 · p. 6
• Authors received color-coded feedback, with green indicating no significant concerns and orange indicating needs for improvement.They were encouraged to validate orange feedback and revise their submissions where necessary.Source: §3 · p. 5

3.1 Deployment

• The voluntary Codabench deployment processed uploaded papers by parsing the PDF, sending each answered checklist question to an LLM, and compiling the responses into a readable HTML report.Source: §3.1 · p. 6
• The deployment encountered PDF and checklist parsing problems involving subsections, spacing, incomplete checklists, and merged special-character sequences.The study states that code improvements addressed these issues.Source: §3.1 · p. 6

3.2 Prompt engineering

• Prompt development favored clear, step-by-step instructions and itemized actionable feedback, while combining scores of 0 and 0.5 into a single improvement-needed signal because of concerns about excessive harshness.Source: §3.2 · pp. 6–7
• Repeated-run testing found significantly lower within-paper score variation than across-paper variation for all questions except ethics, Q9.The reported comparison was significant at p < 0.05 for all but Q9, whose within- and across-paper variances were comparable.Source: §3.2 · p. 7

3.3 Anonymity, confidentiality, and consent

• Authors could participate anonymously, while papers and outputs were kept confidential from NeurIPS reviewers and organizers and participation received IRB approval with written consent.Source: §3.3 · p. 7

4.1.1 Overview of Checklist Usage and Responses

• Across 234 submissions, most checklist questions received Yes responses, while several topics drew NA or No answers and the assistant flagged 8–13 of 15 questions for improvement per submission.NA responses were common for Theory, Impacts, Safeguards, Documentation, Human Subjects, and Risks; No responses were notable for Code and Data and Error Bars.Source: §4.1.1 · pp. 7–8, Figure 3 · p. 8

4.1.3 Survey Responses

• 70% of surveyed authors found the assistant useful, and 70% planned to modify their paper or checklist responses based on its feedback.Positive responses were statistically significant, although enthusiasm and perceived usefulness declined from pre-usage levels.Source: §4.1.3 · p. 9

4.2.1 Characterization of LLM Feedback by Question

• The LLM tended to provide 4–6 distinct feedback points per question, including concrete comments on claims, compute resources, optimizer configuration, implementation-code availability, and experimental details.It also tended to provide generic boilerplate for each question.Source: §4.2.1 · p. 13

4.2.2 Authors’ Descriptions of Submission Changes

• 35 survey respondents described submission changes prompted by feedback, most often adding detail or references to checklist justifications and expanding experiment, dataset, or compute descriptions.Two authors reported correcting checklist answers, while other changes involved introductions, abstracts, limitations, or standard-error discussion.Source: §4.2.2 · p. 13

4.2.3 Analysis of Re-submissions

• Among 40 twice-submitted paper pairs, authors changed checklist answers in 22 cases and justifications in 39; 35 of those 39 changed more than six justifications.When neither answer nor justification changed, LLM evaluations remained unchanged 81%, improved 7%, and worsened 12%; after answer changes, they improved 15% and worsened 2%.Source: §4.2.3 · p. 14

5.1 Gaming the review system

• 14 of 15 checklist questions showed score increases after three adversarial rounds that revised justifications without changing paper content.The attack agent used iterative feedback to select higher-scored responses for 234 papers.Source: §5.1 · p. 15, Figure 12 · p. 16
• The attack LLM added unsupported hardware descriptions and placeholder repository URLs, yet the Checklist Assistant assigned these justifications a score of ‘1’.Source: §5.1 · p. 15
• Automated verification could therefore reward persuasive but fabricated rebuttals that create no substantive changes for readers.Source: §5.1 · pp. 15–16

5.2 Other Limitations

• LLMs may misinterpret complex content, produce false positives or negatives, and miss information in figures, tables, or external links.Authors most frequently cited inaccuracy as an issue, although most surveyed authors still found the tool useful.Source: §5.2 · p. 16
• Processing papers through a third-party API raises privacy and security concerns, while model biases could compromise fairness if outputs were exposed to reviewers.The experiment limited assistant access to authors rather than reviewers, meta reviewers, or program chairs.Source: §5.2 · p. 16

6 Conclusion and future work

• The NeurIPS 2024 deployment demonstrated potential for LLMs to help authors validate whether papers meet submission standards.Source: §6 · p. 17
• Over 70% of authors found the assistant useful and were willing to revise papers based on its feedback, despite lower enthusiasm after use than before.Source: §6 · p. 17
• Inaccuracy and excessive strictness were cited by 20 and 14 of 52 respondents, respectively, limiting reliable interpretation of nuanced scientific content.Source: §6 · p. 17
• Authors who used the assistant made more substantive revisions, but the study could not establish that the assistant caused improvements in submission quality.Source: §6 · p. 17
• Future work should improve scientific-text interpretation, multimodal processing, bias mitigation, expectation calibration, platform integration, and randomized evaluation.Source: §6 · p. 17
• Extending LLM assistance to other review stages requires safeguards against score gaming and continued human oversight.Source: §6 · p. 17

1. Claims

• The abstract and introduction should state the paper’s main claims, contributions, important assumptions, and limitations.Source: §1 · p. 20
• Claims should match theoretical and experimental results and reflect how far those results can be expected to generalize.Source: §1 · p. 20
• Aspirational goals may motivate the work when clearly identified as goals the paper does not attain.Source: §1 · p. 20

2. Limitations

• Authors should discuss limitations, strong assumptions, robustness, claim scope, performance factors, computational scaling, privacy, and fairness.NeurIPS encourages a separate Limitations section and does not penalize honest disclosure of limitations.Source: §2 · pp. 20–21

3. Theory, Assumptions and Proofs

• Theoretical results require complete assumptions and correct proofs, with theorems, formulas, lemmas, and references clearly numbered and cross-referenced.Proofs may appear in the paper or supplement, but informal core proofs should be complemented by formal proofs.Source: §3 · p. 21

4. Experimental Result Reproducibility

• Experimental papers must disclose information needed to reproduce their main results, including suitable access or instructions for models, data, and procedures.The required reproducibility path depends on whether the contribution is an algorithm, architecture, dataset, or model.Source: §4 · pp. 21–22

5. Open access to data and code

• Open-access materials should include sufficient instructions, exact commands and environments, data-preparation details, and scripts for proposed methods and baselines when applicable.Code release is encouraged but not always required; a reasonable reproducibility avenue is required.Source: §5 · p. 22

6. Experimental Setting/Details

• Authors should specify training and test details such as data splits, hyperparameters, selection procedures, and optimizers at a level needed to understand the results.Full details may be supplied through code, appendices, or supplemental material.Source: §6 · pp. 22–23

7. Experiment Statistical Significance

• Error bars should identify captured variability, calculation method, assumptions, and whether they represent standard deviation or standard error.Source: §7 · p. 23
• Asymmetric distributions require error bars that remain within valid ranges, while 1-sigma reporting should be explicitly labeled.Source: §7 · p. 23

8. Experiments Compute Resources

• Reproducibility reporting should specify worker type, memory, storage, per-run compute, total compute, and compute used by omitted preliminary experiments.Source: §8 · p. 23

9. Code Of Ethics

• Ethics guidance asks authors to discuss harms from intended use, incorrect results, and misuse, plus mitigation strategies where negative societal impacts exist.Source: §9 · p. 24

11. Safeguards

• Safeguards guidance covers asset versions, licenses, provenance, consent, anonymity, human-subject details, IRB review, and controlled release for high-risk models.Source: §11 · pp. 24–25

B Prompt

• The prompt compares each checklist answer and justification with the paper and guidelines, then requests terse actionable feedback followed by a separate score.Source: B Prompt · p. 26

C Clustering of LLM Feedback

• Feedback clustering emphasized contribution clarity, documentation consistency, theory–empirical links, real-world applicability, and generalizability.Source: Clarification and Emphasis on Novel… · p. 27, Consistency and Precision in… · p. 27, Clarify and Link Theoretical and… · p. 27

Transparency on Limitations and Future Work

• Transparency feedback combines limitations, ethical considerations, future work, assumptions, computational efficiency, and scalability to strengthen paper integrity and robustness.Source: Transparency on Limitations and Future… · p. 28

Computational Efficiency and Scalability

• Computational feedback addresses efficiency, scalability, environmental limitations, privacy, fairness, societal impacts, and dataset quality or relevance.Source: Computational Efficiency and Scalability · p. 28

Theoretical and Empirical Clarification

• Theoretical and empirical clarification calls for detailed explanations and clearer distinctions between theoretical foundations, proofs, and empirical evidence.Source: Theoretical and Empirical Clarification · p. 28

Clarity and Accessibility for Comprehension

• Accessibility feedback recommends clear explanations, consistent definitions, accurate visual cross-references, and accessible appendices.Source: Clarity and Accessibility for… · pp. 28–29

Transparency on Limitations and Assumptions

• Authors are encouraged to state limitations and assumptions explicitly, including proof completeness, data availability, ethics, and future implications.Source: Transparency on Limitations and… · p. 29

Reproducibility and Methodology Details

• Reproducibility guidance calls for clear experimental frameworks, detailed methodologies, and reporting of dataset licenses, code availability, training details, dependencies, and environments.Source: Reproducibility and Methodology Details · p. 29

Data Accessibility and Management

• Feedback prioritizes data and code availability, licensing, supplemental materials, and documentation to support open science and deeper understanding.It also highlights dataset specifics, model details, limitations, and supplementary-material linkage.Source: Data Accessibility and Management · p. 29, Supplemental Materials and Detailed… · p. 29

Reproducibility and Methodological Detail

• Reproducibility requires exhaustive methodological information covering experimental settings, datasets, code, implementation environments, assumptions, and limitations.Feedback additionally requests anonymized code release and adherence to submission guidelines.Source: Reproducibility and Methodological… · p. 29, Statistical Enhancements and Rigor · p. 30

Open Science and Accessibility

• Open-science feedback stresses licensing, accessibility, documentation quality, novelty and limitations, data integrity, and statistical transparency.Requested statistical improvements include explaining methods, confidence intervals, asymmetric distributions, and computational limitations.Source: Open Science and Accessibility · p. 30, Data Quality and Integrity · p. 30, Statistical Enhancements and Rigor · p. 30

Safeguards

• Safeguards include anonymizing submissions and assets, protecting privacy, documenting data and model availability, and providing responsible-use, oversight, and community-engagement guidance.The feedback also encourages stakeholder involvement, community standards, and interdisciplinary collaboration.Source: Open Science and Accessibility · p. 30, Safeguards · p. 33

Comprehensive Computational Resource Disclosure and Reproducibility

• Authors are asked to disclose computational resources, experiment-specific compute, execution time, preprocessing costs, and computational limitations so readers can assess reproducibility and demands.Technical descriptions should also clarify implementations, failed or preliminary experiments, comparisons, and baselines.Source: Comprehensive Computational Resource… · p. 31, Execution Time and Resource… · p. 31, Enhanced Technical Descriptions and… · p. 31

Comprehensive Ethical Considerations

• Ethical guidance covers privacy, security, anonymization, bias, fairness, inclusivity, legal compliance, misuse, societal and environmental impacts, and responsible research.It also recommends broader-impact statements, future ethical directions, and adherence to methodological, ethical, regulatory, and accessibility standards.Source: Comprehensive Ethical Considerations · p. 31, Future and Societal Implications · p. 31, Impacts · p. 32

Future and Societal Implications

• Across the section, documentation, versioning, artifact and asset verification, accessibility, and community interaction are presented as mechanisms for transparency, reproducibility, credibility, and research uptake.These practices include version and URL details, supplementary-material verification, asset validation, methodological references, and providing code for community feedback.Source: Future and Societal Implications · p. 32, Reproducibility and Open Science · p. 34, Comprehensive Documentation and… · p. 34

Clarity and Highlighting of Novel Contributions

• Feedback urges clearer articulation of novel contributions, scholarly distinctiveness, scope, implications, limitations, future directions, and potential positive or negative impacts.The abstract and introduction should explicitly highlight unique contributions and connect societal-impact claims to technical justification and mitigation strategies.Source: Clarity and Highlighting of Novel… · p. 32, Distinctiveness and Scope Reflection · p. 32, Impacts · p. 32

Dataset and Asset Documentation

• Authors should document datasets, models, assets, versioning, citations, updates, licensing, supplementary materials, and anonymized asset URLs to improve discoverability and reuse.Recommended documentation also includes replicability and benchmarking information, dataset licensing, and licensing information inclusion.Source: Dataset and Asset Documentation · p. 35, Community Engagement and Interaction · p. 35

Human Subjects

• Research reports should explain ethical, societal, environmental, human-subject, crowdsourcing, data-ethics, accessibility, compliance, limitation, and assumption considerations.The guidance also calls for participant instructions, compensation information, experimental-method clarification, and explicit ethical discussion.Source: Human Subjects · p. 36

Methodological and Documentation Transparency

• Methodological transparency should cover implementation details, baselines, architectures, hyperparameters, datasets, standards compliance, accessible documentation, and visual references for key figures and tables.The section also highlights ethics and data-source information, licensing and availability, broader-impact statements, and novelty clarification.Source: Methodological and Documentation… · p. 36

Enhancing Real-world Relevance and Theory-Practice Connection

• Researchers should connect theory to practical applications through detailed case studies, real-world examples, anticipated outcomes, and expanded broader-impact discussion.The guidance specifically emphasizes societal impact and the method’s contributions to real-world scenarios.Source: Enhancing Real-world Relevance and… · p. 36

Comprehensive Ethical and Societal Considerations

• Ethical and societal reporting should address technology’s ethical implications, non-human-subject research, guideline adherence, compliance, and broader impact.These considerations are framed as part of enforcing ethical standards and transparency in research.Source: Comprehensive Ethical and Societal… · p. 37

Real-World Applications and Societal Impact

• Practical-impact discussions should consider healthcare decision-making, data privacy and security, human-subject impacts, environmental effects, deployment guidelines, compliance, and future work.The guidance links research findings to societal impact and future implications.Source: Real-World Applications and Societal… · p. 37

Clarification and Highlighting of Novel Contributions

• The checklist promotes clear methodology, justified answers, guideline adherence, and explicit novelty and contribution claims, with adversarial prompts revising justifications against reviewer feedback and paper content.The attack prompt instructs revisions to be technically sound, concise, guideline-compliant, and linked to appropriate paper sections.Source: Clarification and Highlighting of… · p. 37, Transparency and Clarity in Research · p. 37, D Adversarial Prompt · p. 38

E Adversarial attack with multiple rounds budgets

• Score improvements increased across the first three questions with larger attack budgets, but no improvement occurred for the last question after three rounds.The experiment allowed three revision attempts and evaluated performance across increasing attack budgets; the authors attribute inconsistency partly to the attack algorithm’s simplicity.Source: E Adversarial attack with multiple… · p. 39, Figure 13 · p. 39

Claims, limitations, and theory

• The paper accurately states its contributions and scope, explains its limitations, and appropriately marks theoretical-proof requirements as not applicable to its empirical work.The review noted limitations covering assumptions, scope, performance factors, computational efficiency, privacy, and fairness, while finding no theoretical claims requiring proofs.Source: Checklist Reviews · p. 40

Reproducibility and open access

• The authors report publicly available code and dataset-access information, but the review found that reproducibility claims lacked precise in-paper references and sufficiently detailed instructions.The paper should identify relevant sections, architectures, hyperparameters, preprocessing, dependencies, licensing, environments, and execution procedures rather than relying mainly on GitHub and a website.Source: Checklist Reviews · pp. 42–43

Training and test details

• Although the authors identify Section 3.2 and GitHub for training and test details, the review could not verify completeness and required essential information to appear in the paper itself.Specifically, the review highlighted data splits, hyperparameters, optimizer choices, and hyperparameter-selection procedures as necessary for understanding and reproducing the results.Source: Checklist Reviews · pp. 43–44

Statistical reporting

• Error bars are reported in Figures 2–3 and Appendices E–F, but the review recommended specifying their variability source, calculation method, confidence level, and interpretation.The supplied text does not establish whether the bars represent standard deviation, standard error, or another quantity, nor whether 95% confidence intervals or a sigma level are used.Source: Checklist Reviews · p. 44

Computational resources

• The authors report computer-resource details in Section 3.2 and Appendices E–F, and the review considered this justification broadly consistent with the checklist requirements.These locations cover resources associated with the paper’s few-shot-learning experiments, although the review’s assessment is based on the stated locations rather than detailed verification of their contents.Source: Checklist Reviews · p. 45
• The paper omits detailed compute specifications, execution-time breakdowns, memory and storage requirements, and accounting for preliminary or failed experiments.The review recommends explicit reporting of worker types, configurations, providers, dataset-level runtimes, and total computational costs.Source: Checklist Reviews · p. 45

Ethics and societal impact

• The generic ethics justification does not identify supporting sections, clearly discuss ethical considerations or fairness, or address both positive and negative societal impacts.The review specifically notes that claiming no negative societal impacts fails the requirement to discuss potential risks and mitigation.Source: Checklist Reviews · pp. 45–46

Responsible data release

• The responsible-release justification cites licensing and terms of use but does not identify paper sections or describe safeguards beyond licensing, such as access controls or misuse mitigation.Appendix B contains license information, but the review requests clearer guidance, ethical-use recommendations, and accessibility or monitoring details.Source: Checklist Reviews · p. 47

Asset attribution and licensing

• The paper does not explicitly verify that all external datasets, code, models, and other assets are credited, license-compliant, and subject to clearly stated use conditions.The review recommends expanding Appendix B and documenting third-party assets, restrictions, compatibility, and unresolved licensing questions.Source: Checklist Reviews · p. 48

New-asset documentation

• New-asset documentation points vaguely to appendices and a website without specifying locations for licensing, limitations, preprocessing, consent, or contributor-consent information.The review also requests operational access instructions, backup repositories, and appropriate anonymization during double-blind review.Source: Checklist Reviews · pp. 48–49

Human subjects and IRB

• The paper appropriately marks crowdsourcing, human-subjects, participant-risk, and IRB questions as not applicable, but should clarify data sources, licensing, processing, privacy, and dataset bias.The review considers the current scope free of direct human-subject involvement while recommending stronger responsible-AI reporting for publicly available or licensed image datasets.Source: Checklist Reviews · pp. 49–50