Commitment Checklist: Auditing Author Commitments in Peer Review

• After acceptance, promised changes typically lack a formal enforcement mechanism, and reviewers rarely revisit papers, which can leave reviewers unsure whether their feedback was incorporated.Authors commonly pledge changes in response to reviewer feedback, and those promises can influence decisions.Source: §1 · p. 1
• Rebuttals now contain extensive promises, while manually tracking tens of thousands of commitments is infeasible, motivating exploration of LLM-based document comparison.Commitments range from experiments and proofs to code releases and writing improvements.Source: §1 · p. 1
• The paper presents an empirical study of commitments at ICLR-2025 and EMNLP-2024, using an automated pipeline to extract and categorize promises and verify them against camera-ready papers.The analysis examines prevalence, fulfillment rates, and the types of promises most often unmet.Source: §1 · pp. 1–2
• A non-trivial share of author promises go unmet, especially substantial new work; missing experiments or comparisons are common and among the most frequently unfulfilled commitments.LLM-based checks can identify a majority of these cases automatically.Source: §1 · p. 2
• The proposed Author Commitment Checklist would present rebuttal-extracted commitments to authors for verification at camera-ready submission, aiming to improve accountability.The paper argues that the checklist could discourage insincere promises and help ensure published papers deliver expected improvements.Source: §1 · p. 2

• Open reviews and process reports primarily address decisions and reviewer behavior, while reproducibility checklists encourage practices at submission and post-publication checks are unsystematic.The proposed ACC would formally record and check commitments before publication, adding visibility into whether authors follow through on review promises.Source: §2 · p. 2

3 Methodology

• The study has three components: collecting data from two conferences, extracting and categorizing commitments, and LLM-based verification with human validation.Source: §3 · p. 2

3.1 Data Sources

• The data cover ICLR 2025 and EMNLP 2024 rebuttals and camera-ready papers, with the camera-ready version serving as the target for checking promised changes.ICLR review discussions are publicly accessible through OpenReview, whereas EMNLP 2024 responses and discussions were not publicly released after the conference.Source: §3.1 · pp. 2–3

3.2 Commitment Extraction and Verification

• An author commitment is an explicit response statement promising content beyond the original submission, and the extraction model returns verbatim commitment spans aligned to relevant reviewer comments.The pipeline organizes the promises as atomic commitments paired with related critiques.Source: §3.2 · p. 3

Commitment Verification in the Final Version

• For each commitment, an LLM compares its verbatim text and associated reviewer comment with the final paper, then assigns a binary fulfillment judgment with optional evidence-based justification.Each promise is assessed independently; Table 1 summarizes commitment statistics for the two venues.Source: Commitment Verification in the Final… · p. 3, Table 1 · p. 3

4.1 Volume of Commitments: ICLR vs. EMNLP

• ICLR 2025 authors made 11.8 commitments per paper on average (median 11), versus about 4.1 (median 3) for EMNLP 2024; maxima were 59 and 21.Source: §4.1 · p. 3

4.2 Fulfillment Rates

• 75.89% of ICLR 2025 commitments and 72.91% of EMNLP 2024 commitments were fulfilled; about 24% and 27%, respectively, were unfulfilled.Source: §4.2 · pp. 3–4
• Missing experiments or comparisons were the most common broken promise across both conferences, with time or computational-resource constraints offered as likely reasons.Source: §4.2 · p. 4
• Other broken promises included missing theory or proofs, unreleased code or data, unclarified concepts, and omitted discussion or limitations.Source: §4.2 · p. 4
• Promised restructuring or appendix moves and uncorrected writing also remained among the broken commitments.Source: §4.2 · p. 4

Quadrant IV: Low Severity / High Difficulty

• Missing figures or examples are low-severity, high-difficulty changes that improve presentation rather than scientific validity, and their absence is often tolerated.Source: Quadrant IV: Low Severity / High… · p. 4

4.2 Fulfillment Rates

• Although many broken promises are minor, a substantial share fall in high-severity categories that pose risks to scientific integrity and reproducibility.Source: Quadrant IV: Low Severity / High… · p. 4

4.3 Interpreting Non-Fulfillment and the Role of Transparency

• Non-fulfillment is not simply evidence of bad faith: feasibility varies, and experiments or exploratory analyses may yield negative or inconclusive results.The authors note that the acceptance-to-camera-ready window can be very short, and argue that omissions should be explained to improve transparency rather than treated as grounds for punishment.Source: §4.3 · pp. 4–5

5.1 Human Reference Acceptance and Model Accuracy

• 600 LLM-identified commitments were sampled, with native English speakers annotating them and 200 overlapping cases used to assess consistency.The authors expect results to improve as models improve and more annotated data becomes available for calibration.Source: §5.1 · pp. 5–6
• 74%–78% LLM evaluator acceptance rates were comparable to humans’ 72.86% reference acceptance; the authors say approximately 75% accuracy can limit spurious alarms.The human reference acceptance rate is the proportion of annotator-provided answers judged correct by an evaluator.Source: §5.1 · p. 5
• The proposed checklist should let authors mark cases as not applicable, no longer relevant, or possible detection errors rather than treating model outputs as enforcement.These options allow authors to contest or clarify ambiguous cases.Source: §5.1 · p. 6

5.2 Computational and Financial Cost

• $0.0278 USD is the estimated total LLM cost per EMNLP 2024 paper for promise identification and camera-ready verification.The estimate uses gemini-2.5-Pro for identification and gemini-3-preview for verification.Source: §5.2 · p. 6, Table 5 · p. 6

5.3 Implications for Deployment

• The authors judge routine deployment feasible: the cost is negligible against conference budgets, and a checklist could fit existing workflows without meaningful financial burden.They characterize the checklist as a reminder and documentation tool, not a source of new reviewer labor or strict policing.Source: §5.3 · p. 6, §5.2 · p. 6

6 Author Commitment Checklist

• Non-fulfillment is often attributed to preventable omissions or neglect rather than deliberate misconduct, motivating a checklist that requests evidence or an explicit justification.Examples include missing code or clarifications, as well as overlooked rewording or restructuring.Source: §6 · p. 6

6.1 Design Principles: Transparency over Punishment

• ACC makes commitments observable and fulfillment verifiable, while prioritizing high-severity items for attention.The proposed principles surface verbatim commitment text and its reviewer request, and ask for a final-paper pointer or a brief reason for non-fulfillment.Source: §6.1 · p. 7

6.2 Workflow Integration

• The workflow is trust-first: authors verify their own commitments, while venues may optionally add post-deadline checks and organizer-facing summaries.Commitments can be extracted before decisions, reviewed by authors before camera-ready submission, and optionally checked by the conference afterward.Source: §6.2 · p. 7

6.3 What Authors Actually See: Concrete Examples

• Authors see a human-facing checklist with one atomic commitment per row and a tentative system status they can explicitly override.They confirm fulfillment, explain non-fulfillment, or report a misdetection; the status is a reminder, not a binding judgment.Source: Table 6 · p. 7, §6.3 · p. 8
• The form is intended to reduce accidental omissions and encourage explanations when experiments are negative or resources are constrained, without penalizing authors.It makes outcomes of rebuttal commitments transparent and auditable, aligning review-time discussion with the published record.Source: §6.3 · p. 8

6.4 Anticipated Impact and Scope

• ACC targets common, preventable omissions and can begin as an author-facing reminder, with organizer summaries added if a venue wants stronger accountability.The authors say requiring evidence or justification can reduce silent non-fulfillment and discourage overpromising during rebuttal.Source: §6.4 · p. 8

7 Discussion and Conclusion

• Non-fulfillment can reflect negative results, feasibility constraints, or short camera-ready timelines, so it is not automatically evidence of bad faith.The authors argue for making promises visible and interpretable at camera-ready time rather than silently omitting them.Source: §7 · p. 8
• For papers averaging roughly 4–11 commitments, the authors describe checklist confirmation as a small burden relative to rebuttal writing and final revision.They note that reviewer requests for additional experiments can contribute to pressure to promise more, even when additions are weakly tied to core claims.Source: §7 · p. 8
• Requiring evidence or an explanation may help distinguish requests that affect paper claims from those that mainly add workload, supporting clearer rebuttal communication.The authors present ACC as a practical step toward clearer commitments and better alignment between review-time discourse and the published record.Source: §7 · pp. 8–9

Coverage and representativeness

• The study covers only ICLR 2025 and EMNLP 2024, which may not represent other fields, venues with different rebuttal policies, or journals.The conferences differ in transparency, rebuttal duration, and response format.Source: Limitation · p. 9

Definition and granularity of commitments

• Defining commitments as explicit action statements excludes implicit intentions and can split or merge promises inconsistently, affecting reported counts and fulfillment rates.The boundary between a promise and discussion, as well as extraction granularity, shapes the measurements.Source: Limitation · p. 9

LLM extraction and verification errors

• LLM-based extraction and verification can produce false positives, miss subtle promises, misalign reviewer comments, or overlook indirect and relocated evidence.The authors note that human agreement also indicates ambiguity intrinsic to the task.Source: Limitation · p. 9

Fulfillment as a binary label

• Binary fulfillment labels can overstate compliance or non-compliance when authors make partial progress, motivating graded labels and a distinction between negative attempts and unattempted work.The authors suggest fulfilled, partially fulfilled, and not fulfilled categories as a more faithful evaluation.Source: Limitation · p. 9

Interpretation and normative scope

• Non-fulfillment does not establish bad faith, and the analysis does not estimate author intent, effort, or each promise’s importance to acceptance.The authors therefore frame ACC as documentation rather than enforcement and warn against superficial box-checking.Source: Limitation · p. 9

Potential behavioral effects of ACC

• ACC could change rebuttal behavior by prompting fewer explicit promises, more hedging, or vaguer commitments; this study does not test those effects.The authors say prospective evaluation, such as a venue A/B test, is needed.Source: Limitation · p. 9

A Prompt

• Table 7's prompt identifies author commitments.It also aligns each commitment with reviewer comments.Source: A Prompt · p. 10
• The prompt links commitments to their aligned reviewer comments.Source: A Prompt · p. 10
• Table 7 is used to identify commitments and their reviewer-comment alignments.Source: A Prompt · p. 10

B Prompts for Commitment Classification

• Table 9's prompt classifies unfulfilled promises into specific categories.Source: B Prompts for Commitment Classification · p. 10
• The prompt determines the severity of unfulfilled promises.Source: B Prompts for Commitment Classification · p. 10
• The prompt also assigns unfulfilled promises to difficulty quadrants.Source: B Prompts for Commitment Classification · p. 10

C Quadrant Mapping Logic

• Classification results are mapped to a four-quadrant framework for assessing the impact of broken promises.Source: C Quadrant Mapping Logic · p. 10
• Table 9's prompt classifies unfulfilled commitments into nine categories.Source: Table 9 · p. 12
• The framework distinguishes severity from difficulty.Source: Table 10 · p. 13