Source-linked AI summary
Generating Edit-Inducing Questions for AI Research Manuscripts
Sebastian Joseph, Zichao Wang, Jennifer Healey, Alexa Siu, Junyi Jessy Li, Ani Nenkova
TL;DR
The paper examines whether questions from reviewers and LLMs prompt authors to improve paper drafts. It compares questions with paired paper versions and finds that LLM questions are associated with more extensive edits and account for about half of the changes authors made.
Problem
Authors may find it difficult to anticipate what readers do not know, and the paper asks whether reviewer and LLM questions are edit-inducing and how much of authors’ edits they account for.
Source: §1 · p. 1Method
Results
LLM questions have higher edit rates than reviewer questions, and automatically produced questions account for about half of the edits between initial submissions and final versions.
Source: §6 · p. 4Takeaways & Limitations
Edit-inducing questions can serve as a signal for useful feedback, and language models may benefit AI-paper writers by asking questions as they read segments.
Source: §5 · p. 4, §6 · p. 4The experiments use a single family of closed models, GPT.
Source: Limitations · p. 5
Abstract
from arXiv · showhide
We study the ability of LLMs to generate edit-inducing questions whose answer will improve a paper draft. On a dataset of paired submission and camera-ready papers from ICLR and NeurIPS, we compare the helpfulness of questions from GPT models with or without full paper context to that of human reviewers. GPT produces more edit-inducing questions and its questions are associated with more extensive edits and cover a broader range of edited content compared to questions from reviewers. However, a much smaller percentage of the GPT questions are edit-inducing. Our analyses confirm that automated questions can be beneficial to authors and highlight an example task where proper attending to long context deteriorates reasoning model ability to produce helpful output.
1 Introduction
The paper evaluates whether questions from human reviewers and LLMs prompt author edits by comparing questions with paired paper drafts. LLM questions are associated with more extensive edits and account for a larger share of edits than reviewer questions.
1 Introduction
- Authors may struggle to anticipate what readers do not know, making feedback on drafts a practical way to surface readers’ questions.Source: §1 · p. 1
- Peer reviewers are expected to help improve papers and their presentation, but publication growth is making this delayed feedback harder to obtain.Source: §1 · p. 1
- The study asks whether reviewer and LLM questions are edit-inducing and what fraction of author edits they account for, using author decisions to assess question quality.Source: §1 · p. 1
- The analysis compares initial submissions with final versions to identify added content and checks whether reviewer or LLM questions target that content by searching for answers in the papers.Source: §1 · pp. 1–2
- LLMs generate an order of magnitude more edit-inducing questions than human reviewers, and their questions correspond to more extensive edits and a larger percentage of author edits.The comparison uses changes between submitted and final paper versions.Source: §1 · p. 2
2 Data & Methodology
The study uses paired submissions and camera-ready papers from a PeerQA subset to test questions from reviewers and GPT models against changes in the papers. It generates questions from segments under different context settings and uses automated answer detection to assess whether answers appear in each version.
2 Data & Methodology
- The analysis uses 116 papers from ICLR 2022, ICLR 2023, and NeurIPS 2022, pairing initial submissions with camera-ready versions retrieved through submission histories.PeerQA contains reviewer questions and answers grounded in final paper versions (Baumgärtner et al., 2025).Source: §2 · p. 2
- GPT-4o and o3 generate questions from 512-token segments using full-paper, preceding-segment, or focused-segment context.Segments are formed from paragraphs and split at section boundaries; figures and tables count as paragraphs.Source: §2 · p. 2
- Automated generation yields about 140 questions per paper for GPT-4o segment and 206 for o3 segment, compared with about three reviewer questions per paper.The question-count comparison is reported in Table 2 in Appendix C.Source: §2 · p. 2
- For each question, the system retrieves five similar paper segments with text embeddings and prompts GPT-4o to decide whether any segment answers it.Source: §2 · p. 2
3 Measuring Edit-Inducing Capability
The paper assesses question helpfulness using measures of validity, whether questions induce edits, the extent of associated edits, and the share of paper changes covered by a source’s questions.
3 Measuring Edit-Inducing Capability
- The four helpfulness measures are calculated per paper and reported as averages across papers.Source: §3 · p. 2
- A valid question is not already answered in the submitted manuscript, while an edit-inducing question is a valid question answered in the final version.The paper treats such answers as empirically verified actionable feedback, assuming authors make changes to improve the paper.Source: §3 · p. 2
- Edit rate measures the proportion of characters added or substituted in the final-version segment containing an answer, relative to the original segment’s length.Source: §3 · p. 2
- Edit coverage is the fraction of all changed characters attributable to a source’s edit-inducing questions, indicating how much of the authors’ added content their answers cover.Source: §3 · p. 3
4 Results
GPT models generate far more edit-inducing questions than human reviewers, and their questions are associated with more extensive edits and broader edit coverage. However, only a small share of GPT questions are edit-inducing, and GPT-4o questions are more extractive than o3 questions.
4.1 Human Peer-Review Questions
- 38% of human peer-review questions are edit-inducing, averaging about one per paper and covering around 4% of author edits.The authors attribute the low edit coverage to the scarcity of such questions; authors incorporate substantially more feedback beyond peer reviews.Source: §4.1 · p. 3
4.2 GPT Edit-Inducing Questions
- 11–19 GPT questions per paper are edit-inducing, but only around 7–10% of each model’s questions meet that criterion.The models generate many questions but have limited ability to restrict questions lacking utility.Source: §4.2 · p. 3
- GPT models ask a higher percentage of valid questions than human reviewers, and larger context tends to raise that share, especially for the reasoning model.Even segment-only prompts can produce questions that the rest of the paper does not answer.Source: §4.2 · p. 3
- Figure 2 summarizes question-quality measures; GPT questions correspond to larger edited spans and cover more author edits than peer-review questions.GPT also generates markedly more questions, contributing to its higher edit coverage.Source: §4 · p. 3, §4.2 · p. 3, Figure 2 · p. 4
4.3 Copying Behavior
- GPT-4o questions show significantly greater word overlap with their source spans, consistent with more extractive wording than o3 questions.The authors describe edit-inducing ability as broadly on par between the reasoning and non-reasoning models.Source: §4.3 · pp. 3–4
5 Discussion and Future Work
The discussion identifies the low proportion of edit-inducing LLM questions as a practical concern and leaves effective, inexpensive pruning for future work. It also notes that some unanswered questions may still be useful because page limits or author time can prevent edits.
5 Discussion and Future Work
- The low proportion of edit-inducing LLM questions makes effective, inexpensive pruning an open problem for future work.The authors assume edit-inducing questions are preferable feedback, while noting that unanswered questions may still be good questions.Source: §5 · p. 4
- Page limits or lack of time may prevent authors from editing in response to an otherwise good question.The authors suggest human-subject experiments to investigate why some questions go unanswered.Source: §5 · p. 4
6 Conclusion
The conclusion presents LLMs as potential indefatigable readers for AI-paper writers, generating questions from paper segments. It reports more questions and edit-inducing questions than peer reviewers, with higher edit rates and greater edit coverage.
6 Conclusion
- LLMs may benefit AI-paper writers as indefatigable readers that ask questions arising from paper segments.The conclusion describes automatically generated questions as a potential source of feedback for writers.Source: §6 · p. 4
- LLMs generate an order of magnitude more questions than peer reviewers, and their questions have higher edit rates and account for about half of authors’ edits.The conclusion also describes reasoning-model questions as more valid and objectively less extractive in wording than non-reasoning-model questions.Source: §6 · p. 4
Limitations
The experiments are limited to one closed model family and a restricted set of papers; the appendices specify prompts for generating questions and checking whether chunks answer them.
Limitations
- The study tests only the closed GPT model family, so its findings do not establish how other model families perform.The authors varied context and model type within GPT; they chose one family to keep tables manageable and compare modeling choices.Source: Limitations · p. 5
- Automated answer detection is used extensively; its precision is good, but the system often misses existing answers.Comparison with gold-standard human labels suggests the reported share of edit-inducing GPT questions may be lower than the true share.Source: Limitations · p. 5
- The dataset constraint limits the analysis to papers accepted at the same venue where they were submitted.The authors identify rejected papers later accepted at another venue as a setting for future analysis.Source: Limitations · p. 5
A Question Generation Prompt
- The question-generation prompt asks for insightful follow-up questions that a reader would have after considering the context, while avoiding questions already answered there.It supplies a question typology and requests each question’s exact corresponding text span.Source: A Question Generation Prompt · p. 6
B Answerability Prompt
- The answerability prompt asks whether provided chunks fully answer a question, selecting the fewest sufficient chunks and returning an exact supporting span and brief answer.If no chunk answers the question, the response is No, with no selected chunks and an empty answer.Source: B Answerability Prompt · pp. 6–7
C Total number of generated questions
GPT generates many more questions than peer reviewers, with question counts varying substantially across papers and changing with context differently for 4o and o3.
C Total number of generated questions
- GPT generates many more questions than peer reviewers, while context has opposite effects on 4o and o3 question counts.Full-paper context yields 15 more questions per paper on average for 4o; o3 averages 206 with segment-only context and 163 with full-paper context.Source: C Total number of generated questions · p. 7
- There is a tenfold difference between the paper with the fewest generated questions and the paper with the most.The authors note that the papers themselves modulate question counts.Source: C Total number of generated questions · p. 7
D Assessing Answerability
The answerability pipeline is assessed against PeerQA gold-standard data, and the authors use a top-five segment cutoff as a cost–coverage tradeoff.
D Assessing Answerability
- Precision is good, but the automated answerability pipeline often fails to detect an existing answer.Table 3 evaluates precision, recall, and F1 @ 5 for gpt-4o deciding whether a segment answers a question, against ground-truth PeerQA answers.Source: D Assessing Answerability · p. 7, Table 3 · p. 8
- The top-five segment cutoff balances cost against answer coverage; preliminary tests found the ground-truth answer in those segments 72% of the time.Evaluating every paper segment was prohibitively expensive, while the authors considered top five sufficient to capture the vast majority of existing answers.Source: D Assessing Answerability · p. 7
E Edit-inducing and unanswered questions
Table 4 presents random examples of edit-inducing and unanswered questions from different model and context settings. The examples show questions probing methods, assumptions, experimental choices, and reported findings.
E Edit-inducing and unanswered questions
- Table 4 juxtaposes random examples of edit-inducing and unanswered questions across model and context settings.Source: Table 4 · p. 8, E Edit-inducing and unanswered questions · p. 8
- Some examples request further interpretation of results, such as differences between tracking models or the diminishing benefit of synthetic data as real-shot counts rise.Source: E Edit-inducing and unanswered questions · pp. 8, 10
- Several examples seek methodological clarification, including how direction affects edge weights and how marginal utility is estimated.Source: E Edit-inducing and unanswered questions · pp. 8–9
- Experimental questions also ask for sensitivity analyses, comparison details, or measurements of behaviors and outcomes.Source: E Edit-inducing and unanswered questions · pp. 8–10
- Other questions probe assumptions or design choices, asking about reward calibration, noisy-gradient bounds, and alternatives to chosen features or batching.Source: E Edit-inducing and unanswered questions · p. 10
F Question Style Example
The appendix gives examples of question styles and edit rates, states the dataset-use terms, and specifies the models and decoding settings used in the experiments. It also includes examples of human-written questions and how they are assessed.
F Question Style Example
- 4o questions repeat more content from their source paragraphs than o3 questions, a pattern quantified later in Section 4.3.Source: F Question Style Example · p. 11, Table 5 · p. 11
G Example edit-inducing questions and respective edit rates
- The examples section pairs human-written PeerQA questions with edit rates and the corresponding textual changes.Source: G Example edit-inducing questions and… · p. 11
H License Information
- The PeerQA papers and reviews are publicly accessible through OpenReview, and the authors state their use is solely for non-commercial research.Source: H License Information · p. 11
I Hyperparameter Details
- Experiments used OpenAI’s gpt-4o-2024-11-20 and o3-2025-04-16 with default temperature 1.0 and top-p 1.0, without custom decoding.Source: I Hyperparameter Details · p. 11
- Table 6 gives human-written PeerQA question examples that illustrate question types and how they are assessed.Source: Table 6 · p. 12