Source-linked AI summary
Governing AI Research Through Peer Review: A Mixed-Methods Study of the Longitudinal Effects of Ethics Flags Across Resubmissions
Kento Nishi, Alec Laprevotte, Isaiah Bullock, Mfoniso Andrew
TL;DR
The study asks whether ethics flags steer research after rejection, when the original reviewers no longer oversee the project. The authors compare concern-implicated sections across each paper–resubmission pair and qualitatively code whether changes are absent, rhetorical, procedural, methodological, or both. 83.0% of 446 resubmissions left the concern unaddressed or made mainly rhetorical changes, while 17.0% made notable procedural or methodological changes.
Problem
The study asks whether ethics flags steer research after rejection, when the original reviewers no longer oversee the project.
Source: §1 · p. 2Method
The authors compare concern-implicated sections across each paper–resubmission pair and qualitatively code whether changes are absent, rhetorical, procedural, methodological, or both.
Source: §3 · p. 3Results
83.0% of 446 resubmissions left the concern unaddressed or made mainly rhetorical changes, while 17.0% made notable procedural or methodological changes.
Source: §3 · p. 3Takeaways & Limitations
The authors recommend confidential disclosure of prior ethics flags on resubmission so unresolved concerns can receive independent review at a new venue.
Source: §5 · p. 4, §6 · p. 4The corpus includes only ICLR submissions, and only three of 25 invited researchers responded, limiting claims about broader researcher sentiments.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Selective AI conferences have recently begun enforcing ethics flags and related review requirements, with the goal being to steer research towards safer and more responsible practices before publication. But do these requirements actually steer research as intended? In this paper, we show that authors more often revise how projects are presented following ethics flags than redirect their underlying research agendas. We first study the longitudinal effects of ethics flags by following rejected and withdrawn ICLR submissions with ethics flags into later public resubmissions, tracking manuscript changes after the ICLR review ends, when the original reviewers no longer oversee the project. We qualitatively code these resubmissions into five categories based on what changed after review and find that in 83% of 446 cases, authors leave the flagged concern unaddressed or revise the paper without changing the implicated methods or procedures. Then, we ask: if authors rarely change the research in response to ethics flags, what do they change instead? To answer this, we manually read reviews and rebuttals from 25 cases and directly interview authors about their rebuttal processes and resubmission decisions. We find that authors often concede concerns during rebuttal when reviewers can update their assessments, but drop those concessions after rejection when they do not regard the criticism as a sound reason to change the research. Interview participants describe publication changes as separate from changes to research direction, calling review an "editorial process" that shapes "what stories get seen" and, in another case, saying peer reviews are "mostly to filter out papers." Authors more readily change what they publish than what they study or build; we therefore recommend policy changes, especially disclosure of prior ethics flags upon resubmission so accountability carries over.
1 Introduction
Conference ethics requirements give reviewers a way to steer research toward safer practices, but that leverage arrives after many research choices are already made. The paper asks whether authors carry ethics-related changes into resubmissions and finds that substantive changes are uncommon.
1 Introduction
- Conference peer review gives reviewers leverage to steer research toward safer and more responsible directions.The introduction frames conference influence over research attention and legitimacy as a responsibility.Source: §1 · p. 2
- Reviewers assess submissions after authors have usually chosen the problem, collected data, implemented the system, and completed most experiments.Reviewers can update scores during rebuttal, but lose influence after rejection; a new panel may not re-raise the concern because reviewer judgments are noisy.Source: §1 · p. 2
- Only 17% of 446 resubmissions showed procedural or methodological changes addressing a flagged concern; authors often revised papers without redirecting research.The study tracks rejected or withdrawn ICLR submissions and combines version comparisons with review histories and author interviews.Source: §1 · p. 2
2 Related Work
Prior work connects technical design and evaluation to social consequences and proposes ethics requirements in peer review. Existing studies of conference ethics review examine individual cycles or ethical practice more broadly, rather than whether review changes research over time.
2 Related Work
- Technical design and evaluation can distribute social consequences, a concern discussed across sociotechnical systems and human-computer interaction research [Winner, 1980, Nissenbaum, 1996, Selbst et al., 2019, Stilgoe et al., 2013, Karusala et al., 2024].Source: §2 · p. 2
- Hecht et al. [2021] propose confronting negative impacts during peer review, while Prunkl et al. [2021] argue that broader impact requirements can institutionalize ethical reflection.Source: §2 · p. 2
- Studies of conference ethics review have examined isolated review cycles or ethical practice more broadly, but not whether ethics reviews shape research and how.NeurIPS established ethics guidelines in 2021, and ICLR program chairs reported screening 190 ethics-flagged papers in 2023 [ICLR Program Chairs and Ethics Review Chair, 2023].Source: §2 · p. 2
3 Longitudinal Analysis: Do Authors Change Research After Ethics Flags?
The longitudinal analysis matches rejected or withdrawn ICLR submissions with later resubmissions, then checks whether the methods or procedures implicated by ethics concerns changed. Most resubmissions either leave the concern unaddressed or change its presentation rather than the implicated research.
3 Longitudinal Analysis: Do Authors Change Research After Ethics Flags?
- The study collected 47,893 ICLR submissions and 185,194 public reviews from 2021–2026, identifying 2,498 ethics-flagged submissions and 446 verified resubmissions.The 2021 snapshot lacked public flags; candidate resubmissions were retrieved using title variants, author surnames, and abstract terms, then manually checked.Source: §3 · pp. 2–3
- For each resubmission, researchers compared the parts implicated by the original concern and coded changes to procedures, methods, both, or neither.Changes to discussion of harms, limitations, or claims counted as rhetorical rather than research changes when the implicated methods or procedures stayed the same.Source: §3 · p. 3
- 83.0% of 446 resubmissions showed no observable change related to the concern or mainly rhetorical revisions, while 17.0% made notable procedural or methodological changes.The 76 cases with notable changes included safety evaluations, consent or institutional review, identifier removal, and changes to data access or release.Source: §3 · p. 3, Table 1 · p. 3
4 Qualitative Analysis: What Do Authors Change Instead?
The qualitative analysis combines 25 review histories with interviews of three responding researchers to explore why revisions do or do not follow ethics criticism. It finds that authors may concede concerns during rebuttal yet later treat review-driven revisions as changes to presentation rather than research direction.
4 Qualitative Analysis: What Do Authors Change Instead?
- The analysis covered 25 cases, including 11 apparent research changes, 6 pre-existing practices, and 8 cases with no apparent change; three invited researchers were interviewed.Cases spanned harmful applications, fairness and discrimination, responsible research practice, and privacy, security, and safety.Source: §4 · p. 3
- Authors often responded at length to ethics criticism during rebuttal, including partial concessions that were absent from later resubmissions.In most of the 25 review histories, only one of three to five reviewers raised the concern; some rebuttals treated it as a dispute over disciplinary conventions rather than a reason to change research.Source: §4 · p. 3
- The authors call it filtering when papers are adapted for publication without treating criticism as grounds to redirect research.P1 described mostly similar resubmissions, with changes such as adding a table or metrics.Source: §4 · p. 3
- P2 described review as an editorial process that shaped which stories were seen, while adapting the paper for a computer-science audience and conducting another annotation pass with 5,000 participants.P2 said reviewers focused on a particular aspect rather than the larger picture and that the central criticism concerned presentation.Source: §4 · p. 4
- P3 reported rewriting the introduction and abstract while leaving methods and results essentially unchanged, though publication changes can clarify limitations or support safer release plans.The authors argue that ethics review intended to govern research should encourage longitudinal and deeply procedural changes.Source: §4 · p. 4
5 Discussion: How Should AI Research Be Governed?
The authors recommend carrying ethics flags across resubmissions, reviewing research earlier, and improving reviewer fit. These changes aim to keep unresolved concerns under scrutiny while making feedback more timely and credible.
5 Discussion: How Should AI Research Be Governed?
- Conferences should require confidential disclosure of prior ethics flags on resubmission so unresolved concerns receive independent scrutiny without binding the new venue to the prior verdict.The disclosure would include the concern category and whether authors changed the research or disputed the concern.Source: §5 · p. 4
- Because data collection and experiments may be complete by full-paper review, conferences could require an early-stage ethics review of data collection and experimental design.Later review would remain available for broader critiques.Source: §5 · p. 4
- Better domain matching could make reviewers’ criticism more credible when authors regard feedback as a misunderstanding.The authors describe participants discounting criticism they viewed as repeating an objection to removed text or missing the larger picture.Source: §5 · p. 4
6 Limitations and Conclusion
The study suggests that manuscript revisions do not necessarily redirect underlying research. The authors recommend earlier review, better domain matching, and disclosure of prior flags, while noting limits to the study’s scope and generalizability.
6 Limitations and Conclusion
- The study covers only ICLR submissions, may miss projects abandoned after an ethics flag, and received responses from three of 25 invited researchers.The authors expect identified mechanisms to extend to other top AI conferences, but their prevalence may differ; broader claims about researcher sentiment need more research.Source: §6 · p. 4
- Substantial manuscript revisions do not necessarily mean ethics review redirected the underlying research.The authors recommend that conferences review earlier, match reviewers by domain, and require disclosure of prior flags upon resubmission so rejection does not reset accountability for unresolved concerns.Source: §6 · p. 4
- After rejection, original reviewers lose leverage, and reviewer noise may keep an ethics concern from resurfacing in a later review.The authors say concessions can matter while reviewers still influence the decision.Source: §6 · p. 4