Who Pays for Open Review? Visible Author Reputation and Its Effect on Ratings

1 Introduction

• The study is motivated by calls for open review after a November 2025 OpenReview bug exposed identities across several conferences.Prior evidence included small randomized trials or venues that do not publish their review records, leaving the effect’s size and rating-scale location open.Source: §1 · p. 1
• Ratings rise with author reputation under both mechanisms, but more steeply under open review, with the difference concentrated at borderline ratings.Under the institution proxy, the slope difference is significant for mean-author (p = 0.008) and max-author aggregation (p = 0.041).Source: §1 · p. 1
• With the manuscript fixed, four of five AI reviewers raise ratings by 0.2–0.6 points as stated author reputation moves from low to high.The simulation samples 600 submissions and rates each with five AI models.Source: §1 · pp. 1–2
• The study links ICLR 2026 submissions and author profiles to reputation proxies, then splits submissions by whether an arXiv preprint appeared before review began.The split provides de facto open and blind groups; ICLR publishes the complete review record for every submission.Source: §1 · p. 1
• The authors report that the pattern holds across five reputation proxies, three author-aggregation rules, five open-window definitions, and three input scopes.The proxies include institution reputation, h-index, citation count, publication count, and i10-index.Source: §1 · p. 1

2 Related Work

• Earlier studies find that visible author information favors high-reputation researchers, while hiding identity reduces that advantage.Prior work also reports rating shifts associated with visible signals such as resubmission tags or citations to reviewers’ own work.Source: §2 · p. 2
• Prior evidence comes from cross-venue or cross-year comparisons, small within-venue randomized trials, or studies of a single review mechanism.The paper contrasts open and blind mechanisms on more than 18,000 submissions to one conference and examines where the effect acts on the rating scale.Source: §2 · p. 2
• The study treats preposting before review as an approximate marker of visibility, not a randomized assignment, based on prior evidence that early arXiving’s acceptance effect was small and similar across reputation strata.The cited five-year ICLR study is Zhang et al. (2023).Source: §2 · p. 2
• AI-reviewer experiments can vary stated reputation while holding the manuscript fixed, and this study compares those estimates with human reviews of the same submissions.Prior work finds AI reviewers track paper quality but compress ratings and favor polished text; it also reports favorable ratings for recognized authors and prestigious institutions when reputation is visible.Source: §2 · p. 2

3 Analytical Framework

• The framework assumes expected paper quality is non-decreasing with author reputation; holding it fixed requires fixing the manuscript, which the controlled simulation enforces and the observational split only approximates.Under blind review, the reviewer does not observe author reputation and uses the pool’s mean reputation; under open review, the reviewer observes the author’s reputation.Source: §3 · p. 2
• The open-review premium is positive above the pool’s mean reputation and negative below it, making the reputation effect a transfer toward more established authors rather than a bonus for everyone.The framework represents a review instance by author reputation, paper quality, and review mechanism.Source: §3 · p. 2
• Because expected paper quality is common to both mechanisms, the difference between their reputation slopes identifies the visible-reputation parameter; a blind-review gradient can reflect content alone.The paper estimates this slope difference with the reputation × open interaction in a linear regression.Source: §3 · p. 2
• A rating shift affects outcomes in proportion to the density of ratings at a threshold, so its effect is largest near the acceptance borderline.The framework describes this shift as proportional to γπa.Source: §3 · p. 2

4.1 The ICLR 2026 Corpus

• 18,789 reviewed submissions had ratings, and 13,726 had final decisions; the crawl returned 19,814 submissions, including 18,906 that entered review.Rating analyses use the rated submissions, while acceptance analyses use those with decisions.Source: §4.1 · p. 3
• Each paper’s rating is the unweighted mean of final official review scores on ICLR’s six-level scale from 0 to 10.Final scores are those standing after author response and reviewer discussion.Source: §4.1 · p. 3

4.2 Measuring Author Reputation

• Five reputation proxies comprise institution reputation and four OpenAlex bibliometrics: h-index, citation count, publication count, and i10-index.Institution affiliation comes from OpenReview profiles; bibliometric linkage uses ORCID to resolve authors to OpenAlex records.Source: §4.2 · p. 3
• Institution reputation maps current affiliations to ten tiers, with tier 1 most prestigious and tier 10 least prestigious or unranked.University names are matched to QS World University Rankings 2027; companies receive research-prestige tiers.Source: §4.2 · p. 3
• OpenAlex linkage limits bibliometric coverage: only 2,626 of 5,912 ORCID-bearing first authors resolve, though 88.1% of submissions have at least one ORCID author.Bibliometric measures are available for a minority of first authors but for at least one author on most papers.Source: §4.2 · p. 4

4.3 Labelling Submissions as De Facto Open or Blind

• Submissions are classified by first arXiv version timing: pre-review versions count as de facto open, no version before decisions as blind, and intervening versions are excluded as ambiguous.Four narrower open-window definitions are also tested; Table 2 covers matching and classification for the 13,726 submissions with decisions.Source: Table 2 · p. 3, §4.3 · p. 4

5.1 Quantifying the reputation effect

• Under the institution proxy, open–blind slope gaps were γ = 0.015 (max-author, p = 0.041) and γ = 0.021 (mean-author, p = 0.008).These gaps amount to about 0.06 to 0.19 rating points over the observed reputation range; other proxies show the same direction with weaker separation.Source: §5.1 · pp. 4–5, Figure 1 · p. 5

5.2 Where the Reputation Effect Lands

• 0.025 was the open-paper change in probability at threshold 4.5, versus 0.002 at 2.0 and 0.003 at 7.0 for the institution proxy.The effect peaks around thresholds 3.5–4.5; the blind curve peaks near 3.5 and the open curve near 4.5, while they nearly coincide at the scale extremes. The observational split cannot distinguish reputation from paper quality.Source: §5.2 · p. 5

5.3 Isolating Identity from Content

• 0.5 points was the low-to-high reputation rating rise for claude-opus-5 when the manuscript was held fixed.Four of five AI models showed a rise of 0.2 to 0.6 points; the simulation sampled 600 submissions and rated each under blind and open reputation conditions.Source: §5.3 · p. 6
• Under abstract, abstract-plus-introduction, and full-body inputs, the high–low reputation gap stayed positive at 0.39, 0.32, and 0.32.The reputation effect was therefore insensitive to the amount of the paper shown to reviewers.Source: §5.3 · pp. 7–8
• Across Pconference, Previewer, and Pboth prompts, the high–low reputation gaps stayed positive at 0.21, 0.21, and 0.18.The four reputation conditions retained the same ordering across prompts.Source: §5.3 · p. 8
• Simulated ratings correlated about 0.35 with human ratings in every model and mechanism, but compressed toward the middle of the scale.Human ratings of 2.7 corresponded to simulated ratings between 4.3 and 5.6, while human ratings of 6.4 corresponded to simulated ratings between 5.7 and 6.5.Source: §5.3 · p. 8

5.4 Robustness to Data Construction

• Under all five arXiv timing definitions, the open line climbed more steeply than the blind line.The alternative windows included first versions within one or three months of review start, or more than three or six months before it.Source: §5.4 · p. 8
• Across all four arXiv matching rules, the open profile stayed above the blind profile and the effect peaked in the borderline region.Alternatives to fuzzy title matching with a first-author check used exact titles, omitted the author check, or did both.Source: Figure 6 · p. 7, §5.4 · p. 8

• On ICLR 2026’s complete review record, ratings rose with author reputation in both groups but more steeply under open review, with the difference concentrated at acceptance-deciding borderline ratings.Source: §6 · p. 8
• In a controlled simulation where AI reviewers rated the same manuscripts under different stated author reputations, estimates were consistent with the observational findings.Source: §6 · p. 8
• The findings indicate that anonymity reduces the extent to which visible author standing, rather than content, moves a rating.Source: §6 · p. 8

Bias between open and blind

• The observational rating gap is an upper bound, not a causal estimate, because open-group authors consistently have higher institutional reputation across twelve measure-and-aggregation combinations.The authors therefore treat the observational analysis as descriptive evidence of scale and direction.Source: Limitations · pp. 8–9

Reputation proxies

• Bibliometric reputation proxies are May 2026 snapshots rather than review-time measurements, but their non-decreasing metrics make resulting bias conservative and unable to reverse the reported effects.The proxies include h-index, citation count, and publication count.Source: Limitations · p. 9

AI Reviewer model

• Current AI models may have seen submissions during pretraining, but scoring each paper under multiple author conditions makes such exposure affect all conditions equally.Older-cutoff models were avoided because they usually rate every paper highly and poorly stand in for real reviewers.Source: Limitations · p. 9

Evidence chain

• The evidence chain combines 18,000 submissions, a controlled simulation with manuscripts held fixed, and robustness across five proxies, three aggregation rules, five open-window definitions, and four matching rules.The slope difference is significant under mean-author aggregation and marginally significant under max-author aggregation.Source: Limitations · p. 9

Prompts

• The Pconference and Previewer prompts use the abstract and introduction, with the author line omitted for blind review and otherwise byte-identical prompts.The Previewer prompt adds a reviewer persona and six judgment aspects while leaving the rating scale unchanged.Source: Table 5 · p. 11, Table 6 · p. 11

Reviewer prompt

• The appendix includes the submission-review prompt’s evaluation criteria: presentation, experimental evaluation, writing, clarity, soundness, and strength of the core idea.The prompt addresses a reviewer for ICLR 2026.Source: Appendix · p. 11

Rating scale

• The review prompt asks for an overall rating on the ICLR 2026 scale, ranging from 0 for strong reject to 10 for strong accept.The scale also labels 2 as reject, 4 as borderline reject, 6 as weak accept, and 8 as accept.Source: Appendix · p. 11

Prompt content

• The Pboth prompt’s first nine main-text pages are rendered as images, with headers, page numbers, line numbers, and the author block removed or replaced.Only the content block differs from the abstract-and-introduction version.Source: Figure 8 · p. 12, Figure 9 · p. 13

Simulation figures

• Figures 10 and 11 plot mean simulated against real submission ratings by author-reputation condition across five models, with submissions grouped into seven real-rating strata.Both axes use the ICLR rating scale, and the dotted diagonal marks y = x.Source: Figure 10 · p. 14, Figure 11 · p. 14