Source-linked AI summary
OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Models
Hao Sun, Yunyi Shen, Mihaela van der Schaar
TL;DR
Many datasets used for LLM training and evaluation are limited, synthetic, or static, while systematic exploration of peer-review datasets and methods remains missing. This position paper proposes broader use of OpenReview’s evolving expert deliberations; it identifies applications in review support, open-ended evaluation and post-training, and alignment research, alongside community-led benchmarks and responsible stewardship.
Problem
Datasets for LLM training and evaluation often lack the coverage and structure to capture authentic human judgment, and systematic exploration of peer-review datasets and methods remains missing.
Source: §1 · p. 2Method
The paper examines OpenReview’s records of submissions, reviews, rebuttals, meta-reviews, and decisions as a resource for review support, expert-generated benchmarks, and alignment research.
Source: §1 · pp. 2–3, §5.2 · p. 11Results
The paper argues that OpenReview can support reviewer assistance, open-ended task benchmarks and post-training, and research on reasoning and alignment grounded in expert deliberation.
Source: §1 · pp. 2–3, §5.2 · p. 11, §7 · p. 12Takeaways & Limitations
The authors propose community-driven benchmark development and responsible data stewardship to help realize OpenReview’s potential.
Source: §1 · p. 3Peer-review data is subjective: reviewers may disagree about paper quality, assign divergent scores, or emphasize different aspects.
Source: §7 · p. 12
Abstract
from arXiv · showhide
In the era of large language models (LLMs), high-quality, domain-rich, and continuously evolving datasets capturing expert-level knowledge, core human values, and reasoning are increasingly valuable. This position paper argues that OpenReview -- the continually evolving repository of research papers, peer reviews, author rebuttals, meta-reviews, and decision outcomes -- should be leveraged more broadly as a core community asset for advancing research in the era of LLMs. We highlight three promising areas in which OpenReview can uniquely contribute: enhancing the quality, scalability, and accountability of peer review processes; enabling meaningful, open-ended benchmarks rooted in genuine expert deliberation; and supporting alignment research through real-world interactions reflecting expert assessment, intentions, and scientific values. To better realize these opportunities, we suggest the community collaboratively explore standardized benchmarks and usage guidelines around OpenReview, inviting broader dialogue on responsible data use, ethical considerations, and collective stewardship.
1 Introduction
LLMs increasingly depend on human-centered feedback, yet existing datasets often miss the complexity and disagreement of authentic judgment, while rising submission volumes strain peer review. This paper argues that OpenReview’s evolving record of expert deliberation can support review assistance, open-ended benchmarks, alignment research, and community-guided stewardship.
1 Introduction
- OpenReview preserves continually expanding, structured records of submissions, expert reviews, rebuttals, meta-reviews, and decisions across multiple rounds.These records involve diverse expert perspectives and are grounded in real-world scientific deliberation.Source: §1 · p. 2
- Models trained on OpenReview could help reviewers draft feedback, calibrate scores, identify argumentative gaps, summarize responses, check code, and detect unhelpful language.The paper presents these tools as potential support for more consistent, fair, and informative evaluation amid rising submissions and reviewer fatigue.Source: §1 · pp. 2–3
- OpenReview’s expert-curated evaluations can ground benchmarks for writing, research evaluation, persuasion, and summarization, supporting open-ended LLM and agentic post-training and evaluation.Open-ended tasks are difficult to train and evaluate because they lack scalable, high-quality human feedback.Source: §1 · p. 3
- OpenReview’s evidence-based debate, disagreement, and revision can support score-justification and decision-prediction tasks that assess models’ reasoning about expert judgments.Its rebuttals, conflicting views, and negotiated outcomes also offer a setting to study value pluralism and debate-style alignment in the wild [Khan et al., 2024, Liang et al., 2023a, Du et al., 2023, Sorensen et al., 2024, Feng et al., 2024].Source: §1 · p. 3
- The paper proposes community-driven benchmark development and responsible data stewardship to help realize OpenReview’s potential.It also aims to spark dialogue about challenges and risks in leveraging the platform as a community asset.Source: §1 · p. 3
2 The State of OpenReview Now: Scale, Opportunity, and Emerging Risks
OpenReview is a growing archive of expert deliberation, but its rapid expansion raises concerns about review quality and the persistence of poor practices. A simplified model suggests that early intervention can help prevent low-quality reviews from becoming entrenched.
2.1 The Scale and Structure of Conference Data on OpenReview
- From fewer than 500 to over 11,600 annual submissions, ICLR grew from 2017 to 2025, while estimated reviewers rose from under 1,000 to more than 18,300.Authors increased from about 1,500 to 38,500, and each submission typically receives three or more expert reviews.Source: §2.1 · p. 4, Figure 2 · p. 4
2.2 A Rapidly Growing Community Asset for Learning
- Over 36,000 ICLR interaction threads from 2017–2025 span accepted and rejected submissions, offering examples of scientific exploration, disagreement, clarification, and consensus.The archive is continuously evolving, with new topics, papers, and debates each year.Source: §2.2 · p. 4
2.3 Quality Under Pressure — The Compounding Risk of Rapid Growth
- The model suggests that early action against preferences for low-quality reviews can prevent them from becoming norms that are harder to filter out as the field grows.Reversing a trend later may require a longer period of selection; stopping it can still be useful. The model is extremely simplified and treats review quality as binary despite its dependence on many factors.Source: §2.3 · pp. 5–6, Figure 3 · p. 6
- Protecting OpenReview’s quality requires better reviewer recruitment and training, alongside scalable machine-learning methods for auditing and mitigating quality drift.The paper also recommends taking corrective action before problematic practices become institutionalized.Source: §2.3 · p. 6
3 Assisting and Protecting the Peer Review with OpenReview
The paper reviews current LLM-assisted review efforts and proposes using OpenReview data to support review quality, consistency, and dialogue. It identifies remaining challenges in fairness, calibration, and faithful synthesis of disagreement.
3.1 Existing LLM-Assisted Peer Review in Conferences
- Conferences have deployed LLM tools for ethical and methodological compliance and review feedback, while publishers pilot content-checking and review-drafting tools; AAAI 2026 will experiment with LLM-generated reviews and discussion summaries.Source: §3.1 · p. 7
- OpenReview offers a data-driven foundation for peer-review research compared with systems that use limited, hand-curated inputs.The paper proposes consistent, calibrated reviewer feedback using comparative context and arguments drawn from reviewer cohorts.Source: §3.1 · p. 7, §3.2 · p. 8
3.2 Practices and Opportunities for Data-Driven Support with OpenReview
- Review-generation systems can produce more calibrated, grounded critiques, but most mimic human reviews without deeper integration with reviewing guidelines or accountability structures.The paper proposes task-specific alignment through structured prompting, rubric conditioning, or review-focused alignment objectives.Source: §3.2 · p. 7
- Automated review-quality work targets vague, biased, or uninformative reviews, but fairness, argument soundness, and reviewer calibration remain difficult to evaluate automatically.The paper suggests inverse analysis to detect deviation from expected standards, including overconfidence, inconsistency, or subjective bias.Source: §3.2 · pp. 7–8
- LLMs may mediate rebuttal dialogue by helping authors address reviewer concerns and reviewers assess whether feedback has been answered.Existing datasets support tasks such as rebuttal generation, stance classification, and discourse-structure prediction.Source: §3.2 · p. 8
- Current approaches to score consistency and calibration often lack interpretability or real-time applicability, with limited support for reviewers to understand biases or recalibrate using peer context.For meta-reviews, systems must integrate conflicting perspectives and represent area-chair judgment, while current general-purpose LLMs may miss disagreement nuance or diverge from reviewer consensus.Source: §3.2 · p. 8
4 OpenReview for Open-Ended Task Evaluation and Post-Training
Open-ended research tasks are difficult to supervise and evaluate because quality depends on judgment and context, while common benchmarks offer limited insight into actual research standards. The paper identifies OpenReview papers and review discussions as two potential supervision streams for training and evaluating models.
4.1 Challenges for Open-Ended LLM and Agentic Tasks
- Evaluating agentic research tasks is difficult because judgments such as whether a literature review is good or a research idea promising lack scalable, high-quality supervision.Many current benchmarks are synthetic, short-form, or not scalable, and reveal little about performance under real research-community standards.Source: §4.1 · p. 9
4.2 Two Potential Supervision Streams from OpenReview
- OpenReview papers provide real-world demonstrations of open-ended problem-solving, including research framing, literature positioning, experiments, and claim justification.These examples may support training for multi-stage planning, tool use, fact retrieval, and hypothesis revision.Source: §4.2 · p. 9
- Reviews, meta-reviews, and author responses provide records of expert feedback, decision rationales, and responses to critique for training or evaluating models on research assessment.The paper identifies automated meta-reviews, rebuttal critiques, and scoring models aligned with human preferences as possible future uses.Source: §4.2 · pp. 9–10
5 OpenReview as High-Quality Dataset for Alignment and Reasoning
The paper argues that common alignment and reasoning supervision often lacks the complexity of expert disagreement and deliberation. OpenReview’s multi-round reviews and rebuttals may support research on reasoning from justifications and alignment with context-sensitive, evidence-based assessment.
5.1 Challenges for Alignment and Reasoning Supervision
- Many alignment approaches rely on crowd-sourced annotations, synthetic preferences, or binary votes, which do not capture the depth and disagreement of expert deliberation.Many reasoning benchmarks also emphasize verifiable answers and narrow or synthetic tasks rather than ambiguity, disagreement, or multi-agent deliberation.Source: §5.1 · p. 10
5.2 Opportunities with the OpenReview Dataset
- OpenReview records multi-round, multi-agent dialogue in which participants critique reasoning and negotiate consensus around guidelines and verifiable facts.The paper contrasts these interactions with alignment datasets it characterizes as largely subjective, static, and one-shot.Source: §5.2 · p. 11
- Review rationales and evolving exchanges among authors and reviewers may support studying how assessments follow from arguments and context over multiple interaction rounds.The paper describes this as learning the logic behind review scores from rationales, experimental design, observed outcomes, and stated conclusions.Source: §5.2 · p. 11
- Reviewers’ scores and comments reflect judgments against criteria such as novelty, technical soundness, and significance, and preferences may shift after rebuttals and clarifications.The paper presents these signals as a basis for modeling alignment as contextual and adaptive rather than mere agreement with users.Source: §5.2 · p. 11
6 A Call to Create Standardized Benchmarks Based on OpenReview
Realizing OpenReview’s research potential requires shared benchmarks and responsible community stewardship.
6 A Call to Create Standardized Benchmarks Based on OpenReview
- OpenReview is underused because well-defined tasks and shared evaluation pipelines are lacking; the authors propose benchmarks for review quality, rebuttal generation, argument grounding, and meta-review summarization.They also call for deploying developed methods to intervene in the peer-review system and improve its quality.Source: §6 · p. 11
- Standardizing reviewer score prediction, open-ended task evaluation, post-training, alignment, and reasoning enhancement could support long-term research.The authors say delays in investigating these broader tasks are less likely to compromise OpenReview’s quality or sustainability.Source: §6 · pp. 11–12
- Benchmark development should involve researchers, conference organizers, and practitioners while addressing privacy, anonymization, and representational bias.Areas with more abundant data may dominate training signals and skew evaluation models’ learned priorities.Source: §6 · p. 12
- OpenReview’s continued value as a shared academic asset depends on proactive, collective community stewardship.Source: §6 · p. 12
7 Alternative Views
The authors argue that peer review’s deliberative role should not be reduced to automated scoring, and that disagreement can inform alignment research. They also contend that research tasks may offer reasoning patterns that generalize beyond science.
7 Alternative Views
- Peer review shapes scientific values and standards, so human reviewers must remain responsible for interpreting and controlling LLM tools.The authors warn that overreliance on automation risks eroding peer review’s collaborative and interpretive nature [Ma et al., 2024, D’Arcy et al., 2024].Source: §7 · p. 12
- Reviewer disagreement can be modeled as a source of evidence about conflicts and reasoning, rather than treated as a demand for deterministic consensus.The authors frame this approach as consistent with recent alignment literature [Liang et al., 2023a, Sorensen et al., 2024].Source: §7 · p. 12
- The authors argue that research tasks exercise complex, structured reasoning that may generalize across domains, despite their scientific focus.They cite DeepResearch-type products as explicitly aiming to generalize research workflows into agentic LLM behaviors [OpenAI, 2025].Source: §7 · p. 12