PaperlayerView exact revision on arXiv

Source-linked AI summary

A Gold Standard Dataset for the Reviewer Assignment Problem

Ivan Stelmakh, John Wieting, Sarina Xi, Graham Neubig, Nihar B. Shah

arXiv:2303.16750v2cs.IRcs.DLcs.LG

TL;DR

Venue organizers lack empirical or theoretical evidence to guide choices among strategies for combining paper inputs. The authors release a dataset of expertise evaluations by researchers who had read papers and compare algorithms, finding substantial errors and performance differences across representations.

  • Problem

    Venue organizers lack empirical or theoretical evidence to guide choices among strategies for combining paper inputs.

    Source: §2 · p. 4

  • Method

    The authors publicly release expertise evaluations from scientists who had read the papers for their research and use the dataset to compare existing expertise-estimation algorithms.

    Source: §1 · p. 2, §8 · p. 15

  • Results

    All algorithms make 12%-30% mistakes in easier comparisons and 36%-43% in harder ones; Specter2 performs best with titles and abstracts, while full-text TPMS matches it and generic LLMs lag.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings indicate a need to develop improved algorithms for matching reviewers to papers.

    Source: §8 · p. 16

    The dataset is small (477 data points from 58 researchers) and geographically skewed (74% of participants from the US), limiting its representativeness.

    Source: §1 · p. 2, §4.1 · p. 6

Abstract

from arXiv · show

Many peer-review venues are using algorithms to assign submissions to reviewers. The crux of such automated approaches is the notion of the "similarity score" -- a numerical estimate of the expertise of a reviewer in reviewing a paper -- and many algorithms have been proposed to compute these scores. However, these algorithms have not been subjected to a principled comparison, making it difficult for stakeholders to choose the algorithm in an evidence-based manner. The key challenge in comparing existing algorithms and developing better algorithms is the lack of publicly available gold-standard data. We address this challenge by collecting a novel dataset of similarity scores that we release to the research community. Our dataset consists of 477 self-reported expertise scores provided by 58 researchers who evaluated their expertise in reviewing papers they have read previously. Using our dataset, we compare several widely used similarity algorithms and offer key insights. First, all algorithms exhibit significant error, with misranking rates between 12%-30% in easier cases and 36%-43% in harder ones. Second, most specialized algorithms are designed to work with titles and abstracts of papers, and in this regime the SPECTER2 algorithm performs best. Interestingly, classical TF-IDF matches SPECTER2 in accuracy when given access to full submission texts. In contrast, off-the-shelf LLMs lag behind specialized approaches.

1 Introduction

Automated reviewer assignment depends on expertise similarity scores, but there is no public gold-standard data for principled comparison. The paper releases self-reported expertise data and benchmarks existing algorithms, finding substantial errors that motivate better methods.

1 Introduction

  • A similarity score estimates a reviewer’s expertise for a submission and is used to assign reviewers after scores are computed for reviewer-submission pairs.The paper focuses on computing scores from submission text and reviewer profiles, while subject areas and reviewer bids may also augment them.Source: §1 · p. 2
  • Existing algorithms lack publicly available gold-standard data for development and principled comparison, leaving performance differences across flagship conferences poorly understood.ICML, NeurIPS, and ACL rely on different similarity-computation algorithms.Source: §1 · p. 2
  • The authors survey researchers about their expertise reviewing previously read papers and release a dataset for training or evaluating similarity algorithms.The survey includes researchers ranging from graduate students to senior professors.Source: §1 · p. 2
  • 12%-30% mistakes occur on paper pairs with large expertise differences, while algorithms err with probability 36%-43% on pairs of high-expertise papers.These results show substantial error even in easier comparisons, with still higher error on challenging high-expertise pairs.Source: §1 · p. 2
  • The dataset contains 477 data points from 58 researchers, and 74% of participants are from the US.The authors identify both dataset size and geographical skew as limitations.Source: §1 · p. 2

2 Related literature

Prior work evaluates similarity algorithms through proxies or expertise judgments with different limitations. The paper’s dataset uses public, self-reported expertise evaluations to support comparison without relying on an existing algorithm’s assignments.

2 Related literature

  • OpenReview’s authorship-prediction proxy may not reflect expertise estimation for papers written by other researchers.Its score measures how often an author is predicted among a paper’s top reviewers.Source: §2 · p. 3
  • Earlier external judgments can be noisy, while self-reported datasets face trade-offs involving privacy, brief exposure to papers, willingness to review, or algorithm-selected assignments.Mimno & McCallum (2007) and Zhao et al. (2022) use external judgments; Dumais & Nielsen (1992), Rodriguez & Bollen (2008), and Anjum et al. (2019) use self-reports or bids with differing constraints.Source: §2 · p. 3
  • The authors’ dataset is publicly releasable and uses self-evaluations based on researchers’ deep understanding of papers, without bias from an existing similarity algorithm.It is presented as addressing limitations of prior data collection approaches.Source: §2 · p. 3
  • Different conferences combine automated similarity estimates and human corrections in different ways, with little empirical or theoretical evidence to guide those choices.Human corrections may include reviewer bids, while automated estimates come from similarity-computation algorithms.Source: §2 · pp. 3–4
  • Automated allocation is a separate second step after similarity scores are computed, and assignment objectives include cumulative similarity, fairness, envy-freeness, strategic resistance, and diversity.Cumulative similarity may allocate irrelevant reviewers disproportionately to some submissions, motivating work on balanced assignments.Source: §2 · p. 4

3 Data collection pipeline

The authors collect self-evaluations of expertise for papers researchers have read, aiming to meet requirements for informed ratings, public release, and ratings of one’s own expertise. They release participant and paper representations alongside the evaluations.

3 Data collection pipeline

  • The data-collection design seeks informed evaluations of papers participants read, public release without sensitive information, and ratings of one’s own expertise rather than others’ judgments.The authors explain that bids, confidential reviewer-paper assignments, and external judgments each fail to meet one or more of these desiderata.Source: §3 · p. 4
  • Researchers report expertise levels for papers they have read, providing a list of papers and rating or ranking their own expertise for reviewing them.The collection procedure was performed under Institutional Review Board approval.Source: §3 · p. 4
  • Participants recalled 5-10 papers in their broad research area that they read to a reasonable extent in the last year, excluding papers they authored or that were not freely available online.Recruitment used university and company research-department mailing lists and a call for participation on X; 58 participants passed screening requiring at least one paper published in broad computer science.Source: §3 · p. 5
  • Participants were encouraged to include papers across their expertise range, including difficult near-ties, and rate expertise from 1 to 5 in 0.25 increments.The scale ranges from not qualified to review a paper to having the background needed to evaluate all its aspects; participants were advised to avoid ties and consider tricky papers for algorithms.Source: §3 · p. 5
  • The initial release contains responses from 58 researchers, with entries pairing participants’ expertise evaluations and paper representations.Participant records include Semantic Scholar IDs and bibliographies; paper records include titles, abstracts, authors, publication years, and arXiv identifiers, with freely available PDFs linked when available.Source: §3 · pp. 5–6

4 Data exploration

The dataset comprises experienced computer-science researchers, recent papers, and expertise ratings with a range of score values and pairwise differences. The authors note demographic skew that limits representativeness.

4.1 Participants

  • About 85% of participants have or are pursuing PhDs, and their publication profiles average 54 papers with a median of 20.The authors conclude that most contributors are eligible to review for machine-learning and artificial-intelligence conferences.Source: §4.1 · p. 6
  • Most participants are male and affiliated with US-based organizations, and about 40% are affiliated with one university, so the sample may not represent the broader ML and computer-science communities.The authors encourage further contributions to improve dataset representativeness.Source: §4.1 · p. 6

4.2 Papers

  • Participants evaluated 463 unique papers; 99% have freely available PDFs, and approximately 75% were published in or after 2020.All but one paper is listed on Semantic Scholar, and 99% belong to computer science.Source: §4.2 · p. 7, Table 2 · p. 7

4.3 Evaluations of expertise

  • 477 expertise evaluations cover a range of score values: about 56% of reported papers have scores of at least 4, and pairwise differences have a heavy-tailed distribution.The distribution supports testing similarity methods on both large and small expertise differences.Source: Figure 1 · p. 7, §4.3 · p. 8

5 Experimental setup

Section 5 defines a ranking-based loss for comparing predicted expertise orderings with researchers’ self-reports and describes the similarity algorithms evaluated.

5.1 Metric

  • The loss penalizes incorrect paper orderings by the expertise difference, gives predicted ties half that penalty, and normalizes participant losses against worst-case reversed rankings.Lower values indicate better performance.Source: §5.1 · p. 9

5.2 Algorithms

  • Algorithms estimate expertise by comparing submission text with papers in reviewers’ profiles; TPMS uses TF-IDF, while OpenReview includes embedding-based methods.TPMS is implemented using the open-source version by Xu et al. (2019).Source: §5.2 · pp. 9–10
  • OpenReview methods include general-purpose ELMo, scientific-document embeddings in Specter, multifacet Specter+MFR, and task-specialized adapters in Specter2.Specter explores the citation graph when constructing embeddings.Source: §5.2 · p. 10
  • The ACL matcher learns similarity from positive and negative abstract pairs, while off-the-shelf LLMs are prompted to score reviewer-paper pairs.The ACL model is used without additional fine-tuning despite possible distribution shift from computational linguistics to general machine learning.Source: §5.2 · p. 10

6 Results

Across the evaluated algorithms, Specter2 with max pooling performs best under conference-like modeling choices, while full-text TPMS closes its gap. The section also examines representation and profile-size choices and notes uncertainty in some comparisons.

6.1 Comparison of the algorithms

  • All evaluated algorithms outperform the trivial baseline, and the Specter family outperforms ELMo.The authors note that Specter-family methods use domain-specific embeddings, unlike ELMo’s general-purpose embeddings.Source: §6.1 · p. 12
  • Specter2 with max pooling achieves the lowest loss under modeling choices that mimic conference practice.Profiles use up to 20 recent papers, near the mean profile size reported for NeurIPS 2022.Source: §6.1 · pp. 11–12
  • LLM performance varies: o1-mini is on par with Specter2, while Gemini 2 Flash is worst except for the trivial baseline.Source: §6.1 · p. 12
  • Among Specter2 pooling options, max pooling performs best, while 75th-percentile pooling performs nearly as well.Source: §6.1 · p. 12
  • TPMS is competitive with Specter2 and Specter+MFR, and full-text inputs bring TPMS level with Specter2.TPMS also outperforms Claude Sonnet 3.5 and ACL in the reported comparison.Source: §6.1 · pp. 12–13
  • Some algorithm confidence intervals and relative-difference intervals overlap, so larger datasets are needed for finer comparisons.Source: §6.1 · p. 13

6.2 The role of modeling choices

  • The modeling-choice study uses TPMS to vary paper representation and reviewer-profile size, including title, title-plus-abstract, or full text and profiles of 1–20 papers.The study is motivated by accuracy and computation trade-offs for richer text and by the balance between recent and older profile papers.Source: §6.2 · p. 13

Results

Abstracts improve TPMS over titles alone, and full texts add further improvement, bringing its performance close to Specter2. Reviewer-profile loss curves plateau once profiles contain at least eight recent papers.

Results

  • Figure 2 reports TPMS pointwise loss across parameter choices; Table 4 reports confidence intervals for paper representations with 20 profile papers.The figure omits confidence intervals, which are provided in Table 4.Source: Results · p. 13, Figure 2 · p. 14, Table 4 · p. 14
  • 0.07 improvement from adding abstracts and a further 0.03 from full texts show gains over titles alone.With titles, abstracts, and full texts, TPMS performs quite close to Specter2, which cannot handle full texts.Source: Results · pp. 13–14
  • Loss curves plateau once reviewer profiles include 8 or more recent papers, with no further prediction-quality gain from longer profiles.The paper suggests asking reviewers to provide 10 representative papers.Source: Results · p. 14

7 Easy versus hard stratification

The paper evaluates algorithms separately on easy and hard reviewer–paper triplets, finding stronger performance on easy cases and a substantial challenge on hard ones. Full texts improve TPMS accuracy in both groups.

7 Easy versus hard stratification

  • The evaluation covers 261 easy triplets with high-versus-low expertise and 417 hard triplets with differing high-expertise ratings; low-versus-low pairs are excluded.The authors describe easy cases as having a large expertise gap and hard cases as having a small one; distinguishing high but unequal expertise is important for reviewer matching.Source: §7 · pp. 14–15
  • The evaluation compares predicted and expertise-induced orderings using titles and abstracts, with TPMS also tested using full texts.Accuracy is the fraction of triplets correctly resolved; reviewers’ profiles contain 20 papers.Source: §7 · p. 15
  • For easy triplets, most algorithms detect papers for which reviewers have low expertise with probability close to or above 80%; Specter2 reaches nearly 90%, while Gemini 2 Flash has as low as 57% accuracy.In more than 10% of cases, algorithms cannot distinguish papers a reviewer is well-qualified to review from papers for which the reviewer lacks the necessary background.Source: §7 · p. 15
  • 62% accuracy in the title+abstract regime and 64% in the full text regime make TPMS the best-performing algorithm on hard triplets.The Specter family fares slightly worse than TPMS on hard cases, but error bars are too wide for a decisive comparison; all three LLMs are no better than random.Source: §7 · p. 15
  • Full texts improve TPMS performance on both easy and hard triplets.The paper describes full texts as instrumental to TPMS performance in both groups.Source: §7 · p. 15

8 Discussion

The paper releases a public dataset of expertise judgments from scientists who have read the papers and uses it to compare expertise-estimation algorithms. The discussion highlights substantial algorithm errors and boundaries on interpreting the dataset and results.

8 Discussion

  • The released dataset contains evaluations by scientists who actually read the papers, supporting training and evaluation of expertise-estimation algorithms.The authors present it as a resource for evidence-based algorithm choice by venue organizers.Source: §8 · pp. 15–16
  • Specter2 leads on easy triplets, while TPMS with full text has the highest hard-triplet accuracy; that TPMS result uses a slightly different dataset.The table caption states the winner for each triplet type but does not provide the accuracy values here.Source: §8 · p. 15, Table 5 · p. 16
  • The authors characterize current algorithms as making many errors and call for improved reviewer-paper matching methods.They connect this need to the growing number of submissions.Source: §8 · p. 16
  • The data concern older papers, so incoming citations may provide future algorithms with information unavailable for recent conference submissions.The evaluated algorithms do not use citation relationships.Source: §8 · p. 16
  • Automatically constructed Semantic Scholar profiles may be inaccurate, and survey-interface design may influence the data and evaluations.The authors also flag possible sampling bias and limited representativeness of the participant population.Source: §8 · p. 16

A More details on the survey used for data collection

The survey gathered researchers’ expertise judgments about papers they had read recently. Its instructions sought diverse and challenging examples and specified how scores and participant information would be handled.

A More details on the survey used for data collection

  • Participants were asked to recall papers in their broad research area that they had read to a reasonable extent in the last year and rate their reviewing expertise.They were asked to report five to ten papers, with at least five preferred.Source: A More details on the survey used for… · pp. 20–21
  • The instructions encouraged participants to choose less famous papers and include pairs that were either well-separated or close in their expertise judgments.They also asked participants to avoid ties and include examples that might be tricky for similarity algorithms.Source: A More details on the survey used for… · p. 21
  • The survey stated that submitted data would be released publicly without email addresses and without anonymization.The participant-facing goal was to collect expertise judgments to help improve similarity algorithms.Source: A More details on the survey used for… · pp. 20–21
  • Expertise ratings ranged from 1.0, indicating not qualified to review, to 5.0, indicating background to evaluate all aspects.Intermediate scale descriptions distinguish partial expertise from adequate or broad review capability.Source: A More details on the survey used for… · p. 21

B More details on the ACL algorithm

The ACL algorithm trains an encoder on abstract segments using positive and hard-negative pairs, then scores a reviewer-submission pair using similarities to profile abstracts.

B More details on the ACL algorithm

  • Training uses 45,309 ACL Anthology abstracts, each split into two random disjoint segments that form a positive pair.The segments are allocated into two equally sized groups for constructing examples.Source: B More details on the ACL algorithm · p. 22
  • For each segment, a negative example pairs it with a segment from another abstract selected to maximize cosine similarity.The negative is therefore chosen to be similar under the encoder rather than sampled arbitrarily.Source: B More details on the ACL algorithm · p. 22
  • The training objective is a max-margin contrastive-learning objective, and the encoder mean-pools 1,024-dimensional SentencePiece embeddings.The reported training setup uses a batch size of 64, 20 epochs, and margin δ = 0.4.Source: B More details on the ACL algorithm · p. 22
  • At inference, the similarity score combines the top-three cosine similarities between a submission abstract and abstracts in the reviewer profile.If the profile has fewer than three abstracts, the missing similarity scores are set to zero.Source: B More details on the ACL algorithm · p. 22

C Additional metrics

The appendix defines additional ranking metrics and reports comparisons at several top-k cutoffs. It also specifies the prompt used to elicit reviewer-expertise scores from LLMs.

C Additional metrics

  • Additional evaluations use WKTD@k, MRR@k, NDCG@k, and Precision@k, with MRR, NDCG, and Precision averaged across participants.WKTD is normalized per participant; lower WKTD is better, whereas higher values are better for the other metrics.Source: C Additional metrics · pp. 22–23
  • Tables 6–9 compare algorithms at top 1, 3, 5, and 7 using profiles of 20 recent papers and paper titles and abstracts.The captions state that best values are bolded for each column.Source: Table 6 · p. 23, Table 7 · p. 24, Table 8 · p. 24

D LLM Prompt

  • The LLM prompt asks for a 0–100 expertise-match score based on reviewer-paper titles and abstracts, plus a brief explanation.It instructs the model not to invent information absent from the papers.Source: D LLM Prompt · pp. 25–26

Related papers

Loading 2303.16750v2…