Source-linked AI summary
Reviewer assignment problem: A scoping review
Jelena Jovanovic, Ebrahim Bagheri
TL;DR
Finding suitable reviewers is increasingly difficult, yet a recent systematic synthesis of computational approaches to reviewer assignment was missing. This scoping review searched three databases and synthesized 26 studies; it found varied assignment methods and criteria, alongside diverse evaluation measures and limited public replication materials.
Problem
Finding adequate reviewers is increasingly difficult as scientific-paper production and scholars’ workload increase, while a recent systematic synthesis of RAP research was missing.
Source: §1 · pp. 1–2Method
The authors conducted a scoping review of recent RAP literature from Scopus, Google Scholar, and DBLP, retaining 26 studies for synthesis across five research aspects.
Source: §1 · p. 2Results
The review found varied assignment methods and selection criteria, many custom evaluation measures, and only two studies with publicly available full code and evaluation datasets.
Source: §6.1 · pp. 22–24Takeaways & Limitations
The synthesis provides an overview of recent RAP approaches, criteria, modelling, matching methods, and evaluation to inform further research.
Source: §6 · p. 22Search terminology may have missed studies that used different terms for reviewer assignment, although the authors considered this unlikely after snowballing.
Source: §6.2 · p. 24
Abstract
from arXiv · showhide
Peer review is an integral component of scientific research. The quality of peer review, and consequently the published research, depends to a large extent on the ability to recruit adequate reviewers for submitted papers. However, finding such reviewers is an increasingly difficult task due to several factors, such as the continuous increase both in the production of scientific papers and the workload of scholars. To mitigate these challenges, solutions for automated association of papers with "well matching" reviewers - the task often referred to as reviewer assignment problem (RAP) - have been the subject of research for thirty years now. Even though numerous solutions have been suggested, to our knowledge, a recent systematic synthesis of the RAP-related literature is missing. To fill this gap and support further RAP-related research, in this paper, we present a scoping review of computational approaches for addressing RAP. Following the latest methodological guidance for scoping reviews, we have collected recent literature on RAP from three databases (Scopus, Google Scholar, DBLP) and, after applying the eligibility criteria, retained 26 studies for extracting and synthesising data on several aspects of RAP research including: i) the overall framing of and approach to RAP; ii) the criteria for reviewer selection; iii) the modelling of candidate reviewers and submissions; iv) the computational methods for matching reviewers and submissions; and v) the methods for evaluating the performance of the proposed solutions. The paper summarises and discusses the findings for each of the aforementioned aspects of RAP research and suggests future research directions.
1 INTRODUCTION
Finding suitable, willing reviewers is increasingly difficult as scientific output and demand for peer review rise, while acceptance and completion rates decline. The review addresses the lack of a recent systematic synthesis by examining computational RAP research from 2016 to September 2021.
1 INTRODUCTION
- Reviewer recruitment is difficult and worsening as scientific-paper production and demand for peer review increase, while reviewer acceptance and completion rates decline.Requests sent to a small subset of potential reviewers also limit the pool and introduce selection bias.Source: §1 · p. 1
- A recent systematic synthesis was missing: the latest review known to the authors dated to 2010 and was not systematic.The authors identify this gap amid progress in technologies and computational methods relevant to RAP.Source: §1 · p. 2
- The scoping review covers RAP literature published from January 2016 through September 2021, a period chosen to roughly coincide with significant NLP developments.The authors note NLP’s role in analyzing submission content and candidate reviewers’ publications.Source: §1 · p. 2
- Using scoping-review guidance, the review examines RAP framing, reviewer-selection criteria, reviewer and submission modelling, matching methods, and evaluation.It aims to synthesize recent research and identify challenges ahead.Source: §1 · p. 2
2 BACKGROUND
RAP spans retrieval-based and optimization-based approaches, with different methods suited to individual-paper recommendations or batch assignment. Methods consider expertise alongside criteria such as conflicts, availability, coverage, and reviewer interests.
2 BACKGROUND
- Finding reviewers requires expertise as well as other criteria, including conflict-of-interest absence and availability; voluntary reviewing also makes reviewer interests relevant to acceptance.Submissions may need reviewers for one paper or batches of papers, and rapidly expanding fields add pressure to find expertise.Source: §2 · p. 2
- Paper bidding and keyword matching are widespread but have shortcomings, while advanced automated and semi-automated methods seek more efficient, faster, and less biased assignments.Bidding participation and completion may be limited, while keyword matching struggles with ambiguous and homonymous topics; RAP refers to automated reviewer assignment to submissions.Source: §2 · pp. 2–3
- RAP methods broadly divide into information-retrieval approaches that rank candidates for individual submissions and optimization approaches often used for batch assignment.Retrieval methods build text-based profiles and rank candidate matches; optimization methods maximize matching across submissions subject to constraints such as conflicts, workload limits, and required reviewer counts.Source: §2 · p. 3
- Beyond topical expertise, studies have considered authority, diversity, topic coverage, availability, reviewer interests, and conflicts of interest.Examples include topic-diverse ranking, reviewer panels covering multiple paper topics, interest indicators, and weighting recent publications; conflicts may be supplied or inferred from relationships.Source: §2 · p. 3
- Automated assignment and recommendation are available in commercial editorial and conference systems, yet editors and program chairs still struggle to find enough competent, willing reviewers.The review aims to clarify current solutions, their computational methods, and their performance.Source: §2 · p. 4
3 RESEARCH OBJECTIVES AND QUESTIONS
The review asks which computational approaches have been proposed and evaluated for RAP, then organizes that inquiry into five specific research questions.
3 RESEARCH OBJECTIVES AND QUESTIONS
- The primary question asks what computational approaches have been proposed and evaluated for solving RAP.The five specific questions address RAP framing and approaches, selection criteria, reviewer and submission representations, assignment methods, and evaluation datasets, benchmarks, and measures.Source: §3 · p. 4
- The review’s objective is to provide comprehensive insight into recent computational research on automated reviewer assignment.Its questions cover both how solutions are constructed and how their performance has been evaluated.Source: §3 · p. 4
4 METHODS
The review searched three academic databases, applied eligibility criteria, and charted information across RAP framing, criteria, modelling, computation, and evaluation. It retained 26 articles after screening and backward snowballing.
4.1 Study Selection Procedure
- The review searched Scopus, Google Scholar, and DBLP using Boolean combinations of terms selected to capture computational RAP studies.Eligibility criteria covered the review concept, peer-review context, and evidence-source features such as date, publication type, language, completeness, and evaluation quality.Source: §4.1 · pp. 4–5
- The review excluded older or less complete method reports, toy-example evaluations, and studies without baseline comparisons.The criteria also excluded work outside the defined RAP concept or peer-review context, including expert finding and some specialized review procedures.Source: §4.1 · p. 5
4.2 Results Synthesis Procedure
- Data charting covered reviewer and submission data, RAP framing, assignment criteria, computational methods, and evaluation.Computational methods were charted by reviewer/submission representation, matching estimation, and reviewer assignment or recommendation.Source: §4.2 · pp. 5–6
5 RESULTS
The review synthesized 26 studies on how the reviewer assignment problem is framed, how assignment criteria and computational representations are used, and how solutions are evaluated. It covers topic-based modeling, a range of assignment methods, and evaluation practices that included approach-specific measures, traditional metrics, and qualitative assessment.
5.1 The Selection Evidence Sources
- The search yielded 1,808 records, and after screening, deduplication, abstract review, and backward snowballing, 26 articles were selected for full review.Two reviewers screened independently and resolved disagreements through discussion.Source: §5.1 · pp. 6–7
5.2 Results Synthesis
- Extracted data were organized by research question in supplementary tables SDE_Q1-Q4 and SDE_Q5, with an overview of synthesized findings in Figure 2.SDE_Q1-Q4 covered the first four groups of charting variables, while SDE_Q5 covered method evaluation.Source: §5.2 · pp. 7–8, Figure 2 · p. 9
5.2.1 RQ1: The overall approach to RAP
- Optimization under typical constraints was the largest RAP approach group (10/26 or 38%), followed by recommending top N reviewers (7/26 or 27%).Optimization commonly maximized matching scores while considering topical relevance, conflict of interest, reviewer workload, and reviewers per paper; top-N methods typically ranked candidates by custom, often topic-based relevance.Source: §5.2.1 · p. 10
- The remaining approaches were idiosyncratic optimization (5 / 26 or 19%) and custom algorithms (4 / 26 or 15%).Examples included probabilistic assignment, equilibrium multi-job optimization, hybrid rule-and-optimization methods, classification, and iterative panel assignment for topic coverage.Source: §5.2.1 · p. 10
5.2.2 RQ2: Criteria for reviewer assignment / recommendation
- Topical relevance was the only criterion shared by a large number of studies (18/26, 69%), estimating reviewer competence for a submission’s topics.Studies computed it from their chosen topic-based representations of candidate reviewers and submissions.Source: §5.2.2 · p. 12
- Other recurring criteria included conflict of interest, fairness, and reviewer authority, with fairness requiring equal treatment and the required number of distinct reviewers per submission.Conflict of interest could be assumed available, inferred from academic networks, or formalized as rules; authority measures included h-index, citation counts, PhDs supervised, and publications.Source: §5.2.2 · pp. 12–14
- Some studies assumed submission-reviewer matching scores were already available, while other one-off criteria included interest trends, review quality, diversity, and topical coverage.The assumed scores were used alongside constraints such as conflict of interest or reviewer workload.Source: §5.2.2 · p. 14
5.2.3 RQ3: Computational approaches to reviewer and submission modelling
- The review identified seven groups of computational representations for candidate reviewers and submissions, including topic distributions, TF-IDF vectors, and custom feature sets.Custom features could represent reviewer topical authority, research accomplishments, general research authority, and availability, alongside a submission’s covered conference topics.Source: §5.2.3 · pp. 14–15
- Topic-based models dominated (12/26, 46%), with LDA among the common approaches; term-based, custom-feature, and full-text models were alternatives.LDA-based profiles represented reviewers and submissions as topic vectors or sequences. Term-based approaches included TF-IDF weighting, while other studies used custom features or full-text indexes.Source: §5.2.3 · p. 16
- Only two studies each used word-embedding and semantic-rich deep-learning representations; the review found state-of-the-art NLP underrepresented in RAP research.Representations included Word2Vec-based reviewer and submission profiles, HIEPAR, and sentence-pair models combining embeddings with network-based learning.Source: §5.2.3 · p. 17
- Academic networks modeled researcher relationships for COI detection or fairness, using weighted shortest paths for COI and social influence measures for fair assignment.The networks encoded scholarly relationships such as collaboration; COI was flagged when shortest paths fell below a threshold, while social influence used inverse distance and local connectivity.Source: §5.2.3 · p. 17
5.2.4 RQ4: Computational methods for assigning / recommending reviewers to submissions
- Ranking approaches numbered 7 (27%) and used custom scores; six idiosyncratic algorithms included Word Mover Distance classification and two-phase selection, while one hybrid combined rules with optimization.The hybrid first applied knowledge-based rules to remove potential COI, then optimized discipline-based matching under RAP constraints.Source: §5.2.4 · pp. 17–18
- Optimization was the largest assignment-method group (12 studies, 46%), spanning greedy algorithms, linear programming, and novel or adapted optimizers [30, 34, 53, 13, 9, 21, 11, 17, 22, 52, 36, 51].Guo et al. combined linear programming and graph-based search to ensure optimal k-loop-free assignments.Source: §5.2.4 · p. 18
5.2.5RQ5: Evaluation of RAP solutions
- True benchmarking datasets were unavailable, and only 6 of 26 studies used either of the two existing public datasets; most studies built datasets they did not release.The NIPS dataset included 148 papers and 364 reviewers with relevance judgments, while SIGIR included 73 papers and 189 reviewers annotated with ground-truth topics. Three studies contributed new public datasets; eight other studies created private ground-truth datasets, while ten used private datasets without ground truth.Source: 5.2.5RQ5: Evaluation of RAP solutions · pp. 20–21
- Baseline choices varied: 14 studies used prior RAP methods, 6 used reviewer-submission similarity measures, and 7 used ground truth as an assessment method.Other baselines included language or probabilistic models, text representation and classification methods, adapted optimization algorithms, and random assignment.Source: 5.2.5RQ5: Evaluation of RAP solutions · p. 21
- 19 of 26 studies relied exclusively on RAP-tailored evaluation measures, while 7 used traditional information-retrieval or recommender-system measures, often alongside custom measures.For example, Pradhan and colleagues combined Precision@k and nDCG with four RAP-specific measures.Source: 5.2.5RQ5: Evaluation of RAP solutions · p. 21
- Panel-assignment studies often paired topical coverage with diversity; Mirzaei et al. combined Average confidence and Paper topical coverage, while Yin et al. used Authority, Diversity, Coverage, and F-score.Mirzaei et al.'s measures were originally proposed in Karimzadehgan et al.; Yin et al.'s F-score was the harmonic mean of Diversity and Coverage.Source: 5.2.5RQ5: Evaluation of RAP solutions · pp. 21–22
- A small number of studies supplemented evaluation with manual qualitative analysis of assignments or reviewers’ perceptions.Source: 5.2.5RQ5: Evaluation of RAP solutions · p. 22
6 DISCUSSION
The review finds substantial diversity in how RAP is framed, how reviewers and submissions are represented, and how assignments are made and evaluated. It identifies limitations in the evidence base and proposes directions for future RAP research.
6.1 Summary of Evidence
- RAP approaches were heterogeneous, but fell into four broad groups: top-N recommendation, constrained objective optimization, idiosyncratic optimization, and custom algorithms.The review also notes fairness-focused variants and other distinctive methods within these groups.Source: §6.1 · p. 22
- Topical relevance dominated reviewer-selection criteria; conflict of interest and authority appeared less often, while fairness, review quality, interests, and availability received limited attention.Availability was more often addressed in conference-like settings through reviewer-workload constraints than in individual-reviewer assignment studies.Source: §6.1 · pp. 22–23
- Most studies modelled expertise and submission content with traditional NLP, while only a small subset used embeddings or deep neural representations.A handful also modelled scholarly relationships for conflict-of-interest detection, fairness, or reviewer-panel diversity.Source: §6.1 · p. 23
- Assignment methods grouped into ranking, optimization, and idiosyncratic algorithms, with custom or customized optimization methods forming the largest group.The review also found novel methods in all three groups.Source: §6.1 · p. 23
- Evaluation lacked shared infrastructure: most studies used self-curated data, no true benchmark existed, available RAP datasets were over 15 years old, and only two studies released code and datasets.One additional study provided detailed pseudocode in an appendix.Source: §6.1 · pp. 23–24
6.2 Limitations
- The review may have missed studies using different search terminology or published in other languages, and it excluded one inaccessible article.The authors note that 13 of the 26 selected studies were by Chinese authors, suggesting relevant Chinese-language research may exist.Source: §6.2 · p. 24
6.3 Recommendations for future research
- Future work could test deep-learning recommendation methods, richer language models for reviewer and submission representations, and explanations for increasingly complex assignment algorithms.The review notes that no examined recommendation-framed study explored state-of-the-art recommendation algorithms and that the reviewed decision-support systems lacked explanation facilities.Source: §6.3 · p. 24
- The authors call for public benchmarks with defined procedures, datasets, and measures, plus open code and data to support replication and benchmarking.They describe benchmark creation as challenging because RAP has been framed in varied ways.Source: §6.3 · p. 25