RelevAI-Reviewer: A Benchmark on AI Reviewers for Survey Paper Relevance

1 Introduction

• Human peer review can involve inconsistent judgments, potential bias or conflicts of interest, and substantial time commitments.The study focuses on relevance, one of five identified criteria for evaluating AI reviewers.Source: §1 · p. 1, §1.1 · p. 2

1.1 Problem Formulation

• The task predicts one of four relevance classes for a prompt–paper pair and can rank four candidate survey papers to identify the most relevant.The authors propose training classifiers rather than relying only on cosine similarity or prompt engineering, and make the challenge public.Source: §1.1 · p. 2, §1 · p. 2

1.2 Related Work

• Prior work explored review-quality assessment, plagiarism detection, clinical-paper relevance, and reviewer assignment; cosine similarity provides a relevance baseline.The paper also notes continuing fairness and reliability challenges in integrating AI into peer-review workflows.Source: §1.2 · pp. 2–3

2 Creation of the RelevAI-Reviewer Dataset

• The dataset spans academic fields and represents each paper using its title, abstract, and a synthetic Related Work section.The paper lists Chemistry, Mathematics, and Social Sciences as examples of fields represented.Source: §2 · p. 3, §2.1 · p. 3

2.1 Data Collection and Categorization

• Approximately 30,000 collected papers were refined to 25,164 entries and assigned four relevance levels based on their relation to a reverse-engineered prompt.The highest-relevance papers generate the prompt; the other categories draw on their citations, same-field papers, or papers from different fields.Source: §2.1 · p. 3

2.2 Reverse Prompt Engineering

• Prompts were reverse-engineered from the most-relevant papers’ titles and abstracts using gpt-3.5-turbo at temperature 0.1.The instructions asked for a prompt beginning with “Write a systematic survey or overview about”.Source: §2.2 · p. 3

2.3 Artificial Related Work Section Generation and Quality Control

• When source Related Work sections were unavailable or inadequately parsed, ChatGPT generated artificial sections from the title and abstract of three cited papers; availability and citation-count checks filtered entries.The resulting dataset retained 25,164 instances, each with a prompt and four categorized papers.Source: §2.3 · p. 4

2.4 Final Dataset

• The 25,164 instances contain one prompt and four categorized papers; splitting them into prompt–paper pairs yields 100,656 supervised data points.Similarity distributions differ across relevance categories, with the most-relevant papers showing the highest mean and little overlap with the second-most-relevant category.Source: §2.4 · p. 4, Figure 1 · p. 4

3 RelevAI-Reviewer Methodology

• The methodology trains models on the RelevAI-Reviewer dataset to assess candidate-paper relevance to a generated call-for-paper prompt.The section describes label encoding, evaluation metrics, and classification baselines.Source: §3 · pp. 4–5

3.1 Label Encoding

• One-hot encoding predicts each paper’s class independently, while thermometer encoding combines four papers’ output vectors and ranks their sums.Thermometer labels encode class distinctions across three bits; the resulting sums assign ranks from 3 for highest to 0 for lowest.Source: §3.1 · p. 5, Table 1 · p. 5

3.2 Evaluation

• Evaluation reports mean and standard error for Kendall’s Tau and per-class F1-score.Kendall’s Tau measures ranking agreement using concordant and discordant pairs; F1 is the harmonic mean of precision and recall.Source: §3.2 · p. 5

3.3 Classification

• The classification experiments compare traditional machine-learning models with the more expensive BERT model.The excerpt specifies the comparison but does not report its results.Source: §3.3 · p. 5

3.3.1 Sentence Transform + Scikit-learn Classifiers

• The scikit-learn classifiers use 384-dimensional paraphrase-MiniLM-L6-v2 embeddings to represent prompt and paper text.The authors chose sentence embeddings to support the contextual understanding needed for relevance assessment.Source: §3.3.1 · p. 6

3.3.2 BERT

• The end-to-end sequence classifier uses the bert-based-uncase model, with 12 layers, hidden size 768, and 12 self-attention heads.The model has 110 million parameters and is used for tokenization and training.Source: §3.3.2 · p. 6

4 Experimental Results

• The experiments used an 80%–20% train-test split and varied training-subset size while holding the test sample size constant.The study describes these experiments as assessing data efficiency and the relation between performance potential and dataset size.Source: §4 · p. 6

4.1 Scikit-Learn Classifiers

• The scikit-learn models used prompt–candidate-pair sentence-transformer vectors and were tuned with RandomizedSearchCV.The four classifiers were SVM, KNN, Random Forest, and Logistic Regression via SGD.Source: §4.1 · p. 6
• Among four scikit-learn classifiers, SVC has the highest Kendall’s Tau (0.76) and F1-score.The authors selected SVC for further data-efficiency experiments and report robust performance with limited training data, particularly for identifying the most relevant candidates.Source: §4.1 · p. 6

4.2 BERT

• BERT training used bert-base-uncased with one-hot or thermometer labels, batch size 4, and Adam; one epoch on 80,524 examples took about 1 hour 15 minutes.Training ran on Kaggle P100 GPUs with 16GB RAM.Source: §4.2 · p. 7
• With 80,524 training examples, thermometer encoding improves Kendall’s Tau by approximately 3% over one-hot encoding.One-hot encoding can assign papers from the same prompt to the same category, which poses a problem for ranking but not classification.Source: §4.2 · p. 7
• At approximately 5,032 training examples, both BERT encodings approach a plateau on Kendall’s Tau and F1-score.The paper reports this as less than 10% of the full training size and says increasing training-set size improves performance overall.Source: §4.2 · p. 7

4.3 Comparison of Performances Among Different Baselines

• The cosine-similarity baseline classifies relevance using three thresholds selected on the training set to maximize Kendall’s Tau.The authors compare this baseline with the learned models.Source: §4.3 · p. 7
• BERT models significantly surpass SVC and cosine similarity, with thermometer encoding achieving the highest Kendall’s Tau and the smallest bootstrap standard error.The comparison reports Kendall’s Tau and F1-scores for the models.Source: §4.3 · p. 7

5 The RelevAI-Reviewer Benchmark

• The authors open the relevance-review problem as a Codabench challenge and provide the Relevance-AI dataset and baseline-model code.They describe the work as an initial investigation and invite community participation.Source: §5 · p. 7
• Submissions, training, and testing are processed on the authors’ server using locally installed models, including pre-trained BERT, for data protection.Source: §5 · p. 7

6 Conclusion and Future work

• BERT models, particularly with thermometer encoding, significantly outperform traditional machine-learning approaches in ranking manuscript relevance.The authors describe the work as an initial contribution based on a dataset curated for this task.Source: §6 · p. 8
• The study is limited to survey papers, and its title, abstract, and artificially generated related-work representation may not capture a paper’s full depth.The authors also flag possible bias in prompts generated from available papers, human-cited papers as ground truth, and assigning papers to four classes when two may share a relevance level.Source: §6 · p. 8

A Relevance-AI Dataset Example

• The dataset example pairs a prompt about COVID-19 conspiracy-theory beliefs with candidate papers representing different relevance categories.The example is presented to illustrate differences in the subjects of papers across categories.Source: A Relevance-AI Dataset Example · p. 11, Table 3 · p. 11

B Scikit-learn Models Hyperparameter Tuning

• The four scikit-learn classifiers were tuned with RandomizedSearchCV using three-fold cross-validation, 30 iterations, and PCA component choices from 2 to 200 or none.Source: B Scikit-learn Models Hyperparameter… · p. 12

B.1 KNeighborsClassifier Hyperparameters

• KNN tuning searched neighbor counts from 1 to 20, weighting options, four algorithms, and three distance metrics.Source: §B.1 · p. 12

B.2 RandomForestClassifier Hyperparameters

• Random Forest tuning varied estimator count from 50 to 500, depth, split and leaf sizes, and feature-selection options.Source: §B.2 · p. 12

B.3 SVC Hyperparameters

• SVC tuning searched C values from 10^-3 to 10^2, scale or auto gamma, and linear, polynomial, RBF, or sigmoid kernels.Source: §B.3 · p. 12

B.4 SGDClassifier Hyperparameters

• SGDClassifier tuning varied hinge, log, or modified-Huber loss, regularization, and iteration count from 100 to 999.The appendix describes SGDClassifier as a linear classifier trained with stochastic gradient descent.Source: §B.4 · p. 12

• Table 4 lists each model’s best hyperparameters from RandomizedSearchCV, selected using Kendall’s Tau on the test set.Source: §B.5 · p. 13

C SVC Results

• The second least relevant class remains SVC’s hardest to predict, with final performance substantially below other classes despite gains from more training data.Across 78 to 80,524 training examples, performance on the most relevant, second most relevant, and least relevant classes plateaus; full-set training took about 19.41 minutes.Source: C SVC Results · p. 14, Figure 2 · p. 14

D BERT Results

• BERT training-set experiments use sizes from 157 to 80,524, a fixed 20,132-example test set, and bootstrap-derived error bars.Source: D BERT Results · p. 15

D.1 Performance of BERT Across Different Training Sets

• Beyond 5,032 training examples, none of the three models yields significant performance gains; BERT fluctuates across the first five training sizes while SVC remains stable.Source: §D.1 · p. 15, Figure 3 · p. 15

D.2 F1-score Comparison Across Different Training Sets

• Both BERT encodings score highly on the most and second most relevant classes but struggle on the least and second least relevant classes.The authors attribute this pattern to dataset construction: the original paper is most relevant and a paper it cites is second most relevant, making those categories easier to identify.Source: §D.2 · p. 15, Figure 4 · p. 16, Figure 5 · p. 16

• The experiment bootstraps training data 10 times for each of the first six subset sizes, from 157 to 5,032 examples, and evaluates predictions on a fixed test set.Source: §D.3 · p. 16
• One-hot Kendall’s Tau increases with training size, but its expanding standard error indicates growing uncertainty despite a fixed test set.The authors attribute the uncertainty to variability introduced by bootstrapping the training data.Source: §D.3 · p. 17, Figure 6 · p. 17
• Thermometer Kendall’s Tau fluctuates as training size changes, while its standard error is smaller than in previous iterations.The authors suggest this may indicate more stable predictions or reduced variability in bootstrapped training samples.Source: §D.3 · p. 17, Figure 7 · p. 17

E Cosine Similarity Baseline – Thresholds definition algorithm

• A three-threshold grid is searched on the training set in 0.025 steps, selecting combinations by Kendall’s Tau.Source: E Cosine Similarity Baseline –… · p. 18
• The thresholds divide candidates into four classes: below the first is least relevant, and above the third is most relevant, with the two middle bands assigned intermediate ranks.Source: E Cosine Similarity Baseline –… · p. 18
• The test-set Kendall’s Tau is 0.774, nearly identical to the training-set score of 0.775.The authors note that the train and test similarity distributions appear very similar; the test result is also assessed with bootstrapped samples.Source: E Cosine Similarity Baseline –… · p. 18, Figure 8 · p. 18, Figure 9 · p. 19