PaperlayerView exact revision on arXiv

Source-linked AI summary

Counterfactual Evaluation of Peer-Review Assignment Policies

Martin Saveski, Steven Jecmen, Nihar B. Shah, Johan Ugander

arXiv:2305.17339v1cs.IRcs.DLstat.AP

TL;DR

Peer-review assignment changes are hard to evaluate because observed reviews come only from the policy actually used. This paper uses randomized assignments and partial-identification methods to estimate alternatives; it finds that greater text-similarity weight is associated with higher review quality, while randomization only marginally reduces it.

  • Problem

    Assignment-policy changes are difficult to evaluate because researchers observe review quality only for assignments generated by the algorithm actually used.

    Source: §1 · p. 2

  • Method

    The paper exploits randomized reviewer assignments and uses partial-identification methods based on monotonicity and Lipschitz smoothness to evaluate alternative policies.

    Source: §1 · p. 2

  • Results

    Greater text-similarity weight is associated with higher review quality, while randomization leads to only a very small reduction in review quality.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Partial-identification methods support useful off-policy conclusions despite positivity violations and missing reviews.

    Source: §7 · pp. 12–13

    Inference depends on the amount of on-policy randomness; with little randomness, estimates are limited to policies relatively close to the deployed policy unless assumptions are made.

    Source: §7 · p. 13

Abstract

from arXiv · show

Peer review assignment algorithms aim to match research papers to suitable expert reviewers, working to maximize the quality of the resulting reviews. A key challenge in designing effective assignment policies is evaluating how changes to the assignment algorithm map to changes in review quality. In this work, we leverage recently proposed policies that introduce randomness in peer-review assignment--in order to mitigate fraud--as a valuable opportunity to evaluate counterfactual assignment policies. Specifically, we exploit how such randomized assignments provide a positive probability of observing the reviews of many assignment policies of interest. To address challenges in applying standard off-policy evaluation methods, such as violations of positivity, we introduce novel methods for partial identification based on monotonicity and Lipschitz smoothness assumptions for the mapping between reviewer-paper covariates and outcomes. We apply our methods to peer-review data from two computer science venues: the TPDP'21 workshop (95 papers and 35 reviewers) and the AAAI'22 conference (8,450 papers and 3,145 reviewers). We consider estimates of (i) the effect on review quality when changing weights in the assignment algorithm, e.g., weighting reviewers' bids vs. textual similarity (between the review's past papers and the submission), and (ii) the "cost of randomization", capturing the difference in expected quality between the perturbed and unperturbed optimal match. We find that placing higher weight on text similarity results in higher review quality and that introducing randomization in the reviewer-paper assignment only marginally reduces the review quality. Our methods for partial identification may be of independent interest, while our off-policy approach can likely find use evaluating a broad class of algorithmic matching systems.

1 Introduction

The paper addresses the difficulty of evaluating peer-review assignment policies when only reviews from the policy actually used are observed. It exploits random assignment and develops partial-identification methods to compare alternatives, finding that greater text-similarity weight is associated with higher review quality and randomization reduces it only marginally.

Introduction

  • Automated assignment is needed when large submission or reviewer pools make it infeasible to match every paper to suitable experts manually.Source: §1 · p. 1
  • Evaluating assignment changes is difficult because review quality under alternative algorithms is not observed, while A/B tests are costly and compare only a few decisions.A/B tests can require additional reviews, further burdening already strained review systems.Source: §1 · p. 2

Present Work

  • Randomized assignments make a range of reviewer-paper pairs beyond the optimal match observable, enabling evaluation of alternative policies.The randomness is harvested from assignment methods introduced to mitigate reviewer collusion.Source: §1 · p. 2
  • Higher text-similarity weight is associated with higher review quality, while introducing randomization leads to only a very small reduction in review quality.The evaluation covers TPDP'21 and AAAI'22.Source: §1 · pp. 2–3
  • Monotonicity and Lipschitz-smoothness assumptions support partial-identification bounds that make off-policy estimates more informative despite positivity violations.Monotonicity orders pairs by their bids, text similarity, and subject-area match; Lipschitz smoothness relates covariate similarity to outcome similarity.Source: §1 · pp. 2–3
  • The proposed off-policy evaluation and partial-identification methods may also apply to recommendation, advertising, and ride-sharing matching systems.Source: §1 · p. 3

2 Preliminaries

Peer-review assignment systems combine reviewer-paper signals into scores and optimize assignments subject to allocation constraints. Randomized assignment limits each pair's assignment probability, while review quality is assessed using reviewers' reported expertise and confidence.

Reviewer-Paper Similarity

  • Reviewer-paper similarity scores can combine text similarity, subject-area overlap, and reviewer bids, with the combination itself forming a design choice.The paper gives a convex combination of text similarity and bids as a simple example and notes that other functional forms are used.Source: §2 · p. 3

Deterministic Assignment

  • Deterministic assignment maximizes the sum of assigned-pair similarity scores subject to reviewer-load, paper-reviewer-count, and conflict-of-interest constraints.Source: §2 · p. 3

Randomized Assignment

  • Randomized assignment caps each reviewer-paper pair's marginal assignment probability at organizer-chosen q while maximizing expected similarity.The assignment is sampled after the probabilities are computed.Source: §2 · pp. 3–4

Review Quality

  • Reviewers' self-reported expertise and confidence are used as review-quality measures because assignment similarity may not accurately proxy review quality.Other possible measures include direct assessments of written reviews by meta-reviewers or other reviewers.Source: §2 · p. 4

3 Off-Policy Evaluation

Off-policy evaluation estimates the review quality of an alternative assignment policy using data collected under the deployed policy. Randomized assignments provide observations useful for this comparison, but positivity violations and other missing reviews require imputation or bounds.

Off-Policy Evaluation

  • Reviewer-paper pairs made possible by randomized assignments can support evaluation of alternative policies, but positivity violations arise when the alternative policy assigns pairs impossible under the data-collection policy.Source: §3 · pp. 4–5
  • The estimands are expected review quality under logging policy A and alternative policy B, with outcomes defined as reviewer-paper review-quality measures.The paper uses a potential-outcomes framework and treats the total number of reviews as fixed across policies.Source: §3 · p. 4
  • Horvitz-Thompson estimators use observed assigned reviews, with inverse-probability weights to estimate quality under the alternative policy when positivity holds.The paper notes that many policies of interest go beyond the logging policy's support.Source: §3 · pp. 4–5

Challenges

  • Missing reviews arise from positivity violations, absent reviewers, selective attrition, and manual reassignments, and the paper handles these categories differently.Absent reviewers are imputed using a weighted mean; attrition and removed assignments are bounded, while newly added manual assignments are ignored.Source: §3 · p. 5

4 Imputation and Partial Identification

The section handles outcomes missing because of positivity violations, reviewer absence, or attrition through imputation and partial-identification methods. Its approaches range from strong mean or model assumptions to bounds based on weaker monotonicity and Lipschitz assumptions.

4 Imputation and Partial Identification

  • The framework imputes outcomes where they cannot be estimated, then uses point estimators or bounds to evaluate alternative assignment policies.The unobserved pairs include positivity violations, absent reviewers, and remaining missing reviews.Source: §4 · pp. 5–6

4.1 Mean Imputation

  • Mean imputation assigns unobserved pairs the observed-pair mean, assuming it represents their outcomes—a strong assumption when missingness may depend on outcome.For example, reviewers may skip reviews when assignment quality is poor.Source: §4.1 · p. 6

4.2 Model Imputation

  • Model imputation predicts unobserved outcomes from reviewer-paper covariates, relying on models to generalize the observed covariate-outcome mapping.The observed pairs may be good matches while unobserved pairs are typically less good matches, so this generalization entails strong assumptions.Source: §4.2 · pp. 6–7

4.3 Manski Bounds

  • Manski bounds use the known outcome range to bound the counterfactual mean without assumptions about how positivity violations arise.The bounds are formed by imputing the minimum and maximum possible outcomes for positivity-violating and remaining missing pairs.Source: §4.3 · p. 7

4.4 Monotonicity and Lipschitz Smoothness

  • Monotonicity bounds assume that dominance across covariates implies no lower outcome, fitting surrogate values close to observed outcomes before bounding unobserved ones.Observed outcomes may violate monotonicity; the method constrains surrogate values across observed and unobserved pairs.Source: §4.4 · pp. 7–8
  • Lipschitz bounds assume outcomes change at most in proportion to covariate distance, combining that constraint with outcome-range limits.The method uses surrogate values to address violations among observed outcomes; as L grows without bound, its constraints become vacuous and the Manski bounds are recovered.Source: §4.4 · p. 8

5 Experimental Setup

The experiments apply the framework to randomized reviewer assignments at TPDP’21 and AAAI’22, examining assignment-policy changes and the suitability of the covariate-outcome assumptions. The assumption checks report monotonicity agreement among dominant observed pairs and use observed expertise differences to select Lipschitz constants.

5 Experimental Setup

  • The framework is applied to randomized assignments at TPDP’21 and AAAI’22 to study how policy-parameter changes affect review quality.The analyses were approved by the authors’ institutions’ IRBs.Source: §5 · p. 9

5.1 Datasets

  • TPDP’21 had 95 submissions, 35 reviewers, and three reviews per paper; its randomized assignment used a 0.5 text-similarity weight and q = 0.5.Reviewers reported expertise and confidence as quality measures; organizers manually changed three assignments.Source: §5.1 · p. 9
  • In AAAI’s analyzed first-round stage, randomized assignment assigned 8,450 papers to 3,145 senior reviewers, using similarity weight wtext = 0.75 and q = 0.52.Each paper was assigned to one reviewer, and each reviewer received at most 3 or 4 papers based on primary subject area. Reviewers reported expertise and confidence; organizers changed several assignments, and several reviews were not submitted.Source: §5.1 · p. 9

5.2 Assumption Suitability

  • Among dominant observed pairs, monotonicity held for expertise in 79.8% of TPDP pairs and 76.4% of AAAI pairs, and for confidence in 76% and 78.9%, respectively.Dominance occurred in 65.7% of TPDP observed pairs and 28.0% of AAAI pairs; TPDP used only two covariates.Source: §5.2 · p. 9
  • For expertise, the selected Lipschitz constants correspond to under 10%, 5%, and 1% violations at L = 30, 50, 300 for TPDP and L = 30, 40, 100 for AAAI.The analysis uses normalized ℓ1 covariate distance and imputes a distance of 1 for a missing AAAI covariate. Median distances to the nearest observed pair among positivity-violating pairs were 0.0014 for TPDP and 0.0011 for AAAI; the authors report that even large L can substantially tighten bounds relative to Manski bounds.Source: Figure 1 · p. 10, §5.2 · p. 10

6 Results

In TPDP, greater weight on text similarity improves expertise for some assignments, while randomization has little effect on expertise. AAAI analyses find low text weights worse, but the effect of substantially increasing text weight is unclear.

Analysis and uncertainty

  • The analyses use self-reported expertise; LP ties are broken persistently but arbitrarily, and plotted confidence intervals asymptotically cover the true mean with at least 95% probability.The vertical axis does not start at zero, and similar patterns for confidence appear in Appendix Figure 7.Source: §6 · pp. 10–11

TPDP

  • For TPDP, wtext ∈(0.75, 0.9) significantly improves average expertise, while wtext ∈(0.9, 1] brings positivity violations, wider intervals, and model disagreement.The results suggest that more weight on text similarity than on bids leads to higher-expertise reviews.Source: Figure 2 · p. 11, §6 · p. 12
  • For TPDP, average expertise under policies with q > 0.5 is very similar to the on-policy, apart from a small increase around q = 0.75.This suggests that randomized rather than deterministic assignment does not significantly reduce self-reported expertise, despite a previously documented reduction in expected sum-similarity [22].Source: Figure 2 · p. 11, §6 · p. 12

AAAI

  • For AAAI, all methods indicate worse assignments at low wtext, but the effect of substantially increasing wtext is unclear.Manski bounds are uninformative amid many positivity violations; model, monotonicity, and Lipschitz estimates differ in parts of the range.Source: Figure 2 · p. 11, §6 · p. 12
  • For AAAI, monotonicity and Lipschitz bounds indicate slightly higher expertise than the on-policy when λbid ≥1.Models indicate similar expertise across λbid values, while Manski bounds are too wide to be informative.Source: Figure 2 · p. 11, §6 · p. 12
  • For AAAI, estimates agree that q ≥0.6 has slightly higher expertise than q ∈[0.4, 0.6], but the magnitude is small.The on-policy sets q = 0.52.Source: Figure 2 · p. 11, §6 · p. 12

Additional analyses

7 Discussion and Conclusion

The paper presents partial-identification methods for evaluating off-policy review quality despite positivity violations and missing reviews, and discusses their assumptions and broader relevance to matching systems.

7 Discussion and Conclusion

  • Partial-identification methods support conclusions about off-policy review quality despite large numbers of positivity violations and missing reviews.The authors also identify applications to education, advertising, and ride-sharing, and note that reviewer welfare may trade off against review quality.Source: §7 · pp. 12–13

A Linear Programs for Peer Review Assignment

  • The deterministic-assignment linear program maximizes total similarity subject to reviewer-load, paper-review-count, and conflict constraints, with an optimal binary assignment under total unimodularity.The evaluation framework is agnostic to the assignment objective, although this work focuses on sum of similarities.Source: A Linear Programs for Peer Review… · pp. 16–17

B Stable Unit Treatment Value Assumption

  • SUTVA assumes each review outcome is unaffected by other assignments, but the authors do not test for possible batch effects.Batch effects could arise if a reviewer’s confidence depends on the other papers assigned to them.Source: B Stable Unit Treatment Value Assumption · p. 17

C Covariance Estimation

  • Because randomized assignments constrain marginal probabilities but can correlate reviewer-paper pairs, the authors estimate covariance terms by Monte Carlo sampling one million assignments.A bootstrap analysis found variance-estimate variability below 10^-9 even with ten times fewer sampled assignments.Source: C Covariance Estimation · pp. 17–18

D Coverage of Imbens-Manski Confidence Intervals

  • The authors cannot verify a regularity condition required for one confidence-interval result, although estimates using two interval procedures were nearly identical.The alternative procedure assumes the missing-data fraction is known; in this setting it is nearly, but not exactly, known before assignment.Source: D Coverage of Imbens-Manski Confidence… · p. 18

E Model Implementation

The authors compare imputation models for unobserved reviewer-paper outcomes, using held-out prediction performance to select and assess models.

E Model Implementation

  • Classification, ordinal regression, and collaborative-filtering models impute outcomes; the first two use reviewer-paper covariates, while collaborative filtering models observed outcome-matrix structure.For TPDP the covariates are text similarity and bids; for AAAI they also include subject-area match.Source: E Model Implementation · p. 18
  • Models are evaluated by test-set MAE over ten random 75/25 splits, with 10-fold cross-validation for hyperparameter selection; only the two best-performing models per category are reported.The figure reports mean performance and 95% confidence intervals across the splits.Source: E Model Implementation · pp. 18–19, Figure 3 · p. 19
  • TPDP models perform similarly except cf-svd++ is slightly better, while AAAI collaborative-filtering models perform slightly worse than its classification and regression models.All models outperform a baseline predicting the training-set mean; the authors suggest AAAI’s larger covariate set may explain the pattern.Source: E Model Implementation · p. 19, Figure 3 · p. 19

F Details of AAAI Assignment

This section specifies the AAAI senior-reviewer assignment stage and the score components and parameters varied in alternative policies.

F Details of AAAI Assignment

  • The analyzed AAAI stage randomly assigned one senior reviewer to each of 8,450 first-round papers from 3,145 reviewers, with a probability limit q = 0.52.Reviewers generally had a maximum load of four papers, reduced to three for specified areas or tracks.Source: F Details of AAAI Assignment · p. 19
  • AAAI similarities combine text, subject-area, and bid scores, with specified fallback rules when scores are missing and score adjustments for selected bid and conflict-profile cases.Text similarity uses TPMS scores when available and otherwise ACL Reviewer Matching scores.Source: F Details of AAAI Assignment · pp. 19–20
  • Figure 4 plots the confidence-score Lipschitz-ratio CCDF and marks L thresholds for fewer than 10%, 5%, and 1% violations: TPDP 30, 60, 400; AAAI 20, 30, 70.These are the caption’s threshold values for the respective venues.Source: Figure 4 · p. 20
  • Figure 5 plots distances from relevant unobserved pairs to their closest observed pairs, with median distances of 0.0014 for TPDP and 0.0011 for AAAI.The figure describes relevant pairs as unobserved reviewer-paper pairs under consideration.Source: Figure 5 · p. 20
  • Alternative assignment policies vary only the text-similarity weight, bid-weight parameter, and probability limit.The analyzed assignment stage and its procedure otherwise remain fixed.Source: F Details of AAAI Assignment · p. 20

G Details Regarding Assumption Suitability

The appendix examines evidence relevant to Lipschitz smoothness and the distance of relevant unobserved pairs from observed pairs, then distinguishes similarity cost from review-quality cost.

G Details Regarding Assumption Suitability

  • For confidence, Figure 4 presents the CCDF of L, reporting for each L the fraction of violating observation-pairs (pairs of observed reviewer-paper pairs) under the Lipschitz condition.The corresponding plot for expertise is shown in Figure 1.Source: G Details Regarding Assumption… · pp. 20–21
  • Relevant off-policy pairs are those unsupported on-policy but assigned positive probability by policies varying text weight, and for AAAI also bid weight; Figure 5 reports median nearest-observed distances of 0.0014 for TPDP and 0.0011 for AAAI.The distances are between each relevant unobserved pair and its closest observed pair.Source: G Details Regarding Assumption… · p. 21

H Similarity Cost of Randomization

I Tie-Breaking Behavior

Because LP (2) can have non-unique solutions, the analyses fix tie-breaking to recover actual on-policy assignments and reuse it across off-policies. A stress test shows the methods distinguish deliberately poor policies from high-quality ones.

I Tie-Breaking Behavior

  • Because LP (2) may have multiple solutions, arbitrary tie-breaking is fixed to match actual on-policy assignment probabilities and reused for all off-policies.The venues did not specify their on-policy tie-breaking behavior.Source: I Tie-Breaking Behavior · p. 21
  • For TPDP, a shared random perturbation of similarity scores makes them unique; its random matrix is resampled until the perturbed on-policy LP recovers actual assignments.The perturbation uses λ = 1e-8 and then holds the chosen matrix fixed across policies.Source: I Tie-Breaking Behavior · p. 21
  • For AAAI, each policy uses the largest tested penalty that keeps total similarity within 10^-5 of the original LP, and the procedure recovers on-policy assignments.The tested penalty values are 10^-9, 10^-6, and 10^-3.Source: I Tie-Breaking Behavior · p. 22

J Power Investigation: Purposefully Bad Policies

  • The methods identify minimization policies as worse than maximization policies on both venues, with differences clearer when monotonicity and Lipschitz assumptions address attrition.This suggests the primary analyses explore high-quality regions where assignment quality is often robust to parameter values.Source: J Power Investigation: Purposefully… · p. 22, Table 1 · p. 22

K Results for Confidence Outcomes

The confidence-outcome analyses show substantively similar patterns to the expertise results. Figure 7 varies assignment-policy parameters and reports estimates from the Section 4 methods with confidence intervals.

K Results for Confidence Outcomes

  • Confidence-outcome results are substantively very similar to the expertise results reported in Section 6.Figure 7 presents the confidence analyses.Source: K Results for Confidence Outcomes · p. 22, Figure 7 · p. 23
  • Figure 7 varies wtext and q for TPDP, and wtext, λbid, and q for AAAI, across the Section 4 estimation methods.Its dashed blue and grey lines mark Manski bounds around on-policy and off-policy expertise, respectively.Source: Figure 7 · p. 23
  • The plotted confidence intervals asymptotically contain the true µB with probability at least 95%, and the vertical axis starts above zero to focus on relevant plot regions.The error bands are described in Sections 4 and Appendix D.Source: K Results for Confidence Outcomes · p. 23

Related papers

Loading 2305.17339v1…