PaperlayerView exact revision on arXiv

Source-linked AI summary

Variance-Optimal Off-Policy Evaluation with Conjunct Effect Modeling

Nicolò Felicioni, Michael Benigni, Maurizio Ferrari Dacrema, Paolo Cremonesi

arXiv:2610.08677v1cs.LG

TL;DR

Action-level importance weights can make OPE unstable, while cluster-level corrections depend on local reward-model correctness. VOCEM chooses a variance-minimizing blend between OffCEM and DR; it achieves lower MSE than both across 23 benchmarks.

  • Problem

    DR remains exposed to potentially high-variance action-level weights, while OffCEM uses more stable cluster-level weights but relies on local correctness of the reward model.

    Source: §1 · p. 1

  • Method

    VOCEM estimates from logged data a variance-minimizing coefficient for an unbiased family interpolating between OffCEM and DR.

    Source: §6 · p. 9

  • Results

    Across 15 synthetic and eight real-world benchmarks, VOCEM achieved lower MSE than both DR and OffCEM, with gains under oracle and learned reward models.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The results support adapting residual weights to observed data when DR's action-level correction is unstable.

    Source: §6 · p. 9

    OffCEM's local-correctness condition requires the reward model to preserve within-cluster reward differences, limiting its correction of misspecified differences within a cluster.

    Source: §3 · p. 3

Abstract

from arXiv · show

Off-policy evaluation (OPE) for contextual bandit policies becomes challenging when action-level importance weighting incurs excessive variance. Doubly robust (DR) estimation remains unbiased under common support but retains these high-variance action-level weights. A prior estimator, Off-policy evaluation with Conjunct Effect Model (OffCEM), replaces them with more stable cluster-level weights, at the cost of relying on local correctness of the reward model. In this paper, we show that, under the assumptions required by DR and OffCEM, there exists an unbiased family of estimators that interpolates between OffCEM and DR. Building on this result, we propose the Variance Optimal-CEM (VOCEM) estimator, which selects the interpolation coefficient to minimize variance. We derive the population-optimal coefficient in closed form and show that the resulting estimator has variance no larger than either endpoint, OffCEM or DR. Experiments in controlled synthetic settings and on two large-action benchmarks show that VOCEM improves upon both endpoints in all 23 evaluated conditions, exhibiting greater stability and empirical robustness.

1 INTRODUCTION

Action-level importance weighting can make off-policy evaluation unstable, while cluster-level correction relies on local reward-model correctness. The paper connects OffCEM and DR through an unbiased estimator family and selects its member by variance minimization.

1 INTRODUCTION

  • DR retains action-level residual weights that can be unstable, whereas OffCEM uses cluster-level corrections but requires local correctness of the reward model.Local correctness requires preserving within-cluster reward differences rather than recovering absolute reward.Source: §1 · p. 1
  • A single scalar interpolates between OffCEM and DR, and every fixed member is unbiased under action-level common support and local correctness.Source: §1 · pp. 1–2
  • Across 15 synthetic and eight real-data conditions, VOCEM has lower MSE than OffCEM, while fixed DR is often unstable.The estimator family is unbiased under action-level common support and local correctness.Source: §1 · p. 2
  • Because the family is unbiased, the paper derives a closed-form variance-minimizing coefficient and uses its empirical estimate to select VOCEM.The resulting estimator has variance no larger than either endpoint.Source: §1 · p. 2

2 BACKGROUND

The background formulates OPE as estimating a target policy's value from logged contextual-bandit data. It introduces DR's residual correction and its remaining exposure to action-level importance-weight variance.

2 BACKGROUND

  • OPE estimates a target policy's value from logged contexts, actions, and rewards collected under a logging policy, without deploying the target policy.Source: §2 · p. 2
  • The target-policy value is the context-averaged expected reward under its action distribution, while logged observations use the logging policy's action distribution.Source: §2 · p. 2
  • DR combines a reward-model direct estimate with importance weighting of the logged reward residual, correcting regression error under action-level common support.Source: §2 · pp. 2–3
  • When the reward model is accurate, residuals can be less variable than raw rewards, but DR still uses action-level weights that can become unstable for rarely logged actions.Source: §2 · p. 3

3 ACTION CLUSTERS AND OFFCEM

OffCEM groups actions into clusters and pools policy mass to obtain cluster-level importance weights. Its unbiasedness depends on the reward model preserving within-cluster reward contrasts.

3 ACTION CLUSTERS AND OFFCEM

  • Pooling policy mass within action clusters can make OffCEM's importance weights less variable than action-level weights.Source: §3 · p. 3
  • OffCEM combines target-policy reward predictions with residual corrections weighted at the cluster level, avoiding potentially extreme action-level residual ratios.Source: §3 · p. 3
  • Local correctness requires the reward model's error to be shared within a cluster, preserving within-cluster reward differences while allowing additive error to vary across contexts and clusters.Source: §3 · pp. 3–4
  • Under local correctness, cluster weighting corrects cluster-level offsets while the model supplies within-cluster reward contrasts, yielding OffCEM's unbiasedness.Source: §3 · p. 4

4 VARIANCE OPTIMAL-CEM

The section constructs an unbiased path between OffCEM and DR, then chooses a member by minimizing variance. Its empirical plug-in coefficient yields VOCEM.

4 VARIANCE OPTIMAL-CEM

  • VOCEM connects DR and OffCEM through a one-parameter family, derives its population variance-minimizing coefficient, and estimates that coefficient from logged data.Source: §4 · p. 4

4.1 AN UNBIASED PATH FROM OFFCEM TO DR

  • The interpolated residual weight blends cluster-level and action-level ratios, recovering OffCEM and DR at λ = 0 and λ = 1, respectively.Source: §4.1 · p. 4
  • Every fixed interpolation coefficient is unbiased under action-level common support and local correctness, so minimizing MSE along the path reduces to minimizing variance.Source: §4.1 · p. 4, §4.2 · p. 5

4.2 THE VARIANCE-OPTIMAL COEFFICIENT AND ITS EMPIRICAL ESTIMATE

  • The population coefficient minimizes variance over [0,1]; when A = 0, all path estimators agree, and the coefficient can be estimated from logged-sample averages.The plug-in coefficient is substituted into the estimator to obtain VOCEM.Source: §4.2 · p. 5

5 EXPERIMENTS

The experiments compare DR, OffCEM, and VOCEM across synthetic noise, sample-size, and action-space sweeps, then test learned rewards on EUR-Lex 4K and Wiki10-31K. VOCEM improves on OffCEM across the reported conditions, including with learned rewards.

5.1.2 RESULTS

  • VOCEM’s MSE was below OffCEM’s in all 15 synthetic conditions and all eight learned-reward conditions; all reported pointwise upper confidence bounds were below one.On the real datasets, DR’s MSE ranged from 2.19 to 132.94 times OffCEM’s.Source: Figure 1 · p. 6, §5.1.2 · p. 6, Figure 3 · p. 8

5.1.1 SETUP

  • The synthetic study varied noise heterogeneity, sample size, and action count with a known reward model, isolating correction weights across 300 repetitions per condition.The reward model was set to the true conditional mean; the sweeps included κ = 1 to 16, n = 500 to 8,000, and |A| = 200 to 4,000.Source: §5.1.1 · p. 6

5.1.2 RESULTS

  • MSE ratios were 0.634–0.749 across noise heterogeneity; as |A| rose from 200 to 4,000, the ratio fell from 0.834 to 0.318 while DR rose from 3.34 to 83.38 times OffCEM.The synthetic sample-size sweep instead saw the ratio increase from 0.188 at n = 500 to 0.873 at n = 8,000.Source: §5.1.2 · p. 7
  • In the synthetic sample-size sweep, mean bλ fell from 0.811 to 0.124 as n rose from 500 to 8,000; with 4,000 actions, its mean reached 0.720 and 70% of repetitions selected DR.The coefficient often moves toward OffCEM as sample size grows and toward DR as the action space grows, while VOCEM can back away from DR on logs with unstable weights.Source: §5.1.2 · p. 7

5.2.1 DATASETS AND PROTOCOL

  • The real-data protocol used EUR-Lex 4K and filtered Wiki10-31K labels, with five-fold cross-fitting for the ridge reward model and 100 paired repetitions.The Wiki10-31K filter retained 5,501 labels, while EUR-Lex retained all 3,956 labels.Source: §5.2.1 · pp. 7–8

5.2.2 RESULTS

  • With learned rewards, EUR-Lex’s mean bλ fell from 0.667 to 0.200 as n grew, while Wiki10-31K’s median remained near the DR endpoint.The mean coefficient on Wiki10-31K declined from 0.863 to 0.624; VOCEM makes the endpoint choice separately for each full log.Source: §5.2.2 · p. 9, Figure 4 · p. 9

6 CONCLUSION

The conclusion presents VOCEM as a data-adaptive interpolation between DR and OffCEM. It reports lower MSE than both estimators across synthetic and real-world benchmarks and identifies joint learning of clusters and the coefficient as future work.

6 CONCLUSION

  • VOCEM connects DR and OffCEM through an unbiased one-parameter family and estimates the variance-minimizing coefficient from logged data.The population-optimal coefficient is derived in closed form, and its estimator has variance no larger than either endpoint.Source: §6 · p. 9
  • Across 15 synthetic and eight real-world benchmarks, VOCEM achieved lower MSE than both DR and OffCEM with oracle and learned reward models.The conclusion supports adapting residual weights to observed data when DR’s action-level correction is unstable.Source: §6 · p. 9
  • Jointly learning the action clusters and interpolation coefficient is proposed as future work.Source: §6 · p. 9

A PROOFS FOR SECTION 3: ACTION CLUSTERS AND OFFCEM

The proof of the cluster-projection lemma uses common support and conditional expectation. It distinguishes quantities fixed within a cluster from action-specific weights that still vary within it.

A PROOFS FOR SECTION 3: ACTION CLUSTERS AND OFFCEM

B PROOFS FOR SECTION 4: VARIANCE OPTIMAL-CEM

Under the stated assumptions, the proof establishes an unbiased estimator path from OffCEM to DR and derives a variance-minimizing coefficient with endpoint variance dominance. The appendix also situates VOCEM among stabilization methods and records experimental implementation and synthetic-benchmark details.

B PROOFS FOR SECTION 4: VARIANCE OPTIMAL-CEM

  • Local correctness makes the correction term vanish, establishing unbiasedness for every fixed estimator on the OffCEM–DR path.The proof expands Yλ = Y0 + λde and uses the resulting quadratic variance to obtain the minimizer.Source: B PROOFS FOR SECTION 4: VARIANCE… · p. 15
  • Under Assumptions 1 and 2, the variance-minimizing coefficient lies in [0, 1] and yields variance no greater than either OffCEM or DR.When A = 0, all estimators on the path agree almost surely and λ⋆ is set to 0.Source: B PROOFS FOR SECTION 4: VARIANCE… · p. 15

C RELATED WORK

  • Prior stabilization methods use normalization, weight transformations, shrinkage, or data-driven selection, whereas VOCEM derives an analytic variance optimum within a structured estimator family.The related-work discussion also describes methods that use action structure or marginal density ratios to address variance and support problems.Source: C RELATED WORK · pp. 16–17

D.1 ESTIMATOR IMPLEMENTATION AND REPORTING

  • The reporting protocol pairs methods on the same logs and uses relative squared error, with pointwise bootstrap intervals for the reported comparisons.Synthetic conditions use 300 replications and real-data conditions use 100; the bλ ribbons instead show a clipped standard deviation around the mean.Source: §D.1 · p. 17

D.2 SYNTHETIC BENCHMARK

  • The synthetic benchmark retains the OffCEM generator and seed, uses 50 actions per cluster, and supplies every estimator with the oracle regression bq = q.With κ = 1, the reward construction recovers OffCEM’s homoskedastic reward with variance 3.Source: §D.2 · pp. 17–18

D.3 REAL-WORLD SEMI-SYNTHETIC BENCHMARK

The real-data semi-synthetic benchmark uses EUR-Lex 4K and Wiki10-31K with constructed policies and clustered actions. Its overlap is weak in practice, and the reported oracle-regression OffCEM comparison shows that known probabilities do not prevent high Monte Carlo MSE for action-weighted DR.

D.3 REAL-WORLD SEMI-SYNTHETIC BENCHMARK

  • EUR-Lex 4K and Wiki10-31K provide sparse bag-of-words inputs with 3,956 and 5,501 evaluated actions, respectively.The original test split constructs the logging policy, while evaluation users come from the training split.Source: §D.3 · p. 18
  • The experiment constructs logging policies from a ridge reward model with inverse temperature 30 and evaluates 100 replications using nested log prefixes from 1,000 to 8,000 rounds.Each replication uses 300 training documents as users and a shared random partition of actions into 100 clusters.Source: §D.3 · p. 18
  • The constructed policies satisfy common support, but practical overlap is weak, explaining why action-weighted DR can have very high Monte Carlo MSE despite known probabilities.Source: §D.3 · p. 19
  • The learned reward model uses five-fold group cross-fitting by user, and its additive ridge regression differs from OffCEM’s three-hidden-layer, two-stage neural regression.The coefficient is selected using the complete set of out-of-fold residuals, so the procedure reuses those residuals for selection.Source: §D.3 · p. 19

D.4 ORACLE-REGRESSION ABLATION ON REAL-WORLD DATA

With the true conditional reward mean supplied to every estimator, VOCEM remains below OffCEM in all eight real-data conditions. DR’s much larger relative MSE shows that the advantage persists when reward-model error is removed and correction weights differ.

D.4 ORACLE-REGRESSION ABLATION ON REAL-WORLD DATA

  • 0.347–0.788 on EUR-Lex 4K and 0.156–0.230 on Wiki10-31K: VOCEM’s MSE ratios stay below oracle-regression OffCEM in all eight conditions.The action-weighted DR endpoint has 18.0–191.2 times the corresponding OffCEM MSE.Source: §D.4 · p. 19
  • The gains persist with bq = q, attributing the comparison to correction weights rather than reward-model error.Datasets, policies, cluster partition, sample sizes, paired repetitions, and bootstrap construction remain unchanged from the learned-model protocol.Source: §D.4 · p. 19

D.5 ADDITIONAL OPE BASELINES

The additional-baseline experiments compare IPS and DM with the proposed estimators across synthetic and real-data settings. They report that action-level IPS is unstable, DR improves on IPS but remains less stable than cluster-based corrections, and VOCEM stays below OffCEM in the synthetic sweeps and oracle-regression ablation.

D.5 ADDITIONAL OPE BASELINES

  • Across all three synthetic sweeps, IPS is highly variable and worsens as action count increases, while same-log VOCEM stays below OffCEM.DR is generally more stable than IPS but also deteriorates as the action space grows.Source: §D.5 · p. 20
  • With learned reward regression, action-level IPS has substantially larger MSE than cluster-based estimators on both datasets, while DR improves considerably over IPS but remains less stable.DM is competitive on Wiki10-31K, indicating substantial predictive information in the out-of-fold reward regression there.Source: §D.5 · p. 20, Figure 7 · p. 21
  • With oracle regression, VOCEM remains substantially below OffCEM on both datasets, whereas IPS and DR lie above OffCEM and IPS is especially unstable.The normalized MSE uses oracle-regression OffCEM as its denominator, so its scale differs from the learned-regression comparison.Source: §D.5 · pp. 20–21, Figure 8 · p. 21

E SYNTHETIC EXPERIMENT UNDER DEFICIENT SUPPORT

The deficient-support experiment preserves cluster support while progressively removing logging-policy action support. VOCEM’s relative MSE remains below OffCEM’s under reward-model misspecification, balancing reduced variance against support-induced bias.

E SYNTHETIC EXPERIMENT UNDER DEFICIENT SUPPORT

  • The experiment used n = 3,000, |A| = 1,000, κ = 16, and deficient-support fractions ρ ∈ {0, 0.3, 0.5, 0.7, 0.9}, with cluster support retained.The two reward-model specifications were exact and misspecified; each condition was repeated 300 times.Source: E SYNTHETIC EXPERIMENT UNDER DEFICIENT… · pp. 21–22
  • VOCEM’s relative MSE fell from 0.66 to 0.13 with exact rewards and from 0.76 to 0.52 with misspecification, remaining below OffCEM in the latter setting.Figure 9 reports MSE relative to OffCEM; values span ρ = 0 to ρ = 0.9.Source: Figure 9 · p. 22, E SYNTHETIC EXPERIMENT UNDER DEFICIENT… · p. 22
  • With exact rewards, DR’s relative MSE fell from about 20.3 under full support to well below OffCEM at ρ = 0.9 as fewer weighted correction terms remained active.VOCEM’s estimated mixing coefficient rose from roughly 0.34 to 0.93, moving toward action-level correction as its variance penalty shrank.Source: E SYNTHETIC EXPERIMENT UNDER DEFICIENT… · p. 22
  • With reward-model error, OffCEM remained approximately unbiased under preserved cluster support, while DR developed bias on unsupported actions and VOCEM accepted some of that bias.The misspecified reward model uses bq(x, a) = q(x, a) − 0.75; VOCEM estimates bλ from the complete evaluation log.Source: E SYNTHETIC EXPERIMENT UNDER DEFICIENT… · p. 22
Loading 2610.08677v1…