PaperlayerView exact revision on arXiv

Source-linked AI summary

Budgeted Multi-Source Counterfactual Annotation for Off-Policy Evaluation

Biao Xiang, Ali Eshragh, Yuexing Li, Kai Wang

arXiv:2610.10974v1cs.LGcs.AImath.OC

TL;DR

Limited behavior-policy coverage can make contextual-bandit OPE depend on high-variance reweighting or reward-model extrapolation, while counterfactual annotations can add evidence about unobserved actions. This paper formulates budgeted allocation across annotation sources and characterizes when annotations are valuable. Its analysis establishes a coupled, nonconvex allocation problem and an MM algorithm that monotonically improves the objective.

  • Problem

    In contextual-bandit OPE, logged data reveal rewards only for behavior-policy actions, leaving target-policy actions with poor coverage to be evaluated using high-variance reweighting or reward-model extrapolation.

    Source: §1 · p. 1

  • Method

    The paper formulates budgeted integer allocation of counterfactual annotations across context-action pairs and sources, deriving local value regimes and using MM with dynamic-programming subroutines for the coupled problem.

    Source: §1 · p. 2

  • Results

    The allocation problem is coupled, nonconvex, and not generally submodular; the MM algorithm with dynamic-programming subroutines monotonically improves its objective.

    Source: §8 · p. 10

  • Takeaways & Limitations

    The results guide the purchase and allocation of counterfactual annotations under a limited budget.

    Source: §8 · p. 10

    The framework is limited to contextual bandits and known or estimated source profiles; the ASSISTments experiment uses real LLM annotations in a semi-synthetic environment.

    Source: §8 · p. 10

Abstract

from arXiv · show

Off-policy evaluation (OPE) estimates the value of a target policy from logged data, but limited behavior-policy coverage can force high-variance reweighting or reward-model extrapolation. Counterfactual annotations can add evidence about unobserved actions, yet practical sources, including domain experts and large language models (LLMs), may be costly, biased, or noisy. We study budgeted acquisition of such annotations for contextual-bandit OPE. Given source-specific costs and error profiles, we formulate an integer allocation problem over context-action pairs and annotation sources to minimize the component of estimator variance that depends on the annotation plan. We characterize when annotations are valuable through a first-annotation threshold and local annotation-value regimes. For the coupled multi-source problem, we develop a majorization-minimization algorithm with dynamic-programming subroutines that monotonically improves the objective. Experiments in synthetic clinical and LLM-annotated education bandits show that our allocation method reduces fixed-profile mean squared error (MSE) by 20.58% and 10.77%, respectively, relative to no annotation.

1 Introduction

Limited behavior-policy coverage makes contextual-bandit OPE rely on high-variance reweighting or reward-model extrapolation. The paper studies how to allocate a limited budget across counterfactual annotations that differ in cost and reliability.

1 Introduction

  • Annotation sources differ in cost and reliability, and their value also depends on factual coverage and how strongly the target policy weights the action.Source: §1 · pp. 1–2
  • The allocation problem is discrete and coupled: annotations for different actions in one context jointly shape the reward model used for evaluation.The objective minimizes the part of estimation error that depends on the annotation plan.Source: §1 · p. 2
  • A first-annotation threshold and four local regimes characterize when annotations help immediately, only up to a finite amount, only after a batch, or not locally.Source: §1 · p. 2
  • The coupled, nonconvex, generally non-submodular allocation problem is addressed with a majorization-minimization algorithm and dynamic-programming subroutines that monotonically improve its objective.Source: §1 · p. 2
  • The empirical validation compares the approach with greedy allocation and certified solver benchmarks, tests profile-estimation sensitivity, and evaluates the final estimator using finite pilots and independent samples.Source: §1 · p. 2

2 Related Work

Prior work develops OPE estimators and methods for incorporating counterfactual or auxiliary information. This paper focuses instead on allocating a finite budget for annotations before applying an estimator.

2 Related Work

  • OPE research includes importance-sampling, direct-method, and doubly robust estimators, alongside work on variance reduction, benchmarking, and confidence or estimator-selection methods,,,,,,.Source: §2 · p. 2
  • Counterfactual and auxiliary-data methods study how to use annotations or generated information once available, whereas this paper studies how to allocate a finite annotation budget across heterogeneous sources,, [2], [16].Source: §2 · p. 2
  • Unlike active learning, optimal design, or multiple-logger OPE, this setting acquires counterfactual rewards for a DR reward model, valuing annotations by their effect on OPE variance under the target and behavior policies,.Source: §2 · p. 2

3 Problem Formulation

The paper formulates contextual-bandit OPE with separate data for reward-model fitting and evaluation, then augments reward estimates with draws from independent annotation sources. The allocation chooses how many annotations to obtain for each context-action pair and source.

3 Problem Formulation

  • Contexts, behavior-policy actions, and rewards define the factual data, with each context-action pair having a true reward mean and variance.Source: §3 · p. 2
  • Each annotation source supplies independent draws for a context-action pair, with its expected reward described by an annotation bias and its variance by excess variance.Source: §3 · p. 3
  • Annotations enter the reward estimate rather than the importance-sampling correction, and the estimate uses the factual and annotated observations available for that pair.Source: §3 · p. 3
  • The goal is to choose annotation counts for each source and pair to minimize the DM+-IS estimator's mean squared error, under common support for the target policy.Source: §3 · p. 3

4 Objective Function and Budget-Constrained Allocation

The section isolates the part of DM+-IS mean squared error controlled by annotation allocation and formulates a budget-constrained integer optimization. It also relates profile-estimation and optimization errors to excess OPE MSE.

4 Objective Function and Budget-Constrained Allocation

  • The estimator is unbiased under the stated independent-sample, known-propensity, common-support, and finite-second-moment conditions, while annotations affect variance through bias correction and reward-model variance.The allocation-invariant component can be excluded when minimizing fixed-profile MSE.Source: §4 · pp. 3–4
  • The allocation objective minimizes the part of OPE error that depends on annotation counts, subject to a total cost budget and a one-annotation-per-factual-sample protocol.Source: §4 · p. 4
  • A feasible allocation selected using estimated source profiles has an excess-MSE guarantee that accounts for profile-estimation and optimization errors.As profile errors vanish, the bound reduces to ξalg/N, separating source-profile estimation gains from solver gains.Source: §4 · p. 4

5 Annotation Value: Thresholds, Regimes, and Multi-Source Benefits

The section characterizes when annotations help locally and how source choice and cross-action coupling affect their value. It uses these results to motivate a joint allocation method.

5.1 Local Single-Source Objective and First-Annotation Threshold

  • The first annotation is worthwhile only when its bias satisfies a pair-specific threshold balancing available variance reduction against annotation error.Greater factual coverage and excess annotation variance make the first annotation harder to justify.Source: §5.1 · p. 5

5.2 Four Regimes of Annotation Value

  • Four regimes describe whether the local objective increases, decreases, or reverses direction as annotations accrue; some sources favor finite counts rather than spending the full budget on one pair.In Regime IV, a sufficiently large batch improves on no annotation only when the limiting bias contribution is below the initial objective.Source: §5.2 · p. 5

5.3 Source Selection and Bias Cancellation

  • The multi-source optimum is no worse than the best single-source plan under the same budget and capacities.Mixing can yield modest gains when one source performs well across pairs and offers little bias cancellation.Source: §5.3 · pp. 5–6
  • Because source quality and factual coverage vary across pairs, different sources can satisfy the first-annotation condition at different pairs.At a fixed count, opposite-signed source biases can also reduce reward-model bias, but that benefit must be balanced against source variances and costs.Source: §5.3 · pp. 5–6

5.4 From Local Thresholds to the Full Allocation Problem

  • Cross-action coupling through J2 changes local annotation thresholds and makes the marginal effect depend on effective biases across actions in a context.Together with shared budget constraints, this coupling motivates joint rather than independent allocation.Source: §5.4 · p. 6

6 Budget-Constrained Multi-Source Allocation

The allocation objective is coupled across actions, nonconvex, and integer-constrained. The proposed MM procedure uses a tight surrogate and computational subroutines to find feasible improvements while preserving descent.

6 Budget-Constrained Multi-Source Allocation

  • The allocation objective is non-separable across actions, and valid instances make its continuous relaxation neither convex nor concave, nor submodular nor supermodular.These properties prevent independent pairwise decisions from capturing the joint allocation structure.Source: §6 · p. 6

6.1 Majorization-Minimization Surrogate

  • MM replaces cross-action coupling with a locally tight upper surrogate while retaining the per-pair bias-variance structure, so feasible surrogate-improving updates monotonically decrease the objective.The surrogate majorizes the coupling term while leaving J1 and J3 unchanged.Source: §6 · p. 6, §6.1 · pp. 6–7

6.2 Solving the Surrogate Allocation Problem

  • At each iteration, source-mixture search approximates per-pair values, while dynamic programming exactly aggregates those values across actions within each context.The source mixtures remain approximate even though DP is exact for the resulting tables.Source: §6.2 · p. 7
  • Bisection over a budget multiplier selects among feasible candidates, with the current allocation as a fallback that ensures the next update cannot increase the objective.With other parameters fixed, the stated runtime bound grows linearly with the number of context-action pairs.Source: §6.2 · pp. 7–8

7 Numerical Experiments

Experiments compare annotation value, allocation methods, and profile-estimation robustness in synthetic and education bandits. MM-based allocations reduce MSE relative to no annotation, including when profiles are estimated from finite pilots.

7.1 Annotation Value and Synthetic Validation

  • Figure 1 matches the predicted local regimes: source MSE trends range from monotone decrease to monotone increase, with two sources reversing direction.The decomposition shows variance reduction (J3) competing with bias accumulation (J1) and correction (J2).Source: Figure 1 · p. 8, §7.1 · p. 8
  • 20.58% MSE reduction: the four-source MM allocation lowers scaled MSE from V(0) = 84.23 to 66.90 relative to no annotation.The four-source setting has the lowest scaled MSE among evaluated source subsets.Source: §7.1 · pp. 8–9, Figure 2 · p. 9

7.2 Baselines and Solver Benchmarks

  • MM or greedy-initialized MM achieves the lowest MSE in each environment, with reductions of 10.77%, 20.68%, and 16.11% relative to no annotation.On the I/IV-focused instance, MM improves on both greedy baselines.Source: Table 3 · p. 9, §7.2 · p. 9
  • Greedy-initialized MM has the lowest mean optimality gap; on the full Synthetic and ASSISTments instances, it reaches within 0.2807% and 0.0050%, respectively, of the Gurobi lower bounds.Source: §7.2 · p. 10

7.3 Robustness to Finite-Pilot Profile Estimation

  • 19.13% on Synthetic and 8.47% on ASSISTments: greedy-initialized MM reduces empirical OPE MSE relative to no annotation when source profiles are estimated from finite pilots.The pilot, fitting data, acquired annotations, and evaluation data are mutually independent.Source: §7.3 · p. 10
  • Bootstrap resampling of observed LLM predictions gives a 10.765% mean reference-profile scaled-MSE reduction, versus 10.774% with original profiles, with a 0.0038 percentage-point standard deviation.This sensitivity check complements final OPE MSE results measured using independent evaluation data.Source: §7.3 · p. 10

8 Conclusion

The conclusion presents budgeted counterfactual annotation as an integer allocation problem for contextual-bandit OPE, with an algorithm and results intended to guide limited-budget acquisition. The appendix derives the variance terms and combines them into the allocation objective.

8 Conclusion

  • The framework minimizes DM+-IS estimator MSE through integer allocation, with a majorization-minimization algorithm and dynamic programming that monotonically improves the objective.The analysis identifies cross-action coupling, nonconvexity, and lack of submodularity; the conclusion says the results guide annotation purchasing under a limited budget.Source: §8 · p. 10

A Derivations for the Objective Function

A.1 Term 3

  • The Term 3 derivation calculates reward-model bias for annotated pairs and uses the fixed default when a pair has no annotations.The derivation then expands the bias under the target policy.Source: §A.1 · pp. 13–14

A.2 Term 4

  • The Term 4 derivation calculates reward-model variance for annotated pairs and assigns zero variance to the fixed default when there are no annotations.It then substitutes this variance into the fourth term and expands the expectations.Source: §A.2 · pp. 14–15

A.3 Objective Function

  • The objective function combines the expressions derived in equations (13) and (16).Source: §A.3 · p. 15

A.4 Theoretical bound for source-profile estimation error

This section bounds the effect of estimating source profiles and analyzes when annotations change estimator variance, from a single context-action pair to broader coupled settings. It gives a first-annotation threshold and characterizes how local annotation value changes with annotation count.

A.4 Theoretical bound for source-profile estimation error

  • With independent pilot, reward/annotation, and evaluation samples, source-profile estimation adds no OPE bias, and the paper bounds objective perturbation and its effect on MSE.DM+-IS is unbiased conditional on the pilot-selected allocation under the stated data separation.Source: §A.4 · pp. 15, 16, 18

B Threshold for Valuable Annotation for a single annotation source

B.1 Setup and Notation

  • The single-pair analysis assumes positive factual count and α1; its undercoverage ratio approaches one when behavior-policy probability is small.The zero-factual-count case is treated separately in Appendix B.8.Source: §B.1 · p. 18

B.2 Bias Threshold for a Single Context-Action Pair

  • For one context-action pair, Lemma 2 gives the local-objective change from increasing the annotation count by one, while setting aside cross-action bias correction J2.Source: §B.2 · p. 19

B.3 Proof of Theorem 3 (First Annotation Threshold)

  • When excess variance Δ ≥ (nf+1)σ^2/nf and annotation bias ε ≠ 0, the first annotation strictly increases the local objective.The theorem characterizes when the first annotation does not increase that objective; equality leaves it unchanged, while strict inequality decreases it.Source: §B.3 · p. 19

B.4 Proof of Theorem 4 (Four Regimes of Annotation Value)

This section proves Theorem 4’s classification of local annotation value into four regimes and situates those regimes alongside asymptotic limits, cross-action bias coupling, special cases, and structural properties of the objective. The proof uses objective-component decompositions, quotient-rule Hessian identities, and default definitions for finite annotation changes when counts are zero.

B.4 Proof of Theorem 4 (Four Regimes of Annotation Value)

  • The local objective is constant when a = b = 0; otherwise, it increases, decreases-then-increases, decreases, or increases-then-decreases across four sign regimes.In Regime II, the minimum is at n*; in Regime IV, the maximum is at n̄.Source: §B.4 · p. 21
  • In Regime IV, improvement over no annotation is possible iff ε² < ησ²/nf; when α1ε² < ϕ(0), it occurs only above a finite break-even count nbe.If α1ε² ≥ ϕ(0), annotations do not improve on ϕ(0) at any count; under finite capacity, the break-even count must be feasible.Source: §B.4 · p. 21

B.5 Asymptotic Behavior and Bias Floor

  • Marginal annotation effects vanish at O(1/n²), while the objective approaches the irreducible floor α1ε²; in Regime II, its local minimum still beats the no-annotation value.In Regime II, annotations beyond n* increase ϕ; in Regime III and after the Regime IV maximum, they decrease it.Source: §B.5 · p. 22

B.6 Cross-Action Coupling within a Context

  • Within a context, adding an action annotation changes the objective by δJ = δϕ − δJ2, so its value depends on the existing aggregate bias and cannot be decided independently across actions.The sign of the linear correction depends on Bs · ε: the correction can strengthen or weaken the local benefit.Source: §B.6 · pp. 22–23

B.7 Extension to Arbitrary Context-Action Pairs

  • The local four-regime and diminishing-marginal results omit J2; joint allocation must account for within-context bias coupling and the shared budget.Theorem 5 provides a first-annotation condition using the current aggregate bias when other allocations are held fixed.Source: §B.7 · p. 23

B.8 The Case nf(s, a) = 0

  • When nf = 0, annotation bias is never diluted and the objective is nonincreasing after the first annotation; improvement over baseline requires ε² < (µr − r0)².Thus annotation bias must be strictly smaller in magnitude than the default model’s error for net improvement to be possible.Source: §B.8 · p. 24

C Properties of the Objective Function

  • The full objective is non-negative and bounded, non-separable across actions within a context, neither convex nor concave in its continuous relaxation, and neither submodular nor supermodular.Source: C Properties of the Objective Function · p. 25

C.1 Decomposition and Notation

  • For a context-action pair (s, a), when N > 0, W(s, a) = σ²/N + QD/N²; when N = 0, B(s, a) = r0(s, a) − µr(s, a) and W(s, a) = 0.Source: §C.1 · p. 26

C.2 First and Second-Order Derivatives

  • J1 and J3 decompose over context-action pairs, while J2 decomposes over contexts; at zero counts, finite annotation changes use default definitions rather than rational derivative formulas.The derivative analysis is organized around these decomposition patterns.Source: §C.2 · p. 26
  • The Hessian identities follow by applying the quotient rule to the first-order expressions, with the differentiated pairs assumed to have positive total counts.The proof treats derivatives for the same pair, the same action, and different actions.Source: §C.2 · pp. 26–27
  • For different actions, each B term depends only on annotation counts for its own action, so cross-action second derivatives arise only from products of first derivatives.This isolates the cross-action contribution in the Hessian calculation.Source: §C.2 · p. 27
  • The compact Hessian matrices for J1 and J3 are sums of rank-one and rank-two matrices, and their spectral properties depend on parameters.The proof refers the spectral analysis to §C.3.Source: §C.2 · p. 28

C.3 Structural Properties

This section characterizes the objective’s nonnegative, bounded, nonseparable, and nonconvex structure, and explains how MM updates achieve descent using separable local costs. It also states termination and approximation-error guarantees for fixed-count source search, with exactness in the zero-count case.

C.3 Structural Properties

  • The objective J(n) is nonnegative for every feasible allocation and has a finite maximum over the finite feasible set.Nonnegativity follows from J1 − J2 ≥ 0 and J3 ≥ 0.Source: §C.3 · p. 28
  • A valid two-action instance has a nonzero cross-action finite difference, ruling out any additive per-action decomposition of J.For the constructed instance, J(1, 1) − J(1, 0) − J(0, 1) + J(0, 0) = −1/8.Source: §C.3 · p. 29
  • A single valid instance makes the continuous relaxation neither convex nor concave, and makes J on feasible integer allocations neither submodular nor supermodular.The Hessian has opposite-sign curvature at two feasible configurations; mixed finite differences violate both lattice inequalities.Source: §C.3 · p. 30

D. Analysis of the Majorization-Minimization Algorithm

  • The MM method retains the best feasible candidate, while its fallback update ensures monotone descent and convergence even without exact surrogate minimization.Its candidate search uses local source swaps and dynamic programming; the surrogate is separable across context-action pairs, with convex local cost at fixed total count.Source: §D.1 · pp. 30–31, §D.2 · p. 34, §D.3 · p. 34

D.3 Local Source Search and Exact Count Aggregation

  • The fixed-denominator quadratic extension is convex on the simplex, but need not be strictly convex because its Hessian has rank at most one.Source: §D.3 · p. 35
  • At fixed m, r(s, a)/N0 is constant across source mixtures but varies across counts and must be retained in H♯.Source: §D.3 · p. 35
  • At m = 0, the source simplex is a singleton, so the stored local cost equals the exact cost without a swap search.This also holds when n_f(s, a) = 0; the local gap is zero.Source: §D.3 · pp. 35–36
  • The source search accepts at most Tswap swaps and, without a swap cap, terminates at a swap-local output exactly when δ_s,a(m, λ) = 0.For K = 1 there are no swaps, and the sole source mixture is optimal.Source: §D.3 · p. 35
  • When n_f(s, a) > 0, the approximation gap η_s,a(m) is bounded by α1(s, a)ΔE^2_s,a/[4n_f(s, a)] for every m.For m = 0, the local gap is zero; the count-specific bound also remains valid when n_f(s, a) = 0 and m ≥ 1.Source: §D.3 · pp. 35–36

D.4 Lagrangian Relaxation Analysis

The appendix analyzes the MM solver’s budget relaxation, approximation and descent properties, then reports synthetic-bandit setup and convergence and source-subset checks.

D.4 Lagrangian Relaxation Analysis

  • Dynamic programming aggregates annotation counts exactly by action; Lagrangian relaxation decomposes the problem over contexts, while multiplier search returns a feasible candidate in exact arithmetic.The multiplier search eventually returns the zero allocation or an earlier budget-feasible candidate.Source: §D.4 · p. 36

D.5 Per-Iteration Approximation Error

  • Each MM iteration’s optimality gap is bounded by majorization, inner-greedy, and Lagrangian-relaxation errors.The majorization gap is bounded uniformly; the greedy error certificate includes remaining swap gains when local search stops at its cap.Source: §D.5 · p. 37

D.6 Convergence Guarantees

  • With fixed inputs and deterministic tie-breaking, every changed allocation strictly decreases J, and unrestricted zero-tolerance iterations reach an unchanged allocation finitely.A positive tolerance or finite iteration limit can stop the algorithm before it reaches a fixed point.Source: §D.6 · pp. 37–38

D.7 Computational Complexity

  • Algorithm runtime scales with MM iterations, multiplier trials, contexts, actions, source swaps, and count-aggregation dynamic programming.The per-iteration subproblem-call count includes the zero-multiplier trial, doubling, and bisection.Source: §D.7 · p. 38

E.1.1 Environment and source profiles

  • The synthetic clinical bandit uses 100 contexts, five actions, 1,000 fitting records, and a 10,000 annotation budget.Behavior policies are sampled from a Dirichlet distribution, and the target policy retains a behavior-selected action with probability 0.9.Source: §E.1.1 · p. 39

E.1.2 Source Subset Comparison Under Consistent Optimization

  • With MM optimization across source subsets, the qualitative conclusions remain unchanged; Figure 3’s single-source allocations are optimized rather than random.The multi-source bars are unchanged from Figure 2.Source: §E.1.2 · p. 39, Figure 3 · p. 39

E.1.3 Convergence Trace on the Synthetic Bandit

  • On the synthetic clinical bandit, scaled MSE drops sharply in each subset’s first MM iteration and stabilizes within roughly 5–6 iterations.As more sources become available, the optimizer starts with the lowest-bias source and assigns a small share to higher-variance, lower-cost sources.Source: §E.1.3 · p. 39

E.2.1 Dataset, Context Space, and Action Space

The ASSISTments experiments build a clustered context and action space, define a constructed reward and LLM annotation pipeline, and evaluate allocation and solver benchmarks.

E.2.1 Dataset, Context Space, and Action Space

  • ASSISTments interactions are clustered into 100 contexts and 10 actions, the retained problem_type × original combinations with at least 100 observations each.The dataset contains 6,123,270 interactions; filtering response times retains 6,117,279 rows (99.9%).Source: §E.2.1 · pp. 39–40, Table 5 · p. 40

E.2.2 Reward Function

  • The constructed reward multiplies a data-driven Rasch-model component by an LLM-predictable time-efficiency discount that favors response times near each action’s median.The discount is symmetric, decays linearly in log-space, and floors at 0.3 for extreme mispredictions.Source: §E.2.2 · pp. 40–41

E.2.3 LLM Prompt Design

  • LLM prompts batch 10 context-action pairs, populate pair-specific fields from precomputed lookups, and use 20 independent queries at temperature 0.8 per pair.Mean prompt length is about 3,500 characters, with per-batch overhead below 1 ms.Source: §E.2.3 · p. 41, Figure 6 · p. 42

E.2.4 LLM Annotation Allocation by MM

  • Across four LLM sources, GPT-4o-mini has the lowest mean absolute bias and cost, while GPT-o3-mini costs about 78× more without improving reported quality summaries.Bias and variability estimates use 20 predicted rewards per context-action pair; Table 6 reports those estimates and API costs.Source: §E.2.4 · p. 43, Table 6 · p. 43
  • The optimized three-source configuration {Gemini, Claude, GPT-4o-mini} achieves J = 79.06, a 10.77% reduction over the no-annotation baseline J(0) = 88.61.Figure 8 compares variance objectives across LLM source subsets under MM optimization.Source: §E.2.4 · p. 43, Figure 8 · p. 44

E.3 Exact-Solver Benchmarks and Runtime

  • Small-instance gaps are 0.05% mean and 0.26% maximum; full-instance gaps to Gurobi lower bounds are 0.2807% and 0.0050% in 5.3 and 522 seconds.The full-instance runs did not certify global optimality within the one-hour limit.Source: §E.3 · p. 44

E.4 End-to-End Evaluation with Finite Pilots

With finite pilot estimates, greedy-initialized MM reduces empirical MSE in both tested environments; a bootstrap check finds similar reference-profile reductions, with stated validation limits.

E.4 End-to-End Evaluation with Finite Pilots

  • Greedy-initialized MM reduces empirical MSE by 19.13% on Synthetic and 8.47% on ASSISTments versus no annotation.MSE is evaluated using independent final OPE datasets after allocations are selected from finite-pilot estimates.Source: §E.4 · p. 45, Table 8 · p. 45

E.5 Within-pair bootstrap

  • Bootstrap-resampled profiles yield a 10.765% average reference-profile MSE reduction versus 10.774% with original profiles, but this is not held-out population validation.The evaluation does not measure uncertainty in the simulator’s factual reward reference.Source: §E.5 · p. 45

F.1 Broader Impacts

  • Unreliable annotations or violated evaluation assumptions may produce inaccurate policy comparisons; application-specific validation, including subgroup outcomes, is important before deployment.The stated consequences include treatment or instruction decisions in healthcare and education.Source: §F.1 · p. 46

F.2 Existing Data and Model Services

The appendix identifies the existing student data and hosted LLM services, and records checklist disclosures on reproducibility, study scope, and broader impacts.

F.2 Existing Data and Model Services

  • The experiments use existing ASSISTments 2012–13 records and hosted Gemini-2.5-Flash, GPT-4o-mini, and Claude-Haiku-4.5 services, with no recruitment or interventions.The data-use conditions require reporting and deleting inadvertently discovered personally identifiable information and prohibit using it.Source: §F.2 · p. 46

NeurIPS Paper Checklist

  • The authors identify contextual-bandit scope, reliance on known or estimated source profiles, and the semi-synthetic ASSISTments environment as limitations.Sensitivity to estimated profiles is analyzed in Appendix A.4 and evaluated in Section 7.3.Source: NeurIPS Paper Checklist · p. 47
  • The checklist reports that a GitHub repository provides code, public data sources, preprocessing scripts, and a detailed README for reproducing experiments.Section 7 and Appendix E describe environments, preprocessing, source-profile estimation, and the separation of pilot, fitting, and evaluation samples.Source: NeurIPS Paper Checklist · pp. 48–49
  • The checklist says results report means and standard deviations across 100 independent environment initializations and include bootstrap experiments for robustness.The theoretical statements’ assumptions and proofs are provided in Appendices A–D.Source: NeurIPS Paper Checklist · pp. 48–49

10. Broader impacts

  • The broader-impacts discussion covers benefits of more efficient policy evaluation, risks from inaccurate policy comparisons, and the need for application-specific validation.The experiments use synthetic data and secondary analysis of public ASSISTments records, without new recruitment or interventions.Source: §10 · pp. 51, 53
Loading 2610.10974v1…