PaperlayerView exact revision on arXiv

Source-linked AI summary

Model Selection for Off-Policy Evaluation via Pairwise Bellman Validation

Lars van der Laan, Vasilis Syrgkanis, Nathan Kallus

arXiv:2610.35748v1stat.MLcs.LGecon.EM

TL;DR

Off-policy evaluation lacks a directly evaluable held-out loss for choosing among fitted Q-functions and occupancy-ratio estimates, and common validation approaches can require auxiliary regressions or critic optimization. Pairwise Bellman validation uses candidate-difference critics to provide finite-sample oracle guarantees despite misspecification. On original benchmarks, unweighted PBV leads on Hopper while LSTD-Tournament leads on Deep OPE; under controlled overlap, TA-PBV leads on both benchmarks.

  • Problem

    Held-out transitions provide no supervised validation loss for comparing fitted Q-functions or occupancy-ratio estimates, while common alternatives can require auxiliary regressions or critic tuning.

    Source: §1.1 · p. 1, §3.1 · p. 4

  • Method

    PBV selects among fitted Q-function and occupancy-ratio candidates using penalized Bellman or adjoint Bellman moments against pairwise-difference critics, with oracle guarantees under misspecification.

    Source: §1.2 · pp. 2–3

  • Results

    Unweighted PBV has the lowest mean value error on Hopper while LSTD-Tournament performs best on Deep OPE; TA-PBV has the lowest mean value error on both controlled-overlap benchmarks.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Unweighted PBV avoids occupancy-ratio estimation but requires pairwise coercivity, whereas TA-PBV uses target-occupancy weighting and can add gains when overlap is favorable.

    Source: §7 · p. 10

    Under stochastic transitions, squared TD error may not select Qπ, and unbiased squared Bellman-residual estimation generally faces the double-sampling obstruction.

    Source: §3.1 · pp. 4–5

Abstract

from arXiv · show

Model selection and hyperparameter tuning pose a central challenge in off-policy evaluation (OPE) because errors of fitted Q-functions and occupancy-ratio estimates cannot be directly evaluated from held-out transitions. Bellman-residual and minimax validation methods often require auxiliary regressions or optimization over separate critic classes, adding computational and tuning costs. We introduce pairwise Bellman validation (PBV), a tuning-free minimax procedure for selecting among finite classes of fitted Q-functions and occupancy-ratio estimators using held-out transitions. PBV constructs critics from pairwise candidate differences, requiring only finitely many empirical moment calculations without auxiliary validation models or separate critic optimization. We establish finite-sample oracle inequalities relative to the best available candidate, even under candidate-class misspecification. Our key insight is that Q-function errors and log ratios between candidate and true occupancy ratios provide oracle critics whose Bellman and adjoint Bellman moments control estimation error. Pairwise candidate differences approximate these unknown critics, while penalties account for comparator approximation error. For Q-function selection, we first develop unweighted PBV, which evaluates pairwise Bellman moments under the offline distribution and admits oracle guarantees under a pairwise coercivity condition relating the offline distribution to the target-policy dynamics. We then develop target-aligned PBV (TA-PBV), which uses marginalized importance weighting to provide error control without pairwise coercivity. For occupancy-ratio selection, we develop adjoint PBV (A-PBV), which evaluates adjoint Bellman moments against log-ratio critics. On Hopper and Deep OPE, unweighted PBV performs competitively, while target-occupancy weighting can improve performance in settings with controlled overlap.

1. Introduction

The introduction presents pairwise Bellman validation as a tuning-free way to select fitted Q-functions and occupancy-ratio estimators, with finite-sample oracle guarantees despite candidate misspecification. It motivates the approach by the limits of held-out validation and distinguishes its Q-function and ratio-selection variants from prior work.

1.1. Motivation

  • Held-out transitions provide no supervised validation loss for competing Q-functions or occupancy ratios, while common alternatives add regressions, critic tuning, or optimization costs.LSTD-Tournament avoids a separate critic class, but its guarantees require realizability and conditioning assumptions.Source: §1.1 · pp. 1–2

1.2. Contributions

  • PBV selects finite classes of fitted Q-functions and occupancy-ratio estimators using penalized moments against pairwise-difference critics, without auxiliary validation models or separate critic optimization.Oracle critics based on estimation errors motivate the pairwise comparisons, yielding guarantees relative to the best available candidate even under misspecification.Source: §1.2 · p. 2
  • Unweighted PBV evaluates Q-function moments under the offline distribution and requires pairwise coercivity, while TA-PBV uses target-occupancy weighting to avoid that condition.Unweighted PBV avoids occupancy-ratio estimation; TA-PBV uses importance weighting instead.Source: §1.2 · p. 2
  • A-PBV selects occupancy-ratio estimators using adjoint Bellman residuals against log-ratio critics, without importance weighting or a coercivity condition.Its penalty is proportional to squared Hellinger distance between candidate ratios.Source: §1.2 · p. 2
  • The procedures provide finite-sample oracle inequalities for misspecified candidate classes; the stated validation rates are near-parametric, with logarithmic dependence on candidate-class sizes.For A-PBV and TA-PBV, the joint guarantees separate occupancy-ratio and Q-function approximation errors and quantify how ratio-selection error propagates to Q-function selection.Source: §1.2 · p. 3

1.3. Related work

  • LSTD-Tournament compares candidates with pairwise LSTDQ, whereas PBV uses unnormalized pairwise differences and quadratic critic penalties to support oracle guarantees under misspecification.The comparison also distinguishes PBV's pairwise-coercivity condition from LSTD-Tournament's matrix-conditioning requirements.Source: §1.3 · p. 3

2. OPE setup

The setup defines off-policy evaluation in a time-homogeneous MDP and specifies how independent validation data relate to the offline distribution. It then characterizes the target Q-function and discounted occupancy ratio through Bellman and adjoint Bellman equations.

2.1. MDP and validation data

  • OPE estimates a target policy's discounted value from data collected under another policy, given an initial state–action distribution.The setup defines a time-homogeneous MDP with a target policy and transition kernel.Source: §2.1 · p. 3
  • Candidates are fitted on training data and evaluated on an independent validation sample whose state–action sequence has stationary offline distribution ν.The setup allows a stationary behavior-policy trajectory or independent transitions.Source: §2.1 · p. 4

2.2. Bellman and adjoint Bellman equations

  • The target Q-function yields policy value by averaging under d0 and is a Bellman fixed point; the discounted occupancy ratio yields value by weighting rewards under ν and satisfies an adjoint Bellman fixed-point equation.The adjoint operator is defined through the ν-inner product relation between Pπ and its adjoint.Source: §2.2 · p. 4

3. Validation for Q-function selection

Standard validation cannot directly assess fitted Q-functions on held-out transitions without additional estimation or critic design. Pairwise Bellman validation uses candidate differences as critics, with unweighted guarantees requiring coercivity and target-aligned weighting avoiding that condition.

3.1. Limitations of standard validation criteria

  • Bellman-residual validation is not directly estimable from transitions, while squared one-step TD error can select poorly because squaring adds a candidate-dependent variance term.Direct unbiased Bellman-residual estimation also faces the double-sampling obstruction.Source: §3.1 · pp. 4–5
  • Minimax moment validation tests Bellman residuals against critics, but its usefulness depends on choosing a critic class that distinguishes candidates.Residual-normalized critics can recover Bellman-residual validation, but auxiliary critic design remains a practical challenge.Source: §3.1 · p. 5

3.2. Key ideas behind pairwise Bellman validation

  • Candidate Q-function errors provide oracle critics whose Bellman moments control squared estimation error under coercivity.The moment is linear in the Bellman residual and can be estimated from one-step transitions if the error critic were known.Source: §3.2 · pp. 5–6
  • Pairwise candidate differences approximate the unavailable error critics; worst-case comparison across K candidates requires K² moments and no auxiliary critic fitting.With a realizable candidate equal to Qπ, its difference from another candidate recovers that candidate’s estimation error exactly.Source: §3.2 · p. 6
  • Unweighted PBV’s oracle guarantees require pairwise coercivity under the offline distribution, a condition that need not hold for general offline data.Stationary on-policy sampling is a stated setting where coercivity holds automatically.Source: §3.2 · p. 6
  • TA-PBV weights pairwise moments by the occupancy ratio, yielding target-occupancy Q-error control without offline-distribution coercivity, but requires accurate ratio estimation.Under realizability, the target-occupancy contraction provides the error bound.Source: §3.2 · p. 6

4. Pairwise Bellman validation algorithms

The algorithms penalize pairwise validation moments to account for critic approximation under misspecification. Unweighted PBV uses offline-distribution moments; A-PBV selects occupancy ratios, and TA-PBV uses fitted-ratio weighting for Q-function selection.

4. Pairwise Bellman validation algorithms

  • Under misspecification, unpenalized comparisons with distant candidates can produce large moments and arbitrarily worse selected-candidate error, motivating penalties on pairwise scores.The combined A-PBV and TA-PBV procedure evaluates M² + K² pairwise scores without fitting auxiliary validation models or critics.Source: §4 · p. 7, §4.2 · p. 8

4.1. Unweighted PBV

  • Unweighted PBV evaluates pairwise moments and quadratic penalties under the offline distribution, avoiding occupancy-ratio estimation.The quadratic penalty avoids needing to know the coercivity constant; λ = 1/4 minimizes the population upper bound stated in the analysis.Source: §4.1 · p. 7

4.2. Adjoint PBV and target-aligned PBV

  • A-PBV selects occupancy-ratio candidates using adjoint Bellman moments against pairwise log-ratio critics, with Hellinger penalties; TA-PBV applies quadratic penalties to Q-function moments under the fitted ratio.The oracle critic for ratio selection is log(w/wπ), approximated by log(wa/wb).Source: §4.2 · pp. 7–8

5. Finite-sample guarantees

The paper establishes finite-sample oracle inequalities for unweighted PBV and the two-stage A-PBV/TA-PBV procedure under stated dependence and coverage conditions. Their guarantees differ in coercivity requirements and in how occupancy-ratio estimation error enters Q-function bounds.

5. Finite-sample guarantees

  • Finite-sample oracle inequalities cover unweighted PBV and the two-stage A-PBV/TA-PBV procedure, assuming a behavior-process spectral gap.The stated spectral-gap condition holds with gb = 1 for independent transitions.Source: §5 · p. 8, §5.2 · p. 9

5.1. Unweighted PBV

  • Unweighted PBV’s oracle guarantee assumes pairwise coercivity under the offline distribution and controls error relative to the best candidate.The bound’s dependence on temporal dependence reduces the effective validation sample size from n to ngb, while candidate-class size enters logarithmically.Source: §5.1 · pp. 8–9

5.2. Adjoint PBV and target-aligned PBV

  • TA-PBV gives a Q-function error guarantee without offline-distribution coercivity, with separate terms for candidate approximation, validation, and occupancy-ratio estimation error.The stated horizon dependence is (1 − γ)^−1/2 for Q-function approximation, (1 − γ)^−1 for direct validation error, and (1 − γ)^−3/2 for ratio-error terms.Source: §5.2 · p. 9

6. Experimental investigation

Across original benchmarks, the best method differs by benchmark; under controlled overlap, TA-PBV has the lowest mean value error on both benchmarks. The evaluation compares Q-selection methods on Hopper and Deep OPE, including settings designed to examine target-occupancy weighting.

6. Experimental investigation

  • The comparison includes unweighted PBV, TA-PBV, squared TD error, and LSTD-Tournament across original and controlled-overlap benchmarks.Controlled-overlap data sample one benchmark policy uniformly per trajectory and treat each component policy in turn as the target.Source: §6 · p. 9
  • The Q-classes contain 15 candidates, and A-PBV selects among 28 FORE ratio candidates before TA-PBV selects a Q-function using an independent 3,200-transition validation sample.For TA-PBV, selected ratios are tempered to reduce weight variance and improve stability.Source: §6 · p. 10
  • TA-PBV has the lowest mean value error on controlled-overlap Hopper and Deep OPE; on original benchmarks, PBV leads Hopper while LSTD-Tournament leads Deep OPE.All four bootstrap intervals for comparisons involving TA-PBV on controlled-overlap benchmarks include zero.Source: §6 · p. 10, Table 1 · p. 10

7. Discussion

The discussion presents PBV as a practical approach to offline model selection, with different requirements for unweighted and target-aligned Q-selection. The occupancy-ratio appendix develops A-PBV through pairwise log-ratio critics and a penalized worst-case selector.

7. Discussion

  • PBV and LSTD-Tournament perform competitively, while unweighted PBV avoids ratio estimation but requires pairwise coercivity and TA-PBV trades ratio estimation for avoiding that requirement.The authors recommend unweighted PBV and TA-PBV with tempered ratios as practical choices; target-occupancy weighting can add gains when overlap is favorable.Source: §7 · p. 10

AI use statement

  • Generative AI assisted manuscript drafting, editing, and proof review; the authors independently verified the mathematical arguments and take responsibility for the paper.Source: AI use statement · p. 11

A. Adjoint PBV for occupancy-ratio selection

A.1. From oracle log-ratio critics to pairwise selection

  • Log ratios between candidate ratios approximate the unknown oracle critic, and A-PBV chooses the candidate with the smallest worst-case pairwise score.Under realizability, the true ratio candidate has zero score against every comparison candidate, so the minimax selector recovers it.Source: §A.1 · p. 17

A.2. Penalized score and selector

  • A-PBV penalizes pairwise scores using squared Hellinger distance to account for error from replacing the oracle critic with a comparison candidate.Its implementation evaluates M^2 directly computable pairwise scores without additional nuisance or critic fitting.Source: §A.2 · p. 17

A.3. Theoretical guarantees for occupancy-ratio selection

The occupancy-ratio theory gives A-PBV oracle guarantees relative to the best candidate, including when the candidate class is misspecified. Its finite-sample guarantee accommodates several ways to obtain the initial-distribution moments.

A.3. Theoretical guarantees for occupancy-ratio selection

  • A-PBV's population oracle inequality compares its selector with the best candidate in generalized KL, allowing candidate-class misspecification.Source: §A.3 · p. 17
  • The finite-sample A-PBV result permits initial moments from target-distribution draws, exact expectations, or validation states when d0(ds, da) = νS(ds)π(da | s).The bound's final term is omitted when initial moments are known exactly or the validation-state average applies.Source: §A.3 · p. 18

B. Theoretical guarantees for Q-function selection

  • The Q-function theory section introduces guarantees for target-aligned and unweighted PBV, with a joint TA-PBV result additionally requiring regularity conditions and exact d0 moments or Condition C2 with m = n.Source: B. Theoretical guarantees for… · p. 18, §B.1 · p. 18

B.2. Target-aligned PBV

TA-PBV uses target-occupancy weighting and a quadratic penalty to obtain Q-function oracle control without requiring realizability. Unweighted PBV avoids ratio estimation but requires offline-distribution coercivity.

B.2. Target-aligned PBV

  • TA-PBV's population selector has error within a factor depending only on γ of the best candidate's error, without realizability.If Qπ belongs to the candidate class, every population minimizer equals Qπ in L2(dπ,γ).Source: §B.2 · pp. 18–19
  • TA-PBV's finite-sample bound combines best-candidate error, fitted-weight Hellinger error, and a validation term of order √(log(12K/δ)/(ngb))/(1 − √γ).The validation term vanishes when log K = o(ngb), for fixed γ and envelope constants; if bw = wπ, the weight-error term is zero.Source: §B.2 · pp. 19–20

B.3. Unweighted PBV

  • Unweighted PBV has a validation term of order log K/(ngb) and no weight-estimation term, but its guarantee depends on offline coercivity and an oracle comparison in L2(ν).For fixed positive coercivity and envelope constants, consistency holds when the comparator's Q-function and Bellman residual errors vanish and log K = o(ngb).Source: §B.3 · p. 20
  • Unweighted PBV's stated plug-in value-error bound requires d0 ≪ ν and a finite density-ratio norm c0,ν.Source: §B.3 · p. 20

C. End-to-end doubly robust OPE

Joint TA-PBV selects an occupancy ratio and then a Q-function on a shared validation fold, before evaluating the selected pair with a doubly robust estimator. Its end-to-end analysis covers finite candidate classes, specified sample conditions, and benchmark designs for original and controlled-overlap settings.

C. End-to-end doubly robust OPE

  • Joint TA-PBV selects the ratio and Q-function on the same validation fold, then evaluates the pair with a doubly robust estimator on an evaluation fold that may coincide with validation.Candidates are fitted on training data; initial-distribution moments use empirical samples or exact expectations.Source: C. End-to-end doubly robust OPE · pp. 20–21
  • The end-to-end oracle-bound analysis assumes stationary validation folds with a spectral-gap lower bound, selector regularity, and exact or finite-sample initial-value evaluation.For the selected ratio, Condition B4 follows from Condition A3; the bound includes a product of Q- and ratio-approximation errors and terms in the selection and evaluation error.Source: C. End-to-end doubly robust OPE · p. 21

D.1. Benchmark designs

  • Original benchmarks comprise 120 Hopper environment–policy combinations and 220 Deep OPE environment–dataset–policy combinations; controlled-overlap logging uniformly mixes benchmark policies, bounding the target-to-logging ratio by the number of policies.The controlled-overlap designs include 60 Hopper and 44 Deep OPE environment–target combinations, with five occupancy-ratio data replicates per combination.Source: §D.1 · p. 22

D.2. Candidate classes and selectors

  • Each Q-class contains 15 candidates: simulator-derived values for original Hopper and neural FQE checkpoints for Deep OPE and controlled-overlap Hopper.For TA-PBV, A-PBV first selects among 28 FORE ratio candidates; the selected ratio is held fixed during Q selection, and the primary analysis tempers it with T = 5.Source: §D.2 · p. 22
  • LSTD-Tournament is implemented with normalized candidate differences, absolute moments, and no quadratic penalty; the closed-form criterion is applied directly to candidate predictions.The implementation fits no pair-specific LSTDQ model; the criterion is undefined if any required empirical standard deviation is zero, and ties follow candidate order.Source: §D.2 · p. 23

D.3. Validation and evaluation

The evaluation varies validation-sample sizes in controlled-overlap settings and reports value-error metrics with resampling-based uncertainty. Supplementary analyses compare selector errors and examine initial-state value accuracy and TA-PBV temperature sensitivity.

D.3. Validation and evaluation

  • Controlled-overlap selectors are evaluated at nQ values from 3,200 to 51,200; uncertainty intervals use hierarchical bootstrap draws after averaging inner validation resamples.Paired comparisons preserve method pairing, and intervals hold fitted candidates and reference values fixed.Source: §D.3 · p. 23
  • Initial-state value accuracy is assessed under the initial-state distribution using reset states and independently simulated target-policy returns.For stochastic target policies, independent initial actions are needed to estimate state-value rather than action-value error.Source: §D.3 · p. 23
  • The primary metric is absolute policy-value error, while Table 1 reports initial-state value RMSE and Table S1 reports its uncertainty.The RMSE is the square root of mean estimated state-value MSE across settings and validation resamples.Source: §D.3 · pp. 23–24, Table S1 · p. 24

D.4. Supplementary analyses

  • Figure S1 compares policy-level absolute-value-error differences against LSTD-Tournament, while Figure S2 compares TA-PBV with T = 5 against unweighted PBV; negative differences favor the first-named PBV method.Both figures use nQ = 3,200 and distinguish original from controlled-overlap benchmarks.Source: Figure S1 · p. 25, Figure S2 · p. 26
  • Tables S2 and S3 report paired mean-error differences and setting-level median errors, respectively; Table S4 compares selector errors across the controlled-overlap validation-sample-size grid.Lower errors are favored in the displayed summaries; the captions do not state the winning selector or any table values.Source: Table S2 · p. 27, Table S3 · p. 27, Table S4 · p. 27

D.4.1. Validation-sample-size sensitivity under controlled overlap

  • Table S5 examines TA-PBV temperature sensitivity using mean value error and median and minimum relative effective sample size across benchmark settings.The table uses nQ = 3,200, and T = 5 is the prespecified specification.Source: Table S5 · p. 28

E. Additional related work

This section situates PBV’s nuisance-selection task among related approaches and develops tools and guarantees for several PBV criteria, including occupancy-ratio and unweighted Q-function selection. It also addresses conditions and limitations involving coverage, ratio control, candidate misspecification, normalization, and localized deviation.

E. Additional related work

  • PBV selects a supplied Q-function or occupancy-ratio nuisance for a fixed target policy, unlike methods selecting scalar values or policies.For occupancy ratios, A-PBV selects post hoc among fitted candidates using pairwise log-ratio critics and Hellinger-penalized adjoint Bellman scores.Source: E. Additional related work · p. 28

G.2. Target-relative ratio bounds

  • Existing ratio estimators fit within prescribed classes, whereas A-PBV selects among fitted candidates; its analysis supplies generalized-KL, plug-in value, and pairwise score bounds under Condition C1.The appendix also requires an upper envelope and bounded pairwise log ratios for fitted candidates.Source: E. Additional related work · p. 28, §G.2 · pp. 31–33

E. Additional related work

  • PBV replaces unavailable supervised risks with pairwise criteria, while Bellman-operator contractions yield target-error bounds without generic inverse-problem source conditions.The assumptions remain structural: TA-PBV requires coverage and ratio accuracy, A-PBV requires an upper envelope and bounded pairwise log ratios, and unweighted PBV requires pairwise coercivity. Supervised criteria such as cross-validation and Q-aggregation rely on observed outcomes or losses unavailable for Qπ and wπ.Source: E. Additional related work · p. 29

F. Concentration along the validation trajectory

  • Validation-trajectory concentration combines Bernstein bounds for bounded Markov-chain functionals with martingale concentration for one-step transition innovations.The appendix derives bounds for the predictable and martingale components under its trajectory assumptions.Source: F. Concentration along the validation… · pp. 29–30

G.1. Generalized-KL contraction and adjoint residual identities

  • The adjoint update contracts generalized KL divergence by γ, and a three-point identity for its adjoint residual supports the occupancy-ratio analysis.The contraction is stated when the divergence on the right is finite; the identity assumes common support and finite divergences.Source: §G.1 · pp. 30–31

G.3. Population oracle inequality

  • Under Condition C1, the population A-PBV analysis establishes occupancy-ratio plug-in value and pairwise score bounds, then an oracle inequality relative to the best candidate.These results rely on target-relative ratio bounds; the broader assumptions include coverage and bounded pairwise log ratios for fitted candidates.Source: §G.2 · pp. 32–33, §G.3 · p. 34

G.4. Hellinger oracle inequality

  • The Hellinger-oracle proof compares against the candidate nearest the target ratio in Hellinger distance and bounds adjoint moments using candidate-ratio assumptions.The supplied passage presents the proof setup and moment-bound step, but not the final inequality.Source: §G.4 · pp. 34–35
  • Hellinger oracle control bounds the selected occupancy-ratio error by candidate approximation error and validation and initial-moment errors, with discount-dependent factors.With exact initial moments, the bound uses validation error alone.Source: §G.4 · p. 36

H.1. Population score bounds

  • The population Q-function score comparison yields an oracle inequality by comparing the selected candidate with the candidate of smallest target-distribution error.The proof establishes integrability of the scores under Condition B1.Source: §H.1 · p. 38

H.2. Concentration bound for the Q-function criterion

  • A localized validation-score deviation bound holds simultaneously over fixed candidate pairs conditional on training and weight-construction data, when the validation fold is independent of those data.The result assumes Conditions A1 and B2–B4.Source: §H.2 · p. 39

H.3. Finite-sample oracle inequality

  • Under the stated conditions, the finite-sample Q-function criterion satisfies an oracle inequality using the localized validation-score concentration bound.The proof applies the bound over a collection of at most 2K − 1 candidate pairs.Source: §H.3 · pp. 40–41

I. Proofs for unweighted pairwise Bellman validation

  • The unweighted PBV population selector minimizes the maximum penalized pairwise Bellman score, whose population form is the counterpart of the empirical criterion.Its deterministic score bounds assume pairwise coercivity for candidate differences.Source: §I · p. 42, §I.1 · p. 42

I.1. Population score bounds

  • The cited result states that the coefficient of ∥µk∥2,ν has a unique minimum as λ varies, attained at λ = 1/4.Source: §I.1 · p. 42
  • Unpenalized pairwise validation can select a candidate whose error ratio over the best candidate grows without bound, even with κν = 1.L2-normalized pairwise critics can yield a population oracle inequality, but finite-sample guarantees require control of potentially unstable empirical normalization.Source: §I.1 · p. 43
  • Under Conditions A2 and B5, the normalized pairwise-score minimizer has a population oracle bound, with selected-candidate error controlled by the comparator’s Bellman residual and coercivity constant.The argument bounds the selected candidate’s pairwise difference norm by 2 times the comparator residual divided by κν.Source: §I.1 · p. 43
  • Empirical normalization can be unstable for nearly identical candidates and restrictive for flexible classes, whereas unweighted PBV avoids it through a quadratic critic penalty.For Sobolev-type RKHSs, interpolation inequalities typically give a fractional-power rather than linear supremum-norm bound in the L2 norm.Source: §I.1 · pp. 43–44

I.2. Concentration bound for the unweighted Q-function criterion

  • Under Conditions A1, B2, and B3, Lemma S26 gives a localized deviation bound holding with probability at least 1 − δ simultaneously over a fixed collection of ordered candidate pairs.The collection may be chosen conditional on the training sample.Source: §I.2 · p. 44

I.3. Finite-sample oracle inequality

The finite-sample proof compares the selected candidate with a fixed comparator satisfying the coercivity inequality, using a localized deviation event to establish the bound. Taking the best comparator yields the main-text guarantee.

I.3. Finite-sample oracle inequality

  • With probability at least 1−δ, a localized deviation bound holds over at most 2K−1 candidate-pair indices, with logarithmic factor log(12K/δ).Source: §I.3 · p. 45
  • The proof bounds the selected candidate by empirical comparison with the comparator, maximizing over the nonnegative critic norm.Source: §I.3 · p. 45
  • The proof establishes the main-text bound for any fixed comparator satisfying the coercivity inequality, then takes the best comparator.Source: §I.3 · p. 45
  • For the stated corollary, stationarity and Jensen’s inequality give ∥Pπf∥≤∥f∥ and Eν(f)≥(1−γ)∥f∥².Source: §I.3 · p. 45

J.1. Proof of the joint TA-PBV guarantee

The joint TA-PBV guarantee combines control of the selected occupancy ratio and Q-function, while the doubly robust oracle bound controls Q-error using both nuisance errors. The population value bias vanishes when either the Q-function or occupancy ratio is correct.

J.1. Proof of the joint TA-PBV guarantee

  • Empirically estimated initial moments are absorbed into the validation term, while known exact moments eliminate the initial-sample term.The proof also applies the Q-selection guarantee across M fixed weights and uses a union bound.Source: §J.1 · p. 46
  • The joint TA-PBV bound (8) has a constant depending only on Cw, w, HQ, GQ, and σQ.Its proof combines the ratio guarantee with weight-error control; the two events intersect with probability at least 1−δ.Source: §J.1 · p. 46

J.2. Proof of the doubly robust oracle bound

  • The selected Q-function’s error is bounded by ϵQ + Hν(bw, wπ) + ρδ/√(1−γ).This bound combines candidate Q-error, occupancy-ratio error, and a validation term.Source: §J.2 · p. 46
  • The population value bias vanishes if either Q = Qπ or w = wπ, making the functional doubly robust.The proof expresses the bias through the occupancy-ratio error multiplied by the Q-function’s Bellman error.Source: §J.2 · p. 47
Loading 2610.35748v1…