PaperlayerView exact revision on arXiv

Source-linked AI summary

Near Minimax Rate for Quantile Distributional Reinforcement Learning

Zijie Cheng, Zhihua Zhang

arXiv:2610.29595v1stat.MLcs.LG

TL;DR

As quantile resolution increases, extreme quantiles may become unstable, and it is unclear whether Wasserstein averaging offsets that instability. The paper estimates the projected quantile Bellman fixed point from an empirical MDP and analyzes its Wasserstein error and local asymptotic minimax risk. With endpoint-vanishing reward densities, resolution-independent root-n estimation persists through W2, while higher-order Wasserstein distances can have polynomial m-dependence in their first-order statistical constant.

  • Problem

    As quantile resolution increases, extreme quantiles may become unstable, and it is unclear whether Wasserstein averaging offsets that instability.

    Source: §1 · p. 4

  • Method

    The paper estimates the projected quantile Bellman fixed point from an empirical MDP and analyzes its Wasserstein error and local asymptotic minimax risk.

    Source: §1 · p. 4, §5 · p. 33

  • Results

    With endpoint-vanishing reward densities, resolution-independent root-n estimation persists through W2, while higher-order Wasserstein distances can have polynomial m-dependence in their first-order statistical constant.

    Source: §1 · p. 4, §5 · p. 33

  • Takeaways & Limitations

    Fixed-instance results show that resolution dependence can be intrinsic: optimal local risk stays bounded in a regular instance but grows under endpoint-vanishing densities for p > 2.

    Source: §4.6 · pp. 32–33, §5 · p. 33

    The population projected Bellman map need not be globally contractive in W2, and the cited corollary does not establish W∞ consistency.

    Source: The same constant also satisfies · pp. 14–15

Abstract

from arXiv · show

In this paper, we study quantile distributional policy evaluation, where the return distribution under a given policy is approximated by the quantile Bellman fixed point $\bm\eta_m$. We investigate the statistical accuracy of estimating $\bm\eta_m$ across quantile resolutions under Wasserstein distances. Assuming access to a generative model, we construct a quantile fixed point estimator $\bm\eta_m^{(n)}$ from an empirical Markov decision process and establish non-asymptotic error bounds under Wasserstein distances. When reward densities are uniformly bounded away from zero, we obtain a root-$n$ high-probability bound simultaneously over all resolutions $m$ and all Wasserstein orders $1\le p\le\infty$. When reward densities vanish at the support endpoints, we show that continuity and endpoint monotonicity suffice to preserve the same resolution-independent rate for $1\le p\le2$. For $p>2$, we establish a root-$n$ bound with coefficient $m^{1/2-1/p}$, together with a slower, resolution-independent $n^{-1/p}$ bound for every finite $p>2$. We further obtain resolution-independent $W_\infty$ consistency governed by the continuity modulus of reward quantile functions. To investigate the sharpness of these bounds, we characterize the exact local asymptotic minimax risk at each fixed resolution as the expected Wasserstein-coordinate norm of the efficient Gaussian limit and prove that our estimator attains it. We then examine the resolution dependence of this optimal local risk. A regular one-state instance has resolution-independent local risk constants, whereas a single endpoint-vanishing two-state instance exhibits growth of order $m^{1/2-1/p}$, up to logarithmic factors along a resolution subsequence, for every $p>2$.

1 Introduction

Distributional policy evaluation captures return uncertainty, but quantile estimation must balance growing resolution against instability at extreme quantiles. This paper studies resolution-dependent Wasserstein accuracy, establishing non-asymptotic bounds and fixed-resolution minimax benchmarks.

1 Introduction

  • The central question is whether Wasserstein averaging offsets the instability of increasingly extreme quantiles as the number of atoms grows.Each coordinate has mass 1/m, so coordinatewise error analysis does not determine whether that averaging compensates for small endpoint densities.Source: §1 · p. 4
  • Earlier work left unresolved which resolution dependence is intrinsic to estimation versus induced by the loss or worst-coordinate bounds.The paper addresses this question for the projected quantile Bellman fixed point across Wasserstein distances.Source: §1 · p. 4

1.1 Our Contributions

  • The analysis concerns model-based estimation of a finite-dimensional quantile approximation to a policy’s return distribution from generative-model data.The empirical estimator is the fixed point of the empirical projected distributional Bellman operator.Source: §1.1 · pp. 4–5
  • Root-n bounds are resolution-independent for all Wasserstein orders under uniformly positive reward densities, and through W2 under endpoint-vanishing densities.For p > 2 under endpoint vanishing, the bound has coefficient m^(1/2-1/p), with an alternative resolution-independent n^(-1/p) bound for finite p; W∞ consistency is also resolution-independent and governed by reward-quantile continuity.Source: §1 · p. 4, §1.1 · p. 5
  • At each fixed resolution, the quantile fixed point estimator attains the exact local asymptotic minimax risk characterized by an efficient Gaussian limit.The optimal risk constants remain bounded across resolutions in a regular instance but grow as m^(1/2-1/p), up to logarithmic factors along a subsequence, in an endpoint-vanishing instance for p > 2.Source: §1.1 · p. 5

1.2 Related Works

  • Prior distributional reinforcement learning work includes categorical and quantile representations, while this paper focuses on model-based empirical quantile Bellman fixed points and resolution-dependent error.The related-work discussion distinguishes this focus from prior results on categorical learning, quantile temporal-difference learning, and unprojected return-distribution estimation.Source: §1.2 · p. 6

2 Preliminaries

The preliminaries define the finite-state discounted MDP and return distributions, the metrics used to compare distributions, and the projected distributional Bellman equation. They also state alternative reward-density regimes and regularity conditions for the results.

2.1 Problem Setup

  • The setup uses a finite-state, finite-action discounted MDP, with reward and next-state draws conditionally independent given the state and action.A policy generates trajectories whose discounted return defines the statewise return distribution and whose expectation is the value function.Source: §2.1 · p. 7

2.2 Metrics on the Space of Measures

  • Wasserstein metrics compare quantile functions, while the Kolmogorov–Smirnov metric compares cumulative distribution functions; metrics on statewise distributions are extended by a supremum over states.The setup distinguishes finite-order Wasserstein distances and W∞.Source: §2.2 · pp. 7–8

2.3 Distributional Bellman Operator and Quantile Projection Operator

  • The distributional Bellman operator maps return distributions through reward and transition laws, and quantile projection approximates them with ordered m-atom distributions.The projected operator is a γ-contraction in W∞ and has a unique fixed point η_m.Source: §2.3 · pp. 9–10

2.4 Main Assumptions

  • The paper also assumes bounded continuous reward densities and no boundary coincidence of Bellman-shifted quantiles, supporting regularity at population midpoint quantiles.The density assumptions apply at the fixed MDP for finite-sample bounds, but only at the neighborhood center for the local minimax analysis.Source: §2.4 · pp. 10–11
  • The non-asymptotic results use either uniformly lower-bounded reward densities or continuous densities that vanish monotonically at support endpoints.Endpoint monotonicity excludes oscillatory boundary behavior but imposes no quantitative rate on density decay; the density regimes are complementary.Source: §2.4 · p. 11

3 Main Results

The section defines an empirical quantile fixed point estimator and establishes Wasserstein guarantees whose dependence on quantile resolution varies with reward-density conditions. It also characterizes the estimator’s local minimax risk and gives fixed instances showing when resolution dependence is sharp.

3.1 The Quantile Fixed Point Estimator

  • The estimator is the fixed point of the projected empirical Bellman operator built from empirical joint reward–next-state laws.The empirical joint law is not replaced by a product of its empirical marginals, and the fixed point can be obtained by iterating quantile dynamic programming.Source: §3.1 · p. 12

3.2 Uniformly Positive Reward Densities

  • Under a uniform positive reward-density lower bound, root-n estimation holds simultaneously over all resolutions and Wasserstein orders, including W∞.The result is non-asymptotic and holds on the same event for every resolution.Source: §3.2 · pp. 13–14

3.3 Endpoint-Vanishing Reward Densities

  • With endpoint-vanishing reward densities, continuity and endpoint monotonicity suffice for resolution-independent root-n bounds through W2.The constants are independent of resolution, sample size, confidence level, and Wasserstein order in the stated range.Source: §3.3 · p. 14, The same constant also satisfies · p. 14

The same constant also satisfies

  • Even with endpoint-vanishing densities, W∞ consistency is resolution-independent and governed by the reward quantile functions’ continuity modulus.The modulus is evaluated at the statistical scale.Source: The same constant also satisfies · pp. 15–16

3.4 Local Asymptotic Minimaxity

  • At each fixed resolution, the local asymptotic minimax risk is the expected Wasserstein-coordinate norm of the efficient Gaussian limit, and the paired empirical-joint estimator attains it.The local alternatives range over all joint laws in the Hellinger ball, not just a selected parametric family.Source: §3.4 · p. 18

3.5 Fixed Instances and Sharp Resolution Dependence

  • A one-state instance has local risk constants of order one uniformly over resolution and Wasserstein order.The instance uses a self-transition, uniform rewards on [0, 1], and discount γ=e^-1; its result also shows the general root-n order cannot be improved.Source: §3.5 · p. 18
  • A fixed endpoint-vanishing two-state instance has local risk growth m^(1/2-1/p), up to logarithmic factors along a resolution subsequence, for p>2.For this same instance, the local coefficient remains of order one for 1≤p≤2.Source: §3.5 · p. 19

4 Proof Outlines

This section outlines proofs of the estimator’s error bounds and local asymptotic minimax characterization. It combines empirical Bellman concentration and fixed-point stability with density-adaptive arguments and norm comparisons, and examines fixed-resolution risk in regular and endpoint-vanishing examples.

4.1 Analysis under the Uniform Lower Density Condition

  • Under a uniform lower-density condition, the proof bounds empirical Bellman perturbations at the population fixed point, then applies W∞ contraction to obtain bounds for every Wasserstein order.The same event and expectation bound apply across orders because Wp ≤ W∞.Source: §4.1 · pp. 19–20

4.2 Analysis under Endpoint Vanishing

  • For endpoint-vanishing rewards, localization and density-adaptive CDF control yield resolution-independent Wp bounds for 1 ≤ p ≤ 2, without sample splitting or extra endpoint-decay assumptions.The proof uses averaged squared quantile error and local fixed-point stability; finite smaller resolutions are handled separately.Source: §4.2 · pp. 20, 24

4.3 Larger Wasserstein Orders

  • For larger Wasserstein orders, deterministic norm comparisons transfer the W2 event to bounds for all m and p on the same event.The proof separately uses bounded quantile differences to establish the finite-p > 2 bound.Source: §4.3 · p. 25

4.4 Resolution-Independent W8 Consistency

  • The W∞ proof preserves the reward-quantile continuity modulus through one Bellman update, then uses supremum-norm contraction to establish resolution-independent consistency.For fixed confidence, the resulting supremum error converges to zero in probability.Source: §4.4 · pp. 25–27

4.5 Analysis of the Local Asymptotic Minimax Risk

  • The local minimax analysis reduces distribution-valued loss to an exact coordinate form and seeks uniform expected-loss approximation beyond a CLT at the center.A measurable averaging map compares decision rules; it does not replace the estimator’s midpoint projection.Source: §4.5 · pp. 27–28
  • The paired estimator attains the local asymptotic minimax risk, equal to the expected ℓ_m,p norm of the efficient Gaussian limit.The upper bound applies over the full local ball; a finite-dimensional least-favourable submodel establishes the matching lower bound.Source: §4.5 · pp. 28–30

4.6 Resolution Dependence in the Fixed Instances

  • The regular one-state instance has local risk constants bounded above and below independently of resolution m and Wasserstein order p.A common reward-score direction supplies a positive lower constant across resolutions.Source: §4.6 · pp. 30–32
  • For 2 < p ≤ 8, the fixed two-state, one-action MRP Q* with γ = 1/2 and unit transition probabilities obeys C_{m,p}(Q*) ≤ m^(1/2−1/p) Eℓ_{m,2}(Z_m) ≤ C_{2p}(Q*)m^(1/2−1/p).Source: §4.6 · pp. 32–33

5 Conclusions

The paper establishes resolution-independent estimation rates under positive reward densities and through W2 when endpoint densities vanish, while higher-order Wasserstein distances can be more sensitive to resolution. It also characterizes the fixed-resolution local minimax risk and shows that its resolution dependence can be bounded or grow, depending on the instance.

5 Conclusions

  • Root-n estimation remains possible uniformly over all resolutions and Wasserstein distances when reward densities are uniformly positive.Source: §5 · p. 33
  • With endpoint-vanishing reward densities, the resolution-independent rate persists through W2, while higher-order distances can have stronger resolution dependence.Continuity and endpoint monotonicity suffice through W2 without quantitative boundary-decay restrictions.Source: §5 · p. 33
  • Resolution-independent W∞ consistency is governed by the continuity modulus of reward quantile functions.Source: §5 · p. 33
  • The quantile fixed point estimator attains the exact local asymptotic minimax risk at each fixed resolution; fixed-instance risk can remain bounded or grow polynomially up to logarithmic factors.The growth occurs under endpoint-vanishing reward densities and reflects intrinsic statistical difficulty at a fixed MDP.Source: §5 · p. 33

A Proofs of Supporting Results in Section 4

The appendix proves the supporting results from Section 4 in the order used there, assembling technical results from Appendix B.

A Proofs of Supporting Results in Section 4

A.1 Proof of Lemma 4.1

The proof compares a hybrid target with the population target by coupling reward laws within strata and controlling transition-weight differences. It obtains a pathwise bound with one instance-dependent constant for all resolutions.

A.1 Proof of Lemma 4.1

  • The hybrid target is an exact rewriting of the paired empirical law, not a change of estimator; the expectation bound uses no union over m.Source: §A.1 · pp. 38–39
  • Reward laws in nonempty strata are coupled to their population laws with discrepancy at most dn/c0, using the same action, successor state, and continuation atom.Empty strata have zero empirical weight and do not affect the coupling.Source: §A.1 · p. 38
  • A CDF comparison and inverse bracketing bound the quantile discrepancy uniformly over states and coordinates, using an instance-dependent constant independent of m.The inverse inequality is attributed to Cheng et al. [7].Source: §A.1 · p. 39
  • The pathwise error satisfies ∆n ≤ CM(dn + rn), with one instance-dependent constant valid for every m.Projection is nonexpansive in W∞.Source: §A.1 · p. 39

A.2 Proofs of Lemmas 4.2–4.5

The proofs combine localization and empirical-process control to establish fixed-point bounds without paying a union-bound cost over quantile resolutions. They also handle midpoint-grid shifts in both small- and large-shift regimes.

A.2 Proofs of Lemmas 4.2–4.5

  • The localization thresholds depend only on the fixed instance and the chosen ℓ.The proof selects resolution and error thresholds so the fixed point remains in the common neighborhood.Source: §A.2 · p. 40
  • Along the interpolation path, convexity and a nonnegative row-stochastic coefficient matrix enable Jensen’s inequality in the contraction argument.The path is absolutely continuous, and the resulting bound uses the maximum over states.Source: §A.2 · pp. 40–41
  • For shifts smaller than one quarter-cell, averaging over the midpoint grid controls the envelope terms and rank shift; larger shifts use midpoint quantization, whose error is dominated by the shift.The two regimes together establish the stated midpoint-grid conclusion.Source: §A.2 · pp. 42–43
  • The reward-CDF event yields bounds with constants independent of m, n, and t, without a union bound over resolutions, quantile coordinates, or inputs.The event applies at data-dependent empirical fixed points within the common neighborhood.Source: §A.2 · p. 46

A.3 Proof of Lemma 4.6

The proof controls empirical reward quantiles through the continuity modulus of the reward quantile functions and controls transition-weight perturbations uniformly in resolution. Combining these controls gives the stated W∞ bound.

A.3 Proof of Lemma 4.6

  • For each nonempty stratum, the empirical reward quantile discrepancy in W∞ is at most ωR(dn).The bound follows by bracketing quantiles at probabilities shifted by dn.Source: §A.3 · p. 46
  • The hybrid and population target CDFs differ by at most rn min{F(x), 1 − F(x)}, yielding a quantile comparison uniform in m.For small rn, the proof brackets each midpoint quantile using probability shifts and applies a logit-neighborhood bound; larger rn uses the common support.Source: §A.3 · pp. 46–47
  • Combining the reward and transition comparisons after projection proves the stated W∞ perturbation bound (38).The reward comparison is governed by ωR(dn), while the transition comparison is uniform in m.Source: §A.3 · pp. 46–47

A.4 Proofs of Lemmas 4.7–4.10

These proofs establish the Gaussian limit and risk convergence for the fixed-resolution estimator, and connect its limiting risk to a local minimax bound. They also provide a finite-sample bound on empirical target quantiles.

A.4 Proofs of Lemmas 4.7–4.10

  • At a data-dependent parameter, the empirical target quantile error satisfies rn(s, i) ≤ |S|/n almost surely, improving to 1/n for deterministic transitions.Almost-sure distinctness of rewards in each state-action block limits the contributing observations.Source: §A.4 · p. 48
  • The scaled estimator error has a Gaussian limit, and its expected Wasserstein-coordinate loss converges uniformly over the stated neighborhood, including for p = ∞.The triangular-array central limit argument gives the Gaussian limit; uniform integrability upgrades it to convergence of expected loss.Source: §A.4 · p. 49
  • The canonical gradient is obtained by differentiating the target equation, and its components are bounded at each fixed resolution.Bounded mean-zero scores are dense in the tangent space used to identify the canonical gradient.Source: §A.4 · p. 49
  • The posterior-mean decision has Bayes risk √v RG, which tends to RG as τ increases, supplying the local minimax lower-bound argument.The proof uses a Gaussian prior and convex even norm loss.Source: §A.4 · p. 50

A.5 Proofs of Lemmas 4.11–4.12

The appendix proves contrasting resolution behavior for regular and endpoint-vanishing instances, alongside basic Bellman geometry and return-density properties. In the constructed two-state instance, a covariance lower bound grows along a resolution subsequence.

A.5 Proofs of Lemmas 4.11–4.12

  • For the regular instance, the relevant quantities converge to their positive limiting counterparts at rate O(1/m), with a positive lower bound across resolutions.The proof establishes the bound using an interior interval with positive mass.Source: §A.5 · p. 52
  • In a fixed endpoint-vanishing MRP, the diagonal covariance is at least c m_j (log m_j)^K along a resolution subsequence.The MRP and density perturbations are fixed before choosing the resolution sequence.Source: §A.5 · pp. 53, 55

B.1 Basic Geometry and Paired Empirical Laws

  • The Bellman operator is a γ-contraction in Wasserstein distance, while midpoint projection is nonexpansive in W∞.The proof couples actions, rewards, and successor states identically and optimally couples continuation values; the contraction properties are attributed to Bellemare et al..Source: §B.1.1 · p. 55

B.1.1 Bellman contraction and return densities

  • Return laws have bounded densities and full support on [0, B], with densities that are continuous, positive in the interior, and zero at the endpoints.Full support follows from positive-probability finite reward paths; continuity follows from translation continuity of the reward-density convolution.Source: §B.1.1 · p. 56

B.1.2 Fixed-point geometry and midpoint approximation

The section establishes existence, uniqueness, and measurability of the population and empirical fixed points, then bounds midpoint approximation and controls empirical joint sub-CDF error uniformly.

B.1.2 Fixed-point geometry and midpoint approximation

  • The population and empirical parameter maps are contractions with unique fixed points, and the empirical fixed point is measurable in the data.The unprojected return law is also identified as the unique fixed point.Source: §B.1.2 · pp. 56–57
  • W1(Πmµ, µ) ≤ B/(2m), bounding midpoint-grid approximation for distributions supported on an interval of length B.The bound follows from monotonicity of the quantile function on each grid cell.Source: §B.1.2 · p. 57

B.1.3 Uniform joint-sub-CDF control

  • A DKW-based union bound controls joint sub-CDF errors across |S|²|A| triples, with concentration constants depending only on the finite spaces.The control remains uniform at data-dependent inputs.Source: §B.1.3 · p. 58

B.1.4 Quantile graphs and finitely many resolutions

The section relates CDF error to quantile and grid error, then uses local target-density positivity to obtain stability across any fixed finite set of resolutions.

B.1.4 Quantile graphs and finitely many resolutions

  • A uniform CDF discrepancy of at most e implies |F(G^-1(u)) − u| ≤ e for continuous F and 0 < u < 1.This quantile-graph relation supports localization of inverse-CDF errors.Source: §B.1.4 · p. 58
  • For ordered arrays whose coordinates satisfy the grid CDF-error condition, the W1 distance between supported laws is at most B(e + 1/(2m)).The interval supporting both laws has length B.Source: §B.1.4 · p. 59
  • For each fixed M, positive continuous target densities near selected midpoint quantiles yield max_{1≤m≤M} W∞(η_m^(n), η_m) ≤ C_{M}D_n.The density neighborhoods and constant are instance-dependent; the argument uses inverse bracketing and the empirical contraction.Source: §B.1.4 · pp. 59–60

B.1.5 Stratifying the paired empirical law

This section develops proof tools for analyzing empirical reward laws and quantile graphs, including stratification of paired observations, density-weighted CDF error and quantile properties, and transfer of continuous CDF-error bounds to reflected CDFs. The bounds and properties are established under the stated reward-density conditions, with truncated and untruncated guarantees distinguished.

B.1.5 Stratifying the paired empirical law

  • The stratified empirical law has uniformly concentrated transition-weight and within-stratum reward-CDF errors, with every positive-probability stratum nonempty on the stated event.Conditional on the stratum counts, rewards in each nonempty stratum are i.i.d. from the population reward law; the lemma also bounds the expected sum of the two primitive errors.Source: §B.1.5 · p. 60

B.1.6 Reward inversion and relative mixture weights

  • When the population reward density is bounded below, a sup-norm CDF error d implies a reward-quantile error at most d/c0; relative mixture-weight error r bounds the mixture CDF error by r min(F(x), 1 − F(x)).The weight bound assumes both weight vectors sum to one, preserve zero weights, and have relative error below one on positive weights.Source: §B.1.6 · p. 62

B.2 Density-Weighted Empirical-CDF Bounds

  • The empirical-process supremum bound in this part is an auxiliary reward-distribution estimate inserted into the exact stratum mixture, not a mean-square bound for the Bellman fixed point.Source: §B.2 · p. 63

B.2.1 A decreasing-weight maximal inequality

  • A decreasing-weight maximal inequality bounds a weighted martingale supremum without requiring the weighted process itself to be a martingale or submartingale.Source: §B.2.1 · pp. 63–64

B.2.2 The expected density-weighted empirical error

  • For reward densities positive in the interior and monotone near both endpoints, the expected density-weighted empirical-CDF error is controlled uniformly across any fixed finite collection of reward laws.The constants may depend on that fixed collection.Source: §B.2.2 · p. 64

B.2.3 Exponential confidence by truncating the density weight

  • With probability at least 1 − e^-t, the density-weighted empirical-CDF error is bounded at scale sqrt(t/N), with a constant depending on the reward law but not N or t.The confidence bound uses a truncated density weight; it does not establish a sub-Gaussian tail for the untruncated supremum.Source: §B.2.3 · pp. 65, 67

B.3 Deterministic Quantile Inversion

  • The deterministic quantile-inversion arguments allow zero-density intervals at support endpoints and require neither density differentiability nor regularity of the perturbed CDF beyond the stated lemma hypotheses.Source: §B.3 · p. 67

B.3.1 Active support and quantile graphs

  • Under the stated support and monotonicity conditions, the population quantile is differentiable on probability levels in (0, 1), with derivative 1/f(q(u)); CDF graph selections also obey reflected inequalities.The active density is positive on an interval and bounded below on an interior interval.Source: §B.3.1 · p. 67
  • A continuous bound on the CDF difference also bounds the reflected CDF difference.The proof obtains the reflection formula from the left and right limits of a CDF.Source: §B.3.1 · p. 67
  • Endpoint conditions and monotonicity identify the active support, enable the inverse-function derivative, and yield graph inequalities that also apply at either endpoint of a flat probability level.On compact subintervals of the active support, the density has a positive minimum.Source: §B.3.1 · p. 67

B.3.2 One-sided envelopes at shifted probability levels

This section develops technical bounds for quantile Bellman stability and fixed-resolution estimation, including shifted-level envelope and grid comparisons across resolutions. It also constructs a two-state instance with endpoint-vanishing reward density and low-density intervals to localize an extreme midpoint quantile along a resolution subsequence.

B.3.2 One-sided envelopes at shifted probability levels

  • The inverse-CDF error at a shifted level is bounded by a geometric term and one-sided envelopes whose total integral is at most 14ℓz^2.The envelopes are monotone on their respective sides, and the displacement bound also holds for any graph selection of G.Source: §B.3.2 · pp. 68–69

B.3.3 From shifted probability levels to midpoint sums

  • For mε < 1/4 under the stated geometry, 0 ≤ r ≤ B gives r² ≤ Br, and both rank-shift terms vanish when ε = 0.Source: §B.3.3 · pp. 70–71

B.4 Local Geometry of the Quantile Bellman Map

  • The local neighborhood is centered at the unprojected return law, not at an m-dependent support endpoint.Assumption 4 is in force for the geometric ingredients developed in this subsection.Source: §B.4 · p. 71

B.4.1 A common positive density in the interior

  • For each fixed admissible ℓ, the densities have a common positive lower bound on the interior across resolutions, and the neighborhood is convex under coordinatewise interpolation.The bound and convexity are instance-dependent; boundary densities are monotone on the endpoint strips.Source: §B.4.1 · pp. 72–73

B.4.2 A CDF–density comparison for transition perturbations

  • An instance-dependent CDF–density envelope holds uniformly over resolutions, neighborhood inputs, states, and support points, including when endpoint intervals have zero density.The boundary comparison uses monotonicity on each endpoint strip, with the right-side inequality obtained by reflection.Source: §B.4.2 · p. 73

B.4.3 The midpoint column-average estimate

  • Midpoint logit stability is uniform over resolutions, states, and midpoint indices, with the proof controlling the derivative in logit probability.For sufficiently large resolutions, the common interior density bound and endpoint monotonicity yield a bounded derivative; finitely many smaller resolutions are handled separately.Source: §B.4.3 · p. 74
  • For sufficiently large m, the column-average estimate controls transition-density ratios by splitting midpoint quantiles into boundary and interior regions.The boundary contributions are bounded using monotonicity and reflection, while the continuous ratio mass is at most one; the supplied statement does not show the full inequality.Source: §B.4.3 · pp. 75–76
  • The last two terms in the midpoint column-average estimate vanish uniformly in the stated inputs and shifts, so boundary estimates establish (86).Choosing M_0 sufficiently large makes those terms at most ζ for m ≥ M_0.Source: §B.4.3 · p. 76

B.5.1 Empirical and Hellinger control for general joint laws

  • For general joint laws, scalar encoding of paired observations yields uniform empirical-process control without conditional independence, with constants depending on the experiment only through |S| and |A|.The appendix also records total-variation and event-probability bounds in terms of Hellinger distance.Source: §B.5.1 · pp. 76–77

B.5.2 Fixed-resolution stability and moments

  • At fixed resolution, stability and local moment bounds follow from regularity at the centre and Hellinger closeness, without requiring uniformly bounded alternative densities or differentiable quantile maps.The stability argument uses positive continuous density near the centre quantiles, inverse bracketing, and the contraction of the parameter map.Source: §B.5.2 · pp. 77–78

B.5.3 Small-interval empirical and population increments

  • Small-interval increments are controlled by reducing parameter changes to finitely many threshold bands and bounding nested empirical indicators within each band.The argument uses a conditional DKW bound and sums the resulting envelopes over the finite thresholds and blocks.Source: §B.5.3 · pp. 78–79

B.6.1 A reward density with infinitely many low-density intervals

  • A C1 reward density can have infinitely many low-density shelves while remaining Lipschitz and strictly monotone near both support endpoints.On each shelf its density is comparable to ε_j; on [0, x_j] its derivative is bounded above by Cε_j/x_j.Source: §B.6.1 · p. 80

B.6.2 Enforcing all non-boundary conditions

  • The shelf density and a Beta(2, 2) density can be perturbed to satisfy Assumption 2 for every resolution and coordinate pair while preserving endpoint monotonicity and Lipschitz continuity.In the deterministic-transition construction, every midpoint target density is positive, so the fixed-point equations are continuously differentiable at each fixed resolution.Source: §B.6.2 · p. 81

B.6.3 Downstream small balls and the extreme quantile subsequence

  • For resolutions m_j nearest to (2a_j)^-1, the localized midpoint quantile q_j lies in [x_j/2, x_j], with log m_j = e_j^2 + O(j^2).The construction also gives W_1(ν_m, ν) ≤ C/m and a lower bound ν([0, c_1 x_j]) ≥ e^(-Cj^2) for each fixed c_1 > 0.Source: §B.6.3 · p. 83

Related papers

Loading 2610.29595v1…