PaperlayerView exact revision on arXiv

Source-linked AI summary

Online Inference in Distributional Temporal-Difference Learning

Yang Peng, Liangyu Zhang

arXiv:2608.14408v1stat.MLcs.LG

TL;DR

Policy-evaluation inference has focused on mean returns, while distributional features such as variability and downside risk also matter. This paper develops online inference for return-distribution functionals from a single Markov trajectory and proves that the averaged estimator and its bootstrap share a Gaussian limit, with local CDF theory for nonsmooth targets.

  • Problem

    Inference for policy evaluation has focused on the conditional mean return, which does not determine distributional features such as variability, downside risk, or adverse-outcome probabilities.

    Source: §1 · p. 1

  • Method

    The paper uses nonparametric distributional TD on a single Markov trajectory and develops bootstrap inference, including local CDF expansions for nonsmooth functionals.

    Source: §1 · pp. 2–3

  • Results

    The root-T averaged estimator and conditional bootstrap difference share a centered Gaussian limit in Cramér space, while nonsmooth CDF-equation functionals receive sampling and bootstrap limits.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support bootstrap inference for smooth return-distribution functionals and inference for nonsmooth CDF-equation targets such as return quantiles.

    Source: §1 · p. 2

    The theory assumes a fixed policy and an irreducible, aperiodic Markov chain on a finite state space.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

We study online statistical inference for functionals of the return distribution under a fixed policy. The return distribution is estimated by nonparametric distributional temporal-difference learning from a single Markov trajectory. For the Polyak--Ruppert averaged estimator, we prove that its root-$T$ error converges weakly to a centered Gaussian random element in Cramér space. We also prove that, conditionally on the observed trajectory, the root-$T$ difference between the bootstrap and original averages converges weakly to the same Gaussian limit. These results justify bootstrap inference for smooth statistical functionals, including variance, CVaR, expected shortfall, and expectiles. For nonsmooth statistical functionals, we develop a local asymptotic theory for the estimated return CDF over $T^{-1/2}$-neighborhoods of finitely many thresholds, together with its bootstrap analogue. This theory allows us to conduct inference for nonsmooth statistical functionals characterized by CDF equations, including return quantiles.

1 Introduction

The paper develops online inference for functionals of return distributions estimated by nonparametric distributional TD from a single Markov trajectory. It establishes Gaussian and bootstrap limits for smooth functionals, alongside local CDF theory for nonsmooth functionals defined by CDF equations.

1 Introduction

  • Mean-return inference does not capture variability, downside risk, or adverse-outcome probabilities, which can differ across policies with similar expected returns.Prior statistical inference for policy evaluation has focused on the conditional-mean value function [20] [21] [22].Source: §1 · p. 1
  • The paper develops online inference for return-distribution functionals using nonparametric distributional TD from a single Markov trajectory.Targets include variance, CVaR, expected shortfall, expectiles, quantiles, and parameters defined by CDF equations.Source: §1 · pp. 1–3
  • The method applies functionals to a Polyak–Ruppert average of distribution estimates and bootstraps it by multiplying the recursion step size by an independent multiplier.The distributional TD recursion updates the estimate at the visited state while leaving other state components unchanged.Source: §1 · p. 2
  • For smooth functionals, the averaged estimator has a centered Gaussian limit in Cramér space, and the conditional bootstrap difference converges to the same limit.These results support inference for smooth statistical functionals.Source: §1 · p. 2
  • For nonsmooth functionals characterized by CDF equations, sampling and bootstrap CDF expansions over T^-1/2 neighborhoods yield corresponding limits.The theory applies to return quantiles and other CDF-equation functionals.Source: §1 · p. 2

2 Preliminaries

The preliminaries define a fixed-policy return-distribution target via a contractive Bellman equation, then describe its single-trajectory nonparametric TD estimator and online multiplier bootstrap.

2.1 Return distributions and the distributional Bellman equation

  • With bounded rewards and an irreducible, aperiodic fixed-policy chain, the return-distribution vector is the unique fixed point of a √γ-contractive Bellman operator in supremum Cramér metric.Source: §2.1 · pp. 3–4

2.2 Online distributional TD learning

  • Distributional TD estimates that fixed point nonparametrically from a single policy-generated trajectory, using step sizes αt=a(t+t0)^−κ, 1/2<κ<1, and Polyak–Ruppert averaging for inference.Source: §2.2 · p. 4

2.3 Online multiplier bootstrap

  • The online multiplier bootstrap maintains replicate iterates using each observed transition and independently weighted step sizes, with multipliers taking 0 or 2 with probability 1/2 each.Source: §2.3 · p. 4

3 Inference for smooth functionals

The section establishes a Gaussian limit for the Polyak–Ruppert averaged distributional TD estimator in Cramér space and shows that the online multiplier bootstrap consistently reproduces it. The functional delta method extends these limits to inference for smooth functionals.

3 Inference for smooth functionals

  • The averaged estimator has an asymptotic linear representation and Gaussian limit in Cramér space, which the online multiplier bootstrap consistently reproduces.The analysis represents distributional TD error as a zero-mass signed measure and derives the recursion underlying the Polyak–Ruppert analysis.Source: §3 · p. 5
  • The functional delta method transfers the Gaussian and bootstrap limits to means, variances, CVaR, expected shortfall, and expectiles.Source: §3 · p. 5

3.1 Zero-mass signed measures and error recursion

  • The framework defines each measure’s cumulative function as its mass on [0, x] and completes the Cramér-norm space so probability-measure differences have norm equal to Cramér distance.Source: §3.1 · p. 5

Appendix A. We let T π and T π

The paper establishes Gaussian asymptotics for the Polyak–Ruppert average, shows that the online multiplier bootstrap conditionally reproduces the same limit, and derives inference for smooth functionals. Its appendix supports the asymptotic analysis by establishing bounded invertibility and controlling terms in the averaged error recursion.

Appendix A. We let T π and T π

  • The appendix establishes that the stationary-average operator A is boundedly invertible and states that the first and third terms in the summed error recursion are negligible.These properties support the analysis of the averaged error.Source: Appendix A · p. 6

3.2 Asymptotic normality in Cram´er space

  • The normalized Polyak–Ruppert estimator converges weakly in Cramér space to a centered Gaussian random element with covariance A^-1Σ(A^-1)*.The proof uses an asymptotic linear representation and a Hilbert-space martingale central limit theorem.Source: §3.2 · pp. 6, 7, 9

3.3 Bootstrap validity

  • Conditionally on the observed trajectory, the online multiplier bootstrap error converges to the same Gaussian limit when α1 ≤ 1/2.Bootstrap replicates reuse observed transitions and do not resample the trajectory or explicitly estimate A^-1 or its covariance operator.Source: §3.3 · pp. 10–11

3.4 Inferential consequences and examples

  • Hadamard-differentiable functionals admit bootstrap inference, with mean, variance, expected shortfall, and expectiles given as smooth examples.The theorem requires differentiability in Cramér geometry tangentially to the full space H.Source: §3.4 · pp. 12–13

4 Inference for nonsmooth functionals

The section develops local asymptotic theory for the return CDF and its bootstrap near finitely many thresholds to support inference for nonsmooth functionals characterized by CDF equations, including return quantiles. It also describes residual behavior for generalized-inverse estimators and exact roots.

4.1 Local asymptotics of the return CDF

  • Under 3/5 < κ < 3/4, the return CDF local process converges weakly for each fixed M < ∞ in product supremum norm, with a Gaussian limit.A global Kolmogorov–Smirnov-norm process limit need not hold under the present assumptions, so the result is formulated locally around finitely many thresholds.Source: §4.1 · pp. 15, 17

4.2 Bootstrap validity

  • Conditionally on the observed trajectory, the bootstrap local CDF process converges to the same Gaussian limit and admits an expansion at tight random local thresholds.The random-threshold expansion supports the inferential applications that follow.Source: §4.2 · pp. 17, 18, 20

4.3 Functionals characterized by CDF equations

  • For locally identified CDF-equation functionals with differentiable maps and nonsingular derivative, sampling and bootstrap estimators have Gaussian limits, and bootstrap critical values avoid covariance estimation.The theorem’s sampling covariance depends on the derivative of the defining equation and the Gaussian CDF coordinates.Source: §4.3 · pp. 20–21
  • For median absolute deviation, continuity of the density near m and m±d, positive density at m, and positive combined density at m±d ensure a nonsingular derivative and yield sampling and bootstrap limits.Source: §4.3 · p. 22
  • For an interior return quantile with continuous positive density, exact-root limits scale CDF Gaussian errors by −1/f(q); generalized-inverse estimators are tight in sampling and conditional senses.Source: §4.3 · p. 22
  • Exact roots have zero equation residuals, whereas generalized-inverse estimators have residuals op(1) and op*(1) in probability.The second convergence is in P-probability.Source: §4.3 · p. 23

A Proofs for Section 3 · A.1 Proofs for Section 3.1

The appendix establishes separability of the Cramér-space state representation and proves operator and martingale properties used in the Section 3 analysis.

A.1 Proofs for Section 3.1

  • The Cramér-space completion is separable, and the finite-state product Hilbert space H is therefore separable.The proof embeds measures isometrically into an L2 space and uses that closed subspaces of separable spaces are separable.Source: §A.1 · p. 24
  • The filtration F_t−1 contains the observed trajectory through S_t, and the current transition is drawn conditionally from the fixed policy and MDP kernel.F_t adds the current action, reward, and next state.Source: §A.1 · p. 24
  • The Bellman fixed-point equation implies that (m_t, F_t) is a bounded H-valued martingale difference sequence.Conditioning on F_t−1 gives E[m_t | F_t−1] = 0.Source: §A.1 · p. 24
  • The residuals e_t and operators A_t are uniformly bounded because returns lie in [0, (1 − γ)^−1] and 0 < α_t ≤ 1.The proof uses these bounds to control the update quantities.Source: §A.1 · p. 24
  • The operator A is boundedly invertible and satisfies the stated coercivity bound, with c0 = (1 − √γ) min_s μ^π(s) > 0.The proof obtains invertibility of I − T^π by a Neumann series and then uses A = D_μ^π(I − T^π).Source: §A.1 · p. 24
  • For bounded centered B on the finite state space, the operator Poisson equation has a bounded solution normalized to have stationary mean zero.Geometric ergodicity makes the defining series converge uniformly, and telescoping verifies the equation.Source: §A.1 · pp. 24–25
  • Averaging and geometric ergodicity establish the normalization and uniqueness of the centered solution to the operator Poisson equation.The finite-state chain's geometric-ergodicity bound also ensures uniform convergence of the series.Source: §A.1 · pp. 24–25

A.2 Proofs for Section 3.2

The appendix proves the stability and remainder bounds needed to represent the averaged estimator by Bellman-residual sums, then establishes their Hilbert-space Gaussian limit. Together these steps yield the centered Gaussian limit in Theorem 1.

Lemma A.3: Mean-square bound for the last iterate

  • Coercivity and a corrected Lyapunov function yield a scalar comparison recursion, implying an O(α_t) mean-square bound for the last iterate.The conditional expectation contracts the corrected Lyapunov function up to an O(α_t^2) term.Source: §A.2 · p. 26

Lemma A.4: Remainder bounds for Polyak–Ruppert averaging

  • Abel summation and the last-iterate bound make the first averaging remainder op(1); martingale and telescoping bounds establish the second when κ > 1/2.The normalized endpoint and sum are O_p(T^((κ−1)/2)) = o_p(1).Source: §A.2 · pp. 27–28

Lemma A.5: Hilbert-space CLT for Bellman residuals

  • Finite-dimensional martingale CLTs, followed by projection-tail tightness, establish the Hilbert-space CLT for Bellman residuals.The proof uses bounded residuals for the conditional Lindeberg condition and state-frequency convergence to control projection tails.Source: §A.2 · pp. 28–29

Proof of Theorem 1

  • In the proof of Theorem 1, combining the linear representation with Lemma A.5 gives a centered Gaussian limit with covariance A−1Σ(A−1)∗.Source: §A.2 · p. 29

A.3 Proofs for Section 3.3

This appendix proves the bootstrap results underlying Section 3.3: the averaged bootstrap error reduces to a leading multiplier sum, which has a conditional Gaussian limit and yields conditional weak convergence of the bootstrap average in H.

A.3 Proofs for Section 3.3

  • The bootstrap last iterate satisfies a mean-square bound, and the multiplier leaves the first-order Lyapunov–Poisson drift unchanged.The multiplier enters only the uniformly bounded second-order term.Source: §A.3 · pp. 29–30
  • The bootstrap Polyak–Ruppert expansion makes every summed term except the multiplier sum negligible after normalization.The proof controls the operator and state-selection remainders and establishes the remainder in P-probability.Source: §A.3 · pp. 31–33
  • The leading multiplier sum converges conditionally to a Gaussian limit, with projected convergence and tail control establishing weak convergence.The result holds for almost every observed trajectory.Source: §A.3 · pp. 33–34
  • Theorem 2’s bootstrap average converges conditionally to G in H, in P-probability.The proof combines a conditionally negligible remainder with the conditional Gaussian limit of the leading term.Source: §A.3 · p. 34

A.4 Proofs for Section 3.4 · B Proofs for Section 4

The proof derives functional inference from tight root-T estimation and Hadamard differentiability, establishes the bootstrap analogue, and obtains confidence-interval coverage.

A.4 Proofs for Section 3.4

  • Tightness of ZT in H gives ||θT − θ0||H = Op(T^-1/2), enabling a Hadamard-differentiability expansion for the functional estimator.Source: §A.4 · p. 35
  • For finitely many functionals, stacking the maps and derivatives yields the corresponding joint conclusion.Source: §A.4 · p. 35
  • Conditional weak convergence of the bootstrap process implies its unconditional weak convergence to G.Source: §A.4 · p. 35
  • Uniform Hadamard expansions on compact sets apply to the tight random directions; subtracting the expansions and using linearity yields a negligible remainder.Source: §A.4 · p. 35
  • The conditional continuous mapping theorem and the conditionally negligible remainder establish the asserted conditional weak convergence in probability.Source: §A.4 · p. 36
  • For a scalar nondegenerate Gaussian limit, conditional quantile convergence combined with the sampling result gives the stated confidence-interval coverage.Source: §A.4 · p. 36

B.1 Proofs for Section 4.1

These proofs establish the bounds and convergence results used for pointwise and local inference in Section 4.1, including controls for bootstrap iterates and conditional martingale bounds where stated. The expansion and stochastic equicontinuity hold jointly across all s in S and j = 1, ..., k, alongside the fixed-threshold limit.

B.1 Proofs for Section 4.1

  • Exact and bootstrap TD iterates have densities, yielding Lipschitz CDF iterates with the same deterministic growth bound.The bootstrap update is a nonnegative mixture, which supports the density bound for bootstrap iterates.Source: §B.1 · pp. 36–37
  • An interpolation inequality for CDFs and its statewise application yield uniform CDF last-iterate bounds for the original and bootstrap iterates.The interpolation argument uses Lipschitz continuity and monotonicity to relate pointwise CDF differences to an integral discrepancy.Source: §B.1 · pp. 37–38
  • A deterministic-grid martingale bound controls suprema over compact intervals, with a conditional-in-probability analogue for bootstrap bounds.The proof uses a grid, Freedman’s inequality, and interpolation; the conditional conclusion is stated in p25.Source: §B.1 · pp. 38–39
  • The inverse Bellman series converges absolutely and uniformly on a threshold neighborhood, while the Bellman operators and finite-state averages obey Lipschitz bounds.The operator bounds also apply to truncations of differences or finite-state averages.Source: §B.1 · pp. 39–41
  • The pointwise Polyak–Ruppert expansion follows by applying the inverse Bellman operator and controlling the step-size and random-operator remainders uniformly on shrinking threshold neighborhoods.The remainder proof truncates the inverse series and bounds normalized endpoint, Poisson, and martingale terms; the expansion then removes those terms.Source: §B.1 · pp. 41, 42, 45
  • The pointwise influence vector satisfies a martingale central limit theorem, and local stochastic equicontinuity is established by dyadic chaining and martingale Bernstein bounds.The influence sequence is bounded and has conditional mean zero; the local argument controls increments over nested grids.Source: §B.1 · pp. 46–47

Proof of Theorem 4

  • Stochastic equicontinuity holds jointly over s in S and j = 1, ..., k, and continuity of the densities yields uniform deterministic drift over the same neighborhoods.Source: §B.1 · p. 48
  • Combining the uniform expansion with the fixed-threshold limit proves the asserted convergence, with an op(1) remainder in R^|S|k.The leading martingale sum converges weakly to a centered Gaussian distribution in dimension |S|k.Source: §B.1 · p. 48

B.2 Proofs for Section 4.2

The proofs establish conditional bootstrap convergence for the local return-CDF process by controlling remainder terms, deriving a pointwise expansion and conditional central limit theorem, and proving stochastic equicontinuity near selected thresholds.

Lemmas B.12–B.13

  • The bootstrap remainder bounds are negligible under the joint and conditional laws, supporting the multiplier and expansion results.The proof also obtains a deterministic grid of size ℓ_T=O(T^(1−κ)).Source: §B.2 · pp. 49, 51

Lemma B.14

  • The bootstrap Polyak–Ruppert expansion reduces uniformly over shrinking threshold neighborhoods to a leading empirical sum after four remainder terms are removed.The expansion holds for each fixed state and threshold neighborhood U_T={z: |z−x|≤r_T}, with deterministic r_T↓0.Source: §B.2 · pp. 51–52

Lemma B.15

  • Conditionally on the data, the centered independent summands satisfy covariance convergence and the Lindeberg condition, yielding a conditional multivariate central limit theorem.The covariance convergence follows from the martingale law of large numbers and the Markov ergodic theorem.Source: §B.2 · p. 52

Lemma B.16

  • Conditional stochastic equicontinuity holds uniformly over states and selected thresholds in their neighborhoods.The proof combines the pointwise expansion with a pathwise Bernstein and dyadic-union bound.Source: §B.2 · p. 53

Proof of Theorem 5

  • The proof combines conditional fixed-threshold convergence with the sampling drift to establish the stated conditional local-process convergence.A data-measurable op(1) remainder is also op*(1) in probability.Source: §B.2 · pp. 53–54

B.3 Proofs for Section 4.3

The proofs establish uniform CDF control near the parameter and quantile values of interest, then use localization and vanishing overshoots to derive sampling and bootstrap limits for generalized quantiles.

Lemma B.17

  • Lemma B.17 extends fixed-threshold CDF-error conclusions uniformly over shrinking parameter neighborhoods, for both sampling and bootstrap errors.Source: §B.3 · pp. 54–55

Proof of Theorem 6

  • Theorem 6’s estimating-equation argument uses local invertibility and consistency to localize both estimators at the root-T scale.Source: §B.3 · p. 55
  • Random-threshold expansions and Taylor expansion of the estimating equations yield the sampling limit and the bootstrap limit conditionally in probability.Source: §B.3 · pp. 55–56

Lemma B.18

  • Under an interior quantile, a continuous density positive at that quantile, and its inclusion among evaluation points, Lemma B.18 establishes root-T localization for generalized quantiles and their bootstrap analogues.Source: §B.3 · pp. 56–57

Lemma B.19

  • Under Lemma B.18’s conditions, sampling and bootstrap overshoots vanish, yielding generalized-quantile representations with Gaussian sampling and conditional bootstrap limits.Source: §B.3 · pp. 57–59

Related papers

Loading 2608.14408v1…