PaperlayerView exact revision on arXiv

Source-linked AI summary

Finite-Sample Theory for Fitted Q-Iteration When Actions Are Functions

Gefei Lin, Rui Miao, Xiaoke Zhang

arXiv:2609.36390v1stat.MEcs.LGmath.STstat.MLstat.TH

TL;DR

Offline RL theory for functional actions is limited, and conventional coverage descriptions are difficult when actions are functions. The paper develops smoothness-regularized FQI with critic-relative coverage and proves finite-sample guarantees; experiments report gains over constant-action policies.

  • Problem

    Functional-action FQI lacks finite-sample theory, while the absence of an action density and actions outside behavior support complicate conventional coverage conditions.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    The paper combines smoothness-regularized policy search with critic-relative coverage to transfer logged prediction accuracy to policy-selected actions.

    Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 4

  • Results

    The two-state example achieves regret eOp((NT)^−2/9) despite a positive action-support gap, and Pendulum mean normalized value rises from 0.587 to 0.702 for functional actions versus 0.531 to 0.560 for constant actions.

    Source: §4.2 · p. 8, §5 · p. 9

  • Takeaways & Limitations

    The analysis shows that critic-relative coverage can support polynomially decaying regret bounds for some functional-action FQI examples, even with actions outside behavior support.

    Source: §4.2 · p. 8, §6 · p. 9

    The guarantees rely on coverage conditions requiring directional information in logged data and smooth evaluation over the policy class.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Offline reinforcement learning seeks optimal decision rules from previously collected data. In some applications, a decision can be an entire function, such as a fluence map in radiation therapy or a smooth movement trajectory in robotics. In this paper, we study the finite-sample theory for fitted Q-iteration (FQI) with functional actions in a discounted infinite-horizon setting. Three major difficulties arise in this setting: first, the absence of a Lebesgue probability density for functional actions complicates coverage descriptions; second, conventional coverage requirements can be restrictive; and third, the large functional action space makes greedy optimization in FQI challenging. To address these difficulties, we study smoothness-regularized policy search under a critic-relative coverage condition. This condition measures how well logged data distinguish relevant action-value differences without requiring an action density. Our main theorem gives finite-sample guarantees for learned-policy regret relative to the best value within a fixed smooth class of functional-action policies. The results allow trajectory lengths to be either bounded or growing and Q-functions to be fitted by either functional-input kernel ridge regression or adaptive functional neural networks. For a few examples, we can obtain polynomially decaying regret bounds in the number of logged transitions, up to logarithmic factors, with logarithmically many FQI iterations. Numerical experiments show gains of learned functional-action policies over constant-action policies and support our adoption of a critic-relative coverage condition.

1 INTRODUCTION

Functional actions extend offline reinforcement learning to decisions such as fluence maps and robot trajectories, but theory for offline FQI with such actions is limited. This paper develops finite-sample guarantees and a critic-relative coverage alternative for that setting.

1 INTRODUCTION

  • Offline FQI theory for functional actions remains limited, despite applications where each decision is an entire function.Examples include radiation-intensity fluence maps and smooth robot trajectories.Source: §1 · p. 1
  • Functional actions lack a Lebesgue probability density, and smooth policy actions may lie a positive distance from behavior support, complicating error transfer.Neighborhood probabilities can decay faster than any power of their radius.Source: §1 · pp. 1–2
  • The paper develops critic-relative coverage linking policy smoothness to logged-data information, then proves finite-sample regret bounds for functional-action FQI.The bounds separate approximation, critic, policy-improvement, and finite-iteration errors and cover functional-input KRR and adaptive functional neural networks.Source: §1 · p. 2
  • For a two-state example, regret decays as (NT)^−2/9 up to logarithmic factors, even with positive distance between selected actions and behavior support.Pendulum experiments compare learned functional-action policies with constant-action policies and examine critic-relative diagnostics.Source: §1 · p. 2

2 PROBLEM SETUP FOR FUNCTIONAL-ACTION FITTED Q-ITERATION

The paper formulates discounted infinite-horizon FQI with function-valued actions and stationary deterministic policies. It fits critics from logged trajectories and regularizes policy improvement to control action smoothness.

2 PROBLEM SETUP FOR FUNCTIONAL-ACTION FITTED Q-ITERATION

  • The setting is a discounted, time-homogeneous infinite-horizon MDP with Borel states and action functions in a subset of L2([0, 1]).A policy maps each state to an entire action function.Source: §2 · p. 2
  • The data comprise N independent subject trajectories, with n = NT transitions when each trajectory has length T.The target is the value gap between the best feasible-policy value and the FQI output.Source: §2 · p. 2
  • FQI alternates Bellman-target critic regression with policy improvement using an average-Q objective and an RKHS-norm penalty on policy smoothness.Clipping the critic and regularized-objective error bound the returned policy's RKHS norm when the zero policy is feasible.Source: §2 · p. 3
  • The analysis compares learned policies within a fixed RKHS ball, so its benchmark includes an approximation gap relative to the best feasible-policy value.The penalty-induced norm bound places returned policies in such a ball.Source: §2 · p. 3

3 A COVERAGE ALTERNATIVE FOR FUNCTIONAL ACTIONS

The coverage analysis transfers critic accuracy from logged data to policy-selected actions without requiring an action density. It combines directional information in the logged actions with smoothness of the comparison policy class.

3 A COVERAGE ALTERNATIVE FOR FUNCTIONAL ACTIONS

  • Critic-relative coverage addresses the failure of action-density and neighborhood conditions for functional actions, including actions outside behavior support.The conditions combine logged directional information with smooth evaluation at policy-selected actions.Source: §3 · p. 3

3.1 FROM LOGGED PREDICTION TO POLICY EVALUATION

  • Logged prediction error and policy-evaluation error measure fitted-critic differences on different domains, with the latter taking a statewise supremum over comparison-policy actions.Target-approximation error is controlled separately because policy actions may lie outside behavior support.Source: §3.1 · p. 4
  • A uniform linear transfer may fail, so the analysis allows a Hölder bound from logged prediction error to policy-evaluation error.When the exponent is below one, the coefficient radius of analyzed critic differences must also be controlled.Source: §3.1 · p. 4

3.2 DIRECTIONAL INFORMATION AND SPECTRAL TRANSFER

  • Directional information and smooth evaluation yield transfer even when the uniform linear error ratio is infinite and policy actions lie outside behavior support.The conditions restrict coverage to relevant critic differences and connect action smoothness to weakly observed directions.Source: §3.2 · p. 5
  • In the two-state example, the transfer exponent is 8/9 and logged consistency implies evaluation consistency despite a positive action-support gap.The example's uniform linear transfer coefficient is infinite; a uniformly faster transfer rate requires stronger information or smoothness conditions.Source: §3.2 · p. 5

3.3 TRANSFER AT A PRESCRIBED ACCURACY

  • At prescribed accuracy, an amplification function can express how small logged prediction error bounds policy-evaluation error.A uniform bound of order δ^ϑ* gives the resulting transfer exponent.Source: §3.3 · pp. 5–6

4 FINITE-SAMPLE VALUE GUARANTEE

The finite-sample guarantee controls learned-policy value gap through one-step errors, state propagation, and finite iteration. Under the stated conditions, examples yield polynomial regret rates, including one with a positive action-support gap.

4.2 MAIN VALUE GUARANTEE AND EXAMPLES

  • Under uniform occupancy control and stable propagation with γCB below one, Theorem 1 bounds the learned-policy value gap after M FQI iterations.The result assumes the one-step accuracy and state-transfer conditions.Source: §4.2 · p. 7, §4.1 · p. 7
  • The value bound separates policy-class approximation, critic fitting, policy improvement, and finite-iteration errors.The finite-iteration term decreases geometrically as (γCB)^M, and further iterations have little effect once it falls below the one-step errors.Source: §4.2 · p. 7
  • The rate derivations impose geometric β-mixing on complete transition records, with independent subjects and exponentially decaying lag dependence.The examples also require the premises of Theorem 1 and event probability tending to one.Source: §4.2 · pp. 7–8
  • With logarithmically many iterations, the two-state functional-linear KRR example has regret eOp((NT)^−2/9) despite a positive action-support gap.Its fixed policy ball contains a globally optimal policy, so the approximation gap is zero.Source: §4.2 · p. 8
  • For one-score first-order smoothness, the excess over the approximation gap is eOp((NT)^−1/6); under the stated continuous-state extension, it is eOp((NT)^−1/14).These rates assume ϑ* ≥ 1/2 and zero compatibility gap for the one-score case.Source: §4.2 · p. 8

5 EXPERIMENTS: OFFLINE PENDULUM CONTROL

The modified Pendulum experiment compares smooth functional torque policies with state-dependent constant torques and uses critic-relative diagnostics to assess logged-data coverage. Functional policies achieve higher mean normalized values as sample size grows, while critic-energy ratios suggest action proximity alone may not capture transfer for the fitted critics.

5 EXPERIMENTS: OFFLINE PENDULUM CONTROL

  • Mean normalized value rises from 0.587 to 0.702 for functional actions, versus 0.531 to 0.560 for constant actions, as n increases.Policies were trained on the same 20 datasets; values were estimated using 1,000 independent Monte Carlo trajectories.Source: §5 · p. 9, Figure 1 · p. 9
  • Median nearest-action L2 distances are larger for learned actions than paired behavior actions, suggesting weaker finite-sample neighborhood coverage at learned actions.The comparison uses logged actions at the 32 nearest training states to approximate same-state matching.Source: §5 · p. 9, Figure 1 · p. 9
  • For AdaFNN, median critic-energy ratios remain near 1 across sample sizes, indicating comparable energy under learned-action and logged-pair designs.The ratio uses the adjacent fitted-critic difference; equal energy is marked by the figure’s dotted line.Source: §5 · p. 9, Figure 1 · p. 9
  • The experiment motivates assessing coverage through critic-relative error transfer rather than action proximity alone; for KRR, the ratio rises over n despite a similar distance gap.These experiments illustrate the ideas but do not verify all conditions of the value guarantee.Source: §5 · pp. 8–9

6 DISCUSSION

The discussion summarizes finite-sample value guarantees for offline functional-action FQI, comparing learned-policy regret with the best value in a fixed smooth policy class. The appendices detail the policy domain, numerical representations, and Bellman-envelope tools used in the analysis.

6 DISCUSSION

  • The paper establishes finite-sample guarantees for offline functional-action FQI by combining critic-relative information, policy smoothness, and stable state propagation.The regret benchmark is the best value within a fixed smooth policy class; examples yield polynomial decay in logged transitions, up to logarithmic factors, with logarithmically many iterations.Source: §6 · p. 9

A.1 ONE RKHS POLICY DOMAIN

  • The policy domain uses a measurable L2-valued RKHS, and the analysis localizes returned policies and critics on specified events.The common conditional model and measurability conditions support the Bellman targets and policy-envelope suprema.Source: §A.1 · p. 16

A.2 NUMERICAL POLICY REPRESENTATIONS AND THE NORM PENALTY

  • The theoretical policy update uses a deterministic RKHS-norm penalty; finite-dimensional experiments instead use empirical total curvature rather than computing that full norm.The numerical surrogate is described as coverage-compatible, while the theoretical objective remains the RKHS-norm-regularized one.Source: §A.2 · pp. 16–17

B BELLMAN ENVELOPES AND STATE TRANSFER

  • An upper Bellman fixed point bounds every policy’s value in the comparison ball, and an occupancy identity relates its value gap to the policy’s greedy defect.The fixed point is unique among bounded functions with measurable policy envelopes.Source: B BELLMAN ENVELOPES AND STATE TRANSFER · p. 17
  • The state-transfer conditions separately control final-policy visitation and one-step propagation, with stable propagation requiring γCB < 1.The fixed-point contraction in sup norm requires only γ < 1.Source: B BELLMAN ENVELOPES AND STATE TRANSFER · pp. 17–18

C.1 POLICY EVALUATION AND FUNCTIONAL-ACTION SPECTRAL ENERGY

The section connects smooth policy geometry to the spectral norm used in coverage transfer, then gives projection-based constructions for numerical policy representations. The continuous-state construction verifies feasibility and an objective bound while retaining the coupled-class approximation gap.

C.1 POLICY EVALUATION AND FUNCTIONAL-ACTION SPECTRAL ENERGY

  • The policy RKHS norm controls a stronger functional spectral norm, making policy geometry compatible with the action norm used in coverage transfer.The Sobolev construction provides one route to this control and leaves endpoint values unconstrained.Source: §C.1 · pp. 18–19
  • Two policy-RKHS constructions build spectral evaluation into the class: a weighted Legendre-feature kernel or a Sobolev action component.The Sobolev option is convenient for spline computation and the running example; the deterministic policy norm supplies the spectral radius.Source: §C.1 · p. 18

C.2 AN ENERGY PROJECTION WITH A UNIFORM ACTION ERROR

  • An energy projection onto conforming cubic splines preserves the action-RKHS budget, and radial adjustment preserves action feasibility while bounding policy-optimization loss.For Lipschitz critics, the loss from replacing an action by its feasible projection is bounded by the critic’s Lipschitz constant times the L2 action error.Source: §C.2 · pp. 19–20

C.3 A COMPATIBLE CONTINUOUS-STATE IMPLEMENTATION

  • The continuous-state implementation combines state and functional-action projections to verify membership, feasibility, and an objective bound, but retains Δ(Q).Discontinuous state-cell policies are not used as approximants in this RKHS.Source: §C.3 · pp. 20–21

D POLICY IMPROVEMENT AND SUBJECT CONCENTRATION

  • A positive policy-penalty coefficient alone does not guarantee a sample-size-independent radius; the analysis separately requires localization within the fixed policy ball.The appendix gives a sufficient low-norm comparator calibration.Source: D POLICY IMPROVEMENT AND SUBJECT… · p. 21

D.1 POLICY OPTIMIZATION ERROR AND THE ACTUAL GREEDY DEFECT

This appendix connects policy optimization and regularization errors to the greedy-defect bounds used in the value analysis, then establishes concentration inputs for equal- and unequal-length subject data. The unequal-length extension preserves the pooled empirical objective while using distinct effective sample scales for different bounds.

D.1 POLICY OPTIMIZATION ERROR AND THE ACTUAL GREEDY DEFECT

  • The policy-compatibility gap converts one average-objective policy search into the statewise L2(ν+) greedy-defect bound required by value analysis.For product policy classes with independent statewise maximization, the gap is zero; for coupled RKHS classes it requires verification.Source: §D.1 · p. 22

D.2 WHAT NORM REGULARIZATION SUPPLIES

  • A uniformly bounded low-norm near-optimal comparator localizes returned policies in a fixed RKHS ball when regularization-scale errors are controlled.The stated sufficient condition is that the resulting squared-norm bound not exceed the fixed radius squared.Source: §D.2 · pp. 22–23

D.3 EQUAL-LENGTH SUBJECTS AND JOINT COVERING INPUTS

  • Treating each subject as an independent unit yields concentration at subject-level scale N, while allowing arbitrary dependence within trajectories.Joint covers of unpenalized policy objectives supply the uniform deviation bound; the deterministic RKHS penalty contributes no deviation term.Source: §D.3 · p. 23

D.4 EXTENSION: UNEQUAL AND RANDOM TRAJECTORY LENGTHS

  • For unequal lengths, weighted subject bounds use N2 for bounded-objective deviations but N∞ for relative-risk and Gram bounds.The extension retains all logged transitions in the pooled fitting objective and requires independent subjects, not within-trajectory independence.Source: §D.4 · pp. 24–25

E DETERMINISTIC CLASSES AND RKHS POLICY ENTROPY

The appendix fixes deterministic critic, label, and policy classes before fitting, enabling uniform bounds to cover realized fitted functions and reused-data Bellman labels. It also supplies RKHS policy-entropy bounds and conditions for transferring behavior prediction errors to policy-selected actions.

E DETERMINISTIC CLASSES AND RKHS POLICY ENTROPY

  • Uniform bounds over deterministic label and fit classes allow realized same-data Bellman labels and fitted critics to be substituted together after data reuse.Target approximation must hold over the full admissible state-action domain, including actions outside behavior support.Source: E DETERMINISTIC CLASSES AND RKHS… · p. 26
  • The full continuous-state policy RKHS ball has log covering number at most Cϵ^-5/2 log(1 + C/ϵ) in the stated metric.A separate evaluation-tail condition is needed: a Mercer eigenvalue bound alone does not guarantee the uniform eigenfunction tail used for the cover.Source: E DETERMINISTIC CLASSES AND RKHS… · pp. 27–28

F SMOOTH POLICY ACTIONS AND MOMENT-BASED SECANT IDENTIFICATION

  • Smoothness of functional policies controls policy-action evaluation, while logged second moments supply the information needed to transfer critic errors.The spectral weight operator sets the scale for functional coordinates but does not assert that observed action covariance has those eigenvalues.Source: F SMOOTH POLICY ACTIONS AND… · p. 28

F.1 ACTUAL SECANTS AND EVALUATION AT A PRESCRIBED ACCURACY

  • The actual-secant transfer envelope measures policy-evaluation amplification for fitted-critic differences, and its control need not be uniform as accuracy decreases.Consistency along a sequence of fits requires the envelope at the prediction-error sequence to tend to zero.Source: §F.1 · p. 29

F.2 DIRECTIONAL MOMENTS AND SMOOTH FUNCTIONAL EVALUATION

Directional moment conditions compare information in logged data with the smoothness-weighted directions of actual critic differences, supporting transfer to functional policy actions without requiring local action densities. The appendix also gives an accuracy-dependent nonlinear example and combines the transfer bounds with regression error to control policy evaluation.

F.2 DIRECTIONAL MOMENTS AND SMOOTH FUNCTIONAL EVALUATION

  • Directional information on actual critic secants, combined with smooth-feature control, bounds policy-action evaluation error from behavior prediction error.When information weakens with evaluation accuracy, the transfer rate changes according to the information-decay exponent; the highlighted result identifies the resulting interpolation effect.Source: §F.2 · p. 30

F.3 A NONLINEAR INFORMATION VERIFICATION

  • The nonlinear example has no positive uniform directional-information constant, yet pairs with evaluation error above r satisfy an accuracy-dependent moment condition.Its ambient feature second-moment operator is singular, while the actual coefficient pairs obey a nonlinear relation.Source: §F.3 · pp. 32–33

F.4 FROM BEHAVIOR PREDICTION TO POLICY EVALUATION

  • The regression-to-policy bound combines logged prediction, full-domain approximation, and numerical input errors with a secant-transfer envelope.Approximation only on behavior support cannot control policy actions outside that support; under the stated transfer-envelope rate, the policy-evaluation exponent is 2bQ/(2 + κtr).Source: §F.4 · pp. 33–34

G GENERAL FINITE-SAMPLE VALUE GUARANTEE AND PROOF

The finite-iteration value bound controls regret relative to the fixed smooth policy class, subject to the stated data, measurability, critic-error, and policy-error conditions. When error propagation is stable, the accumulated iteration terms remain bounded uniformly in the number of iterations.

G GENERAL FINITE-SAMPLE VALUE GUARANTEE AND PROOF

  • The general finite-iteration guarantee bounds value error relative to the smooth policy class under the stated uniform critic and policy error conditions.Its bound holds for every finite χ with probability at least Pr(En).Source: G GENERAL FINITE-SAMPLE VALUE… · p. 34
  • When χ = γCB is uniformly bounded below one, accumulated errors remain bounded uniformly in M and the initialization term decays as Vmax(γCB)^M.The comparison with the base policy class enters through the explicit approximation gap aH.Source: G GENERAL FINITE-SAMPLE VALUE… · pp. 34–35
  • For coupled policy classes, the objective-based route retains the policy-compatibility gap; separated statewise optimization can instead use a direct defect bound.Either route supplies the policy-improvement error term in the finite-iteration result.Source: G GENERAL FINITE-SAMPLE VALUE… · pp. 34–35

H KRR PREDICTION AND MOMENT-BASED EVALUATION

This section develops prediction guarantees for KRR and adaptive-basis neural critics in fitted-Q settings, with policy-selected evaluation, ridge calibration, and value-rate results under stated conditions. It also addresses functional-action regression through exact secants and directional-information requirements, and gives approximation guarantees that account for numerical-score error without assuming vanishing discrepancy for raw point samples of L2 actions.

H KRR PREDICTION AND MOMENT-BASED EVALUATION

  • Functional-input kernels retain infinitely many action coordinates, while tensor calculations connect functional-policy regularity to feature energy required for secant transfer.Kernel-section Lipschitz regularity controls joint critic–policy covers and numerical objective comparisons, but does not replace the separate spectral feature condition.Source: H KRR PREDICTION AND MOMENT-BASED… · pp. 36–37

H.1 UNIFORM PREDICTION AND A CONTROLLED FITTED NORM

  • A single full-label event bounds KRR prediction and fitted norms uniformly, allowing same-data FQI iterations and supplying coefficient radii for evaluation at policy-selected actions.The evaluation transfer applies to raw RKHS secants; clipped critics are not treated as RKHS elements.Source: §H.1 · pp. 38–39

H.2 RIDGE CALIBRATION AND EXPLICIT PREDICTION RATES

  • With bounded score, operator, and transfer ratios, boundary ridge tuning gives prediction error O(ϱ_n^1/2) and evaluation error O(ϱ_n^barϑ/2).Interior tuning with vanishing ratios yields prediction error eO_p(ϱ_n^1/2) and, under bounded uniform-information transfer constants, evaluation error eO_p(ϱ_n^ϑ/2).Source: §H.2 · p. 40

H.3 AN EXPLICIT KRR VALUE RATE

  • Under deterministic covering, optimization, transfer, localization, and state-propagation conditions, KRR yields an explicit value rate; the mixing version recalibrates using the effective sample size.The stated iteration requirement is M ≥ c log N for independent subjects, or M ≥ c log(NT) for the mixing version.Source: §H.3 · p. 41

I ADAFNN PREDICTION AND EXACT FUNCTIONAL SECANTS

  • Adaptive-basis clipped ReLU fitted-Q regression uses finite-rank Bellman approximation and prediction bounds over critic and policy parameters, with evaluation based on exact secants and directional information.The state representation may be the identity when states are already finite-dimensional, as in the Pendulum experiment.Source: I ADAFNN PREDICTION AND EXACT… · p. 42
  • The method does not assert vanishing discrepancy for arbitrary raw point samples of L2 actions; exact inner products give zero discrepancy.Numerical cell-average scores are handled using regularity of the learned basis, and point evaluation on an arbitrary L2 equivalence class is not treated as a bounded functional.Source: I ADAFNN PREDICTION AND EXACT… · pp. 42–43
  • Adaptive-basis approximation bounds control reachable Bellman-target error in both behavior-design and policy-evaluation norms, retaining numerical score error.The bounds apply to the implemented class; exact scores have zero discrepancy, while the Lipschitz outer-map case yields an explicit bound on the final approximation term.Source: I ADAFNN PREDICTION AND EXACT… · p. 44
  • With probability at least 1−δ−δdep, adaptive-basis prediction bounds hold uniformly over the full deterministic label class under independent-subject or full-sample mixing designs.For independent subjects, the proposition sets the statistical sample size to N and the dependence terms to zero; the full-sample two-group construction uses its corresponding effective sample size and comparison failure term.Source: I ADAFNN PREDICTION AND EXACT… · pp. 43, 45

I.1 EXACT FUNCTIONAL-COORDINATE SECANTS WITH ALL BASES LEARNED

The section represents differences between AdaFNN critics as exact secants in outer-parameter and learned functional-basis coordinates, without freezing the bases. This representation connects functional-action norms and score accuracy to policy evaluation and its rates.

I.1 EXACT FUNCTIONAL-COORDINATE SECANTS WITH ALL BASES LEARNED

  • Fθ,β − Fθ′,β′ is represented exactly as ⟨dv, ψv⟩ along a path between endpoint critics, with bounded integrated derivatives for outer parameters and functional bases.The intermediate bases need not be micro-network outputs, so the secant does not require a frozen basis or a different fitting procedure.Source: §I.1 · p. 46
  • The secant feature’s functional-basis terms expose how an action’s L2 norm controls policy-evaluation feature energy.Observed actions require only their L2([0, 1]) norm for feature integrability.Source: §I.1 · p. 46
  • The actual-direction comparison uses the full block feature second-moment operator, retaining cross-block terms and dependence on both endpoint critics.For fixed finite bounded-weight architectures the parameter-hull derivative bound is finite, but growing width, depth, or layer norms can enlarge it.Source: §I.1 · p. 47
  • Numerical score accuracy propagates through the outer network and provides a common error budget for fitting, policy objectives, and exact-score secant comparison.The stated bound also accounts for numerical input error and is uniform over endpoints and the full domain.Source: §I.1 · pp. 47–48
  • The prediction rate reflects critic approximation and joint covering complexity, while policy evaluation adds a separate exponent ϑ or ϑ* through secant stability.For the stated specialization, evaluation error is eO(N_stat^-ϑ bQ); the scale-dependent condition replaces ϑ with ϑ* when constants are bounded.Source: §I.1 · pp. 48–49

J POLICY-NORM IMPLEMENTATION AND COVERAGE-COMPATIBLE EMPIRICAL

This section examines empirical policy-smoothness criteria and unequal-trajectory-length scaling for functional-action FQI. It relates these to coverage-compatible spectral bounds and conditional value-rate regimes, while noting that unequal-length scaling alone does not establish coverage or information conditions for the pooled design.

J POLICY-NORM IMPLEMENTATION AND COVERAGE-COMPATIBLE EMPIRICAL

  • Empirical total-curvature sublevels yield classwise spectral envelopes and, with directional-information comparisons, verify the policy-smoothness input for coverage transfer.For the implemented Pendulum splines, the stated bounded-output conditions give a finite deterministic radius at fixed dimension p_u = 12; the radius must retain its dimension dependence if the spline dimension grows.Source: J POLICY-NORM IMPLEMENTATION AND… · pp. 50–51

K A QUANTITATIVE RKHS-COMPATIBLE APPROXIMATION REGIME

  • Under bounded-factor, approximation, entropy, and numerical conditions, the RKHS regime uses ε_app,N = (κ_log,N/N)^(1/(2+d_app)) and logarithmically many FQI iterations.The composite-cover exponent d_app = max{d/s_h, 1/s_b, 1 + p_Lip/2} includes full RKHS policy complexity; the objective-to-greedy conversion contributes ε_app,N^(1/2), even when critic evaluation is faster.Source: K A QUANTITATIVE RKHS-COMPATIBLE… · pp. 51–52

K.1 CONTINUOUS STATES WITH THE COMPLETE FIXED RKHS BALL

  • For continuous states, a full fixed RKHS-ball entropy envelope yields a value rate with approximation error a_H and policy-compatibility gap Δ_εapp,N, under the stated conditions.The gap is O(ε_app,N) for the stated polynomial rate and vanishes when a globally optimal policy belongs to Π_H(B_H); a same-class measurable maximizing selector gives Δ_εapp = 0.Source: §K.1 · p. 53

K.2 RETAINING GROWING TRANSFER FACTORS

  • With growing transfer factors, the value bound has rate a_H + eO_p(ε_app,N^min{b_tr,1/2} + p_Δ ε_app,N), where b_tr depends on information decay and critic-factor growth.For β_info > 0, b_tr = 2(ϑ − p_grow)/(2 + β_infoϑ); when β_info = 0, b_tr = ϑ − p_grow. Setting p_grow = 0 recovers the bounded-factor rate.Source: §K.2 · p. 54

K.3 UNEQUAL-LENGTH INTERPRETATION

  • For the weighted unequal-length extension, the structural concentration scale is N_∞ = n/T_max, while the objective retains its actual scale N_2 = n²/Σ.The supplied passage distinguishes the two scales but does not provide the continuation of the unequal-length interpretation.Source: §K.3 · p. 54
  • For p_C < 1, a verified neural structural power p yields order eO_p(N^{-(1-p_C)p}); the conservative KRR bound uses its own scale and constants.The bound is stated for the neural structural power; the KRR scale and constants must be treated separately.Source: §K.3 · p. 54
  • With p_T = max{1+p_short, p_long+p_longlen}, the length-array calculations give n ≍ N^{p_T}, N∞ ≍ N^{p_T-p_longlen}, and N2 ≍ N^{2p_T-max{1+2p_short,p_long+2p_longlen}}.These expressions apply to N−N^p_long subjects with N^p_short transitions and N^p_long subjects with N^p_longlen transitions, where p_longlen ≥ p_short and 0 < p_long < 1.Source: §K.3 · p. 54
  • The length-array calculations do not verify policy entropy or information for the changing pooled design.For conditional random-length statements, constants must hold uniformly on the stated length event, and the event’s failure probability is added.Source: §K.3 · p. 54

L QUANTITATIVE BETA-MIXING RATES FOR THE FULL-SAMPLE ALGORITHM

This section develops finite-sample, dependence-adjusted value controls for full-sample FQI under beta-mixing dependence, using a partition that retains every transition and a shared event for statistical inputs. It also analyzes an alternative retained-block comparison with explicit omission costs and verifies the Example 1 construction, including separation between behavior-support and policy-selected actions in a weighted norm.

L.1 DEPENDENCE AND A PARTITION THAT USES EVERY TRANSITION

  • Under uniform absolute regularity of complete transition records, alternating block groups retain terminal fragments and sum to the unchanged transition-weighted objective without omission error.Nonstationary marginals and unequal terminal blocks are allowed; the shared independent-block comparison transfers each group’s uniform event once, without multiplying its cost by covers or FQI iterations.Source: L QUANTITATIVE BETA-MIXING RATES FOR… · p. 55, §L.1 · pp. 55–56

L.2 ALL THREE STATISTICAL INPUTS ON ONE EVENT

  • The full-sample block comparison establishes one event controlling objective, prediction, and optional finite-feature Gram inputs, so the temporal gain applies to the fitted full-sample procedure.When the optional empirical derivative criterion is used, the state Gram event gives a relative comparison with c− = 1/2 and c+ = 3/2.Source: §L.2 · pp. 56–57

L.3 KRR USES THE SAME DEPENDENCE EVENT, WITH ITS OWN RIDGE CALIBRATION

  • For full-sample KRR, the same block comparison yields simultaneous raw-norm and prediction bounds; if ϱn → 0 and the stated radii and objective gap are O(ϱn), then δK,n = O(ϱn^1/2).The result assumes bounded targets, full-domain RKHS targets, common conditional compatibility, and a fixed deterministic label cover; clipping does not increase error toward the bounded target.Source: §L.3 · p. 58

L.4 RETAINED-BLOCK COMPARISON AND OMISSION COSTS

  • The alternative retained-block comparison keeps the fitted objective on all transitions but incurs an O(oret) objective cost and failure probability δ + δcoup(bret, qret).For full-sample KRR, oret = O(ϱn) and vanishing coupling failure suffice for a bounded fitted norm; oret = o(ϱn) preserves the stated interior conclusion when other premises hold.Source: §L.4 · pp. 59–60

L.5 DEPENDENCE SCALES AND VALUE CONSEQUENCES

  • With bounded joint entropy and stable propagation, replacing N by Nb yields exponent p = min{ϑ*, 1/2}/(2 + D) and permits M ≥ c log Nb iterations for the stated classes.Here D = dapp for the finite-state product class or Dct for the continuous-state RKHS ball; the coupled case retains p∆εapp.Source: §L.5 · p. 60
  • With d = sh = sb = 1, pLip = 0, ϑ* ≥ 1/2, and ∆n = 0, the finite-state excess rate is (NT)^-1/6 up to logarithms.The one-score choices give DAda = 1.Source: §L.5 · p. 61
  • Under the same smoothness and margin choices, the continuous-state excess value rate is O((NT)^-1/14), reflecting larger policy-class complexity; these sufficient rates are not asserted to be optimal.Here Dct = 5.Source: §L.5 · p. 61
  • For polynomial mixing, the block length is min{T, ⌈[Cmn log²(en)]^(1/(1+ppoly))⌉}; with T ≍ N^plen, a verified power p scales as N^-max{1,(1+plen)ppoly/(1+ppoly)}p up to logarithms.Unequal trajectory lengths use Tmax as the cap and N∞ = n/Tmax in the maxima.Source: §L.5 · p. 61

M VERIFICATION OF EXAMPLE 1

  • For Example 1, the temporal calculation verifies the stronger main-text rate using the full-sample construction.The verification uses the same MDP, behavior law, Sobolev-valued policy RKHS, fixed comparison ball, and full functional linear critic kernel.Source: §L.5 · p. 61, M VERIFICATION OF EXAMPLE 1 · p. 61

M.1 THE MODEL, POLICY CLASS, BEHAVIOR SUPPORT AND MOMENTS

  • The behavior-action support has infinite ζ = 2 weighted norm, while every action selected by the localized policy class has norm at most √c_G B_H.One bound applies to all selected actions and support points.Source: §M.1 · p. 62

M.2 FULL LINEAR CRITICS AND SPECTRAL TRANSFER

These appendices verify spectral transfer and finite-sample value calculations for a full-linear critic, then relate the analysis to prior coverage work and numerical observation limits. They also show how temporal structure and cell-average observations affect the example’s guarantees.

M.2 FULL LINEAR CRITICS AND SPECTRAL TRANSFER

  • Uniform Hölder transfer holds at exponent 8/9 for bounded coefficient secants, although this family has no finite uniform linear error ratio.The exponent cannot be improved uniformly over these secants.Source: §M.2 · p. 63

M.3 STATE STABILITY AND THE GLOBALLY OPTIMAL POLICY

  • The two-state construction has transition probabilities in [3/8, 5/8] and χ² < 1; its global optimum uses actions ±BA v◦ and has value gap 1/2.Discounted contraction establishes uniqueness, and the optimal actions lie in the base class.Source: §M.3 · p. 63

M.4 FULL-LINEAR RIDGE FITTING FOR COVERAGE AND VALUE

  • Full-linear KRR yields δK,n = eO(N^-1/4) and a clipped-critic error bounded by CηN{1 + ∥a∥G}.The error bound uses prediction error for the actual KRR fits and sharpens as N grows.Source: §M.4 · pp. 63–64

M.5 POLICY-NORM SELF-LOCALIZATION AND THE VALUE RATE

  • The independent-subject construction establishes rate eO_p(N^-2/9) with at least c log N FQI iterations.The construction localizes returned policies to a fixed-radius class containing the global optimum.Source: §M.5 · p. 64

M.6 TEMPORAL REFINEMENT FOR EXAMPLE 1

  • Temporal refinement gives effective sample size n = NT with block length one and zero comparison failure when complete-record dependence vanishes at lags k ≥ 2.The independent-subject calculation remains available without this refinement.Source: §M.6 · p. 65

M.7 NUMERICAL OBSERVATIONS WITHOUT IMPOSING SMOOTH OBSERVED PATHS

  • Finite point evaluations are unbounded on arbitrary L2 equivalence classes, so uniform point-quadrature error over the whole L2 ball need not vanish.Cell-average records support a separate bound without requiring smooth observed actions; raw point samples are not converted into averages.Source: §M.7 · p. 65

N FURTHER RELATED WORK

  • Their coverage condition transfers actual critic secants over a fixed RKHS policy class and can yield vanishing nonlinear evaluation error despite an infinite uniform linear error ratio.Unlike finite-dimensional neighborhood arguments, this approach does not require logged actions to be metrically close to policy actions.Source: N FURTHER RELATED WORK · p. 66

O EXPERIMENTAL DESIGN AND IMPLEMENTATION

The experiment compares functional-action policies with a constant-torque comparator across sample sizes and evaluates selected policies with independent simulator rollouts. Its tuning objective uses empirical total curvature as a surrogate for within-action regularity.

O EXPERIMENTAL DESIGN AND IMPLEMENTATION

  • The Pendulum study compares AdaFNN and Nyström KRR functional policies with a state-dependent constant-torque FQI comparator across five sample sizes, using held-out Monte Carlo policy-value estimates.Selected policies are evaluated on disjoint sets of 1,000 trajectories, with 95% percentile intervals resampling the 20 independent datasets and training runs.Source: O EXPERIMENTAL DESIGN AND IMPLEMENTATION · p. 67, §O.1 · p. 68

O.1 DIMENSIONLESS EMPIRICAL TOTAL-CURVATURE POLICY OBJECTIVE

  • The tuned objective replaces direct computation of the policy-RKHS norm with empirical total curvature as a within-action regularity surrogate, not as a second penalty.The tuning grid uses the dimensionless ratio λΩ,n/sΩ.Source: §O.1 · p. 68
  • At n = 8000, increasing the curvature-penalty coefficient reduces mean AdaFNN policy curvature from about 13,625 to numerical zero and selected nonconstant policies from 7 to 0 of 20.The comparison spans the smallest and largest candidate penalty values.Source: §O.1 · p. 68

P ADDITIONAL ADAFNN AND KRR RESULTS

Paired evaluations show higher normalized policy values for learned functional policies than for constant-torque policies, while AdaFNN and KRR exhibit different sample-size trends. Additional diagnostics compare learned actions with nearby logged actions and characterize adjacent-critic energies on held-out and logged designs.

P ADDITIONAL ADAFNN AND KRR RESULTS

  • AdaFNN mean normalized policy values rise from 0.587 to 0.702 as n grows, whereas KRR means fall from 0.710 to 0.645 on the common grid.The state-dependent constant-torque comparator ranges from 0.531 to 0.560; the KRR diagnostics show increasing relative feature leverage and an overall increase in critic-energy ratios, with a dip from n = 4000 to n = 8000.Source: P ADDITIONAL ADAFNN AND KRR RESULTS · p. 68, Figure S1 · p. 69
  • Mean paired AdaFNN functional-minus-constant policy-value differences rise from 0.056 at n = 2000 to 0.141 at n = 32000, with pointwise intervals above zero at all five sample sizes.Evaluations pair policies within seed under a common offline-data and evaluation design.Source: P ADDITIONAL ADAFNN AND KRR RESULTS · p. 69, Figure S2 · p. 69

P.1 EMPIRICAL LOCALITY AND CRITIC DIFFERENCES

  • The median learned-action distance exceeds the behavior-action distance at every tested sample size and neighborhood size k = 16, 32, 64, and 128.The distance measures closeness to recorded actions at nearby states, not probability mass or a population support gap.Source: §P.1 · p. 70, Figure S3 · p. 70
  • The critic-energy ratio compares one realized adjacent-critic difference’s empirical squared energy on the held-out learned-policy graph and logged design; it is not uniform control of the fitted-versus-target family.Selected fits have positive design-energy denominators, with no denominator regularization.Source: §P.1 · p. 70
  • Absolute adjacent-critic energies are reported separately because the ratio alone does not show update scale, and these energies are not errors against a true or Bellman-target critic.The panels report mean squared differences for Q20 − Q19 on held-out learned-policy state–action pairs and the logged design.Source: §P.1 · p. 71, Figure S5 · p. 71
Loading 2609.36390v1…