Statistical Properties of Deep Neural Networks with Dependent Data

• Theorem 1 gives convergence-in-probability rates without requiring stationarity, while Theorem 2 provides non-asymptotic bounds on theoretical and empirical L2-errors under stationary β-mixing data.Source: §1 · pp. 2–3
• The results accommodate general sieve extremum estimation and avoid basis-function and sieve-space conditions that are impractical for adaptive DNNs with diverging depth.They rely instead on entropy with bracketing or interpolation between L∞ and L2 norms.Source: §1 · p. 3
• The DNN applications use fully connected feedforward networks with continuous piecewise-linear activations, unconstrained parameters, and width and depth that grow with sample size.The paper states results under a Hölder smoothness condition and focuses on ReLU while extending to any continuous piecewise-linear activation.Source: §1 · pp. 3–5
• In regression, Theorems 3 and 4 give L2-error results for nonstationary α-mixing and stationary β-mixing data, respectively; with bounded regressors, the rate differs from Farrell et al. (2021, Theorem 1) under i.i.d. data only by a poly-logarithmic factor.The paper notes this result may be useful for inference in some semiparametric settings.Source: §1 · p. 4
• For classification, the paper derives convergence rates for logistic binomial autoregression with covariates and presents this as the first result on DNN statistical properties with dependent data.Source: §1 · p. 4
• The paper excludes recurrent and adaptive architectures, and its DNN results under β-mixing require exponential decay of mixing coefficients.It describes its framework as more general than approaches imposing parameter constraints or structural assumptions on regression functions.Source: §1 · pp. 4–5

1 Introduction

• Section 2 sets out the estimation setting and main results for general sieve estimators.Source: §1 · p. 5
• Section 3 describes the DNN architectures studied and applies Section 2’s results to nonparametric regression and classification, with discussion of alternative architectures.Source: §1 · p. 5
• Section 4 concludes and discusses future research; the appendix gives measurability results, technical proofs, and notation.Source: §1 · p. 5

2 General sieve estimators

• The general dependent-data sieve framework covers nonparametric time-series regression and logistic autoregression, with broader versions considered as applications.The examples include conditional-mean regression and binary-response autoregression.Source: §2 · p. 6

2.1 Sieve extremum estimation and measurability

• Under pointwise compactness and continuity conditions, Proposition 2 ensures a measurable sieve estimator, including for sieve spaces whose L^r metric need not be compact.This addresses the measurability problem that can arise when sieve spaces are uncountable; the result is motivated by DNN sieves with unbounded weights.Source: §2.1 · p. 9

2.2 Convergence rates without stationarity

• For possibly nonstationary data, Theorem 1 gives a convergence-in-probability rate of ρ_n( f̂_n, f0 ) = O_P*(ε_n) under its stated conditions.The theorem applies to pointwise-separable sieve classes and controls sieve complexity using covering numbers.Source: §2.2 · pp. 9–10
• The nonstationary convergence-rate framework uses regularity conditions more general than those typically used in nonparametric sieve estimation, including curvature and Lipschitz requirements on the criterion.Allowing ρn to vary with n is often necessary for nonstationary data.Source: §2.2 · pp. 11–12

2.3 Nonasymptotic error bounds with stationarity and β-mixing

• Under strict stationarity and β-mixing, Theorem 2 gives non-asymptotic probability bounds for sieve-estimator L2(PZ) errors, controlling complexity with pseudo-dimension.The theorem extends to dependent data taking values in unbounded sets.Source: §2.3 · pp. 12, 14, 15
• Under Corollary 1’s stated conditions, the nonlinear ARCH example yields a strictly stationary, geometrically β-mixing sequence.Its coefficients satisfy β(j) ≤ C′βe^-Cβj.Source: §2.3 · p. 13
• Theorem 2 extends Farrell et al. (2021, Theorem 2) using independent blocking and localization with scale-insensitive complexity measures, supporting DNN estimators with unbounded parameters.The proof combines stationarity and β-mixing with an exponential inequality.Source: §2.3 · p. 14

3 DNN estimators

• The target is Hölder-smooth with continuous derivatives through order p−1 bounded by 1, weaker than structural assumptions in much DNN literature (e.g., Kohler & Krzy˙zak 2017; Schmidt-Hieber 2020; Kohler & Langer 2021).Source: §3 · pp. 16–17

3.1 DNN sieve spaces (Nn)

• The sieve uses fully connected feedforward DNNs, and the results apply to any continuous piecewise-linear activation function.ReLU is used for concreteness.Source: §3.1 · p. 17

3.2 DNNs for nonparametric regression

• For stationary β-mixing data, Theorem 4 provides non-asymptotic probability bounds on empirical and theoretical DNN-estimator errors.Source: §3.2 · p. 20
• Theorem 3 applies Theorem 1 to obtain a convergence-in-probability rate for DNN estimators in general nonparametric regression with nonstationary α-mixing data, strictly slower than n^-1/4.Source: §3.2 · pp. 21–22
• Under bounded outcomes and sufficiently fast θ_n convergence, Theorem 4 yields rates as fast as n^(-p/(2(p+d))) log^6(n), differing by log^2(n) from Farrell et al. (2021, Theorem 1).Theorem 4 also provides non-asymptotic error bounds that can imply faster convergence in probability; its rate does not depend on the mixing-coefficient decay rate when mixing is geometric.Source: §3.2 · pp. 22–23
• Theorem 4 rates can support √n-asymptotic normality for a finite-dimensional parameter after DNN estimation of conditional expectations in partially linear regression with stationary β-mixing data.Brown (2024) demonstrates this application for unbounded regressands.Source: §3.2 · p. 23

3.3 DNNs for binomial autoregressions with covariates

• Theorem 5 gives a DNN convergence rate for dependent-data classification, matching the Farrell et al. (2021, Theorem 1) i.i.d. rate up to a logarithmic factor under nonstationary α-mixing.The rate is unaffected by the α-mixing coefficient’s decay rate, and the theorem permits general forms of dependence.Source: §3.3 · pp. 24–25
• Corollary 2 gives two routes to mixing conditions for the binomial autoregression: strict exogeneity permits general noise distributions, while faster-mixing noise and covariates allow some endogeneity.In the latter case, covariates may depend on current noise if they are independent of past noise.Source: §3.3 · pp. 25–26

3.4 Extensions to alternative DNN architectures

• Theorems 3–5 extend to fully connected feedforward DNNs using any continuous piecewise-linear activation with a fixed number of breakpoints, including leaky ReLU.The approximation result extends by increasing network width by only a constant factor, while the cited complexity results also apply.Source: §3.4 · pp. 26–27
• The framework covers feedforward DNNs with varied activation functions and graph structures when an approximation result like Lemma 11 is available.Units may take inputs from any earlier layer; this extension is formalized in Corollary 4.Source: §3.4 · p. 27
• Corollary 4 gives complexity bounds for networks with piecewise-linear, piecewise-polynomial, or sigmoid activations.The bounds define Ξ_n,φ in terms of network parameters and computation units for the listed activation classes.Source: §3.4 · pp. 27–28
• Theorem 1 or 2 can be applied using these bounds when its conditions hold and a suitable approximation result verifies the relevant condition.The required approximation result is analogous to Lemma 11 and is needed to verify Condition (a.1) or (b.2), respectively.Source: §3.4 · p. 28

4 Summary and extensions

• General sieve-estimator results extend DNN analysis to non-i.i.d. data with general dependence and values in unbounded sets.The framework is flexible and applies across a wide range of dependent-data settings.Source: §4 · p. 28
• The analysis covers regression and classification DNNs with fully connected feedforward architectures, unbounded parameters, and sample-size-growing width and depth.Corollary 3 extends the results from ReLU to any continuous piecewise-linear activation, including leaky ReLU.Source: §4 · pp. 28–29
• Brown (2024) shows that a finite-dimensional parameter estimator using DNN-estimated nuisance components achieves √n-consistency and asymptotic normality under dependent data and unbounded regressands.The example is a partially linear regression model.Source: §4 · p. 29
• The sieve framework could support theoretical research on recurrent neural networks under dependence once their approximation power and complexity are better understood.RNNs are a key future direction for time-series settings, where theoretical results remain limited, especially for general recurrent architectures and dependence settings.Source: §4 · p. 29
• The paper leaves computational efficiency, alternative architectures and regularization, and convolutional networks for future work.Convolutional neural networks are noted as standard in applications such as image recognition.Source: §4 · pp. 29–30
• The results may be adapted beyond standard Hölder smoothness using approximation results such as Imaizumi & Fukumizu (2019), a direction left for future research.Source: §4 · pp. 29–30

A Notation

• For nonnegative sequences, ≲ means bounded above by a constant multiple eventually; the appendix also defines related asymptotic and standard measure-theoretic notation.It introduces closure, product and Borel σ-algebras, and outer probability.Source: A Notation · p. 31

MIXING COEFFICIENTS

• For a random sequence, the β- and α-mixing coefficients are written βZ(j), αZ(j), or simply β(j), α(j) when unambiguous, following Dehling & Philipp 2002, Definition 3.1.Source: MIXING COEFFICIENTS · p. 32

B Measurability of extrema of random functions

• Lemma 2 establishes measurability of extrema under its stated measurability and graph assumptions, generalizing Stinchcombe & White (1992, Theorem 2.17-a) without requiring H to be Souslin.The lemma assumes a complete probability space and a complete, separable metrizable ambient space, with H contained in that space.Source: B Measurability of extrema of random… · p. 33
• For sieve spaces Fn, pointwise separability lets Lemma 2 apply; the proof establishes closedness and a measurable graph for the associated correspondence.The correspondence is formed from the vectors of function evaluations at the sample points.Source: B Measurability of extrema of random… · p. 34
• For the function-class optimization in Proposition 2, continuity and compactness support a measurable selector from the optimizing correspondence.The proof also uses pointwise separability and a Polish function space.Source: B Measurability of extrema of random… · pp. 34–35

• Lemma 3’s setup assumes some c ≥ 2 and Hn ⊂ Fn, with c ϵn ≤ ρn(f, ˜fn) for each n ∈ N and all f ∈ Hn, under (a.1), (a.2), and (a.3)(iii).It then states its conclusion for any δ ≥ 0 and n ∈ N.Source: C Proof of Theorem 1 · p. 36
• The proof uses the δ-packing number, defined as the maximum size of a δ-separated subset, to construct a δ-cover from a maximum-size separated set.Source: C Proof of Theorem 1 · p. 37

D Proof of Theorem 2

• The proof applies localization to bound sieve-estimator L2 error, extending to more general estimation settings than the similar steps of Farrell et al. (2021).Source: D Proof of Theorem 2 · p. 42

Appendix D.7 lists the ancillary lemmas used in this section. As before, we write qt(f) :=

• The notation includes m_nt := m_n(Z_t).Source: Appendix D.7 · p. 42

D.1 Main decomposition

• The proof’s main decomposition separates the error into bias, empirical-process, and truncation terms.Source: §D.1 · p. 42

D.2 Truncation term

• The truncation term is bounded by 2µ_n using stationarity and the triangle inequality.Source: §D.2 · p. 43

D.3 Bias term

• The bias-term argument uses stationarity, the minimizing property of f0 over F, and assumptions (b.2)–(b.3).Source: §D.3 · p. 44

D.4 Independent blocks

• Berbee’s Lemma yields independent, identically distributed blocks of length a, although observations within each block may remain dependent.Source: §D.4 · p. 45

D.5 Localization analysis

• The localization setup defines Rademacher and empirical Rademacher complexities using i.i.d. symmetric signs independent of the W_t values.Source: §D.5 · pp. 46–47

D.5.1 Step I: Quadratic process bound

• Step I shows that under certain conditions on r, the quadratic-process bound implies ∥f − f0∥T1 ≤ 2r with probability greater than 1 − e^-δ′.The conditions on r are specified elsewhere in the proof.Source: §D.5.1 · p. 47

D.5.2 Step II: Radius one step tightening

• Given an initial radius r0 at least as large as the estimator’s L2 error, a sufficiently loose r0 yields a tighter bound r1 < r0 with high probability.The step assumes the stated conditions on r0 and δ′; the tightening can be repeated while each new radius continues to meet those conditions.Source: §D.5.2 · pp. 49, 54
• The Step II tightening can be repeated as long as each new radius satisfies the initial conditions for r0.Source: §D.5.2 · p. 54

D.5.3 Step III: Radius tightening lower bound

• Step III constructs a critical radius that is a reasonably tight lower bound for radii to which Step II’s tightening applies.Source: §D.5.3 · p. 54
• For the constructed radius, Step III obtains P(E) ≥ 1 − 1/n by verifying the conditions needed to apply the preceding bound.It states that 2r satisfies (D.13) and (D.14) for δ′ = log(n).Source: §D.5.3 · p. 54

D.5.4 Step IV: Localization

• Step IV uses induction over localization levels, with the accumulated term 5J e^-δ′ equal to e^-δ for the chosen δ′.The passages also state that the induction establishes the relevant property for all j.Source: §D.5.4 · p. 57
• Step IV reduces completion of the proof to showing that the localization radius r* is at most C ε_n(δ, a) for a suitable constant C.The passage defines ε_n as in Theorem 2.Source: §D.5.4 · p. 57

D.6 Empirical error bound

• The empirical-error-bound proof completes Theorem 2 by combining bounds for intermediate terms, including P3 and P4, with prior inequalities.The derivation invokes stationarity and applies earlier results to obtain the needed bounds.Source: §D.6 · pp. 59–60

D.7 Supporting Lemmas

• Lemma 6 shows that measurable transformations preserve upper bounds on both α- and β-mixing coefficients, following Davidson (2022, Theorem 15.1).For W_t = U(Z_t), the lemma states α_W(j) ≤ α_Z(j) and β_W(j) ≤ β_Z(j).Source: §D.7 · p. 60
• Lemma 7 uses an exponential inequality from Merlevède et al. (2009, Theorem 1); stationarity is unnecessary, and the result also applies to β-mixing processes.The passage notes β(j) ≥ α(j).Source: §D.7 · pp. 60–61
• Lemma 8 gives a contraction inequality for Rademacher complexities of sums, under a Lipschitz condition relating differences in g_t to differences in h_t.Its setup uses pointwise separable function spaces and i.i.d. Rademacher sequences; it follows Maurer (2016, Theorem 3).Source: §D.7 · pp. 61–62
• Lemma 9 is stated for uniformly bounded function classes with n ≥ Pdim(G), r ∈ [1, ∞], and δ ∈ (0, 2B].The supplied passage does not include the bound itself.Source: §D.7 · pp. 62–63
• Lemma 10 concerns pointwise separable classes of pointwise bounded functions; its proof invokes Hoeffding’s inequality and Dudley’s entropy integral for a zero-mean sub-Gaussian empirical process.The supplied passages do not show the lemma’s resulting bound.Source: §D.7 · p. 63

• The appendix presents proofs for Lemma 1, Proposition 3, and Section 3’s main theorems, then lists ancillary lemmas in E.4.Source: E Proofs for Section 3 · p. 64
• For Proposition 3(i), the proof uses stationarity and α-mixing of |Yt| to invoke Leadbetter et al. (1983), Condition D(un) and Theorem 3.4.1.Source: E Proofs for Section 3 · p. 65
• For Proposition 3(ii), the proof standardizes Yt, uses measurability and a standard normal distribution to apply Bosq (1998, Corollary 1.1).Source: E Proofs for Section 3 · p. 65
• The proof also bounds correlations by υ|i−k| and uses the assumptions on α(j) to apply Leadbetter et al. (1983, Theorem 6.3.4).Source: E Proofs for Section 3 · pp. 65–66

E.1 Proof of Theorem 3

• The approximation bound from Lemma 11 and (3.3), together with the lower bound on ϵ_n, verifies condition (a.1) of Theorem 1.The proof notes that scaling ϵ_n by a constant does not change the final rate.Source: §E.1 · pp. 66–67
• By (E.5), 1{m_n(Z_t) ≥ M_n} ≤ 1{|Y_t| ≥ B_n} for all n and t, which the proof uses to verify condition (a.3)(iii).The subsequent bound for f in N_n is combined with (E.6).Source: §E.1 · p. 68
• For δ > 0 and n ≥ 4, Lemma 7 and Merlevède et al. (2009, Theorem 1) provide a constant C′ depending only on α.The proof also states that a constant C in the subsequent bound does not depend on n or δ when B_n ≲ n^K_B.Source: §E.1 · p. 69
• Under the stated lower bound on ϵ_n and the assumption K_B < 1/4, Lemma 12 is applied with 4M_n = B_n to obtain a sufficient condition for (a.4)(ii).The proof also uses the definition of λ_n^(q).Source: §E.1 · p. 69

E.2 Proof of Theorem 4

• Theorem 4 follows from Proposition 4 by choosing υ = 0, under Assumptions 1 and 2 and the stated stationary β-mixing condition.The proof applies Theorem 2 after verifying its conditions.Source: §E.2 · pp. 70–71
• The proof verifies the approximation condition using a network approximation bound of order n^-(p/(p+d))(1/2−KB−υ).The cited passages state this bound and require 1/2 − KB − υ > 0.Source: §E.2 · p. 71
• Theorem 2 yields probability conclusions for both ∥f̂n − f0∥2 and ∥f̂n − f0∥2,n, with the latter following by the same reasoning.The supplied passages do not expose the probability threshold or rate expression.Source: §E.2 · p. 73

E.3 Proof of Theorem 5

• Theorem 5 is proved by showing that the conditions for Theorem 1 hold under the proof’s stated setting.The available passage also says condition (a.1) follows by the same reasoning as its earlier verification.Source: §E.3 · p. 73

Appendix E.1.

• For condition (a.2), the proof bounds the relevant terms and sets mn = 2B and Mn = 3B.These choices are stated to establish (a.2)(i) and make (a.2)(ii) and (iii) hold trivially.Source: Appendix E.1 · p. 74
• The process {qt(f)} inherits the mixing properties of {Zt}, enabling application of Merlev`ede et al. (2009, Theorem 1).The appendix also verifies that a condition-dependent expression tends to zero as n → ∞.Source: Appendix E.1 · pp. 74–75

E.4 Supporting lemmas

• Yarotsky’s result supplies a ReLU network approximating f0 within δ, and sufficient sieve depth and width make that network representable in Nn.The approximation is feasible under the stated bounded-weight condition when ∥f0∥∞ ≤ 1 and Bn ≥ 2.Source: §E.4 · p. 76
• With non-decreasing layer widths comparable to Hn, Lemma 13 gives parameter count Wn ≍ Hn^2.Source: §E.4 · p. 77
• Under logarithmic depth and width Hn ≍ n^η log^2(n), Pdim(Nn) ≍ n^(2η) log^7(n).The lemma assumes non-decreasing layer widths comparable to Hn and η > 0.Source: §E.4 · p. 78
• Lemma 15 assumes non-decreasing polynomially bounded weight limits, logarithmic depth, width Hn ≍ n^η log^2(n), and an ≥ Pdim(Nn) eventually.It also introduces a positive sequence δn ≤ 1 eventually; the supplied passages do not show the lemma’s conclusion.Source: §E.4 · p. 78

F Independent block construction

• The construction extends the probability space by a product space and coordinate projection, preserving the distributions of the original block variables.Source: F Independent block construction · pp. 79–80
• Berbee’s Lemma yields a copy of each selected block with the same distribution that is independent of the other sequence elements and previously constructed blocks.The process repeats across blocks until the sequence has the desired properties.Source: F Independent block construction · p. 80