PaperlayerView exact revision on arXiv

Source-linked AI summary

Local polynomial density ratio estimation

Hajo Holzmann, Alexander Meister

arXiv:2609.38412v1math.STstat.MLstat.TH

TL;DR

Estimating a density ratio from independent samples is important, but systematic pointwise analyses of rates and asymptotic normality have been lacking. This paper introduces a local-polynomial estimator and establishes pointwise minimax rates without logarithmic deterioration.

  • Problem

    Systematic pointwise analyses of density-ratio estimation rates and asymptotic normality have been lacking.

    Source: §1 · p. 2

  • Method

    The paper constructs a local-polynomial estimator of the density ratio, requiring smoothness of r rather than of f or g.

    Source: §1 · p. 2

  • Results

    The estimator attains pointwise minimax risk min(m, n)^-2β/(2β+d) without logarithmic deterioration.

    Source: §3 · p. 8

  • Takeaways & Limitations

    A concentration inequality for the estimator is presented as useful for applications such as classification.

    Source: §3 · p. 9

    On a specified subclass, g(x) cannot be estimated consistently, although the paper’s rate guarantees concern the density ratio.

    Source: §1 · p. 2, §3 · p. 8

Abstract

from arXiv · show

We propose a novel local-polynomial estimator of the ratio $r=f/g$ of two $d$-dimensional densities $f$ and $g$, from which independent samples are available. The estimator is shown to achieve pointwise minimax optimal rates over H\"older classes of arbitrary smoothness index without additional logarithmic factors, and with smoothness being only assumed of $r$ but not of $f$ nor $g$. Our analysis remains valid for points on the boundary of the support of the distribution associated to $g$ under a mild geometric assumption on the (unknown) boundary. We also derive a concentration inequality for the estimator, which can be useful in applications to classification, and give a rate in the supremum norm, where the supremum is also taken over boundary support points. The smoothness class over which the rate is obtained is sufficiently large that the individual densities $f$ and $g$ cannot be consistently estimated uniformly over this class. Direct estimators of the partial derivatives of $r$ together with rates of convergence are also provided. We also obtain asymptotic normality of the estimators under quite general assumptions and again including boundary points, with a consistent variance estimator allowing for data-driven studentization. Moreover, we show how our estimator can be used to estimate the Kullback-Leibler information, in a construction which additionally uses debiasing. We provide the parametric rate together with asymptotic normality under sufficient smoothness of $r$ relative to the dimension $d$.

1 Introduction

The introduction presents density-ratio estimation as important across statistical and machine-learning applications, then motivates a local-polynomial approach with pointwise guarantees and boundary coverage. It distinguishes this approach from existing density-estimator, contrast-based, and classification-based methods.

1 Introduction

  • Density-ratio estimation supports covariate-shift weighting, importance sampling, causal inference, divergence estimation, and relative-risk surfaces.For covariate shift, the ratio of source and target marginal densities is used to weight loss functions.Source: §1 · pp. 1–2
  • The proposed estimator targets pointwise minimax rates over Hölder classes of arbitrary smoothness without logarithmic factors, assuming smoothness only of r.The introduction also states validity at boundary points under mild geometric assumptions, asymptotic normality with a consistent variance estimator, and direct estimation of partial derivatives.Source: §1 · p. 2
  • Ratios of density estimators use at most Lipschitz smoothness, while some matching and k-nearest-neighbor methods also require smoothness of both densities.The introduction also notes that contrast-based approaches mainly establish global rates, often with logarithmic factors, rather than pointwise rates or asymptotic normality.Source: §1 · p. 2
  • Classification-based estimators recover the density ratio by transforming an estimated regression function from artificially labeled samples.The approach also requires care with the relative sample sizes and estimates the ratio indirectly.Source: §1 · pp. 2–3

2 The local polynomial density ratio estimator

The estimator fits a local polynomial to the density ratio using quantities from the two independent samples. Its construction accommodates boundary points under local continuity and geometric conditions, and also supports direct derivative estimation.

2 The local polynomial density ratio estimator

  • The local estimator requires f to be absolutely continuous with respect to g nearby and a continuous version of r; boundary evaluation uses a continuous extension.The paper states that its geometric condition includes boundary points of g's support.Source: §2 · pp. 3, 5
  • At polynomial order zero, the estimator reduces to a ratio of kernel density estimates using the same bandwidth.Source: §2 · p. 5
  • The empirical fit solves θ̂ = Ŝ^-1V̂, with the Gram matrix estimated from the Y-sample and the vector term from the X-sample.The intercept of the fitted polynomial gives the estimate of r(x).Source: §2 · p. 6
  • The kernel-localized least-squares importance-fitting version is presented as previously unstudied, to the authors’ knowledge.The estimator is obtained by minimizing a local least-squares criterion.Source: §2 · pp. 5–6
  • The local-polynomial approach also gives direct estimates of partial derivatives of r, unlike LSIF and nearest-neighbor methods.Source: §2 · p. 6

3 Rates of convergence

The section establishes minimax-optimal pointwise rates for density-ratio estimation over Hölder classes, then treats derivative estimation, concentration, and uniform-norm rates. It also shows that individual densities need not be consistently estimable over a relevant subclass.

3 Rates of convergence

  • The pointwise rate analysis considers Hölder smoothness of r on the local neighborhood U.The classes are introduced for β > 0 and require continuous differentiability of r through order ⌈β⌉−1.Source: §3 · p. 6
  • The approximation-bias proposition requires polynomial order l ≥ ⌈β⌉−1.Its bound is stated for (f, g) in F(x).Source: §3 · pp. 6–7
  • The section introduces a trimming sequence am↓0 for mean-squared-error analysis and requires l ≥ ⌈β⌉−1 with h < min(ρ1, ρ5).The stated mean-squared-error bound has constants independent of x, n, m, and h.Source: §3 · p. 7
  • The minimax risk for r(x) is of order min(m, n)^(-2β/(2β+d)), without logarithmic deterioration.The lower-bound argument treats the cases of fixed g and fixed f, yielding the sample-size bottleneck min(m, n).Source: §3 · pp. 7–8
  • Over a subclass satisfying the stated interior-density conditions, g(x) cannot be estimated consistently; a similar statement holds for f(x).The argument assumes a baseline density g0 strictly between the specified density bounds on the local ball.Source: §3 · p. 8
  • Derivative estimation uses trimming and bandwidth h ≍ min(m, n)^(-1/(2β+d)) when |k| ≤ ⌈β⌉−1 and l ≥ ⌈β⌉−1.The trimming sequence is chosen as in Corollary 8.Source: §3 · p. 8
  • The section gives a concentration inequality under Assumption 1 for h < min(ρ1, ρ5), motivated by potential classification applications.The same result applies to the trimmed estimator when am ≤ λ.Source: §3 · p. 9
  • A uniform-norm rate is established over compact G, and the stated rate includes boundary points of g's support.The theorem assumes a Lipschitz kernel and a trimming sequence satisfying the stated logarithmic condition.Source: §3 · p. 9

4 Asymptotic normality

The paper establishes asymptotic normality for pointwise ratio estimates, including at boundary points, and provides studentization using an estimated variance. It also treats derivative estimates and specifies when explicit variance expressions are available.

4 Asymptotic normality

  • The asymptotic-normality results include points on the boundary of g's support.The section describes the boundary inclusion as holding under its assumptions.Source: §4 · p. 9
  • Under Assumption 1 and condition (4.1), the pointwise estimator has an asymptotic normal limit when h→0 and min(m, n)h^d→∞, with estimated variance usable for studentization.The limiting distribution is standard normal after the stated standardization.Source: §4 · p. 10
  • Explicit asymptotic-variance expressions require additional regularity of f and g; at a sufficiently regular boundary, the relevant integrals are taken over the local cone.The approach in Theorem 14 avoids requiring those density regularity conditions.Source: §4 · p. 10
  • The asymptotic-normality analysis also extends to derivative estimators under the stated conditions.The section further reports a multivariate result for joint asymptotic normality of finitely many derivative estimators.Source: §4 · p. 11

5 Estimation of the Kullback-Leibler divergence

The section constructs a debiased estimator of the Kullback–Leibler divergence from a local density-ratio estimator. Under sufficient smoothness of the ratio, it establishes asymptotic normality and the parametric mean-squared-error rate.

5 Estimation of the Kullback-Leibler divergence

  • The section motivates the target as estimating the Kullback–Leibler divergence between g and f using the density ratio.It identifies Φ(x) = −log(x) for the Kullback–Leibler case.Source: §5 · p. 11
  • The construction splits the samples: an initial ratio estimate uses one part, and the remaining observations are used for the debiased functional estimate.The initial estimate is truncated, and the construction uses a first-order Taylor approximation to motivate debiasing.Source: §5 · pp. 11–12
  • Theorem 17 bounds asymptotic risk for the estimator under its stated conditions, including a compact support G and m/n → κ in (0, ∞).The theorem assumes a twice continuously differentiable functional and uses bandwidth h ≍ (n*)^-1/(2β+d).Source: §5 · p. 12
  • For β > d/2, the estimator attains the parametric n^-1 mean-squared-error rate, and its centered, scaled error converges to a Gaussian distribution.Theorem 17 also gives the asymptotic variance V(f, g).Source: §5 · p. 12
  • The smoothness condition differs from Berrett and Samworth (2023): here smoothness is imposed on the ratio r, rather than on both f and g.The section notes that the conditions are not directly comparable.Source: §5 · p. 12

6 Proofs

This section assembles preparatory lemmas and proof steps for results in Section 3. The displayed arguments establish a uniform eigenvalue lower bound for a kernel-based matrix and connect proof steps to the stated bounds.

6 Proofs

  • The proof section introduces the notation used in its arguments and directs readers to the appendix for most details and further proofs.The proof setup fixes a pair (f, g) in F(x) and writes K = K_l.Source: §6 · p. 13

6.1 Preparatory lemmas

  • Under the kernel and density conditions of Lemma 19, the minimum eigenvalue of S is bounded below uniformly over h in (0, ρ5).The lower bound depends only on ρ3, ρ4, d, and l.Source: §6.1 · p. 13

6.2 Proofs of Proposition 6, Theorem 7 and Corollary 8 from Section 3

  • The proof steps combine Lemmas 20 and 21 with Proposition 6 to obtain (3.4), and derive (3.2) from the displayed expansion and bounds.For the trimming term, the argument uses its bandwidth choice and an assumption on log(1/a_m).Source: §6.2 · pp. 14–15

A Remaining proofs for Section 3

These remaining proofs invoke lemmas and proof steps to complete results in Section 3. The excerpts identify conditions used in the arguments, but do not state the resulting bounds in full.

A Remaining proofs for Section 3

  • Lemma 23 assumes Assumption 1 uniformly over G, a Lipschitz kernel, and bandwidths with log(n)/n and log(m)/m both o(h^d).The excerpt introduces the lemma but omits its bound.Source: A Remaining proofs for Section 3 · p. 19
  • Theorem 12's proof is introduced by analogy with Theorem 7, while the surrounding steps apply Lemmas 2 and 20 and a bound from Lemma 21.The excerpts provide proof transitions but not the complete inequality.Source: A Remaining proofs for Section 3 · p. 19
  • The proof of Theorem 13 begins under the condition a_m ≤ λ/2 and follows the proof of Theorem 7 up to its displayed bound.The excerpt says the theorem follows after inserting the choice of h.Source: A Remaining proofs for Section 3 · pp. 19–20

B Proofs for Section 4 on asymptotic normality

This appendix section supplies proofs and lemmas for asymptotic-normality results. It states covariance results under additional assumptions and gives lower eigenvalue bounds for matrices used in the arguments.

B Proofs for Section 4 on asymptotic normality

  • Lemma 24 presents covariance results for the estimator components under Assumption 1 and the additional condition (4.1).It also considers the covariance matrix of S-barθ, where barθ = S^-1V.Source: B Proofs for Section 4 on asymptotic… · p. 20
  • The matrices M and N have minimum eigenvalues bounded away from zero for sufficiently small h under the stated parameter conditions.The respective lower bounds depend on ρ3, ρ4, l, d for M and ρ2, ρ3, ρ4, d, l for N.Source: B Proofs for Section 4 on asymptotic… · p. 20
  • The section considers a sequence of nonzero vectors as h tends to zero while min(m, n)h^d tends to infinity.The excerpt introduces the condition but does not include the result that follows it.Source: B Proofs for Section 4 on asymptotic… · p. 20

3. We intend to use the Lyapounov central limit theorem. Consider the pooled triangular array

The section establishes joint asymptotic normality for selected derivative estimators and supports inference using an estimated covariance matrix.

3. We intend to use the Lyapounov central limit theorem. Consider the pooled triangular array

  • The proof’s covariance argument uses convergence of the estimated sandwich covariance and a population minimum eigenvalue bounded away from zero.The matrix ˆΣm,n is also positive semidefinite.Source: §3 · pp. 22–23
  • For sufficiently small h, the derivative covariance matrix ΓI is invertible with its minimum eigenvalue bounded away from zero, and a multivariate derivative CLT holds when min(m, n)h^d → ∞.The theorem assumes Assumption 1 and (4.1).Source: §3 · p. 23
  • The derivative CLT remains valid when ΓI is replaced by its estimator ˆΓI.Source: §3 · p. 23

C Proof of Theorem 17

The proof controls the expansion remainder and establishes the asymptotic distribution through an equivalent leading term and a central limit theorem for independent averages.

C Proof of Theorem 17

  • The proof takes supremum norms over the domain [ρ7, ρ8].Source: C Proof of Theorem 17 · p. 24
  • The expansion’s second term has asymptotic upper bound O(h^4β), while the first is controlled using Theorem 12 and the stated bandwidth selection.Source: C Proof of Theorem 17 · p. 24
  • The proof shows that √n(ˆΦ − ˜Φ) and √nLm,n have the same L2 limit, while nE[Lm,n²] converges to V(f, g).This uses m/n → κ and the negligible fractions m*/m and n*/n.Source: C Proof of Theorem 17 · p. 25
  • The final distributional step applies the Lindeberg–Lévy central limit theorem separately to the two independent averages.Source: C Proof of Theorem 17 · p. 25

D Proofs of technical lemmas

The technical proofs establish uniform nondegeneracy and control empirical matrix and vector deviations using compactness and concentration arguments.

D Proofs of technical lemmas

Related papers

Loading 2609.38412v1…