PaperlayerView exact revision on arXiv

Source-linked AI summary

Assumption-lean logistic regression with missing covariates

Jyotishka Ray Choudhury, Kabir Aladin Verchand, Richard J. Samworth, Ashwin Pananjady

arXiv:2610.07292v1stat.MLcs.ITcs.LGmath.ITmath.STstat.MEstat.TH

TL;DR

Estimating logistic-regression parameters with missing covariates is challenging when the covariate distribution is unknown and modeling it is undesirable. The paper uses distribution-free Z-estimation with a monotone operator and stochastic approximation; it establishes parametric-rate recovery and lower bounds showing that the nonstandard missingness cost is intrinsic.

  • Problem

    With unknown covariate distributions, imputation can behave poorly in nonlinear regression, motivating parameter estimation without modeling that distribution.

    Source: §1 · pp. 3–4

  • Method

    The paper constructs a monotone-operator estimating equation with the target parameter as its unique zero and solves it using stochastic approximation.

    Source: §2.1 · p. 11, §3.1 · p. 16, §5 · p. 33

  • Results

    The estimator achieves parametric-rate signal recovery without modeling or estimating the covariate distribution, and matching lower bounds establish that its nonstandard missingness dependence is intrinsic.

    Source: §5 · p. 33

  • Takeaways & Limitations

    The statistical cost of missingness in logistic regression can depend on observation probabilities in an intricate, nonstandard way.

    Source: §5 · p. 33

    The lower-bound result establishes sharpness of the upper bound only where that bound is non-trivial; at smaller sample sizes it is vacuous.

    Source: §2.3 · p. 15

Abstract

from arXiv · show

Missing covariates are frequently encountered in supervised learning problems, and classical methods for estimation using such data use carefully chosen imputation schemes for missing data, or likelihood approximations that lead to nonconvex $M$-estimation problems. These methods and their relatives are suitable for scenarios in which the covariate distribution is known, and more broadly, have enjoyed tremendous success in linear models. But even in basic nonlinear problems such as logistic regression in moderate dimensions, such methods can experience drastic failure modes when the covariate distribution is unknown. Motivated by the need for reliable alternatives, we consider the problem of parameter estimation in logistic regression with missing covariates. Crucially, we operate in the assumption-lean setting where the covariate distribution is unknown (but bounded). We design a stochastic approximation method that is based on $Z$-estimation with a novel monotone operator, and establish that our algorithm is computationally efficient and achieves provable signal recovery at parametric rates under the hypothesis that covariates are missing completely at random. Our theory sharply characterizes the $\ell_2^2$ risk of the estimator in terms of the missingness profile, accommodating heterogeneous observation probabilities. Importantly, it shows that our method always outperforms the de facto ``complete-case'' estimator that ignores observations with any missing data. Even in the setting with homogeneous missingness (in which each covariate is observed independently with probability $q$), our bounds exhibit intricate and nonstandard dependence on $q$ that can yield significant improvements over using only complete cases. We complement our upper bounds with new information-theoretic lower bounds that show that this intricate dependence on $q$ is fundamental in a minimax sense.

1 Introduction

The paper studies logistic regression with missing covariates when the covariate distribution is unknown, a setting where imputation and model-based approaches can be unreliable. It proposes distribution-free estimation and analyzes its statistical guarantees under coordinate-wise MCAR missingness.

1 Introduction

  • Unknown covariate distributions make conditional density estimation prohibitive in high dimensions, while parametric assumptions such as Gaussianity can fail when violated.Imputation-based strategies can also behave poorly in nonlinear regression.Source: §1 · pp. 3–4

1.1 A motivating numerical experiment

  • In two bounded, non-Gaussian examples, imputation and Gaussian-likelihood methods remain inconsistent, complete-case estimation performs poorly, and the proposed estimator is consistent and comparable to the oracle.The comparison plots RMSE against sample size under coordinate-wise MCAR missingness.Source: Figure 1 · p. 4, §1.1 · p. 5

1.2 Contributions and organization

  • The proposed distribution-free estimator constructs a monotone operator whose unique zero is the target parameter, then uses unbiased estimates in projected stochastic approximation.It does not impute missing entries or rely on information about the covariate distribution.Source: §1.2 · p. 5
  • The non-asymptotic risk analysis shows the estimator is computationally efficient and always improves on complete-case estimation; a minimax lower bound matches the upper bound up to universal constants under homogeneous missingness.The lower-bound construction uses a Boolean function to conceal parameter perturbations in likelihood ratios under missingness.Source: §1.2 · pp. 5–6

1.3 Related work

  • Related likelihood methods require covariate-distribution knowledge and often face nonconvex optimization, while imputation approaches are typically not distribution-free; the proposed estimator is also an AIPW-family member.The paper distinguishes its estimator from PPI-inspired inference methods that use black-box imputation to improve complete-case analysis.Source: §1.3 · pp. 6–7

1.4 Notation

  • The notation section fixes Euclidean norms and inner products, subset-coordinate notation, positive-semidefinite ordering, and asymptotic comparison symbols.It also defines the paper's probability-measure and divergence notation.Source: §1.4 · p. 7

2 Methodology and main results

This section presents distribution-free stochastic approximation for logistic regression with MCAR covariates, deriving risk bounds that account for heterogeneous observation probabilities and examining homogeneous missingness relative to complete-case estimation. It also establishes the sharpness of a nontrivial upper bound through a missingness-specific three-regime minimax lower-bound argument.

2 Methodology and main results

  • The model assumes bounded covariates from an unknown distribution, a bounded true parameter, and an independently masked MCAR mechanism with known coordinate-wise observation probabilities.The response is always observed, while covariate coordinates may be missing independently of one another and of the data.Source: §2 · p. 8

2.1 Distribution-free estimation via stochastic approximation

  • An unbiased, distribution-free estimate of a strongly monotone population operator enables projected stochastic approximation using only observed data and known coordinate observation probabilities.The operator’s exponential term factorizes across coordinates, enabling construction of an unbiased estimate from the incomplete observations.Source: §2.1 · pp. 9–10

2.2 Upper bounds on the risk of the algorithm

  • The squared-error risk is parametric (n^-1), depends intricately on heterogeneous observation probabilities, and has a missingness penalty no larger than complete-case estimation.The bound’s dependence on observation probabilities can yield a significantly lower missingness penalty than complete-case estimation.Source: §2.2 · pp. 11–12
  • With homogeneous missingness, the bound recovers a rate comparable to complete-case estimation in low dimensions and gives strictly better rates in other regimes, including high dimensions with substantial missingness.For q ≤ q⋆ and R ≤ 2τq(d − 1), the missingness penalty is much smaller than the complete-case penalty q^-d.Source: §2.2 · p. 13

2.3 A minimax lower bound

  • Under homogeneous missingness and the stated curvature and sample-size conditions, the minimax lower bound matches the upper bound’s dependence on n, q, R, and µ across the same three regimes.This establishes minimax optimality of the stochastic-approximation rate up to universal constants in the exponents and a multiplicative factor.Source: §2.3 · pp. 14–15
  • Theorem 4 establishes Corollary 3’s sharpness precisely when its upper bound is non-trivial; below the stated sample-size threshold, that bound is vacuous.When n ≲ n0(R, µ, q), the error bound is only on the order of R^2, matching the all-zero estimator that ignores the data.Source: §2.3 · p. 15
  • The proof combines two Assouad constructions, including a fully observed construction that makes neighboring logistic laws exponentially difficult to distinguish as R grows.It places a large common component in the regression parameter and perturbs orthogonal directions, paralleling Bayesian Cramér–Rao techniques.Source: §2.3 · p. 15
  • A missingness-specific coupling makes two hypotheses indistinguishable whenever fewer than half of the coupled covariates are observed, yielding the three-regime lower bound after hypercube embedding and optimization.The construction uses coupled Rademacher covariates and Boolean-function coefficients so the conditional expectation remains 1 when fewer than half are observed.Source: §2.3 · p. 15

3 Proof of the upper bound

The upper-bound proof validates the population and empirical operators, then controls stochastic approximation through an error recursion.

3 Proof of the upper bound

  • The proof proceeds by establishing properties of Ψ and applying standard stochastic-approximation convergence machinery.Source: §3 · p. 15

3.1 Preliminaries

  • The population operator has θ⋆ as its zero, and the lemma’s two properties make θ⋆ the unique solution to the population variational inequality.Source: §3.1 · pp. 15–16
  • The empirical operator is used because missing data prevent direct evaluation of the population operator; it is unbiased and has a controlled second moment.The second moment is bounded at the level V_q,R.Source: §3.1 · p. 16

3.2 Proof of Theorem 1

  • Theorem 1’s proof closes by inductively controlling the stochastic-approximation error recursion using empirical moment control and operator strong monotonicity.The recursion uses the step-size choice η_t = {α_R(t+2)}^-1.Source: §3.2 · pp. 16–17

4 Proof of the lower bound

The proof establishes minimax lower bounds by constructing experiments that are difficult to distinguish under missingness and extending them across parameter directions with Assouad’s method. It combines a large-radius construction with a missingness-specific construction to obtain bounds across three regimes.

4 Proof of the lower bound

  • Two complementary lower bounds capture exponential difficulty in radius R and regime-dependent penalties in q, Rµ, and µ from MCAR missingness.The proof establishes the bounds separately and combines them to obtain the minimax lower bound.Source: §4 · p. 17

4.1 Proof of Proposition 7

  • Assouad’s lemma uses dimension p + 1 with p ≍ 1/µ; neighboring observed-data laws acquire a multiplicative q factor when the differing coordinate is observed.The construction takes MCAR indicators independent of the complete data.Source: §4.1 · pp. 19–20

4.2 Proof of Proposition 8

  • A second construction separates logistic-regression parameters while hiding their observed-data differences, then embeds the experiment in a hypercube for an Assouad lower bound.The embedding reduces neighboring-law divergence by an additional factor 1/p, and parameter choices yield three regimes.Source: §4.2 · pp. 20–21

4.2.1 A two-point construction

  • The Boolean hiding function makes two well-specified logistic hypotheses indistinguishable whenever at most (m−1)/2 of the first m auxiliary coordinates are observed.The observed-data likelihood ratio is exactly 1 in that case; the construction also bounds divergence between the observed-data laws.Source: §4.2.1 · pp. 24–25
  • The two-point construction supplies a parameter-separation bound and an observed-data χ2-divergence bound, the properties used in the multidimensional construction.The construction compares two well-specified logistic models whose observed-data distributions remain close under missingness.Source: §4.2.1 · p. 25, §4.2.2 · p. 25

4.2.2 A multidimensional Assouad extension

  • Assouad’s extension embeds the two-point experiment across p directions, yielding valid logistic models with separated neighboring parameters and controlled observed-data divergence.Because neighboring models differ only when the randomly selected direction matches, their distinguishability is reduced by a factor of 1/p.Source: §4.2.2 · pp. 26–27

4.2.3 Choice of parameters in the three regimes

  • The passages establish the desired lower bound throughout Rµ ≤ √q and prove Claim 2 whenever Rµ > √q, provided the sample-size condition in Eq. (81) holds.Source: §4.2.3 · pp. 31, 33

5 Discussion

The paper establishes parametric-rate estimation for well-specified logistic regression with missing covariates, without modeling their distribution, and shows that the missingness-dependent cost is intrinsic. The discussion identifies open directions involving weaker distributional assumptions, structured parameters, broader nonlinear models, and missingness mechanisms beyond MCAR.

5 Discussion

  • Matching lower bounds show that the nonstandard dependence of the upper bounds on missingness probability is intrinsic to the problem.The discussion characterizes the statistical cost of missingness in logistic regression as more intricate than in the extensively studied linear-regression problem.Source: §5 · p. 33
  • The guarantees assume bounded covariates and a minimal curvature condition; extending them to unbounded covariates or general joint distributions remains open.The current approach uses the logistic conditional model to construct the population operator, and an analogous equation for misspecified models remains open.Source: §5 · pp. 33–34
  • Whether missingness costs can be characterized and computationally attained for sparse or otherwise structured parameters remains open.The paper also raises instance-optimal stochastic approximation and variance reduction as directions beyond its vanilla scheme.Source: §5 · p. 34
  • Extending the approach to broader nonlinear regression with MCAR covariates would require efficiently computable monotone operators in the AIPW class.The discussion asks whether such operators can also provide minimax optimality guarantees.Source: §5 · p. 34
  • The analysis assumes MCAR missingness with known observation probabilities; broader mechanisms, including MNAR, are left for future study.The authors identify realizable contamination as one setting for studying accuracy, robustness, and computational-efficiency tradeoffs beyond MCAR.Source: §5 · p. 34

A Proofs of results associated with the upper bound

The appendix supplies deferred proofs for the upper-bound results, including the population-operator lemma and a lemma involving the missingness masks.

A Proofs of results associated with the upper bound

A.1 Proof of Lemma 5

  • The proof of Lemma 5 verifies that the population operator vanishes at the target parameter and differentiates it under the expectation using bounded covariates.The argument uses the logistic conditional model and the boundedness of covariates and the parameter set.Source: §A.1 · p. 39

A.2 Proof of Lemma 6

  • The proof of Lemma 6 conditions on the data, uses independent Bernoulli masks for the coordinates, and then takes an expectation over the data.It also invokes the function ϕ_q and a bound involving the coordinate terms T_j.Source: §A.2 · pp. 40–41

A.3 Proof of Corollary 2

These appendix proofs establish the concave-majorant expression used in Corollary 2 and derive bounds used in Corollary 3. The argument treats homogeneous observation probabilities symmetrically and simplifies the resulting bound across parameter regimes.

A.3 Proof of Corollary 2

  • For homogeneous observation probability q, the coordinate-wise optimization is identical up to relabeling, allowing Theorem 1 to yield the corresponding bound.The proof sets one coordinate to zero because the objective does not depend on it.Source: §A.3 · p. 41
  • The proof gives a closed-form least concave majorant for ϕ_q, with cases determined by q and a threshold q⋆.For q ≥ q⋆, the proof establishes monotonicity; for 0 < q < q⋆, it constructs a tangential linear segment.Source: §A.3 · pp. 42–43

A.4 Proof of Corollary 3

  • The Corollary 3 proof bounds K(q, R, µ) separately across three regimes before combining the bounds.It then substitutes the resulting bound into Corollary 2 to obtain the desired result.Source: §A.4 · pp. 44, 46

B Proofs of supporting lemmas associated with the lower bound

B.1 Proof of Lemma 9

The proof of Lemma 9 bounds divergence between neighboring models by conditioning on which coordinate and missingness pattern are observed. When the distinguishing covariate is missing, the response laws coincide; the proof then combines the conditional divergences.

B.1 Proof of Lemma 9

  • The two models can differ only when K = k, an event of probability 1/p, and their divergence is zero when the distinguishing covariate B is unobserved.When B is unobserved, marginalizing over its shared uniform Rademacher distribution gives the same response law under both models.Source: §B.1 · pp. 46–47
  • The proof conditions on K and whether B is observed, then uses independence of D and K to combine the conditional χ2-divergence terms.Source: §B.1 · pp. 46–47
  • For the case where B is observed, the proof compares logistic response laws under the two models and averages over B.It uses the bound α ≤ R/24 in deriving the divergence bound, then applies tensorization of χ2-divergence.Source: §B.1 · pp. 47–48

B.2.1 Proof of Lemma 10(a)

The proof establishes that J is odd and verifies its expectation and conditional expectation under the null. It also derives a uniform upper bound on its infinity norm.

B.2.1 Proof of Lemma 10(a)

  • The proof establishes that J is an odd function.The argument uses that the product defining J ranges over indices with odd |U|.Source: §B.2.1 · p. 48
  • The proof verifies the expectation of J(T) under the null distribution.It invokes independence of the coordinates of T and E0[Tj] = ρ.Source: §B.2.1 · p. 49
  • For |S| ≤ r, the proof establishes the claimed conditional expectation of J(T) given TS.The argument changes summation indices using C := U ∩ S and applies Lemma 18.Source: §B.2.1 · p. 49
  • The proof derives a uniform upper bound on ∥J∥∞ by bounding J(t) for every t ∈ {−1, 1}^m.It uses |χU(t)| ≤ 1, the triangle inequality, Cauchy–Schwarz, and the binomial formula.Source: §B.2.1 · pp. 49–50

B.2.2 Proof of Lemma 10(b)

The proof verifies that the constructed alternative covariate distribution is valid and that the null and alternative specify logistic regression models. It also establishes the stated complete-data likelihood-ratio identity.

B.2.2 Proof of Lemma 10(b)

  • The proof establishes that Eq. (56) defines a valid covariate distribution P1,X under the alternative.It checks positivity and that the proposed Radon–Nikodym derivative integrates to one.Source: §B.2.2 · p. 50
  • Under the null, the constructed conditional distribution is the logistic model with parameter θ0.The proof identifies 2x(a, b)⊤θ0 with s(a).Source: §B.2.2 · p. 51
  • Under the alternative, P1 is a well-specified logistic regression model with parameter θ1.Source: §B.2.2 · p. 51
  • The proof derives the complete-data likelihood ratio and concludes that it satisfies Eq. (57).The derivation rewrites the expression using V = YB and B · J(A) = V · J(T).Source: §B.2.2 · p. 51

B.2.3 Proof of Lemma 10(c)

The proof shows that the second-moment matrices under P0 and P1 agree. It also uses conditional expectations of J(T) to establish needed moment identities.

B.2.3 Proof of Lemma 10(c)

  • For r ≥ 2, the proof establishes E0[J(T) | Tj, Tk] = 1 for distinct j and k.It applies Eq. (52) with S = {j, k}.Source: §B.2.3 · p. 52
  • For r ≥ 1, the proof obtains E0[TjJ(T)] = E0[Tj] using the conditional expectation given Tj.It applies Eq. (52) with S = {j}.Source: §B.2.3 · p. 52
  • Every entry of E[XX⊤] agrees under P0 and P1.The proof combines the identities for AjAk and AjB with the corresponding moment calculations.Source: §B.2.3 · pp. 52–53

B.3 Proof of Lemma 11

The proof identifies the observed-data likelihood ratio as a conditional expectation of the complete-data likelihood ratio. It then evaluates that conditional expectation using the observation pattern.

B.3 Proof of Lemma 11

  • The observed sigma-field can be expressed using Y, M, TS, and the observed value of V when D = 1.The proof notes that observing Aj is equivalent to observing Tj, and when D = 1 observing B is equivalent to observing V.Source: §B.3 · p. 53
  • The observed-data law Q1 is absolutely continuous with respect to Q0, with likelihood ratio equal to the conditional expectation of Lα given O.The proof establishes this relation by testing against bounded measurable functions of the observed data.Source: §B.3 · pp. 53–54
  • On the event D = 0, V is unobserved and independent of O under P0.The proof uses this case distinction when evaluating the conditional expectation.Source: §B.3 · p. 54
  • When |S| ≤ r, the displayed conditional likelihood-ratio expression evaluates to 1 for both D = 0 and D = 1.For D = 1, the proof uses exp(αV){cosh(α) − V sinh(α)} = 1.Source: §B.3 · p. 54

B.4 Proof of Lemma 12

The appendix completes divergence calculations for Lemma 12 and supports lower-bound arguments with independence, moment, and observed-law reductions.

B.4 Proof of Lemma 12

  • The likelihood-ratio calculations used in Lemmas 10(b) and 11 remain valid for every odd m ≥ 1 when α∥J∥∞ ≤ 1/2.They do not require the restriction m ≥ 5.Source: §B.4 · p. 55

B.4.1 Proof of Lemma 16

  • Under P0, V is a uniform Rademacher variable independent of (Y, T1, . . . , Tm), enabling the calculation to use independence from observed subsets of T.The calculation also invokes the missing-completely-at-random property to rewrite an expectation.Source: §B.4 · p. 55

B.4 Proof of Lemma 12

  • Tensorization of the χ2-divergence, together with 1 + x ≤ e^x, yields the stated bound in Eq. (65).The preceding bounds establish Eq. (64).Source: §B.4 · p. 56

B.5 Proof of Lemma 13

  • In the Lemma 13 construction, conditional second moments of (A, B) are unchanged by the selected bit ωk.This follows from the equality of the second-moment matrix under P0 and P1 for m ≥ 5.Source: §B.5 · p. 58

B.6 Proof of Lemma 14

  • For Lemma 14, common appended coordinates have no effect on χ2-divergence, reducing the observed-law pair to the preceding two-point pair.The common coordinates are zero and their mask variables have the same distribution under both laws.Source: §B.6 · p. 59

C Additional details of simulation in Figure 1

The appendix details the Figure 1 simulation procedures and supplies auxiliary Fourier representations for the majority function and Wm,q.

C Additional details of simulation in Figure 1

  • The simulation uses independent two-dimensional covariate blocks, producing decidedly non-Gaussian covariates, and a signal concentrated mainly in the first block.The signal has a spike in coordinate 2, with α = 1.25 and δ = 0.15.Source: C Additional details of simulation in… · p. 60
  • The simulation compares six estimators across (d, q) settings (10, 0.40) and (100, 0.85), using 100 Monte Carlo replications and sample sizes from 200 to 1,000,000.Missingness is homogeneous and introduced independently across observations and coordinates.Source: C Additional details of simulation in… · p. 60
  • The proposed stochastic approximation uses the unbiased stochastic operator in place of the unavailable population gradient and initializes the algorithm at zero.The step-size parameter is set to the minimum eigenvalue of ΣX.Source: C Additional details of simulation in… · p. 61

D.1.1 Fourier coefficients of the majority function

  • For odd m, majority-function Fourier coefficients vanish on even-cardinality sets; Lemma 18 also gives coefficient relations for sets S with |S| ≤ r.The odd-cardinality restriction follows from the vanishing of even-cardinality coefficients.Source: §D.1.1 · pp. 62–64

D.1.1 Fourier coefficients of the majority function; D.1.2 Fourier coefficient representation of Wm,q

  • The coefficients hU depend only on |U|, and hU = 0 when 1 ≤ |U| ≤ r.Lemma 20 represents Wm,q using the hU coefficients and gives an explicit expression in q, ρ, and m.Source: §D.1.1 · pp. 64–65, §D.1.2 · p. 66

D.2 Supporting lower bound lemmas for moderate curvature µ

For moderate curvature, the appendix handles a regime where the earlier Boolean lower-bound construction may not apply directly by specializing a two-point construction to m = 1.

D.2 Supporting lower bound lemmas for moderate curvature µ

  • When µ ∈ (µcrit, 1/3], an m of order µ−1 need not satisfy m ≥ 5, so the earlier construction cannot be applied directly.In this range, µ is bounded below by a universal constant, making a fixed-dimensional construction sufficient.Source: §D.2 · p. 68
  • The proof specializes the two-point construction to m = 1 to establish the lower bound for µ ∈ (µcrit, 1/3].For m = 1, the majority function is f(t) = t.Source: §D.2 · pp. 68–69
  • With total variation distance at most 1/2 and squared parameter separation 8α², Le Cam’s two-point inequality gives the claimed lower bound.The proof obtains these conditions before applying the inequality.Source: §D.2 · p. 69
Loading 2610.07292v1…