Source-linked AI summary
Robust Estimation and Generative Adversarial Nets
Chao Gao, Jiyi Liu, Yuan Yao, Weizhi Zhu
TL;DR
Robust depth-based estimators are statistically optimal but computationally intractable under Huber’s ϵ-contamination model. The paper connects them to f-GANs through f-Learning and shows that suitable hidden-layer discriminator architectures yield statistically optimal robust location estimators for Gaussian and general elliptical distributions. The work also identifies practical training and modeling limitations, including difficult convergence assessment and a technical bounded-parameter assumption.
Problem
Statistically optimal depth-based robust estimators are computationally impractical, motivating feasible procedures under Huber’s ϵ-contamination model.
Source: §1 · pp. 1–2, §2.3 · p. 8Method
The paper interprets depth estimators as variational lower bounds of total variation through f-Learning and uses structured neural-network discriminators in f-GANs.
Source: §1 · p. 3Results
Suitable hidden-layer discriminator classes achieve statistically optimal robust location estimation for Gaussian and general elliptical distributions, including settings where first moments may not exist.
Source: §1 · p. 3Takeaways & Limitations
The f-Learning connection provides a route to compute statistically optimal depth-like robust estimators using GAN training tools.
Source: §1 · p. 3, §2.3 · p. 8GAN convergence is difficult to judge because training can oscillate, and one analysis assumes ∥θ∥∞≤√log p.
Source: §4 · p. 15, §6.1 · p. 18
Abstract
from arXiv · showhide
Robust estimation under Huber's $\epsilon$-contamination model has become an important topic in statistics and theoretical computer science. Statistically optimal procedures such as Tukey's median and other estimators based on depth functions are impractical because of their computational intractability. In this paper, we establish an intriguing connection between $f$-GANs and various depth functions through the lens of $f$-Learning. Similar to the derivation of $f$-GANs, we show that these depth functions that lead to statistically optimal robust estimators can all be viewed as variational lower bounds of the total variation distance in the framework of $f$-Learning. This connection opens the door of computing robust estimators using tools developed for training GANs. In particular, we show in both theory and experiments that some appropriate structures of discriminator networks with hidden layers in GANs lead to statistically optimal robust location estimators for both Gaussian distribution and general elliptical distributions where first moment may not exist.
1 Introduction
The paper addresses the gap between statistically optimal but computationally impractical depth-based robust estimators and feasible procedures under Huber’s ϵ-contamination model. It connects f-GANs and depth functions through f-Learning, then uses discriminator architectures to obtain statistically optimal robust location estimators.
Motivation
- Under Huber’s ϵ-contamination model, unknown contamination can make the sample mean arbitrarily inaccurate, while coordinatewise and geometric medians are statistically suboptimal.The resulting search for statistically optimal and computationally feasible procedures is a central problem in robust statistics and theoretical computer science.Source: §1 · pp. 1–2
- Tukey’s median is statistically optimal, but even approximate computation requires O(e^Cp) time, limiting its practical use.Source: §1 · p. 2
Prior approaches
- Existing computational procedures can achieve the minimax rate up to poly-logarithmic factors in polynomial time, but depth-based estimators retain advantages in objective interpretation and adaptation to nuisance parameters.These advantages matter especially for elliptical distributions, where moments may not exist.Source: §1 · pp. 2–3
Contributions
- The paper identifies f-Learning as an f-GAN subclass that unifies depth-based estimators, GANs, and maximum likelihood estimation through variational lower bounds of total variation distance.The resulting estimators retain favorable statistical properties while being trainable with stochastic gradient methods.Source: §1 · p. 3
- The paper relates JS-GAN optimization to feature matching, showing that hidden layers are necessary because no-hidden-layer discriminators only match linear features and are unsuitable for robust estimation.Source: §1 · p. 3
- Appropriately structured and regularized GAN discriminators achieve rate-optimal robust location estimation for Gaussian and general elliptical distributions, supported by theory and numerical experiments.The elliptical setting includes distributions whose first moment may not exist.Source: §1 · pp. 3–4
2 Robust Estimation and f-GAN
The paper develops f-Learning as a variational framework linking f-GAN objectives to depth-based robust estimators. It recovers Tukey, regression, and covariance-matrix depth procedures through total-variation learning and motivates neural discriminator classes for computationally feasible robust estimation.
2 Robust Estimation and f-GAN
- The f-GAN variational lower bound yields a general learning method in which a fitted distribution makes the best discriminator unable to distinguish it from the empirical distribution.Source: §2 · pp. 4–5
2.1 f-Learning: A Unified Framework
- f-Learning allows discriminator classes to depend on the candidate distribution, and it includes maximum likelihood estimation as a special case.For f(x)=x log x with a distribution-independent class, the resulting procedure is MLE.Source: §2.1 · p. 5
2.2 TV-Learning and Depth-Based Estimators
- TV-Learning recovers Tukey’s median by taking local Gaussian location neighborhoods and letting their radius tend to zero.In this framework, the fitted Gaussian minimizes a variational lower bound of total variation distance.Source: §2.2 · p. 6
- The same TV-Learning construction recovers regression depth and covariance matrix depth estimators through localized perturbations of conditional means and covariance structure.The covariance-depth estimator is stated to achieve the minimax rate under Huber’s ϵ-contamination model.Source: §2.2 · p. 7
2.3 From f-Learning to f-GAN
- Depth-based estimators are statistically optimal but computationally difficult, motivating a return to f-GANs, whose flexible neural discriminator classes offer computational advantages.Source: §2.3 · p. 8
- The paper studies how to choose f and discriminator classes so f-GAN procedures remain robust and achieve minimax rates under Huber’s ϵ-contamination model.The robust mean problem provides the main setting for answering these design questions.Source: §2.3 · p. 8
3 Robust Mean Estimation via GAN
This section studies robust Gaussian mean estimation via GANs, showing how discriminator structure determines statistical robustness and optimization behavior. TV-GAN can attain minimax rates but may optimize poorly, whereas hidden-layer JS-GAN discriminators yield robust, minimax-optimal estimation.
3.1 Results for TV-GAN
- For i.i.d. observations Xi ∼ (1 − ϵ)N(θ, Ip) + ϵQ with n + ϵ2 ≤ c, estimator bθ defined by (16) has a guarantee uniformly over θ and Q with probability at least 1 −e−C′(p+nϵ2).Here c > 0 is sufficiently small, and C, C′ > 0 are universal constants.Source: §3.1 · p. 9
- TV-GAN can fail optimization when contamination and the model distribution are far apart because local discriminator maxima misdirect generator updates.The landscape has opposing local maxima, making it difficult for gradient methods to move between discriminator directions.Source: §3.1 · p. 9, Figure 1 · p. 10
3.2 Results for JS-GAN
- In one-dimensional experiments, linear-discriminator JS-GAN follows the contaminated grand mean, whereas one-hidden-layer JS-GAN remains near the target mean as contamination moves away.The linear estimator is close to 0.2 + 0.8t, while the hidden-layer estimator converges toward θ = 1 for large t.Source: §3.2 · p. 11, Figure 3 · p. 12
- Restricted Jensen-Shannon divergence vanishes exactly when the discriminator feature has equal expectations under the two distributions, reducing JS-GAN to feature matching.Thus, linear discriminators match linear features and recover the sample mean, while nonlinear hidden-layer features can support robustness.Source: §3.2 · pp. 12–13
- Hidden-layer JS-GAN is robust and achieves the minimax rate under the theorem’s conditions.The result applies to bounded-activation two-layer discriminator classes with bounded hidden-layer weight norm, and the stated κ condition is needed only for technical reasons.Source: §3.2 · p. 13
4 Deep ReLU Networks
This section develops regularized deep ReLU discriminator classes for robust Gaussian location estimation and establishes their rate guarantee under a bounded-coordinate condition. A preprocessing procedure removes that condition in the final estimator, while experiments suggest preprocessing is unnecessary.
4 Deep ReLU Networks
- The deep ReLU discriminator uses ℓ1 regularization on all layers except the second-last, which receives ℓ2 regularization to control overfitting.The network combines ReLU activations with a final sigmoid layer.Source: §4 · pp. 14–15
- Under ∥θ∥∞≤√log p, the deep ReLU estimator achieves error rate p log p / n ∨ ϵ^2 with high probability.The guarantee is uniform over θ and contamination distributions Q under the stated sample-size condition.Source: §4 · p. 15
- A coordinatewise-median split-sample preprocessing step shifts the data so the bounded-coordinate condition holds, after which adding the preliminary estimate recovers the same rate.The first half has log n observations and the second half is used for the shifted estimator.Source: §4 · p. 15
- Experiments indicate preprocessing is unnecessary, suggesting the bound ∥θ∥∞≤√log p may be a technical artifact of the Rademacher-complexity analysis.The paper does not claim that a sharper analysis has removed this assumption.Source: §4 · p. 15
5 Elliptical Distributions
This section extends JS-GAN robust location estimation to general elliptical distributions, including cases such as multivariate Cauchy distributions without a finite mean. With an appropriate generator class and shared discriminator structure, the estimator achieves the Gaussian-case convergence rate.
5 Elliptical Distributions
- JS-GAN provides optimal robust location estimation for general elliptical distributions, including multivariate Cauchy distributions whose mean does not exist.This extends the scope beyond robust mean estimation methods designed for finite first moments.Source: §5 · p. 16
- Elliptical distributions are parametrized by center θ, scatter matrix Σ, and shape function h, after restricting the non-identifiable scale representation.The scatter matrix is Σ = AA^T, and the normalization makes the parametrization identifiable.Source: §5 · p. 16
- Theorem 5.1 gives the same convergence rate as in the Gaussian case uniformly over elliptical centers, bounded-scatter matrices, and contamination distributions.The guarantee holds with probability at least 1 − e^(-C′(p+nϵ^2)) under the theorem’s sample-size condition.Source: §5 · p. 16
- The elliptical estimator differs from the Gaussian estimator in its generator class while retaining the same discriminator class, so statistical complexity is determined by the discriminator class.The estimator also outputs scatter and shape estimates, but the paper gives no theoretical guarantee for them.Source: §5 · pp. 16–17
6 Numerical Experiments
The experiments evaluate GAN-based robust estimators across minimax-rate regimes, competing methods, network structures, unknown covariance, and elliptical distributions. They generally support the theoretical rates and show that discriminator design and distributional setting materially affect performance.
6.1 Implementations
- The implementation uses GAN optimization with discriminator and generator updates, early stopping for discriminator-only tuning, and objective-based hyperparameter selection.The reported procedures initialize the generator at the coordinatewise median and return an average estimate over the final epochs.Source: §6.1 · pp. 17–18
6.2 Numerical Supports for the Minimax Rates
- The worst-case ℓ2 errors are approximately linear in ϵ, √p, and 1/√n, empirically supporting the minimax-rate conclusions for TV-GAN and JS-GAN.Figure 5 reports these relationships with standard-deviation bars.Source: §6.2 · p. 19, Figure 5 · p. 21
6.3 Comparisons with Other Methods
- The method does not require prior knowledge of nuisance parameters such as the contamination proportion, with tuning based only on the optimization objective.The comparison experiments report robustness of hyperparameters across sampling distributions when network structures are fixed.Source: §6.3 · pp. 21–22
- JS-GAN and TV-GAN outperform other robust mean-estimation algorithms in most cases, while JS-GAN is stable for separable contamination and competitive for non-separable contamination.TV-GAN performs well for non-separable distributions but can fail when contamination is far from the target distribution because of optimization difficulties.Source: §6.3 · p. 22, Table 4 · p. 22
6.4 Network Structures
- Adding one hidden layer improves TV-GAN over its no-hidden-layer version, whereas deeper JS-GAN networks sometimes help, especially at larger dimensions.The best hidden-layer width depends on sample size, and adding depth does not uniformly improve performance.Source: §6.1 · p. 18, §6.4 · pp. 22–23, Table 5 · p. 23
6.5 Adaptation to Unknown Covariance
- The JS-GAN estimator remains rate-optimal when the covariance matrix is unknown and need not be the identity.Experiments assess both location estimation and covariance estimation under generated nontrivial covariance matrices.Source: §6.5 · p. 23, Table 7 · p. 24
6.6 Adaptation to Elliptical Distributions
- Under Cauchy contamination settings, the proposed method continues to work well while methods relying on moment conditions deteriorate.The elliptical-distribution experiment uses a generator adapted to location and, when needed, scatter structure.Source: §6.6 · pp. 23–24, Table 8 · p. 24
7 Discussions
The discussion explains why hidden layers are central to robust JS-GAN estimation: a logistic discriminator is sharp without contamination but fails under contamination. A controlled one-hidden-layer class instead provides robustness and minimax-rate performance without requiring the unrestricted contaminated optimal discriminator.
Variational Lower Bounds for Robust Estimation
- Without contamination, the logistic-regression discriminator makes the JS-GAN variational lower bound sharp and yields the sample mean, which is minimax optimal.This sharpness follows from the optimal discriminator's logistic-regression form in the uncontaminated Gaussian setting.Source: §7 · p. 25
- With contamination, the logistic-regression discriminator no longer gives a sharp lower bound and does not even produce a consistent estimator.The contaminated population objective differs from the desired Jensen–Shannon divergence.Source: §7 · p. 25
- Although the exact contaminated optimal discriminator can take infinitely many forms, a one-hidden-layer neural-network class is rich enough for robustness while remaining controlled against overfitting.The paper states that this class achieves the minimax rate for robust mean estimation despite not making the variational lower bound sharp.Source: §7 · p. 25
Future Projects
- The paper identifies robust covariance estimation, high-dimensional regression, mixture learning, classification, other f-divergences, integral probability metrics, and GAN optimization landscapes as future directions.These directions extend the framework beyond the robust mean-estimation tasks studied here.Source: §7 · p. 25
8 Proofs
The proofs establish concentration and complexity bounds for the discriminator classes, then apply them to Gaussian and elliptical-distribution robust estimators. The main arguments combine empirical-process control with specially constructed discriminator networks and local behavior of sigmoid-based objectives.
8.1 Some Auxiliary Lemmas
- Auxiliary lemmas control empirical deviations through bounded differences, symmetrization, Rademacher complexity, covering numbers, and Dudley’s entropy bound.The function-class covering number is bounded by N(δ, D, ∥·∥n) ≲p (16e/δ)^O(p), yielding the desired complexity estimate.Source: §8.1 · p. 26
- For Gaussian observations, the auxiliary bounds assume ∥θ∥∞≤√log p and τ = √p log p, with the resulting control holding with probability at least 1 −δ.The proof centers observations as Zi = Xi−θ and bounds the resulting complexity under these conditions.Source: §8.1 · pp. 27–29
8.2 Proofs of Main Theorems
- The main theorem proof summarizes high-probability bounds for the estimator and discriminator objectives under the stated norm constraints.The summary explicitly derives the inequalities with probability at least 1 −δ and uses them to complete the theorem.Source: §8.2 · pp. 29–31
- The Gaussian main-theorem proof defines the discriminator objective Fw,u,b and identifies bθ as an empirical minimizer of F(Pn, η).The proof uses the local linearity of a sigmoid-based population function to relate objective differences to displacement from θ.Source: §8.2 · pp. 29–30
- A hidden-layer ReLU construction realizes discriminators of the form sigmoid(κ sigmoid(euT(x −θ))) within the permitted network class.The construction uses paired positive and negative ReLU coordinates and verifies the required parameter and bias bounds.Source: §8.2 · p. 32
- For elliptical distributions, the proof defines an analogous discriminator objective using EC(η,Γ,g) and transfers the Gaussian-style argument to directional covariance and location bounds.The final steps bound uT(bθ −θ) after controlling uT bΣu over unit directions.Source: §8.2 · pp. 32–33