Source-linked AI summary
Robust Anomaly Detection and Backdoor Attack Detection Via Differential Privacy
Min Du, Ruoxi Jia, Dawn Song
TL;DR
The paper asks whether differential privacy can improve detection of outliers, novelties, and poisoning samples used in backdoor attacks. It develops a theoretical account and evaluates the approach experimentally, reporting improved anomaly-detection utility across these tasks. The paper’s analysis is intentionally broad about how or where outliers arise, while its experiments focus on selected detection settings.
Problem
The paper addresses limited understanding of whether differential privacy, normally used to protect records, can improve outlier, novelty, and backdoor poisoning detection.
Method
The paper combines theoretical analysis of privacy, stability, noise, and outlier counts with experiments using autoencoders, Hadoop log anomaly detection, and poisoned MNIST data.
Results
Differential privacy improves outlier and novelty detection utility, greatly improves Hadoop log anomaly detection over DeepLog, and further improves backdoor attack detection.
Takeaways & Limitations
Applying differential privacy during model training can support anomaly detection while extending the outlier-detection approach to backdoor poisoning samples.
Takeaways & Limitations
The experiments focus on outlier and novelty detection and defense against backdoor attacks, although the analysis allows broader anomaly types.
Abstract
from arXiv · showhide
Outlier detection and novelty detection are two important topics for anomaly detection. Suppose the majority of a dataset are drawn from a certain distribution, outlier detection and novelty detection both aim to detect data samples that do not fit the distribution. Outliers refer to data samples within this dataset, while novelties refer to new samples. In the meantime, backdoor poisoning attacks for machine learning models are achieved through injecting poisoning samples into the training dataset, which could be regarded as "outliers" that are intentionally added by attackers. Differential privacy has been proposed to avoid leaking any individual's information, when aggregated analysis is performed on a given dataset. It is typically achieved by adding random noise, either directly to the input dataset, or to intermediate results of the aggregation mechanism. In this paper, we demonstrate that applying differential privacy can improve the utility of outlier detection and novelty detection, with an extension to detect poisoning samples in backdoor attacks. We first present a theoretical analysis on how differential privacy helps with the detection, and then conduct extensive experiments to validate the effectiveness of differential privacy in improving outlier detection, novelty detection, and backdoor attack detection.
1 Introduction
The paper studies outlier and novelty detection, including poisoning samples as intentional training-data outliers, and examines differential privacy as a way to improve detection utility. It combines theoretical analysis with experiments across anomaly-detection settings.
- Outlier detection finds minority samples within a dataset that differ from its main distribution, whereas novelty detection targets newly observed samples that do not fit it.
- Poisoning examples injected for backdoor attacks can be treated as outliers in the training dataset.
- Differential privacy hides whether an individual record affects aggregate results by adding random noise to the data or analysis procedure.
- The paper presents a theoretical explanation of why differential privacy can help detect outliers in training and testing data, including how outlier counts relate to noise.
- Experiments evaluate differential privacy for outlier detection, novelty detection, and backdoor attack detection.
2 Preliminary
The preliminary material defines differential privacy, dataset adjacency, Gaussian noise, sensitivity, and differentially private deep-learning training. These concepts establish how privacy noise limits record influence while depending on function sensitivity and gradient handling.
- Differential privacy releases aggregation results while guaranteeing that an individual record’s participation is not revealed.
- A randomized mechanism is (ϵ, δ)-differentially private when output distributions on adjacent datasets remain appropriately indistinguishable.
- Adjacent datasets differ in one record, and smaller ϵ provides a stronger privacy guarantee by making outputs more indistinguishable.
- Gaussian differential privacy perturbs a function’s output with N(0, σ^2) noise calibrated to its maximum neighboring-output difference, called L2-sensitivity.
- Differentially private SGD clips each example’s gradient norm and then perturbs the aggregated training update with noise.
3 The connection between differential privacy and outlier detection
The theory connects differential privacy, learning stability, and anomaly separability: privacy can make models less responsive to rare outliers, but excessive noise harms learning. The analysis also predicts dependence on outlier quantity and supports a broad anomaly scope.
- Mechanism: Differential privacy can hide rare training examples, causing models to underfit outliers and become less confident on atypical inputs.
- Definitions: The analysis assumes samples come from an unknown distribution and defines outliers through substantially different loss under an oracle hypothesis.
- Assumptions: The theoretical result applies to differentially private learning algorithms that are uniformly asymptotic empirical risk minimization (UAERM).
- Theoretical result: A larger prediction-loss gap between outliers and normal data makes outlier identification easier, with the theorem providing a lower bound for that gap.
- Privacy trade-off: Stronger privacy improves stability and can increase anomaly separability, but its added noise increases the learning rate, so privacy parameters cannot be too large or too small.
- Outlier quantity: More training outliers make anomaly detection harder and require smaller ϵ and δ to maintain the same novelty-detection performance.
- Scope: The analysis makes no assumptions about how outliers are generated or whether they occur in training or test data, while experiments focus on outlier, novelty, and backdoor detection.
4 Experiments
Experiments evaluate differential privacy for autoencoder-based outlier and novelty detection, Hadoop log anomaly detection with DeepLog, and backdoor poisoning detection. Across these settings, differential privacy improves detection utility, especially by reducing false negatives, while introducing computational and application-scope trade-offs.
- Outlier and novelty detection: Autoencoders identify rare information through reconstruction loss, and experiments vary injected outliers and differential-privacy noise to assess detection utility.The encoder compresses inputs and the decoder reconstructs them; atypical samples are identified through higher loss.
- Outlier and novelty detection: AUPR is used as the primary threshold-independent metric because highly imbalanced anomaly classes make it more informative than AUROC.Additional AUPR and AUROC results are reported in the appendix with similar observations.
- Outlier and novelty detection: The baseline autoencoder performs well at an outlier ratio of 0.1% but drops significantly at 0.5%, whereas clipping and added noise improve anomaly detection in most cases.At a 0.5% ratio, the mini-batch size of 200 yields an average of one outlier per mini-batch, allowing the baseline to learn it.
- Limitations and trade-offs: Differential privacy training can be up to 80 times slower per epoch than baseline training, and a 10% anomalous portion may represent a learnable input pattern rather than outliers.Whether to apply large noise at that scale depends on the application’s requirements.
- Hadoop log anomaly detection: Differential privacy substantially reduces DeepLog false negatives across thresholds, from over a thousand in DeepLog to hundreds or zero in DeepLog+DP.Larger noise reduces false negatives further, although some false positives may be introduced.
- Hadoop log anomaly detection: DeepLog+DP reaches F-measure scores of 96.29% (σ = 0.25, k = 15) and 96.28% (σ = 1, k = 18), above DeepLog’s 90.38%–93.81% range.At 96.28%, DeepLog+DP reports FN = 383 and FP = 762; at 96.29%, it reports FN = 123 and FP = 1040.
- Backdoor attack detection: Backdoor experiments vary poisoning ratio and noise scale while measuring attack effectiveness and poisoning-example detection with AUPR and AUROC.Benign accuracy remains high on clean data, while a 1% poisoning ratio yields a 97.12% backdoor success rate.
5 Related work
Prior work connects outlier and novelty detection with unsupervised learning and differential privacy, while this paper uses privacy-induced stability to improve detection performance. It distinguishes this goal from protecting input privacy during outlier detection and relates its analysis to subgroup performance gaps.
- This paper applies differential privacy to improve anomaly-detection performance, rather than protecting input data during the detection stage.
- Outlier detection identifies rare dataset items unlike the majority, whereas novelty detection targets new observations in low-density regions.
- Differential privacy has been used for privacy protection, robustness to adversarial examples, and bounding inference-attack success.
- The paper relates its approach to evidence that differential privacy can disproportionately reduce accuracy for underrepresented classes and subgroups.
6 Conclusion
The paper applies differential privacy noise to outlier and novelty detection and extends the approach to backdoor attack detection. It combines theoretical analysis with experiments on contaminated image data and Hadoop log anomaly detection to evaluate this approach.
- The paper applies differential privacy noise to outlier and novelty detection, extending the approach to backdoor attack detection.
- Its theoretical results connect anomaly-identification difficulty with privacy parameters and explain effects of privacy and training-set outlier counts.
- Experiments use contaminated MNIST data with autoencoders and a Hadoop file-system log task with DP noise applied to DeepLog.
- The evaluation results show that DP noise is effective for anomaly detection.