Source-linked AI summary
Improving the Robustness of Deep Neural Networks via Stability Training
Stephan Zheng, Yang Song, Thomas Leung, Ian Goodfellow
TL;DR
Deep networks can be unstable under small, naturally occurring visual distortions, complicating large-scale vision tasks even when image semantics remain unchanged. The paper introduces stability training using distorted input copies and an additional stability objective; across classification, ranking, and near-duplicate detection, stabilized models are reported to be more robust while maintaining or improving original performance. The method has an added forward-pass computational cost because it processes distorted examples alongside originals.
Problem
Small distortions from common image processing can significantly alter neural-network embeddings and outputs, challenging robust performance on large-scale noisy visual data.
Method
Stability training combines an additional stability objective with training on distorted copies of inputs to make predictions more constant around the data.
Results
Stabilized models show robust performance and significantly outperform unstabilized models on noisy and corrupted data across near-duplicate detection, similar-image ranking, and classification.
Takeaways & Limitations
Stability training makes networks more robust to random cropping, JPEG compression, and thumbnail resizing while maintaining or improving original-task performance.
Takeaways & Limitations
Stability training effectively doubles the training batch size during the forward pass, introducing significant extra computational cost.
Abstract
from arXiv · showhide
In this paper we address the issue of output instability of deep neural networks: small perturbations in the visual input can significantly distort the feature embeddings and output of a neural network. Such instability affects many deep architectures with state-of-the-art performance on a wide range of computer vision tasks. We present a general stability training method to stabilize deep networks against small input distortions that result from various types of common image processing, such as compression, rescaling, and cropping. We validate our method by stabilizing the state-of-the-art Inception architecture against these types of distortions. In addition, we demonstrate that our stabilized model gives robust state-of-the-art performance on large-scale near-duplicate detection, similar-image ranking, and classification on noisy datasets.
1. Introduction
Small, naturally occurring visual distortions can destabilize deep-network embeddings and predictions even when human-perceived content and labels remain unchanged. The paper proposes stability training and evaluates it across classification, ranking, and near-duplicate detection.
- Motivation: Small distortions from compression, resizing, and cropping can substantially confuse feature extractors despite preserving visual semantics and ground-truth labels.These distortions commonly arise in large-scale, un-curated visual datasets.
- Motivation: Feature instability creates near-duplicate detection failures when distorted duplicates receive feature distances comparable to dissimilar image pairs.Figure 1 illustrates this threshold-confusion failure for JPEG versions of images.
- Motivation: Indistinguishable neighboring video frames can receive sharply different class scores, such as 27% versus 63% for ’fox’.At a confidence threshold above 50%, only the right frame is correctly recognized.
- Approach: Stability training adds a stability objective and trains on distorted input copies, encouraging locally constant predictions while preserving the original learning objective.The method is designed to improve robustness while preventing underfitting on the original task.
- Contributions: The method is presented as a fast, low-cost technique that maintains or improves original performance and is evaluated on near-duplicate detection, similar-image ranking, and classification.Experiments stabilize Inception-based classification and ranking networks.
- Contributions: The paper reports robust performance and significant improvements over unstabilized models on noisy and corrupted data.The contributions include visualizing which perturbations the model becomes robust to.
2. Related work
Related work addresses instability from adversarial perturbations and hard-positive augmentation. This paper focuses instead on naturally occurring perturbations that widely arise in practical visual data.
- Adversarial examples: Adversarial-example research studies intentionally engineered imperceptible perturbations that can change model class labels.The paper documents that similar instability also occurs without intentional human intervention.
- Adversarial examples: This work differs from adversarial-input studies by targeting practically widespread perturbations rather than contrived inputs unlike the original data.The distinction concerns the source and realism of the perturbations considered.
- Data augmentation: Hard-positive data augmentation can improve predictive performance and consistency for represented perturbation classes.Its stability benefits depend on finding visually similar examples that the model currently misclassifies with high confidence.
3. Stability training
Stability training adds an objective that keeps network outputs similar for original and perturbed inputs while preserving the original task objective. It generates perturbed copies during training and adapts the stability distance to feature-ranking or classification tasks.
- 3.1. Stability objective: Stability training combines the original-task loss with a stability loss that encourages similar outputs for an image and its distorted copy.The stability term is weighted by α, while the original loss remains evaluated on the natural input.
- 3.1. Stability objective: The method samples a new perturbed copy for each training example and optimizes the combined objective over model weights.The perturbation-generation mechanism and task-specific distance complete the optimization setup.
- 3.2. Sampling perturbed images x′: Pixel-wise uncorrelated Gaussian noise provides a general perturbation sampler, with uniform per-pixel variance σ² treated as a hyper-parameter.This approach is intended to cover many perturbation classes rather than explicitly selecting each class.
- 3.2. Sampling perturbed images x′: To avoid underfitting, stability training applies the original loss only to natural inputs and uses the stability loss for both natural and perturbed inputs.The paper contrasts this design with ordinary data augmentation using noisy examples.
- 3.3. Stability for feature embeddings: For feature embeddings, stability training perturbs every image in a ranking triplet and preserves the triplet margin relationship using a distance-based objective.The ranking objective requires the query-positive distance plus margin g to remain below the query-negative distance.
- 3.4. Stability for classification: For classification, the method retains cross-entropy for the original task and uses KL-divergence to measure correspondence between natural and perturbed predictions.The KL-divergence is applied to the likelihoods produced on the two inputs.
4. Implementation
The implementation applies stability training to Inception-based classification and ranking networks, exposing them to distorted inputs and combining task and stability objectives during optimization.
- 4.1. Network: Inception serves as the base convolutional architecture for classification and triplet ranking experiments.The ranking network uses Inception to process images and learn similarity features.
- 4.1. Network: Stability training generates perturbed copies with a Gaussian sampler and processes original and perturbed images through the same network.The general setup adds a perturbation sampler and a stability objective layer.
- 4.1. Network: The task objective uses the original output, while the stability loss uses both original and perturbed outputs before their gradients are combined.Triplet ranking additionally processes three images to compute its ranking objective.
- 4.2. Distortion types: The experiments evaluate original and transformed datasets using perturbations from lossy image processes, including JPEG compression, thumbnail resizing, and random cropping.These distortions represent transformations commonly found in real-world visual data.
- 4.3. Optimization: Optimization uses mini-batch stochastic gradient descent with momentum, dropout, RMSprop, batch normalization, and grid-search hyper-parameter tuning.To reduce the doubled forward-pass batch cost, stability training is applied during fine-tuning, with experiments fine-tuning only final fully connected layers.
5. Experiments
Experiments evaluate stability training across near-duplicate detection, feature robustness, similar-image ranking, and ImageNet classification. Stabilized models improve robustness to common distortions while preserving or improving performance on original data.
- Near-duplicate detection: Stabilized ranking features improve near-duplicate detection across thumbnail, JPEG, and cropping distortions at fixed precision or recall.Recall increases by 1.0% at 99.5% precision for thumbnail near-duplicates and by 3.0% at 98% precision for JPEG near-duplicates.
- Near-duplicate detection: 86% of stabilized-feature distances are below 0.1, compared with 76% for baseline features on 650,000 JPEG-50 near-duplicate pairs.The stabilized distance distribution is more concentrated toward 0.
- Near-duplicate detection: Stability training makes previously confusing JPEG and cropping near-duplicates closer in feature space, enabling detection at lower thresholds.The qualitative examples include localized, structured perturbations unlike typical Gaussian-noise samples.
- Similar-image ranking: Stabilized ranking improves ranking scores on both original and transformed evaluation data, while the baseline degrades on all distorted versions.The comparison uses ranking score-at-top-30 on 14,000 hand-labeled triplets.
- Image classification: On ImageNet, both models retain state-of-the-art performance on original data, while stabilization yields higher precision on distorted data.At high distortion levels, the gap reaches 5% to 6% in top-1 and top-5 precision; at the highest JPEG quality, performance is comparable.
6. Conclusion
The paper proposes stability training to make deep networks robust to natural visual distortions. Across detection, ranking, and classification, stabilized models are more robust on noisy data while retaining or improving performance.
- Conclusion: Stability training trains networks to remain constant on input images with small perturbations from cropping, JPEG compression, and thumbnail resizing.The method is described as lightweight and effective.
- Conclusion: Stabilized models show significantly more robust performance than unstabilized models on near-duplicate detection, similar-image ranking, and classification.The conclusion reports robustness across all three noisy-data tasks.