Source-linked AI summary
Enhancing Adversarial Defense by k-Winners-Take-All
Chang Xiao, Peilin Zhong, Changxi Zheng
TL;DR
Gradient-based attacks exploit input gradients to find adversarial examples, and prior defenses that hide gradients can remain vulnerable. This paper replaces ReLU with k-WTA, whose input discontinuities hinder gradient-based search. Across the tested white-box attacks and training setups, k-WTA networks outperform ReLU networks in robustness.
Problem
Gradient-based attacks can find imperceptible input perturbations that change a network's output, while existing gradient-hiding defenses remain vulnerable to gradient estimation.
Source: §1 · p. 1Method
The paper replaces common activations such as ReLU with k-WTA, which retains the largest k activations and zeros the rest.
Source: §1 · pp. 1–2Results
Across tested datasets, architectures, training methods, and white-box attacks, k-WTA networks outperform ReLU networks in robustness.
Source: §1 · p. 2Takeaways & Limitations
The authors conclude that k-WTA's discontinuities can hinder gradient-based adversarial search while remaining compatible with successful network training.
Source: §1 · p. 2, §5 · p. 9The analyzed layer property assumes weights initialized from N(0, 1/l) and zero bias.
Source: §3 · p. 6
Abstract
from arXiv · showhide
We propose a simple change to existing neural network structures for better defending against gradient-based adversarial attacks. Instead of using popular activation functions (such as ReLU), we advocate the use of k-Winners-Take-All (k-WTA) activation, a C0 discontinuous function that purposely invalidates the neural network model's gradient at densely distributed input data points. The proposed k-WTA activation can be readily used in nearly all existing networks and training methods with no significant overhead. Our proposal is theoretically rationalized. We analyze why the discontinuities in k-WTA networks can largely prevent gradient-based search of adversarial examples and why they at the same time remain innocuous to the network training. This understanding is also empirically backed. We test k-WTA activation on various network structures optimized by a training method, be it adversarial training or not. In all cases, the robustness of k-WTA networks outperforms that of traditional networks under white-box attacks.
1 INTRODUCTION
Adversarial examples threaten neural networks, while defenses based on hiding gradients remain vulnerable to gradient estimation. The paper replaces ReLU with k-WTA, whose input-space discontinuities are intended to hinder gradient-based attacks while remaining compatible with training.
1 INTRODUCTION
- The paper proposes replacing common activations such as ReLU with k-WTA, leaving other network components and training methods unchanged.The change is intended to require no significant overhead and to work with nearly all existing networks and training methods.Source: §1 · p. 1
- k-WTA retains a layer's k largest activations and zeros the rest, so an input perturbation can change activation patterns and make the input gradient undefined.The network is C0 discontinuous at certain inputs.Source: §1 · p. 2
- The paper argues that discontinuities are dense in input space but sparse in weight space, allowing attacks to be hindered while training remains successful.The proposed explanation relates the sparsity in weight space to its much higher dimensionality than data space.Source: §1 · p. 2
- Across tested datasets, architectures, training methods, and white-box attacks, k-WTA networks outperform ReLU networks in robustness.The experiments include regular and adversarial training and architectures such as ResNet, DenseNet, and Wide ResNet.Source: §1 · p. 2
1.1 RELATED WORK: OBFUSCATED GRADIENTS AS A DEFENSE MECHANISM
- Unlike prior defenses that hide gradients, k-WTA is argued to create densely distributed input discontinuities that hinder gradient estimation and resist smooth-approximation attacks.The paper contrasts its approach with prior discretization and randomness-based methods and discusses BPDA.Source: §1.1 · pp. 2–3
2 k-WINNERS-TAKE-ALL ACTIVATION
The paper's k-WTA activation passes forward only the largest k values in a layer, with a fixed sparsity ratio setting k relative to layer size. It can use standard training methods, with optional fine-tuning for small sparsity ratios.
2.1 DEEP NEURAL NETWORKS ACTIVATED BY k-WINNERS-TAKE-ALL
- k-WTA retains the k largest values in an input vector and sets the others to zero before passing the vector to the next layer.Ties at the kth-largest value are broken by retaining lower-index elements first.Source: §2.1 · p. 3
- A fixed sparsity ratio γ sets k to ⌊γ·N⌋ for a layer with output dimension N; CNN tensors are treated as one flattened vector.The reported runtime cost is O(N), comparable to ReLU's O(N), with no significant overhead.Source: §2.1 · p. 4
- Unlike LWTA's local groups, k-WTA selects the largest activations across the whole layer; the paper reports greater robustness than LWTA.Both are discontinuous, but the paper says LWTA's discontinuity properties and training effects remain unclear.Source: §2.1 · p. 4
2.2 TRAINING k-WTA NETWORKS
- Standard optimization methods can train k-WTA networks, but γ ≤ 0.2 can slow convergence because fewer active neurons reduce effective network width.An optional iterative fine-tuning procedure gradually reduces γ and is used in practice only when γ < 0.2.Source: §2.2 · p. 4
- Under PGD, k-WTA ResNet18 logits change discontinuously with input perturbations, while its loss changes largely smoothly with network weights during training.The figure contrasts this with ReLU ResNet18, where PGD can readily find adversarial examples as logits change smoothly with input.Source: Figure 3 · p. 5
3 UNDERSTANDING k-WTA DISCONTINUITY
k-WTA’s activation pattern changes can make the network discontinuous across densely distributed input regions. Despite these discontinuities, sufficiently wide networks can remain trainable because different data points tend to have distinct activation patterns.
3 UNDERSTANDING k-WTA DISCONTINUITY
- A k-WTA activation pattern is the set of indices of a layer’s k largest values, and changing that set can abruptly change the next layer’s output.When an index leaves the set, its weight-column contribution vanishes as another column’s contribution begins.Source: §3 · p. 5
- Smaller γ can produce larger discontinuity jumps, while changing γ also affects how densely linear-region boundaries are distributed.The network is piecewise linear within fixed activation patterns, but unlike ReLU’s joined regions, k-WTA’s linear regions are disconnected.Source: §3 · p. 5
- Theorem 1 implies that wider layers need smaller perpendicular perturbations to change an activation pattern, making linear regions finer.The analysis treats positive rescalings of an input as leaving its activation pattern unchanged and measures perturbation by perpendicular distance.Source: §3 · p. 6
- With sufficient width, distinct data points are likely to have disjoint activation patterns, allowing their prediction weights to be optimized almost independently.The theorem assumes that no two data points are positive scalar multiples of one another.Source: §3 · p. 6
- Although individual activation-pattern changes can cause loss jumps, their effects are negligible in the summed training loss when only a small set of patterns changes.The paper attributes this to the relatively low chance that optimization changes any particular data point’s pattern.Source: §3 · p. 6
4 EXPERIMENTAL RESULTS
Across training methods, k-WTA improves white-box robustness over ReLU networks. Experiments also show that sparsity ratio affects accuracy and that k-WTA changes gradient-based attack loss landscapes.
4.1 ROBUSTNESS UNDER WHITE-BOX ATTACKS
- The evaluation compares clean accuracy Astd and robust accuracy Arob across white-box attacks and training methods, and also reports transfer-based black-box results on CIFAR-10, SVHN, and MNIST.The white-box attacks include PGD, Deepfool, C&W, and MIM; training includes natural training, AT, TRADES, and FAT.Source: §4 · p. 6, §4.1 · pp. 6–7
- k-WTA improves white-box robustness over ReLU across the evaluated training methods, with Table 1 reporting worst-case accuracy across attacks for CIFAR-10 and SVHN.The paper reports that this improvement is not tied to a specific training method and comes from replacing ReLU with k-WTA.Source: Table 1 · p. 7, §4.1 · p. 7
- Transfer-based black-box attacks are not always significantly weaker on k-WTA, but the paper reports that gradient-based white-box attacks remain stronger even against an independently trained target copy.Source: §4.1 · p. 7
4.2 VARYING SPARSITY RATIO γ AND MODEL ARCHITECTURE.
- When γ is above approximately 0.1, reducing it has little effect on standard accuracy but increases robust accuracy; below that level, both accuracies fall sharply.The reported trends are from CIFAR experiments across ResNet18, DenseNet121, and Wide ResNet architectures.Source: §4.2 · p. 8, Figure 4 · p. 8
4.3 LOSS LANDSCAPE IN GRADIENT-BASED ATTACKS
- k-WTA models have more non-convex, non-smooth attack-loss landscapes than ReLU models, making estimated gradients less useful for adversarial search.Adversarial training compresses the loss range, and larger γ produces a smoother k-WTA loss surface.Source: §4.3 · pp. 8–9, Figure 5 · p. 8
5 CONCLUSION
The paper proposes replacing widely used activation functions with k-WTA to improve neural-network robustness against adversarial attacks, while leaving other network components and training methods unchanged.
5 CONCLUSION
- Replacing a widely used activation function with k-WTA is the paper’s proposed change for improving neural-network robustness against adversarial attacks.The authors describe the replacement as requiring no significant overhead and as harmless to training.Source: §5 · p. 9
Supplementary Document Enhancing Adversarial Defense by k-Winners-Take-All
White-box attacks exploit model gradients, while adversarial training can be costly and some regularization defenses are brittle under iterative attacks. The section also explains how k-WTA activation-pattern swaps create output discontinuities whose size depends on sparsity.
A OTHER RELATED WORK
- White-box attacks use model gradients to construct adversarial examples, whereas black-box attacks lack model details and may rely on queries or transfer.The section identifies FGSM and PGD as first-order attacks and describes gradient-free black-box methods as less effective than gradient-based white-box attacks.Source: A OTHER RELATED WORK · p. 13
- Adversarial training is a leading defense but can be prohibitively slow at scale; a revised method aims for regular-training time with comparable accuracy.It generates adversarial images during training and requires finding them on the fly; the cited revision targets comparable standard and robust accuracy.Source: A OTHER RELATED WORK · p. 13
- Some regularization defenses without adversarial training are brittle under stronger iterative attacks, while the paper reports k-WTA improves robustness under PGD and other attacks.The reported k-WTA comparison is specifically for use without adversarial training.Source: A OTHER RELATED WORK · p. 14
B DISCONTINUITY JUMP OF Wφk(x)
- A k-WTA swap changes the layer output by (Wj − Wi)x*, with jump magnitude depending on the kth-largest value and thus on sparsity ratio γ.Smaller γ means a smaller effective k for fixed vector length, making the kth-largest value larger; at k = 1, x* is the largest value.Source: B DISCONTINUITY JUMP OF Wφk(x) · p. 14
C THEORETICAL PROOFS
The section sets out the k-WTA network notation and activation pattern used in the theoretical proofs. Fixing activation patterns yields linear regions, while changes in pattern can produce discontinuities.
C THEORETICAL PROOFS
- An n-layer k-WTA network is expressed as nested layer weights, biases, and k-WTA activations, followed by an output layer.The section denotes layer weights and biases by W(i) and b(i), respectively.Source: C THEORETICAL PROOFS · p. 14
- The activation φk retains the top-k entries of an input vector and sets all other entries to zero.Ties are resolved in favor of the smaller index, and A(y) denotes the indices of the retained entries.Source: C THEORETICAL PROOFS · p. 14
- Different activation patterns place Wφ(y) in different linear regions, and the resulting function may be discontinuous.The section notes that wider networks may be more discontinuous with respect to their input.Source: C THEORETICAL PROOFS · p. 14
C.1 DISCONTINUITY WITH RESPECT TO THE INPUT
The theoretical results show that sufficiently wide randomly initialized layers can change activation patterns under small directional perturbations. They also give conditions for distinct input points to have distinct patterns, including a random-data example.
C.1 DISCONTINUITY WITH RESPECT TO THE INPUT
- For a sufficiently wide layer initialized with random Gaussian weights and zero bias, small enough angular changes can yield different activation patterns with high probability.The theorem formalizes this for α in (0.5, 1), under a row-norm condition, and extends the distinction to inputs whose normalized inner product is at most α.Source: §C.1 · pp. 15, 20, 21
- Inputs sharing an activation pattern form a convex set, so a demonstrated pattern change at an intermediate point implies a change at a more distant point on the same line.The convexity follows because a fixed pattern is specified by an intersection of half-hyperplanes.Source: §C.1 · p. 15
- With sufficiently wide random Gaussian weights, a perturbation meeting the theorem’s perpendicular-distance condition changes the activation pattern with high probability.The formal result assumes sparsity ratio γ in (0, 0.48) and states the condition as ||Δx||₂²/||x||₂² ≥ β for x′ = c·(x + Δx), with Δx perpendicular to x.Source: §C.1 · p. 21
- For Bernoulli training points under the stated dimension and sparsity conditions, a sufficiently wide random layer gives every pair a different activation pattern with probability at least 0.9.The example assumes m ≥ Ω(log N), entry probability p in (100 log(N)/m, 0.5), and l ≥ Ω(m/γ · log(m/γ)).Source: §C.1 · p. 22
C.2 DISJOINTNESS OF ACTIVATION PATTERNS OF DIFFERENT INPUT POINTS
The section establishes conditions under which sufficiently wide random layers assign disjoint activation patterns to separated data points, then uses that disjointness to construct a readout vector. It also introduces additional experimental settings and results.
C.2 DISJOINTNESS OF ACTIVATION PATTERNS OF DIFFERENT INPUT POINTS
- For data points with pairwise normalized inner products at most α, a sufficiently wide random Gaussian layer yields disjoint activation patterns with probability at least 1 − δ.The theorem assumes α in (0.5, 1) and fixed k ≥ 1; the proof concludes that activation sets for every distinct pair do not intersect.Source: §C.2 · pp. 24–25
- Given disjoint patterns, the section constructs a vector v by assigning selected active coordinates to the corresponding data labels.For each input, the proof selects an active index not active for the other inputs and sets that coordinate of v using the label and the corresponding weight-row score.Source: §C.2 · p. 25
D ADDITIONAL EXPERIMENTAL RESULTS
- The additional experimental-results section presents experiment settings and further results for evaluating and empirically understanding k-WTA network robustness.Source: D ADDITIONAL EXPERIMENTAL RESULTS · p. 26
D.1 EXPERIMENT SETTINGS
The experiments compare k-WTA networks with ReLU counterparts while keeping other network modules and training procedures unchanged. They use multiple architectures, datasets, and white-box attack configurations.
D.1 EXPERIMENT SETTINGS
- All ReLU activations are replaced with k-WTA while BatchNorm, convolution, pooling, and other modules remain unchanged.Experiments use publicly available ResNet18, DenseNet121, and Wide ResNet implementations in PyTorch.Source: §D.1 · p. 26
- CIFAR-10 and SVHN networks use SGD with momentum 0.9, batch size 256, and a learning-rate schedule of 0.1 through epoch 50 and 0.01 through epoch 80.For naturally trained k-WTA networks at γ = 0.1, incremental training starts at γ = 0.2 and decreases γ by 0.005 every two epochs.Source: §D.1 · p. 26
- White-box attacks use ℓ∞ perturbations of 0.031 on CIFAR-10 and 0.047 on SVHN, with attack-specific iteration settings.PGD uses 40 iterations with random starts and step size 0.003; other attacks use the stated configurations or Foolbox defaults.Source: §D.1 · p. 26
D.2 EFFICACY OF INCREMENTAL TRAINING
Incremental fine-tuning improves both standard and robust accuracy when the k-WTA sparsity ratio is below 0.2, supporting its use at small γ.
D.2 EFFICACY OF INCREMENTAL TRAINING
- When γ < 0.2, incremental fine-tuning improves robust accuracy over regular training.The comparison covers a sweep of sparsity ratios in ResNet18 and Wide ResNet.Source: §D.2 · p. 26
- When γ < 0.2, incremental fine-tuning also improves standard accuracy over regular training.The figure compares training with and without incremental fine-tuning across sparsity ratios.Source: §D.2 · p. 26
- The reported gains make fine-tuning worthwhile when the sparsity ratio is small.The authors identify γ < 0.2 as the condition for the observed improvements.Source: §D.2 · p. 26
D.3 ADDITIONAL RESULTS ON CIFAR-10
Additional CIFAR-10 experiments report that k-WTA improves worst-case robustness over ReLU across tested architectures and training methods. The section also compares k-WTA with LWTA and evaluates transfer attacks.
D.3 ADDITIONAL RESULTS ON CIFAR-10
- Across tested architectures and training methods, k-WTA consistently improves worst-case robustness Arob over ReLU.Arob is measured under the most effective of PGD, C&W, Deepfool, and MIM.Source: §D.3 · p. 26, Table 2 · p. 27
- With the same sparsity ratio, k-WTA consistently outperforms LWTA in both standard accuracy Astd and robustness Arob across tested architectures and training methods.The authors conclude that k-WTA is more suitable than LWTA for defending against adversarial attacks.Source: §D.3 · p. 27, Table 2 · p. 27
- Transfer attacks are more difficult against k-WTA than ReLU, and k-WTA adversarial examples are also more difficult to transfer to other models.The comparison includes transfer attacks between independently initialized versions of the same model; k-WTA also performs better than ReLU under this strongest black-box attack.Source: §D.3 · p. 28, Table 3 · p. 28
D.4 MNIST RESULTS
On MNIST, k-WTA improves robustness under the tested training methods, including natural training. Loss-landscape visualizations show that sparsity and adversarial training affect different aspects of the landscape.
D.4 MNIST RESULTS
- 62.2% robust accuracy is reported for naturally trained k-WTA on MNIST, significantly outperforming the ReLU network.The evaluation uses PGD attacks with perturbation size ϵ = 0.3.Source: §D.4 · p. 28
- k-WTA consistently improves MNIST robustness under all evaluated training methods.The experiments use a 3-layer CNN and evaluate robust accuracy under PGD attacks.Source: §D.4 · p. 28
D.5 LOSS LANDSCAPE VISUALIZATION
- Smaller γ makes the k-WTA loss surface more discontinuous and spiky, while larger γ tends to smooth it.Adversarial training reduces the range of loss values but does not make the loss surface smoother.Source: §D.5 · p. 28, Figure 7 · p. 29