Biologically inspired protection of deep networks from adversarial attacks

1. Introduction

• Small input perturbations can make an image indistinguishable to a human while completely altering a network’s class prediction.The paper frames this as evidence that machine learning models may not fundamentally understand their trained tasks.Source: §1 · p. 1
• Adversarial examples matter scientifically because they highlight a discrepancy between neural-network computations and human vision, and technologically because attacks can generalize across architectures and training data.The authors note that deep networks also serve as foundational models in neuroscience for the visual processing stream.Source: §1 · p. 1
• The proposed approach takes inspiration from active dendritic conductances, which disrupt linear summation and can produce high-threshold, saturated, switch-like neural behavior.The authors suggest this regime may make it difficult for small perturbations to propagate into large output errors.Source: §1 · p. 2
• The paper reports state-of-the-art robustness on gradient-based MNIST adversarial examples without adversarial examples during training, with 2–7% error rates and little to no degradation in original test performance.Its networks also show robustness to targeted iterative attacks, including second-order methods, and improve on adversarially trained networks on fast-gradient attacks.Source: §1 · p. 2
• The paper analyzes robustness through information geometry and reports flat, compressed representations sensitive to few input directions, alongside highly kurtotic weights also found in the brain.It also reports progressively clustered representations for different image classes and says weight kurtosis can confer robustness even in linear classifiers.Source: §1 · p. 2

2. Adversarial Example Generation

• The network maps activity vectors through layer-wise weights and biases, with component-wise nonlinearities transforming neuron inputs into activities.The final activity vector represents the network’s composite transformation from input to output.Source: §2 · p. 2
• A bounded perturbation Δx0 turns a correctly classified input x0 into x0 + Δx0, which the network classifies as a different goal class yG.The bound is intended to keep the perturbation largely imperceptible to a human observer.Source: §2 · p. 2
• The fast gradient sign method linearizes the network’s input-output map near the test example and seeks bounded l∞ perturbations that maximize the network’s cost.The section gives the exact solution to this bounded optimization and uses ε = 0.25 for MNIST, where pixel values are in [0, 1].Source: §2 · pp. 2–3
• Adversarial training interpolates the ordinary cost with a cost on adversarial examples generated from the current model at every iteration.For maxout networks on MNIST, adversarial training with α = 0.5 lowered adversarial-example error from 89.4% to 17.9%.Source: §2 · p. 3
• The authors seek robustness without training on adversarial examples, arguing that intrinsically robust networks should depart from the high-dimensional linear regime associated with gradient-based adversaries.Their proposed direction is to make networks operate in a nonlinear, saturating regime.Source: §2 · p. 3

3. Saturating Networks

• Jacobian regularization directly targets input sensitivity, but storing the Jacobian can be computationally expensive for sufficiently large networks.Its dimensions can become cumbersome to store in memory.Source: §3 · p. 3
• A contractive penalty on layer-wise Jacobians indirectly encourages saturation but can be difficult to compute efficiently for wide layers and can degrade test accuracy by limiting learning capacity.The penalty uses the Frobenius norm of each layer-wise Jacobian.Source: §3 · p. 3
• Saturating autoencoders use an activation penalty to encourage saturation and limit reconstruction of points far from the data manifold.The penalty is described as simple and computable in O(|h|) time.Source: §3 · p. 3
• For sigmoid activations, the complementary function can be taken as |σ′(z)| = |σ(z)(1 − σ(z))| because the sigmoid is monotonic.The function represents an activation’s distance to the nearest saturation region.Source: §3 · p. 3

• The saturating penalty is applied to every layer, including the readout layer before softmax, and the sigmoid complementary function is |σ′(z)|.For ReLU, the complementary function is ReLU itself.Source: §4 · pp. 3–4
• Saturating networks maintain vanilla test accuracy with little degradation while outperforming adversarially trained counterparts on adversarial examples generated from the test set.Table 1 reports classification accuracies; the supplied passages do not give its numerical values.Source: Table 1 · p. 4, §4 · p. 4
• Experiments trained three-layer sigmoid and ReLU MLPs and CNNs on 10-class MNIST, with separately optimized architectures and Adam.The CNN used the stronger penalty only at its last layer.Source: §4 · p. 4
• Annealing the penalty strength during training was essential; finding an annealing schedule was easier for CNNs than for MLPs.The reported maximum λ values were 1.74 for the sigmoid MLP, 3.99 × 10^-8 for the ReLU MLP, and 10^-5 for the CNN.Source: §4 · p. 4

Weight and activity patterns

• Saturating MLPs have weight distributions with a larger range and much greater positive excess kurtosis than vanilla MLPs.Sigmoid weights extend to extreme values at both ends, while saturating ReLU weights extend to extreme negative values; high kurtosis can also act as a linear protection mechanism against adversarial examples.Source: §5 · pp. 4–5, Figure 1 · p. 4
• Pre-nonlinearity activations across 10,000 test examples tend toward extreme values, consistent with operation in the nonlinearities’ saturation regimes.Source: §5 · p. 5, Figure 2 · p. 5

Internal representations

• The representation analysis compares pairwise squared distances among 1,000 randomly selected test examples, with 100 examples drawn from each class.Source: §5 · p. 5
• Saturating networks progressively increase between-class dissimilarity across layers while within-class dissimilarity remains close to zero.This yields tight class clusters that are farther apart than those in vanilla networks, likely requiring larger input perturbations to cross decision boundaries in intermediate and output layers.Source: §5 · p. 5, Figure 3 · p. 5

6. The geometry of saturating networks

• The Fisher-information metric is pulled back to input space, assigning lengths according to the resulting motion in output probability space.The number of large Jacobian singular values indicates how many input directions are locally sensitive.Source: §6 · pp. 5–6
• Along source-to-target interpolations, saturating networks have locally flat length elements interrupted by sharp peaks at probability transitions, whereas vanilla networks change more smoothly.Long input-space movements can produce little output movement in the flat regions.Source: §6 · p. 6, Figure 4 · p. 6
• At probability transitions, vanilla networks inflate multiple Jacobian singular values, while saturating networks inflate only one, suppressing sensitivity in orthogonal input directions.This constrains the directions of expansion available to adversaries.Source: §6 · p. 6, Figure 4 · p. 6
• Trained networks contract input space near decision-volume centers and expand it near decision boundaries; saturating networks do this more extremely than vanilla networks.Source: §6 · p. 6

7. More powerful iterative adversaries

• The first-order attack starts from a correctly classified source image and uses gradient descent on cross-entropy to make a chosen incorrect target label more likely.Adam adapts the learning rate; starts include the least- or most-confident image in each source class.Source: §7 · p. 7
• Adam-based targeted attacks failed on some saturated-network cases even after 30 million iterations, although the images changed and mean pixel distance converged.The vanilla network was misclassified in either confidence condition, usually in fewer than 10 iterations.Source: §7 · p. 7
• After 1000 L-BFGS iterations, most least-confident source images still resisted targeted misclassification in saturated networks; vanilla networks were fooled, usually within 1–2 iterations.With the most-confident sources, only five saturated-network cases were fooled after 1000 iterations.Source: Figure 5 · p. 7, §7 · p. 7
• The second-order comparison uses quasi-Newton L-BFGS to minimize cross-entropy, with an approximate Hessian and a line-searched learning rate.Source: §7 · p. 7

8. Role of weight kurtosis: a linear mechanism for robustness to adversaries

• Saturating networks have high-kurtosis weight distributions in every layer, and such distributions are prevalent in biological neural networks.Source: §8 · p. 8
• The analysis uses an orthogonal-prototype linear classifier and an l∞ fast-sign perturbation; robustness increases when the denominator’s l1 norm is small.The denominator can be large relative to the numerator in high-dimensional spaces, particularly for dense uniform vectors.Source: §8 · p. 8
• Kurtotic sampling is expected to pair a large ||w1||2^2 from extreme values with a small ||w2−w1||1 from the distribution’s peak near zero.The test sampled unit-norm vectors of dimension 20,000 from a Pearson Type VII distribution.Source: §8 · p. 8
• Higher excess kurtosis raises the scaled ratio almost 40% over the Gaussian baseline; saturated-network weights raise it up to 300% over vanilla-network weights.The comparison uses the weight distributions from corresponding layers of saturated and vanilla networks.Source: §8 · p. 8

9. Discussion

• Saturated networks improve performance over adversarially trained networks on fast gradient sign examples and are relatively robust to targeted iterative attacks, including second-order methods.Source: §9 · p. 8
• The proposed biologically inspired strategy finds highly nonlinear networks in a saturated regime without computing adversarial examples.Source: §9 · p. 8
• Information-geometric analyses reveal highly flat, low-dimensional internal representations that still widely separate images from different classes.Source: §9 · p. 8
• Highly kurtotic weights found in the networks and in brains can act as a linear mechanism against adversarial examples.Source: §9 · p. 8