Adversarial Training Without Input Gradients via Low-Rank Householder Expansions

Tiana C. Johnson; Donsub Rim

Source-linked AI summary

Adversarial Training Without Input Gradients via Low-Rank Householder Expansions

Tiana C. Johnson, Donsub Rim

arXiv:2608.26963v1Aug 27, 2026cs.LG

TL;DR

Small-norm adversarial examples motivate reducing the costly input-gradient search used in adversarial training. The paper uses low-rank Householder expansions to extract sensitive input directions from forward-pass activations and trains against them without input derivatives. On MNIST, this approach matches short- and long-horizon PGD training within small relative ℓ2 budgets while costing only 2.8 PGD steps per epoch, but its scope is limited beyond the local regime and to MNIST.

Problem
Adversarial training is robust but costly because its inner maximization is commonly approximated by repeated projected-gradient steps requiring input gradients.
Method
The method uses LRHE to identify sensitive input-space directions from the composed network and forward-pass activation pattern, then replaces the inner maximization with a direction-based penalty.
Results
2.8 PGD steps per epoch matches 3-step PGD robustness for ε ≤ 0.02 and 40-step robustness for ε ≤ 0.012 on MNIST.
Takeaways & Limitations
Input-gradient-free adversarial training can reach similar performance to 40-step training because relevant local geometry is present in the activation pattern.
Takeaways & Limitations
The approach is evaluated on MNIST and its expansion describes a single affine cell, so correspondence with loss-increasing directions is limited beyond the local small-budget regime.

Abstract

from arXiv · show

This work concerns adversarial training against the small-norm adversarial examples that arise from the inherent input instability of a trained deep neural network. Examples in this class are small as measured in the relative $\ell^2$-norm, and therefore lie in the neighborhood of the input on which the model acts approximately linearly, the regime in which the perturbation remains imperceptible. We first show that such examples can be computed directly from the trained network parameters, without input gradient iterations, by means of a linearization called the low-rank Householder expansion (LRHE). The expansion describes the composed affine map rather than any individual layer, and the directions it identifies are read from the activation pattern already available in the forward pass. We then propose a simple adversarial training scheme built on this construction. No differentiation with respect to the input is performed at any point: training requires only additional forward evaluations, with weight parameters updated by the standard backward pass, and the inner maximization of the usual min-max formulation is eliminated entirely. That such a regularizer exists is our main finding: the methods that dispense with the inner search all obtain their local geometry by differentiating with respect to the input, and we show this is not necessary. The regularizer costs the equivalent of $2.8$ PGD steps per epoch, an $8.7\times$ reduction relative to 40-step adversarial training on MNIST and below the cost of 3-step training. The resulting models match three-step PGD adversarial training for relative $\ell^2$ budgets $\varepsilon \le 0.02$ and 40-step training for $\varepsilon \le 0.012$, falling away beyond, consistent with the locality of the expansion.

1. Introduction

Adversarial training remains effective but costly because its inner search requires repeated input-gradient evaluations. This work replaces that search with LRHE directions extracted from the forward-pass activation pattern and reports competitive MNIST robustness at substantially lower cost.

Motivation: K projected-gradient ascent steps make adversarial training roughly K times more expensive than ordinary training.Each step requires a gradient of the loss with respect to the input.
Method: For small relative ℓ2 perturbations, LRHE exploits the piecewise-affine structure of ReLU networks to identify adversarial directions without input-gradient iterations.The network acts as a single affine map within a fixed activation-pattern region.
Method: LRHE directions are available from the forward pass and describe sensitivity of the composed network map rather than an individual weight matrix.The relevant vectors arise from the partial compositions surrounding an activation and the Householder representation.
Method: The proposed training removes the inner search, penalizing responses along directions recomputed at each forward pass without differentiating with respect to the input.Weight parameters are still updated with the standard backward pass.
Experiments: 2.8 PGD steps per epoch is the reported training cost, an 8.7× reduction relative to 40-step adversarial training and below the cost of 3-step training.These results are reported on MNIST.
Experiments: The resulting models match 3-step PGD robustness for relative ℓ2 budgets ε ≤ 0.02 and 40-step robustness for ε ≤ 0.012.The reported parity is limited to the budget range where the expansion remains descriptive.
Scope: The study does not claim state-of-the-art robust accuracy at all budgets or transfer of the approximation to higher-resolution data.Experiments are confined to MNIST and no certificates are provided.

2. Preliminaries

The paper defines sensitivity adversarial examples using relative norm budgets and studies feedforward ReLU networks. Its preliminaries introduce LRHE as an input-dependent low-rank representation whose directions expose nonlinear sensitivity and possible instability.

Adversarial examples: Sensitivity adversarial examples are perturbations whose relative ℓp norm is bounded while changing the model’s softargmax output.The paper focuses primarily on p = 2 and also reports some results for p = 8.
Network model: The model class is a feedforward ReLU neural network composed of affine maps and entry-wise ReLU activations.The network parameters include weight matrices and biases.
Threat model: Adversarial examples are formulated by increasing a loss while holding the clean input and trained weights fixed.The resulting optimization problem and attack procedure define a threat model.
Low-rank Householder expansion: Within a fixed activation pattern, LRHE represents the network’s nonlinear behavior through an input-dependent affine term with rank at most L − 1.The expansion is low-rank when L − 1 is small relative to the layer widths.
Low-rank Householder expansion: LRHE’s column and row spaces are spanned by vectors formed from the network weights and Householder reflection vectors.These spaces describe the low-rank affine transformation associated with the input-dependent term.
Householder representation: The Householder representation of ReLU is distinct from Householder activations because it is a representation used in the expansion rather than an imposed activation definition.The reflection vector is determined from the hidden state during evaluation.
Instability: A dominant singular direction of the input-dependent term can produce large output changes from small input perturbations, even when the input-independent affine term is well-conditioned.This separates linear conditioning of F0 from nonlinear conditioning of Fσ.
Related limitations: Weight-norm control alone does not generally produce adversarial robustness, motivating more targeted treatment of the network’s nonlinear structure.The cited alternative constrains products of layer norms through weight decay.

3. Adversarial training based on LRHE

LRHE identifies network-sensitive input directions from the activation pattern without input differentiation, allowing adversarial training to replace the inner maximization with a regularized minimization. The regularizer acts on composed mappings and is computationally cheaper than multi-step PGD, while remaining local and not strictly loss-maximizing.

LRHE-based training objective: LRHE directions are obtained from the activation pattern and identify directions in which the composed map amplifies most strongly, without differentiating with respect to the input.The expansion uses the composed affine map rather than individual layers.
LRHE-based training objective: The proposed regularizer restricts the adversarial inner maximization to LRHE directions, so the usual min-max problem collapses to a minimization.It retains penalties for perturbations of size ε in directions that strongly affect the model while eliminating K input-gradient computations.
Regularizer construction: The output basis is computed by propagating nonlinear corrections through subsequent affine maps, then used to penalize changes in output logits along those directions.The basis computation requires additional forward evaluations that omit nonlinearities.
Training cost: Training requires one forward and one backward pass per gradient computation, making the regularized objective more efficient than PGD training with multiple parameter- and input-gradient passes.The regularizer's forward cost scales with the number of nonlinear layers, O(L), relative to standard forward evaluation.
Scope and limitations: The LRHE directions depend on the input and network but not the label, so they describe model sensitivity rather than the nearest decision-boundary direction.The correspondence with loss-increasing directions is exact only when amplifying and loss-increasing directions coincide; LRHE also describes a single affine cell near the input.

4. Convolutional Neural Network for MNIST

The MNIST experiments rewrite max-pooling into an equivalent feedforward ReLU representation so LRHE can be applied to the CNN. The resulting clean model and attack setup establish the experimental baseline, while the rewritten pooling layers add basis functions without changing the network function.

Clean training and adversarial examples: The MNIST CNN uses two convolutional-ReLU-max-pooling blocks followed by two linear layers, and standard training achieves 99.48% test accuracy.The model is trained with cross entropy and Adam at learning rate 1e-3 for 20 epochs.
Clean training and adversarial examples: AutoAttack at relative ℓ2 radius ε = 0.075 reduces the clean model's accuracy to 88.40%.The ensemble uses fixed hyperparameters and raises no automatic reliability warnings.
Max-pooling layers in feedforward form: Max-pooling is rewritten as a composition of affine maps and pointwise nonlinearities because LRHE requires strict alternation between these components.Convolutional and fully connected layers are already affine, whereas max-pooling is the non-affine component requiring special treatment.
Max-pooling layers in feedforward form: The rewritten pooling construction is exact, preserves the original CNN function without retraining, and adds one basis function for each introduced nonlinear layer.Two 2 × 2 pooling layers add four basis functions in the resulting feedforward form.

5. LRHE basis results

The LRHE basis identifies input-space directions associated with adversarial sensitivity, while projection onto their orthogonal complement substantially reduces adversarial success.

5.1. LRHE basis attack.: The attack perturbed clean-model inputs along LRHE basis functions using step-sizes α from 0.5 through 3.0 with relative budget ε = 0.1.Inputs were not clipped to [0, 1] so perturbation scaling with α was preserved.
5.1. LRHE basis attack.: 7.79% LRHE basis-attack success exceeded random-vector trials but remained below AutoAttack’s 28.01%.Random-vector trials had median success 1.88%, with a maximum of 1.96% and minimum of 1.81%.
5.2. Projection to the orthogonal complement of LRHE basis.: The projection procedure forms an orthonormal LRHE basis by QR-factorizing the matrix whose columns are the input-basis vectors.Projected adversarial examples are then tested for whether their predicted labels remain adversarial.
5.2. Projection to the orthogonal complement of LRHE basis.: Figure 5 displays six LRHE basis examples across step-sizes α, including original images, perturbed inputs, perturbations, logits, and logit changes.Figure 6 presents original, adversarial, and projected examples with their differences and output-logit changes.

6.1. Training heuristics.

The regularized training objective requires additional tuning, while the selected training configuration preserves clean-model test accuracy and supports comparison under a bounded ℓ∞ threat model.

6.1. Training heuristics.: The regularized objective required more finetuning than the standard misfit-only objective.Using fixed standard choices for λ and γ improved adversarial robustness but reduced test accuracy by 1–2%.
6.1. Training heuristics.: γ was sampled independently at each iteration from a uniform distribution on [−1, 1], while λ was adjusted from λ_base = 1e-2.γ is a dimensionless scale on ϕℓ and is distinct from the perturbation budget ε.
6.1. Training heuristics.: The λ adjustment was performed before the backward-pass computation so the two objective terms remained at roughly the same scale.
6.1. Training heuristics.: 99.53% test accuracy was achieved after selecting the best model across 25 random initializations, matching the original model.Training used Adam for up to 70 epochs with a learning-rate factor of 0.9 every two epochs.
6.1. Training heuristics.: The bounded-ℓ∞ comparison used ε = 0.05 and AutoAttack, which was stronger than the reported PGD configuration.The PGD comparison used 25 runs each of 500 iterations with step-sizes 0.01 and 0.005.

6.2. Adversarial robustness results.

On MNIST, LRHE-based training preserves clean accuracy and matches PGD adversarial training at small relative ℓ2 budgets, while becoming weaker at larger budgets and under ℓ8 attacks.

Clean accuracy: The LRHE-regularized model achieved 99.53% clean test accuracy, comparable to the 99.48% unregularized baseline and PGD-trained models.All four models exceeded 99.4% clean accuracy, so differences were considered inconclusive.
Small-budget robustness: At relative ℓ2 budgets ε ≤ 0.020, LRHE training was on par with 3-step and 40-step PGD training.At ε = 0.010, accuracies were 99.27%, 99.20%, and 99.19% for LRHE, 40-step, and 3-step training, respectively; at ε = 0.020, they were 98.97%, 99.02%, and 98.96%.
Larger budgets: Between ε = 0.012 and ε = 0.014, PGD-40-AT moved ahead, while PGD-3-AT moved ahead between ε = 0.020 and ε = 0.025.At ε = 0.150, PGD adversarial training exceeded the regularized model by 15.70 percentage points under relative ℓ2 perturbations.
Cross-norm robustness: Under ℓ8 perturbations, multi-step training was ahead at every tabulated budget, with a 35.19-percentage-point margin at ε = 0.150.The ℓ8 results probe substantially larger effective ℓ2 displacements and are not directly comparable to the relative-ℓ2 rows.
Computational cost: LRHE-regularized training required 12.7 seconds per epoch, equivalent to 2.8 PGD steps and an 8.7× reduction relative to 40-step training.This timing was below the 13.4 seconds predicted for 3-step training and did not exploit an available partial-product reorganization.
Evaluation scope: The evaluation trained defended models at ε = 0.075 and was conducted only on MNIST, without reporting the conventional ε = 0.3 ℓ8 setting.The authors do not claim that the approximation’s accuracy or usable budget range transfers to higher-resolution data.

7. Conclusion

The conclusion argues that LRHE makes input-gradient-free adversarial training possible by extracting sensitive directions from the forward-pass activation pattern. Its benefits are strongest for small relative ℓ2 perturbations, while its locality limits larger-budget robustness and broader empirical claims.

Conclusion: LRHE identifies sensitive directions from the activation pattern and replaces the min-max inner maximization with a penalty on directions recomputed at each step.Training uses no derivative with respect to the input.
Conclusion: In the MNIST setting, the input-basis subspace had dimension seven out of 784 and removing its orthogonal complement eliminated roughly two thirds of the adversarial effect.A random subspace of the same dimension removed almost none of the effect.
Conclusion: The method’s efficiency and limitations share a source: reflection vectors fixed by the clean activation pattern provide only a local description.Performance becomes substantially worse at large budgets when perturbations cross many activation boundaries.
Future directions: The evaluation remains limited to MNIST, and wider studies, replication across random seeds, and datasets with unsaturated clean accuracy are identified as future directions.The authors avoid claiming that approximation accuracy or budget range transfers to higher-resolution data.
Future directions: The construction may extend beyond feedforward networks because it requires separable nonlinear and affine operations at a given input, as in attention layers.The forward-pass attention matrix would play the role of the activation pattern.
Future directions: The input basis is also proposed as a cheap, differentiation-free tool for sensitivity analysis of trained models beyond robustness training.It identifies input directions to which the model responds most strongly when adversarial examples are absent.