PaperlayerView exact revision on arXiv

Source-linked AI summary

Neither Adversarial Training Nor Purification: Emergent Adversarial Robustness from Oscillatory Predictive Learning

Mohammed-Yassine Habibi, Klea Ziu, Martin Takáč, Makoto Yamada

arXiv:2609.08683v1cs.LGcs.AI

TL;DR

Adversarial robustness is commonly obtained through costly adversarial training or test-time purification, motivating a search for robustness from architectural and representation-learning biases. OPL combines AKOrN oscillatory dynamics with X-PhiNet predictive pretraining and achieves competitive AutoAttack-rand robustness on CIFAR-10 and CIFAR-100 without adversarial training.

  • Problem

    The paper asks whether adversarial robustness can emerge from architectural and representation-learning inductive biases instead of repeated adversarial-example generation.

  • Method

    OPL combines an AKOrN oscillatory encoder with X-PhiNet predictive self-supervised pretraining and evaluates randomized models with AutoAttack-rand and EoT.

  • Results

    76.63 ± 0.76% and 50.44% robust accuracy are achieved on CIFAR-10 and CIFAR-100, respectively, under ℓ∞, ϵ = 8/255, AutoAttack-rand with EoT K = 20.

  • Takeaways & Limitations

    The findings support a complementary route to robustness through structured encoder dynamics and predictive learning without adversarial training or purification.

  • Takeaways & Limitations

    The current setup associates robustness with lower clean accuracy than the highest-capacity adversarially trained baselines, and the efficiency comparison is not protocol-matched.

Abstract

from arXiv · show

Adversarial robustness in computer vision is still largely achieved through adversarial training or test-time adversarial purification, both of which introduce significant computational overhead by generating adversarial examples during training or performing iterative denoising at test time. We study whether empirical robustness can instead emerge from architectural and representation-learning inductive biases. We introduce Oscillatory Predictive Learning (OPL), a two-stage framework that combines Artificial Kuramoto Oscillatory Neurons (AKOrN) with predictive self-supervised pretraining using X-PhiNet. Because our default checkpoint uses randomized initial oscillator states, we compare it with other randomized adversarial defense methods that provide precise, reproducible, and strong attack protocols. Experiments on CIFAR-10 and CIFAR-100, with additional corruption evaluation on CIFAR-10-C, demonstrate that our method achieves competitive results under the AutoAttack-rand evaluation protocol. On CIFAR-10 and CIFAR-100, OPL attains 76.63$\pm$0.76$\%$ and 50.44$\%$ robust accuracy, respectively, under $\ell_\infty$, $ε=8/255$, AutoAttack-rand with EoT $K=20$.

1 Introduction

The paper asks whether robustness can emerge from architectural and representation-learning inductive biases rather than adversarial-example generation. It introduces OPL, combining AKOrN dynamics with X-PhiNet, and reports strong randomized-attack robustness alongside sensitivity to oscillator dimension and evaluation limits.

  • Framework: OPL combines an AKOrN encoder with X-PhiNet predictive self-supervised learning to study robustness without adversarial training.The framework evaluates the interaction between oscillatory dynamics and predictive learning under standardized attacks.
  • Results: 76.63 ± 0.76% robust accuracy is achieved on CIFAR-10 under ℓ∞, ϵ = 8/255, AutoAttack-rand with EoT K = 20.This exceeds DiffPure at 71.29% under the same AutoAttack-rand protocol.
  • Results: 50.44% AutoAttack-rand robustness is reported on CIFAR-100 without adversarial training or adversarial purification.The contribution list reports these results as an alternative to adversarial optimization.
  • Efficiency and limitations: OPL operates in a substantially lower measured training-FLOP regime than representative adversarially trained CIFAR models, but the comparison is contextual rather than protocol-matched.The systems differ in architecture, data, training procedure, and evaluation protocol.
  • Ablations: +11.65 percentage points of robustness amplification occurs when X-PhiNet is applied to AKOrN, increasing robustness from 64.98% to 76.63%.Applying X-PhiNet to a standard ResNet yields near-zero robustness, supporting complementary roles for the two components.
  • Dynamical sensitivity: Robustness collapses near-discontinuously when oscillator dimensionality increases from N = 2 to N = 4, while clean and robust accuracy are not monotonically aligned.The reported sweep concentrates robustness in the N = 2 family despite improved clean accuracy at N = 4.

2 Background and Motivation

The background motivates alternatives to costly adversarial training and purification by connecting robustness to dynamical stability, stochastic defenses, and self-supervised representation learning. OPL studies their interaction rather than treating oscillatory architecture or predictive learning in isolation.

  • Motivation: Adversarial training and purification remain dominant robustness approaches, but they require costly adversarial-example generation or iterative test-time denoising.The paper frames this computational burden as motivation for alternative inductive biases.
  • Dynamical models: Dynamical robustness work links stability to similar trajectories, while AKOrN uses coupled Kuramoto-style oscillator updates.OPL treats oscillatory dynamics as a robustness-supporting inductive bias.
  • Stochastic defenses: Stochastic defenses include randomized smoothing and purification methods that use randomness or iterative denoising before classification.These approaches form part of the broader robustness context in which OPL is evaluated.
  • Self-supervised learning: Predictive self-supervised learning has been studied for stable representations and robustness, including methods such as SimSiam and BYOL.Prior work often uses self-supervised learning as pretraining for adversarial fine-tuning.
  • Iterative inference: Iterative-inference defenses resist perturbations by driving activations toward stable attractors, a philosophy OPL extends through Kuramoto coupling.The paper places OPL alongside deep equilibrium and recurrent reasoning mechanisms.
  • Positioning: OPL studies robustness as an interaction between oscillatory architecture and predictive objective, unlike approaches that consider either direction in isolation.The encoder supplies the oscillatory inductive bias and the SSL framework supplies the predictive learning signal.

3 Preliminaries: Oscillatory Encoders and Predictive Pretraining

The preliminaries describe AKOrN as a coupled oscillatory encoder and X-PhiNet as a predictive self-supervised objective, motivating their complementary roles in robustness-oriented representation learning.

  • Oscillatory Encoders: AKOrN reshapes convolutional features into K oscillator states of dimension N at each spatial location.Here, K = C/N and each oscillator state lies on the unit sphere S^(N−1).
  • Oscillatory Encoders: A Kuramoto layer updates each oscillator using intrinsic rotation, learned coupling, and an input-dependent drive.The dynamics are implemented with convolutional connectivity and unrolled for T discrete steps within each oscillatory block.
  • Oscillatory Encoders: Coupling encourages locally consistent feature configurations by combining each oscillator’s state with neighboring oscillator states.The paper motivates this interaction as preventing localized perturbations from being amplified independently.
  • Predictive Pretraining: X-PhiNet learns representations by predicting one transformed view from another rather than contrasting samples against large negative sets.Its predictive objective is motivated by temporal prediction and is intended to encourage stable latent structure.
  • Predictive Pretraining: X-PhiNet compares outputs from parallel streams at different depths and jointly optimizes symmetric negative cosine and mean squared-error losses.The resulting long-stream encoder is extracted after pretraining.

4 Oscillatory Predictive Learning

Oscillatory Predictive Learning combines AKOrN with X-PhiNet in a two-stage pipeline: predictive pretraining followed by supervised classification fine-tuning, with stochastic initialization evaluated explicitly.

  • 4 Oscillatory Predictive Learning: OPL first pretrains an AKOrN encoder with X-PhiNet and then fine-tunes it for supervised image classification.The pretraining heads are replaced by a classification head in the second stage.
  • Architecture: The encoder uses a convolutional stem followed by L oscillatory blocks, each unrolled for T dynamics steps, before global pooling.The pooled representation feeds either X-PhiNet pretraining heads or a supervised classification head.
  • Architecture: OPL evaluates both random and deterministic initial oscillator states, using AutoAttack-rand with EoT for stochastic models.For deterministic models, the fixed forward pass makes EoT ineffective.
  • Architecture: A single oscillatory unit maps an initial oscillator state and conditional stimulus to a final oscillator state and extracted features.The inputs and outputs are represented over oscillator, spatial, and feature dimensions.
  • Notation: The configuration (C, N, T, L) denotes channel width, oscillator dimension, unrolled steps per block, and number of oscillatory units.The paper reports a default configuration separately.

5 Experiments

The experiments evaluate randomized OPL primarily with adaptive AutoAttack-rand and complementary attack, transfer, and corruption diagnostics. OPL achieves strong robustness without adversarial training, with evidence that oscillatory dynamics and predictive pretraining contribute differently.

  • Evaluation protocols: The evaluation uses AutoAttack-rand as the primary protocol for randomized OPL, with PGD, Square Attack, transfer attacks, and Gaussian-noise robustness as complementary diagnostics.All ℓ∞ evaluations use ε = 8/255, while randomized checkpoints use EoT to account for inference randomness.
  • Primary evaluation: OPL reaches 76.63% ± 0.76% robust accuracy on CIFAR-10 and 50.44% on CIFAR-100 under AutoAttack-rand with EoT K = 20, without adversarial training.Published comparisons are contextual because methods differ in architecture, test-time computation, and attack implementation.
  • Adaptive diagnostics: Under ℓ∞ PGD-100, OPL reaches 76.10% robust accuracy; under ℓ2 PGD-100, it reaches 82.11%.Robust accuracy decreases monotonically as attack strength increases.
  • Adaptive diagnostics: Square Attack yields 81.81% robust accuracy from 86.81% clean accuracy, exceeding AutoAttack-rand's 76.63% in the reported comparison.The ordering is presented as consistent with informative gradients rather than gradient masking.
  • Adaptive diagnostics: Transfer attacks find lower robustness than AutoAttack-rand for OPL, so AutoAttack-rand is not treated as a worst-case lower bound across all attacks.Transfer attacks use PGD-100 examples generated on a ResNet-50 surrogate.
  • Ablations: AKOrN without predictive pretraining reaches 64.98% robustness, whereas X-PhiNet on ResNet-50 reaches 0.31%; the full combination reaches 76.63%.The ablation indicates that oscillatory dynamics provide the primary robustness-supporting component within the tested model family, while predictive pretraining strengthens it.
  • Corruption robustness: OPL maintains the highest accuracy across the evaluated Gaussian-noise range, while AKOrN alone is more stable than X-PhiNet-only and ResNet-50 controls.Each Figure 3 point averages three independently sampled noise draws over the full CIFAR-10 test set.

6 Discussion, Trade-offs, and Limitations

The discussion identifies a narrow dynamical regime associated with robustness and emphasizes trade-offs and unresolved scope. OPL's reported robustness is not a matched-capacity comparison to adversarially trained models and remains limited in dataset coverage and mechanistic understanding.

  • Future directions: The proposed hypothesis that related low-dimensional couplings may show similar robustness phase transitions remains future work in other dynamical or iterative models.The paper frames this transfer beyond the studied setting as a conjecture requiring testing.
  • Trade-offs: OPL reaches 86.81% clean accuracy and 76.63% AA-rand robustness on CIFAR-10, but this is not a matched-capacity or matched-compute comparison with adversarial training.The paper associates the robustness with lower clean accuracy than the highest-capacity adversarially trained baselines.
  • Limitations: The experiments are limited to CIFAR-10, CIFAR-10-C, and CIFAR-100, excluding higher-resolution datasets such as ImageNet.Extending evaluation would require oscillatory-neuron implementations in architectures effective for those settings.
  • Limitations: The paper does not yet provide a mechanistic theory or diagnostic that predicts when a configuration will be robust.The N=2 versus N=4 contrast suggests, but does not establish, the relevant mechanism.

7 Broader Impacts

The paper presents robustness without adversarial training as potentially reducing deployment compute, while noting that easier deployment could also enable misuse in surveillance, filtering, or adversarial settings.

  • Potential benefits: Robustness without adversarial training could reduce deployment cost and improve access to robust models in low-resource or on-device settings.The paper gives medical imaging and edge AI as examples.
  • Potential risks: Lower deployment barriers could also facilitate misuse of robust models in surveillance, filtering, or adversarial settings.The paper therefore frames robustness improvements as having both beneficial and potentially harmful applications.

A Mathematical Derivations: Full Kuramoto ODE, projection operators, X-PhiNet loss functions.

AKOrN Kuramoto layers iteratively update oscillator states using learned rotation, coupling, and input-drive terms, then apply phase-invariant readouts to extract features from the final state.

  • Kuramoto layer: A Kuramoto layer receives an initial oscillator state X and conditional stimulus Y, with K oscillator states at each spatial location.The channel count satisfies C = KN.
  • Kuramoto layer: Each layer applies T discrete Kuramoto updates to the initial state X(0) := X.The output is the final oscillator state X(T).
  • Kuramoto dynamics: The dynamics use learned antisymmetric matrices for intrinsic rotation, learned coupling weights, and an input-dependent drive term.In the convolutional implementation, coupling weights are convolution-kernel connectivity weights.
  • Projection and geometry: A projection operator constrains oscillator velocity to the tangent plane of the unit hypersphere at the current oscillator state.The projector keeps oscillator states on the unit hypersphere.
  • Feature extraction: The phase-invariant readout extracts features from oscillatory states while preserving information encoded in relative oscillator directions.A common phase rotation does not change the readout output.
  • Network construction: Multiple Kuramoto layers are stacked with readout modules between successive layers to create new conditional stimuli.The readout module can include a linear map and learned weight matrices.

A.4 X-PhiNet Predictive learning setup

X-PhiNet combines two self-supervised losses for encoder pretraining, while the surrounding pipeline uses AutoAttack and Auto-PGD to define and optimize adversarial perturbations under constrained budgets.

  • X-PhiNet pretraining: X-PhiNet jointly minimizes Sim-1 and Sim-2 losses, using both to update the encoder by backpropagation.Sim-1 is associated with the hippocampus model and Sim-2 with the neocortex model.
  • X-PhiNet losses: Sim-1 is a symmetric negative cosine loss that minimizes when a predicted representation aligns with a temporally distant signal.The loss is described as analogous to the SimSiam objective.
  • X-PhiNet losses: Sim-2 is an MSE loss between the CA1 output and the entorhinal-cortex representation, and the final X-PhiNet loss sums Sim-1 and Sim-2.Sim-2 is characterized as a slow-learning loss.
  • Stable encoder: A stable encoder uses exponential moving-average updates to maintain more stable parameters and mitigate representation collapse during training.The stable encoder is part of X-PhiNet’s third stream.
  • Adversarial attacks: An adversarial example lies within a distance-bounded feasible set around a correctly classified input and changes the classifier’s predicted class.Common attack distances include l_2 and l_∞ norms.
  • Auto-PGD: Auto-PGD solves the constrained attack objective iteratively using projected updates, adaptive step sizes, momentum, and checkpoint-based restarts.Its checkpoints separate exploration from exploitation and can halve the step size when progress is insufficient or stagnates.
  • Auto-PGD: Auto-PGD checkpoints support exploration early and exploitation later, with increasingly dense checkpoints and restarts from the best point when the step size is halved.The checkpoint schedule is defined recursively from p0 = 0 and p1 = 0.22.

C Experimental Settings and Reproducibility Details

The experiments use a two-stage self-supervised pretraining and supervised fine-tuning pipeline, with X-PhiNet on an AKOrN backbone and stochastic defenses evaluated using AutoAttack-rand with EoT.

  • Training pipeline: The training pipeline first performs self-supervised pretraining and then fine-tunes the pretrained backbone for image classification.Fine-tuning is initialized from the pretrained checkpoint.
  • Experimental setup: The main configuration uses CIFAR-10, X-PhiNet pretraining, and an AKOrN backbone with separate pretraining and fine-tuning stages.The reported configuration is documented for reproducibility.
  • Robustness evaluation: Evaluation uses AutoAttack with ε = 8/255 under the l∞ norm on CIFAR-10 and CIFAR-100.The implementation follows the RobustBench codebase and default evaluation hyperparameters.
  • Robustness evaluation: Because the method is stochastic, experiments use AutoAttack-rand rather than the standard version, with EoT = 20 by default.AutoAttack-rand uses APGD-CE and APGD-DLR.
  • Optimization: Pretraining uses Adam for 700 epochs with batch size 512, learning rate 1 × 10^-4, and weight decay 1 × 10^-5, while fine-tuning runs for 400 epochs.The main CIFAR-10 result averages five independently trained runs.
  • Backbone configuration: The AKOrN backbone uses 128 channels, oscillator dimensionality n = 2, T = 3 recurrent steps, and L = 3 layers.Its interaction operator is convolutional with kernel sizes [9] [7] [5].

D Further Experimental Results

Further experiments examine compute, corruption robustness, pretraining duration, randomness, oscillator dimensionality, and X-PhiNet loss weighting, revealing both gains and sensitivity to configuration choices.

  • Predictive learning frameworks: X-PhiNet is selected after preliminary comparisons of predictive learning frameworks on the same AKOrN backbone.The related experimental table is identified as Table 8.
  • Training compute: 358 times lower overall training cost is reported for OPL than current state-of-the-art adversarially trained models.The comparison is not directly matched because the reference models are evaluated under standard AutoAttack.
  • CIFAR-10-C robustness: 82.61% mean CIFAR-10-C corruption accuracy improves over AKOrN’s 80.69%, a gain of +1.92 percentage points.The improvement is reported across all 15 corruption types.
  • Randomness and dimensionality: Increasing oscillator dimensionality from 2 to 4 with randomness enabled caused a severe robust-accuracy collapse despite strong clean accuracy.Disabling randomness with N = 2 produced a stronger clean/robust tradeoff than the corresponding randomized setting with orientation loss disabled.
  • X-PhiNet loss weights: Moderate MSE weighting and a nonzero orientation term generally improved the overall robustness tradeoff, but precise loss weighting remained consequential.The experiments motivate careful balancing of representation alignment and orientation regularization.
  • Pretraining duration: Increasing pretraining from 400 to 700 epochs improved robust accuracy while preserving strong clean performance, though gains were not strictly monotonic.The result comes from a pretraining-duration sweep.
  • EMA and weight decay: Small pretraining weight decay often stabilized training, while the strongest runs continued to use EMA coefficients near 1.0.These observations motivated the default setting.

D.1 Compute resources

The main experiments used one NVIDIA RTX A6000 GPU, while reported FLOP estimates exclude several overhead categories and do not provide complete wall-clock accounting for exploratory runs.

  • All reported experiments ran on a single NVIDIA RTX A6000 GPU with 48GB of memory.
  • Training FLOP estimates include reported training stages but exclude logging, checkpoint I/O, preliminary failed runs, and adversarial evaluation cost.
  • The study did not systematically record wall-clock time for every exploratory run.
Loading 2609.08683v1…