Source-linked AI summary
Anchoring Adversarial Trajectories to Data Manifolds: A Bilevel Transfer Optimization Framework
Yaohua Liu, Yifan Guo, Jiaxin Gao
TL;DR
Adversarial transfer can be weakened when ambient gradients drive attack trajectories off the data manifold and toward surrogate-specific artifacts. MABT combines manifold anchoring with distributional bilevel optimization and a Hessian-free solver; it reports improved transferability, including a higher overall average ASR.
Problem
Ambient-gradient attacks can drift away from the data manifold into surrogate-specific regions, weakening transfer to unknown victims.
Source: §1 · p. 2, §3.1 · p. 4Method
MABT anchors adversarial states and learns an initialization by minimizing expected transfer risk under surrogate uncertainty, using a Hessian-free solver.
Source: §3.1 · p. 4, §3.2 · p. 5, §3.3 · p. 5Results
39.28% overall average ASR, up from 22.52%, corresponding to a 74.4% relative gain; results also support transfer across surrogate choices and victim architectures.
Source: §4.1 · pp. 7–8Takeaways & Limitations
MABT provides a unified transfer framework across different surrogate choices and victim architectures.
Source: §4.1 · p. 8The surrogate uncertainty distribution is instantiated as a local Gaussian approximation, not an exact distribution over unknown victims.
Source: §3.2 · p. 5
Abstract
from arXiv · showhide
A key bottleneck in adversarial transfer is a trajectory-level geometric disconnect: ambient gradients often drift away from the intrinsic data manifold, causing surrogate-specific overfitting. To rectify this, we propose Manifold Anchored Bilevel Transfer (MABT), a unified framework that anchors adversarial trajectories to the shared semantic subspace. MABT introduces a relaxed manifold-anchoring operator as a semantic rectifier to suppress off-manifold noise. With this constraint, we cast transfer attack generation as a distributional bilevel optimization problem that learns a geometry-aligned initialization by minimizing expected transfer risk under a surrogate uncertainty distribution. We further develop a Hessian-free solver with linear-time complexity to handle the resulting hierarchy. Experiments demonstrate improved transferability for 10 baseline attackers across 28 attack configurations, diverse victim architectures, and defense mechanisms.
1 Introduction
The introduction frames transfer failure as a geometric mismatch: iterative attacks can drift off the data manifold and overfit surrogate-specific artifacts. MABT addresses this with manifold anchoring and bilevel optimization, targeting stronger transfer across attackers and victim architectures.
1 Introduction
- Prior transfer methods improve separate attack components, including gradient updates, input transformations, and surrogate ensembles, while the introduction argues that transferability's underlying mechanism remains less explicit.The cited approaches also include feature-level objectives and generative priors, often involving heuristic augmentations or model priors.Source: §1 · p. 2
- Off-manifold drift can move adversarial trajectories away from the shared semantic subspace, where surrogate-specific artifacts and divergent decision boundaries weaken transferability.The introduction motivates this account by noting that natural images concentrate near a low-dimensional manifold where model boundaries may partially align.Source: §1 · p. 2
1.1 Our Contributions
- MABT combines a relaxed manifold-anchoring operator with distributional bilevel optimization to learn a geometry-aligned initialization and suppress off-manifold noise.The framework also uses a Hessian-free solver to handle the resulting hierarchy.Source: §1.1 · pp. 2–3
- MABT produces larger victim-side semantic feature shifts than GAA [16]+AWT [6], with relative gains of 99.8% on Transformers and 32.9% on CNNs.The feature shift is measured as 1 − cos(fclean, fadv) over 10 attack iterations.Source: Figure 2 · p. 2
- Across 10 baseline attackers and 28 configurations, evaluation on 12 CNN and Transformer victims reports improved transferability under diverse attack and defense settings.The contribution summary gives the evaluation scope but does not quantify the overall improvement.Source: §1.1 · p. 3
2 Related Work
Related work improves transferability through attack refinements, surrogate diversity, learned generators, and manifold priors. The paper distinguishes MABT by explicitly guiding trajectory geometry and learning an initialization through distributional bilevel optimization.
Transfer-based Adversarial Attacks
- Transfer attacks refine gradients and inputs, design feature-level objectives, modify surrogate architectures, or aggregate gradients across models.Examples include momentum [11] [53], input transformations [58] [12] [32], intermediate features [17] [23], and multi-model gradients [39] [27].Source: §2 · p. 3
- Ensembles, generative approaches, and meta-learning seek transferable perturbation priors or generators, but do not explicitly model the adversarial trajectory's geometric evolution.The paper contrasts these strategies with its focus on aligning perturbation evolution to the data manifold and learning transferable geometric priors.Source: §2 · p. 3
Manifold Perspectives in Deep Learning
- Prior manifold-based security work uses geometry for purification, anomaly detection, theoretical explanation, or realistic example generation; MABT instead uses anchoring to guide attack trajectories.The cited examples include purification, detection [33], theoretical analysis, and latent-manifold attacks.Source: §2 · p. 3
Bilevel Optimization and Applications
- Earlier bilevel transfer work [40] uses auxiliary victim models and high-order gradient estimation, whereas MABT minimizes expected transfer risk without extra model access and uses a Hessian-free solver.The paper positions this design as an efficient way to learn geometry-aligned priors.Source: §2 · p. 4
3 Methodology
The methodology first identifies ambient-gradient drift as a source of geometric misalignment, then anchors attack states and learns an initialization under surrogate uncertainty. A regularized-gap approach supports a Hessian-free solution strategy.
3.1 Perspective
- Standard projected gradient attacks optimize in an ambient ℓp ball, and their perturbations can drift off the data manifold into surrogate-specific regions that transfer poorly.MAP is a generalized operator that suppresses off-manifold components while preserving the perturbation budget; practical instances approximate the anchoring effect and can trade computational cost against anchoring strength.Source: §3.1 · pp. 4–5
3.2 Distributional Bilevel Optimization
- Distributional bilevel optimization learns an initialization whose induced perturbation minimizes expected transfer risk over surrogate uncertainty.The inner problem evaluates loss on an anchored state, while the outer objective uses a local Gaussian approximation around a fine-tuned surrogate checkpoint—not an exact distribution over unknown victims.Source: §3.2 · p. 5
3.3 Hessian-Free Solution Strategy
- A regularized gap function encodes lower-level optimality as a differentiable penalty, allowing a value-function approach that avoids differentiating through inner dynamics.The method approximates the hypergradient through one-step probing; the stated analysis guarantees projected first-order stationarity rather than global optimality, and notes practical MAP instances may be non-smooth.Source: §3.3 · pp. 5–6
4 Experiments
Experiments evaluate MABT across surrogate and victim architectures, attack configurations, and defenses. Results report improved transferability, while ablations examine trajectory behavior, component contributions, MAP choices, and hyperparameters.
4.1 Experimental Results
- 39.28% versus 22.52% average ASR, a 74.4% relative gain, with particularly clear gains on heterogeneous ViTs and robust ensembles.The VGG-19 surrogate results also support effectiveness under a structurally distinct surrogate.Source: Table 1 · p. 7, §4.1 · pp. 7–8
- 49.33% versus BETAK’s 33.26% average ASR on GAA, a 48.32% relative improvement; MABT requires no auxiliary victim model.The CNN victim-suite mean includes Inc-v3 for protocol consistency.Source: Table 2 · p. 7, §4.1 · p. 8
- Across architectures, MABT preserves higher victim-side loss basins with flatter curvature, whereas baseline landscapes collapse on unknown victims.Figure 3 plots surfaces along the gradient direction and a random orthogonal direction; the pattern suggests reduced surrogate-specific overfitting.Source: Figure 3 · p. 8, §4.1 · p. 8
- MABT improves aggregate ASR against five representative defenses, with the reported evaluation using a ResNet-50 surrogate.The results suggest manifold-anchored perturbations remain more transferable under diverse defenses.Source: Table 3 · p. 8, §4.1 · p. 8
- MABT consistently outperforms baselines in targeted attacks across diverse settings.The reported results suggest geometry-aware optimization can also benefit targeted transfer.Source: §4.1 · p. 9
4.2 Ablation Results
- Increasing T keeps MABT’s surrogate trajectory controlled while victim-side loss rises; even T = 0 outperforms PGD, and T = 3 offers a strong trade-off.PGD improves surrogate loss rapidly but saturates early on the Inc-v3 victim.Source: §4.2 · p. 9, Figure 4 · p. 9
- Resize yields nearly 2× PGD’s ASR with negligible overhead, while stronger generative priors add about 4–5 ASR points at higher latency.MABT is stable across broad β and λ ranges, but T controls warm-up strength; too-small T under-aligns initialization, while too-large T may overfit the surrogate.Source: §4.2 · pp. 9–10, Figure 4 · p. 9
- MAP and DBO each improve the PGD and PGD+AWT baselines, and their combination performs best; together they improve surrogate–victim feature-shift alignment.PGD shows weaker correlation and larger variance; MAP improves correlation, while DBO shifts initialization toward more transferable regions.Source: Table 4 · p. 9, §4.2 · p. 9, Figure 5 · p. 9
5 Conclusion
MABT combines manifold anchoring with distributional bilevel optimization to address geometric misalignment in adversarial transfer. The evaluation focuses mainly on image classification; dense prediction remains future work.
5 Conclusion
- MABT combines manifold anchoring with distributional bilevel optimization, with a Hessian-free solver, to improve transfer across heterogeneous architectures.The framework is also presented as offering useful insights for black-box transfer robustness evaluation.Source: §5 · p. 10
- The study’s evaluation mainly focuses on image classification.Extending MABT to dense prediction tasks remains future work.Source: §5 · p. 10
- The study may provide useful insights for more reliable robustness evaluation under black-box transfer settings.Source: §5 · p. 10
A Appendix
The experiments ran on an NVIDIA H100 GPU with 80GB memory, and the core implementation is provided in the Supplementary Material.
A Appendix
- Experiments ran on a server equipped with an NVIDIA H100 GPU with 80GB memory.Source: A Appendix · p. 14
- The core implementation is provided in the Supplementary Material to facilitate reproducibility.Source: A Appendix · p. 14
- The full code repository will be made public upon acceptance.Source: A Appendix · p. 14
A.1 Section Overview
The appendix summarizes practical method details, runtime and ablation analyses, extended comparisons, and solver convergence analysis.
A.1 Section Overview
- Appendix A.2 covers the local Gaussian surrogate distribution, finite-sample estimation of expected transfer risk, and practical MAP realizations.It also describes their backward handling.Source: §A.1 · p. 14
- Appendix A.3 reports runtime-fair comparisons, MAP-prior and component ablations, and hyperparameter sensitivity analyses.Source: §A.1 · p. 14
- Appendix A.4 provides extended evaluations on additional attackers, VGG-19 surrogate results, targeted attacks, and defense scenarios.Source: §A.1 · p. 14
- Appendix A.5 presents assumptions and a proof for convergence of the gap-regularized solver.Source: §A.1 · p. 14
A.2 Method Details
The appendix instantiates surrogate uncertainty with sampled variants around a fine-tuned checkpoint and estimates expected transfer risk by finite averaging. MAP implementations share a forward interface but use different anchoring priors and optimization handling.
A.2 Method Details
- The surrogate distribution is approximated by Gaussian parameter perturbations around a fine-tuned checkpoint, with sampled variants fixed during attack optimization.Expected transfer risk is computed by finite-sample averaging.Source: §A.2 · p. 14
- N = 3 surrogate variants are used for efficient risk estimation while capturing local uncertainty, rather than modeling the exact distribution of unknown victims.The surrogate is fine-tuned following MBA [27] for 10 epochs with batch size 1024 and SGD at learning rate 0.001, momentum 0.9, and weight decay 5e-4.Source: §A.2 · p. 14
- All MAP implementations use the same forward anchoring interface, ˜x = PM[x + δ], with differences in how PM is instantiated.Source: §A.2 · p. 14
- Anchoring priors include Resize, Bit-Depth, and FFT filtering, plus DnCNN residual denoising and diffusion-based purification.Resize uses bilinear downsampling and upsampling at scale 0.9; diffusion operators include SD-v1.5, Tiny-SD, and OFA-S, and may be more computationally expensive.Source: §A.2 · pp. 14–15
- Differentiable priors use native autograd, while non-smooth or generative priors use surrogate-gradient approximations; the theory assumes smooth composite settings.Source: §A.2 · p. 15
A.3 Ablation Results
The ablation results are presented in two tables: one examines MAP components with a PGD attacker, and the other examines individual DBO and MAP components.
A.3 Ablation Results
- Table 5 reports an ablation study of different MAP components using a PGD attacker.The supplied caption does not state the table's numerical results.Source: Table 5 · p. 15
- Table 6 reports an ablation study of individual DBO and MAP components.The supplied caption does not state the table's numerical results.Source: Table 6 · p. 15
A.3.1 Ablation of Different MAP Implementations
MABT supports multiple MAP realizations, with measured trade-offs between transferability and computational cost; Resize is selected as the default for its efficiency–performance balance.
A.3.1 Ablation of Different MAP Implementations
- MABT improves transferability across seven practical MAP realizations spanning three categories, with distinct efficiency–performance trade-offs.The comparison is visualized in the Figure 4 bubble chart and reported in Table 5.Source: §A.3.1 · p. 15
- 32.74% and 33.06% average ASR are reported for SD-v1.5 and Tiny-SD, respectively, but both incur substantially higher latency and memory consumption.These are the highest ASR results among the generative priors discussed.Source: §A.3.1 · p. 15
- OFA-S reaches 25.40% average ASR and does not outperform the simpler Resize prior.This compact diffusion variant illustrates that generative priors do not uniformly outperform simpler choices.Source: §A.3.1 · p. 15
- 28.59% average ASR with negligible overhead makes Resize the tested variant with the most favorable efficiency–performance balance, and it is the default MAP realization.Resize is competitive, though not the strongest MAP realization in ASR.Source: §A.3.1 · p. 16
A.3.2 Effectiveness of MAP and DBO
The component ablation reports the strongest performance from jointly optimizing MAP and DBO; additional results describe hyperparameter stability, transfer across settings, and conditions used in the solver analysis.
A.3.2 Effectiveness of MAP and DBO
- Jointly optimizing MAP and DBO achieves the best performance, significantly outperforming either component individually.MAP suppresses surrogate-specific off-manifold noise, while DBO learns a geometry-aligned initialization.Source: §A.3.2 · p. 16
A.3.3 Hyperparameter Analysis
- MABT is stable to β and λ over broad ranges, while small or overly large T can respectively under-align or overfit the initialization.The consistent trends on ResNet-50 and VGG-19 support T = 3, β = 1.0, and λ = 0.01.Source: §A.3.3 · p. 16
A.4 Comparative Results
- MABT maintains strong transferability on heterogeneous ViTs and robust ensembles, and its gains are also reported with a VGG-19 surrogate.With GAA+MBA, average ASR is 57.21%, compared with 32.83% for the original baseline.Source: §A.4 · p. 17, Table 8 · p. 18
- For targeted attacks using a ResNet-50 surrogate, MABT consistently outperforms baselines across diverse settings, often achieving multi-fold improvements in average ASR.Table 9 evaluates targeted ASR against 12 victim architectures.Source: §A.4 · p. 17, Table 9 · p. 19
A.5 Algorithm Analysis
- The solver analysis assumes a non-empty, closed, convex, compact feasible set and smoothness conditions for the reduced transfer risk and regularized gap term.These are local conditions for analyzing stationarity, not global convexity assumptions on the practical adversarial loss.Source: §A.5.1 · p. 18
A.5.1 Assumptions
- The gradient estimator is assumed stochastic, with its variance tied to an effective sample size that is non-decreasing and tends to infinity.The analysis also introduces systematic bias from one-step probing and first-order hypergradient approximation.Source: §A.5.1 · pp. 18–19