PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual-Invariance-Augmented Feature Optimal Alignment for Transferable Adversarial Attacks against Closed-Source MLLMs

Xiaojun Jia, Simeng Qin, Yiming Li, Jie Liao, Sensen Gao, Ke Ma, Yang Liu, Xiaochun Cao

arXiv:2610.06977v1cs.CVcs.AI

TL;DR

Targeted transfer attacks against closed-source MLLMs must convey target semantics across models whose internal representations are inaccessible. IAU-FOA combines global and clustered-local alignment with confidence-adaptive transport and photometric augmentation; experiments report that it consistently outperforms state-of-the-art transferable attack methods on open- and closed-source MLLMs.

  • Problem

    Existing targeted-transfer attacks mainly align global image features, leaving patch-level structures and reliable transfer across heterogeneous closed-source MLLMs insufficiently addressed.

    Source: §I · pp. 1–2

  • Method

    IAU-FOA combines cosine-based global alignment, clustered-patch optimal transport with confidence-adaptive marginals, and photometric augmentation during surrogate-based optimization.

    Source: §I · pp. 2–3, B. Overview of the Proposed Framework · p. 5, C. Visual-Invariance Augmentation · p. 5

  • Results

    Experiments on open-source and closed-source MLLMs show that IAU-FOA consistently outperforms state-of-the-art transferable attack methods.

    Source: §I · p. 3, E. Ablation Study · p. 12

  • Takeaways & Limitations

    Targeted adversarial examples can preserve source-image appearance while steering commercial MLLMs toward target-related descriptions.

    Source: §I · p. 1

Abstract

from arXiv · show

Multimodal large language models (MLLMs) remain vulnerable to transferable adversarial examples, especially in black-box settings where only open-source surrogate models are accessible. Existing targeted transfer attacks mainly align adversarial and target samples using global image-level features, such as encoder [CLS] embeddings. However, such coarse alignment insufficiently exploits patch-level visual structures, limiting transferability across heterogeneous closed-source MLLMs. We propose IAU-FOA, a visual-invariance-augmented feature optimal alignment attack with adaptive unbalanced transport, to improve targeted transferability against closed-source MLLMs. IAU-FOA aligns adversarial and target samples at both global and local levels: a cosine-based objective narrows their global semantic gap, while patch tokens are clustered into compact local patterns and matched through optimal transport for fine-grained feature alignment. Balanced optimal transport enforces fixed marginal masses even for local clusters without reliable counterparts, potentially introducing misleading alignment gradients. We therefore introduce confidence-adaptive unbalanced transport to relax these constraints for weakly matched clusters, aiming to reduce unreliable local alignment and improve adversarial transferability. We further study the effect of input transformations and propose visual-invariance augmentation, which applies bidirectional pixel-intensity rescaling and per-channel white-balance adjustment to simulate exposure, contrast, illumination, and color-temperature variations. This strategy encourages adversarial perturbations to generalize across different visual encoders. Extensive experiments on open-source and closed-source MLLMs show that IAU-FOA consistently outperforms state-of-the-art transferable attack methods. Code is available at https://github.com/jiaxiaojunQAQ/IAU-FOA.

I. INTRODUCTION

IAU-FOA targets the limited transferability of black-box attacks against closed-source MLLMs by combining global and local feature alignment with adaptive transport and visual-invariance augmentation. The paper reports that it consistently outperforms state-of-the-art transferable attack methods on open- and closed-source MLLMs.

I. INTRODUCTION

  • IAU-FOA combines cosine-based global semantic guidance with optimal-transport matching of clustered patch tokens to exploit fine-grained visual structures.Clustering is used to reduce redundant and noisy local correspondences.Source: §I · pp. 2–3
  • Confidence-adaptive transport relaxes marginal constraints for weakly matched clusters, while dynamic ensemble weights respond to relative changes in encoder alignment scores.Higher-confidence clusters receive stronger marginal-deviation penalties; for positive alignment scores, larger relative increases receive larger weights.Source: §I · pp. 2–3
  • Visual-invariance augmentation combines bidirectional intensity scaling and channel-wise gain adjustment to approximate exposure and white-balance variation during optimization.The strategy aims to reduce dependence on surrogate-specific visual statistics.Source: §I · pp. 2–3
  • Experiments on open-source and closed-source MLLMs show that IAU-FOA consistently outperforms state-of-the-art transferable attack methods.The introduction identifies the work as an extension of FOA.Source: §I · p. 3

A. Multimodal Large Language Models

  • MLLMs connect visual encoders to language models, while closed-source systems’ differing architectures and visual pipelines make their representations difficult to access or approximate.The paper notes that commercial systems include GPT-5.5 and Claude-Opus-4.8.Source: §I · p. 1, A. Multimodal Large Language Models · p. 3

B. Adversarial Attacks

  • Black-box targeted attacks must convey specific semantics through open-ended responses, so transfer from accessible surrogate models offers a practical route to unseen victims.The paper distinguishes targeted attacks, which seek specified outputs, from untargeted attacks, which seek incorrect outputs.Source: §I · pp. 1–2, B. Adversarial Attacks · p. 3

C. Transfer-Based Targeted Attacks for MLLMs

  • Prior targeted-transfer methods often align global features [45], while fine-grained adaptive alignment and visual-invariance optimization remain insufficiently explored.Related work also describes approaches using diffusion, self-supervision, multimodal updates, visual patterns, and multi-paradigm features.Source: §I · p. 2, C. Transfer-Based Targeted Attacks for… · pp. 3–4

III. METHOD

IAU-FOA optimizes targeted attacks on accessible surrogate encoders for transfer to unseen closed-source MLLMs. It combines global and local feature alignment across transformed adversarial views, with dynamic weighting over encoders.

III. METHOD

  • Multiple surrogate encoders are used to emphasize feature directions shared across models, with weights adapted to relative changes in alignment similarity scores.The framework integrates visual-invariance augmentation, dual-granularity alignment, adaptive unbalanced local transport, and dynamic ensemble weighting.Source: §III · p. 4

A. Problem Formulation

  • IAU-FOA transfers attacks optimized on an ensemble of accessible surrogate visual encoders to unseen closed-source MLLMs by reducing global and local feature-alignment losses.The perturbation is bounded, and the victim model’s parameters and gradients are unavailable.Source: A. Problem Formulation · pp. 4–5

B. Overview of the Proposed Framework

  • The framework averages gradients across photometrically transformed adversarial copies and combines global semantic alignment with transported local patch-pattern alignment.Dynamic encoder weights are updated at each objective evaluation and held fixed during backpropagation.Source: B. Overview of the Proposed Framework · p. 5

C. Visual-Invariance Augmentation

Visual-invariance augmentation uses global intensity scaling and channel-wise gains to model photometric variation. Separately, the global alignment branch uses cosine distance to bring adversarial and target image representations closer in semantic feature space.

C. Visual-Invariance Augmentation

  • Intensity scaling and channel-wise gains model exposure, brightness, illumination color, and white-balance variation in transformed adversarial views.The default intensity scales are {0.25, 0.5, 1, 2, 4}; channel gains range from 0.5 to 1.5 by default.Source: C. Visual-Invariance Augmentation · p. 5
  • Gradients from transformed copies are averaged, encouraging adversarial examples to remain effective under heterogeneous visual preprocessing pipelines.The global intensity scale approximates brightness and exposure shifts, while channel gains approximate white-balance and illumination-color shifts.Source: C. Visual-Invariance Augmentation · p. 5

D. Global Feature Alignment

  • The global alignment loss is cosine distance between normalized adversarial and target global features; minimizing it increases their semantic similarity.The global feature supplies coarse-grained semantic guidance.Source: D. Global Feature Alignment · p. 6

E. Local Clustering Feature Alignment

Local alignment clusters patch-token features into compact patterns and matches cluster centers through optimal transport. Adaptive unbalanced transport relaxes marginal constraints for low-confidence clusters to reduce unreliable matches in optimization.

E. Local Clustering Feature Alignment

  • Patch-token clustering reduces redundant image information while retaining discriminative local patterns for alignment.K-means partitions patch-feature rows into K clusters and returns their centroids.Source: E. Local Clustering Feature Alignment · p. 6
  • The balanced local transport objective matches adversarial and target cluster masses using cosine-derived costs, but fixed marginals may force unreliable correspondences.The transport plan represents mass moved from adversarial clusters to target clusters; the cost is cosine distance between cluster centers.Source: E. Local Clustering Feature Alignment · p. 6

F. Adaptive Unbalanced Optimal Transport

  • Adaptive unbalanced Sinkhorn uses weaker marginal regularization for low-confidence clusters and stronger regularization for high-confidence clusters.Confidence is estimated from the best target counterpart; with γ = 1, high-confidence clusters receive stronger marginal constraints while low-confidence clusters are relaxed further.Source: F. Adaptive Unbalanced Optimal Transport · pp. 6–7
  • The local loss is normalized transported cosine distance, with transport weights held fixed while gradients pass through similarities.The final IAU-FOA loss combines global semantic alignment with adaptive local transport alignment.Source: F. Adaptive Unbalanced Optimal Transport · p. 7

G. Dynamic Ensemble Model Weighting

The dynamic ensemble weighting rule adapts surrogate encoder contributions using changes in alignment similarity scores across successive objective evaluations. The resulting weights sum to the number of encoders and are held at the equal-weight scale.

G. Dynamic Ensemble Model Weighting

  • The ensemble objective uses each encoder’s distance loss, while dynamic weights are computed separately from alignment similarity scores.The similarity scores are evaluated on the current augmented input and its local transport plan.Source: G. Dynamic Ensemble Model Weighting · p. 8
  • Dynamic ensemble weights are derived from ratios of alignment similarity scores across consecutive objective evaluations, which may include different augmented views.For positive scores well above η, a larger ratio receives a larger weight; weights sum to M, matching equal weighting’s scale.Source: G. Dynamic Ensemble Model Weighting · p. 8

H. Transferability Analysis

The transferability analysis compares balanced optimal transport (OT) with uniform local matching and derives a conditional one-step advantage for OT under stated assumptions. The comparison holds other attack components fixed, while the theoretical result depends on semantic and cross-model gradient separation.

H. Transferability Analysis

  • Balanced OT is compared with uniform local matching at a fixed perturbation while the encoder, augmentation view, target features, and cluster assignments are held fixed.The local alignment gradient is defined with the transport plan held fixed during differentiation.Source: H. Transferability Analysis · p. 8
  • The analysis assumes surrogate costs separate a semantic cluster correspondence and that corresponding clusters have cross-model gradient separation, up to bounded deviations.The correspondence permutation is specified independently of the transport solution.Source: H. Transferability Analysis · p. 8
  • Under the stated assumptions and step-size condition, OT yields stronger victim-model target alignment than uniform local matching after one feasible gradient step, with an objective gap of at least τ > 0.The result is conditional on the updates remaining feasible and the specified smoothness and step-size bounds.Source: H. Transferability Analysis · p. 8

A. Experimental Setup

The experiments evaluate targeted transfer attacks on open-source and commercial MLLMs using a shared image-pair protocol and LLM-as-a-judge metrics. IAU-FOA leads the reported main benchmarks, while sensitivity tests select the default hyperparameters.

A. Experimental Setup

  • The evaluation uses 1,000 clean source images and 1,000 MSCOCO target images, five surrogate models, and an LLM-as-a-judge protocol reporting ASR and AvgSim.An attack counts as successful when GPTScore semantic similarity is at least 0.5; AdvDiffVLM is not constrained by the 16/255 ℓ∞ budget used for other baselines.Source: A. Experimental Setup · p. 9

B. Hyper-parameter Selection

C. Comparative Experimental Results

  • IAU-FOA achieves the highest ASR and AvgSim on all six open-source and all six commercial targets in the main comparison.Tables I and II report results on the same 1,000 source–target image pairs per target model.Source: C. Comparative Experimental Results · pp. 9–10, Table I · p. 10, Table II · p. 10
  • 86.3% ASR and 68.5% AvgSim on open-source targets and 75.0% ASR and 60.0% AvgSim on commercial targets exceed MPCAttack averages.The corresponding MPCAttack averages are 78.5% ASR and 62.3% AvgSim on open-source targets, and 58.9% ASR and 49.3% AvgSim on commercial targets.Source: C. Comparative Experimental Results · p. 10
  • 66.7% average ASR and 54.2% average AvgSim on five recent commercial MLLMs exceed MPCAttack by 21.5 and 13.4 percentage points, respectively.IAU-FOA has the highest ASR and AvgSim on all five targets; gains are largest on the GPT-5.6 family.Source: C. Comparative Experimental Results · p. 11

D. Evaluation under Adversarial Defenses

The evaluation reports stronger attack performance under input defenses and across component comparisons. The analyses support visual-invariance augmentation, confidence-adaptive local transport, and dynamic ensemble weighting as useful parts of IAU-FOA.

D. Evaluation under Adversarial Defenses

  • 0.688 mean ASR across six defended settings, versus 0.495 for MPCAttack and 0.428 for M-Attack-v2; IAU-FOA leads in every defended setting.The average improvements are 19.3 and 26.0 percentage points, respectively, and under DISCO its ASR is 0.587 versus 0.382 for MPCAttack and 0.297 for M-Attack-v2.Source: Table IV · p. 11, D. Evaluation under Adversarial Defenses · p. 11

E. Ablation Study

  • 0.855 mean ASR and 0.659 AvgSim are the full method’s highest ablation results; removing VIA causes the largest ASR drop, 18.0 points.Removing local OT lowers mean ASR and AvgSim by 16.3 and 11.6 percentage points; replacing Adaptive UOT with balanced OT lowers them by 4.3 and 2.4 points, while equal ensemble weighting lowers them by 5.5 and 1.3 points.Source: E. Ablation Study · pp. 11–12, Table V · p. 12

F. Analysis of the Proposed Method

  • VIA achieves 0.855 average ASR and 0.6590 AvgSim, exceeding No Aug by 18.0 and 10.80 percentage points and Admix by 3.7 and 2.50 points.The comparison indicates that augmentation strategy, not just the number of views, matters; VIA varies intensity and channel-wise gains while preserving spatial content.Source: F. Analysis of the Proposed Method · p. 12, Figure 5 · p. 12
  • Adaptive-UOT reaches mean ASR 0.860, versus 0.703 for W/O OT, 0.823 for BOT, and 0.820 for UOT, and leads on all three reported targets.Plain UOT matches BOT on two targets and is slightly worse on Cl-O-4.8; Adaptive-UOT’s largest gain is 8.0 percentage points over UOT on Cl-O-4.8.Source: F. Analysis of the Proposed Method · pp. 12–13, Figure 6 · p. 12
  • Dynamic weighting has the highest mean ASR, 0.855, versus 0.800 for Equal, 0.817 for Random, and 0.832 for Score-Proportional Weighting.It ranks first on all six targets, tying on G-3.5-F and Cl-S-4.6; IAU-FOA’s conclusion also identifies dual-granularity alignment, adaptive transport, dynamic weighting, and VIA as its designs.Source: F. Analysis of the Proposed Method · p. 13, Table VI · p. 13, §V · p. 13

S1. PROOFS AND DETAILS OF THE TRANSFERABILITY ANALYSIS

The proof compares a transport-based local alignment update with uniform local matching under stated assumptions. Its step-size condition is used to ensure a positive objective gap.

S1. PROOFS AND DETAILS OF THE TRANSFERABILITY ANALYSIS

  • The derivation uses a feasible plan and entropy bounds to establish the stated intermediate relation for the transport comparison.The supplied proof steps refer to a feasible plan Q† and bounds H(Q†) = log K and H(Qot) ≤ 2 log K.Source: S1. PROOFS AND DETAILS OF THE… · p. 17
  • The condition in Eq. (36) gives ¯q < q0 − 2ξ/(A + B), hence κ > 0.Source: S1. PROOFS AND DETAILS OF THE… · p. 17
  • The positive directional comparison yields an objective gap τ > 0 when the stated step-size condition holds.For L > 0 the condition ensures τ > 0; for L = 0, τ = αλlκ > 0.Source: S1. PROOFS AND DETAILS OF THE… · p. 17

Finite Solver Accuracy

The finite-plan and implemented-update comparisons require additional bounds beyond the transport mass argument. The result is conditional and does not establish unconditional improvement of the full attack.

Finite Solver Accuracy

  • For an inexact plan, optimization error must also fit within the available comparison margin; the scaling-vector stopping threshold is not itself a bound on ζ.Source: Finite Solver Accuracy · p. 17
  • The comparison holds with remaining branch contributions and ensemble weights fixed; it does not compare full trajectories with changing weights or cluster assignments.Source: Relation to the Implemented Update · p. 18
  • A strict advantage for the implemented update requires the displacement-change and curvature terms together to be smaller than αν.The proof explicitly treats this as an additional sufficient condition, not a consequence of the transport mass bound.Source: Relation to the Implemented Update · p. 18

A Constructive Example of Possible Improvement

A constructed cosine-feature example shows that OT can improve a victim objective over uniform matching under a sign update. The supplementary material also reports visual examples, iteration effects, and the optimization procedure.

A Constructive Example of Possible Improvement

  • With α = 0.01, the constructed sign updates yield t_ot = 0.01 and t_0 = −0.01, with Jv(t_ot) > Jv(0) > Jv(t_0).This is an existence example, not evidence that evaluated MLLMs satisfy the assumptions.Source: A Constructive Example of Possible… · p. 18

S2. VISUALIZATION OF ADVERSARIAL EXAMPLES

  • IAU-FOA has the highest PSNR and SSIM in all three displayed examples, averaging 30.9 dB and 0.769 versus COA’s 30.1 dB and 0.702.These comparisons concern the displayed examples; aggregate image quality over the full evaluation set is not assessed.Source: S2. VISUALIZATION OF ADVERSARIAL… · p. 19, Figure S1 · p. 19

S3. EFFECT OF THE NUMBER OF ATTACK ITERATIONS

  • The highest mean ASR, 85.5%, occurs at 300 attack iterations among the evaluated 100–500 iterations.Source: Figure S2 · p. 20

S4. OPTIMIZATION ALGORITHM

  • Algorithm S1 updates a perturbation using source and target images, surrogate encoders, prepared views, alignment losses, and projected sign steps.It specifies a default of 10 local clusters and holds transport plans, confidence modulation, and ensemble weights fixed during backpropagation.Source: S4. OPTIMIZATION ALGORITHM · p. 21

S5. SOURCE–TARGET SIMILARITY BY ATTACK OUTCOME

Successful attacks were associated with slightly higher clean source–target CLIP similarity than failed attacks across all 12 target models; this post-evaluation analysis does not establish causation. The supplement also specifies the GPT-4o scoring prompt and success threshold used in evaluation.

S5. SOURCE–TARGET SIMILARITY BY ATTACK OUTCOME

  • Successful IAU-FOA attacks had higher clean source–target CLIP similarity than failed attacks across all 12 target models, but differences were small and descriptive.Because samples were grouped by attack outcome after evaluation, the analysis does not establish that greater similarity causes higher ASR.Source: S5. SOURCE–TARGET SIMILARITY BY ATTACK… · p. 22, Table S1 · p. 22

S6. EVALUATION PROMPT

  • GPT-4o returns a subject-level semantic-consistency score from 0 to 1; attacks scoring at least 0.5 count as successful under the same rule across methods and victim models.The score is used directly to compute AvgSim.Source: S6. EVALUATION PROMPT · p. 23

Evaluation Prompt

Loading 2610.06977v1…