PaperlayerView exact revision on arXiv

Source-linked AI summary

FusionAudit: Pathway-Conditioned Robustness Auditing for Native Multimodal Models

Jiaming Zhang, Zherui Li, Siqi Guo, Junhao Dong, Kun Wang, Wei Yang Bryan Lim

arXiv:2610.21667v1cs.CV

TL;DR

Current evaluations separate pixel attacks from rendered-text attacks despite native VLMs integrating visual and textual inputs more deeply. PaRMA uses pathway-specific diagnostics to guide FusionAudit’s strengthened attacks, which raise estimated risk across fusion levels and expose vulnerabilities missed by standard benchmarks.

  • Problem

    Robustness evaluations remain siloed into pixel-space attacks and typographic overlays, without accounting for native VLMs’ integrated visual and textual processing.

    Source: §1 · p. 1

  • Method

    PaRMA organizes audits around pixel perturbation and rendered-text insertion, using pathway-specific diagnostics to guide FusionAudit’s MANTIS and Nat-Evid attacks.

    Source: §1 · p. 2, §4 · p. 3

  • Results

    The diagnostic-driven suite raises level-mean risk over the baseline by 10.6% in L1, 16.9% in L2, and 25.1% in L3.

    Source: §5.2 · p. 6

  • Takeaways & Limitations

    Across 18 VLMs, legacy single-attack proxies yield unstable rankings for native models, and diagnostic-driven attacks expose residual risks missed by standard benchmarks.

    Source: §1 · p. 2, §6 · p. 10

    The L1–L3 taxonomy is observational, L4 remains unaudited, and rendered-text evaluation excludes command-following visual prompt injection.

    Source: §6 · p. 10

Abstract

from arXiv · show

As vision-language models (VLMs) transition from frozen-encoder pipelines to native multimodal architectures, the visual channel increasingly serves as both a perceptual input and a carrier of rendered text that competes with user prompts. Current single-attack, pixel-centric evaluations fail to capture this complexity: they collapse distinct visual attack surfaces into a single scalar and ignore how architectural shifts alter vulnerability. We introduce PaRMA, an evidence-based auditing framework that defines pathway-conditioned risk over two primary image-level interventions: pixel perturbation and rendered-text insertion. Instead of arbitrary attack enumeration, PaRMA uses pathway-specific diagnostics to expose architecture-specific blind spots. Notably, we uncover a see-but-not-deceived signature where native VLMs attend heavily to typographic overlays but visually reject them, and we identify an objective mismatch between encoder-level and answer-level attacks in deep-fusion models. To exploit these findings, we instantiate FusionAudit, introducing two diagnostic-motivated attacks: MANTIS (margin-aware pixel perturbation) and Nat-Evid (host-congruent semantic insertion). Evaluating 18 open-weight VLMs across a novel nativeness taxonomy, we demonstrate that legacy single-attack rankings (e.g., PGD) are unstable for native models. Our diagnostic-driven attacks substantially raise the estimated residual risk, showing that PaRMA provides a falsifiable, evidence-driven auditing methodology for uncovering vulnerabilities missed by standard benchmarks.

1 Introduction

Native multimodal architectures make visual inputs and rendered text compete directly with prompts, but existing evaluations separate pixel attacks from typographic overlays. PaRMA and FusionAudit use pathway-specific diagnostics to expose blind spots and motivate attacks tailored to them.

1 Introduction

  • Native VLMs integrate visual and textual tokens earlier or deeper, while existing evaluations remain siloed into pixel-space attacks and typographic overlays.The visual channel can carry both perceptual input and rendered text that competes with the user’s prompt.Source: §1 · p. 1
  • Legacy encoder-level attacks miss deep decoder gradients, while end-to-end attacks can underweight the answer-commitment position; native models also attend to overlays without adopting them as responses.Visual rejection leaves additional semantic failures unmeasured.Source: §1 · pp. 1–2
  • PaRMA organizes auditing around pixel perturbation and rendered-text insertion, using behavioral and gradient diagnostics to guide MANTIS and Nat-Evid.MANTIS concentrates gradients at the first-divergence token; Nat-Evid naturalizes text payloads to bypass visual rejection.Source: §1 · p. 2
  • Across 18 VLMs, legacy single-attack proxies produce unstable rankings for native models, while diagnostic-driven attacks raise estimated residual visual-channel risk.The contributions also include a graded L1–L3 nativeness taxonomy for organizing audit targets.Source: §1 · p. 2

2 Related Work

Prior work studies modular and native VLM architectures, pixel and typographic attacks, and robustness diagnostics. PaRMA differs by using pathway-specific diagnostics to guide stronger attacks and reporting residual risk after that strengthening.

2 Related Work

  • Native VLMs use multi-layer visual injection or early-fusion backbones, extending beyond modular pipelines and motivating evaluation beyond assumptions inherited from those pipelines.Prior work examined how vision backbones, connectors, and training recipes affect behavior.Source: §2 · p. 2
  • Prior VLM evaluations often treat pixel perturbations and text overlays as separate threat vectors rather than measuring their joint risk in native models.Prior studies adapted pixel attacks to encoder representations, cross-modal alignment, or final responses, while typographic attacks alter image interpretation.Source: §2 · p. 2
  • PaRMA integrates pathway-specific diagnostics into the audit and reports residual risk after diagnostic-driven strengthening, rather than the maximum over a predefined attack collection.Earlier diagnostics were typically used post hoc to explain failures rather than guide subsequent attacks.Source: §2 · pp. 2–3

3 Native Multimodal VLMs: Taxonomy and Threat Scope

The paper organizes VLMs by fusion depth and pretraining strategy, then defines an audit focused on visual-channel interventions. The taxonomy is an organizational framework, and its levels co-vary with other model characteristics.

3 Native Multimodal VLMs: Taxonomy and Threat Scope

  • Moving from modular pipelines to native multimodal training alters gradient paths and makes visual and textual tokens compete through joint early-layer attention.The taxonomy and formalized visual threat scope organize an audit of this changing attack surface.Source: §3 · p. 3

3.1 Nativeness Taxonomy

  • Fusion level co-varies with model family, scale, and training data, so taxonomy-level trends are correlational rather than a strict causal ladder.The taxonomy serves as an organizational axis for the audit.Source: §3.1 · p. 3
  • The L1–L3 taxonomy groups models by fusion depth and pretraining strategy, from projection-bottleneck separation to shallow fusion and deep or early fusion.L1 uses a frozen or lightly adapted encoder; L2 uses a fixed visual prefix; L3 includes multi-layer reinjection or shared transformer blocks from the first layer. L4 is theoretical, and no audited open-weight model meets its threshold.Source: §3.1 · p. 3

3.2 Audit Scope and Visual Pathways

  • The audit fixes the text prompt and intervenes only on the visual input, evaluating attack success on examples answered correctly before intervention.Its pathways are pixel-space perturbation and rendered-text insertion; the audit estimates failure over their union and asks whether behavioral signatures can guide stronger attacks.Source: §3.2 · p. 3

4 Method: PaRMA and FusionAudit

PaRMA turns visual robustness auditing into a pathway-conditioned, diagnostic-driven process, using findings from baseline attacks to motivate stronger interventions and estimate residual risk. FusionAudit applies this process to pixel perturbations and rendered-text insertion.

4.1 Pathway-Conditioned Risk Formulation

  • PaRMA treats pixel perturbation and rendered-text insertion as distinct pathways and strengthens a non-static risk lower bound through pathway-resolved diagnostics.FusionAudit instantiates the protocol by using baseline attack signals to guide diagnostic-motivated augmented attacks.Source: §4 · p. 3, §4.1 · p. 4

4.2 Baselines and Diagnostic Suite

  • Vis-PGD targets vision-encoder features, End-PGD attacks the full answer sequence, and Typo-Evid overlays a task-matched wrong answer at a fixed image region.For rendered-text failures, RUG measures reading-versus-use behavior, while OAR measures attention to the overlay relative to its area.Source: §4.2 · pp. 4–5
  • High task-prompt OAR and positive RUG are consistent with a see-but-not-deceived signature: overlays receive disproportionate attention but are not reliably adopted in responses.This pattern suggests visual distinctiveness in the Typo-Evid baseline.Source: §4.2 · p. 5

4.3 Diagnostic-Driven Augmentation

  • MANTIS concentrates pixel-attack gradient pressure at the first token where fixed canonical correct and wrong continuations diverge, then optimizes the margin there.This augmentation addresses End-PGD’s sequence-wide gradient pressure.Source: §4.3 · p. 5
  • Nat-Evid tests whether host-congruent rendering reduces the read-without-use gap by placing a wrong answer as native-style image content rather than an extrinsic overlay.FusionAudit combines MANTIS and Nat-Evid to compute a strengthened risk estimate, its principal falsifiable audit deliverable.Source: §4.3 · p. 5

5 Experiments

Across 18 VLMs, the pathway-complete audit finds higher risk under diagnostic-driven attacks, while deeper-fusion models show distinct pixel- and rendered-text failure patterns. Stability checks, layer restoration, and generator comparisons further characterize these results.

5.1 Experimental Setup

  • The audit uses 1,500 balanced ChartQA, DocVQA, and MME examples, evaluates 18 models across L1–L3, and conditions ASR on each model’s clean-correct examples.Pixel attacks use an ℓ∞ budget of ε = 8/255 and 40 PGD steps; the MME natural-image subset preserves level-mean ASR patterns within 3.2 points.Source: §5.1 · pp. 5–6

5.2 Main Results

  • The full-audit exclusive additional-risk fraction grows from 10.6% to 16.9% to 25.1%, while absolute residual risk falls from 97.0% to 88.4% to 75.6%.MANTIS-exclusive flips rise from 15.2% in L1 to 24.6% in L3, while Nat-Evid-exclusive flips are 10.4%, 23.7%, and 20.6% across L1–L3.Source: §5.2 · p. 6
  • Risk rises by +10.6% in L1, +16.9% in L2, and +25.1% in L3 when moving from bR0 to bR+.The corresponding 95% CIs exclude zero, supporting additional failures exposed by the diagnostic-driven suite.Source: §5.2 · p. 6, Table 1 · p. 7
  • End-PGD and pathway-complete rankings differ for 8 of 18 models by at least three ranks, while bR+ decreases along the within-family Qwen chain.For example, Gemma3-12B shifts from 2nd under End-PGD to 7th under bR+; the Qwen chain shows the diagnostic-driven risk gap persisting as absolute risk decreases.Source: §5.2 · pp. 6–7

5.3 Pixel Pathway Deep Dive

  • MANTIS has a median standard deviation of 1.3% across three seeds, ASR largely saturates by 30–40 steps, and the L1 > L2 > L3 ordering persists across tested radii.Increasing the budget to 50 steps yields only +2.8%, +1.0%, and +2.2% level-mean gains for Levels 1–3, respectively.Source: §5.3 · p. 7, Figure 2 · p. 8
  • End-PGD flip recovery falls from 92.2% in L1 to 25.6% in L3, consistent with deep-fusion flips surviving restoration as visual features re-enter computation across layers.The answer-level objective in MANTIS adds the most exclusive flips in this setting.Source: §5.3 · pp. 8–9

5.4 Text Pathway Deep Dive

  • Nat-Evid exceeds Typo-Evid by 13.6% in L2 and 15.0% in L3, while native groups show rising Pread and falling Puse from L2 to L3.OAR increases by +138% in L2 and +135% in L3, and a Qwen3.5-2B example attends to an injected banner yet retains the visual answer.Source: §5.4 · p. 9, Figure 3 · p. 9
  • Three exact-text Nat-Evid generators produce highly consistent average ASRs of about 19%, while the Gemini-backed variant is strongest, driven by MME.On 100 natural-image cases, masking the inserted cue restores the correct answer in 98; OCR recovers the exact ground-truth string in 92.3% of eligible chart and document cases.Source: §5.4 · p. 10, Figure 4 · p. 10

6 Conclusion and Limitations

The authors conclude that native multimodal architectures alter VLM attack surfaces, making single-attack, pixel-centric audits insufficient; FusionAudit uses pathway-specific diagnostics and attacks to expose residual risks across 18 VLMs. They also identify scope limitations in the audit and note that MANTIS requires white-box gradient access.

6 Conclusion and Limitations

  • FusionAudit's diagnostic-motivated MANTIS and Nat-Evid exposed substantial residual risks systematically hidden by standard benchmarks across 18 VLMs.Source: §6 · p. 10
  • MANTIS requires white-box gradient access, and the authors release the audit pipeline rather than precomputed adversarial images, although the attacks could also mislead deployed VLMs.Source: §6 · p. 10
  • The L1–L3 taxonomy is observational rather than a causal ladder, L4 remains outside the empirical audit, and adjacent levels differ in training data and scale.Source: §6 · p. 10
  • Rendered-text auditing covers task-relevant wrong-answer injection, not command-following visual prompt injection, which requires compliance-based metrics.Source: §6 · p. 10
  • PaRMA currently instantiates two visual pathways but is designed to extend to audio, video, and embodied tool-use pathways.Source: §6 · p. 10

Related papers

Loading 2610.21667v1…