PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Resource-Exhaustion Attacks on Vision-Language Models via Joint Pixel-Prompt Optimization

Zhaoxiong Ni, Yatie Xiao, Chi-Man Pun, Fei Peng, Qingxiao Guan, Keke Tang

arXiv:2609.05889v1cs.AIcs.CRcs.CV

TL;DR

Autoregressive VLM availability attacks have largely optimized images while holding visible prompts fixed, leaving jointly controllable input surfaces underexplored. JPPO jointly and stagewise optimizes bounded pixels and a budget-constrained prompt, achieving substantial latency and energy amplification across VLMs and benchmarks. The results support treating per-request cost amplification as a security property for multimodal deployment.

  • Problem

    Existing VLM availability attacks primarily use image-only or loop-centric threat models, leaving joint optimization of image and visible-prompt inputs underexplored.

  • Method

    JPPO uses a restricted two-stage framework that coordinates bounded pixel perturbations with budget-constrained visible-prompt construction and selects strong realized-cost states.

  • Results

    Across five VLM families and two benchmarks, JPPO substantially increases generation length, latency, and estimated energy; it exceeds 36.6× latency and 32.7× estimated-energy amplification on BLIP-2.

  • Takeaways & Limitations

    Ablations attribute amplification to multimodal coordination rather than prompt length or either modality alone, motivating cost-aware evaluation and multimodal serving safeguards.

  • Takeaways & Limitations

    Evaluation is primarily exact-model white-box offline construction on open-source models; black-box transfer is weaker and more variable, and estimated energy is a hardware-dependent proxy.

Abstract

from arXiv · show

Resource-exhaustion attacks against autoregressive vision-language models (VLMs) typically assume unimodal threat models, treating the image branch as the primary optimization surface while holding user-visible prompts fixed. Even recent loop-centric variants remain confined to this single-channel paradigm, leaving the exploitation of availability unexplored as a cross-modal optimization problem over jointly controllable input surfaces. We introduce Joint Pixel-Prompt Optimization (JPPO), the first compound adversarial framework elevating the visible prompt to a first-class adversarial variable alongside image perturbations. Under a restricted joint-input threat model, JPPO performs coupled, stagewise optimization over both the pixel and prompt surfaces. This produces synergistic cost amplification, mechanistically distinct from loop-dependent failures, exhibiting negligible loop incidence in our experiments. Evaluating five open-source VLM families on MS COCO and ImageNet under an 8/255 infinity-norm budget, JPPO achieves over 4.6x latency and 5.3x energy amplification on Qwen2.5-VL-7B, and over 36.6x latency with 32.7x energy amplification on BLIP-2. This represents the strongest cost amplification among directly compared baselines while requiring substantially fewer optimization iterations. Ablations confirm this amplification arises from multimodal coordination rather than prompt length or isolated modalities. These findings reveal structural blind spots in current VLM serving defenses, motivating cost-aware robustness evaluation as a first-class security requirement for multimodal deployments.

1 Introduction

Autoregressive VLM serving cost depends on the realized decoding trajectory, creating an availability risk from inputs that induce disproportionately expensive generations. JPPO addresses this risk by jointly optimizing bounded image perturbations and visible prompts, producing multimodal cost amplification.

  • Longer autoregressive generations increase decoding steps, effective context, latency, and energy overhead.
  • Benign-looking inputs can induce disproportionately expensive decoding trajectories, making availability a security concern.
  • JPPO jointly optimizes bounded pixel perturbations and budget-constrained visible prompts under a restricted joint-input threat model.
  • JPPO differs from fixed-prompt image-only attacks and loop-centric failures by using pixel perturbations and prompt-driven continuation pressure together.
  • Under an ℓ∞= 8/255 budget, JPPO achieves over 4.6× latency and 5.3× estimated-energy amplification on Qwen2.5-VL, and over 36.6× latency and 32.7× estimated-energy amplification on BLIP-2.
  • Matched ablations show that JPPO’s amplification comes from multimodal coordination rather than prompt length alone or either branch in isolation.

2 Related Work

Prior multimodal security work largely targets semantic integrity, while availability studies show that image perturbations can increase decoding cost. JPPO extends this line by optimizing the interface-visible prompt alongside the image to create coordinated cost amplification.

  • Autoregressive VLM serving costs remain tightly coupled to input-triggered decoding length despite system-level efficiency optimizations.
  • Prior multimodal attacks primarily target semantic failures through visual, textual, or coordinated image-text manipulation.
  • Availability research includes latency and energy degradation attacks against test-time inputs, training-time manipulation, and generative systems.
  • Image-only VLM attacks amplify decoding cost through pixel perturbations, including continuations containing user-invisible special tokens.
  • Loop-centric attacks strengthen image-only resource exhaustion by inducing cyclic or highly repetitive token generation.
  • JPPO treats the interface-visible prompt as an optimization variable, coordinating pixel initialization with prompt-driven continuation pressure.

3 Problem Formulation and Threat Model

The threat model targets autoregressive VLM services where attackers control an image and visible prompt, while model parameters, infrastructure, and deployment controls remain protected. JPPO measures per-request cost amplification under bounded, interface-compatible inputs, with practical impact depending on transferability and serving conditions.

  • 3.1 Autoregressive VLM Inference and Resource Cost: Generation terminates at an end-of-sequence token or a system-imposed maximum decoding budget.
  • 3.1 Autoregressive VLM Inference and Resource Cost: Autoregressive VLM inference combines multimodal prefilling with decoding, and resource use depends on the realized output trajectory.
  • 3.1 Autoregressive VLM Inference and Resource Cost: The evaluation cost vector records output length, software-observed latency, and an NVML-based estimated-energy proxy.
  • 3.2 Attack Objective: JPPO seeks higher single-request generation length, latency, and estimated energy while preserving input plausibility under an ℓ∞= 8/255 pixel budget.
  • 3.2 Attack Objective: The prompt is restricted to T(p0), the set reachable from a fixed template through budget-constrained prompt construction, while estimated energy serves as the primary objective.
  • 3.2 Attack Objective: Because latency and estimated energy are noisy, non-differentiable, and serving-stack dependent, JPPO uses surrogate objectives during attack construction.
  • 3.3 Threat Model: The attacker controls the user-provided image and visible prompt but cannot modify model parameters, hidden instructions, decoding, scheduling, admission control, or infrastructure.
  • 3.3 Threat Model: Construction occurs offline using an exact model or surrogate, whereas deployment submits the constructed pair through the ordinary interface without target-side gradients or serving signals.

4 Methodology

JPPO coordinates bounded pixel perturbations with budget-constrained visible-prompt evolution through two stages. It separates broad contextual exploration from single-context refinement while selecting the strongest realized-cost paired state.

  • Overview: JPPO jointly optimizes a bounded pixel perturbation and a budget-constrained visible prompt to increase per-request inference cost.The final adversarial state remains within an ℓ∞ constraint and targets availability degradation rather than semantic task failure.
  • Stage I: Stage I maintains visual, spatial, and semantic prompt paths, merging them into a diverse warm-up prompt alongside projected pixel updates.The strongest measured paired state initializes Stage II.
  • State selection: JPPO selects the strongest realized-cost state across both stages rather than relying solely on the final surrogate-loss iterate.The best-state buffer stores paired image-prompt states and their realized metrics, with estimated energy used for ranking.
  • Stage I: Stage I uses prefix-level differentiable losses for pixel updates and realized generation feedback to construct a broad, adaptive prompt scaffold.Generated responses update the active path memory rather than being copied verbatim into the prompt.
  • Stage II: Stage II replaces separate prompt paths with a single running context refined through selective accumulation, novelty filtering, and compression.This stage performs continuation-oriented refinement while continuing perturbation updates.

5 Experimental Methodology

The evaluation compares JPPO with clean, noise, and image-only baselines across five autoregressive VLMs using matched datasets, hardware, decoding, and measurement protocols. It measures generation length, latency, estimated energy, and broader serving conditions while excluding offline attack-construction cost from amplification claims.

  • Models and datasets: JPPO is evaluated on five autoregressive VLMs using 1,000 sampled images each from MS COCO and ImageNet, repeated across three random seeds.Compared methods use the same sampled subsets for each victim model and seed.
  • Attack setting: The default attack uses an ℓ∞ budget of 8/255, 100 iterations in each stage, and 150- and 200-word prompt budgets.A fixed model-specific initial prompt and stage-wise text-space construction are used across models and datasets.
  • Metrics: Latency includes multimodal prefilling and autoregressive decoding, while generation length counts newly generated words before termination or the decoding cap.Decoding is capped at 512 new tokens with nucleus sampling at temperature 1.0 and top-p 0.9.
  • Metrics: Estimated energy is an NVML-based average-power-times-latency proxy, not an integrated hardware energy measurement.Amplification claims concern only final online serving costs and exclude offline attack-construction cost.
  • Baselines and controls: All methods share model checkpoints, sampled images, hardware, decoding caps, sampling policies, and measurement protocols for fair comparison.Baselines include Clean, random bounded Noise, NICGSlowDown, and Verbose Images; JPPO uses 100 Stage-I and 100 Stage-II iterations by default.
  • Aggregated results: Figure 3 reports mean latency amplification across MS COCO and ImageNet, with error bars showing cross-dataset variation.JPPO yields the largest latency and estimated-energy amplification among directly evaluated baselines under the fixed serving protocol.

6 Evaluation

Across datasets and model families, JPPO consistently amplifies serving cost, transfers beyond exact-model white-box settings, and remains effective under defenses and shared-worker conditions. Ablations attribute the effect to coordinated pixel–prompt optimization rather than prompt length, isolated modalities, or a single loss component.

  • Main comparison: JPPO yields the largest latency and estimated-energy amplification among directly evaluated baselines on MS COCO and ImageNet.The comparison uses the fixed top-p/512-token protocol.
  • Main comparison: 4.13× and 4.08× minimum energy and latency amplification for JPPO exceed Neutral-long-prompt’s maxima of 2.04× and 2.40×.Matching the visible-prompt budget alone does not reproduce JPPO’s serving-cost increase.
  • Cross-model transferability: 5.15× average output-length, 3.97× energy, and 4.12× latency amplification persist across off-diagonal source–target transfers.All 40 dataset-specific off-diagonal directions exceed the matched Clean baseline, while retaining 48.7% of exact-model energy gains and 50.7% of latency gains.
  • Cross-model transferability: 18.8% and 24.2% of external black-box cases exceed 1.2× output-length amplification on Gemini 3.1 Pro Preview and Qwen2.5-VL-72B, respectively.Median amplifications are 1.07× and 1.12×, indicating limited but nonzero transfer.
  • Construction and serving costs: JPPO’s final requests incur higher online energy and latency for every model on both datasets, while identical replay can be mitigated by substantial cache-hit rates.Avoiding overload requires 35–50% hits for Qwen2.5-VL and 65–85% for BLIP-2 across the two datasets.
  • Optimization schedule: 5.95×/6.06×/5.40× average length, energy, and latency amplification under phase-based scheduling exceeds random scheduling’s 4.46×/4.96×/4.28×.The fixed-budget transition retains Stage II, with 31.8% of construction runs reaching the final selected optimum during Stage I.
  • Optimization schedule: Refining top-3 or top-5 candidates raises final b_E by 4.30–4.65% or 5.12–6.17% while increasing build-time and build-energy costs.The default therefore retains the highest-energy Stage-I candidate for Stage-II initialization.
  • Output quality: JPPO retains more original-task utility than Verbose Images, despite lower relevance, informativeness, and usefulness than Clean and higher redundancy.Relative to Prompt-only, JPPO improves informativeness and usefulness while reducing redundancy.

7 Limitations

JPPO is evaluated mainly in exact-model white-box settings on open-source VLMs, with weaker and more variable black-box transfer and limited external API testing. The study also leaves several production-relevant mechanisms and measurements outside scope.

  • Evaluation scope: Exact-model white-box evaluation on open-source models is complemented by weaker, more variable black-box transfer and limited external API testing.External API evaluation measures output length because provider-side resource metrics are unavailable.
  • Measurement boundaries: Absolute latency and estimated-energy values depend on the local hardware and software stack, while the NVML-based energy metric is a proxy.The supplied passages identify hardware/software dependence and proxy measurement as boundaries on interpreting absolute costs.
  • Evaluation protocols: Table 9 reports fixed-trace local API evaluation under serving controls, while Table 10 provides matched comparison with LingoLoop using greedy decoding and a 512-token cap.These tables document specific evaluation protocols rather than removing the broader scope limitations.
  • Open scope: Broader evaluations of perceptual noticeability, downstream utility, speculative decoding, request filtering, and production scheduling remain future work.These areas are explicitly identified as important future work rather than completed evaluation dimensions.

8 Conclusion

The paper concludes that JPPO raises autoregressive VLM generation length, latency, and estimated energy through coordinated pixel and prompt optimization rather than either modality or prompt length alone. It frames per-request cost amplification as a security property requiring cost-aware evaluation and deployment safeguards.

  • Contribution: JPPO jointly manipulates bounded pixel perturbations and a budget-constrained visible prompt across five VLM families and two benchmarks.The released artifact supports the core implementation and backend adapters, while full reproduction requires external checkpoints, datasets, environments, and substantial compute.
  • Findings: JPPO substantially increases generation length, latency, and estimated energy while exhibiting low explicit loop incidence under the paper’s detector.The conclusion distinguishes the observed amplification from explicit loop-dependent behavior.
  • Mechanism: Ablations attribute amplification to coordinated multimodal optimization rather than prompt length, pixel perturbation, or either modality alone.The neutral-prompt control and matched modality ablations support this conclusion.
  • Artifact boundary: The artifact is limited to bounded local evaluation on open-source models and is not intended as a turnkey service-abuse package.It omits distributed traffic generation, rate-limit evasion, and turnkey deployment-abuse tooling.
  • Impact: The work identifies increased serving cost, benign-user latency degradation, queueing pressure, and localized availability loss as potential repeated-submission harms.These risks concern shared or non-fully-metered deployments and may consume compute without proportionally useful output.
  • Responsible evaluation: The evaluation uses controlled local open-source deployments and low-volume, non-concurrent external API checks without estimating provider-side energy or hardware cost.The external checks do not intentionally stress service availability or study safety-filter bypass and data exfiltration.
  • Defenses: Evaluated controls include request-rate limiting, safety admission, runtime early stopping, and repeated-input caching, while broader production mechanisms remain deployment-dependent.Timeouts, output caps, and workload-aware scheduling are explicitly outside the current evaluation scope.

C.4 Defense Implementation Details

The defense implementation sanitizes visible prompts and monitors generated text for low novelty or high repetition. Its early-stopping rule provides only limited mitigation against JPPO’s high-cost behavior.

  • Prompt sanitization: Prompt sanitization collapses whitespace, removes abnormal symbol runs, and suppresses consecutive repeated phrases and n-grams.The cleanup scans from larger to smaller n while preserving a valid interface-visible prompt.
  • Runtime monitoring: EarlyStop monitors repeated-word ratio, novelty ratio, and maximum consecutive repeated n-gram count during generation.These are lightweight word-level statistics based on generated-word and distinct-word counts.
  • Stopping rule: Generation terminates when repetition-dominant behavior crosses thresholds of 0.55 repeated-word ratio, 0.20 novelty ratio, or 3 repeated n-grams.The rule searches n values from 1 through min{4, floor(N_w/2)}.
  • Mitigation result: 5.81%: EarlyStop reduces estimated energy by 5.81% on average across five models and two datasets, versus 2.55% for output length and 2.63% for latency.The limited reductions indicate that low-novelty or high-repetition criteria do not eliminate JPPO’s high-cost behavior.

C.5 Shared-Worker and Local API Evaluation Details

The shared-worker and local-API evaluation uses fixed, trace-replayed workloads with explicit timing boundaries and service controls. Results show that JPPO’s online cost remains higher than Verbose Images across all model–dataset settings, while EarlyStop provides limited reduction and Hidden Tail remains comparatively weak.

  • Evaluation setup: The evaluation uses separate Qwen2.5-VL and BLIP-2 services with four RTX 3090 target workers and a fifth RTX 3090 for WildGuard.Metrics are computed separately by model, dataset, configuration, and arrival seed using the same 1,000-image pools.
  • Timing boundaries: Model service time, queueing delay, FIFO response time, API end-to-end latency, and moderation latency are measured at distinct gateway boundaries.The reported service-time quantity is cumulative model service time, not queueing delay or four-worker wall-clock duration.
  • Uncontrolled protocol: The uncontrolled protocol uses four complete replicas sharing one FIFO queue, independent Poisson arrivals, shuffled 1,000-input pools, and no response caching or continuous batching.Stable and overloaded points are classified using total utilization, with P95 inflation normalized to matched no-attack runs.
  • Rate limiting: Request-RL admits requests through a per-identity token bucket with a 5 requests/minute refill rate and burst capacity 2 before moderation or FIFO entry.Rejected requests receive no downstream processing.
  • Early stopping: EarlyStop is applied only after an admitted request begins generation, so it cannot change JPPO admission under a fixed trace.It can only reduce realized service time when the stopping condition is met.
  • Safety admission: 1.2%: JPPO rejection is 1.2% for Qwen2.5-VL and 0.3% for BLIP-2, while pooled Clean false positives are 0.0%.No moderation output fails to parse in the reported evaluation.
  • Cost comparison: JPPO incurs higher online energy and latency than Verbose Images in all ten model–dataset settings, although construction-cost differences remain model-dependent.The comparison direction is consistent across MS COCO and ImageNet.
  • Baseline comparison: Hidden Tail yields nonzero but comparatively weak amplification across evaluated models under the unified protocol.The paper treats it as an appendix-only reference rather than a directly compared main-table baseline.

D.7 Exact Values for Optimization-Space Ablation

The optimization-space ablation shows that joint pixel–prompt optimization produces the strongest resource amplification, exceeding either branch alone. The full three-term objective also outperforms any individual loss component.

  • The joint setting consistently yields the largest output length, estimated energy, and latency on both datasets.This supports multimodal coordination as the source of JPPO’s gain rather than either branch in isolation.
  • All three availability-oriented loss terms contribute to resource amplification, but no single component recovers the full JPPO effect.The full objective remains strongest overall, while the prefix stop-suppression loss is generally strongest among single-loss variants.

D.9 Additional Cross-Model Transfer Results

Cross-model transfer remains nonzero but weaker and asymmetric than exact-model construction. Diagnostic results indicate that JPPO sustains costly, less-novel continuations without generally collapsing into explicit token loops.

  • Transfer is on average weaker than exact-model construction but remains nonzero and asymmetric across source–target directions.The complete output-length and estimated-energy results cover all source–target model pairs.
  • JPPO reduces token-level novelty relative to benign generation, but attacked outputs remain above collapse-to-loop behavior on most models.The novelty statistic uses normalized generated tokens with a sliding window of 32, consistent with the two-dataset loop-incidence results.
  • JPPO often sustains long and costly continuations through broader multimodal steering rather than explicit token-loop failure alone.The comparison with LingoLoop is interpreted under a matched decoding protocol while preserving their different threat models.

D.11 Bottleneck-Ratio Sensitivity Results

Sensitivity studies show that JPPO’s effectiveness depends on bottleneck ratio, perturbation and optimization budgets, prompt growth, decoding caps, and stage scheduling. Strong amplification persists under moderate settings, but several relationships are non-monotonic.

  • Bottleneck-ratio sensitivity: ρ = 0.10 yields the strongest latency and remains near the strongest setting across other metrics, although ρ = 0.04 leads on two specified measures.The default is selected as a strong and stable setting rather than a globally optimal bottleneck ratio.
  • Gradient-based evidence localization: JPPO alters visual evidence localization qualitatively, but Grad-CAM maps are supporting evidence rather than standalone causal proof.Adversarial inputs may produce weaker, more diffuse, more fragmented, or shifted activation patterns.
  • Perturbation and optimization budgets: Increasing perturbations from 2/255 to 8/255 raises all three reported metrics on MS COCO, while larger budgets do not consistently help.Length rises from 196.83 to 373.23 words, latency from 10.07 to 19.58 s, and estimated energy from 1286.49 to 2324.29 J.
  • Stage-wise scheduling: The 200/200 schedule achieves the highest values in its sweep, but optimization strength is not strictly monotonic; 100/100 remains the controlled default.The authors avoid claiming a universally optimal schedule.
  • Prompt budgets: Moderate prompt budgets already induce substantial amplification, and the default 150/200-word setting performs best across both datasets.Controlled prompt growth is favored over unconstrained accumulation through the two-stage warm-up and refinement design.
  • Generation caps: Relaxing the generation cap increases length, latency, and estimated energy, while smaller caps still leave nontrivial overhead over clean inputs.Generation caps therefore bound maximum extractable per-request cost without eliminating the availability surface.
  • Decoding strategies: JPPO achieves the highest absolute cost among evaluated methods across each tested decoding strategy.The comparison includes greedy decoding, top-k decoding with k = 10, and top-p decoding with p = 0.9.
  • Variability: JPPO’s increased mean serving cost is often accompanied by variability comparable to or lower than stronger image-only baselines.This suggests the gains are not driven solely by a small number of extreme outliers.

D.20 Judge-Based Output Evaluation

The judge-based evaluation assesses output quality alongside resource exhaustion, while additional analyses compare caption overlap, representation similarity, and qualitative cost increases. Quality rankings vary by model and metric, and strong amplification can occur without substantial CLIP-space drift.

  • Judge-based evaluation: The evaluation protocol scores informativeness, usefulness, relevance, and redundancy for matched outputs under a common image-grounded judging setup.The judge is instructed not to reward length by itself and evaluates 450,000 candidate outputs.
  • Caption quality: Caption-quality ordering depends on both the target model and metric, with JPPO not uniformly dominating overlap scores.On Qwen2.5-VL, JPPO exceeds Verbose Images on BLEU but trails it on CIDEr; on BLIP-2, it does not dominate caption-overlap metrics.
  • Representation similarity: JPPO adversarial images remain highly similar to originals in CLIP embedding space across reported model–dataset pairs.Strong resource amplification therefore does not require substantial representation-level drift.
  • Qualitative examples: Across all five victim models, attacked inputs produce longer decoding trajectories and report realized length, estimated energy, and latency.The qualitative comparisons use the same image and prompt for clean and attacked conditions.
Loading 2609.05889v1…