PaperlayerView exact revision on arXiv

Source-linked AI summary

Inducing High Energy-Latency of Large Vision-Language Models with Verbose Images

Kuofeng Gao, Yang Bai, Jindong Gu, Shu-Tao Xia, Philip Torr, Zhifeng Li, Wei Liu

arXiv:2401.11170v2cs.CVcs.CR

TL;DR

VLM inference requires substantial resources, and induced energy-latency cost can threaten availability; earlier attacks do not directly transfer to VLMs. This paper crafts imperceptible verbose images to lengthen generated sequences, reporting increases of 7.87× on MS-COCO and 8.56× on ImageNet across four VLMs.

  • Problem

    Inducing high energy-latency cost can exhaust computational resources and reduce VLM availability, while earlier methods cannot directly transfer to VLMs.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    The paper crafts imperceptible verbose images using delayed-EOS, uncertainty, and token-diversity losses with temporal weight adjustment.

    Source: §1 · p. 2

  • Results

    7.87× on MS-COCO and 8.56× on ImageNet: verbose images increased generated-sequence length relative to original images across four VLMs.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The authors propose verbose images as a baseline for inducing high energy-latency cost in VLMs.

    Source: §6 · p. 9

    Earlier methods target LLMs or smaller models, and specific-token objectives are difficult with VLMs' random outputs for the same input.

    Source: §1 · p. 2, §2 · p. 3

Abstract

from arXiv · show

Large vision-language models (VLMs) such as GPT-4 have achieved exceptional performance across various multi-modal tasks. However, the deployment of VLMs necessitates substantial energy consumption and computational resources. Once attackers maliciously induce high energy consumption and latency time (energy-latency cost) during inference of VLMs, it will exhaust computational resources. In this paper, we explore this attack surface about availability of VLMs and aim to induce high energy-latency cost during inference of VLMs. We find that high energy-latency cost during inference of VLMs can be manipulated by maximizing the length of generated sequences. To this end, we propose verbose images, with the goal of crafting an imperceptible perturbation to induce VLMs to generate long sentences during inference. Concretely, we design three loss objectives. First, a loss is proposed to delay the occurrence of end-of-sequence (EOS) token, where EOS token is a signal for VLMs to stop generating further tokens. Moreover, an uncertainty loss and a token diversity loss are proposed to increase the uncertainty over each generated token and the diversity among all tokens of the whole generated sequence, respectively, which can break output dependency at token-level and sequence-level. Furthermore, a temporal weight adjustment algorithm is proposed, which can effectively balance these losses. Extensive experiments demonstrate that our verbose images can increase the length of generated sequences by 7.87 times and 8.56 times compared to original images on MS-COCO and ImageNet datasets, which presents potential challenges for various applications. Our code is available at https://github.com/KuofengGao/Verbose_Images.

1 INTRODUCTION

The paper frames VLM inference as an availability risk because high energy-latency cost can exhaust resources, and proposes imperceptible verbose images to induce longer outputs.

1 INTRODUCTION

  • VLMs deliver multimodal performance but require substantial deployment resources, and inducing high inference energy-latency cost can exhaust resources and reduce availability.The introduction notes that inference accounts for over 90% of machine-learning demand, according to NVIDIA and Amazon Web Services.Source: §1 · p. 1
  • Earlier energy-latency methods were designed for LLMs or smaller-scale models and cannot be directly applied to VLMs.The paper develops verbose images for the VLM setting.Source: §1 · p. 2
  • Energy consumption and latency time are approximately positively linearly related to generated-sequence length, motivating sequence lengthening as an attack route.The paper proposes imperceptible visual perturbations to induce VLMs to generate long sentences.Source: §1 · p. 2
  • Verbose images combine delayed EOS, output uncertainty, and token diversity losses with temporal weight adjustment to encourage longer, varied sequences.The losses target EOS placement, token-level dependency, and sequence-level dependency, respectively.Source: §1 · p. 2
  • 7.87× on MS-COCO and 8.56× on ImageNet: verbose images increased generated-sequence length relative to original images across four VLM models.The paper also reports dispersed visual attention and complex sequences containing hallucinated contents.Source: §1 · p. 2

2 RELATED WORK

Prior work manipulates inference energy and latency through activation computation or longer outputs, while VLMs introduce distinct transfer challenges that motivate a tailored method.

2 RELATED WORK

  • VLMs such as BLIP, BLIP-2, InstructBLIP, and MiniGPT-4 support multimodal tasks through visual-language integration and instruction tuning.BLIP-2 uses a query transformer to bridge a vision transformer and an LLM.Source: §2 · pp. 2–3
  • Sponge samples increase representation computation by maximizing activation L2 norms, while NICGSlowDown increases decoder calls by minimizing EOS and output-token logits.The former targets LLMs; NICGSlowDown targets smaller-scale captioning models.Source: §2 · p. 3
  • Earlier methods do not directly transfer to VLMs because they target other model settings, and specific-token objectives are difficult with random sampled outputs.The paper identifies a need for VLM-specific approaches to induce high energy-latency cost.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries specify the attacker's target and assumptions, then formulate sequence-length maximization under an imperceptibility constraint.

3.1 THREAT MODEL

  • The threat model seeks an imperceptible image perturbation that induces long VLM outputs, with perturbation magnitude constrained in an lp norm.The target may be accessed with full model knowledge or be inaccessible, a more challenging scenario described in appendices.Source: §3.1 · p. 3

3.2 PROBLEM FORMULATION

  • Figure 2 summarizes verbose-image optimization through EOS delay, uncertainty, token diversity, and temporal adjustment of the three objectives.Source: Figure 2 · p. 4
  • The problem formulation treats VLM output as autoregressive tokens and seeks to maximize sequence length N while restricting image perturbation magnitude.The output context can be an image-captioning placeholder or a question for visual question answering and reasoning.Source: §3.2 · pp. 3–4

4 METHODOLOGY

The methodology optimizes imperceptible image perturbations to lengthen VLM outputs, using three losses to delay EOS and disrupt output dependency, with temporally adjusted weights to balance them.

4 METHODOLOGY

  • Three losses target EOS delay, token-level uncertainty, and sequence-level token diversity to lengthen VLM outputs and disrupt output dependency.The method combines them in an overall objective and uses temporal weight adjustment during optimization.Source: §4 · p. 4, §4.1 · p. 5

4.1 LOSS DESIGN

  • Minimizing EOS-token probability at all generated positions encourages VLMs to produce more tokens before stopping.The delayed EOS loss addresses the difficulty of predicting the exact EOS position in non-deterministic generation.Source: §4.1 · p. 4
  • Maximizing output-distribution entropy encourages alternative token predictions, while nuclear norm approximates hidden-state rank to promote token diversity.The uncertainty loss targets token-level dependency; the diversity loss targets sequence-level dependency and can produce more complex sequences.Source: §4.1 · pp. 4–5

4.2 OPTIMIZATION

  • Projected gradient descent optimizes the weighted sum of the three losses while projecting perturbations onto the imperceptibility constraint.The perturbation magnitude ϵ sets the constraint; α is the step size.Source: §4.2 · pp. 5–6
  • Normalization scaling, temporal decay functions, and momentum update the loss weights to balance objectives with different convergence rates.The method incorporates T1(t), T2(t), and T3(t) into the corresponding weights and smooths updates using previous weights.Source: §4.2 · p. 6

5 EXPERIMENTS

Experiments across four VLMs find that verbose images produce longer sequences and the highest energy-latency cost among evaluated methods. Visual and textual analyses, plus ablations, examine attention, hallucinations, loss objectives, optimization modules, and perturbation magnitude.

5.1 EXPERIMENTAL SETUPS

  • Experiments evaluate captioning with four VLMs on 1,000 randomly selected images each from MS-COCO and ImageNet.The models are BLIP, BLIP-2, InstructBLIP, and MiniGPT-4.Source: §5.1 · p. 6
  • Evaluation compares original images, random noise, sponge samples, NICGSlowdown, and verbose images using single-GPU energy, latency, and sequence length averaged over three runs.Perturbation baselines use 1,000 PGD iterations with ε = 8 under an l∞ restriction.Source: §5.1 · p. 6

5.2 MAIN RESULTS

  • 7.87× on MS-COCO and 8.56× on ImageNet: verbose images yielded the longest sequences and highest energy-latency cost among evaluated methods.The length-distribution peak also shifts toward longer sequences for verbose images.Source: Table 1 · p. 7, §5.2 · p. 7, Figure 3 · p. 7

5.3 DISCUSSIONS

  • Grad-CAM maps show attention shifting from a localized object region on original images to more dispersed, uniform focus across verbose images.The authors conjecture that longer sequences are reflected in inaccurate, dispersed visual attention.Source: §5.3 · p. 8, Figure 4 · p. 8
  • Verbose-image outputs contain a broader set of objects and more hallucinated objects than outputs from original images.The study evaluates hallucination with CHAIRi and CHAIRs across four VLMs.Source: §5.3 · p. 8, Table 2 · p. 8

5.4 ABLATION STUDIES

  • Each loss objective individually lengthens sequences, while combining all three yields the best sequence-length results and suggests complementary effects.The objectives delay EOS occurrence, enhance output uncertainty, and improve token diversity.Source: §5.4 · p. 8, Table 3 · p. 9
  • Temporal decay and momentum each contribute to sequence length, and their combination yields the longest outputs on MS-COCO and ImageNet.The comparison is against the baseline without either optimization module.Source: §5.4 · p. 9, Table 4 · p. 9
  • Larger perturbation magnitudes produce longer sequences but make verbose images more perceptible, creating an image-quality versus energy-latency trade-off.Perceptual difference is assessed using LIPIS.Source: §5.4 · p. 9, Table 5 · p. 9

6 CONCLUSION

The conclusion presents verbose images as imperceptible perturbations designed to induce high inference energy-latency cost by prompting VLMs to generate more tokens. Across four VLMs, they increase sequence length by 7.87× on MS-COCO and 8.56× on ImageNet; the authors propose them as a baseline for this attack surface.

6 CONCLUSION

  • Verbose images aim to induce high inference energy-latency cost by prompting VLMs to generate as many tokens as possible through imperceptible perturbations.The approach uses delayed EOS, uncertainty, and token-diversity losses with temporal weight adjustment.Source: §6 · p. 9
  • 7.87× on MS-COCO and 8.56× on ImageNet: verbose images increased generated-sequence length across four VLMs relative to original images.Source: §6 · p. 9
  • The authors propose verbose images as a baseline for inducing high energy-latency cost in VLMs.Source: §6 · p. 9

ETHICS STATEMENT

The authors limit the work to laboratory experiments and do not support using verbose images in real-world scenarios. The appendices describe experimental resources and settings, including four target VLMs and their image-captioning inputs.

ETHICS STATEMENT

  • The authors restrict experiments to the laboratory and do not support verbose images in real-world scenarios.They frame the work as raising awareness of VLM availability security and energy-latency costs.Source: ETHICS STATEMENT · p. 10

Appendix

  • The appendices cover target models, experimental setups, transferability, other multimodal tasks, visual interpretation, and additional analyses.They also include ablations, parameter searches, and visual examples.Source: Appendix · p. 14

A IMPLEMENTATION DETAILS

A.1 TARGET MODELS

  • The four target models are BLIP, BLIP-2, InstructBLIP, and MiniGPT-4, with model-specific image resolutions and captioning prompts.BLIP uses a 224M encoder-decoder and 384 × 384 images; BLIP-2 uses OPT-2.7B, while InstructBLIP and MiniGPT-4 use Vicuna-7B and 224 × 224 images.Source: §A.1 · p. 14

A.2 EXPERIMENTAL SETUPS

The experimental setup optimizes verbose images and comparison attacks with PGD under specified perturbation and generation settings. In the black-box setting, transferable verbose images are less effective than white-box images but still lengthen generated sequences.

A.2 EXPERIMENTAL SETUPS

  • The main experiments optimize sponge samples, NICGSlowDown, and verbose images with PGD for 1,000 iterations and an l∞ perturbation magnitude of 8.The step size is 1; generated sequences are capped at 512 tokens with nucleus sampling p = 0.9 and temperature 1.Source: §A.2 · pp. 14–15
  • The CHAIR metric is used to measure object hallucination, with higher values indicating more hallucinated objects.SEEM is used to segment images and obtain their objects for the metric's ground truth.Source: §A.2 · p. 15

B BLACK-BOX SETTING

  • The black-box setting assumes the victim VLM is inaccessible and transfers verbose images crafted on a known surrogate model.The surrogate is selected from the VLMs other than the target victim.Source: B BLACK-BOX SETTING · p. 15, §A.2 · p. 15
  • On MS-COCO and ImageNet, the length-distribution peaks for verbose images shift toward longer sequences across BLIP, BLIP-2, InstructBLIP, and MiniGPT-4.Source: Figure 5 · p. 16, Figure 6 · p. 16
  • Transferable verbose images produce longer generated sequences than original images in the black-box setting, though they are less effective than white-box images.The evaluation covers transfer across four VLMs; Table 6 reports sequence length, energy consumption, and latency.Source: Table 6 · p. 15, B BLACK-BOX SETTING · p. 16

C MORE TASKS

The paper evaluates verbose images on VQA and visual reasoning with BLIP-2, reporting the highest energy-latency cost among the three multimodal tasks considered.

C MORE TASKS

  • Verbose images achieve the highest energy-latency cost among image captioning, VQA, and visual reasoning in the reported BLIP-2 experiments.VQA uses VQAv2 and visual reasoning uses GQA, with the prompt template “Question: {} Answer:”.Source: C MORE TASKS · p. 16

D MORE RESULTS OF LENGTH DISTRIBUTION

Generated sequence lengths show a bimodal pattern, while joint image-and-text optimization still yields longer sequences than other methods. The appendices also present Grad-CAM visualizations and additional analyses of attack feasibility, cost, and variability.

D MORE RESULTS OF LENGTH DISTRIBUTION

  • Verbose images tend to produce either long or short sequences, with short outputs arising from failed induction or VLM confusion and refusal-like responses.Failed induction is noted particularly for smaller, non-instruction-tuned BLIP and BLIP-2; confusion and refusal-like outputs occur predominantly with instruction-tuned InstructBLIP and MiniGPT-4.Source: D MORE RESULTS OF LENGTH DISTRIBUTION · pp. 16–17

E JOINT OPTIMIZATION OF BOTH IMAGES AND TEXTS

F VISUAL INTERPRETATION

G ADDITIONAL DISCUSSIONS

  • Additional discussions examine whether generation-length limits address the vulnerability, compare three attacks, and report image-embedding distance, verbose-image energy cost, and main-table standard deviations.Appendix H covers sampling-policy and maximum-length ablations; Appendix I reports loss-weight and momentum grid searches.Source: G ADDITIONAL DISCUSSIONS · p. 17

G.1 FEASIBILITY ANALYSIS OF AN INTUITIVE SOLUTION

The section argues that diverse user needs make large generation limits difficult to avoid, then evaluates verbose images against competing attacks, visual similarity, generation costs, and variability.

G.1 FEASIBILITY ANALYSIS OF AN INTUITIVE SOLUTION

  • Diverse prompts require a wide range of sentence lengths, so providers use large token limits; verbose images can approach those limits while staying within them.The paper argues that this makes a simple generation-length limit infeasible as a mitigation.Source: §G.1 · pp. 17–18, Figure 8 · p. 18

G.2 EVALUATION PERFORMANCE OF ENERGY-LATENCY MANIPULATION

  • 226.72 generated tokens for verbose images exceeded sponge samples at 22.53, and both sequence length and captioning performance exceeded NICGSlowDown on BLIP-2 MS-COCO.Table 9 covers attack performance and captioning metrics; the prose reports these sequence-length comparisons.Source: Table 9 · p. 19, §G.2 · p. 19

G.3 IMAGE EMBEDDING DISTANCE BETWEEN ORIGINAL IMAGES AND ATTACKED COUNTERPART

  • The attacked images and original images have similar image-embedding distances across verbose images, sponge samples, and NICGSlowDown.The distance is calculated from CLIP image embeddings.Source: Table 10 · p. 19, §G.3 · p. 19

G.4 ENERGY CONSUMPTION FOR GENERATING ONE ATTACKED IMAGE

  • Attack energy-latency cost and the energy required to generate one verbose image both rise with attack iterations, while generating an image costs more than using it to attack.The authors suggest reusing each generated image by sending copies to the target VLM.Source: §G.4 · pp. 19–20, Table 11 · p. 20

G.5 STANDARD DEVIATION RESULTS

  • Table 12 reports standard deviations for generated-sequence length, energy consumption, and latency time.The supplied text does not state the standard-deviation values.Source: Table 12 · p. 20, §G.5 · p. 20

H ADDITIONAL ABLATION STUDIES

The additional studies test verbose images across sampling policies and generation limits, search loss-weight and momentum settings, and visualize outputs across several VLMs.

H.1 DIFFERENT SAMPLING POLICIES

  • Verbose images induce the longest sequences across greedy, beam, top-k, and nucleus sampling, indicating the result is not sensitive to sampling policy.The default uses nucleus sampling with p = 0.9 and temperature t = 1; beam width is 5 and top-k uses k = 10.Source: Table 13 · p. 21, §H.1 · pp. 21–22

H.2 DIFFERENT MAXIMUM LENGTHS OF GENERATED SEQUENCES

  • Verbose images consistently outperform four other image categories as maximum generation length rises; longer outputs coincide with higher energy consumption and latency.Table 14 reports generated-sequence length, energy consumption, and latency across varying maximum lengths.Source: Table 14 · p. 21, §H.2 · p. 22

I GRID SEARCH

  • The authors conduct grid searches over loss weights and momentum values to select optimization settings.The loss-weight search is described in Table 15–20, and momentum results in Table 21.Source: I GRID SEARCH · p. 22, §I.1 · p. 22, §I.2 · p. 22

I.1 DIFFERENT PARAMETERS OF LOSS WEIGHTS

  • Grid search selected a1 = 10, b1 = −20, a2 = 0, b2 = 0, a3 = 0.5, and b3 = 1, which induced the longest sequences among tested settings.The search results are reported in Tables 15–20.Source: §I.1 · p. 22, Table 15 · p. 23, Table 16 · p. 23

J VISUALIZATION

I.2 DIFFERENT MOMENTUM VALUES

  • The momentum results support including momentum during verbose-image optimization.The tested values are m = {0, 0.3, 0.6, 0.9}.Source: §I.2 · p. 22, Table 21 · p. 23

Related papers

Loading 2401.11170v2…