PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Quantum Layers offer Performance and Security Advantages in Vision-Language Models?

Md Khalid Syfullah, Alvi Ataur Khalil

arXiv:2610.28964v1quant-phcs.AIcs.CVcs.LG

TL;DR

The paper asks how quantum layers might affect vision-language models, where evidence remains limited. It evaluates classically simulated quantum layers in LLaVA-OneVision, finding that outcomes vary with placement and replacement choice. Selected insertion and replacement configurations show accuracy and parameter-count benefits, while generalization remains unestablished.

  • Problem

    The paper investigates how quantum-layer insertion or replacement affects VLM performance and cost, and how hybrid models compare in adversarial robustness.

    Source: §I · p. 2

  • Method

    The study classically simulates quantum circuits in a VLM and compares insertion and replacement configurations with classical baselines.

    Source: §I · p. 2, A. Quantum Circuits · p. 3, B. Technical Framework · p. 4

  • Results

    80.7% development accuracy for SE8 insertion at q2 exceeds classical full fine-tuning by 2.28 percentage points; BR4 replacement at r4 achieves 79.22% accuracy with approximately 13.07 million fewer trainable parameters.

    Source: B. Performance of Quantum Insertion… · p. 5

  • Takeaways & Limitations

    Quantum-layer benefits depend on placement, replacement choice, and training, and the study provides a basis for assessing when quantum integration is useful.

    Source: E. Discussion · p. 5, §VII · p. 6

    Findings are limited to one backbone, one dataset, and one seed, with development data used for checkpoint selection and reporting.

    Source: E. Discussion · p. 6

Abstract

from arXiv · show

As quantum computing moves from the noisy intermediate-scale quantum (NISQ) era toward early fault-tolerant quantum computing (EFTQC), quantum-enhanced AI is shifting from a conceptual idea to experimental research. Quantum components are expected to extend the capabilities of existing AI systems, yet it remains unclear how such components will help within large multimodal models. In this study, we try to answer that question for vision-language models (VLMs) using classically simulated quantum layers. With LLaVA-OneVision as the backbone, we evaluate 4- and 8-qubit basic-ring and strongly entangling circuits at four insertion and four replacement locations. Insertion adds quantum layers to the classical model, while replacement substitutes them for classical components. Using the same setup, we compare performance and computational cost between classical and quantum-classical hybrid models. On the Visual Spatial Reasoning dataset, inserting a quantum layer after the visual projector reaches 80.7% development accuracy, 2.28 percentage-points above classical full fine-tuning. Replacing the final decoder feed-forward sublayer retains 79.22% accuracy with 13.07 million fewer trainable parameters. Quantum insertion can thus improve accuracy, and selective replacement can reduce parameters. These benefits depend strongly on where the quantum layer is inserted or which classical component it replaces: earlier replacements lower accuracy, and fewer parameters do not reliably shorten training or reduce computation. To assess potential security benefits, we evaluated the robustness of quantum-classical hybrid models under six attack settings. Overall, our study maps where quantum layers help and hurt within VLMs, offering a starting point for hybrid VLM design; whether these gains generalize remains open. The code is available at https://github.com/TITANS-Lab-Artifacts/QuantumVLM.

I. INTRODUCTION

The paper asks whether quantum layers can improve VLM accuracy, computational cost, or adversarial robustness, motivated by VLMs’ high-stakes uses, resource demands, and vulnerabilities. It investigates these questions through classical simulations of quantum layers in LLaVA-OneVision.

I. INTRODUCTION

  • VLMs support image-and-text tasks such as VQA, but their high-stakes applications require reliability and robustness despite substantial computational demands and vulnerabilities to multiple attack families.The cited application areas include education, finance, and healthcare.Source: §I · p. 1
  • Prior work on quantum layers in vision and language tasks motivates testing whether they improve VLM performance and robustness rather than assuming such benefits.The paper notes prior work on quantum parameter adaptation, vision classification, language processing, and quantum-classifier adversarial learning.Source: §I · p. 1
  • The study evaluates whether quantum-layer insertion or replacement affects VLM performance, computational cost, and adversarial robustness.It uses LLaVA-OneVision’s Qwen2-0.5B single-image variant and four 4- or 8-qubit circuits.Source: §I · pp. 1–2
  • The study compares insertion and replacement configurations with classical baselines on the same dataset, assessing accuracy, trainable parameters, training time, and memory.It also tests robustness against six attacks under a common protocol.Source: §I · p. 2
  • The introduction frames three questions: how insertion affects performance and cost, whether replacement preserves performance while reducing cost, and how hybrids compare in robustness across attacker-access levels.The attack settings include different levels of attacker access.Source: §I · p. 2

II. BACKGROUND

The background outlines the LLaVA-OneVision components and how they process image and text inputs to produce a True-or-False prediction.

A. VLM Components

  • LLaVA-OneVision’s vision encoder extracts image features, its projector converts them for the language model, and decoder blocks process the combined image-text sequence before the output head predicts True or False.Attention connects information across tokens; feed-forward sublayers transform token features.Source: A. VLM Components · p. 2

B. Attacks on VLMs

The paper describes image-perturbation attacks spanning several levels of attacker access, then identifies gaps in prior quantum VLM research. It proposes a shared-backbone evaluation of insertion and replacement across performance, cost, and robustness.

B. Attacks on VLMs

  • The six attacks modify image pixels using target gradients, output scores, predicted labels, or a separate surrogate model.FGSM, PGD-CE, and PGD-margin use gradients; SPSA uses scores; HSJ uses labels; T-PGD uses a surrogate.Source: B. Attacks on VLMs · p. 2

III. LITERATURE REVIEW

  • Prior multimodal quantum studies examined fusion or adaptation, but did not test quantum-layer replacement in the same VLM under common adversarial attacks.The literature review also notes that prior work had not tested whether quantum methods reduce overall computational cost in VLMs.Source: §III · p. 2
  • Prior work on quantum classifiers and obfuscated gradients motivates direct robustness evaluation across gradient-based, query-based, and transfer attacks.How circuit design and placement affect robustness within the same VLM remained unclear.Source: §III · p. 2
  • The study addresses three gaps by testing four circuits at four insertion and four replacement locations in one LLaVA-OneVision backbone, comparing cost and accuracy with classical baselines and evaluating six attacks.The comparisons include trainable parameters, training time, and memory.Source: §III · p. 3

IV. PROPOSED FRAMEWORK

The framework places quantum layers in a LLaVA-OneVision backbone through insertion or replacement. Insertion locations span image features, projected features, intermediate decoding, and the final representation before prediction.

IV. PROPOSED FRAMEWORK

  • The proposed framework uses a LLaVA-OneVision backbone with quantum-layer insertion and replacement locations.The vision encoder uses SigLIP, and the decoder uses RMSNorm.Source: §IV · p. 3

A. Quantum Insertion Locations

  • Insertion adds a quantum layer without removing an existing component, with four locations ranging from before the visual projector to after final decoder normalization.The positions act on vision-encoder features, projected features, an early decoder representation, or the final prompt-token representation before prediction.Source: A. Quantum Insertion Locations · p. 3

B. Quantum Replacement Positions

Replacement swaps a classical component for a dimension-matched quantum module while leaving the rest of the model unchanged. The four tested positions range from the visual projector to decoder MLPs at three depths.

C. Quantum Replacement Positions

V. METHODOLOGY

The methodology evaluates four quantum-circuit configurations and integrates their outputs through insertion or replacement in a VLM. It uses simulated circuits and specifies different token scopes for the tested placements.

A. Quantum Circuits

  • The study tests basic-ring and strongly entangling circuits with 4 or 8 qubits.Basic-ring layers use trainable Y rotations and CNOT links between neighboring qubits in a closed ring; the strongly entangling circuit uses trainable Rot gates.Source: A. Quantum Circuits · p. 3

B. Technical Framework

  • For each selected token feature, the framework maps normalized features to rotation angles, applies trainable circuit layers, and measures each qubit’s Pauli-Z expectation.A linear output projection maps the measurements to the module output.Source: B. Technical Framework · p. 4
  • Insertion adds the projected quantum output to the feature through a trainable residual scale, whereas replacement uses that output as the component output.The insertion scale is initialized to 0.1; decoder residuals remain in place during replacement.Source: B. Technical Framework · p. 4
  • Insertion updates all retained visual tokens at q1 and q2 but only the final prompt token at q3 and q4; replacement uses exact differentiable state-vector simulation without shots or hardware noise.Source: B. Technical Framework · p. 4

C. Attack Setup

The attack setup perturbs images while holding text and model weights fixed, and evaluates attacks spanning gradient, score, label, and surrogate-model access. All attacks use the first 40 development examples and a shared perturbation budget.

C. Attack Setup

  • Attacks perturb images before preprocessing while text and model weights stay fixed; success means an incorrect prediction under the specified perturbation constraint.Source: C. Attack Setup · p. 4
  • All attacks use the first 40 development examples with ε = 4/255 and cover gradient access (FGSM, PGD), score or label access only (SPSA, HSJ), and no target-model access (transfer PGD).Source: C. Attack Setup · p. 4
  • FGSM uses one step, PGD-CE and PGD-margin use five random starts and 100 updates, SPSA estimates gradients from scores, HSJ uses predicted labels, and T-PGD transfers from a surrogate.The individual attack procedures specify their respective update and perturbation settings.Source: C. Attack Setup · p. 4

D. Evaluation Metrics

Evaluation reports classification and adversarial-robustness metrics alongside parameter, time, and memory costs. Training uses a random VSR split and compares several classical baselines with quantum configurations.

D. Evaluation Metrics

  • The evaluation reports clean accuracy, balanced accuracy, true recall, false recall, and robust accuracy for each attack and their union.Balanced accuracy weights both classes equally; balanced and robust accuracy are defined in Eq. (2).Source: D. Evaluation Metrics · p. 4
  • Resource comparisons use trainable parameters in millions, training time, and peak memory, with confidence intervals from 1,000 image-grouped bootstrap resamples.Source: D. Evaluation Metrics · p. 4

E. Training Setup

  • Training uses the VSR random split with 7,680 training and 1,097 development examples, comparing zero-shot, head-only, projector-only, LoRA, and full-fine-tuning baselines.Runs use one RTX PRO 6000 Blackwell GPU, seed 0, and up to 20 epochs with early stopping.Source: E. Training Setup · p. 4
  • LoRA uses rank 8, scaling 16, and dropout 0.05 on decoder query and value projections.Source: E. Training Setup · p. 4

VI. EXPERIMENTAL RESULTS

The section compares classical and quantum VLM baselines across performance, computational cost, and robustness; among the classical configurations reported here, LoRA achieves the highest accuracy with far fewer trainable parameters than full fine-tuning.

VI. EXPERIMENTAL RESULTS

  • The section compares classical baselines with quantum insertion and replacement models across predictive performance, computational cost, and adversarial robustness.Source: §VI · p. 4
  • Figure 4 reports accuracy across quantum insertion training modes, while Figure 5 reports accuracy under full-backbone training.Source: Figure 4 · p. 5, Figure 5 · p. 5

A. Performance of Classical Baselines

  • 79.0% accuracy for LoRA is the highest classical result, compared with 78.4% for full fine-tuning, while training approximately 0.54 million rather than 893.68 million parameters.Zero-shot and fully frozen evaluation achieve 54.2% accuracy, with True and False recall of 92.4% and 13.9%.Source: A. Performance of Classical Baselines · p. 5

B. Performance of Quantum Insertion and Replacement

Quantum insertion and replacement outcomes vary by configuration: one insertion setting exceeds the classical full-fine-tuning result, while final decoder replacement retains near-baseline accuracy. Training time and memory use do not show consistent resource savings.

B. Performance of Quantum Insertion and Replacement

  • 79.22% accuracy for BR4 at r4 retains accuracy in this run with approximately 13.07 million fewer trainable parameters; replacements at r1 or r2 reduce accuracy to approximately 51–55%.Intermediate replacement at r3 retains approximately 76–77%.Source: B. Performance of Quantum Insertion… · p. 5
  • 80.7% accuracy and balanced accuracy for full-fine-tuned SE8 at q2 exceed classical full fine-tuning accuracy by 2.28 percentage points.The reported result is configuration-specific; changing qubit count or circuit does not consistently improve accuracy at other positions.Source: Table III · p. 5, B. Performance of Quantum Insertion… · p. 5

C. Computational Cost

  • 3.22–7.20 hours for insertion and 3.24–7.81 hours for replacement, versus 6.60 hours classically, do not establish a training speedup because stopping budgets differed.Classical patience was 5, compared with 2 for full quantum runs.Source: Figure 6 · p. 5, C. Computational Cost · p. 5
  • Peak RAM usage ranges from 5.75 to 7.49 GiB for insertion and 4.92 to 8.14 GiB for replacement, compared with 6.29 GiB for the classical model.Source: C. Computational Cost · p. 5

D. Adversarial Robustness

Across the evaluated attacks, quantum hybrids do not provide consistent robustness gains, and clean-performance improvements do not imply stronger robustness. The authors limit their conclusions to one backbone, dataset, and seed, and call for further validation.

E. Discussion

  • Insertion benefits depend on placement and training mode, and neither insertion nor replacement provides consistent robustness gains under the evaluated attacks.The final decoder feed-forward sublayer can be replaced while preserving accuracy in this run, whereas projector and early decoder replacements perform poorly.Source: E. Discussion · pp. 5–6

D. Adversarial Robustness

  • 0% robust accuracy under PGD-CE, PGD-margin, and the six-attack union applies to every model; classical models outperform SE8 at q2 under FGSM and transfer PGD.Classical robust accuracy is 50.0% under FGSM and 25.0% under transfer PGD, versus 40.0% and 17.5% for SE8 at q2; HSJ results warrant caution given initialization failures and limited queries.Source: D. Adversarial Robustness · p. 5, Table IV · p. 6

E. Discussion

  • The conclusions are limited to one backbone, one dataset, and one seed; development data supports both checkpoint selection and reporting, with the best configuration selected from many candidates.The authors identify held-out evaluation, multiple seeds, and parameter-matched classical controls as needed to confirm gains and isolate the circuit's contribution.Source: E. Discussion · p. 6

VII. CONCLUSION

  • The study presents a classical-simulation framework for evaluating quantum-layer insertion and replacement, but establishing quantum advantage requires evidence beyond these comparisons.Future work will evaluate multiple seeds and held-out data, use parameter-matched classical controls and matched training budgets, and strengthen robustness validation.Source: §VII · p. 6

Related papers

Loading 2610.28964v1…