Quantum Vision Transformers

1 Introduction

• For visual recognition, attention can capture more global correlations among image patches than convolutional neural networks without attention [9].The transformer learns coefficients per patch that weight attention to the rest of the image.Source: §1 · p. 2
• The proposed circuits use amplitude encoding and Hamming-weight-preserving gates, which the authors contrast with variational approaches lacking scalability proofs, [21].The authors describe their linear-algebraic tools as making the circuits more NISQ-friendly, with scalability in runtime and model parameters.Source: §1 · p. 2
• The proposed quantum attention approaches are described as adaptable to other applications, including natural language processing.The authors note that transformers have been proven to be particularly efficient in that domain.Source: §1 · pp. 2–3
• Attention coefficients weight each image patch against the others, and the attention layer’s trainable matrices are V and W.The paper focuses its quantum circuits on this attention mechanism.Source: §1 · p. 3
• The paper proposes three quantum transformer types: self-only Orthogonal Patchwise, classical-like Orthogonal, and quantum-native Compound Transformers, with quantum attention complexity lower than classical counterparts.The architectures cover training and inference, with a combined circuit for inference after attention coefficients are trained.Source: §1 · p. 3
• The Compound Transformer loads image patches into uniform superposition, then uses a unitary to multiply the input by a trainable second-order compound matrix.Its operation uses a quantum-native linear algebraic operation on higher-dimensional compound matrices.Source: §1 · p. 3

2 Quantum Tools

• The RBS gate can be implemented natively as FSIM or decomposed into four Hadamard gates, two Ry rotations, and two two-qubit CZ gates.Source: §2 · p. 4

2.1 Quantum Data Loaders for Matrices1

• Each row of a matrix is loaded in superposition using (n-1)+(2n-1)(d-1) gates.The loader extends unary amplitude encoding to load a matrix X ∈ R^n×d as a quantum state.Source: §2.1 · p. 4

2.2 Quantum Orthogonal Layers2

• A quantum orthogonal layer applies a circuit to unary-basis |x⟩, with its restricted unitary defining an orthogonal matrix V that produces |Vx⟩.Source: §2.2 · p. 4
• The new butterfly and X circuits trade resources and expressivity: butterfly has logarithmic depth and higher expressivity, while X uses fewer gates but is less expressive.The butterfly circuit requires gates on all possible qubit pairs; X maintains a path from every input qubit to every output qubit.Source: §2.2 · pp. 4–5
• For Hamming-weight-k inputs, a quantum orthogonal layer outputs the kth-order compound-matrix transform V(k)x rather than Vx.The circuits preserve Hamming weight, so the output remains encoded in the same basis.Source: §2.2 · p. 5
• Although cited work supports trainability and suggests these layers avoid barren plateaus, the existence and implications of exponential local minima remain open.Source: §2.2 · p. 5

3 Quantum Transformers

• Classical attention computes patch-pair coefficients from x_i^T W x_j, then applies a nonlinearity; the proposed quantum approaches differ in how they implement this process.Their theoretical resource comparison includes trainable and fixed parameters, circuit depth, and the number of circuits run in parallel.Source: §3 · pp. 5–6

3.1 Orthogonal Patch-wise Neural Network

• The Orthogonal Patch-wise Network applies the same trainable V to each patch but uses trivial attention, so each patch attends only to itself.A vector loader and quantum orthogonal layer produce Vx_i; tomography retrieves an output whose state size is linear in the qubit count.Source: §3.1 · p. 6, Figure 10 · p. 6

3.2 Quantum Orthogonal Transformer

• The Orthogonal Transformer separately computes squared patch-pair overlaps with W and transformed patches with V, then combines them classically to form each output.The squared-overlap coefficients are positive, with softmax applied classically when needed; the same W is used across patch pairs.Source: §3.2 · p. 6

3.3 Direct Quantum Attention

• Direct Quantum Attention uses trained attention coefficients and V in one circuit to compute each output patch, requiring fewer circuits but more qubits and greater depth than the preceding method.Its matrix loader weights the loaded patches by the corresponding attention coefficients before V is applied.Source: §3.3 · p. 7

3.4 Quantum Compound Transformer

• The Compound Transformer applies one orthogonal layer to all patches, using a second-order compound matrix to combine feature extraction and global patch reweighting.It replaces the separate V and W operations with one circuit that generates all outputs; the authors report a polynomial speedup with respect to n for the compound-matrix operation.Source: §3.4 · pp. 8–9

4.1 Simulation Setting

• The simulations trained three quantum architectures and two baselines on each of 12 MedMNIST datasets, using the same validation method as the dataset benchmarks.Inputs were preprocessed into 16 patches of size 7 × 7, mapped to 16-dimensional features, and passed through four attention layers.Source: §4.1 · p. 9, §4 · p. 9

4.2 Simulation Results

• The Orthogonal Transformer and Compound Transformer outperformed benchmark methods on 7 of 12 MedMNIST datasets; all three attention architectures beat the non-attention quantum baselines on all 12.The reported metrics are AUC and accuracy; overall, quantum transformers reached comparable accuracy to classical equivalents.Source: §4.2 · pp. 9–10, Table 3 · p. 10
• The Compound Transformer uses 80 trainable attention-layer parameters, compared with 512 (2d2) for the Classical Vision Transformer.This resource analysis excludes preprocessing, other transformer-layer components, and the final classification layer; the paper reports fewer parameters overall for quantum transformers.Source: §4.2 · p. 10

5 Conclusion

• The proposed models span trivial attention, a classical-like Orthogonal Transformer, and a Compound Transformer using a quantum-native compound-matrix operation.The compound-matrix multiplication is described as not efficiently achievable classically.Source: §5 · p. 10
• The authors characterize the evidence as preliminary: experiments are limited, and practical advantages depend on future hardware improvements such as noise management and clock speed.They call for further empirical validation using future quantum hardware.Source: §5 · pp. 10–11
• Quantum circuits classified small MedMNIST images as well as or sometimes better than state-of-the-art classical methods while using fewer parameters.The authors say this may help address over-fitting, but call for further validation as quantum computers scale.Source: §5 · p. 11
• The theoretical runtime advantage is difficult to observe on current quantum computers because of limited size, noise, and cloud-access latency.Hardware results become too noisy as qubit count or circuit size increases.Source: §5 · p. 11

A Vision Transformers

• The classical Vision Transformer divides images into patches, processes them through repeated transformer layers, and classifies using the final class-embedding vector.The class embedding is added during preprocessing and its final representation feeds a fully connected classification layer.Source: A Vision Transformers · p. 15

B.1 Quantum Data Loaders for Matrices

• Amplitude encoding stores classical data components as quantum-state amplitudes, and the figure compares parallel, diagonal, and semi-diagonal vector loaders.For d = 8, their respective depths are log(d), d, and d/2.Source: §B.1 · p. 16, Figure 14 · p. 16
• For a matrix, the loader places row norms in a top register and loads the corresponding rows into a lower register, producing a superposition of rows.The construction uses vector loaders, their adjoints, and CNOTs controlled by the top register.Source: §B.1 · p. 16
• Applying an inverse loader after a regular loader makes the probability of measuring the designated state equal to the squared inner product of the encoded vectors.The passage notes that techniques for retrieving the inner-product sign have been developed [5].Source: §B.1 · p. 16

B.2 Quantum Orthogonal Layers

• RBS-gate circuits preserve Hamming weight, so their action on a restricted basis is represented by a smaller real orthogonal matrix.For unary encoding, the reduced matrix is N × N rather than the full 2^N × 2^N unitary.Source: §B.2 · p. 16
• The pyramid circuit uses adjacent-qubit connectivity and, with a final Z gate, can represent all orthogonal matrices of size N × N.The pyramid layout uses N(N −1)/2 RBS gates.Source: §B.2 · pp. 16–17
• The paper introduces butterfly and X circuits; butterfly-layer backpropagation has O(N log N) complexity rather than the pyramid circuit's O(N^2).The same backpropagation method applies to the new circuits, with runtime corresponding to their gate counts.Source: §B.2 · p. 17

C.1 Datasets

• MedMNIST provides 12 preprocessed, two-dimensional medical-image datasets, each with 28 × 28 pixel images; hardware experiments use RetinaMNIST.The collection was selected for its accessible size for circuit simulations and hardware experiments and its medical-image classification scope.Source: §C.1 · p. 17

C.2 Simulations

• Simulations on the 2D MedMNIST datasets show quantum attention architectures reaching accuracy comparable to and sometimes better than standard classical models.The section compares results with those reported in the AutoML benchmark.Source: §C.2 · p. 17

C.2.1 Simulation setting MedMNIST

• The five benchmark networks were trained with cross-entropy and Adam for 100 epochs, using batch size 32 and learning rate 10^-3 with scheduled decay.The learning rate was decayed by a factor of 0.1 after epochs 50 and 75; the best result across three random seeds was selected for each model.Source: §C.2.1 · p. 17

C.2.2 Simulation results MedMNIST

• All quantum neural networks were competitive with the AutoML benchmark and outperformed their classical counterparts on 7 of 12 MedMNIST datasets.Quantum Orthogonal and Compound Transformers outperformed the Orthogonal Fully-Connected and Orthogonal Patch-wise networks most of the time.Source: §C.2.2 · pp. 17–18
• The reported totals were 128 parameters for Orthogonal Patch-wise, 256 for Quantum Orthogonal Transformer, and 320 for Compound Transformer; the classical Vision Transformer requires 512 per attention layer.The quantum totals are model counts; the classical value is for an attention layer, and the resource analysis excludes preprocessing and other shared layers.Source: §C.2.2 · p. 18
• Some classical MedMNIST models reached somewhat better accuracy, but the smallest reported residual network had approximately 10^7 parameters.The passage also notes that automated machine-learning algorithms train numerous architectures to reach their performance.Source: §C.2.2 · p. 18

• RetinaMNIST has 1,080 training, 120 validation, and 400 test images, each with 28×28 RGB pixels and one of five ordinal-regression classes.Source: §C.3 · p. 18

Hardware Description

• Experiments used the 16-qubit ibmq guadalupe and 27-qubit ibm hanoi, with meaningful results reported only for four, five, and six qubits.Higher-qubit experiments, requiring more gates and circuit depth, did not produce meaningful results.Source: §C.3.1 · p. 18, Figure 15 · p. 18, Figure 16 · p. 18
• Device noise and finite-sampling noise are identified as the main noise sources in the hardware computations.Source: §C.3.1 · p. 18
• Noise may help escape local minima or act as data augmentation, but accuracy drops when noise is too large.The passage relates this possibility to neural networks and notes that classical deep learning sometimes adds noise artificially for these purposes.Source: §C.3.1 · p. 19

Hardware Results

• Three quantum proposals were tested with four, five, and six qubits; models were trained in a JAX-based simulator and inferred on all 400 RetinaMNIST test images using IBM hardware.The six-qubit compound setup used 32,000 shots, while other four-qubit configurations used 10,000 shots.Source: §C.3.2 · p. 19
• The Orthogonal Patch-wise network used 16 patches and a four-qubit pyramid, and its quantum-native method showed a small accuracy advantage over an equivalent classical patch-wise network.The experiment used 16 circuits of nine RBS gates per circuit per image.Source: §C.3.2 · p. 19
• The Quantum Orthogonal Transformer used four patches and trainable attention coefficients, while the quantum-attention Orthogonal Transformer attended only to neighboring patches; the last two proposals had accuracy similar to a classical transformer.The former used four-qubit pyramids; the latter used a five-qubit circuit with an X orthogonal layer.Source: §C.3.2 · p. 19
• The six-qubit Compound Transformer was run on ibm hanoi with an X layout, then a reduced “Circuit” orthogonal layer retaining one diagonal of RBS gates to reduce noise in outcomes.This experiment used two circuits of 18 RBS gates and three CNOTs per circuit per image.Source: §C.3.2 · p. 19
• Hardware results used error mitigation by discarding measurements whose Hamming weight differed from the expected unary-state or weight-two superpositions.The procedure was applied at every layer and described as efficient for the hardware demonstration.Source: §C.3.2 · p. 19
• All quantum proposals achieved state-of-the-art test accuracy comparable to classical networks, while the compound transformer occasionally outperformed classical transformers in simulation, where its classical implementation incurs polynomial overhead.Source: §C.3.2 · p. 19

Extended Performance Analysis

• Table 6 extends the MedMNIST performance analysis with AUC and ACC for each test dataset, using benchmark results reported in.Source: D Extended Performance Analysis · p. 19, Table 6 · p. 20
• The ResNet benchmarks distinguish 28×28 from 224×224 inputs, with the larger-resolution ResNet having more parameters.Source: Table 6 · p. 20