Source-linked AI summary
QTrans: A Quantum Transformer for Sentiment Classification
Ren-Xin Zhao, Xinjie Huang, Yahong Liu, Maoyu Ye, Jinjing Shi, Shi Wang, Yaonan Wang
TL;DR
Lightweight models may struggle to capture how contextual cues such as negation and contrast jointly determine sentiment. QTrans uses quantum circuits for attention features and combines quantum attention with a quantum feed-forward network; it outperforms classical baselines across MR, CR, and MPQA.
Problem
Lightweight models trained on small datasets may rely on isolated lexical cues and underrepresent relationships among sentiment-bearing tokens, including negation and contrast.
Source: §I · p. 1Method
QTrans uses independent parameterized quantum circuits to generate query, key, and value features, Gaussian distances for attention, and a quantum feed-forward network with residual connections and layer normalization.
Source: §I · pp. 1–2Results
Across MR, CR, and MPQA, QTrans exceeded the strongest parameter-matched classical baseline by 2.88, 3.17, and 3.79 percentage points, with test accuracies of 72.13%, 69.51%, and 63.45%, respectively.
Source: §V · p. 10Takeaways & Limitations
Ablations show quantum multi-head self-attention improves accuracy on all three datasets, while the quantum feed-forward network helps on MR and MPQA but decreases accuracy on CR.
Source: 1) Ablation Analysis · p. 7
Abstract
from arXiv · showhide
In small-scale binary sentiment classification scenarios, factors such as negation, contrastive shifts, and cross-word dependencies lead to the non-linear coupling of sentiment cues, making it difficult for conventional lightweight models to fully capture the contextual relationships between tokens. To address this issue, we propose a model named QTrans, which uses parameterized quantum circuits to construct query, key, and value features and derives attention coefficients from Gaussian distances between quantum measurements. By further integrating a quantum feed-forward neural network, residual connections, and layer normalization, the model establishes an end-to-end trainable quantum-classical hybrid framework for sentiment classification. Experimental results on the MR, CR, and MPQA datasets show that QTrans achieves test accuracies of 72.13\%, 69.51\%, and 63.45\%, respectively, representing improvements of 2.88, 3.17, and 3.79 percentage points over the best-performing classical baselines for each dataset. Overall, QTrans expands the application of parameterized quantum circuits in lightweight sentiment analysis and lays an experimental foundation for further research into quantum multi-head self-attention for modeling textual relationships.
I. INTRODUCTION
Lightweight models trained on small sentiment datasets may miss how contextual cues jointly determine polarity. QTrans replaces conventional attention projections with quantum circuits and reports higher accuracy than classical baselines on three datasets.
I. INTRODUCTION
- Lightweight models on small datasets may over-rely on isolated lexical cues, mishandling negation, contrast, and distant modifiers that jointly determine sentiment.Such errors can reduce accuracy and weaken generalization in domain-specific settings where labeled data are difficult to obtain.Source: §I · p. 1
A. Classical Self-Attention
- Classical self-attention connects neighboring and distant tokens, but linear query, key, and value projections may limit contextual representations, while scaling the model risks overfitting small datasets.This motivates strengthening projection representations rather than relying only on a larger classical Transformer.Source: §I · p. 1, A. Classical Self-Attention · p. 2
I. INTRODUCTION
- QTrans uses independent parameterized circuits for quantum query, key, and value features, Gaussian relevance coefficients, and a quantum feed-forward network with residual connections and layer normalization.The proposed attention divides measured value features between two heads.Source: §I · pp. 1–2
- 72.13% on MR, 69.51% on CR, and 63.45% on MPQA exceed the strongest classical baselines by 2.88, 3.17, and 3.79 percentage points, respectively.The evaluated QTrans instance uses four qubits.Source: §I · p. 2
B. Fundamentals of Quantum Computing
The section introduces qubits, quantum gates, entanglement, and measurement as foundations for quantum computation. It also identifies the gates and observable used in QTrans.
B. Fundamentals of Quantum Computing
- A qubit is the elementary carrier of quantum information, and an n-qubit state may be entangled rather than decomposable into individual qubit states.The section presents computational basis states and complex probability amplitudes as part of the pure-state representation.Source: B. Fundamentals of Quantum Computing · p. 2
- Quantum states evolve through unitary gates, including rotations and CNOT operations that can introduce multi-qubit correlations.The CNOT matrix uses the ordered computational basis {|00⟩, |01⟩, |10⟩, |11⟩}, with the first qubit as control.Source: B. Fundamentals of Quantum Computing · p. 2
- Measurement of a Hermitian observable produces a real-valued expectation value determined by the quantum state and selected observable.In QTrans, Pauli-Z is measured at the circuit output; other listed gates support state preparation, encoding, trainable rotations, or entanglement.Source: B. Fundamentals of Quantum Computing · p. 2, Table I · p. 3
C. Parameterized Quantum Circuits
A parameterized quantum circuit maps classical inputs to measured outputs through input encoding and trainable quantum operations. Its parameters can be optimized for a learning objective.
C. Parameterized Quantum Circuits
- A parameterized quantum circuit is a trainable unitary process controlled by classical parameters.Its general construction takes an input vector and a parameter vector.Source: C. Parameterized Quantum Circuits · p. 3
- The circuit combines an input-dependent encoding operator with a trainable variational unitary.These operators act on a state prepared from the classical input and circuit parameters.Source: C. Parameterized Quantum Circuits · p. 3
- Measuring an observable defines a parameter-dependent real-valued mapping from classical input, whose parameters can be optimized for a learning objective.The mapping depends jointly on the encoding operator, variational unitary, and measured observable.Source: C. Parameterized Quantum Circuits · p. 3
III. QUANTUM TRANSFORMER FOR SENTIMENT CLASSIFICATION
QTrans combines sequence embeddings, quantum multi-head self-attention, a quantum feed-forward network, and sentiment readout. Its framework uses residual paths and normalization while classifying from the terminal [SEP] representation.
III. QUANTUM TRANSFORMER FOR SENTIMENT CLASSIFICATION
- QTrans has four stages: sequence embedding, quantum multi-head self-attention, a quantum feed-forward network, and sentiment readout.The attention uses Gaussian similarities between measured quantum query and key features to quantify token relevance.Source: §III · p. 3, Figure 1 · p. 4
A. Overall Framework
- Token and positional embeddings form the input sequence, with embedding width matched to the circuit's qubit count; the experiments use four qubits.The sentiment label is the sequence-level training target and is not included in the input embedding matrix.Source: A. Overall Framework · p. 3
- The causal-padding mask lets each valid position attend only to itself and earlier valid positions, enabling terminal [SEP] to aggregate the sentence for classification.The sequence is truncated or padded to fixed length L before processing.Source: A. Overall Framework · p. 3
- After attention, a residual addition and layer normalization precede the quantum feed-forward network, whose output is added to the retained representation through a second residual path.The terminal [SEP] representation maps to positive and negative logits, and softmax produces output probabilities.Source: A. Overall Framework · pp. 3–4, Figure 1 · p. 4
B. Quantum Multi-Head Self-Attention
QTrans replaces scaled dot-product attention with Gaussian similarity over quantum-measured query and key features, while measured value features supply the content to aggregate. Two measurement heads are fused classically, with a residual connection applied outside the attention calculation.
B. Quantum Multi-Head Self-Attention
- Gaussian similarity between measured query and key features determines token relevance, while independently parameterized query, key, and value circuits supply attention features and content.The measured value features are aggregated using the normalized attention coefficients.Source: B. Quantum Multi-Head Self-Attention · p. 4
- Token embeddings are bounded as rotation angles, then encoded through a Hadamard layer and data-dependent RY rotations before trainable RX-RY and repeated CNOT-ring layers.The experiments use n = 4 qubits and Dq = 2 repeated blocks.Source: B. Quantum Multi-Head Self-Attention · p. 4, Figure 2 · p. 5
- Each of two heads uses scalar query and key measurements and an n/2-dimensional value vector formed by splitting the even-width measured value output.The query and key branches use Z1 and Z2 measurements, while the value branch retains all n measurements.Source: B. Quantum Multi-Head Self-Attention · p. 5
- Masked Gaussian coefficients normalize over admissible keys, assigning future and padded positions exactly zero attention weight before value aggregation.The normalized coefficients are equivalent to a masked softmax over negative squared query-key distances.Source: B. Quantum Multi-Head Self-Attention · p. 5
- The two head outputs are concatenated and mixed by a small classical matrix that preserves residual width, with the residual connection applied outside attention.The same circuit topology serves the four branches, but their trainable parameters are independent.Source: B. Quantum Multi-Head Self-Attention · pp. 4–5
C. Quantum Feed-Forward Neural Network
After quantum attention establishes token context, QTrans applies an independent quantum feed-forward branch position by position. Its nonlinear quantum feature map and classical transformations return each token to the residual width before classification.
C. Quantum Feed-Forward Neural Network
- The quantum feed-forward network transforms each token independently after quantum attention and uses parameters independent of the query, key, and value circuits.It applies no second token-to-token attention matrix.Source: C. Quantum Feed-Forward Neural Network · p. 5
- A quantum projection supplies a nonlinear, entanglement-dependent feature map, GELU adds a classical activation, and a final linear mapping restores n-dimensional residual width.The measured quantum outputs are collected as G ∈ R^(L×n).Source: C. Quantum Feed-Forward Neural Network · p. 5
- The feed-forward output enters a residual addition before the terminal [SEP] representation is classified.With n = 4 and Dq = 2, the four quantum branches contain 64 quantum parameters, jointly optimized with the classical components and embeddings.Source: C. Quantum Feed-Forward Neural Network · p. 6
IV. EXPERIMENTS
The experiments assess QTrans in binary sentiment classification through component substitutions, comparisons with lightweight classical models, and input-noise tests.
IV. EXPERIMENTS
- Separate experiments replace quantum multi-head self-attention and the quantum feed-forward network with classical counterparts to examine the contributions of both quantum components.Source: §IV · p. 6
- The study compares QTrans with Tiny Transformer, BiLSTM-Attention, and TextCNN using identical data partitions and nearly equal parameter scales.The comparison evaluates classification performance.Source: §IV · p. 6
- Input sensitivity is tested by injecting Gaussian noise with standard deviations 0.1, 0.3, and 0.5 into token embeddings.Source: §IV · p. 6
A. Datasets
The evaluation uses three binary sentiment datasets spanning movie reviews, product reviews, and general opinion expressions. Samples are deduplicated, stratified, and balanced across train, validation, and test splits.
A. Datasets
- MR covers sentence-level movie-review polarity, CR customer evaluations of products, and MPQA contextual-polarity opinion expressions.Together, the datasets represent three sentiment scenarios with different linguistic characteristics.Source: A. Datasets · p. 6
- After duplicate removal, samples are stratified into 70% / 15% / 15% training, validation, and test splits, with positive and negative examples downsampled to a 1:1 ratio within each split.The procedure prevents samples from crossing split boundaries.Source: A. Datasets · p. 6
B. Experimental Settings
The section fixes a four-qubit, two-head QTrans setup and reports controlled component swaps: quantum attention improves accuracy across all three datasets, while quantum feed-forward effects vary by dataset.
B. Experimental Settings
- The instantiated circuit uses n = 4, entangling depth Dq = 2, and two attention heads.The configuration is identified in Table III.Source: B. Experimental Settings · p. 6, Table III · p. 6
- The cited table reports dataset statistics after strict class balancing, without specifying the balancing ratio, split proportions, duplicate filtering, or resulting dataset sizes in this passage.Source: Table II · p. 6
- QTrans has 24382 trainable parameters, with less than 1% parameter difference from every baseline.The comparison uses four-dimensional token embeddings for the classical baselines to keep parameter scales nearly equal.Source: B. Experimental Settings · p. 6, Table IV · p. 7
1) Ablation Analysis:
- Attention replacement raises test accuracy from 69.4% to 72.1% on MR, 68.0% to 69.5% on CR, and 56.7% to 63.4% on MPQA.The results are validation-selected test accuracies from the component-wise ablation.Source: Figure 3 · p. 7, 1) Ablation Analysis · p. 7
- Quantum feed-forward replacement changes accuracy from 70.5% to 72.1% on MR, 70.2% to 69.5% on CR, and 62.3% to 63.4% on MPQA.These changes correspond to gains of 1.6 points on MR and 1.1 on MPQA, but a 0.7-point decrease on CR.Source: Figure 3 · p. 7, 1) Ablation Analysis · p. 7
- The ablation comparisons change one component at a time, keep remaining components and settings unchanged, and use checkpoints selected by lowest validation cross-entropy.The ablation and model-comparison experiments were trained independently, so absolute checkpoint values are only compared within each experiment.Source: 1) Ablation Analysis · p. 7
- Quantum self-attention improves accuracy across all three datasets, whereas the quantum feed-forward replacement has dataset-dependent results.The attention replacement is more consistent; the feed-forward replacement improves MR and MPQA but decreases CR accuracy.Source: 1) Ablation Analysis · p. 7
2) Performance Beyond Lightweight Classical Models:
QTrans outperforms the lightweight classical models in accuracy and macro-F1 across MR, CR, and MPQA. Its embedding-noise evaluation shows comparatively modest changes at low noise and degradation under stronger perturbations, while not testing quantum hardware noise.
2) Performance Beyond Lightweight Classical Models:
- QTrans ranks first in macro-averaged F1 on all three datasets, with strict 1 : 1 class balancing supporting the conclusion that its accuracy gains do not merely favor one sentiment class.The comparison uses validation-selected checkpoints.Source: 2) Performance Beyond Lightweight… · pp. 7–8
- Across 120 epochs, all models’ training losses generally decrease while several test losses rise slightly; reported checkpoints are selected by validation loss.Fig. 4 displays accuracy and cross-entropy loss curves, with dots marking validation-selected checkpoints.Source: Figure 4 · p. 8, 2) Performance Beyond Lightweight… · p. 8
- QTrans achieved test accuracies of 72.13%, 69.51%, and 63.45% on MR, CR, and MPQA, respectively, exceeding each dataset’s strongest classical baseline by 2.88, 3.17, and 3.79 percentage points.The comparison involved three lightweight classical models.Source: 2) Performance Beyond Lightweight… · p. 7
3) Robustness to Embedding Noise:
- At σ = 0.1, test accuracies change only moderately; the 0.1-percentage-point MR increase is within graphical and experimental fluctuation, not a deterministic noise benefit.The reported comparison evaluates σ values of 0, 0.1, 0.3, and 0.5.Source: 3) Robustness to Embedding Noise · pp. 8–9
- At σ = 0.3 and 0.5, MR and CR accuracies decrease visibly as losses increase, while MPQA accuracy varies comparatively little across noise levels.MPQA’s noiseless baseline is lower than those of MR and CR.Source: 3) Robustness to Embedding Noise · p. 9
- The noise experiment tests sensitivity to perturbed classical representations, not gate errors, decoherence, or readout errors on quantum hardware.The authors identify noisy simulators and real quantum devices as settings for separate investigation.Source: 3) Robustness to Embedding Noise · p. 9
V. CONCLUSION
QTrans combines quantum attention and a quantum feed-forward network in a trainable quantum-classical architecture for small-scale binary sentiment classification. It exceeds parameter-matched classical baselines across three datasets, while the fixed-seed, small-dataset, simulator-based evaluation leaves broader validation and hardware noise for future study.
V. CONCLUSION
- Independent parameterized quantum circuits generate query, key, and value measurements, and Gaussian-projected coefficients determine token relevance.A position-wise quantum feed-forward network refines contextualized representations.Source: §V · pp. 9–10
- The quantum attention and feed-forward components are integrated with residual connections and layer normalization in an end-to-end trainable quantum-classical architecture.The architecture uses a terminal [SEP] readout.Source: §V · p. 10
- 72.13%, 69.51%, and 63.45% test accuracy on MR, CR, and MPQA exceed the strongest parameter-matched classical baseline by 2.88, 3.17, and 3.79 percentage points, respectively.Macro-averaged F1 results show the same ranking.Source: §V · p. 10
- Evaluation is limited by fixed-seed testing, small datasets, simulator-based implementation, and the absence of quantum hardware noise.The authors propose testing repeated seeds, larger and more diverse corpora, alternative measurement-head configurations, and realistic hardware noise.Source: §V · p. 10