PaperlayerView exact revision on arXiv

Source-linked AI summary

Wave Is All You Need: A Quantum Language Model Based on Fully Connected Quantum Boltzmann Machine

Jun Liu

arXiv:2610.34492v1quant-phcond-mat.stat-mechcs.CLcs.LG

TL;DR

Classical self-attention uses discrete matrix operations that can disrupt continuous semantic representations and incur substantial computational overhead. This paper develops FC-QBM self-attention and an FC-QBM language model, using unitary evolution to encode positions and token interactions. In experiments, the model retains over 99% of noiseless convergence performance under noise and shows preliminary capabilities in five-character quatrain generation.

  • Problem

    Classical self-attention relies on discrete matrix operations that disrupt continuous semantic representations and incur substantial computational overhead.

    Source: §I · pp. 2–3

  • Method

    The paper builds an FC-QBM self-attention mechanism and language model, using unitary evolution of semantic quantum states to encode positions and token interactions.

    Source: §I · p. 5, C. Equivalence Between Fully Connected… · p. 15

  • Results

    Over 99% of noiseless convergence performance is retained under noise, and five-character quatrain experiments preliminarily show associative and divergent thinking, imagery generation, and human-aligned word ordering.

    Source: B. Noise Robustness Experiments · p. 38, §VI · pp. 41–42

  • Takeaways & Limitations

    The FC-QBM mechanism is mathematically equivalent to classical self-attention and provides a basis for the model’s preliminary poetry-generation capabilities.

    Source: §VI · pp. 41–42

Abstract

from arXiv · show

This work proposes a self-attention mechanism based on the fully connected quantum Boltzmann machine (FC-QBM), and further constructs a fully connected quantum Boltzmann machine language model (FC-QBM-LM) using the mechanism as the core engine. By performing unitary evolution on semantic quantum states, the mechanism implicitly implements positional encoding and self-attention computation, eliminating the need to decompose continuous semantic streams into discrete matrices for explicit matrix multiplication, with the measurement results naturally satisfying the normalization condition. Specifically, this work rigorously derives the computational principle of the self-attention mechanism based on FC-QBM, and demonstrates that this mechanism is mathematically equivalent to the classical self-attention mechanism. The mathematical principles governing the training and generation of FC-QBM-LM are thoroughly derived, and the two corresponding algorithms are subsequently developed. An 8-qubit FC-QBM-LM is finally constructed and trained on the corpus of Tang poetry. Under the 8-qubit setting, the quantum layer implements the entire positional encoding and self-attention computation pipeline with only 44 parameters. In the noiseless scenario, the model exhibits favorable convergence. Furthermore, the quantum circuit proposed in this work, which terminates with Y-axis rotation gates, achieves better convergence than the counterpart terminating with X-axis rotation gates. Under the small-sample regime and across eight distinct noise intensity levels, the model demonstrates strong noise robustness, retaining over 99% of its noiseless convergence performance. Experiments on five-character quatrain generation reveal that the model preliminarily exhibits three core capabilities: associative and divergent thinking, content generation across imagery and human-aligned word ordering.

I. INTRODUCTION

Classical self-attention models global token dependencies but relies on discrete matrix operations that disrupt continuous semantic representations and incur high computational cost. The paper proposes FC-QBM self-attention as a functionally equivalent alternative that encodes positions and token interactions through quantum-state evolution.

I. INTRODUCTION

  • The paper proposes an FC-QBM self-attention mechanism and quantum language model, deriving the mechanism and evaluating an eight-qubit model on Three Hundred Tang Poems.The evaluation covers convergence, noise robustness, and five-character quatrain generation.Source: §I · p. 5

A. Review of Classical Self-Attention Mechanism

  • Classical self-attention maps token embeddings to query, key, and value matrices, then uses scaled, softmax-normalized scores to combine value vectors into contextualized outputs.The output represents positional information and pairwise token interactions.Source: A. Review of Classical Self-Attention… · pp. 6–8
  • Classical attention’s discrete matrix operations disrupt continuous semantic structure and incur computational and storage costs that grow as O(n^2d_model).The introduction also notes quadratic sequence-length costs and resource demands for classical attention.Source: §I · p. 3, A. Review of Classical Self-Attention… · pp. 9–10
  • The proposed architecture is mathematically functionally equivalent to classical self-attention and is expected to reduce resource consumption through quantum parallelism and entanglement.The resource-reduction claim is presented as an expectation.Source: A. Review of Classical Self-Attention… · p. 10
  • FC-QBM implicitly encodes token positions and pairwise interactions through unitary evolution of semantic quantum states, without decomposing continuous semantic streams into discrete matrices.Measurement probabilities naturally satisfy normalization, removing the need for softmax.Source: A. Review of Classical Self-Attention… · p. 10

B. Computational Principle of Self-Attention Mechanism Based on Fully Connected Quantum Boltzmann Machine

The FC-QBM attention computation begins with an amplitude-encoded token sequence and applies a Hamiltonian derived from a fully connected Boltzmann machine. Unitary evolution and final qubit rotations produce measurement probabilities that encode positional and interaction information.

B. Computational Principle of Self-Attention Mechanism Based on Fully Connected Quantum Boltzmann Machine

  • The token sequence is amplitude-encoded as a real vector whose entries prepare the initial quantum state.The vector has dimension 2^N for an N-qubit circuit.Source: B. Computational Principle of… · p. 11
  • The circuit Hamiltonian represents an FC-QBM with trainable unit biases and pairwise connection weights.Its classical counterpart is an energy function over binary unit values.Source: B. Computational Principle of… · p. 11
  • Unitary evolution under the Hamiltonian, followed by Y-axis rotations and projective measurement, yields a probability vector over the N-qubit basis states.The derivation obtains the probability for each basis outcome and assembles those values into the output vector.Source: B. Computational Principle of… · pp. 12, 14

C. Equivalence Between Fully Connected Quantum Boltzmann Machine Self-Attention Mechanism and Classical Self-Attention Mechanism

FC-QBM phases encode token order and pairwise interactions, and subsequent RY rotations produce measurement probabilities equivalent to classical self-attention outputs. The construction uses basis-vector amplitudes to represent token embeddings and yields normalized probabilities without a separate softmax step.

C. Equivalence Between Fully Connected Quantum Boltzmann Machine Self-Attention Mechanism and Classical Self-Attention Mechanism

  • Each token embedding is represented using d_model basis-vector amplitudes, and the resulting measurement probabilities are equivalent to classical self-attention results.The probabilities encode token positional and interaction information.Source: C. Equivalence Between Fully Connected… · pp. 14–15
  • FC-QBM bias phases encode token order, while phases from two-qubit interactions encode pairwise token interactions.RY rotations then yield measurement probabilities carrying positional and interaction information.Source: C. Equivalence Between Fully Connected… · p. 15
  • Unitary evolution implicitly computes attention without discrete-matrix decomposition, and measurement returns a naturally normalized probability distribution that requires no separate softmax.The resulting probability vector can be applied to downstream generation and recognition tasks.Source: C. Equivalence Between Fully Connected… · p. 16

QUANTUM BOLTZMANN MACHINE LANGUAGE MODEL

The model prepares an amplitude-encoded quantum state from the ordered token embeddings. It determines a qubit capacity sufficient for the encoded vector and uses the resulting state as the circuit input.

A. Initial State Preparation and Quantum Circuit Construction

  • Token embeddings are concatenated in sequence order, shifted to nonnegative values, L1-normalized, and square-rooted before amplitude encoding.The resulting vector preserves one embedding’s elements for each token.Source: A. Initial State Preparation and… · p. 17
  • The qubit count is chosen so the circuit’s 2^N-dimensional Hilbert space can hold the encoded vector, with extra basis amplitudes padded with zeros.The required qubit count is the minimum that satisfies the capacity constraint.Source: A. Initial State Preparation and… · p. 18
  • The vector is assigned to a as the token sequence’s amplitude encoding, preparing the initial state |ψ0⟩ for the N-qubit circuit.The circuit uses FC-QBM gates RZ(2bi), RZ(2wij), and CNOT, plus trainable Y-axis rotations RY(2θi).Source: A. Initial State Preparation and… · p. 18

B. Construction of Classical Neural Network Layers

The quantum circuit’s measurement probabilities feed a classical neural network that compresses them and predicts the next token embedding.

B. Construction of Classical Neural Network Layers

C. Model Training

FC-QBM-LM trains its quantum and classical parameter groups iteratively using backpropagation and an embedding-prediction loss. The output linear layer’s parameters are updated from loss gradients.

C. Model Training

  • The model has quantum-circuit parameters and parameters for each of its two classical linear layers, updated through iterative optimization using the corpus and pretrained embeddings.The parameter groups include b_i, w_ij, θ_i, the first-layer weights and biases, and the second-layer weights and biases.Source: C. Model Training · p. 20
  • Forward propagation predicts the next-token embedding, backpropagation computes parameter gradients, and mean squared error measures its difference from the ground-truth embedding.This procedure is used to simulate positive-phase energy minimization and negative-phase contrastive-divergence learning.Source: C. Model Training · p. 20

1. Parameter Updates for Second Classical Linear Layer

  • Gradients for the second classical linear layer’s weights and biases are computed from the loss and used to update those parameters.The update uses the second layer’s learning rate η2.Source: §1 · p. 21

2. Parameter Updates for First Classical Linear Layer

The training derivation updates the first classical layer and quantum layer from loss gradients. Quantum-layer gradients use parameter-shift estimates based on measurement outcomes.

2. Parameter Updates for First Classical Linear Layer

  • The first classical linear layer’s weights and biases are updated using loss gradients and its learning rate η1.Source: §2 · pp. 21–22

3. Parameter Updates for Quantum Layer

  • Quantum-layer parameters b, w, and θ affect the final state and measurement probabilities, and their gradients are approximated with parameter shifts of +π/4 and −π/4.Measured outcome frequencies approximate the basis-state probabilities used in the gradient computation.Source: §3 · pp. 23–24

D. Generation Process

Generation predicts an embedding for the next token, scores candidate tokens against it, and samples from a filtered set before repeating autoregressively.

D. Generation Process

  • A forward pass produces the next-token embedding, which is compared with token vectors to compute similarity scores.The scores are scaled, ranked, and used to select the top-m candidates.Source: D. Generation Process · p. 25
  • Tokens failing the filtering rules are removed, then softmax converts the remaining candidate scores into sampling probabilities.A token is sampled from this distribution for the next position.Source: D. Generation Process · pp. 25–26
  • The generated token is appended to the sequence, and the same procedure repeats autoregressively until termination.The basic flow retrieves the token closest to the predicted embedding.Source: D. Generation Process · pp. 25–26

E. Algorithm

The section presents training and generation algorithms for an 8-qubit FC-QBM language model, whose quantum circuit implements positional encoding and self-attention.

E. Algorithm

LANGUAGE MODEL

  • The model processes 10 tokens as a 250-dimensional input, pads it to 256 dimensions, and encodes it in an 8-qubit circuit.The Tang-poetry embedding matrix has vocabulary size 2,481 and embedding dimension 25.Source: LANGUAGE MODEL · pp. 28–29
  • The quantum layer implements positional encoding and self-attention with 44 parameters, while the full model has 36,165 trainable parameters.The two classical linear layers account for 36,121 parameters.Source: LANGUAGE MODEL · p. 31
  • Training iterates over the Tang Poems corpus, using forward propagation and backpropagation to update parameters until each epoch is complete.The quantum layer outputs a probability vector, and the classical layers use it to predict the next-token embedding.Source: LANGUAGE MODEL · p. 32
  • Generation retrieves the token closest to the predicted embedding, appends it to the sequence, and repeats until termination.The embedding matrix is trained on Three Hundred Tang Poems.Source: LANGUAGE MODEL · pp. 32–33

V. EXPERIMENTAL RESULTS AND ANALYSIS

The experiments evaluate convergence, noise robustness, and five-character quatrain generation; the reported convergence comparison favors terminal Y-axis rotation gates over X-axis gates.

V. EXPERIMENTAL RESULTS AND ANALYSIS

  • The evaluation plan includes model convergence, noise robustness, and five-character quatrain generation experiments.The experiments use PennyLane and PyTorch.Source: §V · p. 34

A. Model Convergence Experiments

  • The convergence experiment compares otherwise identical circuits that differ only in their terminal rotation gates.The proposed circuit uses Y-axis gates; the control uses X-axis gates.Source: A. Model Convergence Experiments · p. 34
  • The experiments use a PennyLane state-vector simulator and evaluate convergence with mean MSE across token samples for each epoch.Training runs for 30 epochs; each epoch takes about 4.5 hours on average.Source: A. Model Convergence Experiments · p. 34
  • Under noiseless conditions, both circuits show favorable convergence, and the terminal-RY circuit converges better than the terminal-RX counterpart.The comparison uses epoch-wise mean MSE curves.Source: Figure 4 · p. 35, A. Model Convergence Experiments · p. 35

B. Noise Robustness Experiments

The experiments report noise robustness under a small-sample regime and preliminary capabilities in five-character quatrain generation. The conclusion relates these findings to an FC-QBM self-attention mechanism that encodes positions and token interactions.

B. Noise Robustness Experiments

  • Noise tests used 501 tokens from the first eight poems across 30 batches and nine scenarios, from noiseless through 8× baseline noise.The baseline was 0.5% noise for single-qubit gates and 1.0% for two-qubit gates.Source: B. Noise Robustness Experiments · pp. 35–36
  • Under the small-sample regime, the model retained over 99% of noiseless convergence performance across noise intensities.The reported robustness result is preliminary and limited to small-scale experiments; verification on real quantum devices is left for future work.Source: Figure 5 · p. 37, Table II · p. 38, B. Noise Robustness Experiments · p. 38

C. Five-Character Quatrain Generation Experiments

  • In five-character quatrain generation, the model showed associative and divergent links, human-aligned word ordering, and improving poetic qualities across epochs.Coherence, fluency, antithesis, and imagery rendering improved with training; the authors describe the model as preliminary and report a notable gap from professional large language models.Source: C. Five-Character Quatrain Generation… · pp. 39–41

VI. CONCLUSION

  • FC-QBM encodes token positions and interactions through unitary evolution, mathematically matching classical self-attention with 44 parameters in the 8-qubit layer.Measurement results naturally satisfy normalization without softmax computation.Source: §VI · p. 41
  • Under noiseless conditions, the model converged favorably, and the circuit terminating with RY gates converged better than the RX-terminated counterpart.The conclusion also reports over 99% retention of noiseless convergence performance in the small-sample noise experiments.Source: §VI · p. 41

Related papers

Loading 2610.34492v1…