PaperlayerView exact revision on arXiv

Source-linked AI summary

The Quantum Shortcut: Complex Phase-State Dynamics Reduce the Optimization Steps of Sequence Models

Ahmed Nebli, Hadi Saadatdoorabi, Christopher Keibel, Kevin Yam

arXiv:2608.14691v1cs.LGquant-ph

TL;DR

The paper asks whether replacing real-valued hidden states with a complex phase-based substrate can accelerate training. It instantiates this substrate in Mamba and Transformer models and finds faster optimization, with sustained gains for state-space models but diminishing gains for attention.

  • Problem

    The paper asks whether a complex phase-based substrate also trains faster, because optimization steps determine training computation and data consumed to reach a target quality.

  • Method

    The study instantiates a deployable complex substrate with gated phase-based recurrences and applies it to state-space and attention-based sequence models.

  • Results

    Approximately one third as many steps for state-space models and one half as many for attention models reached every measured validation loss across three byte-level corpora; the state-space gap widened after warmup.

  • Takeaways & Limitations

    Complex substrates accelerate early optimization in both backbones, while the advantage persists and widens for state-space models but decays toward zero for attention models.

  • Takeaways & Limitations

    Each experimental cell is a single training run without error bars, and the authors treat attention margins as potentially seed-level until replicated.

Abstract

from arXiv · show

Sequence models are conventionally distinguished by their backbone, the mechanism that routes information across positions, such as attention or recurrence. This paper varies a choice that is prior to the backbone and shared by nearly all current models: the \emph{substrate}, the number system in which the hidden state is represented together with the form of the map from state to prediction. The prevailing substrate is a real-valued state with an affine--softmax readout; we study a complex-valued alternative drawn from the mathematics of quantum theory, in which information is carried by the phases of the state and scores are quadratic Born forms. Prior work proved an idealized version of this substrate representationally stronger than any real model with a linear readout; we ask whether it also trains faster. Relaxing the two properties that block deployment, exact unitarity and the Born vocabulary readout, we instantiate it in the Mamba state-space model and an attention-based Transformer. At 253M parameters, matched to within $0.02\%$ and trained under one fixed protocol on three byte-level corpora, the complex models reach every measured validation loss in approximately one third (state-space) and one half (attention) of the optimization steps of their real counterparts. The two backbones then diverge. Once the learning-rate warmup ends, the state-space advantage continues to widen, from $0.321$ to $0.354$ bits per character on OpenWebText and from $0.368$ to $0.396$ on FineWeb, which an artifact of the warmup ramp would not do; the attention advantage instead decays toward zero on every corpus, and is therefore an effect of early training.

1 Introduction

The paper treats the hidden-state number system and readout as a substrate axis distinct from the backbone, motivated by phase-based forgetting, interference, and limitations of real affine–softmax models. A deployable complex substrate relaxes ideal unitarity and Born vocabulary readout, yet reaches validation targets in substantially fewer optimization steps.

  • Substrate as a design axis: The substrate comprises the hidden-state field and the map from state to next-token distribution, separate from the model backbone.The prevailing substrate uses a real-valued state with an affine transformation followed by softmax.
  • Motivation: Complex states store information in phase, allowing interference between fixed-magnitude components and changing how forgetting affects retained information.Real attenuation shrinks state entries and can rapidly degrade information over long sequences.
  • Motivation: Real affine–softmax readouts have log-probability rank at most d + 1, while real recurrences attenuate propagated gradients by ρ^T over T steps.These limitations are independent of the backbone and motivate measuring optimization steps to reach target loss.
  • Deployable substrate: The deployable complex substrate replaces exact unitarity with contractive, input-dependent recurrence and applies Born scoring in a linear-attention kernel while retaining a tied softmax vocabulary readout.The idealized version used unit-norm states, exactly unitary evolution, and a quadratic Born-rule output distribution.
  • Empirical result: 253M parameters and one training protocol yielded validation targets in approximately one third of the optimization steps of real Mamba and one half of real Transformer across three byte-level corpora.The corpora span a 150× range of size, and the paper names this reduction the quantum shortcut.

2 Preliminaries and Notation

This section establishes the complex-state substrate through Hilbert-space states, density matrices, unitary evolution, and quadratic Born-rule measurements. It also fixes notation used throughout the paper.

  • Complex states and the density matrix: A complex state |ψ⟩ has density matrix ρ=|ψ⟩⟨ψ|, whose diagonal entries encode occupation magnitudes and off-diagonals encode relative phases.Quadratic functionals can read the off-diagonal phase information, unlike linear functionals of the state.
  • Unitary evolution and the Cayley transform: A unitary operator preserves inner products and norms, while the Cayley transform parameterizes unitary evolution using skew-Hermitian matrices.With A=−iHΔt, the transform corresponds to Crank–Nicolson discretization and preserves norms exactly for any step size.
  • Born-rule measurement: The Born rule assigns outcome v probability p_v=|⟨e_v|ψ⟩|^2, making measurement a real quadratic form of the state and a linear functional of its density matrix.This quadratic-versus-linear distinction underlies the representational separation developed later.
  • Notation: The section also defines σ as the logistic function, ⊙ as the Hadamard product, tr as the trace, and T, N, H, and V as sequence, state-width, head-count, and vocabulary-size notation.These conventions are used throughout the paper’s subsequent substrate and model descriptions.

3 The Wave-Function Substrate and Its Idealization

The idealized wave-function substrate combines unitary complex-state dynamics with a quadratic Born readout, and its representational guarantee motivates the deployable substrate as a controlled relaxation. Its key mechanism is interference between state components under later inputs.

  • From idealization to deployment: The deployable substrate is a controlled relaxation of the idealized model, whose exact form and guarantee provide the reference for those relaxations.The paper states the idealized substrate and its guarantee while referring to prior work for constructions and proofs.
  • Idealized substrate: The idealized model uses a unit-norm complex state, Hamiltonian-driven evolution, exact unitary updates, and a quadratic Born readout.The Cayley discretization preserves ∥ψ_t∥ = 1 at every step and for every sequence length.
  • Representational guarantee: Born readouts span an N^2-dimensional real family of Hermitian measurement functionals, compared with 2N dimensions for linear state readouts.The guarantee concerns the interaction between the complex state and quadratic readout, not norm preservation alone.
  • Interference: Interference is the mechanism formalized by the guarantee: state components encode competing readings, and later inputs favor one reading over another.The example concerns disambiguating an occurrence of “bank” using a later token.

4 Related Work

The paper’s substrate combines ingredients developed across state-space, complex/unitary, linear-attention, structure-preserving numerical, oscillatory, and quantum-inspired modeling. Its unresolved contribution is to test whether this deployable combination changes optimization speed inside production backbones.

  • State-space models: State-space models evolved from HiPPO’s memory-preserving recurrences through S4’s trainable convolutional systems to Mamba’s input-dependent selective recurrence and hardware-aware scan.The paper’s recurrence belongs to this family but uses a diagonal, input-dependent linear recurrence evaluated by the same associative scan in Mamba’s block.
  • Complex and unitary networks: Complex and unitary networks established stable deep training and gradient conditioning, but exact unitarity prevents forgetting and limited prior validation mainly to memory-oriented benchmarks.The paper relaxes this constraint to study complex dynamics in competitive language-modeling backbones.
  • Attention alternatives: Linear-attention and rotary-attention work supplies learned decay gates and phase-based information storage, while differing in their nonnegative feature-map accumulation, fixed positional frequencies, and linear inner-product scores.These threads provide separate halves of the paper’s attention realization rather than the full substrate.
  • Dynamical systems and numerics: Oscillatory and structure-preserving dynamics motivate phase-based memory and integrator-aware stability, while Crank–Nicolson preserves the unitary structure of the continuous flow exactly at every step size.The paper contrasts complex phase storage with real oscillators, whose position–velocity coordinates couple information and decay under damping.
  • Quantum-inspired modeling and output expressivity: Quantum-inspired language models introduced complex states and Born-type readouts, but the training question—whether a production-compatible substrate changes learning speed—remained open.The paper supplies this training-side investigation at 253M parameters, while related output-head work addresses rank limitations of affine–softmax readouts.

5 The Deployable Complex Substrate

The deployable substrate uses complex phase states with gated diagonal recurrences, relaxing exact unitarity while preserving phase transport and selective forgetting. It applies Born-style quadratic scoring where inexpensive and retains a conventional tied-softmax vocabulary head.

  • Gated phase recurrence: The complex recurrence separates a decaying magnitude channel from a phase channel transported without distortion, placing carried information in the preserved phase.The recurrence remains contractive in magnitude, while phase information arrives undistorted and its observable strength depends on accumulated magnitudes.
  • Efficient evaluation: The diagonal recurrence is evaluated exactly by a chunked scan in O(N) arithmetic per step with O(log C) sequential depth.Diagonal structure avoids the O(N^2) cost of a dense transition, while complex arithmetic adds only a constant factor to the O(N) term.
  • Readout: The Born score is used as a positive semidefinite quadratic attention kernel, while both realizations retain a conventional tied-softmax vocabulary head.The quadratic score is nonnegative, normalizable without exponentials, and linear in the density matrix.
  • Readout: Relative phase can reduce a fixed-magnitude interaction from 1 at θ = 0 to 0 at θ = π, enabling phase opposition that real readouts cannot represent.The recurrence controls this relative phase through the Cayley phase.
  • Complex state: Each head carries a width-N = 16 complex state stored as 2N real numbers and initialized from a unit-modulus complex prior.The complex initialization prevents interference cross terms from vanishing at the first step.
  • Gated phase recurrence: The recurrence multiplies a unit-modulus Cayley phase, which carries information in the state argument, by a sigmoid gate that controls forgetting.The gate relaxes exact unitarity and restores selective, input-dependent forgetting.

6 Theoretical Analysis

Theoretical analysis identifies two candidate mechanisms: affine–softmax readouts impose rank-dependent optimization floors, while complex recurrences transport relative phase additively without distortion. Both results are delimited and do not derive the measured step ratios, instead motivating substrate design through reachable-set and retention mechanisms.

  • Readout rank: An affine–softmax head constrains centered logits to rank at most d + 1, creating a strictly positive excess-loss floor when rank(eL⋆) > d + 1.The floor vanishes otherwise and applies after any number of gradient steps.
  • Readout rank: At the trained vocabulary head, d = dmodel = 1024 exceeds the byte vocabulary’s centered target rank bound V − 1 = 255, so the floor is inactive.The analysis instead identifies narrow heads and attention scoring as settings where rank constraints may matter.
  • Phase transport: For an isolated homogeneous component, the recurrence accumulates phase additively and preserves phase sensitivity at ∂θs,j = 1, independently of horizon and gates.Magnitude derivatives retain the accumulated contraction, whereas phase transport is gate-independent.
  • Phase transport: The phase lemma applies to separate input contributions, not their aggregate, whose phase can change nonlinearly through interference.The state-to-state Jacobian still has gate singular values, so the result is decoupling rather than unconditional losslessness or a gradient guarantee.
  • Theoretical contribution: Together, the results motivate a substrate whose score is not confined by state width and whose information coordinate is distinct from the coordinate attenuated by contraction.The paper explicitly states that these theoretical results establish candidate mechanisms rather than explain the measured optimization-step ratios.

7 Experimental Setup

The experiments compare real and complex versions of Mamba and Transformer under one byte-level training protocol, controlling parameter count, tuning, corpus, and backbone. Shared budgets, reproducible logs, and matched counterparts isolate substrate effects while defining bits per character as the evaluation metric.

  • Models: Four models pair real Mamba with complex H-Mamba and real Transformer with complex H-Transformer, with the complex Transformer replacing softmax attention by a sum-normalized Born score.The attention pair therefore differs in the functional form and normalization of its score.
  • Data: 256 byte values form the vocabulary, and performance is reported in bits per character (bpc), with lower values better.Byte-level modeling removes tokenizer variation as a confound.
  • Optimization: One unchanged protocol uses AdamW, a cosine learning-rate schedule from 6 × 10^-4 to 6 × 10^-5, 750 warmup steps, unit-norm gradient clipping, bf16, and activation checkpointing.Applying the same schedule to all four models prevents protocol choices from favoring one substrate.
  • Budget: 1000 steps are allocated on OpenWebText and FineWeb, versus 1500 on enwik8; comparisons ordinarily use the common 1000-step budget.Beyond roughly 1000 steps, complex models begin overfitting enwik8's 100 MB corpus, with validation loss climbing toward 2.2–2.3 bpc by step 1500.
  • Runs, controls, and reproducibility: 0.02% is the maximum parameter-count mismatch between counterparts, with twelve runs spanning four models and three corpora.Training logs and the parser used for tables and figures are released, making each quantity re-derivable.

8 Results

Across three corpora and two backbones, complex models reach matched validation losses in fewer optimization steps than real counterparts, with stable backbone-specific speedups. After warmup, the state-space advantage widens while the attention advantage contracts toward zero, though complex models generally retain better terminal loss within the budget.

  • Per-step convergence: The complex model requires fewer steps than its real counterpart in all 18 validation-target comparisons.The comparison uses interpolated steps to reach common validation losses.
  • Post-warmup behavior: +0.321 to +0.354 bpc on OpenWebText and +0.368 to +0.396 on FineWeb describe the state-space gap widening after warmup.The post-warmup interval is 250 steps and lies in the cosine-decay phase, so this evidence excludes an artifact of the warmup ramp, not every schedule interaction.
  • Post-warmup behavior: +0.026 and +0.034 bpc are the attention gaps at step 1000 on the two web corpora, with the trend extrapolating toward zero.The attention advantage contracts on all three corpora on both sides of the warmup boundary and is therefore concentrated in early training.
  • Terminal quality and data limit: H-Mamba improves terminal loss over real Mamba by 0.315–0.396 bpc across the three corpora, while H-Transformer improves on both web corpora but trails on enwik8.The H-Mamba gains correspond to a 20–24% reduction in per-byte perplexity; on enwik8, the complex models later overfit the finite corpus.

9 Discussion

The complex substrate reduces optimization steps across both backbones, with the larger state-space gain attributed to conditioning and the attention gain combining conditioning with a richer Born score. The discussion also identifies training-cost implications, scaling uncertainties, and experiments needed to isolate the mechanisms and test generalization.

  • 9.1 Effect: Approximately threefold for state-space and twofold for attention, the complex substrate reduces steps to every measured target across corpora spanning a 150× size range.The discussion attributes the effect to replacing the real substrate with a deployable complex one.
  • 9.2 Mechanisms: Threefold acceleration in H-Mamba is attributed to conditioning, while H-Transformer’s twofold acceleration can additionally use the richer Born interaction rule.H-Mamba retains its baseline’s tied softmax readout, whereas H-Transformer uses a genuinely richer Born score.
  • 9.5 Open tests: A factorial design over real or complex recurrence and affine–softmax or Born score is required to isolate conditioning from score rank, while the effect should also be tested on a third backbone and longer sequences.The proposed tests include complex long-convolution or gated linear attention and measuring whether the state-space ratio grows with sequence length.
  • 9.3 Training cost: Approximately 0.57B tokens are saved reaching 2.0 bpc on FineWeb, with H-Mamba using approximately 0.23B tokens versus approximately 0.79B for real Mamba.Each training step processes 1,048,576 tokens, so fixed-batch step reductions translate directly into token savings.
  • 9.3 Training cost: At 80,000 versus 160,000 tokens per second, attention throughput offsets the twofold step ratio, yielding data efficiency at wall-clock parity.The state-space comparison requires an optimized real-baseline kernel that was unavailable in the setup.
  • 9.4 Scaling: The measured advantage is expected to persist at subword vocabularies because it originates in recurrence and attention score mechanisms rather than substrate-specific dependence on vocabulary size, but this remains untested.The idealized readout instead scales unfavorably at 30k to 128k-token vocabularies, requiring state width of order V and readout cost O(NV), or surrendering exact rank advantage to softmax.

10 Limitations

The conclusions are limited by single-run experiments, non-identical parameter-matched models, one size and budget regime, and incomplete convergence. Interpretation is also bounded by theory–implementation gaps, while complex arithmetic currently imposes throughput and wall-clock costs.

  • Measurement: Every experimental cell uses a single training run, so the study reports no error bars or direct estimate of seed variance.The authors calibrate plausible seed variance against the smallest observed effect.
  • Measurement: The paired models are parameter-matched but functionally non-identical, differing in the real Mamba’s SwiGLU branch and the attention score’s functional form and normalization.Per-layer gradient plots are supporting diagnostics for conditioning, not direct measurements of long-range gradient transport.
  • Measurement: 253M parameters and 1000 to 1500 steps define the sole size and budget regime, which remains far from convergence on the web corpora.The reported conclusion therefore concerns the measured loss range and persistence of per-step advantage under these budgets and models.
  • Interpretation: The deployable substrate relaxes both properties required by Nebli et al.’s representational theory, so the Ω(N 2) separation applies to the idealized model rather than the trained models.Theorem 6.1 bounds the readout in isolation under an assumption about training probabilities and does not quantify the full effect.
  • Cost and evaluation scope: Complex arithmetic decomposes into real operations on current accelerators, and fused Born-attention reaches half the throughput of its real counterpart.Consequently, the attention pair’s step advantage does not yet produce wall-clock savings.

11 Conclusion

The paper extends prior representational results for complex substrates by showing a corresponding training advantage. Its deployable substrate reaches measured validation losses substantially faster than real Mamba and attention-based models.

  • Conclusion: Complex substrates were previously shown to be representationally more powerful than real substrates on idealized synthetic tasks.The prior result concerned an idealized model class rather than deployment-oriented training.
  • Conclusion: This paper establishes the corresponding claim about training efficiency.The conclusion frames the contribution as extending the prior representational result to optimization behavior.
  • Conclusion: Approximately one third of real Mamba’s optimization steps sufficed for the deployable complex substrate to reach every measured validation loss.The deployed form is contractive rather than unitary, with the Born rule confined to the attention score.
  • Conclusion: Approximately one half of real attention-based models’ optimization steps sufficed for the deployable complex substrate to reach every measured validation loss.The passage contrasts the complex model’s optimization speed with both real Mamba and real attention-based models.

A Substrate Pseudocode

This section gives pseudocode for implementing the complex state-space recurrence and attention score, including a diagonal Cayley transition and a chunked parallel scan. The transition uses unit-modulus phase factors with selective gates, while the scan accumulates magnitudes and phases across chunks.

  • Algorithms 1–2 implement the recurrence without sequential evaluation, while Algorithm 3 implements the attention score.
  • The diagonal Cayley transition maps phase parameters to unit-modulus coefficients and combines them with sigmoid selective gates.The coefficient is λ ← (1 + iϕ) ⊘ (1 − iϕ), with |λ_k| = 1, and the returned transition has |a_k| = α_k.
  • The chunked parallel scan accumulates log magnitudes and phases, constructs polar transition products, and carries the final state to the next chunk.It clamps cumulative log magnitudes to [−30, 0] before forming the polar products.

B Per-Step Arithmetic Accounting

The diagonal Cayley recurrence requires O(N) arithmetic per step per head, while causal Born attention uses squared-modulus scores and causal running-sum normalization. Although the recurrence has an approximately 3× arithmetic ratio at N = 16 versus a dense real transition, memory bandwidth, complex overhead, and shared projection costs prevent that ratio from directly determining wall-clock time.

  • Diagonal Cayley recurrence: O(N) arithmetic per step per head: forming the transition coefficient and updating the state each require about 6N real operations.Together, these recurrence operations cost roughly 12N real operations.
  • Causal Born attention: Causal Born attention computes scores from squared moduli of query-key inner products rather than softmax probabilities.The score uses |⟨q_i | k_j⟩|^2/(D τ_i) for j ≤ i.
  • Causal Born attention: Causal Born attention applies running-sum normalization and returns results through a fused kernel with single-precision accumulation.The algorithm specifies causal running-sum normalization before returning ψ.
  • Diagonal Cayley recurrence: At N = 16, the diagonal Cayley recurrence has an approximately 3× arithmetic ratio relative to a dense real transition.The passage contrasts this arithmetic ratio with observed wall-clock time.
  • Diagonal Cayley recurrence: Memory bandwidth, complex arithmetic overhead, and shared projection costs prevent the approximately 3× arithmetic ratio from being realized directly in wall-clock time.At N = 16, the recurrence kernel is memory-bandwidth-bound because the state resides in registers; projection costs cancel in the comparison.

C Training Loss

Training-loss curves mirror validation results: complex realizations lead by the same per-step margins and separate at the same early optimization step. This indicates the acceleration reflects learning rather than a validation-split artifact.

  • Figure 7 compares training loss against optimization step for complex realizations and the real Transformer.
  • The complex realizations lead on training loss by the same per-step margins observed on validation, with separation emerging at the same early step.
  • The matching training and validation orderings indicate that the acceleration is a property of learning, not an artifact of the validation split.

D Reproducibility

The experiments use a fixed byte-level data partitioning and evaluation procedure, with detailed loss logging and released training logs supporting reproducibility.

  • Data and evaluation: Each corpus is partitioned 90/5/5 into training, validation, and test sets after being read as raw bytes.Inputs are segmented into non-overlapping windows of 1024 bytes, with the target advanced by one byte.
  • Data and evaluation: Bits per character is defined as cross-entropy in nats divided by ln 2.
  • Logging and release: Training loss is recorded every 10 steps, while validation loss is recorded every 20 steps over 20 validation batches and the full validation split at completion.All twelve training logs are released.
Loading 2608.14691v1…