PaperlayerView exact revision on arXiv

Source-linked AI summary

Circuit Hypernetworks for Quantum-Augmented Diffusion Language Models

Xiaoqiang Wang, Mengyang Xiong, Jun Dai, Bang Liu

arXiv:2609.24657v1quant-phcs.CL

TL;DR

Evaluating wider quantum circuits inside language models is computationally demanding, and prior integrations have not established whether greater circuit width improves language-model quality. HyperQ emits token-conditioned circuits in residual branches of a frozen masked-diffusion model; across six benchmarks, its average score rises from 47.65 at 16 qubits to 54.30 at 64 qubits.

  • Problem

    Prior quantum integrations evaluated components at fixed resource levels, leaving unresolved whether increasing circuit width improves language-model quality while keeping training and evaluation practical.

    Source: §1 · pp. 2–3

  • Method

    HyperQ uses per-block hypernetworks to emit token-specific coordinates on a shared fixed-degree IQP circuit skeleton and adds projected readouts to a frozen masked-diffusion backbone.

    Source: §1 · p. 3, §4.2 · p. 19

  • Results

    54.30 at 64 qubits exceeds the frozen backbone (49.59) and its low-rank adapter (50.63) by 4.71 and 3.67 points, respectively.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The findings support token-conditioned circuit emission as a tractable architectural approach to diffusion language modelling without classically hard readouts.

    Source: §1 · pp. 4–5

    The constant derivative variance characterizes circuit-coordinate gradients at initialisation, not the frozen backbone or the complete optimisation trajectory, and does not guarantee trainability in general.

    Source: §4.3 · p. 22

Abstract

from arXiv · show

Language models can be adapted by changing the computations applied to individual tokens. Quantum circuits offer one such approach, but evaluating wider circuits inside a large model can be computationally demanding. Here we introduce HyperQ, which adds token-conditioned quantum residual branches to a frozen masked-diffusion language model. A quantum residual branch is a module in each transformer block that reads a token's hidden state, emits the coordinates of that token's circuit, executes it, and adds the measured values back through a residual connection. The backbone remains frozen, and only the added branches are trained. Within each branch, a lightweight circuit hypernetwork emits token-specific rotation angles, coupling strengths, and measurement axes in a shared sparse circuit structure. The required expectation values have an exact classical expression whose evaluation cost grows linearly with the qubit count, enabling circuits from 16 to 64 qubits to be trained within a 1.1-billion-parameter backbone. Across downstream benchmarks, increasing circuit width raises the average score from 47.65 to 54.30. At 64 qubits, HyperQ exceeds the backbone and its low-rank-adapted counterpart by 4.71 and 3.67 points, respectively. HyperQ is fine-tuned on 20,000 prompt-response pairs, compared with 200,000 for the classical baselines. These findings support token-conditioned circuit emission as a tractable architectural approach to quantum-augmented language modelling.

1 Introduction

The section asks whether token-conditioned quantum circuits can improve language modelling as circuit width increases, while remaining trainable and practical to evaluate. HyperQ emits circuits from token representations in a frozen masked-diffusion model, and its six-benchmark average rises with tested width.

1 Introduction

  • Prior quantum integrations largely evaluate components at fixed resource levels, leaving open whether increasing qubit count improves language-model quality.A width study also faces trainability concerns from vanishing gradients and the cost of exact statevector simulation inside a large model.Source: §1 · p. 2
  • Token-level emission must adapt to each contextual representation while preserving usable gradients and computationally practical readouts as width increases.Existing circuit-search and sampling approaches select discrete structures or produce circuits for a fixed task rather than emitting one within each token computation.Source: §1 · p. 3
  • HyperQ replaces task-level circuit selection with a per-token branch that emits circuit coordinates from hidden states and adds projected readouts to query, key and value tensors.The masked-diffusion backbone stays frozen; only the added branches and their low-rank projections are trained.Source: §1 · pp. 3–4
  • The emitted circuits use a two-body IQP family on a fixed ring-and-chord skeleton, whose degree-four connectivity gives Θ(n) readout cost and reference derivative variance 1/48 across tested widths.The variance claim is specific to the stated circuit family and independent reference initialisation.Source: §1 · p. 4
  • 54.30 at 64 qubits exceeds the frozen backbone (49.59) and its low-rank adapter (50.63) by 4.71 and 3.67 points, respectively.Across 16, 32 and 64 qubits, the six-benchmark average is 47.65, 52.80 and 54.30; HyperQ uses 20,000 prompt-response pairs versus 200,000 for classical baselines.Source: §1 · p. 4, Figure 1 · p. 5
  • The results support token-conditioned circuit emission as a tractable architectural approach to diffusion language modelling without relying on classically hard readouts.The paper explicitly makes no claim of quantum advantage.Source: §1 · pp. 4–5

2 Results

HyperQ adds token-conditioned quantum residual branches to a frozen masked-diffusion model, with continuous circuit emission outperforming fixed and searched circuits as width increases. The results also examine token-specific circuit supports, linear-cost readout, hardware evaluation, and decoding behavior.

2.1 HyperQ quantum residual architecture

  • Each of 22 transformer blocks maps each token’s hidden state to circuit coordinates, then adds projected measurement readouts to the classical query, key, and value tensors.The shared ring-plus-chord edge set has 2n edges, with four couplings per qubit; the model retains iterative masked denoising.Source: §2.1 · pp. 5–6

2.2 Classical baseline comparison

  • 47.65 at 16 qubits exceeds Llama-2-7B at 47.20, and HyperQ reports the standard six-benchmark suite for direct comparison with classical baselines.A same-backbone low-rank adapter raises LLaDA from 49.59 to 50.63, while 64-qubit HyperQ exceeds the adapted backbone by 3.67 points.Source: §2.2 · pp. 7–8

2.3 Per-token circuit emission

  • Both automatic circuit routes outperform fixed hand-designed families at every width, while the emitted circuit’s margin over the best fixed ansatz grows from 1.78 points at 16 qubits to 3.68 at 64.At 32 qubits, the hypernetwork reaches 52.80 in one pass versus 52.24 for the best three-circuit ensemble.Source: §2.3 · pp. 8–9, Table 3 · p. 9
  • 54.30 at 64 qubits exceeds the best fixed ansatz by 3.68 points and the 3-motif IQP search by 2.35 points; emission also leads at 16 and 32 qubits.The corresponding hypernetwork averages are 47.65 at 16 qubits and 52.80 at 32 qubits.Source: §2.3 · p. 9

2.4 Emitted circuit motifs

  • At 16 qubits, 55.8% of tokens retain both coupling layers, while 23.2% retain only the ring; supports vary by token within a shared edge-set skeleton.The branch averages 2.58 active couplings per qubit, and only 0.1% of tokens retain every edge at 64 qubits.Source: §2.4 · p. 10

2.5 Circuit synthesis and compilation

  • The strict-IQP block’s expectation values have Θ(n) evaluation cost, allowing 16-, 32-, and 64-qubit circuits without constructing a full 2^n statevector.The same budgeted circuits are used for ablation and on-device runs.Source: §2.5 · p. 12

2.6 Quantum register scaling

  • The emitted circuit gains 1.50 points from 32 to 64 qubits, while the best fixed ansatz gains 0.56; on hardware, mean losses are 0.34, 0.68, and 1.12 points at 16, 32, and 64 qubits.The bounded hardware evaluation uses paired 256-item subsets and does not test native hardware training or establish quantum advantage.Source: §2.6 · pp. 12, 15, 16

2.7 Any-order text decoding

  • HyperQ maintains generation quality across tested decoding orders, and degrades more slowly than its backbone as more tokens are committed per forward pass.Confidence-ordered unmasking performs best; gains over the backbone concentrate on RACE and HellaSwag and remain close to neutral on ARC-e and GSM8K.Source: §2.7 · p. 16

3 Discussion

HyperQ’s emitted circuits outperform the tested fixed and searched circuits across the evaluated widths, but the results do not establish scaling beyond those widths. The discussion also frames exact, efficient readout as a trade-off: the circuit family is classically simulable, so the study does not claim quantum computational advantage.

3 Discussion

  • At 64 qubits, HyperQ exceeds the best tested hand-designed ansatz by 3.68 points and the tested 3-motif IQP search by 2.35 points.Both margins widen across the three tested widths.Source: §3 · p. 16
  • Fixed ansatz families improve little after 32 qubits, while the emitted circuit continues improving through 64; these results establish no scaling law beyond the tested widths.The comparison supports the architectural claim only within the measured range.Source: §3 · pp. 16–17
  • Figure 7 reports that the autoregressive model collapses on reversed question answering and infilling, while the bidirectional models hold up.The figure also compares six-benchmark averages across decoding orders and quality against tokens committed per forward pass.Source: Figure 7 · p. 17
  • The ring-and-chord edge set keeps every qubit at degree four, giving exact expectation-value readout cost Θ(n) and derivative variance 1/48 at initialization.The variance result uses the specified independent reference initialisation and does not guarantee trainability across optimization.Source: §3 · pp. 17–18
  • The same two-body structure makes the readout classically simulable, so HyperQ tests architectural bias rather than a classically intractable computation.The authors claim no quantum computational advantage.Source: §3 · p. 18
  • A computational result would require a non-efficiently-simulable circuit family and a task where quantum execution supplies a meaningful separation.The discussion identifies direct evaluation of that separation as necessary, rather than inference from downstream language benchmarks or hardware execution alone.Source: §3 · p. 18

4 Methods

Section 4 describes HyperQ’s token-conditioned quantum residual branches within a frozen masked-diffusion denoiser, including its sparse circuit family and trainable circuit emission. It also details exact circuit-readout evaluation, optimisation and execution, and the fine-tuning, benchmark, and hardware-validation protocols.

4.1 Masked discrete diffusion modelling

  • Training minimizes masked-token cross-entropy under uniform-rate corruption, while inference refines a fully masked sequence for 128 rounds using bidirectional context.Each example is capped at 1,024 tokens; between rounds, the model commits high-confidence predictions and re-masks the rest.Source: §4.1 · pp. 18–19

4.2 Qubit encoding and circuit family

  • A fixed ring-and-chord IQP circuit has degree four per qubit; a rank-16 token code emits its angles and readout axes, enabling O(n) expectation evaluation.Across a balanced cut of an eight-qubit register, emitted couplings yield 2.65 bits of entanglement entropy on average; with inert couplings, the state is a product state with zero entanglement.Source: §4.2 · pp. 19–21

4.3 Circuit trainability analysis

  • For this ring-plus-one-chord-layer family, coordinate-gradient variance stays 1/48 at 16, 32, and 64 qubits because every qubit has degree 4; it does not explain downstream accuracy gains.Source: §4.3 · p. 22

4.4 Circuit emission hypernetwork

  • HyperQ replaces finite motif selection with a continuous hypernetwork that emits token-specific circuit coordinates from a rank-16 code.The 3-motif comparison uses four categorical slots, yielding 81 architectures per token, and trains selection with a straight-through estimator; the continuous branch emits 5n real values.Source: §4.4 · pp. 22–23

4.5 Circuit synthesis and transpilation

  • Synthesis compiles the fixed edge set once per register width, keeping routing, gate count, and depth constant across tokens, then checks readouts against tensor-network contraction within 10^-4.Commuting interior rotations can be packed into parallel layers, and hardware runs batch transpiled circuits in an EstimatorV2 job.Source: §4.5 · p. 25

4.6 HyperQ training and optimisation

  • HyperQ freezes the backbone and trains only the quantum branch on 20,000 prompt-response pairs for three epochs across eight A100 GPUs.This subset processes one tenth as many tokens as the baselines.Source: §4.6 · p. 25
  • The branch is attached to all 22 transformer blocks; freezing the 1.1-billion-parameter backbone gives an approximate per-token backbone cost of 4N rather than 6N floating-point operations.The trainable branch still incurs forward, activation-gradient, and weight-gradient passes.Source: §4.6 · pp. 25–26
  • Θ(n) forward and backward evaluation computes the exact circuit readout without statevector construction, parameter-shift rules, or sampling estimators.At 32 and 64 qubits, the closed form is the training-time engine; it agrees with statevector and tensor-network calculations within 10^-4 in the validation range.Source: §4.6 · pp. 26–27
  • At ω = π/2, the mean magnitude of gradients for the other couplings falls from 0.385 to 3.7 × 10^-5; the trained branch has fewer than 0.2% of couplings in this suppressed regime, versus 3.2% when the range extends to π/2.The emitted couplings are bounded below π/2 using an ωmax = 1.2-radian squashing range.Source: §4.6 · p. 26
  • A 16-to-32-qubit increase multiplies coupling-dependent memory by 2.00, while a 16-to-64-qubit increase multiplies it by 4.00; sparse support also keeps mean readout magnitude at 0.16 across widths.On a complete graph, the corresponding memory factors are 4.13 and 16.8; at 64 qubits the mean readout magnitude with n − 1 cosine factors is 2.6 × 10^-13.Source: §4.6 · p. 26

4.7 Datasets and evaluation protocols

  • HyperQ is fine-tuned on a uniformly sampled 20,000 prompt-response pairs, one tenth the tokens used for the classical baselines, with only the rank-16 branch and circuit heads updated.Training uses three epochs, a global batch of 256 pairs, and a fixed sequence cap of 1,024 tokens; the inherited objective masks response positions while leaving prompts uncorrupted.Source: §4.7 · pp. 27–28
  • Zero-shot evaluation covers 12 tasks and reports a six-benchmark average over ARC-e, HellaSwag, PIQA, BoolQ, RACE, and GSM8K against LLaDA-1.1B and TinyLlama-1.1B.Multiple-choice candidates are scored by conditional log-likelihood under the diffusion bound, estimated with up to 128 Monte Carlo mask samples per item.Source: §4.7 · p. 28
  • Hardware transfer uses 10,000-shot execution on IBM Quebec and Brisbane, scoring 256 randomly selected items per benchmark under both analytic and hardware readouts with identical weights.This paired validation covers a task subset rather than a full benchmark sweep and targets 10^-2 precision for each expectation value consumed by the backbone.Source: §4.7 · p. 28

Data availability.

The six zero-shot benchmarks are public; the instruction-tuning corpus follows the supervised fine-tuning setting reported by LLaDA [13].

Data availability.

  • The six zero-shot benchmarks are public, and the instruction-tuning corpus follows LLaDA’s supervised fine-tuning setting [13].Source: Data availability · p. 28

Related papers

Loading 2609.24657v1…