Variational Quantum Transformer Architecture for Synthetic Language Generation

• Transformers offer an autoregressive interface with contextual processing, while QNLP represents linguistic structure through tensorial or circuit-like models [7].Source: §1 · p. 1
• The proposed model retains transformer-style next-token prediction but replaces attention and feed-forward blocks with variational quantum circuits.It is evaluated on deterministic and lexicographic grammar tasks over four tokens.Source: §1 · p. 2
• Quantum variants learn nontrivial grammar regularities and solve the deterministic task in individual runs, but remain less accurate and stable overall than the classical baseline.Source: §1 · p. 2

Transformers and QNLP

• The model uses transformer-style next-token probabilities as a design pattern, with parallel heads, encoder integration, decoder conditioning, and measured output.This connects to QNLP’s tensor-network or circuit-like representations [7], near-term executions [13], and language-to-circuit software pipelines [14].Source: §2 · p. 2

Variational circuits under NISQ constraints

• Variational quantum circuits encode data into quantum states, apply trainable gates, and optimize parameters from measurement-derived outputs.Source: §2 · p. 2
• Near-term constraints favor small circuit width and depth and simple readout, motivating a compact model for studying sequence behavior rather than replacing classical transformers at scale.Source: §2 · pp. 2–3

3 A NISQ-Compatible Quantum Transformer

• The model estimates a next-token distribution from a context window, with trainable circuit parameters and a four-token vocabulary in the experiments.The context length is four, matching the four computational-basis probabilities of the two-qubit readout state.Source: §3 · p. 3

3.1 Input Encoding

• Each context token is angle-encoded into a four-qubit register, with wire index representing token position instead of a separate positional encoding.Source: §3.1 · p. 3

3.2 Encoder, Connector and Decoder

• Encoder blocks use two variational heads and an integration circuit to reduce the context to a two-qubit state; decoder blocks condition on that state while retaining a path from the original context.Each head and integration circuit returns a two-qubit reduced state.Source: §3.2 · pp. 3–4

3.3 Measurement Readout and Variants

• The final two-qubit density matrix supplies next-token probabilities through its diagonal entries, and the tested variants use at most six active qubits.Variant names encode encoder depth, decoder depth, and variational depth; their parameter counts follow V + D(18V + 2).Source: §3.3 · p. 4

• The controlled tasks use three letter tokens and a separator, with texts formed from three-character words separated by spaces and tokenized character by character.The deterministic task uses a periodic word cycle; the lexicographic task contains length-three words with nondecreasing characters.Source: §4 · pp. 4–5
• Each grammar uses 25 training words and 40 test words; generation greedily continues for 100 tokens from the first four test tokens.Quantum models train for 20 epochs, while the 700-parameter classical baseline trains for 800 epochs as a sanity-check comparison, not a parameter-matched competitor.Source: §4 · p. 5
• Evaluation reports generated-continuation token accuracy and a grammar score that combines four rule-based validity components.For lexicographic generation, grammatical validity can be high even when exact agreement with a sampled test sequence is low.Source: §4 · p. 5

Evaluation

• Performance is aggregated over three seeds, and the reported token and grammar metrics use the same autoregressive generation protocol.Losses are most meaningful within a model family because quantum and classical models use different training schedules.Source: §5 · p. 5

Deterministic grammar

• The classical transformer solves the deterministic task in all runs, while q_e2_d2_v3 reaches grammar score 0.755 ± 0.213 and some deeper variants solve it perfectly in one seed.The quantum variants do not do so consistently, and their large standard deviations indicate unstable optimization.Source: §5 · pp. 5–6, Table 1 · p. 6

Lexicographic grammar

• 0.828 ± 0.299 grammar score for q_e3_d3_v2 is the strongest lexicographic quantum result, while the classical baseline has token accuracy 0.557 ± 0.080 and perfect grammar score.High validity can come from degenerate repeated-word outputs, so validity alone does not assess diversity or distribution matching.Source: §5 · p. 6, Table 1 · p. 6
• The lexicographic task separates exact sequence prediction from grammatical validity, so token accuracy and grammar score capture different outcomes.The classical baseline's moderate token accuracy accompanies perfect grammar validity.Source: §5 · p. 6

Discussion

• The architecture retains a transformer-style next-token interface while replacing its sequence-processing blocks with variational quantum encoder, connector, and decoder blocks.The authors characterize this as feasibility rather than superiority over a classical transformer.Source: §6 · pp. 6–7
• Figure 1 presents grammar-level validity for the classical baseline and quantum variants on both tasks, with seed-level outcomes shown alongside mean scores.Bars show means over three seeds, error bars show one standard deviation, and black points show individual seeds.Source: Figure 1 · p. 7
• Increasing variational depth appears more useful at the tested scale than adding another encoder–decoder layer, but training remains sensitive to initialization and optimization.The large standard deviations indicate that the current procedure does not reliably find good solutions.Source: §6 · p. 7

Limitations

• The evidence is limited to synthetic tasks and exact simulation; the direct two-qubit readout ties the present vocabulary to four tokens.The study does not test semantic representation, compositional meaning, natural-language generalization, or noisy and shot-based execution.Source: §6 · p. 7

Conclusion

• The proposed model preserves autoregressive next-token prediction while replacing classical sequence-processing blocks with variational quantum circuits and a direct two-qubit measurement readout.Its blocks include encoder heads, connector circuits, and decoder blocks.Source: §7 · pp. 7–8
• Quantum models learn nontrivial grammar structure, including perfect deterministic generation in individual runs and high lexicographic validity in the strongest variant.The classical baseline remains more accurate and stable, and quantum outcomes are sensitive to initialization.Source: §7 · p. 8
• The results support a viable architecture for controlled autoregressive QNLP experiments, not a claim of quantum advantage.Future work should address larger vocabularies and contexts, optimization stability, and shot-based or noisy execution.Source: §7 · p. 8

A Detailed Circuit Diagram

• The appendix diagram depicts the full circuit-level layout of the two-encoder, two-decoder quantum transformer instance.The diagram was moved to the appendix to keep the main paper focused on the architectural definition and experimental results.Source: Figure 2 · p. 10, A Detailed Circuit Diagram · p. 10
• Red marks embedding circuits and blue marks individual heads in the circuit diagram.Source: Figure 2 · p. 10
• Yellow marks encoder integration circuits, green connector blocks, and orange decoder blocks.Source: Figure 2 · p. 10