PaperlayerView exact revision on arXiv

Source-linked AI summary

All You Need Is Non-Commutative Words

Carla M. Quispe Flores, Stanley Salvatierra, Renan Cabrera

arXiv:2608.29314v1cs.CL

TL;DR

The paper addresses how to represent sequential text without separately added positional signals or modular components for related capabilities. It encodes words as unitary matrices and composes them in order, deriving attention, chunking, readout, and continual-learning mechanisms from the same algebra. On standard text-classification benchmarks, it matches or exceeds bag-of-words baselines while using a dense 64-coordinate encoding instead of a roughly 30,000-dimensional vocabulary space.

  • Problem

    Bag-of-words and self-attention are permutation-invariant in their core operations, so positional information must be added separately to represent token order.

  • Method

    The method maps each word to a learnable unitary operator and composes operators by ordered matrix multiplication, with shared algebraic mechanisms for attention, chunking, readout, and continual learning.

  • Results

    The method matches bag-of-words baselines on standard text-classification benchmarks, with 86.53% IMDB accuracy and 87.68% AG News accuracy under the adaptive rotation budget.

  • Takeaways & Limitations

    A 64-coordinate real encoding can support order-sensitive text representation and multiple derived capabilities within a single algebraic framework.

  • Takeaways & Limitations

    At fixed operator capacity, continual-learning expansion limits how much a new task can be represented without mixing into the frozen block.

Abstract

from arXiv · show

We represent lexical tokens as unitary matrices and encode each sentence as their ordered product. The noncommutativity of matrix product captures word order without positional encodings (PEs). The same algebra yields several capabilities, including antisymmetric self-attention with no query, key, or value projections, and parallel composition of variable-length text chunks at a reduced attention cost. Furthermore, it provides a canonical-coset readout layer that encodes all true unitary degrees of freedom compactly, while supporting continual learning through nested group extensions that enlarge the operator space with each new task preserving prior representations exactly. Across standard text-classification benchmarks, the method matches or exceeds bag-of-words baselines. Achieving higher accuracy on IMDB and comparable performance on AG News. Notably, this is accomplished by replacing the conventional $\sim$30,000-dimensional vocabulary space with a dense, 64-parameter real-valued encoding, highlighting the expressive efficiency of our parameterization.

I. INTRODUCTION

The paper proposes representing words as learnable unitary operators whose noncommutative products encode word order intrinsically, without positional encodings. The same algebra supports QKV-free attention, parallel chunk composition, compact readout, and continual learning, while using a 64-dimensional real coordinate space.

  • Motivation: Bag-of-words and self-attention representations require separately added positional information because their core operations are permutation-invariant.Prior unitary recurrent models handle order through recurrence but retain conventional vector word embeddings.
  • Proposed framework: Each token becomes a learnable unitary group element, and an ordered matrix product forms a document operator whose noncommutativity preserves word order.The architecture maps token identifiers to Chevalley-basis coordinates, constructs Hermitian generators, exponentiates them, and applies a canonical-coset readout.
  • Derived capabilities: The framework derives order-sensitive self-attention from one learned Hermitian operator without query, key, or value projections.Its score is antisymmetric, avoiding the three full projection matrices used by conventional directed attention.
  • Derived capabilities: Continual learning uses nested group extensions to enlarge the operator space while preserving prior task representations exactly.The paper also reports canonical-coset coordinates as a compact readout of the intrinsic unitary degrees of freedom.
  • Efficiency and evaluation: The method replaces a roughly 30,000-dimensional vocabulary space with a dense 64-coordinate encoding and matches bag-of-words baselines on standard text-classification benchmarks.The 64 coordinates arise from the real basis of 8×8 Hermitian matrices in u(8).

B. Adaptive Per-Word Rotation Budget

The adaptive per-word rotation budget assigns rotation magnitudes from word embeddings rather than treating all words equally. This allocates larger rotations to informative content words and smaller rotations to low-information function words while controlling phase accumulation.

  • Motivation: A global rotation budget treats all words equally, although content words carry stronger discriminative signals than function words.The proposed motivation is to allocate rotation magnitude according to word importance.
  • Adaptive budget: Each word predicts its own rotation budget from its embedding using a learned weight vector and scalar bias.The budget mechanism adds approximately 301 parameters and applies softplus to ensure positivity.
  • Phase control: The adaptive budget dynamically controls accumulated eigenphase to prevent phase aliasing in long sequences.A fixed global budget can over-rotate long documents toward phase wrap-around, whereas εw varies with word importance.

C. Order for Free: Non-Commutative Matrix Product

Ordered products of unitary word operators encode sequence order intrinsically through non-commutativity, eliminating positional encodings. The same structure supports QKV-free antisymmetric attention and parallel chunking with reduced score-pair computation.

  • Order-sensitive representation: Group closure composes variable-length sequences into a single unitary document operator without padding to a fixed dimension.Each word contributes a unitary operator, and their time-ordered product remains in U(8).
  • Order-sensitive representation: Non-commuting matrix products make the final document representation order-sensitive, so swapping words changes the representation without positional encodings.The document operator is formed from prefix products and can otherwise be flattened into a 128-dimensional real vector.
  • QKV-free attention: 64 parameters in one Hermitian matrix A provide direction-sensitive attention instead of three full query, key, and value projection matrices.The score is computed directly from unitary prefix products and is exactly antisymmetric for Hermitian A.
  • Variable-length chunking: Chunking L words into L/c ordered products reduces score-pair computation by a factor of 1/c^2 relative to the unchunked baseline.Associativity permits parallel evaluation, while each chunk remains a valid unitary operator when its rotation budget is scaled appropriately.
  • QKV-free attention: The attention design replaces positional encodings and WQ, WK, WV projections with direct scoring of unitary prefix products through one Hermitian matrix A.Order is supplied by non-commutativity, and antisymmetry is built into the score.

F. Canonical-Coset Readout Layer

The canonical-coset readout represents a unitary document operator using its exact intrinsic degrees of freedom rather than an overcomplete flattened tensor. Its coordinates have a geometric interpretation based on successive column tilts and alignment.

  • Intrinsic coordinate representation: For n = 8, flattening the real and imaginary parts yields 128 numbers, although the unitary operator has only 64 real degrees of freedom.The constraint P†P = I ties exactly half of the flattened entries, making the 128-dimensional representation overcomplete.
  • Coset decomposition: Canonical coset coordinates retain only the genuinely new information contributed by each column after preceding columns are fixed.The decomposition uses nested quotient factors together with residual diagonal phases.
  • Geometric interpretation: Each column’s coset coordinates lie in a closed ball, with radius 0 representing alignment to its target axis and radius 1 representing full perpendicularity.The surviving alignment is quantified by sqrt(1 − r_k^2), and the ball constraint follows from column unit length.
  • Intrinsic coordinate representation: For n = 8, the canonical-coset readout produces a 64-dimensional coordinate vector containing the exact intrinsic degrees of freedom of U(8).The vector comprises 56 ball coordinates and 8 diagonal phases before batch normalization and the linear head.

G. Coset-Tower and Continual Learning

The coset tower expands the unitary operator space through nested group extensions while freezing previously learned blocks. This gives exact algebraic parameter isolation and permits task deletion by linear projection, with a measurable capacity trade-off.

  • Nested group extensions: Nested extensions U(n) ⊂ U(n+k) append new coordinate shells while preserving prior representations in the top-left block.The construction applies the canonical coset decomposition block-wise to organize newly added coordinates.
  • Exact preservation: The embedding diag(G, I_k) is a group homomorphism, so embedded word operators, prefix products, and document operators remain exactly embedded.Products of embedded operators equal the embedding of their products.
  • Capacity growth: For n = 8 and k = 4, the added shell contains 80 coordinates: four vector shells contribute 76 coordinates and four phases contribute the remainder.The shell decomposition lists B22, B20, B18, and B16 plus four phases.
  • Exact preservation: Training updates only the new shell while the original Hermitian block receives zero gradient, providing exact algebraic parameter isolation.The shell contains the mixing block and the new internal block; the frozen block preserves prior task structure.

III. RESULTS AND DISCUSSION

The results evaluate adaptive rotation budgets, QKV-free attention, and chunked composition for sequence modeling. Across these experiments, the algebra supports competitive classification accuracy and longer-context processing without explicit positional embeddings.

  • A. Global vs. Adaptive Per-Word Rotation Budget: 86.53% IMDB accuracy and 87.68% AG News accuracy are achieved with predicted per-word rotation budgets.These results exceed the flat global-budget baseline and tightly compete with hashed bag-of-words.
  • B. Attention variants: 64 parameters from an 8 × 8 Hermitian matrix generate the antisymmetric attention score without query, key, or value projections.The score is computed from a single learned Hermitian operator.
  • B. Attention variants: 84.4% IMDB and 87.1% AG News accuracy make the QKV-free attention model competitive with conventional attention variants.It statistically matches the similarly parameterized conventional transformer and outperforms PaTH-lite, while remaining below the bag-of-words ceiling.
  • C. Chunked Attention: Chunk-level unitary operators allow consecutive-word products to be evaluated in parallel while preserving group closure for sequences up to 1024 words.The method reduces the quadratic cost of full token-level attention by composing variable-length chunks.

D. Reading out in Canonical Coset Coordinates

Canonical-coset coordinates provide a compact, non-redundant readout, while nested unitary extensions support exact task preservation and deletion. The approach has a capacity trade-off for newly added tasks and remains limited to small text-classification operators.

  • D. Reading out in Canonical Coset Coordinates: The canonical-coset readout maps the document operator to 64 intrinsic dimensions, avoiding the redundant 128-dimensional complex-matrix flattening.It isolates the exact non-redundant unitary degrees of freedom through nested Householder spherical shells.
  • D. Reading out in Canonical Coset Coordinates: 79.4% IMDB and 84.8% AG News accuracy are achieved with the 64-dimensional canonical-coset readout at ε = 0.15.The flattened representation reaches 82.6% on IMDB and 84.9% on AG News under the same comparison.
  • E. Coset-tower: The coset-tower embeds a frozen U(8) task-A block in U(12) and trains task B solely in an 80-coordinate shell.The base block receives zero gradient after task A is frozen.
  • E. Coset-tower: Accuracy on task A remains unchanged through extension and deletion, whereas standard finetuning reduces it to 53.2%.The reported task-A accuracy is preserved exactly in the extension and remains untouched when the shell is projected to zero.
  • E. Coset-tower: Deleting the shell by linear projection removes task B in one step while leaving task A unchanged, without retraining.Task B returns to its 25% chance level after deletion.
  • E. Coset-tower: The shell constrains task-B capacity: it reaches 69.2% versus 86.6% when trained alone in full U(12), a 17.4-point gap.This gap measures task-B performance unavailable without mixing into the frozen block and limits fixed-width tower iteration.

Appendix A: Bounding εw

The appendix bounds per-word and accumulated eigenphases under normalized Hermitian generators, including non-commuting products. These bounds motivate wrap-safe chunking and adaptive rotation budgets for long documents.

  • Single-word bound: Unit Frobenius normalization of each Hermitian generator bounds every effective word eigenphase by |λa| ≤ εw.The bound follows because the spectral norm does not exceed the Frobenius norm.
  • Single-word bound: When εw < π, each word induces a small, unambiguous rotation without phase aliasing.Its eigenphases remain inside the principal branch (−π, π].
  • Document accumulation: For non-commuting generators, eigenphase increments are not additive; their order-dependent departure begins at order εwiεwj in the Baker–Campbell–Hausdorff expansion.Non-commutativity encodes word order while the accumulated norm bound still holds.
  • Document accumulation: A total budget ΦL < 2 keeps every document eigenphase strictly within the principal branch and prevents 2π wrap.For small total budgets, the accumulated eigenphase is bounded by ΦL up to cubic corrections.
  • Wrap-safe chunking and adaptive budget: Chunking with εw = ε/c caps each c-word chunk budget at ε independently of chunk size, making every chunk wrap-safe when ε < 2.This supports parallel evaluation of ordered chunk products while preserving the bound.
  • Wrap-safe chunking and adaptive budget: Adaptive word-dependent budgets allocate smaller rotations to low-information words and larger portions of the phase budget to discriminative content words.The resulting accumulated phase tracks document information content rather than raw length.

Appendix C: Householder Construction of the Coset Coordinates and the Ball Constraint

The Householder sweep converts a unitary operator into canonical coset coordinates and residual phases, with each coset vector constrained to a unit ball. An explicit inverse enables exact reconstruction.

  • Householder construction: Each factor in the unitary decomposition is represented as a product of two Householder reflections.The reflections recursively reduce the operator while preserving the decomposition structure.
  • The ball constraint: Each coset vector lies in the closed unit ball because it represents the perpendicular component of a unit-norm column after removing its leading phase.Its radius measures the column’s tilt from the target axis, while the remaining square-root term measures alignment.
  • Householder construction: A Householder sweep aligns successive columns, recursively reduces the remaining block, and collects residual phases in a diagonal factor.The coset vectors and phases are read off at each step.
  • Inverse map: The inverse map reconstructs the normalized Householder vector from the coset vector and its diagonal phase.This explicit inverse yields round-trip reconstruction exact to numerical precision, with error 1.1 × 10^-15.
Loading 2608.29314v1…