PaperlayerView exact revision on arXiv

Source-linked AI summary

QSV: Quat-Sphere-Vision for Coupled Quaternion Attention on Spherical Lattices

Nicholas Foley, Devin Marinelli, Donny Moore, Diego Enriquez, Amanda Fernandez

arXiv:2609.30592v1cs.LGcs.CV

TL;DR

Standard attention learns routing and message transformation with separate projections, and QSV asks whether one learned geometric object can serve both roles. QSV uses a relative quaternion for both routing and feature transport; matched results show standard attention on the same graph performs at least as well, while transport ablations indicate an asymmetry between the roles.

  • Problem

    Standard attention learns how strongly tokens attend and how their messages are transformed with separate projections that share no structure.

    Source: §1 · p. 1

  • Method

    QSV replaces the projection triple with one learned unit quaternion per token, whose relative quaternion supplies both the attention score and sandwich-product feature transport.

    Source: §1 · p. 1, §6 · p. 7

  • Results

    87.3% vs. 85.9% mean: parameter- and graph-matched standard attention performs at least as well as QSV; removing transport costs 4.3 points on CIFAR-10 and 4.1 on CIFAR-100.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    In the coupled kernel, nearly all learned pairwise computation resides in transport, while routing weights can be replaced by uniform aggregation with little accuracy change.

    Source: §1 · p. 1, §6 · p. 7

    Evaluation covers CIFAR-10 and CIFAR-100 at one model scale; persistence at larger scale, higher resolution, on natively spherical data, or against stronger baselines remains unevaluated.

    Source: §5 · p. 7

Abstract

from arXiv · show

In standard attention, three separately learned projections decide how strongly a token attends to each neighbor ($W_Q$, $W_K$) and how the attended features are transformed before aggregation ($W_V$). We study Quat-Sphere-Vision (QSV), a sparse spherical vision model that replaces this projection triple with a single learned unit quaternion per token: the relative quaternion $r_{ij} = q_i^{*} \otimes q_j$ supplies both the attention logit $\operatorname{Re}(r_{ij})$ and a sandwich-product feature transport $x \mapsto r_{ij} \otimes x \otimes r_{ij}^{*}$, with messages passed over sparse kNN graphs on concentric Fibonacci spheres. Ablations that change only the targeted component show the two roles to be asymmetric. Removing the transport reduces test accuracy by about four percentage points on CIFAR-10 and CIFAR-100 (single runs per CIFAR-100 variant), while replacing the learned attention weights with uniform averaging leaves it essentially unchanged. Parameter-matched controls then remove the geometry itself: standard attention on the same graph exceeds QSV (mean $87.3\%$ vs. $85.9\%$), and the same model on a flat 2D lattice reaches $91.1\%$, within $2.1$ points of a ResNet-20 trained under the same pipeline (single run). In the coupled kernel, nearly all of the learned pairwise computation resides in the transport channel.

1 Introduction

QSV replaces standard attention’s separate routing and value projections with a single learned quaternion that supplies both an attention score and feature transport. Its transport channel matters more than its routing weights, while matched controls show stronger results for standard attention on the same graph and for a flat-lattice variant.

1 Introduction

  • QSV replaces WQ, WK, and WV with one learned unit quaternion per token, whose relative quaternion supplies both an alignment score and a sandwich-product feature transform.The real part gives the score, while the sandwich product rotates the imaginary part of the feature quaternion.Source: §1 · p. 1
  • Tokens occupy Fibonacci-lattice vertices on concentric spheres, connect through sparse k-nearest-neighbor edges, and use Lambert patch sampling, Hamilton structured linears, and a quaternion FFN.Unit quaternion parameters are trained with Riemannian Adam on S3.Source: §1 · p. 1
  • Removing transport reduces test accuracy by 4.3 points on CIFAR-10 and 4.1 points on CIFAR-100; replacing learned routing with uniform aggregation changes accuracy by at most 0.4 points on both.Disabling both roles or freezing the quaternion field at random values collapses accuracy to about 65%.Source: §1 · p. 1
  • 87.3% vs. 85.9% mean: parameter- and graph-matched standard attention performs at least as well as QSV, while a flat-lattice variant reaches 91.1%.The flat-lattice result is within about two points of a same-pipeline ResNet-20.Source: §1 · p. 2
  • The matched controls indicate that the quaternion kernel form and spherical layout are replaceable in this CIFAR-10/100 setting, while the transport channel matters more than routing.The flat-lattice matched-attention variant gains 3.8 points; the analysis also attributes much of the fine-texture generalization bottleneck to the spherical layout itself.Source: §1 · p. 2

2 Related Work

Related work places QSV alongside quaternion attention, equivariant attention, rotation-based positional encodings, spherical networks, and studies of attention weights. These connections distinguish QSV’s learned transport-and-routing coupling from approaches that retain separate paths or use fixed geometric structure.

2.1 Quaternion attention models

  • Quaternion-valued attention and graph message-passing models retain separate Q, K, and V paths; QuatE instead uses a per-relation quaternion transform and an inner-product score, unlike QSV’s two-sided transport and per-vertex quaternion.QSV’s uniform-routing ablation also finds the quaternion-derived score replaceable when transport remains.Source: §2.1 · p. 2

2.2 Equivariant attention and capsule routing

  • Equivariant attention separates invariant aggregation weights from equivariant value kernels, whereas QSV couples these roles without claiming equivariance; its routing result extends prior findings that fixed or uniform capsule routing can suffice.The cited capsule-routing studies include Paik, Kwak, and Kim (2019), Gu and Tresp (2020), and Byerly, Kalganova, and Dear (2021).Source: §2.2 · p. 2

2.3 Rotation based positional encoding

  • RoPE-style methods rotate Q and K but leave V untouched, while QSV derives both the routing logit and value transform from the same learned per-token relative rotation.GeoPE and RiemannFormer also apply rotations on Q/K paths rather than reporting learned value-path transport.Source: §2.3 · p. 2

2.4 Spherical and gauge-equivariant networks

  • Gauge-equivariant spherical networks transport features between per-vertex frames using a fixed mesh-derived connection; QSV instead learns a quaternion connection and makes no equivariance claim.Source: §2.4 · p. 2

2.5 Token mixers and the role of attention weights

  • Prior work finds attention weights often dispensable; QSV decomposes one geometric kernel more finely, finding routing dispensable given transport while transport is not, and its algebraic form interchangeable with matched attention.The QSV overview depicts sparse propagation across five Fibonacci shells and labels the kernel ablations.Source: §2.5 · p. 3, Figure 2 · p. 3

3 Methods

QSV places image tokens on learnable concentric spherical lattices and couples attention scoring with feature transport through relative quaternions. Its method combines sparse kNN propagation with quaternion-specific updates and pooling, evaluated on CIFAR-10 and CIFAR-100.

3.1 Token layout and geodesic patch embedding

  • Five concentric spherical shells use Fibonacci-lattice vertices; a Lambert projection samples and bilinearly interpolates 5 × 5 image patches for the first shell.The first-shell patch projection is differentiable with respect to vertex positions.Source: §3.1 · p. 3

3.2 Quaternion attention

  • Each token’s features are reshaped into quaternion tensors, and one relative quaternion supplies both the attention score and sandwich-product feature transport.Quaternion features are perturbed by a data-dependent Lie-algebra MLP; Hamilton structured linears and a quaternion FFN handle post-aggregation updates.Source: §3.2 · p. 3, Lie algebra perturbation · p. 3, Hamilton structured linears · p. 4
  • The forward pass combines within-shell and cross-shell propagation, followed by two final within-shell rounds and quaternion-native pooling.Cross-shell steps apply the quaternion attention update and Hamilton block with a residual.Source: §3.2 · p. 3, Regularizers and pooling · p. 4

3.3 Sparse kNN connectivity

  • Sparse kNN graphs connect each target only to its nearest neighbors, yielding about 0.05 of the dense within-shell edge count.Cross-shell connectivity uses Euclidean kNN and final within-shell rounds use angular kNN; on the fixed-radius shells, the two rankings coincide.Source: §3.3 · p. 4

3.4 Optimization on S3 and the shells

  • Quaternion parameters use Riemannian Adam, with tangent-space gradients mapped through the Lie algebra and exponential-map retraction.Moments are stored at the Lie algebra identity, avoiding inter-iteration parallel transport.Source: Riemannian Adam · p. 4, §3.4 · p. 4
  • Periodic tangential repulsion updates spread shell positions, while auxiliary losses regularize quaternion smoothness, head diversity, and learned temperatures.After the final shell, features are pooled using a mean quaternion direction and softmax attention over vertices before classification.Source: Sphere Coulomb tangent repulsion · p. 4, Regularizers and pooling · p. 4

3.5 Datasets

  • Experiments use CIFAR-10 and CIFAR-100 with 45,000 training, 5,000 validation, and 10,000 test images each; test accuracy is measured at the best-validation checkpoint.CIFAR-100 uses the same pipeline and configuration apart from the class count and repulsion neighbor-count adjustment.Source: §3.5 · p. 4

4 Experiments

Experiments assess QSV’s trainability, separate the quaternion kernel’s routing and transport roles, and compare matched attention, geometry, and optimization controls. The controls show that the flat-lattice attention model outperforms the spherical models on CIFAR-10.

4.1 Trainability and accuracy

  • QSV reaches up to 86.9% test accuracy, with test accuracy staying within 85.7–86.9% across broad learning-rate, quaternion-step-size, repulsion, and seed variants.The authors do not present these numbers as competitive; Table 1 provides context.Source: §4.1 · pp. 4–5

4.5 A flat-lattice control

  • 91.1% test accuracy on the flat-lattice control is 3.8 points above spherical attention and within 2.1 points of same-pipeline ResNet-20 (93.2%, single run).The flat-lattice result is a three-seed mean; the ResNet-20 result is a single run.Source: Table 1 · p. 5, §4.5 · p. 6

4.2 Which part of the kernel matters

  • Transport removal costs 4.3 CIFAR-10 and 4.1 CIFAR-100 accuracy points, while uniform routing changes accuracy by only −0.2 and −0.4, respectively.Disabling both roles or freezing the quaternion field at random values also collapses accuracy, to 64.1 and 66.5 in single runs.Source: Table 2 · p. 5, §4.2 · p. 5

4.3 Routing diagnostics

  • Attention weights remain near uniform (entropy about 0.99 of maximum; peak weights 1.2–1.4× uniform), and freezing β = 5 changes accuracy by −0.4 within seed noise.Warm-started cross-shell temperatures return to the prior fixed-point range, while uniform, learned-soft, and forced-sharp aggregation reach the same accuracy in the reported probe.Source: §4.3 · pp. 5–6, Figure 3 · p. 6

4.4 A matched attention baseline

  • 87.3% test accuracy (three-seed mean) for matched standard attention matches or slightly exceeds QSV’s 86.9% best run and 85.9% default mean.At d = 256, QSV’s kernel uses about 87K parameters versus 1.32M for attention; QSV’s compiled training throughput is about 11× lower.Source: §4.4 · p. 6

4.6 Optimization and geometry

  • The optimization stack stabilizes quaternion training and perturbations, while sphere repulsion preserves uniform vertex coverage despite having little accuracy effect.Without repulsion, vertices drift toward concentrated regions within a dozen epochs; disabling it matched or slightly improved validation accuracy in these runs.Source: §4.6 · p. 7

4.7 Where the error is

  • Vehicle and global-shape classes are most reliable, while fine-texture animal classes are weakest; clean-training accuracy of 95.7% versus 86.5% test accuracy points mainly to generalization error.The model does not interpolate the clean training set, and preliminary graph-size and mixup/cutmix runs did not close the gap.Source: §4.7 · p. 7

5 Limitations

QSV falls short of strong CIFAR-10 convolutional and transformer baselines, with the flat-lattice control suggesting much of the deficit is associated with the spherical layout. The evidence is limited in dataset and model scale, and the authors do not claim input-rotation equivariance.

5 Limitations

  • The flat-lattice control suggests much of QSV’s CIFAR-10 deficit against strong convolutional and transformer baselines traces to the spherical layout.Those baselines commonly exceed 90%.Source: §5 · p. 7
  • QSV’s errors are class-structured, with fine-texture animal classes weakest, while the authors suggest spherical tokenization and sparse graph transport preserve global structure better than fine texture.The flat-lattice variant is presented as the natural remedy for planar images.Source: §5 · p. 7
  • The study covers CIFAR-10 and CIFAR-100 at one model scale, leaving larger-scale, higher-resolution, natively spherical, and stronger-baseline evaluations untested.The matched-attention comparison used three CIFAR-10 seeds and a single-run CIFAR-100 baseline; the flat-lattice comparison used three CIFAR-10 seeds only.Source: §5 · p. 7
  • The authors do not claim rotation equivariance: free per-vertex quaternions and a non-equivariant perturbation MLP leave any preserved symmetry unresolved.They describe the construction as a learned quaternion connection between per-vertex frames.Source: §5 · p. 7

6 Conclusion

QSV couples attention routing and feature transport through one learned unit quaternion per token. Its transport role matters more than its learned routing weights, but matched controls outperform it, and further tests are proposed.

6 Conclusion

  • QSV uses one learned unit quaternion per token as both routing weight and sandwich-product feature transport.A relative rotation supplies both roles.Source: §6 · p. 7
  • Matched controls report 87.3% for standard attention on the same graph versus QSV’s 85.9% mean, while the flat lattice reaches 91.1% against 93.2% for a same-pipeline ResNet-20.The flat-lattice and ResNet-20 figures are single runs.Source: §6 · p. 7
  • The kernel roles are unequal: transport reduces accuracy by 4.3/4.1 points, routing changes accuracy within noise, and disabling both yields 64.1.The reported transport deltas span CIFAR-10 and CIFAR-100.Source: §6 · p. 7
  • Future work includes matched comparisons with equivariant-attention baselines and quaternion-routing prior work, plus testing the asymmetry at larger scale.The cited works are Fuchs et al. 2020, Brehmer et al. 2023, and Zhao et al. 2020.Source: §6 · p. 7

A Default Configuration

The default configuration specifies a five-shell Fibonacci sphere graph, optimization and regularization settings, and implementation details. Reported per-seed results and throughput provide additional reference points for the CIFAR-10 setup.

A Default Configuration

  • The default model uses d = 256, H = 16, five shells with vertex counts [128, 192, 256, 320, 384], and per-shell neighbor counts [14, 14, 16, 16, 16].The resulting edge ratio is approximately 0.05 using the per-shell dense denominator.Source: A Default Configuration · p. 9
  • Optimization uses AdamW (weight decay 0.05) with a onecycle cosine-shaped schedule peaking at 10−3; the quaternion group uses Riemannian Adam at a constant 2.5 × 10−3.The schedule uses 10% warmup, an initial division factor of 10, and a final division factor of 100. Position learning rate is 5 × 10−5 with no weight decay, and the ˆε logits train at 3× the base rate.Source: A Default Configuration · p. 9
  • Regularization uses label smoothing and 0.1 attention and output dropout, alongside quaternion smoothness and head-diversity penalties; augmentation uses padded crops, horizontal flips, and TrivialAugmentWide.The cited penalty coefficients are ξ = 10^-3 and λdiv = 10^-2.Source: A Default Configuration · p. 9
  • The quaternion perturbation bound is sigmoid-parameterized per shell, while sphere repulsion uses 8 neighbors and is applied every 10 batches.The repulsion neighbor count is 16 on CIFAR-100, and its target step decays to zero over roughly the first 40% of training.Source: A Default Configuration · p. 9
  • Per-seed CIFAR-10 test accuracies include 85.9/86.0 for baseline, 81.3/81.7/82.1 for transport-to-identity, and 85.3/86.1 for uniform routing.The passage also reports values for absolute-real-part, geometric, and identity value-projection variants.Source: A Default Configuration · p. 9
  • On an RTX A6000, compiled QSV trains at approximately 470 images/s, compared with approximately 5,350 images/s for compiled matched attention.The passage reports an approximately 14-hour full QSV run.Source: A Default Configuration · p. 9

Related papers

Loading 2609.30592v1…