PaperlayerView exact revision on arXiv

Source-linked AI summary

Subspace Clifford Networks

Zhongping Ji

arXiv:2610.37226v1cs.CV

TL;DR

Sparse rolling limits the width of Clifford wedge interactions by sampling channel pairs, but full pairwise wedge components scale quadratically and five-shift rolling expands channels to 10D. Subspace Clifford Networks group channels into small subspaces, and experiments on CIFAR-100 report reduced parameters and training time, with hierarchical variants also improving accuracy over evaluated shift-based hierarchical baselines.

  • Problem

    Computing all pairwise wedge components across D channels creates O(D²) intermediate channels, while five-shift rolling expands features to 10D and requires multiple memory-intensive operations.

    Source: §1 · pp. 1–2

  • Method

    Subspace Clifford Networks replace global cyclic shifts with grouped interactions for k ∈ {2, 3, 4}, sandwiched between learnable 1 × 1 projections and implemented with a fused CUDA kernel.

    Source: §1 · p. 2, §5 · p. 11

  • Results

    Hier-Subspace-CAN (k = 3) reaches 80.12% Top-1 accuracy, exceeding the 4-shift hierarchical baseline by +0.59% with 60% fewer parameters and 37% less training time.

    Source: §4.2 · p. 9

  • Takeaways & Limitations

    The proposed models reduce parameter count and training time across isotropic and hierarchical regimes, while hierarchical variants improve accuracy over the evaluated shift-based hierarchical baselines.

    Source: §4.2 · pp. 8–9

    The reported rolling-cost comparison specifies 10D expansion and 10D² MACs per pixel for the setting with |S| = 5 shifts.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

In this work, we propose Subspace Clifford Networks, a reformulation that replaces global cyclic rolling with grouped subspace interactions. Instead of sampling sparse cyclic diagonals across the entire $D$-dimensional channel space, we project the features into an aligned space $\R^{\tilde{D}} = \bigoplus_{g=1}^{G} V_g$ partitioned into $G$ low-dimensional groups $V_g \cong \R^k$ with $k \in \{2, 3, 4\}$. Within each $k$-dimensional group, we compute all $\binom{k}{2}$ wedge coordinates and concatenate them with $\operatorname{SiLU}$-activated coordinate-wise products, while sandwiching the grouped interaction between learnable $1\times 1$ linear projections to enable cross-group channel mixing. This design reduces intermediate feature expansion from $2|\mathcal{S}|D$ (typically $10D$ for 5 shifts) down to $1.5\tilde{D}$ ($k=2$), $2.0\tilde{D}$ ($k=3$), or $2.5\tilde{D}$ ($k=4$). Furthermore, exploiting the small, fixed group size $k \in \{2, 3, 4\}$, we implement a fused CUDA kernel that evaluates the forward interaction and a recomputation-based analytical backward pass using thread-local variables without allocating global-memory buffers for elementwise intermediates. Finally, we extend the backbone from an isotropic columnar layout to a multi-resolution Hierarchical Clifford Network with configurable per-stage subspace dimensions. Experiments on CIFAR-100 show that the proposed models substantially reduce parameter count and training time across both regimes, while the hierarchical variants also improve classification accuracy over the evaluated shift-based hierarchical baselines.

1 Introduction

Global wedge interactions create a computational trade-off: all channel pairs require quadratic intermediate width, while sparse rolling limits width by sampling cyclic pairs. Subspace Clifford Networks replace that sampling with complete within-group interactions and reduce expansion while supporting fused execution and hierarchical variants.

1 Introduction

  • All-pairs wedge components scale quadratically with D, whereas coordinate-wise products scale linearly.The original sparse-rolling approach samples cyclic channel pairs to keep interaction width linear, but samples only a predefined subset.Source: §1 · p. 1
  • Five-shift rolling concatenates 10 feature maps of width D, increasing the subsequent projection’s parameter count and multiply-accumulate operations.Separate shifts, products, and concatenations also launch multiple GPU kernels and allocate intermediate tensors in global memory.Source: §1 · pp. 1–2
  • The proposed grouped interaction replaces sparse global cyclic shifts with all pairwise wedge coordinates in groups of size k ∈ {2, 3, 4}, between learnable 1 × 1 projections.The input projection mixes channels before grouping, and the output projection maps concatenated group outputs back to R^D.Source: §1 · p. 2
  • For group size k = 4, the expansion is 2.5D̃, a 75% reduction compared with the 10D expansion of 5-shift rolling.Source: §1 · p. 2
  • The fused CUDA implementation combines interaction operations and a recomputation-based analytical backward pass without intermediate global-memory buffers.The reported contributions include a 17.2× operator speedup and 50% lower peak extra allocated memory for the evaluated k = 4 benchmark.Source: §1 · p. 2
  • The hierarchical architecture supports configurable subspace dimensions across stages of a multi-resolution feature pyramid.The contribution also describes a contiguous 5 × 5 SpatialDiffusion module.Source: §1 · p. 2

2 Related Work

The work builds on lightweight Clifford interactions for real-valued vision features, replacing cyclic shift approximations with grouped subspace interactions and fused execution.

Clifford Algebra in Deep Learning

  • The approach replaces CliffordNet’s cyclic shift approximation with grouped subspace interactions and a fused CUDA kernel.CliffordNet combines coordinate-wise inner products and shifted wedge differences over real-valued 2D vision feature maps without separate FFN blocks.Source: §2 · p. 2

Efficient Vision Backbones & Channel Mixing

  • Modern lightweight backbones use structural priors such as depthwise separable convolutions or channel split and shuffle, while many architectures pair spatial token mixing with channel-mixing MLPs.The cited MLP designs typically use a 4× channel expansion ratio; the proposed approach instead sandwiches low-dimensional subspace interactions between 1 × 1 projections.Source: §2 · p. 2

3 Methodology

The methodology recaps CliffordNet’s feature-interaction formulation and sparse rolling implementation, then replaces global shifts with grouped subspace interactions. The new design uses pre-projection channel mixing and local group interactions to reduce intermediate width.

3.1 Recap: Feature Evolution and the Clifford Interaction Ansatz

  • CliffordNet models each layer’s features as a D-dimensional vector field over a preserved 2D spatial grid and motivates updates as a discrete dynamical system.The context field aggregates local neighborhood information over the grid, and the interaction operator combines state and context.Source: §3.1 · p. 3
  • The Clifford interaction replaces a scalar dot product with coordinate-wise products concatenated with wedge features, followed by a learnable 1 × 1 contraction projection.The wedge coefficients represent signed areas in coordinate planes.Source: §3.1 · p. 3

3.2 Recap: Sparse Rolling Realization and Its Costs

  • Sparse rolling computes shifted inner and wedge slices for predefined offsets, concatenates them, projects back to D channels, and adds the result through a gated geometric residual update.With five shifts, the concatenated width is 10D; the contraction projection requires 10D² MACs per pixel, and standard execution uses multiple shift and concatenation operations.Source: §3.2 · pp. 3–4

3.3 Proposed Method: Subspace Clifford Interaction

  • The proposed Subspace Clifford Interaction replaces global cyclic shifts with grouped interactions to reduce intermediate channel expansion while evaluating all wedge pairs within local groups.The rolling realization samples only selected cyclic off-diagonals rather than all pairs within local channel groups.Source: §3.3 · p. 4, §3.2 · p. 4

3.3.1 Channel Grouping and Pre-Projection Mixing

  • The method aligns channel width to a multiple of k and applies Batch Normalization followed by a learnable 1 × 1 input projection to mix channels and generate state and initial context streams.A 3 × 3 depthwise convolution computes the spatial context stream; the differential parameter λ changes the coordinate-wise inner branch but leaves the wedge branch invariant in exact real arithmetic.Source: §3.3.1 · p. 4

3.3.2 Within-Group Interaction for 𝑘∈{2, 3, 4}

  • At each spatial location (h, w), the aligned channel space is partitioned into G contiguous k-dimensional coordinate subspaces, whose group slices of U and V are used for within-group feature computation.Source: §3.3.2 · pp. 4–5

1. Coordinate-Wise Inner Branch (p(𝑔)

The inner branch retains coordinate-wise products rather than summing them into a single scalar.

1. Coordinate-Wise Inner Branch (p(𝑔)

  • The inner branch retains coordinate-wise products rather than summing them into a single scalar.Source: §1 · p. 5

2. Complete Within-Group Wedge Branch (p(𝑔)

Subspace Clifford interactions retain coordinate-wise diagonal products and complete within-group wedge coordinates, with group size controlling output width. The design extends to hierarchical stages and a fused CUDA implementation, while reducing post-projection cost relative to five-shift rolling.

3.3.3 Tensor Interpretation of the Group Representation

  • Each group retains its k coordinate-wise products and all k choose 2 antisymmetric wedge coordinates, rather than collapsing the diagonal products to one scalar.The diagonal products receive SiLU activation before the output projection.Source: §3.3.3 · p. 6

3.4 Block Design and Hierarchical Multi-Resolution Architecture

  • The block separates channel-wise spatial diffusion from nonlinear cross-channel reaction, applying the spatial sub-step before the subspace interaction.The optional spatial sub-step uses a 5×5 depthwise convolution, batch normalization, and SiLU.Source: §3.4 · p. 6
  • Post-projection cost falls from 10D^2 in five-shift rolling to 1.5D_tilde–2.5D_tilde for grouped interactions.Table 1 compares per-token interaction complexity; its Post-Proj MACs count only the contraction projection.Source: Table 1 · p. 7, §3.4 · p. 7
  • The hierarchical backbone uses configurable stage widths and depths, halves spatial resolution between stages, and permits group size k to vary by stage.Early high-resolution stages may use k=2, while deeper stages may use k=3 or 4.Source: §3.4 · p. 7

3.5 Fused CUDA Kernel Optimization

  • A fused templated CUDA kernel evaluates grouped interactions and recomputes local products in the backward pass instead of storing elementwise intermediates in global memory.Threads load each group’s inputs into thread-local arrays and write the interaction output once.Source: §3.5 · pp. 7–8

4 Experiments

On CIFAR-100, the experiments compare subspace and shift-based Clifford networks across isotropic and hierarchical architectures, assessing accuracy, model and training cost, depth, and the fused CUDA operator. Hierarchical subspace variants improve the accuracy–cost trade-off, while the fused kernel reduces operator time and peak extra memory.

4.2 Main Results and Comparison with Shift-Based CliffordNet

  • 78.32% accuracy: the 12-layer isotropic k=2 model uses 0.756M parameters and trains in 29.46 min, versus CliffordNet-Nano’s 77.82%, 1.4M parameters, and 57.64 min.The isotropic k=4 model reaches 78.46% with 0.952M parameters and trains in 31.54 min; it is 0.59% below CliffordNet-Lite but uses 63% fewer parameters and trains 1.85× faster.Source: §4.2 · pp. 8–9, Table 2 · p. 9
  • At k=3, Hier-Subspace-CAN reaches 80.12% Top-1 accuracy with 1.017M parameters, exceeding the 2.535M 4-shift hierarchical baseline by +0.59%, with 60% fewer parameters and 37% less training time (43.24 min vs. 68.68 min).Source: §4.2 · p. 9
  • 80.19% accuracy: Hier-Subspace-CAN (k=4) uses 1.120M parameters and 43.59 min, outperforming the 4-shift hierarchical baseline by 0.66%.The 24-layer variant reaches 81.21% accuracy with 2.209M parameters and trains in 68.97 min, improving accuracy by 1.68% over the 12-layer 4-shift baseline.Source: §4.2 · p. 9, Table 3 · p. 10
  • 81.21% accuracy: the 24-layer hierarchical model uses 2.209M parameters and trains in 68.97 min, versus 79.53%, 2.535M parameters, and 68.68 min for the 12-layer 4-shift baseline.The 18-layer model reaches 80.68% accuracy with 1.650M parameters in 57.12 min; stage widths remain fixed as depth increases.Source: §4.2 · p. 9, Table 3 · p. 10

4.3 Ablation Studies

  • 80.25%: the Hybrid (2, 4, 4) schedule slightly exceeds uniform k=4 accuracy while using fewer parameters and less training time.The paper attributes the reduced early-stage channel expansion to using k1=2 in the high-resolution first stage; the inverted Hybrid (4, 3, 2) reaches 79.80%.Source: §4.3 · p. 10, Table 4 · p. 10
  • 80.19% accuracy with both interaction branches, compared with 79.02% for inner-only and 78.49% for wedge-only configurations.The authors describe the coordinate-wise products and antisymmetric wedge differences as complementary features.Source: §4.3 · p. 10, Table 4 · p. 10
  • 80.19% accuracy with SpatialDiffusion, versus 79.02% without it; removing the module reduces training time by 28%.SpatialDiffusion adds 0.045M parameters, from 1.075M to 1.120M.Source: §4.3 · p. 10, Table 4 · p. 10

4.4 Operator Benchmark: Fused CUDA Kernel vs. Native PyTorch

  • 17.18× faster: for k=4, the fused CUDA kernel reduces combined forward–backward time from 48.67 ms to 2.83 ms and peak extra memory from 640.0 MiB to 320.0 MiB.Across tested configurations, the maximum absolute difference from native PyTorch autograd is within 10^-7 to 10^-6.Source: §4.4 · pp. 10–11

5 Conclusion

Subspace Clifford Networks replace global cyclic channel shifts with grouped interactions between learnable 1×1 projections. On CIFAR-100, hierarchical variants improve the evaluated accuracy–cost trade-off, and the fused kernel speeds up the k=4 operator benchmark.

5 Conclusion

  • The reformulation uses groups of size k∈{2, 3, 4} to compute wedge coordinates and SiLU-activated coordinate-wise products, reducing intermediate expansion by up to 75% versus 5-shift rolling.The fused CUDA kernel achieves a 17.2× operator speedup and 50% reduction in peak extra allocated memory on the evaluated k=4 benchmark.Source: §5 · p. 11
  • On CIFAR-100, the 12-layer hierarchical model reaches 80.25% accuracy with 1.09M parameters, and the 24-layer variant reaches 81.21% while using fewer parameters than 12-layer shift-based baselines.The backbone is extended from single-resolution isotropic columns to multi-resolution hierarchical networks with configurable per-stage subspace dimensions.Source: §5 · p. 11

Related papers

Loading 2610.37226v1…