Learning Holographic Reduced Representations with Clifford Variational Autoencoders

1 Introduction

• Unstructured data embedding remains an open VSA problem, despite algebraic methods for embedding structured data and fractional-power encodings for real-valued data.The paper focuses on the Holographic Reduced Representations algebra.Source: §1 · p. 2
• Prior learned HRR-compatible features were used for supervised augmentation rather than learning the representation itself, while RAAMs encode and decode structured hierarchies.The introduction positions Clifford-VAE as learning embeddings for unstructured data within the HRR framework.Source: §1 · pp. 2–3
• The Clifford torus is the maximal product-of-spheres factorization, enforcing unit Fourier magnitudes that preserve exact HRR invertibility.Factors larger than S1 would permit magnitude variation within their subspaces and break exact invertibility.Source: §1 · p. 3
• The approach addresses the lack of perceptual grounding in randomly initialized high-dimensional symbols and embedding methods that disregard unitary-vector constraints.The introduction notes that random-symbol cognitive maps can represent a landmark's location without an implicit perceptual account of the landmark.Source: §1 · pp. 3–4
• Clifford-VAE replaces the Gaussian latent distribution with independent von Mises distributions and imposes conjugate symmetry before a discrete Fourier transform.The learned embeddings are intended to be compatible with HRR/FHRR vector symbolic algebras.Source: §1 · p. 4
• Clifford-VAE improves over the strongest baseline on CIFAR-10 by an average 29% and supports effective binding, unbinding, and bundling across experiments.It also reports best-in-class supervised performance on FashionMNIST with nℓ=100 and VSA-test performance better than canonical random HRR vectors, matching FHRRs.Source: §1 · p. 4

2.1 Mitigating Surface-Area Collapse of Hyperspherical Priors

• The unit hypersphere’s surface area peaks at d=7 and then decays toward zero, becoming vanishingly small beyond d≈17.The Clifford torus (S1)^(d−1) instead has total measure (2π)^(d−1), which grows with d; its independent circle concentrations avoid relying on one shared concentration parameter.Source: §2.1 · pp. 4–5, Figure 1 · p. 5

2.2 Semi-supervised Learning (k-NN)

• Clifford-VAE improves over Gaussian N on CIFAR-10 k-NN accuracy at every evaluated dimension and label budget, and is best-in-class on FashionMNIST at nℓ=100.The evaluations use k=5 neighbors on MNIST; Table 1 spans label budgets and latent dimensions, and vMF did not converge at d=128.Source: §2.2 · pp. 5, 7, Table 1 · p. 5, Table 4 · p. 7
• The Gaussian VAE attains the lowest CIFAR-10 generation FID, while Clifford-VAE degrades more gracefully with dimension than Power Spherical.At d=256, Clifford reconstructions are qualitatively comparable to the evaluated Gaussian baselines despite the unit-magnitude constraint on Fourier coefficients.Source: §2.2 · p. 8, Table 5 · p. 8, Figure 3 · p. 8

2.3 VSA Operation Benchmarks

• Clifford-VAE matches unitary vectors on bundle capacity and iterated self-binding; other VAE embeddings never match its self-binding performance.For role-filler capacity, it outperforms random vectors at lower dimensions, but methods converge as d increases beyond 512.Source: §2.3 · p. 10, Figure 6 · p. 10, Figure 7 · p. 11
• Clifford-VAE’s associative-memory decoding accuracy is comparable to FHRR/unitary HRR vectors, with encoding cost matching the HRR binding operator.In Fourier space, the stated complexity is O(kn). The comparison does not account for the single VAE-encoder forward pass needed to produce a vector.Source: §2.3 · p. 11, Table 6 · p. 11

3 Discussion

• On CIFAR-10, Clifford-VAE beats the strongest baseline (N) across evaluated dimensions and sample sizes by 29% on average, ranging from +13.4% at d = 4096 to +48.4% at d = 128.The gains are generally most prevalent with nℓ=100 labeled samples; additional labels close the gap between the method and baselines.Source: §3 · p. 11
• On FashionMNIST, Clifford-VAE is best-in-class with nℓ=100, improving by ≈4.1%, and remains within ±2% of competitors at higher label budgets.Source: §3 · p. 11
• On MNIST, Clifford-VAE matches or exceeds the standard VAE; hyperspherical variants decline as dimensionality increases, while at d = 128 it outperforms all tested priors.The performance gap relative to the strongest baseline narrows as d increases, and all models improve with more labeled samples.Source: §3 · pp. 11–12
• Clifford-VAE has worse image-reconstruction FID than the standard VAE, but is more robust to dimensionality increases than Power Spherical VAE.The dimensionality robustness advantage was not preserved under iterated VSA operations; the authors suggest the embedding may trade reconstruction fidelity for robustness to algebraic-operation noise.Source: §3 · p. 12
• Random HRR symbols can separate conceptual from perceptual processes, whereas learned embeddings are semantically meaningful and decodable, with potential use in cognitive models and practical HDC systems.Source: §3 · p. 12
• The evaluation covers MNIST, FashionMNIST, and CIFAR-10 and benchmark VSA tasks, not more sophisticated downstream machine-learning tasks or VSA computations.The unit-magnitude Fourier-coefficient constraint may limit tasks where varying magnitudes carry useful information; conjugate symmetry also doubles decoder input dimension without adding useful information.Source: §3 · p. 12
• Clifford-VAE combines HRR algebraic guarantees with learned probabilistic representations, addressing the representation stage while leaving hierarchy disambiguation and reusable structure composition for future work.A proposed next step is to constrain learned representations by their utility in VSA-defined functions as well as by the Clifford torus.Source: §3 · p. 12

4.1.1 Reconstruction Losses

• FashionMNIST and CIFAR-10 use L1 reconstruction loss, while MNIST uses BCE on dynamically binarized inputs.Training uses Adam for up to 500 epochs with early stopping on total loss.Source: §4.1 · p. 13, §4.1.1 · p. 13

4.1.2 KL Annealing

• MNIST uses a linear β warm-up from 0 to 1 over 100 epochs; FashionMNIST and CIFAR-10 use a triangular schedule with period 250, cycling from βmin = 0.1 to βmax = 1.0 after warm-up.The table defines cyclical scheduling as a triangular oscillation between βmin and βmax after warm-up.Source: §4.1 · p. 13, Table 7 · p. 13, §4.1.2 · p. 13

4.1.3 KL Divergence Derivation

• The posterior is a product of independent circular distributions, so its KL divergence against the uniform Clifford-torus prior factorizes into per-circle terms.The prior entropy is (d −1) log(2π), and the posterior differential entropy is computed in closed form.Source: §4.1.3 · p. 14, Table 8 · p. 14
• The derivation considers von Mises and Power Spherical circular factors, with the latter specialized to S1 and assigned a closed-form entropy.The Power Spherical normalizing constant is specialized to d = 2 for the circle.Source: §4.1.3 · p. 14

4.2 Sampling from the Clifford Torus

• The Clifford torus is the product manifold (S1)^(d−1), constructed from d−1 independent circle angles.The angles are represented as unit-vector pairs in R2 for the sampled representation.Source: §4.2 · p. 14
• The inverse Fourier transform yields real vectors whose non-DC Fourier coefficients have magnitude exactly 1.Conjugate symmetry is used because an unconstrained inverse Fourier transform of complex frequencies would produce complex-valued output.Source: §4.2 · p. 15
• Independent samples from von Mises or Power Spherical distributions provide the circle angles, which are assembled into a conjugate-symmetric frequency vector before inverse Fourier transformation.The Power Spherical and von Mises sampling schemes are reparameterizable so gradients flow to posterior parameters.Source: §4.2 · p. 15
• In the 3D projections, Gaussian samples fill the interior, normalized Gaussian and Power Spherical samples concentrate on the sphere, and Clifford constructions occupy the product-of-circles torus.Figure 8 compares Gaussian distributions with and without L2 normalization, Power Spherical, CT-vM, and CT-PS samples.Source: Figure 8 · p. 15, §4.2 · p. 16

4.3 Architecture

• MNIST uses an MLP with encoder dimensions 784 → 256 → 128 and a mirrored decoder, while FashionMNIST and CIFAR-10 use four residual blocks with channels [64, 128, 256, 512].The convolutional network maps 32×32 inputs to a 2×2 spatial map; the Clifford decoder receives 2d-dimensional input, versus d for other decoders.Source: §4.3 · p. 16

4.4 Supervised Learning Protocol (k-Nearest Neighbor)

• After unsupervised training, each encoder is frozen and its latent features are evaluated with k = 5 nearest neighbors using cosine distance, or Euclidean distance for Gaussian variants.Evaluation uses label budgets of 100, 600, and 1000 and reports held-out top-1 accuracy averaged over 20 MNIST trials or 30 FashionMNIST and CIFAR-10 trials per dimension.Source: §4.4 · p. 16

4.5 Statistical reporting

• Results report the mean and a bootstrap-estimated 95% highest-density interval, based on 20,000 resampled means.A best mean is marked a clear winner when its interval is disjoint from every other entry’s; overlapping intervals indicate statistical ties.Source: §4.5 · p. 16

• The authors received no funding for this work.Source: Funding Declaration · p. 16

• FashionMNIST is shown at d = 256 and d = 512 in Figures 9 and 10.Source: Figure 9 · p. 20, Figure 10 · p. 20
• FashionMNIST is shown at d = 1024 and d = 2048 in Figures 11 and 12.Source: Figure 11 · p. 20, Figure 12 · p. 20
• FashionMNIST is shown at d = 4096 in Figure 13.Source: Figure 13 · p. 21

• CIFAR-10 is shown at d = 256 and d = 512 in Figures 14 and 15.Source: Figure 14 · p. 21, Figure 15 · p. 21
• CIFAR-10 is shown at d = 1024 and d = 2048 in Figures 16 and 17.Source: Figure 16 · p. 21, Figure 17 · p. 22
• CIFAR-10 is shown at d = 4096 in Figure 18.Source: Figure 18 · p. 22

• Figure 19 presents MNIST VSA benchmarks for d ∈ {2, 5, 10}.Source: Figure 19 · p. 23
• Figure 20 presents MNIST VSA benchmarks for d ∈ {20, 40}.Source: Figure 20 · p. 24