PaperlayerView exact revision on arXiv

Source-linked AI summary

Superposed Latent Autoencoder

Quanling Zhao, Jiaying Yang, Tianqi Zhang, Ziyang Hao, Fatemeh Asgarinejad, Flavio Ponzina, Tajana Rosing

arXiv:2609.01158v1cs.LGcs.AI

TL;DR

Conventional autoencoders meet tight storage budgets by shrinking latents, potentially discarding representational capacity. SLAE instead stores multiple wider latents together through learned superposition and recovery, improving reconstruction and downstream classification at matched storage.

  • Problem

    Under limited storage, conventional autoencoders shrink latent representations, raising whether wider representations can be stored together without increasing per-example memory.

  • Method

    SLAE transforms wide latents into storage-friendly codes, binds them with randomized keys, superposes them in shared memory, and learns to recover each latent before decoding.

  • Results

    Across five image datasets, SLAE improves the reconstruction–memory tradeoff, reducing MSE by up to 56% and improving CIFAR-100 classification by up to 16.79 percentage points at matched storage.

  • Takeaways & Limitations

    The results support preserving wide representations and sharing storage as an alternative to independently shrinking each latent under tight memory budgets.

  • Takeaways & Limitations

    Exact recovery is generally impossible because K storage codes contain more degrees of freedom than the shared memory, leaving interference that recovery must suppress.

Abstract

from arXiv · show

Autoencoders typically meet tight latent-memory budgets by making each latent representation smaller, sacrificing representational capacity. We ask a different question: can multiple wider latents be stored together instead? We introduce the Superposed Latent Autoencoder (SLAE), which preserves high-capacity latent representations while sharing storage through learned superposition. SLAE transforms latents into storage-friendly codes, binds them with randomized keys, superposes multiple codes into a single memory tensor, and learns to recover each latent before decoding. Under the same storage budget, SLAE replaces irreversible dimensional bottlenecks with structured interference that can be suppressed. Across CIFAR-10/100, SVHN, STL-10, Tiny ImageNet, and a wide range of memory budgets, SLAE substantially improves the reconstruction--memory tradeoff, reducing reconstruction error by up to 56% over conventional autoencoders at matched storage. Further analysis shows that SLAE's advantage comes from making wider representations usable under the same storage budget. These gains also extend beyond reconstruction: the information preserved by SLAE improves downstream classification by up to 16.79 percentage points under the same memory budget. Our results suggest a new principle for representation compression: instead of making every latent smaller, keep representations wide and let them share memory.

1 INTRODUCTION

SLAE addresses the information bottleneck created when conventional autoencoders shrink latents to meet storage limits. It instead stores multiple wider representations together through learned superposition, improving reconstruction and downstream utility at matched storage.

  • Conventional autoencoders reduce latent dimensionality or spatial resolution under limited storage, sacrificing representational capacity.
  • SLAE lets multiple wider latents share one memory, keeping average storage per example unchanged while providing richer representations.
  • SLAE transforms latents into storage-friendly codes, binds them with keys, superposes them, and learns recovery before decoding.
  • SLAE replaces irreversible capacity loss with structured interference that storage and recovery networks learn to suppress.
  • 56% reconstruction error reduction and 16.79 percentage-point downstream classification improvement are reported at matched storage.

2 RELATED WORK

The related work situates SLAE among latent-capacity studies, information-superposition methods, and approaches that amortize representation storage across multiple examples.

  • Prior autoencoder research studies how latent dimensionality, architecture, and data structure affect compression quality.
  • SLAE preserves wider representations and amortizes their storage across multiple examples rather than further shrinking each latent.
  • Information superposition has roots in distributed representations and has been applied to shared neural parameters and lossy feature compression.

3 MOTIVATION AND PROBLEM FORMULATION

The paper asks whether shared storage can provide higher-capacity latents at the same per-example budget, especially when conventional latents are capacity limited.

  • When a latent is strongly capacity limited, adding dimensions produces large reconstruction gains, with diminishing benefit as expressiveness increases.
  • At budget B, a conventional example uses B scalars, whereas SLAE gives each example a wider latent containing KB scalars for K-way sharing.
  • Conventional autoencoders trade storage for capacity independently, while SLAE trades wider representations for interference from shared storage.
  • The study focuses on convolutional image autoencoders, while the underlying principle is described as applicable in principle to other latent representations.

4 SUPERPOSED LATENT AUTOENCODER

SLAE separates reconstruction-friendly latent representations from storage-friendly codes, then uses randomized binding, superposition, and learned recovery to share memory.

  • 4 SUPERPOSED LATENT AUTOENCODER: SLAE decouples reconstruction latents from storage codes through separate learned storage and recovery pathways.
  • 4 SUPERPOSED LATENT AUTOENCODER: Each input is encoded into a wide latent, adapted for storage, bound with a slot-specific key, and superposed into shared memory.
  • 4 SUPERPOSED LATENT AUTOENCODER: Randomized orthogonal binding preserves code information and norm while making other slots appear as weakly aligned interference after unbinding.
  • 4 SUPERPOSED LATENT AUTOENCODER: Only the shared memory is stored, so SLAE gives each example a wider latent at the same average storage cost as a narrower conventional latent.
  • 4 SUPERPOSED LATENT AUTOENCODER: Inverse binding retrieves a demixed code containing the target plus interference, and recovery maps it back to a decoder-friendly latent.
  • 4 SUPERPOSED LATENT AUTOENCODER: Training combines superposed reconstruction, latent recovery, clean-path anchoring, and storage-code decorrelation.

5 UNDERSTANDING LATENT SUPERPOSITION

SLAE’s analysis separates the benefits of wider latents from the costs of shared-storage interference. Randomized binding disperses interference across feature directions, while learned recovery determines whether useful information remains accessible.

  • SLAE recovers the desired storage code before learned recovery, while the other K −1 codes remain as cross-slot interference.
  • Exact recovery is generally impossible because K codes contain more degrees of freedom than the shared memory.The superposition map is non-invertible for arbitrary storage codes.
  • Randomized binding preserves the energy of each interfering code, so aggregate interference can remain comparable to or larger than the desired code.
  • For any fixed feature direction, randomized cross-slot interference has zero mean, reducing systematic directional bias.Its magnitude is characterized by variance, while learned storage and recovery networks exploit this structure.
  • Randomized binding disperses interference across high-dimensional channel space rather than making its total norm small.
  • SLAE improves when the capacity gained from a wider latent exceeds the superposition–recovery penalty.The strongest gains are expected when latent capacity limits performance and superposition remains recoverable.

6 EXPERIMENTS

Experiments evaluate SLAE across datasets, latent geometries, storage budgets, and superposition factors, finding improved matched-storage reconstruction and downstream utility in regimes where conventional latents are capacity-limited. Analyses attribute the gains to shared storage of wider representations rather than additional depth, while showing that stronger superposition increases recovery difficulty.

  • Reconstruction at matched storage: SLAE improves matched-storage reconstruction across all five datasets at B = 512, reducing MSE by 8.2–19.9%.Global best-under-budget comparisons report peak reductions of approximately 35%, 31%, and 56% on CIFAR-10, CIFAR-100, and SVHN, respectively.
  • Reconstruction at matched storage: SLAE improves over Plain at all evaluated storage budgets on CIFAR-10, CIFAR-100, and SVHN, while gains on STL-10 and Tiny ImageNet are more regime-dependent.Global comparisons show improvements at 13/18 and 10/18 budget points on STL-10 and Tiny ImageNet, with peak reductions of approximately 17% and 41%.
  • When does SLAE help: The realized gain balances capacity gain against recovery penalty: median gain changes from +7.7% at K = 2 to −12.3% at K = 8.Across the same configurations, capacity gain increases in 97.8% of cases from K = 2 to K = 8, while recovery penalty increases in all cases.
  • Source of gains: SLAE consistently outperforms the depth-matched Independent Bottleneck AE, isolating superposition as the key difference.The control uses comparable compression machinery and additional depth but stores examples independently.
  • Downstream utility: SLAE improves downstream CIFAR-100 classification over matched-storage Plain AE in 11 of 12 comparisons.The same CIFAR-style ResNet-18 and training protocol are used, with evaluation on the original clean test set.
  • Downstream utility: Downstream gains are generally largest under tighter memory budgets and diminish as the Plain representation becomes less capacity-limited.This pattern links reconstruction improvements to greater downstream utility.

7 CONCLUSION

SLAE compresses latent representations by keeping them wide and superposing multiple examples in shared storage instead of shrinking each latent independently. Across five image datasets, this improves reconstruction and preserves information useful for downstream classification under the same storage budget.

  • CONCLUSION: SLAE keeps latent representations wide and stores multiple examples together through superposition rather than shrinking each latent independently.This replaces irreversible capacity loss with a structured interference-and-recovery problem.
  • CONCLUSION: Across five image datasets, SLAE improves the reconstruction–memory tradeoff on all datasets, reducing MSE by up to 56%.The strongest gains occur under tight memory bottlenecks.
  • CONCLUSION: SLAE improves CIFAR-100 classification by up to 16.79 percentage points at the same storage budget.The result indicates that information preserved for reconstruction remains useful downstream.
  • CONCLUSION: The results suggest that preserving rich representations and sharing their storage can outperform shrinking each representation in isolation under tight memory budgets.The paper identifies potential relevance to memory-constrained representation learning and other memory-critical systems.

C COMPONENT STUDY

The component study evaluates SLAE modules under a fixed CIFAR-10 storage setting and finds that randomized binding, the storage adapter, and learned recovery each contribute to reconstruction quality. Slot conditioning has negligible effect in this setting.

  • Component contributions: Removing randomized binding increases MSE by more than 7× and reduces SSIM from 0.8845 to 0.6287.The result supports slot-specific randomized transforms as necessary for structuring cross-slot interference.
  • Component contributions: Removing the learned storage adapter increases reconstruction MSE by 10.2%.This supports separating the decoder-facing latent from the representation used for superposition.
  • Component contributions: Removing the learned recovery module degrades MSE by 2.6%, indicating benefits beyond inverse binding alone.The module provides learned post-demixing adaptation.
  • Component contributions: Removing slot conditioning has negligible effect in the evaluated CIFAR-10 setting.The shared recovery network appears able to handle slot-specific interference patterns without an explicit slot identifier.

D PARAMETER OVERHEAD

SLAE adds learned storage and recovery modules to preserve wider latents, with modest overhead in the regimes where its reconstruction benefits are strongest. Overhead rises at very large channel widths, while ablations indicate the gains are not explained solely by added parameters.

  • Architecture: SLAE adds learned storage-adapter and recovery modules beyond the shared encoder–decoder, while fixed randomized binding keys are excluded from parameter counts.The modules are compared against the corresponding Wide Plain AE with the same wide latent geometry.
  • Overhead: 6.7% is the median SLAE-specific learned parameter overhead across the full experimental grid.The overhead is measured as the additional learned parameters from the storage adapter and recovery network relative to the encoder and decoder.
  • Overhead: 4.3% additional learned parameters occur in the CIFAR-10 H = 2, K = 2, B = 128 setting, while Tiny ImageNet H = 4, K = 2, B = 512 incurs 6.1%.These settings use wider latents in tight-memory regimes where preserving capacity yields large reconstruction benefits.
  • Limitations: C = 256 configurations incur considerably higher parameter costs because the current recovery network scales with latent channel dimension.This identifies parameter-efficient recovery at very large latent widths as a direction for future work.
  • Ablations: A depth-matched Independent Bottleneck AE with comparable compression machinery does not reproduce SLAE’s reconstruction gains.This comparison indicates that the gains cannot be attributed solely to additional model capacity.
  • Ablations: Removing the storage adapter increases MSE by 10.2%, whereas removing the recovery network increases it by 2.6%; both variants retain most of full SLAE’s benefit.The binding–superposition–unbinding operations themselves are parameter-free.

H ROBUSTNESS TO CHECKPOINT-SELECTION PROTOCOL

The checkpoint-selection robustness analysis removes the evaluation split from model selection while retaining matched-storage comparisons. SLAE keeps its advantage across all four tested settings, including the comparatively modest Tiny ImageNet case.

  • Protocol: The robustness protocol excludes the evaluation split from checkpoint selection, using training data for model selection and evaluating the split only once after training.The repeated experiments retain the corresponding configurations and training setup from the main experiments.
  • Results: SLAE retains its matched-storage advantage in all four settings when evaluation-split reconstruction is not used for checkpoint selection.The tested settings include two CIFAR-10 superposition factors and one each for CIFAR-100 and Tiny ImageNet.
  • Results: 55.3% and 54.8% MSE reductions occur on CIFAR-10 for K = 2 and K = 4, respectively, with 52.8% on CIFAR-100 and 10.1% on Tiny ImageNet.These are relative MSE reductions of SLAE over Plain under the alternative checkpoint-selection protocol.
  • Additional reconstruction results: Figures 10–12 compare Plain AE and SLAE at matched average storage using SSIM and MSE across CIFAR-10/100, STL-10, Tiny ImageNet, and SVHN.The curves vary latent geometry H and superposition factor K ∈ {2, 4, 8}; higher SSIM and lower MSE indicate better reconstruction.
  • Capacity–interference tradeoff: Moderate superposition, K = 2, is generally the most reliable operating point, while larger K increases latent capacity but makes recovery more difficult.The capacity–interference decomposition explains this as a widening tradeoff between capacity gain and recovery penalty.

K DERIVATION OF RANDOMIZED-BINDING INTERFERENCE

Randomized binding does not remove interference energy; it disperses interference across channel directions. In high dimensions, this makes interference nearly orthogonal to fixed feature directions, while exact recovery still depends on learned recovery networks and empirical penalties.

  • Binding construction: SLAE binds each storage code with a randomized orthogonal transform built from permutation, sign, and normalized Walsh–Hadamard factors.The relative transform between slots becomes a randomized signed permutation.
  • Interference moments: Interference from multiple slots has zero mean along a fixed direction, with independent conditional terms allowing their variances to add.The derivation conditions on the retrieved slot key and uses independence across interfering keys.
  • Interference interpretation: Randomized binding does not eliminate interference energy; it randomizes its orientation and disperses its contribution across d = KC channel dimensions.The complete interference vector can remain large even when directional projections are small.
  • Interference interpretation: For fixed code energy, interference along a fixed feature direction has standard deviation decreasing as 1/√d, approximately preserving downstream inner products.This describes directional dispersion rather than a guarantee that the full interference vector has small norm.
  • Scope: The analysis characterizes randomized binding and unbinding for fixed storage codes but does not imply exact recovery or describe the nonlinear trained adapter and recovery network.Those effects are evaluated through the recovery penalty in the empirical analyses.
  • Memory efficiency: At matched quality, SLAE’s memory-reduction factor R compares the minimum evaluated Plain AE budget with the minimum SLAE budget; R = 2 means half the storage.R = 1 denotes parity, while R < 1 means SLAE requires more storage.
  • Memory efficiency: SLAE reaches several representative quality targets with a 2× storage reduction on CIFAR-10, CIFAR-100, and SVHN in the most constrained geometry.Savings diminish as H increases; STL-10 and Tiny ImageNet show mixed savings and mostly parity elsewhere.

M COMPARISON WITH ALTERNATIVE LATENT COMPRESSION STRATEGIES

SLAE preserves a wide latent while sharing storage, outperforming dimensional-compression baselines and complementing numerical precision reduction. At aggressive compression, combining moderate superposition with INT8 gives the strongest reported CIFAR-10 result among the compared strategies.

  • Experimental comparison: The comparison evaluates reduced-width autoencoders, PCA, quantization, SLAE, and combinations at matched per-image latent-memory bit budgets.Experiments use CIFAR-10 with H = 2 and a C = 64 Wide Plain latent as reference.
  • Dimensional compression: At 2× compression, K = 2 SLAE obtains MSE 0.00393, compared with 0.00522 for PCA and 0.00735 for the reduced-width AE.These correspond to MSE reductions of 24.7% versus PCA and 46.6% versus the reduced-width AE.
  • Dimensional compression: At 4× compression, K = 4 SLAE achieves MSE 0.00806, compared with 0.01053 for PCA and 0.01267 for the reduced-width AE.The advantage over dimensional-compression methods persists at the higher compression ratio.
  • Precision compression: FP16 at 2× and INT8 at 4× introduce negligible degradation relative to the FP32 Wide Plain reference, showing that quantization complements rather than replaces superposition.Quantization reduces bits per value, whereas SLAE reduces the number of independently stored representations.
  • Combined compression: At 8× compression, K = 2 SLAE with INT8 shared memory achieves MSE 0.00395, SSIM 0.88461, and PSNR 24.04 dB, a 37.3% MSE reduction versus direct INT4.Quantizing K = 2 shared memory from FP32 to INT8 changes MSE only from 0.00393 to 0.00395.
  • Combined compression: At the same 8× budget, the reported MSEs are 0.00395 for K = 2 + INT8, 0.00806 for K = 4 + FP16, and 0.01493 for K = 8 + FP32.The comparison shows the cost of assigning more compression burden to superposition.

N LIMITATIONS

SLAE’s strong reconstruction–memory tradeoff is accompanied by limitations involving aggressive superposition, recovery-network scaling, fixed superposition factors, binding choices, grouping, and evaluation scope. The discussion also presents superposition as a potentially complementary compression axis and identifies directions beyond image reconstruction.

  • N LIMITATIONS: K = 2 is the most reliable operating point, while K = 4 and K = 8 are beneficial only in selected regimes.More aggressive superposition provides wider latent representations but increases the recovery penalty.
  • N LIMITATIONS: The recovery architecture becomes parameter-expensive at very large channel widths, despite modest overhead in capacity-limited regimes.The overhead is modest where SLAE performs best but grows with very wide latents.
  • N LIMITATIONS: Each superposition factor is trained separately, preventing the current model from dynamically changing K at deployment.Variable K within a single model could enable more flexible memory allocation.
  • O DISCUSSION: Superposition is presented as an additional compression axis that can be composed with dimensionality, precision, redundancy, quantization, and coding-efficiency methods.The paper reports an initial demonstration in which moderate superposition combined with quantization improves the memory–quality tradeoff over aggressive reliance on either mechanism alone.
  • N LIMITATIONS: The current system uses fixed randomized orthogonal binding and random example grouping, leaving alternative binding and learned grouping strategies for future work.Alternative designs may improve interference separation or recovery, while learned grouping could reduce interference through more compatible storage codes.
  • O DISCUSSION: The principle may extend beyond image autoencoders to replay buffers, stored neural features, multimodal or language-model representations, and vector-memory systems.These settings may prioritize downstream task performance or retrieval fidelity rather than pixel-level reconstruction.
Loading 2609.01158v1…