PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrieval-Centric Deep Learning in Growing Nonparametric Neural Networks

Maximilian Schlegel, Rajai Nasser, Seijin Kobayashi, Yanick Schimpf, Oliver Sieberling, Robert Obryk, Kazuki Irie, João Sacramento, Johannes von Oswald

arXiv:2610.03858v1cs.LGcs.AI

TL;DR

The paper asks how to replace linear attention in the dual form of gradient-trained neural networks with more powerful attention while retaining principled learning rules. It derives functional-gradient rules for kernelized growing networks and relates advanced linear-attention variants to optimizers for conventional networks. It reports promising performance and sample efficiency for RCDL, and a hybrid method that matches nonparametric performance at a small fraction of its size.

  • Problem

    Replacing the linear attention underlying deep networks with more powerful attention functions is an open question, and directly inserting softmax does not yield principled optimization because it invalidates the linear case’s optimal key-value representations.

    Source: §1 · p. 2

  • Method

    The paper derives functional-gradient learning rules for growing kernel-attention layers, including RBF and weighted-softmax kernels, and connects MesaNet and DeltaNet updates to conventional-network optimizers.

    Source: §1 · p. 2, §3 · p. 3, §3.1 · p. 4

  • Results

    RCDL shows promising performance and sample efficiency on image classification and teacher-student tasks, while the hybrid algorithm matches nonparametric performance at a small fraction of its size.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results establish a principled retrieval-centric learning approach and connect advanced linear-attention variants to optimization of conventional fixed-size neural networks.

    Source: §1 · p. 2

    RCDL’s memory and compute requirements grow linearly with the number of training datapoints, which the authors describe as generally prohibitive.

    Source: §6 · p. 10

Abstract

from arXiv · show

We investigate a general-purpose layer for deep learning that, instead of compressing arbitrary-size training data into fixed-size weight matrices, stores a new pair of key-value representations for every data point during training, and retrieves and recombines these representations through an attention mechanism at inference time - resulting in a growing neural net (NN). While Irie et al. (arXiv:2202.05798) have put forward this perspective from the classic duality expressing any linear layer in a deep NN trained by gradient descent as linear attention (LA) over the training data points, replacing LA by more powerful attention functions, as they suggest, turns out to be non-trivial: we show that naively applying learning rules from the LA case to advanced kernels does not lead to principled optimization. Here we fill this gap and develop functional gradient-based learning rules for kernelized attention layers, based on radial basis function (RBF) and softmax-like kernels - establishing the principled "retrieval-centric deep learning" (RCDL) paradigm. Empirically, we demonstrate the promising performance and learning-efficiency of RCDL on image classification and synthetic teacher-student learning tasks. Moreover, we show that replacing LA in the dual form of NNs by advanced LA variants, namely MesaNet/DeltaNet, yields a formal connection to recently proposed optimizers for conventional fixed-size NNs, offering a novel perspective on deep learning optimization.

1 Introduction

The paper asks whether the linear attention underlying deep networks can be replaced with more powerful attention, and develops retrieval-centric learning rules for growing networks. It also connects advanced linear-attention variants to conventional neural-network optimizers and explores hybrid capacity management.

1 Introduction

  • The paper extends the linear-attention mechanism underlying deep networks’ dual form because linear attention is considered suboptimal for sequence processing relative to softmax attention.The exact linear-layer duality stores training datapoints as key-value pairs and retrieves outputs through unnormalized dot-product attention [1].Source: §1 · pp. 1–2
  • RCDL derives learning rules for growing networks using positive-definite kernels and a proposed weighted-softmax kernel, with promising performance and sample efficiency on image classification and teacher-student tasks.Directly inserting standard softmax does not preserve the optimal key and value representations from the linear case.Source: §1 · p. 2
  • Replacing linear attention with DeltaNet and MesaNet updates yields optimizers connected formally to a recently proposed optimizer and second-order optimization, with competitive performance to Muon.Source: §1 · p. 2
  • Because RCDL’s model size grows at every step with a nonlinear kernel, the paper presents a hybrid rule that matches the nonparametric learner’s performance at a small fraction of its size.The hybrid combines parametric gradient updates with model-growing steps triggered when layer inputs are sufficiently distinct from past ones.Source: §1 · p. 2

2 Background: Dual Form of a Linear Layer

A gradient-trained linear layer can be expressed as retrieval over its training history: each training input and its error signal contribute a key-value pair, and inference applies linear attention to a query.

2 Background: Dual Form of a Linear Layer

  • Gradient descent updates the layer’s weight matrix by adding the outer product of the error signal and the current input at each training step.Source: §2 · p. 3
  • Unfolding these updates expresses an arbitrary query’s output as the initial-weight output plus a sum of error signals weighted by input-query dot products.Source: §2 · p. 3
  • The sum is unnormalized linear attention, with training inputs as keys, error signals as values, and the new input as query.Source: §2 · p. 3
  • This dual view suggests that improving the simple linear-attention mechanism could improve a fundamental mechanism shared by deep learning systems.Source: §2 · p. 3

3 Growing Nets with Non-Linear Attention

The paper replaces linear attention with kernels that cannot be reduced to finite-dimensional linear maps, creating growing layers whose key-value representations require principled learning rules. It develops such rules for positive-definite kernels and weighted-softmax attention.

3 Growing Nets with Non-Linear Attention

  • Nonlinear kernels that are not reducible to finite-dimensional maps produce growing layers, so the key-value representations must be chosen by a principled loss-minimizing learning algorithm.Source: §3 · p. 3

3.1 Positive Definite Kernel Attention

  • For a positive-definite kernel, gradient descent in its implicit feature space is equivalent to appending a key-value pair, with the input as key and the negative learning-rate-scaled output gradient as value.The experiments use the radial basis function kernel.Source: §3.1 · p. 4
  • For positive-definite kernels, the rule retains the linear case’s key and value definitions and provides loss-minimizing updates in the RKHS.Source: §3.1 · p. 4

3.2 Softmax and Weighted Softmax Attention

  • Standard softmax complicates infinitesimal database growth because adding a key changes the denominator; the paper introduces weighted softmax so new key-value pairs can be added smoothly and derives a growing-database formulation.Source: §3.2 · p. 4

Appendix B.1 provides further remarks on the functional space of weighted softmax functions.

The appendix describes the functional space underlying weighted softmax updates, addresses negative update weights, and presents a hybrid rule for reducing model growth. It also connects advanced linear-attention formulations to conventional neural-network optimization.

Appendix B.1 provides further remarks on the functional space of weighted softmax functions.

  • A Gaussian-mixture density over key-value pairs induces the same discrete weighted-softmax retrieval function, with predictions independent of the mixture variances under the stated covariance conditions.The appendix relates adding a key-value pair to adding an exponential basis function in the functional weight space.Source: Appendix B.1 · pp. 4–5
  • Functional gradient descent on the weighted-softmax objective adds a database tuple per observation; a mini-batch adds distinct tuples centered at its queries.The new key is a scaled query, while weight and value depend on prediction error and local density.Source: Appendix B.1 · pp. 5–6

3.2.1 Handling negative weights: clipping and gauge invariance

  • When an update would create a negative weight, clipping can enforce non-negativity, while gauge invariance offers an alternative offset that guarantees a positive new weight.Source: §3.2.1 · p. 6

3.3 Sublinear Growth with Hybrid Non-Parametric and Parametric Learning

  • The hybrid rule optimizes stored pairs parametrically and grows the database only when a new key is farther than threshold τ from existing keys, curbing memory growth while preserving fully nonparametric performance.Source: §3.3 · p. 6

4 Advanced Linear Attention Perspectives

  • Replacing linear attention with the mesa-layer yields the FOOF optimizer, whose inverse-matrix factor whitens inputs and gives an approximate second-order update for deep networks.The paper also considers DeltaNet as an additional reference and describes its update as online gradient descent on a layer-local, time-dependent loss.Source: §4 · p. 7

5 Experiments

Experiments test growing nonparametric layers and related optimizer ideas across synthetic regression, CIFAR classification, and mixed or hybrid setups. They report strong benchmark and one-pass results, alongside capacity and optimization constraints.

5 Experiments

  • On CIFAR-10/100, growing weighted softmax and RBF outperform several standard baselines, while FOOF performs best on CIFAR-100 and Muon achieves top accuracy on CIFAR-10.FOOF is on par with the growing variants on CIFAR-10; the growing models use plain SGD.Source: Figure 2 · p. 8, §5 · p. 9
  • In teacher-student regression, the overparameterized Reference MLP performs best, while Growing Weighted Softmax reaches significantly lower loss than the parametric softmax baseline.The parametric baseline uses a very wide MLP with width matching the final number of training samples.Source: §5 · p. 8, Figure 1 · p. 8
  • In a single pass over non-augmented CIFAR-10, the growing methods outperform all parametric models, including Muon and Mesa.The figure also compares their accuracy with kNN and SVM results from methods that are not single-pass.Source: Figure 2 · p. 8, §5 · p. 9
  • On CIFAR-10, adding hidden layers improves GWS performance before it saturates at two, temperature effects are non-monotonic, and linearizing the exponential causes a severe performance drop.The post-training ablation supports operation in the nonlinear regime.Source: §5 · p. 9, Figure 3 · p. 9
  • Compared with parametric softmax, GWS fits the training data faster and reaches higher test accuracy without the widening train-test gap found across tested parametric configurations.The parametric model was trained for 2000 epochs versus 50 for the growing model.Source: §5 · p. 9, Figure 4 · p. 9
  • After one CIFAR-10 epoch, a ViT with RCDL feed-forward blocks reaches 54.4% test accuracy, versus 48.4% for the purely parametric ViT.Other layers, including self-attention, remain parametric.Source: §5 · p. 10
  • The hybrid layer makes memory growth sublinear while preserving strong one-epoch CIFAR-10 performance, with test accuracy degrading gracefully as capacity is compressed.The hybrid combines parametric updates with conditional nonparametric growth.Source: Figure 5 · p. 9, §5 · p. 10

6 Discussion, Related Work, and Conclusion

  • RCDL is an end-to-end cascade of kernel machines and may support continual learning by preserving activations, but its memory and compute grow linearly and advanced optimizer rules remain unknown.The discussion identifies deriving rules for optimizers such as Adam and Muon as an open challenge.Source: §6 · p. 10

A Derivation of the Growing Layer Algorithm for Positive Definite Kernels

This appendix derives growing-layer database updates from functional gradient descent and implicit RKHS optimization, extending the constructions to weight decay, momentum, and weighted softmax. It also addresses limitations of naive softmax updates and develops approaches for handling negative weights while maintaining strictly positive updates.

A.1 The Growing Kernel Layer; A.2 The Implicit Weight Matrix View

  • A positive-definite-kernel layer stores key-value pairs and is equivalent to a finite-rank linear operator over infinite-dimensional RKHS features; composing layers with nonlinearities forms deep networks.Source: §A.1 · p. 16, §A.2 · p. 16

A.3 Derivation of the Learning Algorithm

  • SGD on an implicit infinite-dimensional weight matrix is equivalent to appending each batch input as a key and its scaled negative gradient as a value.Source: §A.3 · p. 17

A.4.1 Weight Decay and Database Pruning

  • Weight decay scales every stored value by (1 − γ) at each step, so pairs whose values fall below a threshold can be pruned to bound database size.Source: §A.4.1 · p. 17

A.4.2 Momentum

  • Momentum can be computed exactly without storing per-pair velocity vectors by using each pair’s creation timestamp to calculate its coefficient during retrieval.Combining momentum with weight decay decays stored values for regularization and scales the retrieval output for momentum.Source: §A.4.2 · pp. 17–18

B.1 Remark: the functional space of weighted softmax functions

  • For a finite weighted-softmax database, entries sharing a key can be combined into one tuple with summed weights and the corresponding weighted value.The weighted-softmax function space requires the weight measure’s moment-generating function to be finite everywhere.Source: §B.1 · p. 18

B.2 Continuous softmax function induced by the Gaussian mixture density & Discrete weighted softmax function

  • The continuous softmax induced by a Gaussian-mixture density is exactly equivalent to the discrete weighted-softmax function defined by its database.Source: §B.2 · p. 18

B.3 Further discussion on the standard (unweighted) softmax case

  • A naive free-value gradient update for standard softmax fails global function preservation because adding a key changes the denominator for other queries.The derivation also requires the post-update output gradient, creating a circular dependency in the general case.Source: §B.3 · p. 20

C Functional Gradient Descent in the Weighted Softmax Space

  • Functional differentiation in the weighted-softmax space yields a database-tuple update; its denominator term corresponds to a new key Σ_k q_t, and its numerator term depends on Σ_v.The derivation establishes Fréchet differentiability and represents the functional gradient using the Riesz theorem.Source: C Functional Gradient Descent in the… · p. 20, §C.2 · p. 21, §C.3 · p. 23

C.4 Constructive Update Analysis

  • Functional gradient descent in Hσk,σv is constructively equivalent to growing the database with specific, tractable update rules.Source: §C.4 · p. 23

C.5 Negative Weights and Numerical Stability

  • Negative weights arise when the prediction and loss gradient have negative inner product, but can make the softmax denominator approach zero; clipping is proposed for robustness.Source: §C.5 · p. 24

C.6 Functional Gradient Descent in the Decoupled Weight Space

  • Decoupling numerator and denominator weight functions makes nonnegative denominator clipping a valid descent step while preserving the numerator’s ability to reduce error.Source: §C.6 · p. 25

C.7 Gauge-Invariance: Another Method for Solving the Negative Weights Issue

  • Clipping discards the denominator update when the prediction-gradient inner product is negative, motivating an augmented, gauge-invariant alternative.Source: §C.7 · p. 25

C.7.1 Augmented Functional and Gauge Symmetry

  • Adding a global bias yields an exact symmetry: shifting value vectors and the bias together leaves the augmented retrieval function unchanged.Source: §C.7.1 · pp. 25–26

C.7.2 Gauge Fixing for Strictly Positive Weight Updates

  • Dynamically choosing a virtual gauge makes the new weight strictly positive for any nonzero error gradient, without clipping.The new weighted value vector points along the negative error gradient.Source: §C.7.2 · p. 27

C.7.3 Canonical Return and Locality of the Database Update

The gauge transformation returns the database to its canonical zero-bias form while leaving all historical tuples unchanged. The accompanying algorithms append one tuple per update and choose a hyperparameter scaling that makes weight changes proportional in order to the gradient length.

C.7.3 Canonical Return and Locality of the Database Update

  • The gauge-invariant update is local: it appends one tuple while leaving every previously stored tuple unchanged.Returning from the virtual gauge to the canonical gauge does not require modifying historical database entries.Source: §C.7.3 · pp. 27–28

C.7.4 Algorithm Summary

  • Algorithm 1 computes a strictly positive weight, a canonical weighted value, and a new key before appending the tuple to the database.Its inputs include the current database, query, loss gradient, learning rates, gauge parameter, and temperature.Source: §C.7.4 · p. 28

C.7.5 Hyperparameter Choice that Yields Changes in the Order of the Gradient Length

  • Setting α_t := α||g_t|| changes the weight update from quadratic to linear order in the gradient length.The stated motivation is to make added updates be in the order of the gradient length.Source: §C.7.5 · pp. 28–29

D Optimization Dynamics in the Growing Database

The section extends optimization dynamics for growing databases to weight decay, momentum, and their combination, then connects linear-attention updates to DeltaNet and MesaNet formulations. Decay enables pruning of low-influence memories, while momentum can be implemented without explicit velocity buffers.

D. Optimization Dynamics in the Growing Database

D.3 Combined Dynamics

D.1 Weight Decay

D.2 Momentum

  • Momentum can be implemented without storing an explicit velocity function by recording each tuple’s creation time and scaling its retrieval contribution.The resulting growing network can benefit from accelerated convergence without the storage overhead of explicit velocity buffers.Source: D Optimization Dynamics in the Growing… · p. 29, §D.2 · p. 30

D.3.1 Setup and Update Equations

  • The combined recurrence uses a momentum coefficient, a per-step weight-decay factor, and a learning rate to update the velocity and function state.The velocity is initialized to zero, and the function state begins at its initial value.Source: §D.3.1 · p. 31

D.3.2 Derivation of the Closed-Form State

  • The derivation unrolls the velocity and state recurrences to express the current function state directly from the history of gradients.It defines effective momentum and learning-rate quantities and derives a compact closed-form state expression.Source: §D.3.2 · pp. 31–32

E.1 DeltaNet

  • For DeltaNet, the update can be interpreted as gradient descent preceded by targeted forgetting along the current activation direction.The update also corresponds to gradient descent on a layer-local, time-dependent loss.Source: §E.1 · p. 32

E.2 MesaNet: Problem formulation and connection to FOOF

  • The Mesa layer is formulated as online minimization of a time-discounted objective, with a regularization center given by an exponential moving average of past weights.The section presents a modified Mesa formulation with a decay factor and an additional regularization center.Source: §E.2 · pp. 32–33

E.3 Derivation of the Recursive Update

The derivation expresses the Mesa update recursively and identifies a consistency condition under which its residual vanishes. Two regularization regimes meet that condition: a static version matching FOOF and a cumulative version used in practice.

E.3 Derivation of the Recursive Update

  • The Mesa update is derived by expanding recursive sufficient statistics to express the current solution from the previous state and current observation.The derivation treats both the matrix and vector statistics recursively.Source: §E.3 · p. 33, §E.3.2 · p. 33

E.3.3 Sufficient Condition for the Delta Rule

  • The residual vanishes if and only if the regularization-center update satisfies the stated consistency condition.The text says this condition is satisfied by definition.Source: §E.3.3 · pp. 33–34
  • Under that condition, setting the target to the previous prediction minus the scaled output gradient yields the stated database-state update.The update is described in terms of the state W and Φ.Source: §E.3.3 · p. 34

E.4 Proposed Algorithms

  • The authors identify two distinct regimes that satisfy the consistency condition.The passage introduces these regimes as proposed algorithms.Source: §E.4 · p. 34, §E.3.3 · p. 34

E.4.1 Algorithm A: Static Regularization

  • With static regularization, Algorithm A corresponds exactly to the FOOF optimizer.Source: §E.4.1 · p. 34

E.4.2 Algorithm B: Cumulative Regularization

  • The static approach can grow the norm of Φ and reduce the prior’s importance, so the authors propose cumulative regularization and use it in practice.The cumulative strength is defined recursively, and the resulting objective is an averaged weighted loss.Source: §E.4.2 · p. 34

F.1 Tasks and Model Selection

The experiments cover CIFAR image classification and synthetic teacher-student regression, with reported results averaged over five seeds. Hyperparameter selection is generally tailored to each plotted metric, with a specific shared selection for Figure 2.

F.1 Tasks and Model Selection

  • Experiments report means and standard deviations over five seeds, with comparable shared hyperparameter ranges for Figure 2 and additional method-specific parameters.For other figures, hyperparameters are selected from subsets relevant to the result and detailed in the corresponding model subsections.Source: §F.1 · p. 35
  • For most plots, hyperparameters are selected separately to optimize the displayed metric; Figure 2 shares runs selected to maximize final test accuracy.That selection can differ from runs that minimize training loss, which may achieve lower loss but potentially worse generalization.Source: §F.1 · p. 35

F.1.1 Image Classification

  • CIFAR-10 and CIFAR-100 classification uses normalized, augmented 32 × 32 RGB images flattened to 3072-dimensional vectors.Training augmentation includes reflect-padding and random crops, plus epoch-wise horizontal flips; test images are normalized without augmentation.Source: §F.1.1 · p. 35

F.1.2 Teacher-Student Regression

  • The synthetic regression tasks use unit-sphere inputs and targets from a randomly initialized nonlinear residual MLP teacher.The reported setup uses input and hidden dimensions of 100, two residual hidden blocks, and scalar outputs.Source: §F.1.2 · p. 35

F.2 Model Architecture

The experiments use a shared ResNet-MLP backbone and instantiate its blocks with growing, hybrid, or fixed-size alternatives. The supplied passages describe these configurations and evaluation settings, but provide no comparative performance values for this section.

F.2 Model Architecture

  • The shared backbone is a pre-normalization ResNet-MLP whose residual blocks use dense, Delta, Mesa, RBF, or Weighted Softmax modules.The block output is added to the residual stream, followed by final normalization and an output block.Source: §F.2 · pp. 35–36

F.3 Weighted Softmax

  • Growing Weighted Softmax adds tuples to its database in batches, clips negative weights to zero, and uses separate learning rates for weights and values.For numerical stability, the described update drops division by the previous database normalization.Source: §F.3 · p. 36

F.3.1 Gauge-Invariant Weighted Softmax

  • Gauge-Invariant Weighted Softmax chooses a gauge offset so each added weight is nonnegative by construction, avoiding the clipping step.It otherwise shares the database, initialization, and prediction of Weighted Softmax.Source: §F.3 · p. 36, §F.3.1 · p. 37

F.4 Hybrid Weighted Softmax

  • Hybrid Weighted Softmax trains stored tuples by backpropagation and adds a query only when its nearest stored key is farther than threshold τ.The method clips δ to nonnegative values for stability and initializes the database with zero keys and values.Source: §F.4 · p. 37

F.5 Parametric Softmax

  • Parametric Softmax keeps a fixed-size key-value parameterization and updates it jointly by gradient descent rather than growing the database.In the teacher-student setup, training weighted-softmax parameters caused severe instability, so the reported baseline uses unweighted softmax.Source: §F.5 · pp. 37–38

F.6 RBF

  • The RBF experiments evaluate RCDL modules on CIFAR-10 and CIFAR-100 using the shared ResNet-MLP backbone.Source: §F.6 · p. 38

F.7 Mesa (FOOF)

  • The Mesa update is evaluated on CIFAR-10 and CIFAR-100 by applying it to the dense layers in the ResNet-MLP backbone.The matrix inverse is implemented with Newton-Schulz iteration; the supplied passage says CIFAR generalization saturates with 10 iterations.Source: §F.7 · p. 39, Table 8 · p. 40

F.8 Delta

  • The Delta rule is evaluated on CIFAR-10 and CIFAR-100 by applying its weight update to dense layers in the ResNet-MLP backbone.Source: §F.8 · p. 40

F.8.1 Additional Experimental Results: Extended training for deep networks with Delta regularization

Extended CIFAR-10 training finds that DeltaNet-like decay improves generalization more than multiplicative weight decay. The section also describes MLP and nonparametric baselines used for comparison.

F.8.1 Additional Experimental Results: Extended training for deep networks with Delta regularization

  • 74% vs. 67% final test accuracy: Delta rule decay substantially outperforms plain SGD, while weight decay yields only marginal generalization improvement.Both decay mechanisms slow the increase in training accuracy; combining them adds no improvement.Source: §F.8.1 · p. 41, Figure 6 · p. 42, Figure 7 · p. 42

F.9 MLP Baselines

  • On CIFAR-10, modest DeltaNet-like decay improves test accuracy and reduces the train-test gap, whereas multiplicative decay improves generalization only marginally.The comparison uses a 10-layer residual MLP trained with SGD.Source: Figure 6 · p. 42, Figure 7 · p. 42

F.10 Nonparametric Baselines

  • The nonparametric baselines for Figure 2/Right include kernel ridge regression (LS-SVM) on one-hot targets and k-nearest neighbors.The kernel ridge model uses a Gram matrix and predicts by evaluating the kernel against training images.Source: §F.10 · p. 42, §F.10.1 · p. 43

F.10.1 Kernel ridge regression (least squares (LS-)SVM)

  • Kernel ridge regression uses channel-wise normalized, flattened RGB images and sets its bandwidth using a median-distance heuristic.The sample median is approximated from pairwise distances among 2048 subsampled images.Source: §F.10.1 · p. 43

F.10.2 k-Nearest Neighbors (kNN)

  • The kNN baseline tests Euclidean distance and unnormalized dot-product similarity, with uniform or similarity-based neighbor weighting.The reported Figure 2/Right result comes from sweeping the similarity metrics, weighting choices, and k values 5, 10, 15, 20, and 25.Source: §F.10.2 · p. 44

F.11 Vision Transformer Experiments

The section evaluates a Growing Weighted Softmax Module in a ViT and discusses the memory and computational costs of growing modules. Weighted Softmax and RBF training loss decreases faster than that of other models.

F.11 Vision Transformer Experiments

  • In a single-epoch CIFAR-10 experiment, the Weighted Softmax ViT replaces each block’s feed-forward network with a Growing Weighted Softmax Module.The remaining components use standard parametric learning; the comparison model uses standard softmax attention and feed-forward networks.Source: §F.11 · p. 44

F.12 Computational considerations

  • A 3072-by-3072 float32 growing layer adds 2.5 × 10^6 tuples and uses 61.5GB in the reported CIFAR experiments.The 50-epoch CIFAR experiments shard the database across four TPUs; the 100-dimensional teacher-student setup uses approximately 8GB and needs no sharding.Source: §F.12 · p. 45
  • Growing-module updates avoid computing gradients for the large databases, unlike Parametric Weighted Softmax, whose equally sized gradients add memory and training cost.Advanced optimizers such as Adam add first and second moments, further increasing expense and slowing training relative to growing Weighted Softmax.Source: §F.12 · pp. 45–46
  • Figure 8 shows Weighted Softmax and RBF training loss decreasing much faster than that of the other models.Figure 9 plots test accuracy for nonparametric methods as a function of training samples seen.Source: Figure 8 · p. 46, Figure 9 · p. 46

G.1 Training loss on CIFAR-100

The section reports the CIFAR-100 training-loss comparison and similar final test accuracies for Gauge-Invariant Weighted Softmax, standard Weighted Softmax, and RBF. It then situates RCDL relative to retrieval and growing-network approaches.

G.1 Training loss on CIFAR-100

  • For CIFAR-100, the paper reports the respective training-loss numbers in Figure 8, complementing its CIFAR-10 training-loss results reported in Figure 2/Middle.Source: §G.1 · p. 46

G.2 Gauge-Invariant Weighted Softmax

  • 59.4 ± 0.4% (Weighted Softmax), 58.8 ± 0.2% (RBF), and 58.4 ± 0.4% (Gauge-Invariant) are the reported final test accuracies.The Gauge-Invariant variant guarantees non-negative weights through its gauge choice rather than clipping; the methods perform comparably in this setting.Source: §G.2 · p. 46

H Further Related Work and Discussions

  • Unlike differentiable memory models, RCDL retrieves over training datapoints, accumulating key-value allocations as more training data is processed.The paper compares this growing behavior conceptually to Turing machines and transformer inference-time resource use.Source: H Further Related Work and Discussions · p. 47
  • RCDL differs from deep kernel learning, which uses a deep network to preprocess features for a single-layer Gaussian Process, and from label-retrieval Hopfield layers.The paper contrasts RCDL’s learned representations in a deep kernel machine with infinite-width theories’ question of representation learning versus lazy training.Source: H Further Related Work and Discussions · p. 47
  • RCDL grows through fine-grained key-value pairs, with each addition derived as a functional-gradient-descent step on the global loss.The paper contrasts this principled growth with approaches that add coarse-grained parametric modules or allocate units heuristically.Source: H Further Related Work and Discussions · p. 47
  • RCDL integrates retrieval within the network, contrasting with LLM retrieval used to complement knowledge in fixed-size weight matrices.The paper describes inference-time attention as recombining features stored in the growing network.Source: H Further Related Work and Discussions · p. 47
Loading 2610.03858v1…