PaperlayerView exact revision on arXiv

Source-linked AI summary

Nested Learning: The Illusion of Deep Learning Architectures

Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, Vahab Mirrokni

arXiv:2512.24695v1cs.LGcs.AI

TL;DR

Modern deep learning models face unresolved challenges in continual learning, self-improvement, and effective context-to-memory transfer. Nested Learning models learning systems as interconnected optimization levels and introduces CMS-based Hope, which improves continual learning and long-context reasoning while remaining competitive as a general backbone.

  • Problem

    Deep learning architectures provide expressive computation but do not generally let contextual information update long-term parameters, limiting continual acquisition of knowledge and skills.

    Source: §1 · p. 1, §1.1 · p. 3

  • Method

    Nested Learning models architectures and optimizers as multi-level associative memories, introducing expressive updates, Continuum Memory System, and the self-modifying Hope architecture.

    Source: §4.4 · p. 19, §7.2 · p. 29, §10 · p. 40

  • Results

    Hope improves continual learning and long-context reasoning capabilities while remaining competitive as a general backbone.

    Source: §9.1 · p. 36, §9.2 · p. 36, §10 · p. 40

  • Takeaways & Limitations

    Nested Learning reframes pre-training, in-context learning, and continual learning as context compression and reuse across levels and timescales.

    Source: §10 · p. 40

    Catastrophic forgetting is not solved in general, despite Hope and CMS reducing it in the empirically studied tasks.

    Source: §10 · p. 40

Abstract

from arXiv · show

Despite the recent progresses, particularly in developing Language Models, there are fundamental challenges and unanswered questions about how such models can continually learn/memorize, self-improve, and find effective solutions. In this paper, we present a new learning paradigm, called Nested Learning (NL), that coherently represents a machine learning model with a set of nested, multi-level, and/or parallel optimization problems, each of which with its own context flow. Through the lenses of NL, existing deep learning methods learns from data through compressing their own context flow, and in-context learning naturally emerges in large models. NL suggests a philosophy to design more expressive learning algorithms with more levels, resulting in higher-order in-context learning and potentially unlocking effective continual learning capabilities. We advocate for NL by presenting three core contributions: (1) Expressive Optimizers: We show that known gradient-based optimizers, such as Adam, SGD with Momentum, etc., are in fact associative memory modules that aim to compress the gradients' information (by gradient descent). Building on this insight, we present other more expressive optimizers with deep memory and/or more powerful learning rules; (2) Self-Modifying Learning Module: Taking advantage of NL's insights on learning algorithms, we present a sequence model that learns how to modify itself by learning its own update algorithm; and (3) Continuum Memory System: We present a new formulation for memory system that generalizes the traditional viewpoint of long/short-term memory. Combining our self-modifying sequence model with the continuum memory system, we present a continual learning module, called Hope, showing promising results in language modeling, knowledge incorporation, and few-shot generalization tasks, continual learning, and long-context reasoning tasks.

1 Introduction

The introduction argues that stacking layers has not solved continual learning, adaptation, computational-depth, and optimization challenges in deep models, especially LLMs. It presents Nested Learning as a unifying paradigm that models architectures and training as interconnected nested optimization problems with multiple context flows and update frequencies.

1 Introduction

  • Deep models gain expressive power from depth, but added layers may not improve computational depth, parameter capacity, optimization outcomes, or continual adaptation.These limitations motivate designs beyond simply stacking layers, particularly for fast adaptation, continual learning, and out-of-distribution generalization.Source: §1 · pp. 1–2
  • LLMs remain largely static after deployment, retaining new information only in immediate context while facing costly adaptation, weak generalization, or catastrophic forgetting beyond in-context learning.The introduction compares this limitation to anterograde amnesia: pretrained knowledge persists, but new long-term memories are not formed.Source: §1 · p. 2

1.1 Human Brain Perspective and Neurophysiological Motivation

  • Human continual learning motivates online memory consolidation, neuroplasticity, reusable neural structure, and multi-timescale processing, whereas current models use fixed test-time weights and largely uniform update rates.Brain oscillations organize computation across frequencies, with slower Delta and Theta activity associated with memory consolidation and learning.Source: §1.1 · pp. 3–4

1.2 Contributions and Roadmap

  • Nested Learning represents models and training as interconnected nested, multilevel, or parallel optimization problems, giving each component its own context and internal gradient flow.It treats learning algorithms and architectures as related concepts operating at different system levels, unifying knowledge transfer across meta-learning, in-context learning, recurrent networks, and hypernetworks.Source: §1.2 · p. 4
  • NL interprets gradient-based training and optimizers as associative memories compressing context or gradients, and introduces Delta Gradient Descent whose updates depend on both current inputs and weight state.The paper also presents a neural learning module in which architecture and optimization form a uniform system of components optimizing internal objectives at different frequencies.Source: §1.2 · p. 5
  • Continuum Memory Systems distribute memory across update frequencies, enabling fast adaptation, persistent knowledge, and partial recovery of forgotten information; evaluations test NL across continual learning, long-context, language, memory, and optimizer tasks.The paper further introduces Multi-scale Momentum Muon and reports evidence for continual learning, multilevel computation, and self-referential processes.Source: §1.2 · p. 5

2 Preliminaries

This section defines the paper’s notation and reviews gradient descent, meta-learning, fast weight programmers, and in-context learning. It frames these concepts for the later Nested Learning formulation, including a unified view of in-context learning as adaptation from context.

Notations

  • The paper represents inputs, memory states, keys, values, queries, memory-module parameters, and nested-learning levels using explicit notation, with memory modules implemented as residual MLPs.Nested levels are indexed by either level or update frequency.Source: §2 · p. 6

Gradient Descent

  • Gradient descent is presented as a standard optimizer whose update is equivalently steepest descent, a proximal step on a first-order objective approximation, or an FTRL formulation.The proximal view exposes an implicit bias toward small L2-distance moves.Source: §2 · p. 6

Meta Learning

  • Meta-learning automates model-design decisions through two-level optimization, with an outer procedure selecting parameters for an inner learner to maximize performance across tasks.The review notes both supervised and increasingly common unsupervised outer loops, including applications to sequence models.Source: §2 · p. 6

Fast Weight Programmers

  • Fast weight programmers are recurrent networks with matrix-valued short-term memory, whose slow programmer generates queries, keys, and values while online rank-one updates write the fast-weight matrix.Memory is retrieved by matrix–vector multiplication, yielding a compact learnable key–value memory with constant state size across time.Source: §2 · pp. 6–7

In-context Learning

  • The paper adopts a general definition of in-context learning as a model’s ability to adapt to and learn from a given context, connecting it to associative memory across architectures and objectives.This broadens the concept beyond Transformer models trained with next-token prediction.Source: §2 · p. 7

3 Nested Learning

This section presents Nested Learning as a framework for modeling neural systems as interconnected, multi-level optimization processes that compress context flows through associative memories. It further examines how nested optimization levels transfer knowledge and connect their outputs through distinct mechanisms.

3 Nested Learning

  • Nested Learning models neural systems as integrated sets of nested, multi-level, and potentially parallel optimization problems, each with its own context flow.The framework aims to reinterpret existing deep-learning components and optimization procedures within this representation.Source: §3 · p. 7, §3.1 · p. 8

3.1 Associative Memory

  • Associative memory maps keys to values, while learning acquires an effective mapping by optimizing its quality; the optimization compresses mappings into lower-dimensional parameters.Keys and values may be tokens, gradients, or other events, so the formulation connects memory, learning, and data compression.Source: §3.1 · pp. 7–8
  • Backpropagation in a linear layer can be viewed as associative memory that maps each data sample to its corresponding prediction error or local surprise signal.The gradient is zero when the loss is minimized, making it an error signal for the predicted output.Source: §3.1 · p. 8
  • Gradient descent is a 1-level associative memory, whereas momentum creates a 2-level process whose inner memory stores gradients before updating the slow weights.For linear attention, the inner memory update and outer projection-layer training likewise form a 2-level optimization process.Source: §3.1 · p. 9

3.2 Nested Optimization Processes

  • Nested systems order components by update frequency, with each level containing optimization problems and its own gradient flow; nested associative-memory systems specialize this structure to memory-learning processes.Stacking levels enables different abstraction and update frequencies and can support higher-order in-context learning, latent computation, multiple memories, or more expressive optimizers.Source: §3.2 · pp. 10–12
  • A neural learning module jointly represents architecture and optimization because their interaction determines model outputs, especially in continual settings without separate training and test phases.The model generates gradient data for its optimizer, while the optimizer changes subsequent model states, creating feedback between architecture and optimization.Source: §3.2 · pp. 12–13

3.3 Knowledge Transfer Between Levels

  • NL distinguishes parametric and non-parametric direct connections, conditioning a lower-frequency memory on higher-frequency parameters, outputs, or context without backpropagating across levels.In these variants, each block’s state is treated as a hyperparameter for the other; examples include linear Transformers/FWP and softmax attention.Source: §3.3 · p. 13
  • Backpropagation-based transfer places both levels in one gradient flow while allowing their states to update at different frequencies.The forward pass conditions one memory on the other, while the backward pass jointly optimizes both states; the paper connects this design to continuum memory systems.Source: §3.3 · pp. 13–14
  • Initialization-based transfer makes a higher-level problem learn an initial point that enables a lower-level problem to adapt rapidly, as in MAML (Finn et al. 2017).From the nested-learning perspective, the inner problem follows its own context and objective, while the higher-level problem learns the best initialization across possible lower-level contexts.Source: §3.3 · p. 14
  • Generation-based transfer lets one block generate another block’s weights or context, encompassing hypernetworks and optimization processes such as gradient descent, Adam (Kingma et al. 2014a), and AdaGrad.Hypernetworks generate target-network weights, whereas optimization processes receive gradients generated by the architecture as optimizer context.Source: §3.3 · p. 14
  • Designing a neural learning module requires choosing the optimization problems and their frequencies, then specifying how knowledge transfers between levels.Under different transfer choices, meta-learning, MAML, hypernetworks, and learned optimizers appear as instances of the broader nested-learning framework.Source: §3.3 · pp. 14–15

4 Optimizers as Learning Modules

This section reframes backpropagation and modern gradient-based optimizers as nested, self-referential associative-memory systems that compress error, gradient, or sample information while retaining state to control updates. It develops more expressive update rules for long-context optimization, continual learning, and state-dependent adaptation.

4.1 Backpropagation as an Associative Memory

  • Backpropagation compresses mappings from each layer’s input to its local error signal, forming an associative memory rather than simple linear attention on gradients.The update is self-referential because the associative-memory values are generated by the memory itself.Source: §4.1 · p. 15

4.2 Momentum-based Optimizers as Associative Memories

  • Momentum compresses past gradients through a two-level optimization process, while Adam, RMSProp, AdaGrad, and related optimizers can be reformulated as associative memories.The inner memory learns gradient statistics or mappings, and the outer process uses that state to update weights.Source: §4.2 · pp. 16–17

4.3 Long Context in Optimizers: An Example of Continual Learning with Orthogonal Tasks

  • With β=0.9, the last 6 gradients contribute at least 50% and the last 43 at least 99%, leaving global information beyond 43 steps below 1%.This limited memory can shift toward the current task’s gradient subspace, causing catastrophic forgetting because old subspaces are no longer retained.Source: §4.3 · p. 18

4.4 More Expressive Designs for Momentum as an Associative Memory

  • Delta momentum reaches the toy time-varying-curvature solution faster than standard momentum by using gradient-dependent weight decay to decay or stop memory when needed.The broader designs increase capacity through delta-rule objectives, nonlinear or higher-order mappings, and richer memory structures such as MLPs.Source: §4.4 · pp. 19–20, Figure 4 · p. 20

4.5 Going Beyond Simple Gradient Descent and Momentum

  • Delta Gradient Descent incorporates the previous weight state into updates, producing an adaptive decay term based on the current data sample.It replaces state-independent dot-product updates with an L2-regression objective, making the rule better suited to dependent contexts such as sequences.Source: §4.5 · p. 20
  • Gradient descent is self-referential rather than a conventional linear recurrence because its values depend on the memory state and thereby control the learning process.At each step, the generated value is produced from the memory and the current input.Source: §4.5 · p. 21
  • Generalized Gradient Descent treats backpropagation as a self-referential associative memory that compresses training samples and maps them to self-generated keys and values.Its mapping quality is measured by a loss, while a retention term keeps new-instance solutions near the current state.Source: §4.5 · p. 21
  • The same formulation extends to Generalized Momentum, while conventional momentum remains an associative memory whose keys and values are supplied by a lower-frequency level.The L2-regression formulation is identified as a special case of this broader framework.Source: §4.5 · p. 21
  • In continual learning, optimizer states retain knowledge about gradient distributions, the loss landscape, and optimization objectives, so discarding momentum at pretraining’s end can impair learning new capabilities.Conventional parameters store data knowledge, whereas lower-frequency optimization levels store knowledge about how the model optimizes itself and about the objective space.Source: §4.5 · p. 21

5 Existing Architectures as Neural Learning Modules

The section reframes modern sequence architectures as nested associative-memory optimization systems, with attention, Hebbian, Delta, and other rules corresponding to distinct objectives and update mechanisms. From the Nested Learning perspective, their apparent heterogeneity reflects differences in optimization level, timescale, objective, and learning rule within a uniform reusable structure.

5 Existing Architectures as Neural Learning Modules

  • Modern sequence models can be represented as nested associative memories that learn key–value mappings by optimizing an internal attentional-bias objective.This framework treats architectures as memory modules whose optimization algorithm and function class define the memory architecture.Source: §5 · p. 21
  • Softmax attention is a non-parametric solution to an ℓ2 regression objective, while restricting optimization to past c tokens yields sliding-window attention.More expressive attention variants also fit the associative-memory formulation through optimal non-parametric mapping solutions.Source: §5 · p. 22
  • Hebbian recurrent models recover linear-attention variants by optimizing dot-product key–value similarity with gradient descent, whereas Delta-rule models optimize an L2 regression objective.Different kernels, retention factors, weight decay, and learnable parameters produce diverse variants within these optimization-based formulations.Source: §5 · pp. 22–23
  • Beyond Hebbian and Delta rules, OjaNet, non-Euclidean objectives, and Omega updates expand associative-memory architectures with stability constraints, alternative norms, and cached past inputs.Non-Euclidean learning showed better empirical performance and robustness on long-context tasks than Delta-rule variants, while Omega reduces to an online update in a special case.Source: §5 · p. 23
  • When memory initialization is not meta-learned, gating in modern sequence models can act as persistent memory that stores pretraining knowledge alongside in-context adaptation.With meta-learned initialization, recurrent memory modules instead include a second level that adapts their state to context.Source: §5 · p. 23

5.1 Revisiting the Human Brain Perspective of Nested Learning

  • Nested Learning characterizes modern architectures as uniform, reusable sets of feedforward networks optimized at different levels and timescales, making their heterogeneity an illusion of observed solutions.Components differ primarily in optimization level, internal objective, and update rule, rather than in fundamentally different structural principles.Source: §5.1 · p. 24

6 Takeaways and Revisiting Common Terms

Nested Learning reframes memory, learning, in-context learning, pre-training, and continual learning as interconnected processes distributed across levels, contexts, and update frequencies. It also unifies architectural components while emphasizing that knowledge transfer and optimizer design depend on interactions between levels and architectures.

Memory and Learning

  • Memory is distributed throughout model parameters: any input-caused update stores memory, while learning is the process of effectively acquiring and encoding it.Continuum memory applies updates at different frequencies, storing memories across time scales and improving resistance to catastrophic forgetting.Source: §6 · p. 24

A General Note on Parameters of a Model

  • All parameters represented in Nested Learning contribute to performance and expressivity, including momentum and recurrent hidden states that store loss-landscape or contextual knowledge.These parameters are updated at different frequencies and may lose compressed knowledge when context changes because knowledge transfer between levels is absent.Source: §6 · p. 25

In-Context Learning

  • In-context learning is adaptation to a level’s context flow; it occurs across all levels, while Transformer conditioning on full context is only its non-parametric form.Its performance still depends on a powerful low-frequency level that enables fast adaptation by higher-frequency parameters.Source: §6 · p. 25

(Test-Time) Learning/Memorization

  • Test-time training and memorization are parametric in-context learning whose acquired knowledge disappears when the current context is removed, making “test-time” terminology misleading for continual learning.In continual learning, there is no clear training-versus-test boundary; a neural learning module either receives information or operates as an isolated system.Source: §6 · pp. 25–26

Pre-training and Test Time

  • Pre-training is one instance of in-context learning with the entire pre-training dataset as context, while continual learning spans phases whose knowledge may not transfer to more persistent levels.The training/test distinction results from disconnecting knowledge transfer between high- and low-frequency levels.Source: §6 · p. 26

Neural Learning Module as an Inter-Connected System

  • Nested Learning views architectures as interconnected systems: recurrent models add computational levels, and architecture-specific optimizers are needed because architectures generate the gradients that optimizers must manage.Vanilla and learned optimizers are both associative-memory modules that compress information into parameters, differing in frequency and context flow.Source: §6 · pp. 26–27

7 Continuum Multi-Timescale Memory System

The Continuum Memory System (CMS) extends long-/short-term memory by assigning knowledge-storage MLP blocks to multiple update frequencies, whose parameters compress their context into abstract knowledge. CMS supports continual learning through distributed knowledge retention, efficient scheduled updates, and sequence parallelization, while enabling multi-scale optimizers and pre-trained initialization.

7.1 Continuum Memory System (CMS)

  • CMS chains MLP blocks with different context chunk sizes and update frequencies, generalizing the Transformer’s single persistent-memory block.Each block’s parameters compress their own context into abstract knowledge; the conventional Transformer is the special case with one block and zero update frequency.Source: §7.1 · p. 27
  • CMS mitigates catastrophic forgetting because knowledge lost from an updated block can remain in lower-frequency blocks and transfer back through their initialization pathways.This creates a temporal loop that helps preserve important knowledge during continual updates.Source: §7.1 · p. 28
  • CMS can be efficient in practice by updating only a small number of scheduled parameters at each time and enabling sequence parallelization for higher-frequency levels.Tokens from different positions within a chunk can be processed in parallel when no sequential dependency occurs inside that chunk.Source: §7.1 · pp. 28–29

7.2 Continuum Memory System In Optimizers

  • M3 demonstrates CMS in optimizers by combining two-scale momentum memories with weighted aggregation and Newton–Schulz transformations to retain information about long-past gradients.M3 combines Adam (Kingma et al. 2014a), Muon (Jordan et al. 2024), and CMS, but may incur computational overhead when scaling to larger networks.Source: §7.2 · p. 29

7.3 Ad-hoc Level Stacking: Initializing CMS with Pre-Trained Models

  • Ad-hoc level stacking initializes CMS blocks from pre-trained MLP weights so higher-frequency levels can adapt faster to their context flow.The higher-frequency learning rate controls adaptation capacity; as it approaches zero, blocks remain close to their pre-trained states.Source: §7.3 · pp. 29–30

8 Hope: A Self-Referential Learning Module with Continuum Memory

Hope combines self-modifying Titans with a Continuum Memory System to address Transformers’ limited computational depth, static parameters, and weak context adaptation. Its chunk-wise training enables parallel computation while preserving adaptive memory updates.

8 Hope: A Self-Referential Learning Module with Continuum Memory

  • Transformers have limited computational depth and static context-time parameters, so their contextualization is bounded by fixed projections and stored knowledge.Short convolutions and canon layers mix local tokens but remain limited in adapting globally; self-referential Titans address this limitation.Source: §8 · pp. 30–31

8.1 Deep Self-Referential Titans

  • Deep self-referential Titans make all memory components adaptive in-context and generate their own values, while meta-learned initial states support fast adaptation, stability, and noise robustness.The design uses adaptive keys, values, learning rates, retention gates, and memory states; only q_t is a non-adaptive projection.Source: §8.1 · pp. 31–32

8.2 Fast and Parallelizable Training

  • Chunk-wise training computes keys, values, learning rates, weight decays, and gradients in parallel, yielding a fast parallelizable dual form for self-referential Titans.The method uses separate chunk sizes for the memory module and the other memories, with update frequency f_□ = L/C_□.Source: §8.2 · p. 32
  • Hope-enhanced architecture achieves the best accuracy on CLINC, Banking, and DBpedia class-incremental learning among continual-learning methods, including ICL.The comparison is shown across the three datasets in Figure 6.Source: Figure 6 · p. 33

8.3 Hope Neural Learning Module

  • Hope combines self-modifying Titans with Continuum Memory System because the former provides expressive adaptation while the latter supplies higher-capacity, persistent storage across update frequencies.Hope’s forward pass places self-modifying Titans before the Continuum Memory System; experiments also use L2-normalized q and k and local convolutions with window size 4.Source: §8.3 · p. 33

9 Experiments

The experiments evaluate Nested Learning through Hope’s continual learning, long-context understanding, language modeling, recall, formal-language, and ablation tasks, alongside the M3 optimizer. Across these evaluations, Hope generally outperforms relevant baselines, while M3 improves solution quality with comparable large-model training efficiency.

9.1 Hope: Continual Learning and Long Context Understanding

  • Hope shows the best continual-learning performance across evaluated baselines, and its multi-level memory improves adaptation through knowledge transfer between update frequencies.Compared with ICL, InCA, and EWC, Hope’s advantage is attributed to multiple in-context-learning levels and cross-level knowledge transfer.Source: §9.1 · p. 34
  • Increasing Hope’s memory levels improves in-context and long-context performance, whereas increasing the lowest-frequency update lowers performance; a lowest frequency of 2K offers improved efficiency with near-512 performance.The lowest-frequency memory is the most persistent, so higher frequency makes long-term memory weaker but more adaptive.Source: Figure 7 · p. 34, §9.1 · pp. 35–36

9.2 Hope: Long Context Understanding

  • Hope achieves the best performance across all needle-in-a-haystack tasks and difficulty levels, while Hope-Attention outperforms Transformers.The results support the importance of self-referential updates and the Continuum Memory System, especially at longer context lengths.Source: Table 1 · p. 35, §9.2 · p. 36

9.3 Hope: Language Modeling and Common-Sense Reasoning

  • Hope outperforms all baselines on average across both language-modeling and common-sense-reasoning benchmarks, and its performance gain with parameter scaling is higher than that of other attention-free models.Source: §9.3 · p. 38

9.4 Hope: In-context Recall Tasks and MAD Synthetic Benchmark

  • Hope outperforms all attention-free baselines on short in-context recall and achieves the best results on the MAD recall, memorization, compression, and copying tasks.On short recall tasks, Transformers remain best, but Hope narrows the gap while exceeding other attention-free models.Source: §9.4 · p. 38, Table 3 · p. 38, Table 4 · p. 38

9.5 Language Recognition Tasks

  • Hope achieves perfect scores on all formal-language recognition tasks, matching nonlinear recurrent models while retaining parallelizable training when needed.This combination supports scaling Hope to larger language-modeling settings without sacrificing recurrent-task performance.Source: Table 5 · p. 38, §9.5 · p. 39

9.6 Hope: Ablation Studies and Scaling

  • Ablations show that every Hope component contributes positively, since replacing or removing self-modifying optimization, momentum, weight decay, CMS, or projection choices damages perplexity and/or accuracy.The evaluation compares average language-modeling perplexity and common-sense-reasoning accuracy across architectural variants.Source: §9.6 · p. 39, Table 6 · p. 39

9.7 Expressive Optimizers

  • M3 obtains the best ViT training and test losses versus AdamW and Muon, while its large-language-model training efficiency is on par with AdaMuon but slower than Muon.The efficiency tradeoff is attributed to M3’s use of multiple momentum memories.Source: Figure 11 · p. 39, Figure 12 · p. 39, §9.7 · p. 40

10 Conclusion

The conclusion presents Nested Learning as a unified view of architectures, optimizers, and learning behaviors through multi-level associative-memory compression, while positioning Hope and CMS as promising but incomplete steps toward continual learning and long-context reasoning.

Conclusion

  • Nested Learning models machine learning systems as interconnected, multi-level optimization problems whose associative memories compress and reuse context across levels and timescales.This reframes pre-training, in-context learning, and continual learning as manifestations of the same underlying mechanism, while recasting backpropagation, momentum, and preconditioning as associative-memory mechanisms.Source: §10 · p. 40
  • NL motivates generalized updates including Delta Gradient Descent, Delta Momentum, and Multi-scale Momentum Muon, alongside nested-associative-memory interpretations of sequence architectures.The conclusion also introduces Continuum Memory System as a memory formulation generalizing traditional long-term/short-term memory blocks.Source: §10 · p. 40
  • Hope and CMS show promising reductions in catastrophic forgetting and improvements in long-context reasoning while remaining competitive as a general backbone, but catastrophic forgetting is not solved generally.The conclusion attributes forgetting to compression under limited network capacity, which forces retention of new information at the expense of older information.Source: §10 · p. 40
  • NL is proposed as a roadmap in which progress should come from exploiting additional levels in learning systems rather than building ever-deeper static networks.The proposed design axis spans continual learning, long-context reasoning, modern optimizers, and self-modifying models.Source: §10 · p. 40

A Generalized Formulation for Nested Learning and Nested Systems

This section generalizes nested systems and nested systems of associative memories (NSAM). A nested system organizes optimization problems across ordered levels, while NSAM specializes this structure to learned memory mappings.

Generalized Nested System

  • A generalized nested system has K ordered levels, with each level containing a set of optimization problems.The definition indexes levels by k, where 1 ≤ k ≤ K.Source: A Generalized Formulation for Nested… · p. 50
  • Each optimization problem is specified by an objective, context data, and parameter set, and is optimized using gradient descent.The context is the data optimized on, while the parameters belong to the individual problem.Source: A Generalized Formulation for Nested… · p. 50

Nested System of Associative Memories

  • A nested system of associative memories extends the same K-level organization to sets of memory-learning optimization problems.Its problems are defined in terms of ground-truth mappings, objectives measuring learned-mapping quality, and memory parameters.Source: A Generalized Formulation for Nested… · p. 50

B Adam, AdaGrad, and Other Similar Optimizers as Associative Memory Modules

The section reframes momentum and adaptive optimizers as associative memories that compress gradient information about past data. Under this view, Adam, AdaGrad with momentum, and related optimizers differ by the global properties they memorize and the structure of their updates.

B Adam, AdaGrad, and Other Similar Optimizers as Associative Memory Modules

  • Momentum can be interpreted as a key- or value-less associative memory that compresses past gradients to improve current weight updates.A more powerful momentum term would ideally memorize past gradients and map them to expressive global properties of prior data.Source: B Adam, AdaGrad, and Other Similar… · p. 50
  • Adam is an optimal associative memory under the stated L2 regression objective, learning a mapping from gradients to their variance as a global property of past samples.Its first- and second-moment memories update at the same frequency and can be computed independently in parallel.Source: B Adam, AdaGrad, and Other Similar… · p. 51
  • The formulation is AdaGrad with momentum and therefore generalizes AdaGrad, with the AdaGrad case recovered when 𝛽1 = 1.Source: B Adam, AdaGrad, and Other Similar… · p. 51
  • The same perspective re-formulates Adam- and AdaGrad-related methods—including RMSProp, SignSGD, NAdam, AMSGrad, RAdam, Lion, Shampoo, and Soap—as associative memories compressing gradients.The cited connections treat these methods as related optimizer variants or approximations of preconditioning terms.Source: B Adam, AdaGrad, and Other Similar… · p. 51

C Delta Gradient Descent with Normalization

This section reformulates gradient descent as associative memory and extends it to Delta Gradient Descent using an L2 regression objective. The derivation assumes normalized inputs and optimizes the resulting objective by taking its gradient.

C Delta Gradient Descent with Normalization

Related papers

Loading 2512.24695v1…