PaperlayerView exact revision on arXiv

Source-linked AI summary

Nested Learning: The Illusion of Deep Learning Architectures

Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, Vahab Mirrokni

arXiv:2512.24695v1cs.LGcs.AI

TL;DR

The paper addresses how machine-learning systems can continually learn, memorize, and self-improve despite the limitations of static deep architectures. It introduces Nested Learning, which models architectures and optimizers as multi-level context-compression systems, and develops expressive optimizers, Continuum Memory Systems, and the self-modifying Hope architecture. Hope and CMS improve continual learning and long-context reasoning while remaining competitive as a general backbone, though catastrophic forgetting is not solved in general.

  • Problem

    Static deep architectures and deployed LLMs have limited continual learning and adaptation beyond immediate context, while existing remedies may be costly, dependent on external components, poorly generalizing, or vulnerable to catastrophic forgetting.

  • Method

    Nested Learning represents models and training as interconnected multi-level optimization problems, interpreting architectures and optimizers as associative memories that compress context and motivating CMS, expressive updates, and self-modifying models.

  • Results

    Hope and CMS improve continual learning and long-context reasoning capabilities while remaining competitive as a general backbone.

  • Takeaways & Limitations

    The paper presents update levels and frequency-based memory as an additional design axis for developing continual-learning, long-context, and self-modifying models.

  • Takeaways & Limitations

    Catastrophic forgetting is not solved in general because limited network capacity can force forgetting when new information is compressed.

Abstract

from arXiv · show

Despite the recent progresses, particularly in developing Language Models, there are fundamental challenges and unanswered questions about how such models can continually learn/memorize, self-improve, and find effective solutions. In this paper, we present a new learning paradigm, called Nested Learning (NL), that coherently represents a machine learning model with a set of nested, multi-level, and/or parallel optimization problems, each of which with its own context flow. Through the lenses of NL, existing deep learning methods learns from data through compressing their own context flow, and in-context learning naturally emerges in large models. NL suggests a philosophy to design more expressive learning algorithms with more levels, resulting in higher-order in-context learning and potentially unlocking effective continual learning capabilities. We advocate for NL by presenting three core contributions: (1) Expressive Optimizers: We show that known gradient-based optimizers, such as Adam, SGD with Momentum, etc., are in fact associative memory modules that aim to compress the gradients' information (by gradient descent). Building on this insight, we present other more expressive optimizers with deep memory and/or more powerful learning rules; (2) Self-Modifying Learning Module: Taking advantage of NL's insights on learning algorithms, we present a sequence model that learns how to modify itself by learning its own update algorithm; and (3) Continuum Memory System: We present a new formulation for memory system that generalizes the traditional viewpoint of long/short-term memory. Combining our self-modifying sequence model with the continuum memory system, we present a continual learning module, called Hope, showing promising results in language modeling, knowledge incorporation, and few-shot generalization tasks, continual learning, and long-context reasoning tasks.

1 Introduction

The paper introduces Nested Learning (NL) to represent models and training as interconnected optimization problems with multiple context flows and update timescales. It uses this perspective to address limitations in static deep architectures and develop more expressive optimizers, memory systems, and self-modifying models.

  • Motivation: Stacking layers improves expressive power for static tasks but does not necessarily increase computational depth, adaptation, continual learning, or out-of-distribution generalization.The paper identifies these as limitations of treating depth as the primary design axis.
  • Motivation: LLMs remain largely static after deployment, with adaptation mainly confined to in-context learning and other approaches facing costs, external dependencies, limited generalization, or catastrophic forgetting.The paper frames this as a central challenge for continual acquisition of capabilities.
  • Nested Learning Paradigm: Nested Learning models architectures and optimization as interconnected, multi-level, and parallel optimization problems, each with its own context and internal gradient flow.It also treats learning algorithms and architectures as different levels of a common system.
  • Optimizers and Architectures: NL interprets pre-training and gradient-based optimizers as associative-memory processes that compress context into parameters, motivating architecture-specific optimizers and new learning updates.The paper describes pre-training as in-context learning over the entire pre-training dataset and optimizers as gradient-compressing memories.
  • Continuum Memory and Hope: The Continuum Memory System distributes memory across update frequencies, while Hope combines this design with a self-modifying sequence model for continual learning and long-context reasoning.Higher-frequency components adapt quickly, whereas lower-frequency components retain more persistent knowledge.

2 Preliminaries

The preliminaries define the paper’s notation, optimization background, meta-learning framing, and matrix-valued memory mechanisms. They establish the concepts used to analyze nested learning and associative-memory-based sequence models.

  • Notations: The paper represents inputs, memory states, keys, values, and queries with matrix and vector notation, and uses residual MLPs as memory-module architectures.The memory state is denoted M_t, while bold lowercase vectors refer to time-indexed inputs.
  • Gradient Descent: Gradient descent is presented as a standard optimizer whose update can also be characterized as steepest descent, a proximal step, or an FTRL formulation.These equivalent views support later optimization analyses.
  • Meta Learning: Meta-learning is framed as a two-level optimization procedure in which an outer process learns decisions about model parameters, objectives, or optimizers across tasks.The outer loop may use supervised or unsupervised processes.
  • Fast Weight Programmers: Fast Weight Programmers use a matrix-valued recurrent state that is updated online and read through matrix–vector multiplication as a compact key–value memory.The matrix state replaces the vector states used in traditional recurrent networks.
  • In-context Learning: In-context learning is introduced as using knowledge acquired during pre-training to infer and perform a new task from context such as examples or instructions.

3 Nested Learning

Nested Learning reframes models and their optimization processes as interconnected associative memories and nested optimization problems. This view explains how different components compress context at different levels and update frequencies, motivating more expressive continual-learning systems.

  • Associative Memory: Associative memory maps keys to values, while learning acquires an effective operator for that mapping from data.In this paper, keys and values may be tokens, gradients, or subsequences.
  • Associative Memory: Backpropagation can be viewed as associative memory that maps each data sample to the local surprise signal of its prediction.The gradient acts as an error signal that is zero when the loss is minimized.
  • Associative Memory: Gradient descent with momentum forms a 2-level associative memory: an inner process stores gradients, while an outer process updates the slow weights.Plain gradient descent is described as a 1-level associative memory.
  • Nested Optimization Processes: Training linear attention with gradient descent is a 2-level optimization process, with outer updates to projection layers and inner updates to the memory.The inner memory update is equivalent to unnormalized linear attention, while projection layers act as slow weights.
  • Nested Optimization Processes: Nested Learning represents models as interconnected nested, multi-level, and/or parallel optimization problems, each with its own context flow.The paradigm supports processing at different abstraction levels and update frequencies.
  • Nested Optimization Processes: The optimization process is itself an associative memory whose inputs are gradients generated by the architecture, creating feedback between model dynamics and optimizer behavior.This perspective links optimizer choice to gradient-pattern compression and memory management.

4 Optimizers as Learning Modules

This section reframes backpropagation and momentum-based optimization as nested associative-memory systems that compress error or gradient information. It then motivates more expressive objectives and update rules, including delta-based and generalized gradient descent.

  • 4.1 Backpropagation as an Associative Memory: Backpropagation maps each layer’s input to its local error signal, so neural-network training can be viewed as compressing these mappings into associative memory.The local error signal measures the surprise of a layer’s output given its input.
  • 4.1 Backpropagation as an Associative Memory: Backpropagation is self-referential because the associative-memory values are generated by the update process itself, not precomputed as in simple linear attention.
  • 4.2 Momentum-based Optimizers as Associative Memories: Momentum forms a two-level optimization: an inner memory compresses past gradients, while the outer process uses that state to update the weights.The momentum objective differs from the original task objective and evaluates the quality of its mappings.
  • 4.2 Momentum-based Optimizers as Associative Memories: Adam and related optimizers can be interpreted as associative memories that compress gradients, with Adam optimally predicting gradient variance under an L2-regression objective.The framework also connects AdaGrad and other variants to associative-memory formulations.
  • 4.3 Long Context in Optimizers: An Example of Continual Learning with Orthogonal Tasks: Standard momentum has limited memory capacity because its low-pass filtering emphasizes recent gradients; with β=0.9, the last 6 and 43 gradients contribute at least 50% and 99%, respectively.The authors argue that this weakens access to information beyond recent steps.
  • 4.4 More Expressive Designs for Momentum as an Associative Memory: More expressive momentum variants improve memory management by using richer objectives and learned mappings, including orthogonalization and higher-frequency processes.The higher-frequency level can learn an orthogonal mapping that a lower-frequency process uses to optimize weights.
  • 4.5 Going Beyond Simple Gradient Descent and Momentum: Delta Gradient Descent incorporates the previous weight state and applies an adaptive decay term determined by the current data sample.This extends ordinary updates beyond dependence on only the current elements.
  • 4.5 Going Beyond Simple Gradient Descent and Momentum: Generalized Gradient Descent defines backpropagation as a self-referential associative memory that compresses training samples and maps them to self-generated values.Generalized Momentum adapts the same perspective to momentum, whose keys and values are generated by a lower-frequency level.

5 Existing Architectures as Neural Learning Modules

This section interprets modern neural architectures as learning modules composed of associative-memory mappings and optimization processes operating at different levels and time scales. Under this view, attention, recurrent rules, and other architectures differ through their internal objectives, retention mechanisms, and context windows.

  • 5 Existing Architectures as Neural Learning Modules: Modern sequence models can be analyzed as associative memories that learn mappings from keys to values through internal optimization or non-parametric solutions.This viewpoint covers Transformers and recurrent architectures used in language models.
  • Softmax Attention: Softmax attention is reformulated as a non-parametric solution to an L2 regression objective, and restricting optimization to the past c tokens yields sliding-window attention.
  • RNNs with Hebbian Rule: Hebbian recurrent architectures use dot-product similarity as their internal objective, recovering variants of linear attention through different retention gates and kernels.Examples include linear attention, RetNet, RWKV, and related models.
  • RNNs with Delta Rule: Delta-rule recurrent models replace Hebbian updates with an L2-regression objective that learns mappings between keys and values.The family includes DeltaNet, Longhorn, and RWKV7.
  • Beyond Conventional Learning Rules: Non-Euclidean learning rules use Lp regression objectives and have been reported to improve empirical performance and robustness on long-context tasks compared with Delta-rule variants.
  • Beyond Conventional Learning Rules: Omega rules extend online updates by using a cached local window of past inputs, while the full-context special case can collapse back to an online update.The local window size is denoted by c.
  • 5.1 Revisiting the Human Brain Perspective of Nested Learning: Nested Learning represents optimization processes and architectures as nested or parallel optimization problems whose memory structures and objectives operate across levels.The framework treats groups of neurons as having internal objectives and update mechanisms.
  • 5.1 Revisiting the Human Brain Perspective of Nested Learning: The apparent heterogeneity of deep architectures is attributed to overlooking the NL axis of optimization levels and time scales, producing an illusion of architectural depth.The paper characterizes neural learning modules as feedforward networks optimized at different levels and time scales.

6 Takeaways and Revisiting Common Terms

Nested Learning reframes neural models as interconnected, multi-level learning systems in which adaptation, memory, and in-context learning arise from context-specific updates. This perspective revisits continual learning, pre-training, architectures, and parameters as parts of one unified system.

  • Memory and Learning: Nested Learning defines memory as an input-caused neural update and learning as acquiring useful memory, distributed throughout the model’s parameters.Under this view, gradient-based updates at any level constitute memory rather than belonging to a separate memory block.
  • A General Note on Parameters of a Model: All parameters appearing in a Nested Learning representation contribute to model performance and expressivity, not only those optimized during pre-training.The framework consequently broadens the set of model components regarded as parameters or knowledge-bearing entities.
  • In-Context Learning: Nested Learning treats each neural-network level as having its own context flow, making adaptation at any level a form of in-context learning.The framework defines in-context learning broadly as adapting to and learning from a given context.
  • (Test-Time) Learning/Memorization: Test-time training and test-time memorization are instances of parametric in-context learning whose acquired knowledge disappears when the current context is removed.The distinction becomes misleading in continual learning because there is no clean separation between test time and training time.
  • Pre-training and Test Time: Nested Learning interprets pre-training as in-context learning over an ultra-large context, namely the entire pre-training dataset.Training and test-time distinctions result from disconnecting knowledge transfer between the highest-frequency and lower-frequency levels.
  • Continual Learning: Continual learning depends on whether knowledge acquired at one level transfers to more persistent levels; without such transfer, updates may remain temporary.The framework therefore views both pre-training and test-time processing as forms of continual internal computation, while persistence varies by level connectivity.
  • Existing Architectural Backbones and Hybrid Models: Nested Learning views modern architectures as feedforward layers with their own context flows and optimization problems, while recurrent modules add computation levels to MLP blocks.This perspective also emphasizes that neural learning modules are interconnected, so component designs can affect one another.

7 Continuum Multi-Timescale Memory System

The Continuum Memory System organizes MLP memories across update frequencies and transfer patterns to support persistent knowledge, longer contexts, and continual learning. Its variants include nested, sequential, and independent arrangements, while M3 demonstrates the same multi-scale idea for optimization.

  • 7.1 Continuum Memory System (CMS): CMS represents memory as MLP blocks operating at different frequencies, with each block updated according to its own context chunk size.The blocks compress their respective context flows into parameters through an optimizer-defined update mechanism.
  • Nested Continuum Memory Systems: Nested CMS meta-learns each higher-level block’s initial state from the preceding level, enabling higher-order in-context learning across reinitialized context flows.A block is reinitialized after completing its designated optimization context.
  • Sequential and Independent CMS: Sequential CMS connects blocks so each level’s output feeds the next, whereas independent CMS combines separately operating blocks through an aggregation function.The independent variant can use a learnable weighted sum to aggregate block outputs.
  • CMS Design Helps with Continual Learning: CMS can reduce catastrophic forgetting because updating one block leaves knowledge stored in other blocks unchanged.The design distributes knowledge across blocks with different update schedules rather than concentrating all updates in one memory.
  • Is CMS Efficient Enough?: CMS is designed to be efficient by updating only a small number of parameters at each time and enabling sequence parallelization at higher-frequency levels.The paper summarizes these properties as making CMS fast in practice.
  • 7.2 Continuum Memory System In Optimizers: M3 applies CMS to optimizer momentum, combining recent-gradient compression with a second memory for information from longer-past gradients.It replaces a simple associative-memory term with an independent two-level CMS and aggregates the momentum terms.
  • 7.2 Continuum Memory System In Optimizers: M3 is a proof-of-concept whose computational overhead may challenge scaling to larger networks.Its central design delays memory updates to obtain longer-context momentum.

8 Hope: A Self-Referential Learning Module with Continuum Memory

Hope combines self-modifying Titans with a Continuum Memory System in a nested architecture whose memory components adapt at different levels and update frequencies. Its chunk-wise training generates and updates these components in parallel, supporting efficient self-referential learning.

  • Nested Learning Perspective: Nested Learning represents sequence models as associative memories, with each feedforward block having its own context, update frequency, and internal objective.Transformers are described as two-level systems: parameter optimization followed by in-context learning through a non-parametric solution.
  • Nested Learning Perspective: Transformer projections remain fixed after pre-training, bounding contextualization by knowledge stored in the key, value, and query projection blocks.The paper identifies this static behavior as a limitation for in-context self-modification and adaptation.
  • Self-Referential Titans: The associative-memory formulation maps keys to values and retrieves them with queries, using learning rates and retention gates within nested optimization problems.Initial states for the memory modules are meta-learned across sequences and contexts.
  • Self-Referential Titans: Self-referential Titans generate latent values for their own memory components, allowing the model to modify its components and learning process in context.The design extends adaptive memory to keys, values, queries, learning rates, and retention gates, while addressing shared key-value representations as a suboptimal design.
  • Fast and Parallelizable Training: Chunk-wise training uses update frequency f_□ = L/C_□ and computes chunk elements and gradients in parallel before processing each chunk.The experiments use separate chunk sizes for the memory module and the other memories in self-referential Titans.
  • Hope Neural Learning Module: Hope combines self-modifying Titans followed by a Continuum Memory System as a neural learning module.The paper presents this combination as the Hope architecture.

9 Experiments

The experiments evaluate Hope, CMS, and Nested Learning across continual learning, long-context understanding, language modeling, reasoning, recall, formal language recognition, vision, and optimizer efficiency. Hope generally outperforms baselines, benefits from additional memory levels, and maintains long-context performance, while some efficiency and continual-learning limitations remain.

  • Continual Learning and Long Context Understanding: Hope outperforms ICL and DuoAttention across memory-level and update-frequency variants on the evaluated long-context and in-context learning benchmarks.More memory levels improve in-context learning and long-term memory, while higher lowest-frequency values reduce performance; a 2K lowest frequency offers efficiency close to the 512 setting.
  • Continual Learning and Long Context Understanding: In continual translation, ICL suffers a dramatic performance drop, whereas increasing Hope’s memory levels improves results and Hope-3 nearly recovers its non-continual capability.The comparison uses ChRF for Manchu→English and Kalamang→English translation.
  • Long Context Understanding: Hope maintains good performance through 10M-token contexts, while large models fail around 128K–256K and Titans and ARMT decline rapidly after 1M.The passage attributes Hope’s long-context result mainly to CMS and notes that fine-tuning helps lower-frequency levels adapt memory management.
  • Language Modeling and Common-Sense Reasoning: Hope achieves the best average performance across the reported language-modeling and common-sense-reasoning benchmarks, with larger parameter scales yielding higher gains than other attention-free models.The comparison is reported in Table 2.
  • Recall and Language Recognition: Hope achieves the best results on the MAD recall, memorization, compression, and copying benchmark and perfect scores on all evaluated formal-language-recognition tasks.On formal language recognition, Hope matches other nonlinear recurrent models such as LSTM and SRWM.
  • Optimizer and Efficiency Experiments: On ImageNet, the experiments compare optimizers under controlled ViT settings, while language-model efficiency experiments find M3 relatively slower than Muon because it uses multiple momentum memories.The optimizer comparisons include Muon, AdaMuon, and M3 at 140M and 1.3B language-model scales.

10 Conclusion

Nested Learning models machine learning systems as interconnected multi-level optimization problems whose memories compress and reuse context across levels and timescales. The paper derives expressive optimizers and introduces CMS and Hope, which improve continual learning and long-context reasoning while remaining competitive as a general backbone, but catastrophic forgetting is not solved generally.

  • Conclusion: Nested Learning represents architectures and optimizers as interconnected associative memories that compress tokens, gradients, or higher-level signals at different levels and update frequencies.This framing unifies pre-training, in-context learning, and continual learning as context compression and reuse.
  • Conclusion: The paper derives generalized gradient updates, introduces CMS as a continuum-memory formulation, and combines CMS with self-modifying Titans in the Hope architecture.Examples of generalized updates include Delta Gradient Descent, Delta Momentum, and Multi-scale Momentum Muon.
  • Conclusion: Hope improves continual learning and long-context reasoning capabilities while remaining competitive as a general backbone.This is the paper’s stated conclusion about the combined self-modifying Titans and CMS design.
  • Limitations: Catastrophic forgetting is not solved in general because limited network capacity can force forgetting when compressed information must accommodate new information.The authors characterize Nested Learning as a roadmap toward better use of multiple levels rather than a completed solution.

A Generalized Formulation for Nested Learning and Nested Systems

The generalized Nested Learning formulation organizes systems into ordered levels containing optimization problems, each with its own objective, context, parameters, and gradient-based update. The associative-memory extension interprets these optimization problems as learning mappings from contexts to memory parameters.

  • Generalized Nested System: A generalized nested system contains K ordered levels, with each level consisting of a set of optimization problems.The formulation allows multiple optimization problems to be organized across levels.
  • Generalized Nested System: Each optimization problem is specified by an objective, context, parameter set, and gradient-descent optimization process.The context is the data optimized on, while the parameter set contains the problem’s trainable parameters.
  • Nested Systems of Associative Memories: A nested system of associative memories extends the formulation by assigning memory-learning optimization problems to each level.These problems evaluate the quality of learned memory mappings.

B Adam, AdaGrad, and Other Similar Optimizers as Associative Memory Modules

The paper interprets momentum-based optimizers as associative memories that compress gradient information into update states. Under particular objectives and choices of accumulated statistics, this view recovers momentum, AdaGrad with momentum, and Adam.

  • Momentum as Associative Memory: Momentum can be interpreted as a key- or value-less associative memory that compresses past gradient terms into the momentum state.A more expressive momentum should preserve useful information about past gradients to improve current weight updates.
  • Momentum as Associative Memory: The associative-memory objective seeks a momentum mapping from gradients to a global property of past data rather than simply mapping gradients to 1.More expressive global properties can incorporate more compressed information from past data.
  • Recovering Known Optimizers: With the squared-sum statistic and λ→0, the generalized update recovers gradient descent with momentum.This is presented as a simple case of the proposed formulation.
  • Recovering Known Optimizers: Using the variance of preceding data samples as the memory statistic yields an update equivalent to Adam.The formulation identifies Adam as an optimal associative memory under the stated L2 regression objective.
  • Recovering Known Optimizers: The formulation also connects the generalized update family to AdaGrad with momentum and generalizes AdaGrad when β1=1.The paper situates these relationships alongside connections among Adam, RMSProp, SignSGD variants, NAdam, AMSGrad, RAdam, and Lion.

C Delta Gradient Descent with Normalization

This section reformulates gradient descent as associative memory and extends it to Delta Gradient Descent using a more expressive L2 regression objective under normalized inputs.

  • Gradient descent can be reformulated as an associative memory process.
  • Each update learns the negative gradient direction using terms dependent only on the current gradient.
  • Delta Gradient Descent extends this process with an L2 regression objective based on the update target u_t = −∇_y_t L(W_t; x_t).
  • The derivation assumes normalized inputs satisfying ∥x_t∥2 = λ, as in normalized memory systems or networks with normalization layers.
Loading 2512.24695v1…