PaperlayerView exact revision on arXiv

Source-linked AI summary

Where Should Experience Live? Hierarchical Hebbian Memory for Continual Vision Transformers

Mohammed Yusuf Mujawar, Noorbakhsh Amiri Golilarz

arXiv:2609.00358v1cs.CVcs.NE

TL;DR

Vision Transformers need a way to rapidly acquire and organize new visual associations across different persistence timescales. The paper proposes Hierarchical Hebbian Memory with learned routing and causal updates, achieving strong recognition results and differentiated delayed-memory behavior. Its findings support combining Hebbian association with learned memory allocation for online visual experience.

  • Problem

    Vision Transformers rely on slowly updated parameters, limiting rapid storage and organization of new associations across few-shot and continual visual learning.

  • Method

    Hierarchical Hebbian Memory combines Working, Routed Episodic, and Semantic Memory with a controller for routing, plasticity, retention, contribution, and consolidation.

  • Results

    Across Omniglot and CORe50, the hierarchy maintains strong recognition while learned multi-bank retrieval reaches 47.50% versus 24.17% for a single persistent bank and 25.00% without memory.

  • Takeaways & Limitations

    Hebbian association and learned memory routing can organize online visual experience across rapid, persistent, and consolidated timescales within Vision Transformers.

Abstract

from arXiv · show

Vision Transformers provide strong visual representations but typically rely on slowly updated parameters, limiting their ability to organize newly acquired information across different memory timescales. This work proposes \textit{Hierarchical Hebbian Memory}, a three-level memory architecture composed of rapid Working Memory, persistent Routed Episodic Memory, and slower Semantic Memory. A learned controller regulates memory contribution, read and write routing, plasticity, retention, and consolidation. A causal read-before-write lifecycle ensures that the current outcome cannot influence the prediction it supervises. The architecture is evaluated on Omniglot 5-way 1-shot recognition and CORe50 continual object recognition. With Swin-Tiny, the hierarchical model reaches 97.39\% accuracy on Omniglot and 95.37\% final accuracy on CORe50 when combined with experience replay. Learned multi-bank retrieval reaches 47.50\% delayed-association accuracy, compared with 24.17\% for a single persistent bank and 25.00\% without memory. After intervening distractors, Episodic Memory retains approximately 0.96 cosine similarity with stored associations, while Working Memory falls to approximately 0.05. These results show that Hebbian association and learned memory routing can jointly organize online visual experience across rapid, persistent, and consolidated memory timescales within Vision Transformers.

I. INTRODUCTION

The paper addresses how Vision Transformers can rapidly acquire and organize visual associations that persist beyond individual episodes. It proposes a three-level memory hierarchy with learned control over allocation, routing, plasticity, retention, and consolidation.

  • Vision Transformers mainly store knowledge in slowly updated parameters, limiting rapid association of new information from few observations.
  • Prior Hebbian Vision Transformer methods control temporary associative memory but do not organize experience that must persist beyond a single episode.
  • A single persistent state risks interference, whereas resetting memory after each short context prevents useful information from being retained.
  • Hierarchical Hebbian Memory separates rapid Working Memory, persistent Routed Episodic Memory, and slower Semantic Memory.
  • A task ID-free controller regulates memory allocation and strength through separate read and write routing, plasticity, retention, contribution, and consolidation decisions.
  • The architecture extends temporary Hebbian associative memory into persistent multi-timescale storage using a causal online update process.

B. Neurocognitive Memory Organization

The paper draws on functional distinctions among working, episodic, and semantic memory to motivate computational memory systems with different persistence and retrieval roles. It positions the proposed approach within memory-augmented Transformer and continual-learning research.

  • Working memory temporarily maintains information relevant to current processing, episodic memory preserves contextual experiences, and semantic memory represents less event-dependent knowledge.
  • Context-dependent encoding and retrieval motivate computational operations that coordinate storage and retrieval through a controller and episodic banks.
  • Consolidation reorganizes selected experience into more stable representations, motivating Semantic Memory's selective receipt of information from Episodic Memory.
  • Hierarchical and episodic Transformer systems separate immediate processing from information retained across longer language contexts.
  • Memory-augmented Transformer research identifies retrieval, interference, forgetting, and coordination as recurring challenges across state-based, external, parameter-based, and hybrid mechanisms.
  • The proposed method combines rapid Hebbian association, allocation across persistent episodic banks, and slower semantic consolidation within Vision Transformer learning.
  • Separate read and write routes independently control where an experience is stored and which memory is later retrieved.
  • Replay is evaluated as an external rehearsal mechanism rather than another level of the internal memory hierarchy.

III. HIERARCHICAL HEBBIAN MEMORY

The architecture implements three memory timescales in a causal online lifecycle: existing memory is read for prediction before the current outcome can update memory. Hebbian association and learned control govern memory formation and contribution.

  • The hierarchy uses rapid temporary Working Memory, persistent multi-bank Episodic Memory, and slower Semantic Memory under a learned controller.
  • The lifecycle reads existing memory, predicts, observes the outcome, plans and optimizes, commits the detached update, and periodically consolidates.
  • Prediction uses only memory available before the current observation is written, preventing the current target from entering memory before its supervising prediction.
  • The associative mechanism forms key-value associations through Hebbian fast-weight operations related to prior Vision Transformer memory models.
  • The general Hebbian update combines the previous memory state with a new association while controlling update strength, retention, and memory norm.
  • Working Memory is read before writing, gated in its contribution to the backbone representation, and updated after prediction and observation.
  • Working Memory resets at few-shot episode boundaries, enabling rapid adaptation without indefinitely carrying temporary associations.

C. Semantic Memory and Consolidation

Semantic Memory is the slowest hierarchy level and selectively receives eligible information from persistent Episodic Memory. Its update policy supports longer retention while separating changing contextual associations from consolidated information.

  • Semantic Memory receives selected episodic candidates rather than storing every observation.
  • Consolidation eligibility depends on accumulated evidence including new writes, utility, episodic support, and consistency across stored information.
  • Eligible associations are promoted toward the slower Semantic Memory state, while other information remains episodic.
  • For few-shot evaluation, Semantic Memory is developed during training and then frozen, while Working and Episodic Memory reset for each episode.
  • In continual learning, Semantic Memory persists across experiences and receives periodic eligible consolidations from episodic storage.

D. Controller and Transformer Integration

The controller uses current visual representations and pre-existing memory state to regulate access, routing, plasticity, retention, and consolidation across the hierarchy. A read-before-write lifecycle preserves causal prediction, while integration details vary across Transformer backbones.

  • D. Controller and Transformer Integration: The controller combines the current pooled representation with existing memory state to produce routing, gating, plasticity, and retention signals.These controls govern memory contribution and episodic read and write decisions.
  • D. Controller and Transformer Integration: Adaptive Hebbian controls are incorporated into persistent memory allocation and retrieval rather than only episode-specific fast-weight memory.
  • D. Controller and Transformer Integration: Prediction reads existing Working, Episodic, and Semantic states before the outcome is available, after which the detached memory operation is committed.Persistent states are detached between stream steps to avoid backpropagation through the complete stream history.
  • D. Controller and Transformer Integration: The primary continual controller receives no task identity, although experience boundaries remain known to the experimental protocol.This defines a task ID-free continual-learning setting with known experience boundaries.
  • D. Controller and Transformer Integration: Swin-Tiny integrates memory at Stage 4, whereas DeiT-Small and ViT-Small read one shared hierarchy at blocks 3, 6, 9, and 12 and write persistently at block 12.

IV. EXPERIMENTAL SETUP

The experiments cover rapid few-shot adaptation on Omniglot and persistent continual recognition on CORe50, using task ID-free routing with known experience boundaries. Comparisons include replay variants, backbone transfers, and controls that isolate persistent-memory functions.

  • IV. EXPERIMENTAL SETUP: Omniglot uses a 5-way 1-shot episodic protocol with one labeled support example per class and prototype-based query classification.Support samples can update episode-specific memory, whereas query samples are read only.
  • IV. EXPERIMENTAL SETUP: CORe50 evaluates repeated observations of object classes across sequential experiences and changing acquisition conditions with a fixed complete test set after each experience.Episodic and Semantic Memory persist across the stream, while Working Memory operates on a shorter timescale.
  • IV. EXPERIMENTAL SETUP: Current labels affect memory only after prediction, preserving task ID-free continual learning with known experience boundaries.
  • IV. EXPERIMENTAL SETUP: The main Swin-Tiny comparison includes a base model, Experience Replay, Latent Replay, and the full hierarchy combined with matched Experience Replay.Latent Replay stores intermediate representations, and matched replay capacity is retained for the replay comparisons.
  • IV. EXPERIMENTAL SETUP: Additional controls compare a single persistent bank, routed Episodic Memory without Semantic Memory, and the complete hierarchy with replay.DeiT-Small and ViT-Small provide architecture-transfer comparisons, while detailed mechanism analysis centers on Swin-Tiny.

C. Memory and Evaluation Controls

The study evaluates memory controls through few-shot accuracy, delayed retrieval, routing and bank analyses, semantic consolidation, and efficiency measurements. On Omniglot, the hierarchy reaches 97.39%, while its additional capacity supports the controller and three memory levels.

  • C. Memory and Evaluation Controls: Few-shot accuracy is measured on held-out episodic classification, while continual recognition uses final fixed test accuracy and area under the accuracy trajectory.
  • C. Memory and Evaluation Controls: Memory analyses test delayed retrieval, routing associations, bank utilization, routing entropy, inter-bank similarity, ablations, and semantic consolidation activity.These probes examine retrievability and functional effects of particular memory states.
  • C. Memory and Evaluation Controls: Efficiency is measured through parameter count, prediction latency, prediction-plus-observation latency, and memory usage.Separating prediction timing from the complete path isolates the additional cost of the online memory operation.
  • C. Memory and Evaluation Controls: 97.39% Omniglot accuracy is achieved by Hierarchical Hebbian Memory, compared with 96.14% for the Swin base, 96.74% for fixed Hebbian, and 96.94% for Fully Adaptive Hebbian Routing.
  • C. Memory and Evaluation Controls: 33.56M parameters are used by the hierarchical configuration, compared with 27.51M for the base backbone and 29.94M for Adaptive Hebbian Routing.The additional capacity supports the controller and three memory levels.

B. Few-Shot Memory Ablations

Few-shot memory ablations retain strong recognition across several allocation, routing, access, and semantic-lifecycle choices. In continual recognition, the hierarchy matches the displayed replay peak and obtains higher final accuracy in two of three reported seeds.

  • B. Few-Shot Memory Ablations: All reported Omniglot memory controls remain above 97% accuracy across allocation, access, and semantic-lifecycle choices.
  • B. Few-Shot Memory Ablations: 97.45% is reached by K=1 Persistent, 97.36% by learned Top-2 routing, 97.49% by read-only control, and 97.60% by Semantic Freeze K=1.
  • B. Few-Shot Memory Ablations: 95.37% is reached by both Experience Replay and Full Hierarchy + Replay at the displayed precision.The hierarchical configuration additionally maintains Working Memory, eight routed persistent Episodic Memory banks, and Semantic Memory.
  • B. Few-Shot Memory Ablations: The hierarchy obtains higher final accuracy than Experience Replay in Seeds 1 and 2, while both display 95.37% in Seed 0.

D. Online Continual Performance

On CORe50, the hierarchy maintains strong continual recognition while routed persistent memory substantially improves delayed association retrieval and preserves associations across distractors.

  • 0.9811 online AUC is reached by Full Hierarchy + Replay, compared with 0.9809 for Experience Replay and 0.9776 for the Base model.These values are reported for the selected Swin-Tiny Seed 1 evaluation.
  • 95.37% final accuracy is achieved by both Experience Replay and Full Hierarchy + Replay for Swin-Tiny.The hierarchy matches the displayed peak recognition accuracy while operating all three memory levels.
  • 47.50% delayed-association accuracy is achieved by Learned Multi-Bank retrieval, compared with 24.17% for a Single Persistent Bank and 25.00% without memory.Forcing retrieval through an incorrect bank reduces accuracy to 0.00%.
  • 23.33 percentage points separate Learned Multi-Bank routing from the Single Persistent Bank, while the Wrong-Bank Intervention reduces retrieval to 0.00%.The comparison indicates that the persistent storage location affects later retrieval.
  • 100.00% retrieval is reached when an oracle supplies the correct context bank during storage and retrieval.This provides an upper-bound control for recoverability when the appropriate episodic location is known.
  • After intervening distractors, Episodic Memory retains approximately 0.96 cosine similarity, whereas Working Memory falls to approximately 0.05.The result separates persistent episodic retention from rapidly changing working associations.

H. Routing Organization and Semantic Consolidation

Routing distributes experience across episodic banks while retaining class- and context-related organization, and semantic consolidation updates selectively on a slower schedule.

  • 0.98 normalized utilization entropy shows broadly distributed routing across all available episodic banks.All eight episodic banks remain active, while mean off-diagonal bank cosine is 0.62.
  • 0.56 class-routing NMI indicates a relationship between object class and learned bank allocation.The persistent bank states are neither identical nor completely independent, with mean off-diagonal cosine 0.62.
  • Approximately 0.49 context-routing NMI measures context-dependent routing in the delayed association evaluation.Together with bank-sensitive retrieval results, this indicates that routing influences where associations remain accessible.
  • 4, 3, and 5 semantic consolidations are accepted across the primary CORe50 hierarchy evaluations, while 372, 409, and 408 candidate promotions are rejected.Semantic Memory therefore updates selectively rather than promoting every episodic write.

I. Efficiency

The hierarchy adds moderate computational overhead while supporting strong few-shot and continual recognition and functionally routed, multi-timescale memory.

  • 33.62M parameters and 61.27 ms prediction latency result from Full Hierarchy, versus 27.56M parameters and 55.84 ms for the Swin-Tiny model.Prediction followed by observation and the associated memory operation requires 67.41 ms, with peak VRAM increasing from 2.40 GB to 2.58 GB.
  • 97.39% Omniglot accuracy is achieved by the hierarchical Swin configuration, compared with 96.14% for the Swin base.Fixed Hebbian memory reaches 96.74%, and Adaptive Hebbian Routing reaches 96.94%.
  • 95.37% final CORe50 accuracy is reached by both Experience Replay and Full Hierarchy + Replay for Swin-Tiny.The hierarchy operates Working Memory, multiple persistent Episodic Memory banks, and Semantic Memory at this accuracy.
  • 47.50% delayed retrieval is achieved by Learned Multi-Bank retrieval, compared with 24.17% for a single persistent bank and 25.00% without memory.Incorrect-bank retrieval falls to 0%, while the oracle condition reaches 100%.
  • 0.96 cosine similarity remains in Episodic Memory after distractors, while Working Memory falls to 0.05.All eight episodic banks remain active, with class-routing NMI 0.56 and utilization entropy 0.98.
  • Hebbian association and learned memory allocation are combined within Vision Transformers across rapid, persistent, and consolidated memory timescales.The architecture uses Working Memory, Routed Episodic Memory, and Semantic Memory under a learned controller.
Loading 2609.00358v1…