Lifelong Learning of Spatiotemporal Representations with Dual-Memory Recurrent Self-Organization

• Replay can support consolidation without explicitly storing past samples, which matters when agents have restricted access to prior experience.The introduction notes that replay may be especially useful when classes are unbalanced or a sample is encountered only once.Source: §1 · pp. 1–2
• Existing lifelong-learning approaches often use limited-complexity static-image datasets and do not account for temporal structure in progressively available sequential input.Lifelong learning also requires retaining prior experience as new tasks arrive, despite catastrophic forgetting and restricted access to past experience.Source: §1 · pp. 1–2
• The growing dual-memory architecture learns object instances in an unsupervised episodic memory and develops compact category representations in a task-regulated semantic memory.Both recurrent self-organizing memories can expand; the semantic memory uses task-relevant labels to regulate neurogenesis and neural updates.Source: §1 · p. 2
• The episodic memory periodically replays internally generated neural activity to both memories without sensory input, mitigating catastrophic forgetting during incremental learning.Source: §1 · pp. 2–3
• The model outperforms current lifelong-learning approaches in three incremental learning scenarios on CORe50 continuous object recognition.The benchmark includes 50 objects across 10 categories, with sequences captured under varied conditions.Source: §1 · p. 3

• Dual-memory approaches include GeppNet+STM, which replays stored novel knowledge during sleep-interleaved training, and FearNet, which consolidates hippocampal memories into a prefrontal network.Other systems use generative pseudo-rehearsal, but one requires an oracle to select the memory module during prediction.Source: §2 · p. 3
• Regularization methods protect learned knowledge by constraining weight updates, but can trade off old-task and new-task performance when neural resources are limited.Examples include LwF, EWC, and online synaptic-importance estimation; their specific constraints differ.Source: §2 · pp. 3–4
• Architecture expansion accommodates new knowledge, while exemplar replay combines stored old examples with new data; these strategies can increase model complexity or memory footprint.Expansion can grow with the number of tasks, whereas iCaRL stores example data points.Source: §2 · p. 4
• Earlier recurrent self-organizing networks learned temporal structure through prediction-driven dynamics, but lacked task-relevant plasticity regulation and could not learn without external sensory input.Source: §2 · p. 4

3 Proposed Method

• GDM combines a deep convolutional feature extractor with recurrent episodic and semantic memories that grow neurons and connections in response to sequential input.G-EM learns unsupervised from sensory experience; G-SM receives its activation trajectories and uses input annotations to regulate structural plasticity.Source: §3 · p. 4

3.1 Gamma-GWR

• Gamma-GWR integrates temporal context into self-organizing dynamics, creating neurons when network activity and BMU habituation meet insertion conditions.Its connections form through co-activation, with aging links and isolated neurons eligible for removal.Source: §3.1 · pp. 5–6

3.2 Episodic Memory

• G-EM learns fine-grained episodic representations from image-frame features, with insertion threshold controlling spatial resolution and context descriptors controlling temporal depth.Temporal synapses link consecutively activated neurons, enabling sequence-selective activation trajectories; episodic neurons also store instance and category labels.Source: §3.2 · pp. 6–7

3.3 Semantic Memory

• G-SM uses category labels to regulate neuron growth and updates, developing compact representations of statistical regularities across episodic experience.Its neurons code over longer temporal windows, with temporal slowness arising from learned spatiotemporal regularities.Source: §3.3 · p. 7

3.4 Memory Replay

• After each learning episode, G-EM generates and replays temporally ordered neural-activity trajectories to both memories, preserving input temporal structure without storing the sequences explicitly.Each trajectory has length K_EM + K_SM + 1, and the set of trajectories is replayed to both memories after an episode.Source: §3.4 · p. 8

4.1 Feature Extraction

• CORe50's 50 objects span 10 categories and 11 sessions; the same pre-trained VGG features were compressed from 2048 to 256 dimensions.The experiments used 128 × 128 RGB images at 5 Hz and focused on context-aware networks rather than comparing feature extractors.Source: §4.1 · pp. 8–9

4.2 Batch Learning

• 79.43% instance-level and 93.92% category-level accuracy with temporal context improved on VGG+FT baseline results by 10.35% and 13.69%, respectively.G-EM grew more neurons than G-SM, while habituation reduced update rates and helped protect consolidated knowledge as new experience arrived.Source: Table 2 · p. 10, §4.2 · p. 10, Figure 3 · p. 11

4.3 Incremental Learning

• 75.93% ± 2.23 instance-level and 85.53% ± 1.35 category-level accuracy under incremental learning were 3.5% and 8.39% below batch results.Categories were presented one at a time and not revisited; accuracy declined as categories accumulated, and exposure order affected final results.Source: §4.3 · p. 11, Figure 4 · p. 11

4.4 Incremental Learning with Memory Replay

• 82.14%±2.05 instance-level and 91.18% ± 0.25 category-level accuracy with replay, versus 75.93%±2.23 and 85.53% ± 1.35 without.Accuracy on the first five instances rose from 69.25% ± 2.01 to 80.41% ± 1.35, and on the first category from 79.53% ± 5.23 to 89.21% ± 3.37.Source: §4.4 · p. 12, Figure 5 · p. 13

4.5 Continuous Object Recognition

• Comparisons are hindered by other methods' lack of recurrent dynamics, and evaluation always assigns labels, leaving unseen-class detection outside the tested setting.The authors identify novelty detection for unseen classes as a useful addition.Source: §4.5 · pp. 13–14
• With memory replay, the proposed method achieved state-of-the-art CORe50 accuracy of 87,94% (NI), 86.14% (NC), and 87.06% (NIC), over 20% higher in each scenario than the previous best cumulative approach.Source: §4.5 · p. 13

5.1 Summary

• GDM significantly outperforms state-of-the-art lifelong-learning methods on three CORe50 incremental learning tasks.Source: §5.1 · p. 14
• The model pairs recurrent episodic memory for fine-grained spatiotemporal experience with semantic memory for compact category representations.Source: §5.1 · p. 14

5.2 Growing Recurrent Networks with Memory Replay

• Gamma-GWR grows neurons when needed after existing units train, unlike Gamma-GNG’s fixed growth interval.Source: §5.2 · p. 15
• Gamma-GWR is computationally incremental, but its growth mechanisms do not resemble biologically plausible neurogenesis or synaptogenesis; adult hippocampal neurogenesis remains contested.Source: §5.2 · p. 15
• Periodic replay of episodic neural-activity trajectories consolidates knowledge without storing training sequences and significantly improves incremental-learning accuracy.Source: §5.2 · p. 15
• Connection and isolated-neuron deletion were not investigated, and age thresholds are difficult to set when consolidated neurons remain inactive for long periods.Source: §5.2 · pp. 15–16
• ART shares adaptive unsupervised growth with GWR, but some variants depend on training order and lack extensive evaluation on recent lifelong-learning benchmarks.Source: §5.2 · p. 16

5.3 Conclusion

• The authors identify multisensory learning, such as audio-visual streams, as a future extension and frame GDM as a step toward embodied lifelong-learning agents.Source: §5.3 · p. 16