Source-linked AI summary
Overcoming Catastrophic Forgetting in Graph Neural Networks
Huihui Liu, Yiding Yang, Xinchao Wang
TL;DR
Catastrophic forgetting is well documented in sequential learning, but prior methods largely target CNNs and overlook graph-structured inputs handled by GNNs. The paper introduces topology-aware weight preserving (TWP), which protects task-important parameters and topological aggregation in a plug-and-play module. Across three GNN backbones and five datasets, TWP consistently achieves the best performance among compared methods.
Problem
Continual learning methods have largely focused on CNNs with grid-structured inputs, while GNNs handling non-grid graph data remain vulnerable to catastrophic forgetting.
Method
TWP measures parameter importance using task-related and topological information, penalizing changes to important parameters while preserving topological aggregation in arbitrary GNNs.
Results
TWP consistently achieves the best performance among compared methods across GAT, GCN, and GIN backbones evaluated on five datasets.
Takeaways & Limitations
The approach provides a continual-learning scheme tailored to graph data that is applicable to both node-level and graph-level tasks.
Takeaways & Limitations
The continual-learning setting assumes disjoint tasks and removes data from a completed task once its learning is finished.
Abstract
from arXiv · showhide
Catastrophic forgetting refers to the tendency that a neural network "forgets" the previous learned knowledge upon learning new tasks. Prior methods have been focused on overcoming this problem on convolutional neural networks (CNNs), where the input samples like images lie in a grid domain, but have largely overlooked graph neural networks (GNNs) that handle non-grid data. In this paper, we propose a novel scheme dedicated to overcoming catastrophic forgetting problem and hence strengthen continual learning in GNNs. At the heart of our approach is a generic module, termed as topology-aware weight preserving~(TWP), applicable to arbitrary form of GNNs in a plug-and-play fashion. Unlike the main stream of CNN-based continual learning methods that rely on solely slowing down the updates of parameters important to the downstream task, TWP explicitly explores the local structures of the input graph, and attempts to stabilize the parameters playing pivotal roles in the topological aggregation. We evaluate TWP on different GNN backbones over several datasets, and demonstrate that it yields performances superior to the state of the art. Code is publicly available at \url{https://github.com/hhliu79/TWP}.
Introduction
Continual learning methods have largely targeted CNNs, leaving GNNs vulnerable to catastrophic forgetting because graph topology affects information propagation. The paper proposes TWP, a plug-and-play module that preserves both task-relevant parameters and topological aggregation, and reports consistently strong results across GNNs and datasets.
- Deep models can forget previously learned tasks when trained sequentially, causing inferior performance on earlier tasks.
- CNN-oriented continual learning methods overlook graph topology, despite interconnections being central to information propagation and aggregation in GNNs.
- 7.48% accuracy is lost on task 1 after learning task 3 when a CNN-based method is applied to graph data.
- TWP estimates parameter importance from task-related and topology-preserving objectives, then penalizes changes to significant parameters during new-task learning.
- The experiments report that TWP consistently achieves the best performance among the compared methods.
- TWP is designed as a portable module for arbitrary GNN architectures and is evaluated with GAT, GCN, and GIN backbones across five datasets.
Related work
Related continual-learning research has used rehearsal, parameter regularization, parameter isolation, and model-reuse strategies. These approaches have also introduced pseudo-rehearsal and generative methods as memory and computation costs grow with task count.
- Rehearsal methods store samples from previous tasks and replay them while learning a new task.
- Memory-module approaches can incur rapidly increasing computation and memory costs as the number of tasks grows, motivating pseudo-rehearsal and generative methods.
- Regularization methods preserve important parameters from earlier tasks by slowing or limiting their updates.
- Parameter-isolation methods assign different parameter subsets to different tasks while reusing prior knowledge.
Preliminaries
GNNs represent graph-structured data through node features, adjacency relations, and neighborhood aggregation, with architectures differing in how aggregation weights are obtained. The continual-learning formulation trains a shared model sequentially on disjoint tasks whose earlier data become unavailable.
- Graph Neural Networks: A graph is specified by nodes, node features, and an adjacency matrix encoding relations among nodes.
- Graph Neural Networks: A GNN computes each node’s hidden representation by aggregating information from neighboring nodes through layer transformations and activation functions.
- Graph Neural Networks: The aggregation strategy is a core component of GNNs, represented by a matrix that determines how neighbor information is combined.
- Graph Neural Networks: GCNs use an aggregation matrix derived from a first-order spectral approximation and fixed by the graph adjacency matrix.
- Graph Neural Networks: GATs compute aggregation weights through pair-wise attention, while GINs extend the expressive power of GNNs.
- Problem Formulation.: Continual learning presents a sequence of disjoint tasks with separate training and testing node sets, and data from completed tasks is no longer available.
- Problem Formulation.: The goal is to learn one shared GNN that performs well on new graph-related tasks while remembering old tasks.
Method
TWP preserves GNN knowledge by identifying parameters important to both task performance and graph topology, then stabilizing those parameters while retaining capacity for future tasks. The module extends to arbitrary GNNs by constructing topology information when attention is unavailable.
- TWP module: TWP combines minimized-loss preserving and topological-structure-preserving submodules to identify crucial network parameters.Parameter importance reflects contributions to the task-related objective and topology-related objective.
- Minimized loss preserving: Parameter importance for the task loss is estimated from the magnitude of each parameter’s loss gradient.The gradient approximates a parameter’s contribution to the loss under infinitesimal perturbation.
- Topological structure preserving: For GATs, TWP models attention coefficients between a center node and its neighbors as local topological information.The method computes parameter sensitivity to these coefficients and aggregates the resulting importance scores.
- Importance scoring: The final parameter importance combines task-related and topology-related scores using hyper-parameters λl and λt.Parameters with low importance can change for subsequent tasks, whereas high-importance parameters are protected.
- Continual learning objective: When learning a new task, TWP penalizes changes to parameters important for previous tasks while optimizing the new task’s loss.This preserves previously learned knowledge through stability constraints on important parameters.
- Plasticity and stability: An l1 regularizer on current-task importance scores preserves future learning capacity by reducing the number or magnitude of highly important parameters.The hyper-parameter β controls the capacity reserved for future tasks; higher β preserves more capacity.
- Extension to general GNNs: TWP extends to GCNs and GINs by adding a non-parametric attention mechanism that constructs topology information for arbitrary GNNs.These constructed attention weights are used only by TWP, while the original graph remains responsible for feature aggregation and updates.
Experiments
The experiments evaluate graph continual learning across node- and graph-level tasks, covering transductive and inductive datasets with multiple task configurations.
- The evaluation uses four node-classification datasets and one graph-classification dataset.The node datasets are Corafull, Amazon Computers, PPI, and Reddit; the graph dataset is Tox21.
- Corafull contains nine five-way tasks, while Amazon Computers contains five two-class tasks.
- PPI contains 12 ten-class tasks, and Reddit contains eight five-class tasks under an inductive setting.
- Tox21 contains 8014 compounds and 12 binary classification tasks, one for each target.
Baselines
The study compares TWP with CNN-oriented continual-learning baselines adapted to graphs, using average performance and average forgetting to assess learning quality and retention.
- The baselines include Fine-tune, LWF, EWC, MAS, and GEM, with Joint train serving as an approximate upper bound.
- Fine-tune provides a lower bound because it uses no continual-learning mechanism, whereas Joint train accesses data from all learned tasks.
- Average performance measures mean test performance across learned tasks, while average forgetting measures performance loss after subsequent tasks.
Experimental Setup
The setup tests TWP across GAT, GCN, and GIN backbones, reports task-specific metrics, and averages results over five random seeds.
- The evaluation uses GATs, GCNs, and GINs to assess TWP across different GNN architectures.For GCNs, attention coefficients are computed before estimating parameter importance; GINs are adapted for graph-level classification.
- Task performance uses accuracy for Corafull and Amazon Computers, and micro-averaged F1 score for PPI and Reddit.
- Table 1 highlights the best and second-best results, with higher or lower values preferred according to each metric.
- Results are reported as the mean and standard deviation from five runs with random seeds.
Node Classification Task
Across node-classification experiments, TWP consistently achieves strong retention and overall performance, while preserving topological information during sequential task learning.
- Node Classification Task: TWP achieves the best or second-best average performance and average forgetting across node-classification datasets and GNN backbones.
- Node Classification Task: TWP shows minimal forgetting on the first task and remains close to the Joint train upper bound as more tasks are learned.
- Node Classification Task: Figure 4 tracks each task’s performance as new tasks are added, whereas Figures 5 and 6 show first-task and average performance over time.
- Node Classification Task: TWP further improves over EWC and MAS, showing that topology-aware preservation strengthens parameter-protection methods for graph continual learning.
- Node Classification Task: On Corafull with GATs, TWP’s performance degrades more slowly than baselines and is generally similar to Joint train.
Graph Classification Task
On the Tox21 graph-level task, experiments with GAT and GCN backbones use AUC to assess continual-learning performance, with TWP improving forgetting performance.
- Tox21 is used to evaluate the graph-level continual-learning task with GATs and GCNs as base models and AUC as the metric.
- TWP boosts forgetting performance by a significant margin on the graph-level experiment.
Ablation Study
The ablation study separates task-loss importance, topology-aware importance, and parameter-capacity preservation, finding that topology and minimized importance scores improve performance.
- Using only the original task loss reduces the method to EWC, which measures parameter importance from the task-related loss alone.
- TWP achieves consistently better performance than the task-loss-only variant on two datasets, supporting the value of preserving topological information.
- Minimizing importance scores across parameters further improves overall performance by preserving model capacity for future tasks.
Conclusion
The paper presents a dedicated, topology-aware continual-learning method for GNNs that preserves important parameters and extends across GNN architectures and graph-task settings.
- The proposed TWP module measures parameter importance using both task-related loss and graph topological information.
- During new-task learning, penalizing changes to important parameters helps the model remember old tasks.
- TWP can be extended to arbitrary GNNs and is evaluated on node-level and graph-level tasks.