PaperlayerView exact revision on arXiv

Source-linked AI summary

Simplifying approach to Node Classification in Graph Neural Networks

Sunil Kumar Maurya, Xin Liu, Tsuyoshi Murata

arXiv:2111.06748v1stat.MLcs.LG

TL;DR

Combining feature propagation and transformation in GNN layers makes it difficult to distinguish feature importance from the role of the MLP. FSGNN decouples feature generation from learning and uses soft selection; it outperforms state-of-the-art GNNs on node classification, with gains up to 51.1%.

  • Problem

    Combining feature propagation and transformation in GNN layers makes it difficult to distinguish feature importance from the role of the MLP.

    Source: §1 · p. 1, §3.1.1 · p. 3

  • Method

    FSGNN separates feature generation from representation learning and uses learned scalar weights to soft-select input features in a two-layer model.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    Up to 51.1% higher node classification accuracy than state-of-the-art GNN models, with FSGNN outperforming both shallow and deep models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    FSGNN’s learned parameters provide information about feature importance, and the experiments support feature selection as a direction for GNN architectures.

    Source: §8 · p. 9

    FSGNN does not completely reduce the effect of noisy or irrelevant features without an explicit forgetting scheme.

    Source: §7.2 · p. 7

Abstract

from arXiv · show

Graph Neural Networks have become one of the indispensable tools to learn from graph-structured data, and their usefulness has been shown in wide variety of tasks. In recent years, there have been tremendous improvements in architecture design, resulting in better performance on various prediction tasks. In general, these neural architectures combine node feature aggregation and feature transformation using learnable weight matrix in the same layer. This makes it challenging to analyze the importance of node features aggregated from various hops and the expressiveness of the neural network layers. As different graph datasets show varying levels of homophily and heterophily in features and class label distribution, it becomes essential to understand which features are important for the prediction tasks without any prior information. In this work, we decouple the node feature aggregation step and depth of graph neural network, and empirically analyze how different aggregated features play a role in prediction performance. We show that not all features generated via aggregation steps are useful, and often using these less informative features can be detrimental to the performance of the GNN model. Through our experiments, we show that learning certain subsets of these features can lead to better performance on wide variety of datasets. We propose to use softmax as a regularizer and "soft-selector" of features aggregated from neighbors at different hop distances; and L2-Normalization over GNN layers. Combining these techniques, we present a simple and shallow model, Feature Selection Graph Neural Network (FSGNN), and show empirically that the proposed model achieves comparable or even higher accuracy than state-of-the-art GNN models in nine benchmark datasets for the node classification task, with remarkable improvements up to 51.1%.

1. Introduction

Combining feature aggregation and transformation makes it difficult to distinguish feature importance from neural-network expressiveness. The paper separates feature propagation from learning and proposes a soft-selection GNN that reports higher node-classification accuracy than state-of-the-art models.

  • Many existing GNN variants are designed for homophily and may perform poorly on heterophily datasets, where linked nodes can have different labels.Source: §1 · p. 1
  • Combining aggregation and transformation obscures which features and neural-network components contribute to prediction, so the paper studies them separately.Source: §1 · pp. 1–2
  • Experiments on multiple node-classification benchmarks identify hop-feature selection as essential for higher prediction accuracy.Source: §1 · p. 2
  • The paper reports that feature-generation requirements differ between homophily and heterophily graphs.Source: §1 · p. 2
  • FSGNN1 is a simple two-layer GNN that uses soft selection to learn feature importance during training.Source: §1 · p. 2
  • Up to 51.1% higher node-classification accuracy is reported for the proposed model than for other state-of-the-art shallow and deep GNNs.Source: §1 · p. 2

2. Preliminaries

The preliminaries define graph and node-feature notation and frame GNNs as neighborhood aggregation followed by learned transformations. They set up node classification and distinguish feature-generation choices across GNN architectures.

2. Preliminaries

  • The setup represents an undirected graph with adjacency matrix A and node features X; Figure 1 contrasts aggregation with and without self-loops.The adjacency matrix records edges between nodes, while X contains their feature vectors.Source: §2 · p. 2, Figure 1 · p. 2

2.1. Graph Neural Networks

  • Conventional GNN layers aggregate neighborhood features and apply learned transformations, while heterophily motivates separating a node’s features from its neighbors’.The conventional self-loop formulation cumulatively aggregates self and neighbor features; H2GCN separates them to avoid aggregating dissimilar features.Source: §2.1 · p. 2
  • For multi-hop features, the formulation uses a symmetric normalized adjacency without self-loops and allows concatenating hop features before the final layer.The two-layer GNN formulation is presented using the adjacency matrix A.Source: §2.1 · p. 2

2.2. Node Classification

  • Node classification trains a GNN to predict node labels from features and graph structure, using two-hop aggregation, row-wise softmax, and cross-entropy on labeled training examples.After training, the model predicts labels for test nodes.Source: §2.2 · p. 3

2.3. Feature Generation in GNNs

  • Feature generation varies with aggregation choices and architecture: models may combine aggregation with representation learning in a layer or precompute features beforehand.Generated node features can capture homophilic or heterophilic properties and can also use feature-space proximity or other criteria.Source: §2.3 · p. 3

3. Feature Selection in Graph Neural Networks

The section examines how aggregated node features contribute to node classification by separating feature generation from representation learning. Experiments compare individual features, all features, and feature subsets across datasets with different homophily and heterophily properties.

3.1.1. Model Design

  • The experiments decouple feature generation from representation learning because conventional stacked GNN layers make feature importance and the role of the MLP difficult to distinguish.They use a two-layer neural network with separate linear transformations for each input feature, row-wise L2-normalization, and sum or concatenation aggregation.Source: §3.1.1 · p. 3

3.1.2. Input Features

  • The study considers seven features from the node itself and up to three hops, with and without self-loops, and tests Single_feature, All_feature, and Sub_feature settings.Sub_feature evaluates 119 feature subsets; All_feature uses sum or concatenation aggregation.Source: §3.1.2 · pp. 3–4

3.1.3. Datasets and Hyperparameters

  • The experiments cover nine datasets with varying homophily and heterophily properties, and search 54 hyperparameter combinations for each input feature setting.The searched parameters are learning rate, weight decay, and dropout.Source: §3.1.3 · p. 4

3.2. Analysis of results

  • The most informative features varied by dataset: self-looped features led on Cora, Citeseer, and Pubmed, while first-hop no-loop features led on Squirrel and Chameleon.Node features were best on Wisconsin, Texas, Cornell, and Actor; the authors connect these patterns to differing homophily and heterophily properties.Source: §3.2 · p. 4
  • Sub_feature selection significantly improved performance over Single_feature and All_feature settings on all datasets, while less informative features could act as noise.The findings motivate feature selection as a way to reduce the effect of uninformative features in a shallow network.Source: §3.2 · p. 4
  • In Sub_feature, the performance gap between concatenation and sum aggregation was smaller than in All_feature, where sum performed significantly worse on several datasets.The cited examples for the All_feature comparison are Chameleon, Wisconsin, Texas, Cornell, and Squirrel.Source: §3.2 · p. 4

4. Proposed Architecture

The proposed approach assigns trainable scalar weights to input feature matrices and constrains them with softmax to favor more relevant features. FSGNN combines this soft-selection mechanism with L2-normalization in a shallow, two-layer model.

4. Proposed Architecture

  • Softmax-constrained scalar weights scale each input feature matrix, allowing the model to learn higher weights for relevant features and lower weights for less useful ones.The authors call this a soft-selection procedure because it does not make a binary choice of features.Source: §4 · p. 5
  • The feature weights can also be viewed as softmax regularization on polynomial-filter coefficients or as a weighting scheme over input features.The polynomial-filter view allows P to be an adjacency, Laplacian, or PageRank-based matrix.Source: §4 · p. 5
  • Each input feature matrix has its own weight matrix because the scalar feature weights affect the magnitudes and gradients reaching the linear layer.Initially equal weights normalize to 1/L for L feature matrices.Source: §4 · p. 5

Feature Selection Graph Neural Network

  • FSGNN is a shallow, two-layer model that transforms precomputed features with a distinct linear layer for each feature matrix, then applies L2-normalization and softmax-regularized scalar weights.The weighted outputs are concatenated, passed through ReLU, and mapped to output logits for cross-entropy loss.Source: Feature Selection Graph Neural Network · pp. 5–6

5. Related Work

Related work spans methods for propagation, attention, sampling, and deeper GNN architectures, as well as approaches targeting heterophily. The paper positions its feature-selection approach alongside prior work that decouples feature generation and representation learning.

Related GNN architectures

  • Prior work addresses propagation with GraphSAGE [17], FastGCN [18], GAT, APPNP, JK, Geom-GCN, SimP-GCN, and CPGNN, while deeper models address over-smoothing with methods including DropEdge and GCNII.The passage also identifies RevGNN and noise regularisation [39] as approaches to training deep GNNs.Source: §5 · p. 6

Heterophily

  • H2GCN, CPGNN, TDGNN, Geom-GCN, and GPRGNN are presented as models designed for heterophily, though GCNs can perform strongly under certain conditions [40].The related-work discussion contrasts traditional GNN performance on homophily graphs with their weaker generalization to heterophily graphs.Source: §5 · p. 6

Feature selection and related approaches

  • SGC and SIGN [27] decouple feature generation and representation learning but correspond to the paper’s Single_feature and All_feature settings; prior weighting schemes may be fixed or lack explicit regularization.The authors also note that the compared models do not use no-loop features by design, which they describe as limiting learning capability on heterophily datasets.Source: §5 · p. 6

6. Experiments

The experiments evaluate node classification on nine commonly used datasets and compare FSGNN with other GNN models. They describe dataset properties, preprocessing, task splits, input settings, and baseline selection.

6. Experiments

  • The experiments assess the proposed model on real-world datasets for node classification and compare it with other GNN models.The study uses nine fully supervised datasets, including citation, webpage, actor-co-occurrence, and topic-page graphs, characterized as homophily or heterophily datasets.Source: §6 · p. 6, §6.1 · p. 6

6.1. Datasets

  • The nine datasets include citation networks considered homophilic and webpage, actor-co-occurrence, and topic-page graphs considered heterophilic; the comparisons use public splits from.The paper defines homophily ratio as the fraction of edges connecting nodes with the same label.Source: §6.1 · p. 6

6.2. Preprocessing

  • Preprocessing follows and: node features are row-normalized, and symmetrically normalized adjacency matrices with and without self-loops are stored as sparse matrices.Source: §6.2 · p. 6

6.3. Settings and Baselines

  • Each dataset is split by class into 48% training, 32% validation, and 20% testing, with mean accuracy reported over 10 random splits.The model uses 64-dimensional embeddings and is evaluated with conventional homophily/heterophily-specific inputs and with all features; comparisons include 10 baselines.Source: §6.3 · p. 7

7. Results

FSGNN is comparable to state-of-the-art on homophily datasets and reports substantial gains on several heterophily datasets. Ablations and analyses show that useful hops vary by dataset, while performance is relatively stable as aggregation depth increases.

7.1. Node Classification Results

  • FSGNN reports 51.1% improvement on Squirrel and 18.8% on Chameleon, with comparable and consistent performance across homophily datasets.Different models perform best on different homophily datasets.Source: §7.1 · p. 7

7.2. Comparison with Sub_feature

  • On five datasets FSGNN matches Sub_feature at hidden dimension 64; on the others, performance is comparable but somewhat lower.The model learns to reduce irrelevant features’ effects, but without an explicit forgetting scheme it cannot completely remove noisy features.Source: §7.2 · p. 7

7.3. Ablation Studies

  • Across 1080 hyperparameter combinations, proposed designs usually improve average accuracy, but dataset-specific exceptions and sensitivity make exploration important.Cora and Citeseer average performance is better without softmax regularization; Actor is the only dataset where adding normalization reduces accuracy.Source: §7.3 · pp. 7–8, Table 4 · p. 8

7.4. Soft-Selection Parameter Analysis

  • Learned feature weights favor self-looped features on homophily datasets, while Chameleon and Squirrel benefit most from self and first-hop no-self-loop features.On Wisconsin, Cornell, Texas, and Actor, self-features receive the greatest weights, suggesting graph structure plays a limited role in accuracy there.Source: §7.4 · p. 8

7.5. Over-smoothing Analysis

  • With 3, 8, 16, and 32 hops, performance varies little and FSGNN does not suffer over-smoothing; higher-hop features can receive low weights.Higher accuracies were reported for Citeseer (77.46%), Cornell (88.11%), and Squirrel (74.15%).Source: §7.5 · pp. 8–9, Figure 4 · p. 8

8. Conclusion

The conclusion presents feature selection as a direction for GNN architecture exploration and introduces FSGNN. Experiments report strong node-classification performance, feature-importance information from learned parameters, and avoidance of over-smoothing with a shallow model.

8. Conclusion

  • The experiments support feature selection as a direction for exploring GNN architectures.The paper introduces FSGNN based on its experimental observations.Source: §8 · p. 9
  • FSGNN outperforms current state-of-the-art GNN models on node classification, according to the paper’s experiments.The learned parameters provide information about feature importance.Source: §8 · p. 9
  • The paper reports that its shallow model provides high prediction accuracy and avoids over-smoothing.Source: §8 · p. 9

Appendices

Increasing the hidden dimension of a two-layer MLP under the reported feature settings can approach or exceed state-of-the-art accuracy, with diminishing returns at larger dimensions.

A.1. Node classification on higher dimensions

  • Increasing the hidden dimension of a two-layer MLP can approach or exceed state-of-the-art accuracy under the reported feature settings.The comparisons cover Single_Feature, All_Feature, and Sub_Feature at dimensions 64, 128, 256, and 512.Source: §A.1 · p. 10
  • More model parameters improve classification accuracy, but accuracy gains diminish as hidden dimensions increase.Source: §A.1 · p. 10
  • Mean node-classification accuracy is compared across three feature settings and four hidden dimensions.Source: §A.1 · p. 10

A.2. Experiments with no non-linear activation

The appendix examines a two-layer MLP under Sub_Feature settings without an intermediate ReLU. Accuracy rises on three datasets and declines only slightly on most others, with larger drops on Chameleon and Squirrel.

A.2. Experiments with no non-linear activation

  • The appendix frames the comparison as a test of removing the intermediate ReLU from the Sub_Feature two-layer MLP.Other learning settings are kept the same.Source: §A.2 · p. 10
  • Removing ReLU further improves accuracy on Cora, Texas, and Cornell; most other datasets decline only slightly, except Chameleon and Squirrel.The comparison uses Sub_Feature with hidden dimension 256 and removes only the ReLU between layers.Source: §A.2 · pp. 10–11, Table 6 · p. 10
  • For the examined datasets, simple graph convolution with hop-feature selection provides sufficient information for a simple two-layer model to perform well.Source: §A.2 · p. 11

B. Implementation Details of FSGNN

The section documents FSGNN’s experimental setup and hyperparameter selection for reproducibility. It reports the software and hardware environment, grid-search settings, and patience value.

B. Implementation Details of FSGNN

  • Experiments use PyTorch 1.6.0 on Python 3.8, with an Nvidia V100 GPU, 16 GB graphics memory, and CUDA 10.2.89.The setup details are provided for reproducibility.Source: B. Implementation Details of FSGNN · p. 11
  • The authors grid-search learning rate, layer weight decay, and inter-layer dropout, using ReLU, Adam, and patience 100 across datasets.The passage identifies Table 7 as the search space and Tables 8 and 9 as the best hyperparameters for the 3-hop and 8-hop configurations.Source: B. Implementation Details of FSGNN · p. 11
  • Table 9 is identified as the hyperparameter table for the 8-hop all-features model.The passage provides the table’s scope but no hyperparameter values.Source: Table 9 · p. 11
Loading 2111.06748v1…