PaperlayerView exact revision on arXiv

Source-linked AI summary

Auto-GNN: Neural Architecture Search of Graph Neural Networks

Kaixiong Zhou, Qingquan Song, Xiao Huang, Xia Hu

arXiv:1909.03184v2cs.LGstat.ML

TL;DR

GNN architecture design requires laborious tuning, and existing NAS methods do not directly accommodate GNN search spaces, architecture sensitivity, or unstable parameter sharing. AGNN addresses these issues with a reinforcement-learning controller that makes targeted modifications and a constrained sharing strategy. Experiments report that AGNN-discovered architectures outperform handcrafted models and other search methods across transductive and inductive tasks.

  • Problem

    GNN performance depends on architecture choices, while existing NAS methods do not directly handle GNN-specific search spaces, sensitivity to small changes, and unstable parameter sharing.

  • Method

    AGNN searches a predefined GNN space using a reinforcement-learning controller that modifies selected action classes and constrains parameter sharing among homogeneous architectures.

  • Results

    AGNN-discovered architectures consistently outperform state-of-the-art handcrafted models and other search methods on transductive and inductive learning tasks.

  • Takeaways & Limitations

    AGNN provides a GNN-specific architecture-search framework whose controller searches efficiently and whose constrained shared weights can remain effective in offspring networks.

Abstract

from arXiv · show

Graph neural networks (GNN) has been successfully applied to operate on the graph-structured data. Given a specific scenario, rich human expertise and tremendous laborious trials are usually required to identify a suitable GNN architecture. It is because the performance of a GNN architecture is significantly affected by the choice of graph convolution components, such as aggregate function and hidden dimension. Neural architecture search (NAS) has shown its potential in discovering effective deep architectures for learning tasks in image and language modeling. However, existing NAS algorithms cannot be directly applied to the GNN search problem. First, the search space of GNN is different from the ones in existing NAS work. Second, the representation learning capacity of GNN architecture changes obviously with slight architecture modifications. It affects the search efficiency of traditional search methods. Third, widely used techniques in NAS such as parameter sharing might become unstable in GNN. To bridge the gap, we propose the automated graph neural networks (AGNN) framework, which aims to find an optimal GNN architecture within a predefined search space. A reinforcement learning based controller is designed to greedily validate architectures via small steps. AGNN has a novel parameter sharing strategy that enables homogeneous architectures to share parameters, based on a carefully-designed homogeneity definition. Experiments on real-world benchmark datasets demonstrate that the GNN architecture identified by AGNN achieves the best performance, comparing with existing handcrafted models and tradistional search methods.

1 INTRODUCTION

GNN architectures require substantial manual tuning, while existing NAS methods face GNN-specific search-space, efficiency, and parameter-sharing challenges. AGNN addresses these challenges with a tailored search formulation, controller, and sharing strategy.

  • Motivation: GNN architectures require extensive tuning because design choices such as attention heads and hidden dimensions affect performance across graph datasets.Handcrafted architectures can become suboptimal when transferred to other graph-structured datasets.
  • Challenges: Existing NAS methods cannot be directly applied because GNN convolutions use action sequences and slight modifications can substantially change representation capacity.Traditional controllers resample complete architectures, making it difficult to identify which modification changes performance.
  • Challenges: Parameter sharing can become unstable among heterogeneous GNN architectures, motivating constraints on how weights are shared during search.The paper frames stable parameter sharing as a central research question alongside efficient search-space exploration.
  • Contributions: AGNN formally defines GNN NAS, designs an efficient controller, constrains sharing among heterogeneous architectures, and reports consistent gains over handcrafted models and other search methods.The stated experiments evaluate the discovered architectures against existing handcrafted and search-based approaches.

2 PROBLEM STATEMENT

The paper formulates GNN architecture search as validation-metric optimization over a predefined architecture space, with parameters learned on training data. AGNN searches by modifying selected action classes while retaining the best architecture as the search state.

  • Objective: The objective is to find architecture f* in search space F whose trained parameters maximize validation metric M on Dvalid.Training minimizes loss on Dtrain, while architecture selection uses validation performance.
  • Requirements: GNN search requires a graph-convolution-based search space, an efficient controller for small architecture changes, and stable transfer of weights among architectures.The formulation highlights these three characteristics as specific to the GNN search problem.
  • AGNN framework: AGNN represents each graph-convolutional layer with six action classes and uses independent RNN encoders to choose new actions instead of resampling complete architectures.The action classes include choices such as hidden dimension and activation function.
  • AGNN framework: The controller retains the best architecture and updates it only when an offspring architecture performs better; otherwise, it continues from the existing best architecture.This retained architecture provides the starting point for subsequent architecture modifications.

3 SEARCH SPACE

The search space describes message-passing GNN layers through six action classes governing transformations, attention, aggregation, combination, and activation. It covers several established GNN families while yielding a very large architecture space.

  • Message passing: Each GNN layer aggregates neighbor representations, combines them with the node’s own representation, and applies an activation function.These operations define the message-passing computation underlying the search space.
  • Action classes: The search includes candidate hidden dimensions, attention functions, attention-head counts, aggregate functions, combine functions, and activation functions.Available choices include dimensions {4,8,16,32,64,128,256}, heads {1,2,4,6,8,16}, aggregates {SUMMATION, MEAN, MAXPOOLING}, and eight activations.
  • Architecture modification: Figure 1 illustrates modifying one action class across a retained 3-layer architecture, replacing activation functions with ELU, ReLU, and Tanh.The controller evaluates the resulting subarchitectures and selects a new best architecture using decision entropy.
  • Attention functions: The attention-function table specifies alternative attention operations and identifies concatenation and trainable vectors or matrices used in those operations.The caption names || as concatenation and denotes trainable vectors and WG as a trainable matrix.
  • Search-space scale: The framework represents an n-layer architecture as a string of length 6n, with 14,112 combinations per layer and approximately 2.8 × 10^12 three-layer architectures.The search space includes established models such as Chebyshev, GCN, GraphSAGE, GAT, and LGCN.

4 REINFORCED CONSERVATIVE CONTROLLER

The controller combines conservative exploitation with reinforcement-learning-guided, partial architecture mutations to search GNN architectures efficiently. It modifies uncertain action classes and updates their encoders using performance changes from offspring architectures.

  • Controller design: RCNAS retains the best architecture, mutates selected action classes, and trains reinforcement-learning encoders to guide subsequent modifications.Its three components are a conservative explorer, guided architecture modifier, and reinforcement-learning trainer.
  • Controller design: Independent RNN encoders generate new action sequences for each architecture component, such as hidden dimensions or activation functions.Each encoder receives a subarchitecture string formed by removing the concerned action class from the retained architecture.
  • Guided modification: Decision entropy identifies which action classes should be modified, prioritizing classes whose current decisions are more uncertain.The action guider samples from six action classes, with larger entropy increasing a class’s sampling probability.
  • Guided modification: When one class is selected, the controller makes a minimum-level mutation; selecting all six classes resamples the whole architecture like a traditional controller.Activation functions across all convolutional layers can be replaced together when that class is selected.
  • Reinforcement-learning trainer: The trainer applies REINFORCE, using offspring-minus-best performance as the reward for modifying each selected action class.The baseline reduces policy-gradient variance, while the reward represents the performance variation caused by the modification.

5 CONSTRAINED PARAMETER SHARING

The constrained sharing strategy addresses instability when transferring weights between heterogeneous GNN architectures. Sharing is allowed only when layer shapes and functions are compatible, while batch normalization and skip-connection parameters are excluded.

  • Motivation: Heterogeneous architectures differ in trainable-weight shapes or output statistics, making traditional parameter sharing unsuitable for GNN search.Parameter sharing can reduce computation cost, but direct transfer is unstable across such architectural differences.
  • Constrained sharing: The proposed strategy restricts inheritance from ancestor architectures to offspring architectures that satisfy explicit compatibility constraints.The constraints are designed to limit which previously trained parameters an offspring may inherit.
  • Constraint 1: Graph-convolution parameters are shared only when ancestor and offspring layers have the same input and output tensor shapes.The trainable matrix W^(k) and attention transform weight require matching shapes for direct sharing.
  • Constraint 3: Batch-normalization and skip-connection parameters are excluded from parameter sharing under the third constraint.This restriction is explicitly part of the constrained strategy illustrated in Figure 2.
  • Constraint 2: Sharing additionally requires matching attention and activation functions because they determine aggregation and the output interval.These functions substantially determine the layer’s output statistics.

6 EXPERIMENTS

Experiments evaluate AGNN for node classification across transductive and inductive settings, comparing it with handcrafted architectures and alternative search methods. AGNN generally achieves stronger performance and search efficiency, while parameter sharing trades computation cost against model performance.

  • Experimental settings: Experiments use Cora, Citeseer, and Pubmed for transductive learning and PPI for inductive learning.These datasets are benchmarked for node classification, with citation networks used in the transductive setting.
  • Baselines: The evaluation compares AGNN with handcrafted GNN architectures and other neural architecture search approaches.Handcrafted baselines include Chebyshev, GCN, GraphSAGE, GAT, and LGCN, with setting-specific restrictions.
  • Performance comparison: AGNN discovers architectures that outperform handcrafted models and other search methods across the reported transductive and inductive comparisons.The discovered models generally improve classification performance while exploring larger attention-head and hidden-dimension configurations.
  • Parameter sharing: Architectures found without parameter sharing generally outperform those found with sharing, but require about 0.5 GPU days and therefore create a performance–computation trade-off.The reported runtime uses a single Nvidia GTX 1080Ti GPU, and unshared training costs a few times more than shared training.
  • Search efficiency: AGNN finds well-performing architectures more efficiently than GraphNAS and random search during the search progress.The top-10 architectures found by AGNN have better averaged performance on PPI and Citeseer.
  • Parameter sharing: Constrained parameter sharing yields better performance than relaxed sharing for most discovered architectures by restricting sharing to homogeneous architectures with similar output statistics.Warm-up epochs for batch normalization and skip connections further help shared weights transfer to newly sampled architectures.
  • Architecture modification: Smaller architecture modifications improve search efficiency, with s = 1 achieving the best top-10 averaged performance on PPI and Citeseer.Small modifications also preserve similar structure and output statistics, making transferred weights more likely to remain effective.

7 RELATED WORK

The paper relates AGNN to graph neural networks built around message passing and to neural architecture search methods based primarily on reinforcement learning or evolutionary algorithms.

  • Graph Neural Networks: GNN research includes recursive neural networks, graph convolutional networks, and graph attention networks for learning node representations.Most approaches use message-passing graph convolutions involving information transformation, neighbor aggregation, and self-updating.
  • Neural Architecture Search: NAS frameworks commonly use reinforcement learning or evolutionary algorithms to explore neural architectures.RL methods use RNN controllers and policy gradients, whereas evolutionary methods use population initialization, mutation, and crossover.

8 CONCLUSION

AGNN is presented as a framework for finding neural architectures for node classification, combining a tailored search space, RCNAS controller, and constrained parameter sharing. Experiments report competitive performance across transductive and inductive learning tasks.

  • AGNN finds neural architectures for a given node classification task.
  • Its search space, RCNAS controller, and constrained parameter sharing strategy are designed for message-passing-based GNNs.
  • The discovered architectures achieve competitive performance on both transductive and inductive learning tasks.
  • The RCNAS controller searches for well-performing architectures more efficiently, while shared weights remain effective in offspring networks under constraints.
  • Future work includes applying AGNN to graph classification and link prediction and adding more advanced graph convolutions to the search space.
Loading 1909.03184v2…