PaperlayerView exact revision on arXiv

Source-linked AI summary

GNN-FiLM: Graph Neural Networks with Feature-wise Linear Modulation

Marc Brockschmidt

arXiv:1906.12192v5cs.LGstat.ML

TL;DR

Standard GNN message passing often computes messages from source nodes without using target representations for feature-wise interactions. The paper introduces GNN-FiLM, which uses target states to compute element-wise affine message transformations, and evaluates it against reimplemented baselines across three tasks. GNN-FiLM is competitive with or outperforms GNN-MLP and established baselines, while RGDCN is impractical to train.

  • Problem

    Standard GNN message transformations generally omit target-node representations, while richer source–target transformations are computationally impractical.

  • Method

    The paper introduces GNN-FiLM, which combines learned message-passing functions with target-conditioned, dynamically computed element-wise affine transformations.

  • Results

    Across three tasks, GNN-FiLM is competitive with or outperforms GNN-MLP and established baselines, while GNN-MLP also outperforms better-published techniques in the experiments.

  • Takeaways & Limitations

    The experiments support including simple source–target MLP baselines and using substantial, consistent hyperparameter search when evaluating GNN architectures.

  • Takeaways & Limitations

    RGDCN is infeasible to train because it is extremely sensitive to parameter initialisation, so no experimental results are reported for it.

Abstract

from arXiv · show

This paper presents a new Graph Neural Network (GNN) type using feature-wise linear modulation (FiLM). Many standard GNN variants propagate information along the edges of a graph by computing "messages" based only on the representation of the source of each edge. In GNN-FiLM, the representation of the target node of an edge is additionally used to compute a transformation that can be applied to all incoming messages, allowing feature-wise modulation of the passed information. Results of experiments comparing different GNN architectures on three tasks from the literature are presented, based on re-implementations of baseline methods. Hyperparameters for all methods were found using extensive search, yielding somewhat surprising results: differences between baseline models are smaller than reported in the literature. Nonetheless, GNN-FiLM outperforms baseline methods on a regression task on molecular graphs and performs competitively on other tasks.

1. Introduction

The paper identifies a limitation in standard graph message passing: messages generally depend on source-node representations, while target-node information is underused. It proposes GNN-FiLM, which uses target representations to compute efficient feature-wise transformations, and evaluates it against baselines across three tasks.

  • 1. Introduction: Standard neural graph methods initialise node representations, transform source states into messages, aggregate them, and update target nodes.In most variants, the message is determined by the source node and potentially the edge type.
  • 1. Introduction: Target-node representations are generally excluded from message computation, except that attention models use source–target agreement to assign one weight across all message dimensions.This limits how target information modulates incoming messages.
  • 1. Introduction: Pairing source and target states could enable richer interactions, but common linear message transformations provide only additive interactions.More complex transformations are often impractical because message transformation dominates GNN computation.
  • 1. Introduction: FiLM offers an efficient compromise by using the target-node representation to compute an element-wise affine transformation for graph messages.The approach adapts hypernetwork-style interaction without predicting the weights of a full neural network.
  • 1. Introduction: The paper introduces GNN-FiLM, generalises GAT and GIN to multi-relational graphs, and evaluates methods on classification, regression, and ranking tasks.The experiments cover small and large graphs and use a unified framework with consistent hyperparameter optimisation.

2. Model

GNNs update node representations by aggregating transformed neighbor information, with existing variants differing in how edge types, attention, recurrence, and nonlinear transformations are handled. GNN-FiLM conditions message transformations on target-node representations, enabling feature-wise modulation while retaining a tractable structure.

  • Graph Neural Networks: GNN layers repeatedly update node representations using information from neighboring nodes, either by unrolling one update function or stacking layers with distinct parameters.The computation begins from initial node representations and produces successive representations h^(1), h^(2), …, h^(T).
  • Existing GNN variants: Existing message-passing variants differ in their update mechanisms, including recurrent units in GGNN, nonlinearities in RGCN, attention weights in R-GAT, and edge-specific MLPs in R-GIN.These formulations extend graph learning to typed edges and use edge-type-dependent parameters or functions.
  • Existing GNN variants: Across these models, messages are primarily determined by the source representation, while the target representation is updated, treated as an incoming message, or used only to weight relevance.This limitation motivates models that use target-node information directly in message computation.
  • Graph hypernetworks: GNN-MLP0 applies a message-passing function to the concatenation of source and target states, differing from R-GCN in how each message is formed.It instantiates the edge-specific MLP with a single linear layer.
  • Graph hypernetworks: A direct dynamic transformation conditioned on target representations would require producing a D × D matrix from D inputs, yielding O(D^3) parameters with a simple linear implementation.Chunking node representations into C parts of dimension K = D/C is proposed as a mitigation.
  • Graph hypernetworks: GNN-FiLM uses the target-node representation to condition a linear message transformation, focusing on separate representation chunks and applying feature-wise modulation.When each chunk has dimension 1, the method coincides with element-wise affine modulation; this provides richer source-target interaction than additive formulations.

3. Evaluation

The evaluation compares re-implemented GNN architectures across PPI, QM9, and VarMisuse, using task-specific training and validation procedures. GNN-FiLM performs competitively overall, while baseline rankings and reported performance differ across tasks.

  • Evaluation tasks: The study evaluates GNNs on PPI node classification, QM9 molecular-property regression, and VarMisuse program-variable selection.These tasks differ in graph structure, scale, prediction target, and required propagation depth.
  • Evaluation setup: Baseline methods were re-implemented in a unified framework, with implementations, task code, reproducibility scripts, and searched hyperparameters released.The baselines include GGNN, R-GCN, R-GAT, R-GIN, GNN-MLP0, and GNN-MLP1.
  • Evaluation setup: GNN-FiLM was compared using early stopping and repeated training runs with different random initialisations, while hyperparameters were selected through grid or staged searches.PPI models were trained ten times after selection; QM9 models were trained five times per property.
  • PPI: On PPI, all re-implemented models exceeded the published GAT result, and GNN-FiLM improved slightly over the four literature baselines while converging in about 150 epochs versus 400–700.The reported improvements were associated with edge-type generalisation, self-loops, dropout, and larger node representations.
  • QM9: On QM9, GNN-FiLM outperformed the standard baselines on all tasks and the GNN-MLP variants on most tasks.The results are reported as average error rates and standard deviations across separately trained property models.
  • VarMisuse: On VarMisuse, R-GCN performed best and substantially outperformed GGNN, contrary to earlier findings; complex models appeared to overfit, while model-specific solvable-example rates were similar.At least one trained R-GCN correctly predicted 2.2% of SEENPROJTEST examples exclusively by architecture, compared with 2.7% for R-GIN.

4. Discussion & Conclusions

The experiments challenge reported differences among GNN baselines and show that simple GNN-MLP models often outperform established techniques. GNN-FiLM performs competitively, while reproducible evaluation with extensive hyperparameter search reveals important baseline effects.

  • GNN-FiLM and GNN-MLP outperform established baseline models in experiments, with GNN-MLP doing so without a substantial runtime penalty.
  • Extensive experiments indicate that several literature results may reflect insufficient hyperparameter search and missing comparisons with obvious baselines.
  • On PPI, GATs show no advantage over GGNNs or R-GCNs, contrary to findings by Veličković et al. (2018).
  • On VarMisuse, R-GCNs substantially outperform GGNNs, contradicting findings by Allamanis et al. (2018).
  • Across all three tasks, GNN-MLP models outperform better-published techniques such as GGNNs, R-GCNs, and GATs.
  • Independent reproducibility efforts and comparisons that include obvious baselines have substantial value for evaluating GNN results.

A.1. PPI

The PPI experiments use a full grid search over representation size, propagation depth, and input dropout. These settings define the main model-capacity and regularization choices explored for the task.

  • A full grid search evaluates all combinations of hidden size, graph layers, and graph-layer input dropout keep probability.
  • Hidden size is searched over 192, 256, and 320 dimensions for per-node representations.
  • Graph depth is searched over 2, 3, 4, and 5 propagation steps or layers.
  • Graph-layer input dropout keep probability is searched over 0.8, 0.9, and 1.0.

A.2. QM9

The QM9 experiments sample 500 hyperparameter configurations across optimization, architecture, normalization, activation, and residual-design choices. The reported setup also includes R-GAT attention and GGNN cell alternatives, with results summarized in the PPI ablation table.

  • For all models, 500 configurations are considered by uniformly sampling hyperparameter settings from specified options.
  • The search varies hidden size across 64, 96, and 128 and graph depth across 4, 6, and 8 propagation steps.
  • The configurations vary dropout keep probability, layer normalization, dense-layer spacing, residual-connection spacing, and graph activation function.
  • Optimization choices include learning rate 0.0005 or 0.001 and RMSProp or Adam, while GGNN uses an RNN, GRU, or LSTM cell.
  • R-GAT configurations use 4, 8, or 16 attention heads.
  • Table 4 reports (R-)GAT ablation results on the PPI task.

A.3. VarMisuse

The VarMisuse experiments use a full grid search over representation size, propagation depth, dropout, recurrent cell, and attention-head settings. This search targets deeper propagation and architecture-specific choices for the task.

  • A full grid search evaluates all combinations of hidden size, graph layers, and graph-layer input dropout keep probability.
  • Hidden size is searched over 64, 96, and 128 dimensions for per-node representations.
  • Graph depth is searched over 6, 8, and 10 propagation steps or layers.
  • Graph-layer input dropout keep probability is searched over 0.8, 0.9, and 1.0.
  • GGNN configurations use GRU or LSTM cells, while R-GAT configurations use 4 or 8 attention heads.

B. R-GAT Ablations on PPI

The ablations show that changes to the R-GAT formulation, especially edge-type handling, substantially affect PPI performance. Additional gains came from dropout and larger node representations.

  • The R-GAT reimplementation uses input and output linear layers that differ from the original GAT implementation.It projects input features to the hidden-layer size and applies a linear layer to final node representations to obtain logits.
  • The biggest performance jump came from switching from the original GAT formulation to the R-GAT formulation.
  • Distinguishing dataset edges from self-loop edges appears especially important for PPI performance.The self-loops preserve state at each node while receiving a distinct edge type.

C. Training Curves

Training curves compare validation performance across models for PPI and VarMisuse. On PPI, GNN-FiLM converges very quickly, while VarMisuse curves report validation accuracy across random-seed runs.

  • GNN-FiLM shows very fast convergence on the PPI validation-loss curves.Its convergence is visible across runs using 10 random seeds.
  • GNN-MLP0 converges equally fast in training steps but is slower in wall-clock time because each epoch takes longer.
  • The VarMisuse curves track validation accuracy during training across runs using 5 random seeds.
Loading 1906.12192v5…