PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta-Aggregator: Learning to Aggregate for 1-bit Graph Neural Networks

Yongcheng Jing, Yiding Yang, Xinchao Wang, Mingli Song, Dacheng Tao

arXiv:2109.12872v1cs.CVcs.LG

TL;DR

Cumbersome GNN architectures can be difficult to use in resource-constrained settings, and binarized GNNs can struggle to distinguish graph topologies. The paper introduces adaptive meta aggregators for 1-bit GNNs, reporting results superior to the state of the art across domains and models.

  • Problem

    Cumbersome GNN architectures challenge resource-constrained deployment, while binarized GNNs can lack the expressive power to distinguish graph topologies.

    Source: §1 · p. 1, §3 · p. 4

  • Method

    The framework binarizes GNN features and parameters, then uses GNA to select an aggregator or ANA to generate a hybrid based on binarized graph features.

    Source: §1 · pp. 1–2

  • Results

    Experiments across domains report that the proposed meta aggregators achieve results superior to the state of the art on both 1-bit and full-precision GNNs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The meta aggregation schemes are presented as usable in compact 1-bit GNNs and extendable to full-precision models.

    Source: §6 · p. 8

    GNA depends on its candidate aggregators, while ANA's mathematical form limits which aggregators it can approximate.

    Source: §5.3 · p. 8

Abstract

from arXiv · show

In this paper, we study a novel meta aggregation scheme towards binarizing graph neural networks (GNNs). We begin by developing a vanilla 1-bit GNN framework that binarizes both the GNN parameters and the graph features. Despite the lightweight architecture, we observed that this vanilla framework suffered from insufficient discriminative power in distinguishing graph topologies, leading to a dramatic drop in performance. This discovery motivates us to devise meta aggregators to improve the expressive power of vanilla binarized GNNs, of which the aggregation schemes can be adaptively changed in a learnable manner based on the binarized features. Towards this end, we propose two dedicated forms of meta neighborhood aggregators, an exclusive meta aggregator termed as Greedy Gumbel Neighborhood Aggregator (GNA), and a diffused meta aggregator termed as Adaptable Hybrid Neighborhood Aggregator (ANA). GNA learns to exclusively pick one single optimal aggregator from a pool of candidates, while ANA learns a hybrid aggregation behavior to simultaneously retain the benefits of several individual aggregators. Furthermore, the proposed meta aggregators may readily serve as a generic plugin module into existing full-precision GNNs. Experiments across various domains demonstrate that the proposed method yields results superior to the state of the art.

1. Introduction

The paper targets compact GNNs for resource-constrained settings, where existing applications can depend on cumbersome architectures. It proposes adaptive meta aggregation for 1-bit GNNs and reports results superior to the state of the art across benchmarks and in full-precision models.

1. Introduction

  • Existing GNN-based applications can require cumbersome architectures, while resource-constrained settings motivate compact models that avoid expensive floating-point operations.SuperGlue is cited as requiring 12M parameters for state-of-the-art performance; distillation can reduce parameters but still relies on floating-point operations.Source: §1 · p. 1
  • Binarizing 32-bit features and parameters makes a lightweight GNN, but limited expressive power can make distinct graph topologies indistinguishable during aggregation.With binarized features, a conventional mean aggregator can produce identical results for graphs with different topologies.Source: §1 · pp. 1–2, Figure 2 · p. 2
  • The proposed meta aggregators adapt aggregation to graph features: GNA selects one candidate aggregator, while ANA can generate a hybrid behavior that retains strengths of several aggregators.GNA uses learnable selection; ANA can approximate a single aggregator or dynamically generate a hybrid aggregation behavior.Source: §1 · p. 2
  • GNA and ANA can also be added as portable modules to full-precision GNNs to enhance expressive capability.The reported full-precision results replace GAT's predefined aggregator with GNA or ANA and introduce few additional parameters.Source: §1 · p. 2
  • The proposed meta aggregators achieve results superior to the state of the art on 1-bit GNNs and full-precision models across benchmarks.The framework is described as lightweight and competitive, with resource-constrained applications such as edge computing as a target.Source: §1 · p. 2

2. Related Work

Related work covers GNN architectures and applications, alongside model compression and binarization. The paper distinguishes its approach as a GNN-specific binarization framework and a study of learning aggregators from aggregators.

Graph Neural Networks

  • Prior work applies GNNs across domains including semantic segmentation, object detection, pose estimation, interaction detection, and visual SLAM.GraphSAGE addresses scalability on large graphs through neighborhood sampling and aggregation, while graph attention networks introduce an attention mechanism.Source: §2 · p. 2
  • Despite broad GNN applications, the cited related work identifies compression of cumbersome GNNs for resource-constrained deployment as an underexplored need.The passage names mobile-terminal deployment as an example setting.Source: §2 · p. 2

Generalized Aggregation Functions

  • Unlike generalized aggregation-function studies [24] [34], this work studies learning aggregators from other aggregators, described as functions of functions.The paper characterizes this as a first dedicated study of functionals of GNNs.Source: §2 · p. 2

Network Binarization

  • CNN binarization includes direct quantization and optimization-based methods, while this work focuses on direct binarization tailored to GNNs for lightweight models.The related work says optimization-based methods use scaling factors to improve representation ability but reduce efficiency.Source: §2 · p. 3

3. Vanilla Binary GNN and Pre-analysis

The section develops a vanilla GNN that binarizes weights and features, then identifies reduced topology discrimination as a key limitation. Combining aggregators improves the preliminary ZINC results, but the repeated aggregation motivates more efficient adaptive aggregation.

Vanilla 1-bit GNN Models

  • Binarizing 32-bit input features and learnable weights to 1 bit yields a compact GNN trained with the straight-through estimator.The estimator addresses the sign function’s near-zero gradients during backpropagation.Source: §3 · p. 3
  • With compact node features and weights, binarized GCN relies on 1-bit XNOR and bitcount operations for graph processing.The section presents this as an efficient, lightweight model suited to edge computing.Source: §3 · pp. 3–4, Figure 3 · p. 4

Challenges and Possible Solutions

  • 0.669 ± 0.070 MAE for vanilla binarized GCN versus 0.407 ± 0.018 for full-precision GCN on ZINC, averaged over 25 runs.The section also reports 0.599 ± 0.017 MAE for summing five standard aggregators and 0.614 ± 0.045 for concatenating aggregator outputs; both mixed approaches require multiple aggregation passes per layer.Source: §3 · p. 4
  • The smaller discrete feature space makes 1-bit outputs less distinguishable, hampering neighborhood aggregation’s ability to discriminate graph topologies.In the illustrated binarized layer, both max and mean fail to distinguish two topologies; the section notes the challenge is greater when aggregation precedes transformation, because inputs are limited to 1 or −1.Source: §3 · p. 4, Figure 4 · p. 4
  • Combining aggregation schemes can distinguish some topologies that individual schemes miss, but summing or concatenating their outputs requires multiple aggregation passes per layer.The reported preliminary ZINC MAEs are 0.599 ± 0.017 for summing five standard aggregators and 0.614 ± 0.045 for concatenation.Source: §3 · p. 4

4. Meta Neighborhood Aggregation

The section introduces meta aggregation as a learnable way to adapt neighborhood aggregation efficiently. GNA selects one candidate aggregator per node and layer, while ANA generates a hybrid; training uses graph encodings to guide aggregation and the straight-through estimator to optimize the 1-bit GNN.

4.1. Overview

  • Meta aggregation adaptively and efficiently adjusts neighborhood aggregation, with GNA selecting one candidate and ANA learning a formulation that can combine several aggregators.The framework is presented as a response to the computational burden of mixed multi-aggregators.Source: §4.1 · pp. 4–5, Figure 5 · p. 5

4.2. Greedy Gumbel Aggregator

  • GNA uses encoded graph features and stochastic Gumbel sampling to select a single candidate aggregator for each node and layer.Its one-hot decision is conditioned on the node and layer; the Gumbel-softmax relaxation supports gradient propagation through the discrete choice.Source: §4.2 · pp. 5–6

4.3. Adaptable Hybrid Aggregator

  • ANA’s learned aggregation can approximate max or mean behavior depending on the graph-encoder output, and can be adapted to approximate other aggregators or combine their benefits.Higher encoder values yield behavior similar to max, while smaller values yield an effect similar to mean; the text also describes approximating min and variance.Source: §4.3 · p. 6

4.4. Training Strategy

  • At each layer, a lightweight 1-bit graph auto-encoder extracts information for the meta aggregator, and the resulting model is optimized with the straight-through estimator.The meta module either selects an aggregator or produces a diffused one that combines aggregation behaviors.Source: §4.4 · pp. 6–7

5. Experiments

The experiments evaluate the meta aggregators across graph regression, node classification, and 3D object recognition, then discuss their strengths and weaknesses.

5.1. Experimental Settings

  • The evaluation covers ZINC graph regression, PPI node classification, and ModelNet40 3D object recognition, using GCN, GAT, and DGCNN architectures.ZINC results average 25 independent runs; comparison methods keep the first and last GNN layers full-precision while binarizing the others.Source: §5 · p. 7, §5.1 · p. 7

5.2. Results

  • On ZINC graph regression, GNA and ANA show favorable train and test MAE while maintaining compact model size, with paired t-tests reported against vanilla 1-bit GNNs.Results average 25 independent runs with 25 seeds; the paper says the p-values statistically validate the method's effectiveness.Source: §5.2 · p. 7
  • On full-precision GAT, replacing the predefined aggregator with GNA or ANA outperforms state-of-the-art approaches by a large margin while adding few parameters.The comparison includes approaches cited as [3] [10] [32] [21].Source: §5.2 · p. 7
  • For PPI node classification, GNA and ANA perform on par with 32-bit models, use a lighter architecture, and outperform the fixed-aggregator vanilla 1-bit model.Table 3 reports micro-averaged F1 scores.Source: Table 3 · p. 7, §5.2 · p. 8
  • More than 10% gains in both overall accuracy and mean class accuracy are reported for 1-bit GNNs with meta aggregation on ModelNet40.The authors also report that the learned feature structure is closer to that of the full-precision GNN.Source: §5.2 · p. 8, Table 4 · p. 8

5.3. Discussions

  • GNA depends on the quality of its candidate aggregator pool, whereas ANA's mathematical form limits which aggregators it can approximate.The discussion identifies these constraints as potential bottlenecks to future improvement for each variant.Source: §5.3 · p. 8

6. Conclusions

The paper proposes adaptive aggregation schemes to improve the topological discrimination of compact 1-bit GNNs, and reports that they yield results on par with full-precision models across domains. It also identifies compact, lightweight visual transformers as a direction for future work.

6. Conclusions

  • The paper proposes learnable aggregation schemes for compact 1-bit GNNs, adapting by selecting one aggregator or generating a hybrid that retains strengths of several aggregators.The schemes are also described as extendable to full-precision GNNs.Source: §6 · p. 8
  • Across domains, 1-bit GNNs with the proposed meta aggregators yield results on par with cumbersome full-precision models.Source: §6 · p. 8
  • Future work will seek to generalize the proposed aggregator to compact, lightweight visual transformers.Source: §6 · p. 8

Related papers

Loading 2109.12872v1…