PaperlayerView exact revision on arXiv

Source-linked AI summary

Bayesian graph convolutional neural networks via tempered MCMC

Rohitash Chandra, Ayush Bhagat, Manavendra Maharana, Pavel N. Krivitsky

arXiv:2104.08438v2cs.LG

TL;DR

Bayesian inference offers principled parameter uncertainty quantification, but MCMC demands have limited its use in deep learning. This paper combines graph convolution with parallel tempered MCMC and Langevin-gradient proposals; maximum accuracy is generally on par with canonical GCNNs, while mean accuracy is around 45% lower on CiteSeer.

  • Problem

    MCMC's computational requirements have limited Bayesian inference for deep learning, despite its role in uncertainty quantification for model parameters.

    Source: §I · p. 2

  • Method

    Bayes-GCNN uses spectral graph convolution and parallel tempered MCMC with Langevin-gradient proposals to sample model parameters.

    Source: §I · p. 2, A. MODEL AND LIKELIHOOD FUNCTION · p. 3

  • Results

    Bayes-GCNN's maximum accuracy is generally on par with canonical GCNNs across benchmark problems, while its mean accuracy is around 45% lower for CiteSeer.

    Source: §VI · p. 10

  • Takeaways & Limitations

    Bayes-GCNN provides an alternative training approach with probabilistic weights and biases for principled uncertainty quantification.

    Source: §VI · p. 10

    The adaptive Langevin proposal may mildly violate MCMC's Markov assumption because its step length depends on previous gradients and the Q-ratio may be approximate.

    Source: §V · p. 8

Abstract

from arXiv · show

Deep learning models, such as convolutional neural networks, have long been applied to image and multi-media tasks, particularly those with structured data. More recently, there has been more attention to unstructured data that can be represented via graphs. These types of data are often found in health and medicine, social networks, and research data repositories. Graph convolutional neural networks have recently gained attention in the field of deep learning that takes advantage of graph-based data representation with automatic feature extraction via convolutions. Given the popularity of these methods in a wide range of applications, robust uncertainty quantification is vital. This remains a challenge for large models and unstructured datasets. Bayesian inference provides a principled approach to uncertainty quantification of model parameters for deep learning models. Although Bayesian inference has been used extensively elsewhere, its application to deep learning remains limited due to the computational requirements of the Markov Chain Monte Carlo (MCMC) methods. Recent advances in parallel computing and advanced proposal schemes in MCMC sampling methods has opened the path for Bayesian deep learning. In this paper, we present Bayesian graph convolutional neural networks that employ tempered MCMC sampling with Langevin-gradient proposal distribution implemented via parallel computing. Our results show that the proposed method can provide accuracy similar to advanced optimisers while providing uncertainty quantification for key benchmark problems.

I. INTRODUCTION

Graph neural networks represent data through graph structure, enabling node-wise propagation for non-Euclidean data. This paper applies Bayesian inference to graph convolutional networks using tempered MCMC, addressing computational challenges in Bayesian deep learning.

A. GRAPH NEURAL NETWORKS

  • Graph neural networks propagate information using graph structure, making node input order irrelevant and supporting unstructured, hierarchical data.Source: §I · p. 1

I. INTRODUCTION

  • MCMC's computational demands for large datasets and parameter sets have limited Bayesian deep learning, while parallel computing and gradient-based proposals offer a route forward.Source: §I · p. 2
  • Bayesian graph convolutional networks use tempered MCMC with parallel computing and Langevin-gradient proposals.The parallel framework exchanges tempered MCMC replica states between processes.Source: §I · p. 2

A. GRAPH NEURAL NETWORKS

  • A graph comprises vertices representing data elements and edges representing their relationships; nodes send feature-vector messages to immediate neighbours.Source: A. GRAPH NEURAL NETWORKS · p. 2
  • GNNs learn node state embeddings for downstream tasks, with propagation variants differing in how they aggregate neighbourhood information.Source: A. GRAPH NEURAL NETWORKS · p. 2
  • Graph CNNs include spatial and spectral convolution approaches, which respectively use nearby-node information or process the graph through its Laplacian spectrum.Spectral methods can capture complex patterns but are more computationally expensive; spatial methods can process node batches for speed and efficiency.Source: A. GRAPH NEURAL NETWORKS · p. 2

B. BAYESIAN DEEP LEARNING

Bayesian deep learning has been limited by canonical MCMC's demands on models with many parameters and complex architectures. The literature described here includes variational methods and newer gradient-based and tempered MCMC approaches.

B. BAYESIAN DEEP LEARNING

  • Canonical MCMC limitations have constrained Bayesian deep learning for models with many parameters and complex architectures.Source: B. BAYESIAN DEEP LEARNING · p. 3
  • Variational inference has been applied across CNNs, RNNs, LSTMs, and graph models, but the authors report finding no variational graph-CNN work.Source: B. BAYESIAN DEEP LEARNING · p. 3
  • Recent Bayesian neural-network MCMC work includes gradient-based Langevin and Hamiltonian proposals, GPU computation, tempered sampling, surrogate assistance, and transfer learning.Source: B. BAYESIAN DEEP LEARNING · p. 3

III. METHODOLOGY

Bayes-GCNN combines spectral graph convolution with tempered MCMC and Langevin-gradient proposals to classify graph nodes. Its likelihood compares training labels with model outputs, while a parameter prior supports posterior sampling.

III. METHODOLOGY

  • Bayes-GCNN uses spectral convolution and tempered MCMC with Langevin-gradient proposals for graph-node classification.Source: §III · p. 3

A. MODEL AND LIKELIHOOD FUNCTION

B. LANGEVIN-GRADIENT PROPOSAL DISTRIBUTION

Bayes-GCNN samples weights and biases with MCMC, using Langevin-gradient proposals, parallel computing, and tempering to address large parameter spaces and multimodal posteriors.

B. LANGEVIN-GRADIENT PROPOSAL DISTRIBUTION

  • The sampling scheme combines Langevin-gradient proposals, parallel computing with inter-process communication, and tempering to address large models and multimodal posteriors.The authors describe these components as means to optimize sampling when GCNNs may have hundreds of thousands of parameters.Source: B. LANGEVIN-GRADIENT PROPOSAL… · p. 4
  • Langevin-gradient proposals combine Gaussian noise with a single-epoch gradient step, using either SGD or Adam to guide sampling.The paper names the SGD-based proposal LG and the Adam-based proposal adapt-LG.Source: B. LANGEVIN-GRADIENT PROPOSAL… · p. 4
  • The proposal is drawn from a multivariate normal distribution, with its mean shifted by a scaled gradient of the log posterior.The gradient scale is fixed for SGD-based proposals and adaptive for Adam-based proposals; the covariance uses a scaled identity matrix.Source: B. LANGEVIN-GRADIENT PROPOSAL… · p. 4
  • Because Langevin-gradient proposals are asymmetric across chain steps, the Metropolis–Hastings acceptance calculation requires the conditional proposal-density ratio.The proposal density is denoted Q(θ⋆_n|θ_n).Source: B. LANGEVIN-GRADIENT PROPOSAL… · p. 4
  • Tempered MCMC runs an ensemble of replicas at different temperatures, flattening higher-temperature posteriors and periodically exchanging neighboring replica states.The temperature ladder's maximum is user-defined; higher temperatures help replicas escape local minima.Source: B. LANGEVIN-GRADIENT PROPOSAL… · p. 4

C. BAYES-GCNN FRAMEWORK

The Bayes-GCNN framework runs tempered MCMC replicas in parallel and exchanges their states through inter-process communication. It configures a graph-CNN architecture for the graph-based classification task and supports user-set sampling parameters.

C. BAYES-GCNN FRAMEWORK

  • Bayes-GCNN executes tempered MCMC replicas in parallel, exchanging their states at regular intervals through inter-process communication.The framework also specifies graph-based input data and a GCNN architecture for classification.Source: C. BAYES-GCNN FRAMEWORK · p. 4
  • Sampling begins with tempered replicas, whose proposals are evaluated using log-likelihood and Metropolis–Hastings acceptance before the algorithm switches to canonical MCMC.The switch is controlled by Rswitch; tempered samples are burn-in and are excluded from the true posterior.Source: C. BAYES-GCNN FRAMEWORK · p. 4
  • After replica swaps, the algorithm removes burn-in and combines the remaining samples with the corresponding replicas’ posterior distributions for analysis.The manager process decides whether neighboring replicas swap, and inter-process communication exchanges their chain positions.Source: C. BAYES-GCNN FRAMEWORK · p. 5
  • Users set the replica count, maximum temperature, swap interval, sample limit, and Langevin-gradient rate alongside the GCNN configuration.The Langevin-gradient rate controls how often proposals use Langevin gradients rather than random walks.Source: C. BAYES-GCNN FRAMEWORK · p. 5

IV. EXPERIMENTS AND RESULTS

The experiments assess Bayes-GCNN's computational efficiency, proposal-distribution choices, and prediction accuracy on established benchmark datasets.

IV. EXPERIMENTS AND RESULTS

  • The evaluation examines computational efficiency of tempered MCMC sampling, effects of proposal distributions, and prediction accuracy on established benchmark datasets.The passage states evaluation dimensions but reports no outcomes or values.Source: §IV · p. 5
  • One evaluation dimension is the computational efficiency of tempered MCMC sampling.Source: §IV · p. 5
  • The experiments also assess how different proposal distributions affect the method.Source: §IV · p. 5

A. DATASET DESCRIPTION

The experiments use Cora, CiteSeer, and PubMed citation networks for graph-node classification, with a training split of 20 labels per class and 1,000 test nodes.

A. DATASET DESCRIPTION

  • Cora, CiteSeer, and PubMed are citation-network benchmarks, with publications as nodes, citations as edges, and word-presence vectors describing publications.Source: A. DATASET DESCRIPTION · p. 5
  • Each dataset uses 20 training labels per class and 1,000 test nodes, a split the authors say is used in prior literature [29].The authors note that the number of training instances is relatively low.Source: A. DATASET DESCRIPTION · p. 5
  • Cora contains 2,708 papers and 1,433 dictionary words, while CiteSeer contains 3,327 papers and 3,703 dictionary words.The passage identifies seven Cora classes and six CiteSeer classes.Source: A. DATASET DESCRIPTION · p. 5
  • PubMed contains 19,717 diabetes papers, 44,338 citation links, and a 500-word dictionary, with nodes classified into three diabetes categories.Source: A. DATASET DESCRIPTION · pp. 5–6

B. EXPERIMENT DESIGN

The experiments evaluate Bayes-GCNN’s tuning choices and benchmark performance, finding near-comparable accuracy to literature methods alongside prediction uncertainty quantification. They also examine sampling behavior and convergence across citation-network datasets.

B. EXPERIMENT DESIGN

  • Experiments used a maximum temperature of 2, swap interval of 2 samples, up to 48,000 samples, and switched to canonical MCMC after 60% of samples.Random-walk proposal noise had standard deviation 0.005; the adaptive and canonical Langevin learning rates were 0.01 and 0.1, respectively.Source: B. EXPERIMENT DESIGN · p. 6
  • The evaluation used Cora, CiteSeer, and PubMed, with Bayes-GCNN topology reported by dataset and implementations built with PyTorch, PyTorch Geometric, and Python multiprocessing.The supplied table captions identify dataset class/training/test counts and model parameter totals, but provide no values.Source: B. EXPERIMENT DESIGN · p. 6, Table 1 · p. 7, Table 2 · p. 7

C. RESULTS

  • For Cora, adapt-LG rate was positively associated with sample acceptance, while higher rates did not appear to improve performance and computing gradients was expensive.A rate of 0 produced much lower training and test accuracy, indicating random-walk proposals alone could not train Bayes-GCNNs.Source: C. RESULTS · pp. 6–7
  • Increasing Cora replicas from 2 to 4 reduced computational time without much change in classification performance; neighboring-replica swap percentage rose gradually through 8 replicas.The authors attribute the time and efficiency gains to parallel replica processes, one per processing core.Source: C. RESULTS · p. 7, Table 4 · p. 7
  • Bayes-GCNN (adapt-LG) offers almost comparable classification accuracy to literature results, with slightly lower mean performance but comprehensive prediction uncertainty quantification.Using LG rather than adapt-LG significantly deteriorates performance.Source: C. RESULTS · p. 7
  • Selected weight posteriors were unimodal for Cora and CiteSeer, while PubMed showed one unimodal and one bimodal posterior; all datasets’ trace plots showed high chain correlation.Only two weights among thousands were plotted, and the authors say explaining dataset differences requires further analysis.Source: C. RESULTS · pp. 7–8
  • PubMed showed higher test-accuracy variance over time than Cora and CiteSeer as log-likelihood increased alongside training and test accuracy.The passage suggests this variance could relate to the application problem, dataset size, or Bayes-GCNN architecture.Source: C. RESULTS · p. 8
  • After a 60% burn-in, selected weight IDs across replica chains had Gelman–Reubin diagnostic values close to 1, which the authors interpret as convergence.The diagnostic used posterior distributions from all replica chains.Source: C. RESULTS · p. 8

V. DISCUSSION

Bayes-GCNN offers performance comparable to traditional methods while providing principled uncertainty quantification. The discussion highlights a potential Markov-assumption caveat for adapt-LG and unresolved challenges in diagnosing convergence for large models.

V. DISCUSSION

  • Bayesian inference via tempered MCMC provides GCNN performance comparable to traditional methods while offering principled uncertainty quantification.The paper presents this as a proof-of-concept; Table 5 compares Bayesian GCNN results with established methods.Source: §V · p. 8, Table 5 · p. 9
  • Adapt-LG's dependence on past gradients may mildly violate MCMC's Markov assumption, making the Q-ratio approximate and requiring careful checks.The authors identify this as a subject for future work.Source: §V · p. 8
  • Convergence diagnostics remain challenging for deep models: Gelman–Rubin effectiveness with many parameters is not well studied, and autocorrelation and effective sample size need evaluation.The paper notes deep learning models can feature tens of thousands of parameters.Source: §V · p. 9
  • The authors identify Bayesian inference for other graph architectures, including graph-LSTM and graph attention networks, as future work.They also propose exploring other datasets and Hamiltonian MCMC.Source: §V · p. 10

VI. CONCLUSION

Bayes-GCNN uses tempered MCMC with adaptive Langevin-gradient proposals and parallel computing. Its maximum accuracy is generally on par with canonical GCNN across benchmark problems, while representing weights and biases probabilistically.

VI. CONCLUSION

  • The maximum accuracy of Bayes-GCNN is generally on par with canonical GCNN across benchmark problems, although its mean accuracy is around 45% lower for CiteSeer.The authors characterize this as an alternative training approach that quantifies uncertainty in model parameters.Source: §VI · p. 10
  • Bayes-GCNN uses tempered MCMC sampling via parallel computing with adaptive Langevin-gradient proposals.The method represents weights and biases probabilistically, avoiding repetitive experiments for that representation.Source: §VI · p. 10
  • Probabilistic weights and biases provide uncertainty quantification that canonical optimizers do not offer on their own.The paper identifies this capability as useful for certain problems.Source: §VI · p. 10

CODE AND DATA

The authors report that a Python-based Bayes-GCNN implementation and its data are available.

CODE AND DATA

Related papers

Loading 2104.08438v2…