PaperlayerView exact revision on arXiv

Source-linked AI summary

Be Confident! Towards Trustworthy Graph Neural Networks via Confidence Calibration

Xiao Wang, Hongrui Liu, Chuan Shi, Cheng Yang

arXiv:2109.14285v3cs.LGcs.AI

TL;DR

Although GNNs achieve high accuracy, their confidence is often under-confident, raising a trustworthiness problem for interpreting predictions. The paper introduces topology-aware CaGCN calibration and calibrated self-training, with experiments showing improved calibration and accuracy while preserving classification accuracy; its explanation of under-confidence and extension beyond node classification remain limited.

  • Problem

    GNNs can achieve high accuracy without producing confidence estimates that reliably reflect prediction correctness, and this trustworthiness gap matters for safety-critical use.

  • Method

    CaGCN learns topology-aware, nonlinear, node-specific calibration transformations from GNN logits while preserving class order, and CaGCN-st uses calibrated confidence for pseudo-label selection.

  • Results

    GNNs are generally under-confident, while CaGCN is statistically significantly better than uncalibrated models and other baselines on calibration results.

  • Takeaways & Limitations

    Calibrating GNN confidence can make predictions more trustworthy and provide higher-confidence pseudo labels for self-training without sacrificing the original classification accuracy.

  • Takeaways & Limitations

    The work provides limited explanation for why GNNs are under-confident and leaves extension of CaGCN to other graph tasks for future study.

Abstract

from arXiv · show

Despite Graph Neural Networks (GNNs) have achieved remarkable accuracy, whether the results are trustworthy is still unexplored. Previous studies suggest that many modern neural networks are over-confident on the predictions, however, surprisingly, we discover that GNNs are primarily in the opposite direction, i.e., GNNs are under-confident. Therefore, the confidence calibration for GNNs is highly desired. In this paper, we propose a novel trustworthy GNN model by designing a topology-aware post-hoc calibration function. Specifically, we first verify that the confidence distribution in a graph has homophily property, and this finding inspires us to design a calibration GNN model (CaGCN) to learn the calibration function. CaGCN is able to obtain a unique transformation from logits of GNNs to the calibrated confidence for each node, meanwhile, such transformation is able to preserve the order between classes, satisfying the accuracy-preserving property. Moreover, we apply the calibration GNN to self-training framework, showing that more trustworthy pseudo labels can be obtained with the calibrated confidence and further improve the performance. Extensive experiments demonstrate the effectiveness of our proposed model in terms of both calibration and accuracy.

1 Introduction

GNNs achieve strong predictive accuracy, but trustworthy use requires confidence to reflect correctness. The paper finds that GNNs are typically under-confident and proposes topology-aware calibration methods to improve reliability while preserving accuracy.

  • 1 Introduction: Trustworthy prediction requires confidence to reflect the likelihood that the predicted class is correct, especially in safety-critical applications.High-confidence predictions may be acted on automatically, whereas lower-confidence cases can be deferred to safer strategies.
  • 1 Introduction: GNNs are usually under-confident, unlike many modern neural networks that are often over-confident, leaving accurate predictions concentrated at low confidence.This motivates calibrating GNN predictions so they can better support trustworthy and safety-critical use.
  • 1 Introduction: Experiments evaluate GCN and GAT confidence against accuracy and show that GNNs are far from well-calibrated.The study frames confidence calibration as necessary for making GNN predictions more trustworthy.
  • 1 Introduction: The proposed CaGCN learns topology-aware, nonlinear calibration transformations that assign node-specific temperatures while preserving the original class ordering.Using graph topology allows adjacent nodes to receive related confidence adjustments without changing the original classification accuracy.
  • 1 Introduction: CaGCN-st first calibrates GNN confidence and then uses high-confidence predictions as pseudo labels in self-training.The method is designed to exploit accurate predictions that ordinary confidence thresholds may overlook.

2 Notation and Preliminary Study

The paper defines calibration by comparing predicted confidence with correctness and studies representative GNNs using reliability diagrams and confidence distributions. Across datasets, GNNs commonly achieve higher accuracy than their reported confidence suggests, indicating under-confidence and unreliable confidence estimates.

  • 2 Notation and Preliminary Study: Perfect calibration requires each node’s confidence to equal the true probability that its prediction is correct.The setup considers semi-supervised node classification on an undirected attributed graph with adjacency matrix A and node features X.
  • 2 Notation and Preliminary Study: Reliability diagrams compare confidence bins with average accuracy, using deviation from the diagonal to represent miscalibration.The study uses equal-sized confidence bins and plots confidence on the x-axis against average accuracy on the y-axis.
  • 2 Notation and Preliminary Study: Across four datasets and multiple label rates, most bins have higher average accuracy than average confidence, showing that GCNs and GATs are generally under-confident.The experiments use Cora, Citeseer, Pubmed, and CoraFull with 20, 40, or 60 labeled nodes per class.
  • 2 Notation and Preliminary Study: Confidence distributions show many correct predictions in the low-confidence range, indicating that current GNN confidence estimates are unreliable.Figure 2 distinguishes confidence distributions for correct and incorrect test-node predictions.

3 Confidence Calibration on GCNs

CaGCN calibrates GNN confidence by propagating logits over graph topology, learning node-specific temperatures while preserving class order and classification accuracy.

  • 3.1 CaGCN: GCNs as Calibration Function: Confidence homophily motivates topology-aware calibration because neighboring nodes in a well-calibrated graph should have similar confidence.The method compares total variation before and after calibration to assess confidence smoothness across adjacent nodes.
  • 3.1 CaGCN: GCNs as Calibration Function: CaGCN propagates confidence through graph topology, using a calibration GCN to learn node-specific temperatures for post-hoc scaling.The calibration function is trained from classification-GCN logits and optimized on held-out validation data.
  • 3.1 CaGCN: GCNs as Calibration Function: Unlike temperature scaling, CaGCN uses graph topology and can learn a non-linear calibration function rather than applying one shared linear transformation.Temperature scaling uses the same scalar for every node, whereas CaGCN derives a distinct temperature from the graph.
  • 3.2 The Accuracy-Preserving Property: The improved CaGCN preserves accuracy because positive node-specific temperature scaling leaves the ordering of class logits unchanged.The construction applies the same transformation to every logit dimension for a node, maintaining the original class order.
  • 3.2 The Accuracy-Preserving Property: For finite logits, a unique positive temperature can produce any confidence in (1/K,1), enabling perfect calibration in principle.The result follows from Proposition 2 and the bounded-logit assumption described for GCN outputs.
  • 3.3 Calibration Objective: CaGCN optimizes validation-set NLL together with a regularizer that raises confidence for correct predictions and lowers it for incorrect ones.The regularizer adjusts the gap between the largest and second-largest prediction probabilities.

4 Self-training with Confidence Calibration

CaGCN-st calibrates GCN confidence before selecting pseudo labels for self-training, addressing high-accuracy predictions that otherwise fall in low-confidence ranges.

  • 4 Self-training with Confidence Calibration: Table 2 evaluates calibration with ECE across models, citation networks, and label rates, comparing uncalibrated models with calibrated methods.The table caption specifies ECE with M=20 and reports variability and paired t-test significance markers.
  • 4 Self-training with Confidence Calibration: Uncalibrated self-training can omit many accurate predictions because GNNs distribute substantial high-accuracy predictions in low-confidence intervals.Calibrated confidence is intended to make these predictions available for pseudo-label selection.
  • 4 Self-training with Confidence Calibration: CaGCN-st first calibrates GCN predictions, then uses calibrated confidence to select unlabeled nodes and their pseudo labels for self-training.The pipeline trains a classification GCN, calibrates its logits with CaGCN, and uses the resulting confidence for pseudo-label selection.

5 Experiments

Experiments show that CaGCN improves confidence calibration over uncalibrated GNNs and competing post-hoc methods, while CaGCN-st achieves the strongest self-training accuracy across evaluated datasets and label rates.

  • 5.1 Confidence Calibration Evaluation: CaGCN is statistically significantly better than uncalibrated models and other baselines on ECE.Uncalibrated GCN and GAT models generally have the highest ECE, indicating poor calibration.
  • 5.1 Confidence Calibration Evaluation: CaGCN remains well-calibrated after calibration across the appendix reliability-diagram comparisons.
  • 5.2 Classification Evaluation of Self-Training: CaGCN-st consistently outperforms all baselines on every dataset and label rate at the 0.05 significance level.Table 3 reports node-classification accuracy for GCN and its self-training variants.
  • 5.2 Classification Evaluation of Self-Training: Confidence-calibrated self-training methods perform better because calibration shifts more correct predictions into the high-confidence range while leaving incorrect predictions basically unchanged.
  • 5.2 Classification Evaluation of Self-Training: CaGCN-st consistently outperforms GCN-st, indicating that calibrated confidence produces more correct pseudo labels.The comparison is reported in the self-training ablation study.

6 Related Work

Related work covers graph neural network architectures and established confidence-calibration methods developed primarily for computer vision and natural language processing.

  • Graph Neural Networks: GCN and GAT learn graph representations through graph convolution and attention-based aggregation, respectively.The related work also situates these models among spectral, localized-filter, and neighborhood-aggregation approaches.
  • Confidence Calibration: Confidence calibration research includes post-hoc methods such as Platt scaling, temperature scaling, matrix scaling, and vector scaling.These methods transform model outputs to improve the correspondence between confidence and correctness.

7 Conclusion

The paper concludes that GNN predictions are under-confident and presents CaGCN and CaGCN-st to improve calibration and self-training performance, while identifying limitations in explanation and task scope.

  • 7 Conclusion: CaGCN addresses GNN under-confidence with topology-aware calibration that preserves classification accuracy, while CaGCN-st uses calibrated confidence for pseudo-label generation.
  • 7 Conclusion: The proposed models are reported to be effective in both calibration and accuracy.
  • 7 Conclusion: Extending CaGCN beyond node classification remains future work because the homophily property of edges requires further study.
  • 7 Conclusion: Trustworthy GNN predictions are identified as important for applications including disease prediction, traffic-state prediction, and autonomous-driving object detection.
  • 7 Conclusion: The work provides only a limited explanation for the under-confidence problem.

A Evaluation Metrics

The evaluation uses calibration metrics that compare predicted confidence with correctness, including ECE and Brier Score, on citation-network datasets and a fixed experimental hardware setup.

  • A.1 Expected Calibration Error (ECE): Expected calibration error measures the expected difference between confidence and prediction accuracy.Because the true correctness probability is unknown, the paper uses a bin-based approximation.
  • A.1 Expected Calibration Error (ECE): The approximate ECE averages absolute accuracy-confidence differences across equally spaced confidence bins.Each bin contributes according to its number of predictions.
  • Evaluation Metrics: Brier Score evaluates probabilistic prediction accuracy and is lowest when predicted probability equals the true probability of correctness.
  • Experimental Setup: Evaluation uses Cora, Citeseer, Pubmed, and CoraFull citation networks, with 500 validation nodes, 1,000 test nodes, and 20, 40, or 60 labeled nodes per class.

B.2 Additional Experimental Details for Calibration

The calibration experiments use specified official implementations, optimization settings, and baseline code, with dataset statistics reported separately in Table 5.

  • B.2 Additional Experimental Details for Calibration: Temperature scaling uses learning rate 0.01 and a maximum of 50 iterations, while matrix scaling uses the same learning rate and 400 iterations.Matrix scaling also includes an off-diagonal regularization term to reduce overfitting.
  • B.2 Additional Experimental Details for Calibration: The TS-st method uses learning rate 0.001, with 50 iterations on Cora, Citeseer, and Pubmed and 25 on CoraFull.Additional experimental settings are provided in Table 9.
  • B.2 Additional Experimental Details for Calibration: The study implements the four self-training baselines, co-training, union, and intersection, in PyTorch using their official implementation, while GCN and GAT baseline code is identified by its MIT-licensed repositories.The paper states that related experimental details and code are included in the supplemental material and README.
  • B.2 Additional Experimental Details for Calibration: The paper states that acquired code complies with provider licenses and contains no personally identifiable information or offensive content.The listed baseline repositories include implementations for GCN and GAT.

C.1 Additional Results for Calibration

Additional calibration results show that CaGCN generally improves calibration metrics, reliability, and confidence distributions across models, datasets, and label rates.

  • Metric-Based Calibration: CaGCN is generally better than other baselines on NLL and BS at the reported 0.05 and 0.01 significance levels.The comparisons are reported in Tables 6 and 7 while excluding matrix scaling because it does not preserve accuracy.
  • Reliability Diagrams: Reliability diagrams show well-calibrated predictions after calibration across datasets, models, networks, and label rates.Figures 8 and 9 compare predictions before and after calibration.
  • Confidence Distributions: Calibration shifts many correct predictions into higher-confidence ranges while contrasting them with incorrect predictions.Figures 10 and 11 visualize confidence distributions before and after calibration.

C.2 Additional Results for Self-Training

Additional self-training experiments evaluate GAT accuracy, threshold selection, training trends, and the parameters used by CaGCN-st and TS-st.

  • GAT Self-Training: Our method achieves the best node-classification results among GAT and its self-training variants.The comparison is reported in Table 8.
  • Threshold Study: Both excessively high and low confidence thresholds harm performance because they respectively exclude correct predictions or admit less-confident nodes.Figures 5 and 6 show accuracy trends for different label rates.
  • Parameter Settings: Table 9 summarizes parameters for CaGCN-st and TS-st, including calibration weight decay, self-training epochs, stages, and threshold.These parameters define the reported self-training configurations.
  • Training Trends: The experiments report test-accuracy and NLL trends over epochs for GCN on three datasets.Figure 7 presents the changing trends used in the experimental analysis.

D Why GCNs are poorly calibrated

The analysis attributes poor GCN calibration to a mismatch between optimizing classification accuracy and modeling probabilities, producing under-confident predictions.

  • Accuracy–NLL Mismatch: On Cora, optimizing NLL reduced accuracy from 81.5% to 79.5%, while the highest accuracy occurred before NLL reached its minimum.The same analysis was conducted on Cora, Citeseer, and CoraFull with label rate L/C = 20.
  • Accuracy–NLL Mismatch: The study explains GNN miscalibration as under-fitting NLL without under-fitting accuracy, contrasting probability modeling with classification performance.This analysis is motivated by prior observations about neural-network overfitting to NLL.
  • Visual Evidence: Reliability diagrams and confidence-distribution figures compare GCN and GAT calibration before and after calibration across models and label rates.The figures provide visual analyses of the calibration behavior discussed in this section.
Loading 2109.14285v3…