PaperlayerView exact revision on arXiv

Source-linked AI summary

Graph Barlow Twins: A self-supervised representation learning framework for graphs

Piotr Bielak, Tomasz Kajdanowicz, Nitesh V. Chawla

arXiv:2106.02466v3cs.LG

TL;DR

Graph self-supervised learning often relies on negative samples that are difficult to define, while negative-sample-free alternatives can use asymmetric architectures. Graph Barlow Twins instead uses a symmetric encoder and cross-correlation loss, achieving performance on par with or better than state-of-the-art methods while converging substantially faster. Its reported scope is graph node classification, with further graph tasks and other data types left for future study.

  • Problem

    Contrastive self-supervised graph methods require negative samples whose appropriate definition depends on the downstream task.

  • Method

    Graph Barlow Twins computes the cross-correlation matrix of two distorted graph views processed by the same symmetric encoder.

  • Results

    Graph Barlow Twins achieves results on par with or better than state-of-the-art self-supervised graph methods across eight real-world datasets and converges an order of magnitude faster.

  • Takeaways & Limitations

    The method reduces computation cost while maintaining decent downstream performance, supporting efficient processing of larger graph datasets.

  • Takeaways & Limitations

    Further studies are needed for link prediction, graph classification, and extensions to data types more specific than graphs.

Abstract

from arXiv · show

The self-supervised learning (SSL) paradigm is an essential exploration area, which tries to eliminate the need for expensive data labeling. Despite the great success of SSL methods in computer vision and natural language processing, most of them employ contrastive learning objectives that require negative samples, which are hard to define. This becomes even more challenging in the case of graphs and is a bottleneck for achieving robust representations. To overcome such limitations, we propose a framework for self-supervised graph representation learning - Graph Barlow Twins, which utilizes a cross-correlation-based loss function instead of negative samples. Moreover, it does not rely on non-symmetric neural network architectures - in contrast to state-of-the-art self-supervised graph representation learning method BGRL. We show that our method achieves as competitive results as the best self-supervised methods and fully supervised ones while requiring fewer hyperparameters and substantially shorter computation time (ca. 30 times faster than BGRL).

1. Introduction

Graph Barlow Twins addresses the difficulty of defining graph negative samples by using a symmetric, negative-sample-free self-supervised framework. It evaluates this framework across transductive and inductive node-classification settings.

  • Motivation: Negative samples are difficult to define for graphs because appropriate choices depend on the downstream task.Possible negatives include non-neighbors or nodes from another graph component.
  • Proposed framework: Graph Barlow Twins uses the cross-correlation matrix of two distorted graph views to optimize node representations.The framework applies a Barlow Twins loss to graph data.
  • Proposed framework: The framework uses a fully symmetric architecture without negative samples or special asymmetry techniques.Both distorted views pass through the same encoder with symmetrically backpropagated gradients.
  • Evaluation: The method is evaluated on five smaller benchmarks, ogb-arxiv, PPI, and ogb-products in transductive and inductive node-classification settings.Experiments use GCN-based and GAT-based encoders.
  • Results: Graph Barlow Twins achieves results analogous to state-of-the-art methods while converging substantially faster.The contribution statement reports comparisons across multiple node-classification settings.

2. Related works

Prior self-supervised graph representation methods largely use contrastive learning and negative sampling. BGRL avoids negative samples but introduces an asymmetric and conceptually complex encoder pipeline.

  • Self-supervised learning: In computer vision, BYOL and SimSiam demonstrate negative-sample-free self-supervised learning using siamese architectures and anti-collapse techniques.BYOL uses a slow-moving average target network, while SimSiam does not require momentum encoders or large batches.
  • Self-supervised graph representation learning: Self-supervised graph methods adapted contrastive approaches such as DGI and GraphCL to learn graph representations.These methods use GNNs and graph-specific objectives.
  • Self-supervised graph representation learning: Contrastive graph methods use negative sampling, whose high complexity and graph-specific interpretation limit the setting.The appropriate negative counterpart can vary with the task.
  • Self-supervised graph representation learning: BGRL removes the need for negative samples through online and target encoders but uses an asymmetric pipeline.Its design also includes techniques such as gradient stopping to prevent trivial solutions.

3. Proposed framework

Graph Barlow Twins generates two augmented views of an attributed graph, encodes both with the same network, and trains that encoder using cross-correlation-based redundancy reduction.

  • Generating graph views via augmentation: The framework begins with an attributed graph and generates two graph views using sampled augmentation parameters.The algorithm takes an attributed graph, an augmentation function, an encoder, epochs, and a learning rate as input.
  • Computing node embeddings: The same encoder processes both augmented views to produce two node-embedding matrices.The encoder is pretrained within the proposed framework.
  • Optimizing encoder weights: The empirical cross-correlation matrix of the two embedding matrices is driven toward the identity matrix by the training loss.Encoder weights are optimized by backpropagating the loss gradient.
  • Generating graph views via augmentation: Edge dropping and node-feature masking are selected as the two graph augmentations.Edge dropping removes edges according to a Bernoulli-sampled mask.

11 end

The framework combines symmetric graph encoding with a Barlow Twins loss that enforces augmentation invariance and reduces redundancy. Its design omits momentum encoders, gradient stopping, and predictor networks.

  • Encoder network for node embeddings: Graph Barlow Twins represents graphs through node and edge structures, while allowing encoders for edges or whole graphs when supported.The experiments use GCN- and GAT-based encoders for node embeddings.
  • Encoder network for node embeddings: Both augmented graph views pass through the same encoder, producing embedding matrices Z(1) and Z(2).The approach eliminates the projector network because its GNN embeddings are low-dimensional.
  • Loss function: The encoder receives normalized embedding matrices before the empirical cross-correlation matrix is computed.Normalization uses a batch-wise mean of zero and standard deviation equal to one.
  • Loss function: The loss drives diagonal cross-correlation entries toward one for augmentation invariance and off-diagonal entries toward zero for decorrelation.The λ parameter trades off invariance and redundancy reduction.
  • Symmetric optimization: Gradients are backpropagated symmetrically without momentum encoders, gradient stopping, or predictor networks.The authors also report no performance gain from investigating the Hilbert-Schmidt Independence Criterion.

4. Experiments

The experiments evaluate Graph Barlow Twins across small, medium, and large graph benchmarks under transductive and inductive settings. Results use frozen encoder embeddings for downstream node classification and compare performance and convergence with BGRL.

  • Datasets: The evaluation covers six real-world graph datasets, including WikiCS, Amazon, Coauthor, ogb-arxiv, ogb-products, and PPI.The suite includes smaller and larger benchmarks, with PPI comprising multiple graphs.
  • Training: Graph Barlow Twins trains graph encoders on two augmented views using shared augmentation parameters, AdamW, cosine annealing, and a linear warmup.Augmentation parameters are selected by grid search, with training configured for 500 epochs in the described procedure.
  • Experimental setting: The experiments use transductive learning for six datasets, while ogb-products and PPI also represent inductive settings.In the transductive setting, the full graph and node features are observed during encoder training, but labels remain hidden.
  • Encoder model: The encoder is a two-layer GCN with batch normalization and PReLU after the first layer, while the final layer has no normalization or activation.The GCN uses symmetrically normalized adjacency with added self-loops; activation is applied after batch normalization.
  • Results: 500-900 epochs: Graph Barlow Twins converges earlier than BGRL, which converges and reports results at 10 000 epochs on the smaller and medium benchmarks.Graph Barlow Twins outperforms BGRL at 1000 epochs and achieves comparable results to state-of-the-art methods.
  • Results: 1.5 pp: on ogb-arxiv, Graph Barlow Twins remains within 1.5 percentage points of BGRL while converging in about 300-400 epochs instead of 10 000.The ogb-arxiv graph contains about 170 thousand nodes and 1.1 million edges.

4.4. Inductive experiments

The framework is evaluated on inductive node-classification settings spanning PPI and the large-scale ogb-products graph. It achieves results comparable to or better than BGRL while supporting batched training and substantially faster computation.

  • 4.4.1. PPI: The PPI experiments use a 3-layer GAT encoder with skip connections and attention heads selected for stronger performance than standard GCN layers.The first two layers use four heads of size 256, while the final layer uses six heads of size 512.
  • 4.4.1. PPI: On PPI, G-BT achieves a Micro-F1 score of 70.49 using a GAT encoder, with results on par with BGRL.Training uses a batch size of 1 graph for 500 epochs, with results averaged over 20 model initializations.
  • 4.4.2. ogb-products: The ogb-products experiment uses inductive node classification with neighbor sampling, batch size 512, and 100 training epochs because the graph does not fit into GPU memory.Training uses only training-set nodes and edges among them.
  • 4.4.2. ogb-products: On ogb-products, G-BT highly outperforms BGRL on both validation and test node-classification sets.The graph contains about 2.5 million nodes and 61 million edges, requiring inductive neighbor sampling and batched training.
  • 4.5. Training time comparison: Across considered models, G-BT takes the least time for a single training iteration in virtually all cases and speeds computation versus BGRL by about 17-42 times.The comparison measures single-epoch duration over 10 training epochs, with evaluation time held constant.
  • 4.4. Inductive experiments: G-BT supports both full-batch and mini-batch training, although batched neighbor sampling produces an expected performance decrease.Batches sample the k-hop neighborhood for a k-layer encoder.

5. Ablation and hyperparameter sensitivity study

The study examines augmentation choices, encoder architectures, training time, batching, and loss weighting. Results favor combining node feature masking with edge dropping and using a two-layer GCN, while symmetric settings simplify tuning and computation.

  • Augmentation hyperparameters: Using the same augmentation hyperparameters for both graph views reduces the grid-search space from 1296 to 36 combinations, without a substantial test-accuracy difference.The shared setting is motivated by the model’s symmetric architecture and lower hyperparameter-search complexity.
  • Training time comparison: G-BT requires the least single-epoch training time in virtually all considered model comparisons.The timing comparison averages durations over 10 training epochs.
  • Batched setting: In batched experiments, G-BT is retrained across batch sizes using neighbor sampling, with an expected decrease in performance compared with full-batch training.The augmentation hyperparameters and epoch counts are reused from the full-batch experiments.
  • Augmentation functions: Using both node feature masking and edge dropping provides the best augmentation results in the ablation study.Node feature masking alone reaches 75.9% accuracy, edge dropping alone reaches 72%, and no augmentation reaches 67%.
  • Encoder architectures: The two-layer GCN achieves the best encoder result, while the one-layer GCN is only 1.1pp worse than the baseline.The three-layer GCN shows a performance drop, possibly related to oversmoothing, and the MLP-based encoder reaches about 68% accuracy.
  • Encoder architectures: Deeper GNN models can have sharply increasing time and space complexity on highly dense graphs.This provides a practical cost alongside the observed performance drop for the three-layer GCN.
  • Loss sensitivity: The loss coefficient λ controls the trade-off between invariance and redundancy reduction, with λ = 1/256 producing the best reported performance.Smaller λ values perform better than larger ones, while λ = 1 makes the two terms equally important and deteriorates performance.
  • Projector network: The method omits the projector network because low-dimensional GNN embeddings already address the dimensionality-reduction role it served in the original Barlow Twins model.Projector dimensionality choices, including no projector, are evaluated experimentally.

6. Conclusions

Graph Barlow Twins uses cross-correlation matrices from distorted graph views to learn symmetric, augmentation-invariant, less redundant representations. Across real-world node-classification tasks, it matches or exceeds state-of-the-art SSL methods while converging faster and reducing computation cost.

  • Graph Barlow Twins computes the embedding cross-correlation matrix from two distorted views and enforces it toward the identity matrix.The diagonal term promotes invariance to augmentations, while the off-diagonal term decorrelates embedding components.
  • The framework is fully symmetric and does not require special techniques to produce nontrivial embedding vectors.
  • On 8 real-world datasets, Graph Barlow Twins achieves results on par with or better than state-of-the-art SSL graph representation methods in transductive and inductive node classification.
  • Graph Barlow Twins reduces computation cost through faster convergence while retaining decent downstream-task performance.The authors connect this efficiency to processing larger graph datasets and tasks such as node classification, link prediction, and graph classification.
  • Further studies are proposed for additional negative-sample-free approaches, graph tasks, and extensions to data types more specific than graphs.

Appendix A. Augmentation hyperparameters

The augmentation setup uses one shared pair of hyperparameters for both graph views and selects them by grid search. Values above 0.5 remove too much graph information, while the ogb-products experiment uses a shorter training schedule.

  • Graph Barlow Twins uses a single pair of augmentation hyperparameters pA and pX for both graph views.The authors state that this shared set is sufficient for their symmetric architecture, so reported values from other works cannot be reused directly.
  • The framework is trained for 500 epochs with a 50-epoch learning-rate warmup in the standard setup.
  • The augmentation search evaluates pA and pX over {0, 0.1, . . . , 0.5}.
  • Values greater than 0.5 remove too much information from the graph.
  • For ogb-products, training uses 10 epochs and a 2-epoch warmup because of the dataset’s large size.The same augmentation hyperparameter values are evaluated, and the best-performing configurations are summarized in Table A.8.

Appendix B. Training setup

Training uses AdamW with cosine-annealed learning rates and linear warmup, while downstream evaluations are performed periodically on frozen embeddings and the best checkpoint is reported.

  • All datasets use AdamW with weight decay 10^-5 and cosine annealing with a linear warmup period.
  • Frozen embeddings are evaluated in downstream tasks after each configured evaluation interval.
  • The reported model is the best-performing one among the scheduled evaluation points.For example, a 1000-epoch run with a 500-epoch interval evaluates at epochs 0, 500, and 1000.

Appendix C. Encoder architecture

The framework uses GCN encoders for small-to-medium datasets, a larger GCN for ogb-arxiv, and a GAT encoder for PPI. Architecture choices include low-dimensional embeddings, skip connections, and attention-head aggregation.

  • Small-to-medium datasets use a 2-layer GCN-based encoder, with input feature dimension k and embedding dimension d.
  • The ogb-arxiv dataset uses a slightly larger 3-layer GCN-based model.
  • Weight standardization and layer normalization did not benefit Graph Barlow Twins, and training became unstable with fluctuations and loss peaks under that setting.
  • The inductive PPI experiment uses a 3-layer GAT encoder because GATs perform better than GCNs on this dataset.
  • The PPI architecture concatenates attention-head outputs in the first two layers, averages them in the last layer, and uses linear skip connections.

Appendix D. Code and reproducibility

The appendix documents a reproducible experimental pipeline implemented with PyTorch-Geometric and DVC, including dataset preprocessing, hyperparameter searches, model training, evaluation, and runtime comparisons.

  • Implementation: PyTorch-Geometric implements all models, while DVC runs the experimental pipeline and supports reproducibility.The full pipeline can be run with dvc repro, or individual stages with dvc repro <stage name>.
  • Pipeline stages: Dataset preprocessing downloads each dataset and, when applicable, generates train/validation/test node splits.
  • Pipeline stages: G-BT training and evaluation are provided for full-batch and batched settings, with separate batched stages for ogbn-products.
  • Comparative experiments: The experiments compare shared versus separate augmentation hyperparameter sets for graph views and measure average training-epoch times across seven methods.The runtime comparison includes DeepWalk, DGI, MVGRL, GRACE, BGRL, and G-BT; augmentation comparisons produce Figure 2.
  • Comparative experiments: The appendix includes stages for BGRL hyperparameter searches and training, raw-feature and DeepWalk evaluations, and configuration and script directories.Hyperparameters are stored in experiments/configs/, while experimental Python scripts are stored in experiments/scripts/.
Loading 2106.02466v3…