Source-linked AI summary
GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph
Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit Singh, Guangzhong Sun, Xing Xie
TL;DR
Textual-graph representation learning must combine node text with neighborhood information, but cascaded models encode node text independently before graph aggregation. GraphFormers nests GNNs alongside transformer layers and uses progressive training to fuse these signals iteratively. Across three large-scale datasets, it outperforms SOTA cascaded baselines with comparable running efficiency.
Problem
Cascaded Transformers-GNN models independently encode node text before graph aggregation, limiting information exchange between linked nodes during text encoding.
Method
GraphFormers nests layerwise GNN components alongside transformer layers and progressively trains on manipulated data followed by original data.
Results
GraphFormers consistently outperforms SOTA cascaded Transformers-GNN baselines on DBLP, Wiki, and Product with comparable running efficiency.
Takeaways & Limitations
The results support iterative fusion of text encoding and graph aggregation for higher-quality textual-graph representations.
Takeaways & Limitations
Full GraphFormers encoding requires related neighborhood nodes to be encoded from scratch, incurring repetitive computation costs.
Abstract
from arXiv · showhide
The representation learning on textual graph is to generate low-dimensional embeddings for the nodes based on the individual textual features and the neighbourhood information. Recent breakthroughs on pretrained language models and graph neural networks push forward the development of corresponding techniques. The existing works mainly rely on the cascaded model architecture: the textual features of nodes are independently encoded by language models at first; the textual embeddings are aggregated by graph neural networks afterwards. However, the above architecture is limited due to the independent modeling of textual features. In this work, we propose GraphFormers, where layerwise GNN components are nested alongside the transformer blocks of language models. With the proposed architecture, the text encoding and the graph aggregation are fused into an iterative workflow, {making} each node's semantic accurately comprehended from the global perspective. In addition, a {progressive} learning strategy is introduced, where the model is successively trained on manipulated data and original data to reinforce its capability of integrating information on graph. Extensive evaluations are conducted on three large-scale benchmark datasets, where GraphFormers outperform the SOTA baselines with comparable running efficiency.
1 Introduction
Textual-graph methods combine language models for node text with GNNs for neighborhood information, but cascaded processing prevents cross-node context during text encoding. GraphFormers nests GNNs within transformer layers and adds progressive training, achieving stronger benchmark performance with comparable efficiency.
- Textual-graph representation learning generates low-dimensional node embeddings from textual features and neighborhood information.
- Cascaded Transformers-GNN models independently encode node text before rear-mounted GNNs aggregate the resulting embeddings.
- Because cascaded encoding lacks node information exchange, linked-node context cannot mutually enhance semantic interpretation during text modeling.The paper illustrates this with “transformers,” whose intended meaning becomes clearer from a neighboring machine-translation node.
- GraphFormers nests layerwise GNN components alongside transformer layers, iteratively exchanging neighborhood information before producing increasingly informative node representations.The architecture fuses text encoding and graph aggregation rather than performing them as two consecutive stages.
- Progressive training first uses manipulated data and then original data to strengthen GraphFormers’ integration of graph information.
- On DBLP, Wiki, and Product, GraphFormers outperform cascaded Transformers-GNN baselines with comparable running efficiency.Representation quality is measured by link prediction accuracy.
2 Related Work
Related work develops textual-graph representation by combining pretrained language models with graph neural networks. The dominant cascaded architecture independently encodes node features and then aggregates them through GNNs.
- Textual graph representation is studied across natural language processing, information retrieval, and graph learning.
- Pretrained language models capture textual semantics, while graph neural networks aggregate neighborhood information for node representations.
- The cascaded architecture independently encodes node features before using GNNs to generate final representations.This paradigm is associated with GraphSage and widely adopted by subsequent textual-graph methods.
3 GraphFormers
GraphFormers learns node embeddings from textual features and neighbourhood information by nesting layerwise GNN components alongside Transformer layers. It iteratively exchanges graph information and text representations, adds progressive training, and uses unidirectional aggregation to reduce redundant computation.
- GraphFormers: GraphFormers represents each node using its textual feature and neighbourhood information for relationship-aware node embeddings.The training objective uses embeddings to predict whether pairs of nodes are connected.
- GNN-nested Transformers: Graph-augmented token embeddings are processed by Transformer layers, whose final center-node embedding becomes the representation.The workflow initializes token embeddings, performs graph aggregation, concatenates graph messages with token embeddings, and applies Transformer encoding.
- GNN-nested Transformers: Graph aggregation and Transformer-based text encoding are performed iteratively across multiple nested layers.Node-level embeddings are processed by a layerwise GNN, dispatched back to token sequences, and then encoded by the Transformer.
- GNN-nested Transformers: The GNN aggregates [CLS]-based node embeddings with multi-head attention and uses position bias to distinguish center and neighbour relationships.The position bias differentiates center-to-center, center-to-neighbour, and neighbour-to-neighbour interactions.
- Encoding Complexity: GraphFormers has per-layer encoding complexity O(M^2 + MP^2) for M nodes with P tokens per node.The graph aggregation costs O(M^2), while text encoding costs O(MP^2).
- Model Simplification: Unidirectional Graph Aggregation: Unidirectional graph aggregation lets only the center node reference its neighbourhood while neighbour encodings remain independent and reusable.Caching these intermediate neighbour representations avoids repetitive encoding and preserves similar performance empirically.
- Model Training: Two-Stage Progressive Learning: The model is trained progressively, first on polluted nodes to encourage graph-information use and then on original data to fit the target distribution.The two stages minimize separate objectives until convergence.
4 Experimental Studies
Experiments on three million-scale textual graph datasets evaluate GraphFormers for link prediction against matched baselines. GraphFormers consistently achieve stronger representation quality, benefit from neighbourhood information and progressive learning, and retain comparable efficiency.
- Data and Settings: Experiments use DBLP, Wiki and Product textual graphs, evaluating link prediction from node texts and sampled neighbourhoods.DBLP uses paper titles, Wiki uses entity-introduction sentences, and Product connects products appearing in common browsing sessions.
- Overall Evaluation: GraphFormers consistently outperform all baselines, including cascaded Transformers-GNN methods, with 2.9%, 4.8% and 6.5% relative improvements across the datasets.The comparison uses matched-capacity text encoders and graph aggregators for GraphFormers and cascaded baselines.
- Overall Evaluation: Adding neighbourhood information improves prediction over the PLM-only baseline, and GraphFormers maintain advantages over PLM+Max as neighbour size increases.On DBLP, increasing the number of neighbours enlarges the advantage initially, although marginal gains eventually diminish.
- Overall Evaluation: Pretrained language model methods significantly outperform simplified text-encoder baselines, showing that text-encoder capacity is crucial for textual graph representation.The comparison includes GraphFormers, cascaded Transformers-GNN baselines, PLM-only, TNVE and IFTN.
- Overall Evaluation: Representation quality depends on graph type and aggregator: non-GAT aggregators generally perform better on DBLP and Wiki, where neighbour semantics differ by graph.Product and DBLP are described as homogeneous, whereas Wiki is heterogeneous; neighbour information may therefore contribute differently to link prediction.
5 Conclusion
GraphFormers integrates GNNs alongside each pretrained-language-model transformer layer, then adds progressive training and unidirectional aggregation to improve textual graph representation efficiently.
- GraphFormers nests GNNs alongside transformer layers so textual semantics and graph information are integrated throughout representation learning.
- Two-stage progressive training strengthens GraphFormers’ representation quality by training on manipulated data before original data.
- Unidirectional graph aggregation removes unnecessary computation cost while retaining the proposed representation-learning framework.
- On three large-scale textual graph datasets, GraphFormers outperform cascaded Transformer-GNN methods with comparable running efficiency and scalability.
A.1 Masking Strategy
GraphFormers uses span masking to mask text spans iteratively until a target masking ratio is reached, following settings from prior work.
- GraphFormers samples and masks text spans at each iteration until the masked-token ratio reaches its threshold.
- Span lengths follow a geometric distribution with p = 0.2 and are clipped at lmax = 10.
- The masking strategy masks 15% of input tokens, replacing them with [MASK], random tokens, or unchanged tokens.
A.2 GraphFormers’ Workflow
GraphFormers repeatedly extracts node representations, exchanges them through a GNN, augments token embeddings, and processes them with transformer attention before returning the center-node representation.
- The workflow extracts [CLS] token embeddings as node-level representations at each step.
- The GNN component processes node-level embeddings with a learnable bias vector and multi-head attention.
- GNN-processed node embeddings are concatenated with original token-level embeddings to form graph-augmented token representations.
- The transformer processes graph-augmented embeddings using asymmetric multi-head attention in the relevant layers.
- The center node’s node-level embedding is returned as the graph representation.
B Training Details
GraphFormers’ implementation initializes token embeddings, applies an ordinary first transformer layer, then iterates GNN processing and asymmetric transformer updates to produce output embeddings.
- Training uses up to 100 epochs, dataset-specific learning rates, early stopping on P@1, and Adam optimization.
- The workflow takes initial token-level embeddings as input and returns output embeddings hx.
- The implementation tracks batch size, graph node count, sequence length, hidden dimension, nested-layer count, and a learnable node bias.
- The first transformer layer uses original multi-head attention before the nested GNN-transformer iterations.
- Each subsequent iteration applies multi-head attention to node representations, concatenates them with token embeddings, and uses asymmetric transformer attention.