PaperlayerView exact revision on arXiv

Source-linked AI summary

Label-free Node Classification on Graphs with Large Language Models (LLMS)

Zhikai Chen, Haitao Mao, Hongzhi Wen, Haoyu Han, Wei Jin, Haiyang Zhang, Hui Liu, Jiliang Tang

arXiv:2310.04668v3cs.LGcs.AIcs.CL

TL;DR

GNNs need abundant high-quality labels, while LLMs can annotate without them but are costly and do not naturally capture graph structure. LLM-GNN combines LLM-generated annotations with GNN prediction, using difficulty-aware selection and confidence-aware filtering. It outperforms several label-free baselines and reaches 74.9% accuracy on OGBN-PRODUCTS for under $1.

  • Problem

    GNNs require abundant high-quality labels, while LLMs can predict without labeled data but have difficulty capturing graph structure and incur high inference costs.

    Source: §1 · pp. 1–2

  • Method

    LLM-GNN uses LLMs to generate confidence-aware annotations for difficulty-aware selected nodes, optionally filters them, and trains a GNN for prediction.

    Source: §1 · p. 2, §3 · p. 3, §3.1 · p. 3

  • Results

    LLM-GNN significantly outperforms SES, TAG-Z, and BART-large-MNLI; on OGBN-PRODUCTS it achieves 74.9% accuracy for under $1, comparable to manually annotating 400 randomly selected nodes.

    Source: §1 · p. 2, §4.3 · p. 8

  • Takeaways & Limitations

    On massive-scale OGBN-PRODUCTS, LLM-generated annotations can support GNN predictions without human annotations at a cost under $1.

    Source: §1 · p. 2, §6 · p. 9

Abstract

from arXiv · show

In recent years, there have been remarkable advancements in node classification achieved by Graph Neural Networks (GNNs). However, they necessitate abundant high-quality labels to ensure promising performance. In contrast, Large Language Models (LLMs) exhibit impressive zero-shot proficiency on text-attributed graphs. Yet, they face challenges in efficiently processing structural data and suffer from high inference costs. In light of these observations, this work introduces a label-free node classification on graphs with LLMs pipeline, LLM-GNN. It amalgamates the strengths of both GNNs and LLMs while mitigating their limitations. Specifically, LLMs are leveraged to annotate a small portion of nodes and then GNNs are trained on LLMs' annotations to make predictions for the remaining large portion of nodes. The implementation of LLM-GNN faces a unique challenge: how can we actively select nodes for LLMs to annotate and consequently enhance the GNN training? How can we leverage LLMs to obtain annotations of high quality, representativeness, and diversity, thereby enhancing GNN performance with less cost? To tackle this challenge, we develop an annotation quality heuristic and leverage the confidence scores derived from LLMs to advanced node selection. Comprehensive experimental results validate the effectiveness of LLM-GNN. In particular, LLM-GNN can achieve an accuracy of 74.9% on a vast-scale dataset \products with a cost less than 1 dollar.

1 INTRODUCTION

GNNs can use graph structure effectively but require abundant high-quality labels, while LLMs can label without them but are costly and do not naturally capture graph structure. LLM-GNN combines LLM annotations with GNN prediction, using difficulty-aware selection and confidence-based filtering to improve annotation quality.

1 INTRODUCTION

  • GNNs rely on abundant high-quality labels, but human annotation of complex, million-node graphs can be difficult, time-consuming, and resource-intensive.LLMs can provide predictions without labeled data, but their high inference cost and limited ability to capture graph structure constrain their use on large datasets.Source: §1 · pp. 1–2
  • GNNs and LLMs have complementary strengths: GNNs use graph structure efficiently with adequate labels, while LLMs can perform without high-quality labels but are costly.The paper frames its challenge as obtaining annotations that balance quality with diversity and representativeness.Source: §1 · p. 2
  • LLM-GNN uses LLMs to annotate selected nodes, then trains a GNN on those annotations to predict labels for other nodes.Selection accounts for LLM annotation difficulty; confidence-aware annotations and post-filtering refine annotation quality.Source: §1 · p. 2
  • 74.9% accuracy on OGBN-PRODUCTS costs under $1 and is comparable to manually annotating 400 randomly selected nodes.The annotations are generated by LLMs, without human annotations.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries describe text-attributed graphs and two GNN node-classification pipelines, distinguishing fixed labeled training sets from active selection. Both assume ground-truth labels, although graph annotation can be costly, error-prone, and difficult to diversify.

2 PRELIMINARIES

  • A text-attributed graph represents nodes with raw text attributes and sentence embeddings, alongside an adjacency matrix encoding graph connectivity.The study focuses on this graph type and notes potential extension to other graph types.Source: §2 · p. 2
  • The default GNN pipeline trains on a fixed set of ground-truth-labeled nodes and predicts labels for the remaining nodes.Source: §2 · p. 3
  • Graph active learning selects nodes from the pool to maximize the performance of GNNs trained on their labels.Source: §2 · p. 3
  • Both pipelines assume ground-truth labels, yet graph annotation can be expensive and error-prone, while improving diversity can make quality harder to ensure.The passage describes human graph annotation as potentially biased toward nodes sharing characteristics within a small subgraph.Source: §2 · p. 3

3 METHOD

LLM-GNN combines LLM annotation with GNN prediction through active node selection and confidence-based quality control. Its method balances annotation reliability with graph-selection criteria and label diversity.

3 METHOD

  • LLM-GNN uses LLMs to annotate selected nodes and trains GNNs on those annotations for prediction.Difficulty-aware selection and optional confidence-based filtering aim to provide training nodes with high annotation quality, representativeness, and diversity.Source: §3 · p. 3, §3.1 · pp. 3–4

3.1 AN OVERVIEW OF LLM-GNN

  • The overview combines difficulty-aware selection, confidence-aware annotation, optional post-filtering, and GNN training and prediction.Confidence scores help identify annotation quality and remove lower-confidence nodes before GNN training.Source: §3.1 · p. 3, Figure 1 · p. 3

3.2 DIFFICULTY-AWARE ACTIVE NODE SELECTION

  • Closer-to-center nodes tend to receive more accurate LLM annotations, motivating C-Density as a proxy for annotation reliability.Difficulty-aware selection aggregates the C-Density rank with traditional active-selection scores; the candidate set is kept small to control cost.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 CONFIDENCE-AWARE ANNOTATIONS

  • Zero-shot hybrid prompting is adopted because its confidence can indicate annotation quality among the tested strategies.The study also considers vanilla, reasoning-based, TopK, consistency-based, and few-shot prompts; reasoning-based prompts increase query time and cost and can produce format errors.Source: §3.3 · p. 5

3.4 POST-FILTERING

  • Post-filtering ranks annotations by confidence, C-Density, and change of entropy to balance annotation quality with label diversity.A small change-of-entropy score signals that removing a node could adversely affect diversity; the score is recomputed as nodes are removed.Source: §3.4 · p. 6

3.5 GNN TRAINING AND PREDICTION

  • The pipeline trains GNNs, primarily GCN, on selected annotations and can weight cross-entropy by LLM confidence to address label noise.The trained GNN is then used to predict labels.Source: §3.1 · p. 4, §3.5 · p. 6

4 EXPERIMENT

The experiments evaluate LLM-GNN’s settings, selection and filtering strategies, performance and cost against other label-free methods, budget sensitivity, and the training behavior of LLM annotations.

4.1 EXPERIMENTAL SETTINGS

  • Experiments use six text-attributed graph datasets, GPT-3.5-turbo-0613 annotations, a budget of 20 times the class count, and GCN models.Accuracy is evaluated on the remaining unlabeled nodes without a validation set; experiments use three seeds and fixed training durations of 30 epochs for smaller datasets and 50 for larger ones.Source: §4.1 · pp. 6–7

4.2 RQ1. IMPACT OF DIFFERENT ACTIVE SELECTION STRATEGIES

  • Post-filtering often improves traditional selectors; FeatProp-based methods perform consistently well, while pure C-Density can cause class imbalance and poor GNN performance.For PUBMED, the selected annotations all belonged to one class; weighted loss and post-filtering are more effective than choosing a suitable clustering count on large datasets.Source: §4.2 · p. 8

4.3 (RQ2.) COMPARISON WITH OTHER LABEL-FREE NODE CLASSIFICATION METHODS

  • LLM-GNN significantly outperforms SES, TAG-Z, and BART-large-MNLI, while LLMs-as-Predictors perform better at 2,124× its OGBN-PRODUCTS cost.Table 3 reports costs in dollars, with LLM inference time proportional to expense.Source: Table 3 · p. 8, §4.3 · p. 8

4.5 (RQ4.) CHARACTERISTICS OF LLMS’ ANNOTATIONS

  • LLM annotations show different training dynamics from synthetic noisy labels, with substantially less overfitting.Synthetic labels are generated by uniformly flipping a fraction of ground-truth labels matched to the assumed quality of LLM annotations.Source: §4.5 · p. 9

4.4 (RQ3.) HOW DO DIFFERENT BUDGETS AFFECT THE PERFORMANCE OF OUR PIPELINES?

  • As the CORA annotation budget rises from 35 to 1,120, performance tends to improve gradually, but growth is more limited than with ground-truth labels.The results indicate a performance–cost trade-off; Figure 4 compares top-performing Table 2 methods with random selection.Source: §4.4 · p. 8, Figure 4 · p. 9

5 RELATED WORKS

The related work describes graph active learning’s selection criteria and distinguishes prior uses of LLMs for graphs from this paper’s annotator role.

5 RELATED WORKS

  • Graph active learning selects nodes under a limited query budget, commonly targeting informativeness and diversity through community partitions, influence scores, or direct performance objectives.Prior approaches include partition-based selection (Ma et al., 2022), influence-based selection (Zhang et al., 2021c;b;a), and reinforcement learning (Gao et al., 2018; Hu et al., 2020a; Zhang et al., 2022).Source: §5 · p. 9
  • Prior graph-focused LLM work uses models as predictors or enhancers, whereas this paper uses them as annotators to train an efficient model without ground-truth labels.The cited predictor work includes Chen et al. (2023), Wang et al. (2023a), and Ye et al. (2023); enhancer work includes He et al. (2023a).Source: §5 · p. 9

6 CONCLUSION

The conclusion presents LLM-GNN as a response to the neglected annotation process in node classification and reports experiments validating the pipeline.

6 CONCLUSION

  • LLM-GNN combines confidence-aware LLM annotations, difficulty-aware selection, and confidence-based post-filtering to improve annotation quality.The conclusion frames the pipeline as addressing the annotation process in node classification.Source: §6 · p. 9
  • Comprehensive experiments validate the effectiveness of the proposed pipeline.Source: §6 · p. 9

A.1 GRAPH ACTIVE LEARNING METHODS

The appendix reviews graph active-learning selectors, zero-shot classification baselines, and related work on LLM annotation and zero-shot node classification.

A.1 GRAPH ACTIVE LEARNING METHODS

  • Graph active-learning baselines select nodes using feature-space density, graph partitions, aggregated features, degree, PageRank, or influence-based scores.The cited methods include Density-based selection, GraphPart, and Degree-based selection (Ma et al., 2022), FeatProp (Wu et al., 2019), AGE (Cai et al., 2017), and RIM (Zhang et al., 2021b).Source: §A.1 · p. 14

A.2 ZERO-SHOT CLASSIFICATION BASELINE MODELS

  • Zero-shot baselines include semantic-space matching in SES, prompt- and topology-based logits in TAG-Z, and MNLI-fine-tuned BART-Large-MNLI.SES and TAG-Z are attributed to Li & Hooi (2023); BART-Large-MNLI to Lewis et al. (2019).Source: §A.2 · p. 14

B MORE RELATED WORKS

  • Prior work studies LLMs as data annotators, while zero-shot node classification transfers knowledge to unseen categories; this paper’s label-free setting begins with no labels.Prior zero-shot approaches use label descriptions, label consistency, or contrastive learning.Source: B MORE RELATED WORKS · pp. 14–15

C DATASETS

The study uses six commonly adopted node-classification datasets, with text-attributed versions selected for LLM annotation. It also describes prompt formats and strategies used to obtain annotations.

C DATASETS

  • Six datasets are used: CORA, CITESEER, PUBMED, WIKICS, OGBN-ARXIV, and OGBN-PRODUCTS.Text-attributed versions from Chen et al. (2023) are used for all except WIKICS, whose raw attributes come from its dataset repository.Source: C DATASETS · p. 15

D PROMPTS

  • The prompts request a Python dictionary-like output, and a self-correction prompt addresses outputs that do not follow instructions.The self-correction prompt is shown in Table 5.Source: D PROMPTS · p. 15, Table 5 · p. 15
  • The paper illustrates zero-shot consistency and few-shot consistency prompts; few-shot is used only for comparison, not in Section 4.For one-shot prompts, LLMs automatically generate top K predictions and confidence scores following Zhang et al. (2023).Source: D PROMPTS · p. 15, Table 6 · p. 16, Table 7 · p. 16

E COMPLETE RESULTS FOR THE PRELIMINARY STUDY ON EFFECTIVENESS OF CONFIDENCE-AWARE PROMPTS

The appendix points to complete results for the preliminary confidence-aware prompt study, including accuracy and cost across six datasets and confidence-group plots.

E COMPLETE RESULTS FOR THE PRELIMINARY STUDY ON EFFECTIVENESS OF CONFIDENCE-AWARE PROMPTS

  • The complete preliminary-study results on confidence-aware prompts are presented in Table 8, Figure 6, and Figure 7; Figure 6 depicts group accuracies sorted by confidence generated by LLMs.Source: E COMPLETE RESULTS FOR THE PRELIMINARY… · p. 16, Figure 6 · p. 17
  • Table 8 reports annotation accuracy and cost for multiple prompt strategies across six datasets.The caption states that yellow marks the best result and green the second-best; cost is relative to zero-shot prompt token consumption.Source: Table 8 · p. 17
  • The reported datasets are CORA, CITESEER, PUBMED, OGBN-ARXIV, OGBN-PRODUCTS, and WIKICS.The table fragment gives accuracy and cost columns for each dataset.Source: E COMPLETE RESULTS FOR THE PRELIMINARY… · p. 17
  • The provided table fragment distinguishes accuracy from cost for each prompt strategy and dataset.The table caption defines cost by token consumption relative to zero-shot prompts.Source: E COMPLETE RESULTS FOR THE PRELIMINARY… · p. 17, Table 8 · p. 17
  • Figure 6 plots group accuracy after sorting nodes by LLM-generated confidence.Here, k denotes the number of nodes selected in the groups.Source: Figure 6 · p. 17
  • Figure 7 also relates group accuracy to LLM-generated confidence rankings.Its caption uses k for the number of selected nodes in the groups.Source: Figure 6 · p. 17

F.1 LABEL-LEVEL OBSERVATIONS

The appendix examines class-level annotation errors, their relationship to clustering-center distance, confidence-based prompt strategies, model settings, and comparisons with synthetic noisy labels.

F.1 LABEL-LEVEL OBSERVATIONS

  • LLM annotation quality varies substantially by class, and errors tend to concentrate in one or two classes rather than spread uniformly.For WIKICS, accuracy is 100% for class c0 and 31% for c7; the matrices cover WIKICS, CORA, and CITESEER.Source: §F.1 · p. 17, Figure 8 · p. 18
  • WIKICS class c7 tends to flip into c8, but c8 does not flip into c7, unlike the more uniform synthetic-noise pattern.The paper suggests the classes’ similar meanings may explain the direction of some errors; annotation distributions also shift, with WIKICS c8 and CORA c6 becoming majority classes.Source: §F.1 · pp. 17–18, Figure 9 · p. 18, Figure 11 · p. 19

F.2 RELATIONSHIP BETWEEN ANNOTATION QUALITY AND C-DENSITY

  • On PUBMED, the relationship between annotation quality and distance to clustering centers is less evident than on other datasets.The paper offers exceptionally high annotation quality or LLM shortcuts in text attributes as possible explanations; Figure 12 encodes group accuracy and accumulated average accuracy against distance.Source: §F.2 · p. 19, Figure 12 · p. 19

G EFFECTIVENESS OF CONFIDENCE GENERATED BY LLMS

H HYPERPARAMTERS

  • For small datasets, the stated settings are learning rate 0.01, weight decay 5e-4, hidden dimension 64, and dropout 0.5.The small-scale group comprises CORA, CITESEER, PUBMED, and WIKICS.Source: H HYPERPARAMTERS · p. 20
  • For large datasets, the settings are learning rate 0.01, weight decay 5e-4, hidden dimension 256, and dropout 0.5.The large-scale group comprises OGBN-ARXIV and OGBN-PRODUCTS.Source: H HYPERPARAMTERS · p. 20

I COMPARING LLMS ANNOTATIONS TO SYNTHETIC NOISY LABELS

  • Figure 14 distinguishes GNN training and test performance for ground-truth labels, corrected LLM annotations, uncorrected LLM annotations, and synthetic noisy labels.The caption assigns solid lines to test performance and dashed lines to training performance; no outcome comparison is stated in the supplied text.Source: Figure 14 · p. 20

J EXTRA RESULTS FOR THE COMPARATIVE STUDY IN TABLE 2

The appendix reports extra comparisons of confidence-related strategies and gives a theoretical motivation for selecting nodes closer to clustering centers.

J EXTRA RESULTS FOR THE COMPARATIVE STUDY IN TABLE 2

  • Weighted cross-entropy usually improves performance, while applying post-filtering and difficulty-aware selection together usually performs poorly and does not surpass baselines.The gap from the baselines is described as very small; Table 9 is captioned as an ablation of different combinations, and Table 10 covers degree- and pagerank-based selection.Source: J EXTRA RESULTS FOR THE COMPARATIVE… · p. 21, Table 9 · p. 21, Table 10 · p. 21

K THEORETICAL MOTIVATION

  • The theoretical analysis aims to explain why C-Density can help select nodes with high annotation quality.It frames annotation accuracy as minimizing discrepancies between pseudo-labels and ground-truth labels, with the selected node indices entering the objective.Source: K THEORETICAL MOTIVATION · p. 21
  • Under the stated Gaussian assumption, minimizing distance to clustering centers reduces the objective’s distance-related term, motivating preference for closer nodes.The paper says the unknown distribution parameters leave σ_i to minimize; smaller σ_i corresponds to smaller distance to clustering centers and better annotation quality.Source: K THEORETICAL MOTIVATION · p. 22

L DESIGN PHILOSOPHY BEHIND LLMGNN

LLM-GNN’s design rationale is that LLM annotations can be imperfect and structure-aware prompting does not reliably resolve structural reasoning limits. The authors therefore separate text-based annotation from GNN structural learning and treat annotation quality as important to performance and scaling.

L DESIGN PHILOSOPHY BEHIND LLMGNN

  • Less than 70% overall annotation accuracy shows node labeling remains difficult for GPT-3.5 and GPT-4, with GPT-4 offering little accuracy improvement.The comparison uses randomly selected nodes from Cora and CiteSeer.Source: L DESIGN PHILOSOPHY BEHIND LLMGNN · p. 22, Table 11 · p. 23
  • Both LLMs can assign plausible alternate categories, but single-label evaluation accepts only one, exposing dataset-specific annotation bias.The example contrasts a ground-truth “Database” label with both models’ “Information Retrieval” prediction.Source: L DESIGN PHILOSOPHY BEHIND LLMGNN · pp. 22–23, Table 12 · p. 23
  • Structure-aware prompting did not effectively improve annotation quality; LLMs use structural information only in limited ways, unlike GNN message passing.The paper reports a large performance gap when replacing the GNN with an MLP.Source: L DESIGN PHILOSOPHY BEHIND LLMGNN · p. 23, Table 13 · p. 23, Table 14 · p. 24
  • The authors use text-only information for annotation and structural information during GNN training, based on the current effectiveness of this division.They chose a structure-free prompt and GPT-3.5-turbo rather than GPT-4, whose improvement they found very limited.Source: L DESIGN PHILOSOPHY BEHIND LLMGNN · pp. 22–23
  • Designing more effective structure-aware prompts remains future work; agent-based multi-round prompting may improve performance, and the framework supports varied prompt designs.The authors describe new prompt designs as a way to enhance the framework’s effectiveness.Source: L DESIGN PHILOSOPHY BEHIND LLMGNN · p. 23
  • Correcting a portion of incorrect annotations may improve LLM-GNN performance, making annotation quality a potential lever for improving results.The paper examines performance trends after randomly correcting some wrong annotations.Source: L DESIGN PHILOSOPHY BEHIND LLMGNN · p. 24, Table 15 · p. 24
  • At low budgets, the performance difference between high- and low-quality annotations is <20, compared with >20 at high budgets; PubMed performance is better than Cora’s.The authors relate this scaling pattern to annotation quality and suggest better annotations may permit a higher performance upper bound as budgets grow.Source: L DESIGN PHILOSOPHY BEHIND LLMGNN · p. 24, Table 16 · p. 24

Related papers

Loading 2310.04668v3…