Source-linked AI summary
Who Should Teach? Confidence-Aware Dual-Teacher Learning for Few-Shot Node Classification on Text-Attributed Graphs
Hojin Kim, Sujin Yoon, Sungsu Lim, Dongwon Lee, David Yoon Suk Kang
TL;DR
Existing LLM-based TAG methods do not account for node-level variation in supervision reliability, even though GNNs and LLMs provide complementary structural and semantic signals. CoTeach dynamically selects the more reliable teacher using confidence-aware routing, and experiments show improved few-shot performance with lower LLM token usage and favorable cost-effectiveness.
Problem
Existing LLM-based TAG methods generally overlook substantial node-level variation in the reliability of LLM-derived supervision, despite complementary structural signals from graph topology.
Method
CoTeach uses confidence-aware routing between a structure-focused GNN teacher and a semantic LLM teacher, retaining reliable predictions to supervise a student model.
Results
CoTeach achieves the best overall performance across most few-shot settings and reduces LLM token usage by up to 67.6× versus TAPE and 8.3× versus LLMGNN.
Takeaways & Limitations
Adaptive teacher selection provides effective supervision for few-shot node classification while reducing unnecessary LLM utilization and associated monetary costs.
Abstract
from arXiv · showhide
Text-Attributed Graphs (TAGs) integrate graph structures and node-associated textual attributes, and recent studies have increasingly leveraged Large Language Models (LLMs) to improve TAG learning in few-shot settings. However, existing approaches typically utilize LLM-derived information uniformly across all nodes, despite substantial variations in its reliability, while also incurring considerable monetary costs. We argue that the most appropriate source of supervision may differ across nodes, as Graph Neural Networks (GNNs) and LLMs exhibit complementary strengths in exploiting structural and semantic information, respectively. To this end, we propose CoTeach, a Confidence-aware dual-teacher learning framework that dynamically selects the more reliable teacher for each node. Experimental results demonstrate that CoTeach consistently improves few-shot node classification performance while reducing unnecessary LLM utilization and associated monetary costs.
1 Introduction
TAG learning combines graph topology with node-text semantics, but existing methods do not fully exploit language-model knowledge or account for node-level variation in LLM supervision reliability. CoTeach addresses this by selecting between structural and semantic teachers according to confidence in few-shot learning.
- Text-Attributed Graphs jointly encode relational information through graph topology and semantic information through node-associated text.
- Traditional TAG methods combine shallow text representations or pretrained language models with GNNs but cannot fully exploit rich linguistic knowledge from language models.
- Recent LLM-based TAG paradigms use LLMs for annotation, representation enhancement, or data generation, especially in label-scarce and few-shot settings.
- Motivation: LLM-derived supervision can vary substantially in reliability across nodes, while graph topology provides complementary structural supervision through neighboring nodes.
- Proposed Framework: CoTeach uses confidence-aware selection between a GNN teacher for structural knowledge and an LLM teacher for semantic knowledge, retaining reliable supervision while reducing unnecessary LLM inference costs.
- Contributions: The paper evaluates CoTeach comprehensively on four real-world TAGs to assess effectiveness and cost efficiency.
2 CoTeach: Proposed Framework
CoTeach combines a structure-aware GNN teacher, a semantic LLM teacher, and a student model that receives supervision from the more reliable teacher for each node. Confidence thresholds route nodes, filter LLM predictions, and support joint hard-label and soft-distribution training.
- Overview: CoTeach consists of a GNN teacher, an LLM teacher, and a student model for confidence-aware supervision in few-shot TAG learning.
- GNN Teacher: The GNN teacher uses graph topology to produce each node’s pseudo-label and confidence score, separating highly confident nodes from nodes requiring further examination.
- LLM Teacher: Nodes with insufficiently confident structural predictions are passed to the LLM teacher, which uses node text and candidate label descriptions for semantic reasoning.
- LLM Teacher: The LLM teacher retains only predictions whose confidence exceeds a predefined threshold, assigning the retained nodes to V_LLM for student supervision.
- Student Model: The student receives pseudo-labels and class-probability distributions from the teacher assigned to each node, combining hard pseudo-label supervision with soft distribution distillation.
- Student Model: The final objective combines GNN-teacher and LLM-teacher losses, with α controlling their relative contributions.
3 Evaluation
CoTeach is evaluated on four real-world TAGs against GNN, PLM, GNN+PLM, and GNN+LLM baselines using few-shot node classification. Results show that combining confidence-aware GNN and LLM supervision improves accuracy, remains competitive across backbones, and reduces LLM token consumption.
- Experimental Settings: Experiments cover Cora, Citeseer, Pubmed, and WikiCS, with k∈{3, 5, 7, 10} labeled nodes per class and five-run test-accuracy reporting.
- Experimental Settings: CoTeach is compared with 10 methods spanning backbone GNNs, PLMs, GNN+PLM, and GNN+LLM approaches.
- Ablation Study: Combining both teachers consistently achieves the best performance across WikiCS few-shot settings, while either teacher alone outperforms no teacher.
- Accuracy Comparison: CoTeach_S achieves the best overall average rank at 1.6 and exceeds the strongest baseline by 6.9% on Cora and 4.9% on Pubmed when k=3.
- Cost-Effectiveness: CoTeach reduces LLM token usage by up to 67.6× versus TAPE and 8.3× versus LLMGNN, while LLM4NG remains more token-efficient but has a worse average rank.
- Hyperparameter Sensitivity: CoTeach is generally robust to hyperparameter choices, with best performance under moderate confidence thresholds and sufficient seed-node coverage.
4 Conclusions
The paper concludes that CoTeach uses confidence-aware selection between GNN and LLM teachers for few-shot TAG learning. Across four real-world TAGs, it achieves the best overall performance across most settings, with future work extending beyond node-level tasks.
- CoTeach adaptively selects GNN- and LLM-based supervisory signals according to prediction confidence because LLM-supervision reliability varies across nodes.
- Experiments on four real-world TAGs show the best overall performance across most settings.
- Future work will extend CoTeach beyond node-level tasks to edge-level and graph-level learning scenarios.
GenAI Usage Disclosure
The authors disclose selective use of generative AI in the research process.
- Generative AI models, including ChatGPT-3.5 Turbo, served as core components of CoTeach’s LLM teacher.
- The disclosed models were not used for data collection, preprocessing, experimental evaluation, or result analysis.
- GenAI-assisted writing support was also disclosed as part of the research process.