PaperlayerView exact revision on arXiv

Source-linked AI summary

Augmenting Low-Resource Text Classification with Graph-Grounded Pre-training and Prompting

Zhihao Wen, Yuan Fang

arXiv:2305.03324v1cs.IR

TL;DR

Low-resource text classification is difficult when labeled samples are scarce, and many texts also have graph relationships that existing approaches do not fully exploit. G2P2 jointly pre-trains text and graph representations and applies prompting, with reported advantages over baselines in zero- and few-shot tasks.

  • Problem

    Low-resource classification has few or no labeled examples, and existing approaches do not fully exploit the graph relationships grounding many text documents.

    Source: §1 · p. 1

  • Method

    G2P2 jointly pre-trains graph and text encoders with three graph-interaction contrastive strategies, then uses prompting for downstream classification.

    Source: §1 · p. 2

  • Results

    G2P2 outperforms the best baseline by around 2–8% in 5-shot classification and significantly outperforms baselines in zero-shot classification.

    Source: §4.2 · pp. 7–8

  • Takeaways & Limitations

    The reported zero-shot results indicate that graph-grounded contrastive pre-training is effective without labeled data, a setting relevant to evolving classes.

    Source: §4.2 · p. 8

    Handcrafted prompt effectiveness varies across models and datasets, making prompt engineering labor-intensive and uncertain.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

Text classification is a fundamental problem in information retrieval with many real-world applications, such as predicting the topics of online articles and the categories of e-commerce product descriptions. However, low-resource text classification, with few or no labeled samples, poses a serious concern for supervised learning. Meanwhile, many text data are inherently grounded on a network structure, such as a hyperlink/citation network for online articles, and a user-item purchase network for e-commerce products. These graph structures capture rich semantic relationships, which can potentially augment low-resource text classification. In this paper, we propose a novel model called Graph-Grounded Pre-training and Prompting (G2P2) to address low-resource text classification in a two-pronged approach. During pre-training, we propose three graph interaction-based contrastive strategies to jointly pre-train a graph-text model; during downstream classification, we explore prompting for the jointly pre-trained model to achieve low-resource classification. Extensive experiments on four real-world datasets demonstrate the strength of G2P2 in zero- and few-shot low-resource text classification tasks.

1 INTRODUCTION

Low-resource text classification is important because labeling is costly, while existing text and graph approaches do not fully use network relationships and fine-grained text semantics together. G2P2 jointly pre-trains graph and text representations through graph interactions, then uses prompting for zero- and few-shot classification.

1 INTRODUCTION

  • Low-resource classification addresses settings with few or no labels, which are costly and time-consuming to obtain.Supervised deep learning has achieved success especially with large, high-quality labeled datasets.Source: §1 · p. 1
  • Existing PLMs and prompting do not exploit graph relationships, while conventional GNN text features are coarse-grained and miss subtle textual semantics.Common GNN inputs include bag-of-words or aggregated shallow word embeddings.Source: §1 · pp. 1–2
  • G2P2 jointly aligns text and graph representations through text-node, text-summary, and node-summary contrastive interactions.This graph-grounded pre-training addresses the loose coupling of text and graph representations in a naïve language-model-plus-GNN pipeline.Source: §1 · p. 2
  • For downstream classification, G2P2 uses handcrafted discrete prompts for zero-shot settings and continuous prompts for few-shot settings.Its context-based prompt initialization also uses graph structure between texts to provide a more informative starting point.Source: §1 · p. 2
  • The paper presents G2P2 as a first joint text-and-graph pre-training approach for low-resource classification and evaluates it on four real-world datasets.The reported experiments cover zero- and few-shot text classification.Source: §1 · p. 2

2 RELATED WORK

Prior graph-text methods either treat text coarsely or keep graph and text encoders decoupled, while language prompting offers discrete and continuous forms. Related learning paradigms differ in their label requirements and support for novel classes.

Graph neural networks

  • Prior graph methods for text tasks use coarse-grained text or decoupled graph and text encoders; GLEM integrates both but is not designed for low-resource learning.The cited approaches include knowledge-graph, heterogeneous-graph, and multimodal methods.Source: §2 · p. 2

Language pre-training and prompting

  • Prompting adapts PLMs with task templates, including discrete natural-language prompts and continuous prompts.The passage notes prompting applications in classification, machine translation, relation extraction, and GNN node classification.Source: §2 · p. 2

Zero- or few-shot paradigms

  • Unlike semi-supervised and meta-learning paradigms, this setting needs label-free pre-training and no labeled meta-training phase, while graph interactions provide varied text relationships.The passage also contrasts graph-based zero-shot classification with approaches that rely on curated side information for class prototypes.Source: §2 · p. 2

3 PROPOSED APPROACH

G2P2 jointly pre-trains text and graph encoders using graph-based contrastive interactions, then applies prompting for zero- or few-shot classification. Its setup uses label-free self-supervised pre-training; the downstream task classifies queries among support-set classes, with zero-shot relying on class metadata.

3.1 Preliminaries

  • Low-resource classification predicts query labels from a small labeled support set, while zero-shot classification has no labeled examples and relies on class metadata.Pre-training uses label-free data rather than training and testing episodes.Source: §3.1 · p. 3

3.2 Overview of G2P2

  • G2P2 has two stages: graph-grounded contrastive pre-training, followed by graph-grounded prompting for low-resource classification.The framework jointly trains a text encoder and a graph encoder, then uses prompt-assisted inference.Source: §3.2 · p. 3, Figure 1 · p. 3

3.3 Graph-grounded contrastive pre-training

  • A transformer encodes each document and a GNN encodes its graph node, with their representations aligned in a dual-modal embedding space.The three interaction types are text-node, text-summary, and node-summary.Source: §3.2 · p. 3, §3.3 · pp. 3–4
  • Text-node contrastive training matches each document with its corresponding graph node while separating nonmatching document-node pairs.The method uses the document-node bijection to define matching pairs and optimizes similarity against the other pairs.Source: §3.3 · pp. 3–4
  • Text-summary contrastive training aligns each document’s text embedding with a summary embedding formed from its graph neighbors.Neighboring documents provide the summary; a fixed number of neighbors is sampled for efficiency.Source: §3.3 · p. 4
  • Node-summary contrastive training aligns each graph node embedding with the neighborhood-based summary text embedding.The three contrastive losses are integrated into the overall pre-training objective, with λ balancing summary-based interactions.Source: §3.3 · p. 4

3.4 Prompting joint graph-text model

  • Zero-shot classification uses handcrafted discrete prompts with class-label text, while few-shot classification uses continuous prompts tuned on labeled support examples.Few-shot tuning freezes the dual encoders and updates the prompt embeddings; graph contexts initialize these prompts by averaging embeddings from support documents and sampled neighbors.Source: §3.4 · p. 5

4 EXPERIMENTS

Experiments evaluate G2P2 on four graph-grounded text corpora across zero- and few-shot classification, followed by analyses of its interaction strategies, prompting, efficiency, and generalization. G2P2 outperforms baselines in the reported low-resource settings, including when only one or two labeled examples are available.

4.1 Experimental setup

  • The evaluation uses four public graph-grounded text corpora and averages results over 5-way tasks covering all classes.Each task has a K-shot support set and a same-sized validation set; remaining examples form the query set.Source: §4.1 · pp. 5–6

4.2 Performance of low-resource classification

  • G2P2 beats the best baseline by around 2–8% with statistical significance in 5-shot classification.PLMs generally outperform GNNs, while further fine-tuning PLMs gives mixed results; G2P2 also significantly outperforms P-Tuning v2, and continuous prompts improve on G2P2-p.Source: Table 2 · p. 7, §4.2 · pp. 7–8
  • G2P2 generally performs best across shot counts and remains relatively robust with only 1 or 2 shots.Performance decreases for all approaches as shots are reduced, but baselines suffer more under extreme low-resource settings.Source: Figure 3 · p. 8, §4.2 · p. 8
  • G2P2 and G2P2+d significantly outperform the baselines in zero-shot classification.Handcrafted discrete prompts can outperform label text alone, though finding effective templates requires engineering and their results vary across models and datasets.Source: Table 3 · p. 8, §4.2 · p. 8

4.3 Model analyses

  • Learning continuous prompts improves performance over label text alone, and context-based initialization gives a small but consistent advantage over random initialization.The comparison evaluates prompt-tuning variants in the 5-shot setting.Source: §4.3 · p. 8, Table 4 · p. 9
  • Removing L1 substantially hurts performance, while adding L2 or L3 to L1 improves it and all three losses together outperform one- or two-loss combinations.The authors identify text-node interaction as the fundamental pre-training component and report benefits from the additional graph interactions.Source: §4.3 · p. 8, Table 4 · p. 9
  • Performance is generally better and stable for slightly bigger λ (e.g., ≥10); prompt length has little effect, making a small M (e.g., 4) an efficient choice.Source: §4.3 · p. 9
  • Prompt-tuning achieves 2.1–18.8x speedups over fine-tuning baselines, and G2P2 performs very well in the inductive setting.The inductive evaluation pre-trains on a subset and classifies on a disjoint subset; prompt-tuning updates 2,048 parameters compared with more than 100M for fine-tuning.Source: Table 5 · p. 9, §4.3 · p. 9, Table 6 · p. 9

5 CONCLUSION

G2P2 jointly pre-trains graph and text encoders with three graph-interaction contrastive strategies, then prompts the resulting model for downstream low-resource classification. Experiments show advantages in zero- and few-shot settings, while the approach requires a graph to complement the texts.

5 CONCLUSION

  • G2P2 combines three graph interaction-based contrastive strategies in pre-training with prompting the jointly pre-trained graph-text model for downstream classification.The paper reports advantages in zero- and few-shot text classification.Source: §5 · p. 9
  • Experiments show G2P2's advantages in zero- and few-shot text classification.Source: §5 · p. 9
  • The approach requires a graph to complement the texts; synthetic graphs based on word co-occurrences or other relations are suggested when an organic graph is unavailable.Further exploration of synthetic graphs is left to future work.Source: §5 · p. 9

Related papers

Loading 2305.03324v1…