PaperlayerView exact revision on arXiv

Source-linked AI summary

Why Does Graph Learning Fail to Fully Benefit from a Text Teacher?

Fumiaki Kimino, Ryoma Sato

arXiv:2608.25741v1cs.LGcs.CL

TL;DR

The paper asks why adding a text teacher to cross-domain graph learning does not reliably improve predictive performance. It combines FUG’s feature-universal graph encoder with GLEM’s alternating text–graph optimization, and finds that teacher influence is limited by several representation-transfer and objective conflicts. The resulting analysis identifies six factors explaining the insufficient improvement.

  • Problem

    Graph models trained on one dataset may not transfer effectively to another, motivating cross-domain graph learning that also incorporates textual information.

  • Method

    The paper combines FUG, which handles differing node-feature dimensionalities, with GLEM, which alternates language-model E-steps and GNN M-steps, and evaluates staged variations of E-step influence.

  • Results

    FUG+GLEM did not clearly outperform FUG alone: standard-probe accuracy was 0.7459 for FUG-only versus 0.7480 for FUG+GLEM-ITT, while balanced accuracy decreased from 0.6016 to 0.6001.

  • Takeaways & Limitations

    Text-teacher alignment alone does not guarantee better graph learning; transfer pathways, node-specific textual information, and target-task alignment also require careful design.

  • Takeaways & Limitations

    The M-step jointly preserves source-side FUG geometry and approaches the teacher anchor, producing a compromise representation because the graph, text, and target-classification spaces may not align.

Abstract

from arXiv · show

Graph neural networks (GNNs) are widely used to represent complex interactions and relationships among entities. We investigate a multimodal model that combines two complementary ideas: a self-supervised method that enables a GNN encoder pretrained on one dataset to operate directly on another dataset with a different node-feature dimensionality, without rebuilding the model or realigning the data; and an alternating optimization method that updates a language-model module in an E-step and a GNN module in an M-step, rather than jointly training a large language model and a GNN end to end on a large graph. Despite expectations, the combined model did not sufficiently improve predictive performance. We identify six factors: (1) an external anchor in the E-step has a strength-safety trade-off: a weak anchor has little effect, whereas an overly strong anchor can damage the graph representation; (2) the knowledge of the E-step teacher is not injected directly into the GCN embedding Z; (3) the representation space constructed in the M-step is not optimized for the same objective as the E-step teacher space, resulting in a compromise representation for target classification; (4) GCN propagation averages a node's own textual information with information from its neighbors; (5) cosine alignment does not guarantee axes that are discriminative for classification, so stronger geometric alignment with the E-step text anchor need not sufficiently improve the target decision boundary or classification performance; and (6) the force that preserves the source-side self-supervised geometry in the M-step conflicts with the force that moves the representation toward the E-step teacher. We support these observations through a staged set of experiments that varies the influence of the E-step.

1 Introduction

The paper combines FUG’s feature-universal graph pretraining with GLEM’s alternating text–graph optimization for cross-domain multimodal graph learning, but finds no clear improvement over FUG alone and investigates why.

  • GNNs represent relational structure, but models trained on one graph dataset may not transfer effectively to another.
  • FUG lets a GNN encoder pretrained on one dataset process another dataset with different node-feature dimensionality without rebuilding or explicit feature alignment.
  • GLEM alternates language-model updates in an E-step with GNN updates in an M-step instead of jointly training both modules end to end.
  • FUG+GLEM combines FUG in the GLEM M-step with a large text-attributed graph in the E-step to support cross-domain node representations for classification and link prediction.
  • Experiments found no clear improvement over FUG alone, motivating analysis of why the E-step teacher failed to sufficiently improve the M-step graph representation.

2 Background

FUG constructs feature-format-independent graph representations through column-wise feature encoding and self-supervised losses, while GLEM alternates text and graph optimization through a variational EM framework.

  • 2.1 FUG: FUG is a label-free self-supervised model that generates basis transformations to map arbitrary node-feature formats into a unified representation.
  • 2.1 FUG: FUG processes each feature column with a Dimension Encoder, producing per-feature vectors that enable datasets with different feature counts to share fixed-dimensional representations.
  • 2.1 FUG: FUG jointly optimizes Dimension Encoder, connected-node representation, and global representation-distribution losses.
  • 2.1 FUG: The connected-node loss treats graph edges as positive pairs and brings representations of adjacent nodes closer, assuming homophilous graphs often have similar neighboring labels or semantics.
  • 2.2 GLEM: GLEM integrates graph structure and language information by alternating an E-step language-model update with an M-step GNN update.
  • 2.2 GLEM: In the E-step, the language model uses observed labels and GNN pseudo-labels; in the M-step, the GNN learns from observed labels and language-model pseudo-labels.
  • 2.2 GLEM: GLEM uses an ELBO because unlabeled-node labels are unobserved, then optimizes its variational distribution and graph model in alternating steps.

3 Proposed Method

FUG+GLEM-ITT combines an independently trained text teacher with FUG through alternating E-like and M-step updates. Auxiliary cosine anchors preserve text-derived and pre-propagation information while the GCN representation remains optimized with the FUG self-supervised objective.

  • 3 Proposed Method: FUG+GLEM-ITT replaces GLEM’s graph module with FUG and uses an independently trained TextHead that does not imitate the current GCN representation.TextHead uses raw text-hash features, source-label supervision, and dropout-based self-supervision.
  • 3.2 An EM-Like Framework for FUG+GLEM-ITT: Training alternates short TextHead refreshes that produce a new text anchor with M-step updates of the FUG encoder.The procedure includes TextHead pretraining, FUG warm-up, and six EM-like iterations; each E-like refresh lasts two epochs and each M-step lasts 18 epochs.
  • 3.2 An EM-Like Framework for FUG+GLEM-ITT: An anchor loss moves the post-propagation GCN embedding toward a separate reference representation using cosine similarity.The four references include TextHead, MLP-only, raw text-hash, and external or random text anchors.
  • 3.2 An EM-Like Framework for FUG+GLEM-ITT: The M-step combines the FUG self-supervised loss with TextHead, MLP-only, raw-text, and external-anchor losses.The reported weighting is LM = 0.80LFUG + 1.80Ltext + 1.10LMLP + 0.70Lraw + 0.25Lext.
  • 3.3 Representations Used for Evaluation: Evaluation compares Full GCN Z with text-only representations and concatenated late-fusion ensembles under linear probing.The evaluated text references are Raw Text Hash and Raw MPNet; ensembles include [ZGCN; Xhash] and [ZGCN; XMPNet].

4 Experimental Setup and Main Result

The study transfers from Amazon Digital Music reviews to the OpenAlex paper network. FUG+GLEM-ITT did not clearly outperform FUG-only, showing only a small standard-probe gain and a slight balanced-accuracy decrease.

  • 4 Experimental Setup and Main Result: The source domain is Amazon Digital Music, while the target domain is the OpenAlex paper network.Amazon provides the labeled review network; OpenAlex provides the target scholarly paper network.
  • 4.2 Main Experimental Result: 0.21 percentage points was the approximate standard-probe accuracy improvement from FUG-only to FUG+GLEM-ITT with an external TextHead anchor.Standard-probe accuracy was 0.7459 for FUG-only and 0.7480 for FUG+GLEM-ITT on OpenAlex.
  • 4.2 Main Experimental Result: 0.6001 versus 0.6016 balanced accuracy represented a slight decrease for FUG+GLEM relative to FUG-only.These results did not support the hypothesis of a reliable improvement of at least 1%.

5 Analysis

The staged experiments show why external text teachers did not clearly improve FUG: teacher information reaches GCN Z through constrained alignment, while competing objectives and propagation reshape it. Stronger teacher alignment therefore does not reliably produce better target classification.

  • Experimental design: The experiments progressively increase teacher externality and semantic strength from self-copy GCN output, through raw text hashing, to frozen MPNet embeddings.Exp2 uses the current GCN representation, Exp3 uses a fixed lexical anchor, and Exp4 uses contextual semantic embeddings.
  • Bottlenecks: MPNet reached 0.7888 as a raw representation, but Full GCN Z reached only 0.7437 with MPNet as teacher, whereas late fusion reached 0.7845.This places the bottleneck in transferring MPNet’s semantic information into GCN Z rather than in MPNet’s target-relevant information.
  • Representation objectives: FUG’s self-supervised graph objective and the teacher’s semantic or lexical objective are not necessarily aligned with the target classification objective, producing a compromise representation.In Exp4, MPNet alignment improved while Full GCN Z balanced accuracy remained 0.7437.
  • Propagation effects: GCN propagation can mix node-specific textual information with neighborhood information, potentially diluting discriminative features; this pattern was observed mainly for balanced accuracy.Full GCN Z underperformed Raw Text Hash in balanced accuracy, but the evidence does not establish general degradation across all metrics or directly prove the mechanism.
  • Alignment and optimization: Cosine alignment improved in Exp3 and Exp4 without corresponding gains in Full GCN Z performance, showing that geometric similarity alone does not ensure classification-effective axes.The M-step simultaneously preserves source-side FUG geometry and approaches the text anchor, and candidate selection likewise suggests these objectives are not fully consistent.

6 Conclusion

The paper explains why combining FUG with GLEM-ITT did not reliably improve graph learning despite text-teacher alignment. Its findings emphasize designing the transfer pathway, preserving discriminative textual information, and evaluating whether it reaches the graph representation.

  • Conclusion: Additional experiments and analysis provide evidence about why FUG+GLEM-ITT failed to outperform the FUG baseline.The investigation focuses on the mechanism underlying the limited benefit from the E-step teacher.
  • Conclusion: Introducing a strong text teacher and aligning graph outputs with teacher representations does not by itself guarantee better graph representation learning performance.The conclusion identifies teacher design, transfer into GCN Z, preservation after propagation, and target-boundary alignment as distinct considerations.
  • Implications: Knowledge-transfer success should be assessed by whether discriminative information reaches GCN Z, rather than by cosine alignment alone.This recommendation follows the paper’s finding that stronger geometric alignment need not improve target classification.
  • Implications: The implications extend beyond this FUG+GLEM-ITT implementation to methods that seek to improve graph learning with textual information.The paper presents the findings as broader design guidance for multimodal graph-learning methods.

A Results of the Additional Experiments

The additional experiments compare cross-domain transfer, representation stages, alignment, and checkpoint selection across FUG and FUG+GLEM-ITT settings. They show that stronger text alignment and EM-like updates do not consistently improve target classification or source-side selection.

  • Table 2 compares cross-domain transfer performance across the experimental settings, including FUG-only and FUG+GLEM variants.
  • 0.7845 is the GCN+MPNet ensemble value, while 0.7724 is the GCN+Raw-Text-Hash ensemble value.
  • Exp2 selects the pre-update warm-up checkpoint rather than any post-distillation checkpoint based on source validation balanced accuracy.
  • Table 3 reports OpenAlex standard-probe test accuracies for Raw Text Hash, MLP-only, Full GCN Z, and ensemble representations.
  • Negative ∆Hash→GCN and ∆MPNet→GCN values indicate that Full GCN Z underperforms the corresponding raw-text representation.
  • Table 5 relates changes in alignment metrics to Full GCN-Z classification performance, while Table 6 reports candidate metrics and model selection.
  • The composite score selects em6 over SWA despite SWA having the highest source-side valid_bacc.

B Implementation and Evaluation Details

The implementation uses text-derived graph inputs, domain-specific data construction, label-free target preprocessing, and checkpoint procedures for evaluating cross-domain transfer. Reported runs use retained original graph edges and linear-probe evaluation.

  • FUG+GLEM-ITT uses loss weights WFUG = 0.80, Wtext = 1.80, WMLP = 1.10, Wraw = 0.70, and Wext = 0.25.
  • Digital Music reviews and OpenAlex papers are constructed as graph nodes with text-derived labels and undirected retained edges.
  • Each node uses 2,048 concatenated word- and character-level hashing dimensions followed by ℓ2 normalization.
  • Digital Music metadata tuples are split 60:20:20, keeping duplicate-review tuples within the same split to prevent leakage.
  • OpenAlex retains 6,984 nodes across 20 classes and uses a 60:20:20 linear-probe split of 4,190, 1,397, and 1,397 nodes.
  • OpenAlex features are standardized using all unlabeled retained-node features, making the preprocessing label-free but transductive.
  • Reported runs retain original edges; the k = 10 nearest-neighbor fallback is used only when no retained edge exists and is not used here.
  • Checkpoint selection procedures are summarized for FUG+GLEM-ITT and Exp1–Exp4, with source validation and alignment metrics defined separately from target evaluation.

C.1 Detailed Analysis of Text-to-GCN Projection

Text teachers are projected into the lower-dimensional GCN-Z space rather than injected directly, so compression and projection can discard classification-relevant semantic structure. The same dimensionality mismatch affects Raw Text Hash representations.

  • MPNet embeddings are projected from 768 to 1,024 dimensions before cosine alignment with GCN Z.
  • The MPNet teacher is used as a projected anchor rather than being fed directly into the GCN, potentially losing semantic structure.
  • Raw Text Hash compresses 2,048 dimensions into the 1,024-dimensional GCN-Z space before alignment.
  • This projection can reduce classification-relevant directions, with Raw Text Hash accuracy of 0.7623 versus approximately 0.7459–0.7480 for Full GCN Z.
  • Cosine alignment transfers a target vector rather than reproducing the teacher’s semantic space wholesale.

C.2 Mismatch between the Objectives of the Teacher and FUG Representation Spaces

The M-step jointly optimizes teacher alignment with FUG self-supervised objectives, propagation, and source-graph structure, producing a compromise representation. Improved alignment therefore need not improve target classification.

  • The numerical coefficient λ_dim = 200.0 alone does not establish that the dimension loss dominates optimization.
  • Teacher alignment is one of several jointly optimized signals, alongside FUG objectives, GCN propagation, and source-graph structure.
  • The resulting GCN Z is a compromise representation rather than a reproduction of the teacher space alone.
  • In Exp4, mpnet_cos increased from 0.0095 to 0.4341, while Full GCN Z balanced accuracy remained 0.7437.
  • Closer alignment with MPNet therefore did not necessarily align GCN Z with directions useful for the OpenAlex classification boundary.

C.3 Comparison of Balanced Accuracy Before and After GCN Propagation

Across the reported experiments, raw text representations often matched or exceeded post-propagation GCN representations in balanced accuracy, although this ordering may not generalize across metrics. The results show performance differences before and after propagation but do not directly establish textual-information dilution.

  • 0.7888 balanced accuracy for Raw MPNet exceeded 0.7437 for Full GCN Z on the OpenAlex test set.
  • 0.7623 balanced accuracy for Raw Text Hash also exceeded 0.7437 for Full GCN Z on the OpenAlex test set.
  • In FUG-only experiments, Full GCN Z reached 0.7459 balanced accuracy, below Raw Text Hash at 0.7623 and MLP-only at 0.7480.
  • With TextHead as an external anchor, Full GCN Z achieved 0.7480, below Raw Text Hash at 0.7623 and the ensemble at 0.7717.
  • GCN aggregation combines a node representation with neighbor representations through normalized adjacency and learned transformations, whose effectiveness depends on graph-label structure.
  • The reported balanced-accuracy ordering does not necessarily hold for other metrics, and the experiments do not directly demonstrate that propagation dilutes textual information.

C.4 Trade-off Between FUG SSL and Teacher Alignment

The M-step balances the source-side FUG self-supervised objective against several teacher-alignment objectives rather than optimizing the teacher anchor alone. This creates a compromise whose alignment, source validation, and target-performance effects depend on the relative influence of these objectives.

  • The M-step simultaneously preserves FUG self-supervised geometry and aligns GCN Z with a frozen MPNet semantic anchor, creating potentially conflicting objectives.
  • The selected em6 model had valid_bacc 0.4766 at source validation, slightly below warmup's 0.4788, but achieved the highest composite score.
  • SWA achieved valid_bacc= 0.4789 but had a lower composite score than em6, showing that source validation performance and text-anchor proximity are not fully aligned.
  • The model-selection composite score weights source valid_bacc at 0.25 and ood_cos at 0.35, while also incorporating text_cos and other alignment terms.
  • The M-step uses WFUG = 0.80 and Wtext = 1.80, but coefficient values alone do not determine effective influence because loss and gradient scales may differ.
  • Retaining FUG SSL while aligning post-propagation ZGCN with multiple anchors produces a compromise representation.
  • The experiments are consistent with weak teacher influence providing limited benefit, while stronger alignment may alter FUG geometry or reduce source validation performance.
Loading 2608.25741v1…