PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Fixed Features: Architecture-Dependent Sensitivity to Node Representations under Heterophily

Priyanath Maji, Sidharth Gaur, Rajavinoth Paul Durai

arXiv:2609.33764v1cs.LGcs.AIcs.IR

TL;DR

Existing heterophilic GNN comparisons usually fix node features, leaving open whether architecture behavior depends on representation. The paper varies representations across two large-scale benchmarks and finds architecture-dependent gains, while architecture rankings remain stable across representations.

  • Problem

    Architecture comparisons typically hold node representation fixed, leaving unclear whether architecture behavior changes with the available feature information.

    Source: §1 · p. 2

  • Method

    The study crosses multiple static and contextual node representations with GNN architectures on two fixed heterophilic benchmark graphs.

    Source: §3 · p. 3

  • Results

    Contextual gains range from 2.38 to 13.67 percentage points across architectures on Roman-Empire and from 0.20 to 6.78 points on Amazon-Ratings.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Architecture and representation should be treated jointly in heterophilic GNN evaluation because representation changes produce architecture-dependent gains.

    Source: §5 · p. 8, §7 · p. 9

    The primary analysis covers two large-scale heterophilic benchmarks, which is insufficient to establish a general theory of representation benefits across architectures.

    Source: §6 · p. 8

Abstract

from arXiv · show

Graph Neural Networks (GNNs) perform well on homophilic graphs but struggle in heterophilic settings, where connected nodes often carry dissimilar labels. Existing evaluations typically compare architectures under a fixed node-feature representation, leaving unclear whether conclusions about heterophily robustness remain stable as the input representation changes. We address this question by constructing parallel feature variants of two large-scale heterophilic benchmarks, Roman-Empire and Amazon-Ratings, pairing each graph with representations ranging from static fastText vectors to contextual Transformer embeddings and evaluating seven GNN architectures across these representations. We find that the effect of representation varies across architectures: on Roman-Empire, the contextual gain ranges from 2.38 percentage points for GCN-sep to 13.67 points for GAT, with H2GCN gaining 8.77 points. On Amazon-Ratings, where node text is limited to short product titles, GAT improves by 6.78 points from fastText to MPNet, while GCN-sep changes by only 0.20 points. These results show that architectural performance is conditional on node representation: the same representation change can produce different magnitudes of performance gain across architectures, so architecture and representation cannot be treated as independent evaluation factors. A rank-correlation analysis on these two benchmarks further shows that the relative ordering of architectures remains highly stable across representations, isolating differential sensitivity, rather than ranking instability, as the primary effect.

1 Introduction

The paper asks whether GNN architecture comparisons under heterophily depend on the node representations supplied to the models. It varies representations across two large-scale benchmarks to examine architecture-dependent effects.

1 Introduction

  • Existing architecture comparisons usually fix node features, leaving unclear whether architecture behavior depends on the input representation.The paper frames this as an open question for heterophilic GNN evaluation.Source: §1 · p. 1
  • On Roman-Empire, contextual representations can encode occurrence-specific syntactic information that static embeddings assign identically to repeated words.On Amazon-Ratings, short product titles may yield a smaller, qualitatively different benefit from contextualization.Source: §1 · p. 2
  • The study crosses multiple representations with seven GNN architectures on fixed graph structures, using an MLP as a feature-only reference.It reports architecture-dependent gains, including 4.24 percentage points for GAT-sep on Roman-Empire and 1.46 points for GraphSAGE on Amazon-Ratings.Source: §1 · p. 2

2 Related Work

Prior work establishes message-passing challenges under heterophily, the importance of reliable benchmarks, and the varied information encoded by node representations. This study builds on those strands by varying representations while holding each graph fixed.

Heterophily and Message Passing

  • Heterophily-focused architectures use strategies such as ego-neighbor separation, multi-hop aggregation, residual mappings, and adaptive aggregation.The paper includes established architectures and separation-based variants without claiming their mechanisms as new.Source: §2 · p. 2

Benchmark Reliability

  • Platonov et al. introduced Roman-Empire and Amazon-Ratings as larger alternatives after showing that small benchmarks can yield unstable architectural comparisons.The paper reconstructs these benchmarks and varies node representations while keeping graph structure unchanged.Source: §2 · p. 3

Node Representations in Graph Learning

  • Static lexical and contextual Transformer embeddings encode different kinds and amounts of information, without a universal quality ordering.The paper uses an MLP control to assess task-relevant informativeness empirically before comparing representations across GNN architectures.Source: §2 · p. 3

3 Rationale and Hypothesis

The study tests whether contextual representations provide task-relevant information and whether representation gains are uniform across architectures. Its design varies representations while holding graph structure and evaluation protocol fixed.

3 Rationale and Hypothesis

  • Contextual embeddings are hypothesized to help Roman-Empire because syntactic roles depend on sentence context, while short Amazon-Ratings titles may yield a smaller effect.Static embeddings cannot distinguish repeated lexical items used in different syntactic roles.Source: §3 · p. 3
  • The study contrasts the hypothesis that stronger representations narrow architectural performance differences with the alternative that gains depend on the receiving architecture.The alternative treats representation improvements and architectural design as complementary contributors.Source: §3 · p. 3
  • Crossing representations with architectures on fixed graphs and under a fixed protocol tests for uniform versus architecture-dependent performance changes.Architecture-dependent gains would make the interaction between node features and graph processing relevant to evaluation.Source: §3 · p. 3

4 Methodology

The methodology crosses established GNN architectures with multiple node representations on heterophilic benchmarks, while adding feature-only and homophilic controls. It standardizes evaluation, dataset processing, and feature generation across the study.

4.1 Experimental Design

  • Seven GNN architectures spanning aggregation, separation, multi-hop, and residual strategies are crossed with static and contextual features, alongside an MLP baseline.The study also tests selected models on WebKB and citation-network controls.Source: §4.1 · pp. 3–4

4.2 Experimental Setup

  • Macro F1 is the primary metric because both main datasets have severe class imbalance; accuracy and balanced accuracy are also reported.The setup uses ten predefined splits, selects hyperparameters on Split 1, and fixes each selected configuration across folds.Source: §4.2 · p. 4

4.3 Dataset Engineering Pipeline

  • A standardized dataset format separates graph structure from feature matrices, allowing one reconstructed graph to pair with independently generated embeddings.The processing pipeline and processed datasets are released for reproducibility.Source: §4.3 · p. 5

4.4 Heterophilic Datasets

  • Roman-Empire nodes are word occurrences labeled by syntactic role, while Amazon-Ratings nodes are products labeled by discretized average rating and use product titles as text.Roman-Empire features include fastText and Transformer embeddings; Amazon-Ratings uses fastText and two SBERT variants.Source: §4.4 · pp. 5–6

4.5 Homophilic Datasets

  • WebKB pages and Cora and PubMed papers provide additional benchmark settings with bag-of-words node features and classification labels.WebKB classifies pages into five categories, while Cora and PubMed predict research field.Source: §4.4 · p. 6, §4.5 · p. 6

5 Results and Discussion

Across Roman-Empire and Amazon-Ratings, representation changes yield architecture-dependent performance gains rather than uniform shifts. Yet architecture rankings remain stable across representations, making differential sensitivity—not ranking instability—the primary distinction.

Performance on Large-Scale Heterophily

  • 87.76 Macro F1 with BERT and 87.00 with RoBERTa make GAT-sep strongest on Roman-Empire with contextual features, but its score falls to 58.81 with the custom representation.The custom representation is a deliberately weak stress test; GCN-sep also falls from 85.79 with BERT to 67.39 with the custom representation.Source: §5 · pp. 6–8
  • A 0.07-point MPNet lead for GAT-sep over GraphSAGE on Amazon-Ratings is nominal, and neither reported reversal is statistically significant.The reported Holm-adjusted p-values are 0.18 and 1.0.Source: §5 · p. 6

Feature Representation Effects Are Architecture-Dependent

  • 53.44 Macro F1 with fastText rises to 76.33 with BERT and 76.43 with RoBERTa for the Roman-Empire MLP, showing contextual features help without graph propagation.The MLP also improves from 41.31 with fastText to 44.21 with SBERT and 44.59 with MPNet on Amazon-Ratings.Source: §5 · pp. 6, 8
  • 2.38 percentage points for GCN-sep to 13.67 for GAT on Roman-Empire, and 0.20 for GCN-sep to 6.78 for GAT on Amazon-Ratings, mark the contextual-gain ranges.The corresponding benchmark tables report Macro F1 across feature representations.Source: §5 · p. 6, Table 2 · p. 7, Table 3 · p. 7

Behavior on Small-Scale and Legacy Benchmarks

  • H2GCN leads the three small WebKB benchmarks with Macro F1 scores of 56.49, 62.88, and 61.95, but these are robustness checks given reliability concerns.On homophilic Cora and PubMed, architectures cluster more closely and the MLP remains below most GNNs.Source: §5 · p. 7

Hypothesis Evaluation

  • Contextual gains are smaller and less uniform on Amazon-Ratings, where node text consists primarily of short product titles, than on Roman-Empire's syntactic task.The paper relates Roman-Empire's contextual gains to predicting syntactic relations.Source: §5 · p. 7

6 Limitations

The findings are limited by the small number of primary benchmarks, differences among embedding families, and the short product-title text available for Amazon-Ratings.

6 Limitations

  • Two primary benchmarks are insufficient to establish a general theory of which representations benefit particular architectures.WebKB and citation-network experiments are complementary robustness checks, not direct replications.Source: §6 · p. 8
  • Embedding families vary in contextualization, pretraining data, objectives, and dimensionality, so they are not a universal feature-quality scale.The authors suggest controlled variation of representation properties or direct manipulation of feature informativeness.Source: §6 · p. 8
  • Amazon-Ratings uses product titles rather than full reviews or descriptions, which may partly explain its modest representation differences.Richer product text or non-textual attributes could test whether those effects generalize beyond short titles.Source: §6 · p. 9
  • The custom Roman-Empire Transformer lacks external semantic pretraining and is a deliberately weak feature condition, not a typical alternative embedding.Its results do not establish that pretrained Transformers universally outperform independently trained representations.Source: §6 · p. 9
  • Independent tuning means observed differences may reflect interactions among representation, architecture, and optimization configuration.Targeted GCN/GAT controls do not fully match all architectural parameters and computational factors.Source: §6 · p. 9

7 Conclusion

The paper concludes that node representation changes affect GNN architectures by different amounts, even as architecture rankings remain largely stable. It recommends evaluating architecture and representation jointly, while identifying broader and more controlled studies as future work.

7 Conclusion

  • Representation sensitivity varies substantially across architectures, while their relative ranking remains largely stable across representations.The reported sensitivity ranges are 2.38–13.67 percentage points on Roman-Empire and 0.20–6.78 points on Amazon-Ratings; the MLP control also indicates that representations differ in task-relevant information.Source: §7 · p. 9
  • Architecture conclusions should be interpreted jointly with node representations, which should be treated as an evaluation factor rather than fixed background.A single feature representation can obscure the magnitude of representation-dependent gains even when it does not obscure which architecture performs best.Source: §7 · p. 9
  • Future work should test broader graph domains, control feature informativeness, and expand factorial studies across architectures, representations, and graph regimes.Suggested controls include systematic feature corruption, dimensionality control, and task-specific probing.Source: §7 · p. 9
  • The impact statement flags pretrained language-model representations as potentially increasing computational and memory requirements and inheriting training-data biases.The experiments do not evaluate downstream decision systems or make application-specific performance, fairness, or safety claims.Source: §7 · p. 9

A GenAI Usage Statement

  • The authors report using LLMs only as supplementary tools for debugging, conceptual clarification, visualization generation, and language refinement.They state that all analyses, experimental configurations, quantitative results, and final conclusions were finalized by the authors.Source: A GenAI Usage Statement · p. 11

B Dataset Characteristics and Homophily

  • Adjusted homophily characterizes same-label edge connectivity relative to a degree-weighted null model, with values from −1 to 1.Positive values indicate more same-label connectivity than expected, and negative values indicate less; the measure is descriptive, not a direct measure of neighborhood-feature informativeness.Source: B Dataset Characteristics and Homophily · p. 11

C Relation to HeTGB

  • The study distinguishes changes in architecture ranking from differences in the magnitude of architecture-specific responses to representation changes.Its graded representation ladder and rank-correlation analysis support this distinction; the authors also describe lower sensitivity for separated variants in all tested naive/separation pairs, while noting that this pattern is not statistically established.Source: C Relation to HeTGB · pp. 11–12

D Model Architectures

  • The evaluated architectures span additive aggregation, ego-neighbor separation, concatenative aggregation, and residual designs, alongside an MLP control.GCN uses fixed adjacency weights while GAT learns attention coefficients; separation variants use distinct pathways for ego and neighbor information, and GCNII uses residual connections and identity mapping.Source: D Model Architectures · pp. 12–13

E Analytical Perspective on Architecture-Representation Interaction

The analysis explains how representation changes can propagate differently through GNN architectures, while treating its derivation as a local mechanistic interpretation rather than a predictor of benchmark outcomes. It also describes paired fold comparisons as supplementary because cross-validation folds share training data.

E Analytical Perspective on Architecture-Representation Interaction

  • A first-order decomposition separates changes propagated through an existing graph operator from changes to the operator when it depends on node representations.This allows the same input perturbation to produce different hidden-representation changes across architectures.Source: E Analytical Perspective on… · p. 13
  • For feature-independent propagation, representation sensitivity depends on how the architecture transforms and propagates the change, including its propagation operators and parameters.The analysis describes this for operators determined only by graph structure, such as normalized GCN propagation.Source: E Analytical Perspective on… · pp. 13–14
  • In attention-based propagation, a representation change can alter both message content and the relative weights assigned to messages.The operator’s dependence on node representations supplies a pathway absent from feature-independent propagation.Source: E Analytical Perspective on… · p. 14
  • Architectures with multiple information pathways can carry representation changes through direct node transformations as well as neighborhood propagation.The formulation is schematic, and the derivation holds learned parameters fixed rather than decomposing the independently tuned empirical gains.Source: E Analytical Perspective on… · p. 14
  • 2.38–13.67 percentage-point gains on Roman-Empire and 0.20–6.78 points on Amazon-Ratings illustrate that contextual representation gains vary across architectures.The analysis explains this possibility through different propagation mechanisms, but does not predict which architecture benefits most on a particular dataset.Source: E Analytical Perspective on… · pp. 14–15
  • The analysis is local and first-order, abstracting away optimization, hyperparameter selection, depth effects, and dataset properties that can influence trained-model test performance.Paired t-tests assess consistency across folds, but their p-values are supplementary because folds share training data; Table 5 reports selected GAT-sep versus naive GAT comparisons.Source: E Analytical Perspective on… · p. 15, F Statistical Comparisons · p. 15, Table 5 · p. 15

G Architectural Ranking Stability and Interaction Analysis

Across Roman-Empire and Amazon-Ratings, architecture rankings remain highly stable across feature representations even as performance sensitivity differs substantially by architecture. Nominal reversals were not statistically supported after the stated multiple-comparison criteria.

G Architectural Ranking Stability and Interaction Analysis

  • Architecture rankings remain highly stable across both benchmarks, including invariant ordering among Roman-Empire’s fastText, BERT, and RoBERTa representations.Table 6 reports Spearman and Kendall ranking correlations, while its inversion counts describe pairwise order reversals.Source: G Architectural Ranking Stability and… · p. 15, Table 6 · p. 16
  • 13.67 percentage points for GAT versus 2.38 for GCN-sep on Roman-Empire, and 6.78 for GAT versus 0.20 for GCN-sep on Amazon-Ratings, show differential representation sensitivity.Despite these contrasting gains, the relative architecture ordering remains highly stable across representations.Source: G Architectural Ranking Stability and… · p. 15
  • None of the twelve nominal architecture-pair-by-representation-pair reversals across the benchmarks survived both the bootstrap-interval and Holm-adjusted significance criteria.The analysis therefore found zero statistically supported rank reversals under its stated family-wise error criterion.Source: G Architectural Ranking Stability and… · pp. 15–16

H Capacity and Structural Control Experiments

Capacity and structural controls on Roman-Empire test whether naive-versus-separation architecture gaps are explained by model size or macro-architecture. The reported controls indicate that capacity or shallower baseline configurations alone do not explain the gaps.

H Capacity and Structural Control Experiments

  • 36.29% versus 87.76% Macro F1 for matched 5-layer, 128-dimensional BERT GAT and GAT-sep shows a large gap despite matching depth and hidden dimension.Under RoBERTa, high-capacity naive GAT reaches 39.13%, while independently selected GAT-sep reaches 87.00%.Source: H Capacity and Structural Control… · p. 16, Table 7 · p. 17
  • 65.07% versus 85.79% Macro F1 on the corresponding BERT setting shows that a capacity-matched naive GCN does not reproduce GCN-sep performance.The naive GCN’s hidden dimension was chosen to approximately match the dominant hidden-layer parameter count of GCN-sep.Source: H Capacity and Structural Control… · pp. 16–17, Table 7 · p. 17
  • The controls do not establish universal superiority of an architectural mechanism, but show that selected baseline capacity or shallower configurations alone do not explain the observed differences.The experiments were conducted on Roman-Empire using BERT and RoBERTa representations.Source: H Capacity and Structural Control… · pp. 16–17

I Additional Limitations and Implementation Notes

The section qualifies representation sensitivity by distinguishing large gains from overall effectiveness and noting limits in the explored configurations. It also clarifies how the feature-only baseline and independent tuning affect interpretation.

I Additional Limitations and Implementation Notes

  • GCNII depth was limited to 10 layers on Amazon and small WebKB benchmarks, versus 32 or 64 on Roman, potentially limiting its best attainable WebKB performance.The restriction reflected the higher training cost of deeper configurations.Source: I Additional Limitations and… · p. 17
  • 44.30% to 76.43%: the feature-only MLP rises from the custom Roman-Empire representation to RoBERTa, showing that node-level information varies across representations.The MLP is a feature-only reference estimating predictive information available without neighborhood propagation.Source: I Additional Limitations and… · p. 17
  • Independent tuning means performance differences reflect representation, architecture, and selected optimization configuration together; matched budgets or fixed hyperparameters could offer further insight.The authors identify matched parameter counts, computational budgets, or jointly fixed hyperparameters as possible alternatives.Source: I Additional Limitations and… · p. 17

J Interpreting Feature Sensitivity for Naive GAT

  • 13.67 percentage points: Naive GAT rises from 38.96% Macro F1 with fastText to 52.63% with RoBERTa, yet remains the lowest-performing GNN across fastText, BERT, and RoBERTa.Its standard deviation increases from 1.23 percentage points with fastText to 2.67 with BERT and 3.40 with RoBERTa.Source: J Interpreting Feature Sensitivity for… · p. 17
  • A large contextual gain reflects representation-dependent change, not evidence that GAT is most effective or uniformly benefits most from contextual features.The result illustrates sensitivity, not an architecture ranking by responsiveness.Source: J Interpreting Feature Sensitivity for… · pp. 17–18

K Per Class Analysis

Per-class results show uneven performance across Roman-Empire syntactic roles, while H2GCN improves one class relative to the feature-only MLP. The accompanying figures and tables describe performance views and hyperparameter configurations across benchmarks.

K Per Class Analysis

  • Approximately 64% and 58% F1: Roman-Empire Class 7 (compound modifiers) and Class 9 (direct objects) outperform Class 14 (passive nominal subjects) at 32%.The differences suggest contextual node representations provide unequally useful information across syntactic roles.Source: Figure 1 · p. 18, K Per Class Analysis · p. 18
  • H2GCN substantially improves Class 14 relative to the feature-only MLP, suggesting propagation can add information useful for some neighboring-node-dependent categories.This comparison uses the same node features for H2GCN and the MLP.Source: K Per Class Analysis · p. 19
  • Figures 3–5 depict Macro F1 scaling trends across Roman-Empire and Amazon-Ratings feature spaces and cross-validation performance footprints across WebKB and Planetoid.Figure 3 spans restricted corpus, static lexical, and deep contextual features; Figure 4 spans static lexical, sentence, and base contextual features.Source: Figure 3 · p. 19, Figure 4 · p. 20, Figure 5 · p. 20
  • Figures 6–7 compare Roman-Empire per-class F1 across architecture transitions and feature spaces; Tables 8–9 report overall and balanced accuracy across feature spaces for Roman-Empire and Amazon-Ratings benchmarks.Tables 10–11 give optimal hyperparameter configurations for large-scale Roman-Empire and Amazon-Ratings benchmarks across feature spaces, and small-scale WebKB heterophilic benchmarks.Source: Table 8 · p. 19, Figure 6 · p. 21, Table 9 · p. 21

Related papers

Loading 2609.33764v1…