PaperlayerView exact revision on arXiv

Source-linked AI summary

TANGO: Graph Neural Dynamics via Learned Energy and Tangential Flows

Moshe Eliasof, Eldad Haber, Carola-Bibiane Schönlieb

arXiv:2508.05070v1cs.LG

TL;DR

Deep and long-range graph learning faces challenges including vanishing gradients, over-smoothing, and over-squashing, while task-driven energy learning raises the question of how to guide GNN representations. TANGO combines learned energy descent with energy-preserving tangential flows; the paper reports strong benchmark performance and theoretical stability guarantees under mild assumptions.

  • Problem

    Graph neural networks face vanishing gradients, over-smoothing, and over-squashing, and the paper asks how to learn a task-driven energy landscape whose minimization corresponds to a downstream task.

    Source: §1 · pp. 1–2

  • Method

    TANGO incorporates a learnable Lyapunov energy into message passing and combines energy-minimizing gradient descent with a tangential flow that evolves along energy level sets while preserving energy.

    Source: §1 · p. 2

  • Results

    TANGO achieves competitive or better performance than strong, widely used baselines, and under mild assumptions satisfies Lyapunov conditions for stable dynamics.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The tangential component supports continued feature evolution in flat or ill-conditioned energy landscapes, and the paper relates this property to mitigating oversquashing.

    Source: §7 · p. 9

    The theoretical analysis assumes a connected input graph and an energy function that is twice differentiable and bounded from below.

    Source: §4 · p. 5

Abstract

from arXiv · show

We introduce TANGO -- a dynamical systems inspired framework for graph representation learning that governs node feature evolution through a learned energy landscape and its associated descent dynamics. At the core of our approach is a learnable Lyapunov function over node embeddings, whose gradient defines an energy-reducing direction that guarantees convergence and stability. To enhance flexibility while preserving the benefits of energy-based dynamics, we incorporate a novel tangential component, learned via message passing, that evolves features while maintaining the energy value. This decomposition into orthogonal flows of energy gradient descent and tangential evolution yields a flexible form of graph dynamics, and enables effective signal propagation even in flat or ill-conditioned energy regions, that often appear in graph learning. Our method mitigates oversquashing and is compatible with different graph neural network backbones. Empirically, TANGO achieves strong performance across a diverse set of node and graph classification and regression benchmarks, demonstrating the effectiveness of jointly learned energy functions and tangential flows for graph neural networks.

1 Introduction

TANGO addresses challenges in deep and long-range graph learning by learning a task-driven energy landscape and evolving node features through energy descent and tangential motion. The paper reports competitive or better performance than strong baselines and theoretical guarantees under mild assumptions.

1 Introduction

  • Deep or long-range GNN learning faces vanishing gradients, over-smoothing, and over-squashing, while some prior approaches rely on relatively simple energy functionals such as Dirichlet energy.Source: §1 · p. 1
  • The motivating question is how to learn a task-driven energy whose minimization corresponds to a downstream task, rather than data likelihood as in generative energy models.Source: §1 · p. 2
  • TANGO combines a gradient-descent flow that minimizes learned energy with a tangential flow that evolves along energy level sets while preserving energy.The framework incorporates a learnable Lyapunov energy into message passing.Source: §1 · p. 2
  • Under mild assumptions, TANGO satisfies Lyapunov conditions for stable dynamics, and its tangential component helps mitigate oversquashing through expressive yet controlled propagation.Source: §1 · p. 3
  • TANGO performs competitively with or surpasses strong, widely used baselines across graph-learning benchmarks.The introduction reports this overall comparison without specifying metrics or benchmark-by-benchmark values.Source: §1 · p. 3

2 Mathematical Background

The mathematical background describes continuous dynamical systems and the Lyapunov conditions used to assess stability. A Lyapunov function is nonnegative near an equilibrium, vanishes there, and does not increase along trajectories.

2 Mathematical Background

  • A continuous dynamical system has state h(t) in R^d at time t and can be represented by a first-order ODE with continuous vector field F.Source: §2 · p. 3
  • An equilibrium h* satisfies F(h*) = 0, and a Lyapunov function V is continuously differentiable, nonnegative near h*, zero at h*, and has nonpositive trajectory derivative there.Source: §2 · p. 3
  • The Lyapunov conditions mean energy is lower-bounded by zero and does not increase along system trajectories; the stated criterion concludes stability when dV/dt ≤ 0 in a neighborhood.Source: §2 · p. 3

3 Method

TANGO treats node features as the state of graph dynamics, with network depth corresponding to time in a continuous-system view. Its method combines energy descent and energy-preserving tangential motion, then implements these flows as graph neural network layers.

3 Method

  • TANGO represents node features as a matrix H(t), with continuous time interchangeable with discrete network depth ℓ when describing layerwise features.Source: §3 · p. 3

3.1 Optimizing Features with Energy Tangential and Gradient Steps

  • The feature dynamics balance energy-gradient descent with a direction orthogonal to the gradient, so the first flow reduces energy while the tangential flow preserves it.Source: §3.1 · p. 4
  • Tangential motion is introduced because gradient flow alone may converge slowly and restrict training dynamics; its combination with descent can offer a better overall descent direction.The passage also notes that many gradient-flow steps imply many effective network layers and high computational costs.Source: §3.1 · p. 4
  • A predicted feature-update direction is projected to remove its component shared with the energy gradient, producing an orthogonal tangential direction.When the energy gradient is zero, the tangential direction is defined as the predicted update itself.Source: §3.1 · p. 4

3.2 TANGO Graph Neural Networks

  • TANGO discretizes the continuous dynamics with forward Euler to form graph neural layers, using a step size and coefficients that balance descent and tangential terms.Source: §3.2 · p. 4
  • The energy is computed with a graph neural network and an MLP, while pooled features feed an MLP and sigmoid to produce a bounded, nonnegative descent coefficient.Nonnegativity supports valid gradient descent, and the bounded value is chosen to maintain stable training.Source: §3.2 · pp. 4–5
  • A dedicated TANGENTGNN predicts node-feature updates, which are projected to form the tangential component and scaled by a learned coefficient.Source: §3.2 · p. 5

4 Theoretical Properties of TANGO

TANGO combines energy descent with a tangential flow, and its analysis examines energy dissipation, evolution in flat landscapes, and the tangent direction’s convergence benefits. Under stated conditions energy is non-increasing, while the tangential component supports feature evolution and can be learned to match a Newton direction.

Energy dissipation

  • With α_G ≥ 0 and a nonzero energy gradient, TANGO’s energy is non-increasing along trajectories.The analysis assumes a connected graph and a twice-differentiable energy function bounded from below.Source: §4 · p. 5

Feature evolution in flat energy landscapes

  • When the energy gradient is zero but the tangential direction is nonzero, TANGO can still evolve features in flat energy landscapes.Unlike gradient flows, its tangential component permits feature evolution in these regions.Source: §4 · p. 5

Benefit of the tangent direction

  • In the 50-layer comparison, TANGO propagates information through the bottleneck more effectively than gradient flow.The paper presents this comparison as empirical validation of its theoretical results.Source: Figure 2 · p. 6, §4 · p. 6
  • Gradient descent struggles when the Hessian is ill-conditioned, a condition associated with graph bottlenecks and oversquashing.The paper relates this difficulty to poor information propagation between nodes.Source: §4 · p. 6
  • TANGO can learn a tangential direction and step size that make its combined update match the Newton direction, which offers quadratic convergence under the stated assumptions.The proposition assumes an invertible Hessian and β_G = 1.Source: §4 · p. 6

5 Experiments

Across the evaluated graph-learning benchmarks, TANGO improves its backbone performance and often outperforms strong baselines. The experiments cover synthetic graph-property tasks, five GNN benchmarks, peptide long-range prediction, and heterophilic node classification.

5 Experiments

  • The benchmark suite spans synthetic long-range graph-property prediction, peptide graphs, GNN datasets, and heterophilic node classification, with consistent downstream improvements over TANGO’s baseline models.The paper also compares against MPNN-based, diffusion-based, higher-order, and attention-based GNNs.Source: §5 · p. 7

5.1 Graph Property Prediction

  • All TANGO variants achieve the lowest (best) error across the three synthetic graph-property tasks, and TANGOGPS reduces Eccentricity error by over 1.2 points versus PH-DGN and over 2.0 versus SWAN.The reported result is mean test log10(MSE), with lower values better; TANGO also improves the GatedGCN backbone.Source: §5.1 · p. 7, Table 1 · p. 7

MPNNs

  • On graph-property prediction, TANGO consistently improves GatedGCN, showing that the method can strengthen a simple MPNN backbone on tasks requiring distant-node information.The tasks are graph diameter, single-source shortest paths, and node eccentricity on synthetic graphs.Source: §5.1 · p. 7

5.2 GNN Benchmarking from Dwivedi et al. [31]

  • Across five benchmarks, TANGO consistently improves its backbone performance and often outperforms other strong baselines.The benchmarks include ZINC-12k, MNIST, CIFAR-10, CLUSTER, and PATTERN; ZINC-12k uses MAE and the others use accuracy.Source: §5.2 · p. 7, Table 2 · p. 8

5.3 Long-Range Benchmark

  • On Peptides-func and Peptides-struct, TANGO outperforms standard MPNNs, transformer-based GNNs, DE-GNNs, and most Multi-hop GNNs.The comparison uses the long-range graph benchmark setting with a 500K parameter budget, and the authors report a broader comparison in Table 10.Source: §5.3 · p. 8

5.4 Heterophilic Node Classification

  • On heterophilic node-classification datasets, TANGO is competitive and often outperforms state-of-the-art methods, including on larger graphs and complex heterophilic scenarios.The reported comparisons include baselines from [36] [68].Source: §5.4 · p. 8

6 Related Work

Prior work connects GNNs to dynamical systems and energy-based modeling, but TANGO distinguishes itself by learning a task-specific graph energy and adding a tangential flow. It focuses on predictive objectives rather than generative modeling.

Deep GNNs and Dynamical Systems

  • Earlier dynamical-systems views of GNNs use continuous-depth dynamics, heat-equation updates, or gradient flows, whereas TANGO learns graph-adaptive, task-specific energy and adds a learnable tangential component.The paper contrasts this with prior gradient-flow interpretations of message passing.Source: §6 · p. 9

Learning Energy Functions in Neural Networks

  • Unlike generative energy-based models that capture data likelihood, TANGO’s learned energy is tailored to downstream prediction, with inference framed as descending toward minima aligned with correct outputs.The related work also distinguishes TANGO from approaches that assume a fixed or implicit energy function.Source: §6 · p. 9

7 Conclusions

The paper concludes that TANGO’s joint energy-descent and tangential dynamics support feature evolution in flat or ill-conditioned landscapes and strong benchmark performance. Its related-work discussion connects the design to oversquashing remedies and optimization methods that supplement gradient directions.

7 Conclusions

  • TANGO jointly models energy descent and tangential flow, allowing feature evolution in flat or ill-conditioned landscapes and relating the approach to mitigation of oversquashing.The conclusion reports strong performance across 15 synthetic and real-world benchmarks, outperforming message-passing, diffusion-based, and attention-based GNNs.Source: §7 · p. 9
  • The paper identifies higher-order differential operators for tangential flow and analysis or regularization of the learned energy landscape as future research directions.Source: §7 · p. 9

A Additional Related Work

  • Oversquashing remedies include graph rewiring, feature-dependent graph changes, global-attention models, and non-local dynamics, with some approaches facing scalability or computational-cost concerns.The related work notes that full attention has quadratic complexity and some dense propagation methods have increased computational demands.Source: A Additional Related Work · p. 18
  • TANGO’s energy-preserving tangential updates are presented as learnable corrective directions that complement gradient descent without explicitly approximating the Hessian.The discussion connects this motivation to quasi-Newton methods such as conjugate gradients and LBFGS.Source: A Additional Related Work · p. 18

B Proofs of Theoretical Results

The proofs establish energy dissipation and continued feature evolution in flat energy regions, and show conditions under which TANGO can learn a Newton direction with quadratic energy convergence.

Energy Dissipation

  • When α_G(H) is nonnegative, energy is non-increasing along TANGO trajectories, and it decreases when α_G(H) is positive.The energy-dissipation result assumes a nonzero energy gradient; the appendix assumes a connected graph and an energy function twice differentiable and bounded from below.Source: B Proofs of Theoretical Results · pp. 18–19

Feature Evolution in Flat Energy Landscapes

  • If the energy gradient is zero but the tangential direction is nonzero, TANGO can still evolve node features in a flat energy region.In this case the gradient term vanishes and the tangential flow remains.Source: B Proofs of Theoretical Results · p. 19

Convergence of Gradient Descent

  • Gradient descent on the energy has linear convergence, with a rate expressed using the maximal and minimal eigenvalues.For graph-Laplacian problems, λ_min is the first non-zero eigenvalue of the Hessian.Source: B Proofs of Theoretical Results · p. 19

Quadratic Convergence Direction

  • With β_G = 1 and an invertible Hessian, TANGO can learn a tangential direction and step size that match the Newton direction, yielding quadratic energy convergence.The stated construction assumes the combined direction D is zero.Source: B Proofs of Theoretical Results · pp. 19–20

C Complexity and Runtimes

TANGO’s cost comes from energy-gradient computation and a projected tangential flow; under linear-complexity message-passing subnetworks, its unrolled cost scales with steps, graph size, and feature dimension.

Complexity

  • For L unrolled steps, TANGO has complexity O(L · L_gnn · (|V| + |E|) · d), assuming linear-complexity message-passing subnetworks of equal depth.Here L_gnn is the number of layers in each subnetwork and d is the feature dimensionality.Source: C Complexity and Runtimes · p. 20
  • Each step computes the learned energy gradient and projects a separate message-passing tangential vector field onto the gradient’s orthogonal complement.The projection costs O(nd) per step; pooled-feature MLPs compute the scalar coefficients.Source: C Complexity and Runtimes · p. 20

Runtimes

  • Runtime comparisons use the Questions dataset with 8 layers and 256 channels on an NVIDIA RTX6000 Ada GPU, comparing TANGO backbones with baseline backbones and other methods.The table reports training runtimes in milliseconds per epoch.Source: C Complexity and Runtimes · p. 20, Table 4 · p. 20

D Experimental Details

The experimental details describe a synthetic oversquashing demonstration and evaluation protocols spanning graph-property prediction and the Dwivedi et al. graph benchmarks.

D Experimental Details

  • The experimental details specify NVIDIA RTX6000 Ada hardware and a broad comparison with classical, heterophily-specific, dynamical, transformer, higher-order, and state-space GNN baselines.The code is implemented in PyTorch and is stated to be publicly released upon acceptance.Source: D Experimental Details · pp. 20–21

D.1 Synthetic Example from Figure 2

  • On a Barbell graph, TANGO’s combined energy and tangential flows propagate an initially localized signal through all nodes, unlike the 50-step Dirichlet-energy gradient flow’s bottlenecked propagation.All node features start at zero except the leftmost node, initialized to one.Source: §D.1 · p. 21

D.2 Graph Property Prediction

  • Graph-property prediction uses 7,040 synthetic graphs for shortest paths, node eccentricity, and graph diameter, with 25–35 nodes per graph.The collection includes 5,120 training, 640 validation, and 1,280 test graphs; evaluation follows Gravina et al. [41].Source: §D.2 · p. 21

D.3 Graph Benchmarks from Dwivedi et al. [31]

  • The Dwivedi et al. benchmarks cover molecular regression, image-superpixel classification, and synthetic community or pattern tasks, using the official data splits.The suite includes ZINC-12k, MNIST, CIFAR-10 superpixels, CLUSTER, and PATTERN.Source: §D.3 · pp. 21–22

D.4 Long Range Graph Benchmark

The appendix describes peptide and heterophilic-node benchmarks, tuning procedures, and further comparisons; the reported comparisons show strong TANGO performance, similar to or better than recent methods on heterophilic tasks.

D.4 Long Range Graph Benchmark

  • The long-range graph benchmark uses 15,535 peptide graphs with about 2.3 million nodes for functional classification and structural regression.Peptides-func has 10 functional categories; Peptides-struct predicts physical and geometric attributes, with evaluation following Dwivedi et al. [30].Source: §D.4 · p. 22

D.5 Heterophilic Node Classification

  • The heterophilic evaluation covers Roman-Empire, Amazon-Ratings, Minesweeper, Tolokers, and Questions, using official splits and task-specific classification metrics.Accuracy is used for Roman-Empire and Amazon-Ratings; ROC-AUC is used for the other three tasks.Source: §D.5 · pp. 22–23

D.6 Hyperparameters

  • Hyperparameter tuning spans unrolled dynamics steps, feature dimension, shared ENERGYGNN and TANGENTGNN depth, and the forward-Euler step size.The grids cover graph-property prediction, long-range graph learning, graph benchmarks, and node classification.Source: §D.6 · p. 23, Table 6 · p. 23

E.1 Heterophilic Node Classification

  • Table 7 compares TANGO with recent methods on heterophilic node classification, where it performs similarly to or better than state-of-the-art methods.The table reports mean test scores and standard deviations across Platonov et al. splits.Source: §E.1 · p. 23, Table 7 · p. 24

E.2 Additional Comparisons

  • The broader comparisons in Tables 10 and 11 also show strong TANGO performance against recent methods.The appendix describes these comparisons as more comprehensive than the focused comparisons in Section 5.Source: §E.2 · p. 24

E.3 Ablation Study

The ablations compare TANGO with variants that remove energy-gradient structure or the tangential constraint, using representative datasets and two backbones. Both the learned energy gradient and the structured tangential update contribute to the reported downstream performance.

E.3 Ablation Study

  • The two controlled ablations use ZINC-12k, Roman-empire, Peptides-func, and Diameter with GatedGCN and GPS backbones.Original backbone performance is also reported for reference.Source: §E.3 · p. 24
  • TANGO consistently outperforms TANGO-NON-ENERGY, while both variants benefit from adding a tangential component.TANGO-NON-ENERGY replaces the energy-gradient term with intermediate backbone features, removing the guarantee that the direction is a gradient of a scalar energy.Source: §E.3 · pp. 24–25
  • TANGO-NON-TANGENT improves over the baseline backbone but performs worse than full TANGO when its added direction is not orthogonal to the energy gradient.The authors interpret this comparison as evidence for the tangential constraint and its contribution to using the learned energy function.Source: §E.3 · p. 25

MPNNs

  • The MPNNs results block lists score rows for methods including GAT, GCN, CO-GNN, and SAGE.Source: MPNNs · p. 27

Graph Transformers

  • The Graph Transformers results block lists score rows for Exphormer, GPS variants, GT, and other transformer-based methods.Source: Graph Transformers · p. 27

Heterophily-Designated GNNs

  • The Heterophily-Designated GNNs results block lists score rows for CPGNN, FAGCN, FSGNN, GBK-GNN, GloGNN, GPR-GNN, H2GCN, and JacobiConv.Source: Heterophily-Designated GNNs · p. 27

Graph SSMs

Ours

  • The Ours results block lists rows for TANGOGatedGCN and TANGOGPS.Source: Ours · p. 27
Loading 2508.05070v1…