PaperlayerView exact revision on arXiv

Source-linked AI summary

Graph is a Substrate Across Data Modalities

Ziming Li, Xiaoming Wu, Zehong Wang, Jiazheng Li, Yijun Tian, Jinhe Bi, Yunpu Ma, Yanfang Ye, Chuxu Zhang

arXiv:2601.22384v2cs.LGcs.AI

TL;DR

Graph structure is widely used but is typically learned separately for each task and discarded, preventing it from persisting as a reusable intermediate representation. G-Substrate organizes graphs as a shared substrate through a unified schema and interleaved role-based training, and experiments show consistent gains over task-isolated and naive multi-task methods. The framework's reuse of graph states also makes cross-domain bias propagation a deployment concern.

  • Problem

    Graph representations are usually built for individual objectives and discarded, motivating a way to preserve and accumulate graph structure across heterogeneous contexts.

  • Method

    G-Substrate uses a unified graph state space with consistent structural conventions and interleaved training that exposes the same graph to multiple functional roles.

  • Results

    G-Substrate consistently outperforms task-isolated and naive multi-task baselines across tasks from multiple domains and modalities.

  • Takeaways & Limitations

    Shared structural compatibility combined with role-based training supports graph representation reuse across heterogeneous tasks and modalities.

  • Takeaways & Limitations

    Reusing graph states may propagate systematic representation biases across domains, and the study does not examine how modality, domain, or role composition shapes representation formation.

Abstract

from arXiv · show

Graphs provide a natural representation of relational structure that arises across diverse domains. Despite this ubiquity, graph structure is typically learned in a modality- and task-isolated manner, where graph representations are constructed within individual task contexts and discarded thereafter. As a result, structural regularities across modalities and tasks are repeatedly reconstructed rather than accumulated at the level of intermediate graph representations. This motivates a representation-learning question: how should graph structure be organized so that it can persist and accumulate across heterogeneous modalities and tasks? We adopt a representation-centric perspective in which graph structure is treated as a structural substrate that persists across learning contexts. To instantiate this perspective, we propose G-Substrate, a graph substrate framework that organizes learning around shared graph structures. G-Substrate comprises two complementary mechanisms: a unified structural schema that ensures compatibility among graph representations across heterogeneous modalities and tasks, and an interleaved role-based training strategy that exposes the same graph structure to multiple functional roles during learning. Experiments across multiple domains, modalities, and tasks show that G-Substrate outperforms task-isolated and naive multi-task learning methods. The codebase, model, and datasets are available at https://github.com/zmli6/G-Substrate.

1. Introduction

Graphs recur across domains and tasks, but their structure is usually learned in isolated contexts and discarded rather than preserved. G-Substrate addresses this by treating graph structure as a shared substrate compatible across modalities and functional roles.

  • Motivation: Graphs represent relational information across computer vision, language, chemistry, and graph algorithmic tasks.Examples include scene graphs, event graphs, molecular graphs, connectivity, shortest paths, and structural reasoning.
  • Motivation: Existing systems commonly build graph representations for one objective and discard them after training, limiting reuse across tasks and modalities.Task-specific graph representations can serve as supervision targets while structurally similar patterns recur across different inputs.
  • Motivation: G-Substrate asks how graph structure can persist and accumulate across heterogeneous learning contexts without unifying task semantics.Its focus is aligning structural patterns that recur across domains.
  • Approach: G-Substrate uses a unified structural schema and interleaved role-based training to coordinate graph learning across modalities and functional roles.The schema establishes representation compatibility, while interleaving exposes the same graph to multiple roles during learning.
  • Results: Across multiple domains and modalities, G-Substrate consistently outperforms task-isolated and naive multi-task baselines.The schema and role-based interleaving are complementary, with their combination producing the strongest performance.

2. The G-Substrate Framework

G-Substrate treats graph structure as a reusable intermediate substrate rather than a task-bound artifact. It combines a unified structural schema with interleaved role-based training so graphs from heterogeneous contexts can remain compatible and be reused across functional roles.

  • 2.1. Perspective: Graph is a Structural Substrate: G-Substrate reframes graph structure as a reusable substrate that persists across domains and modalities instead of being reconstructed and discarded per task.The framework organizes learning contexts around shared graph structures.
  • 2.1. Perspective: Graph is a Structural Substrate: A graph is represented as structural triples G = {(u, r, v)}, preserving entity identities and typed relations while excluding task-specific objectives and execution logic.Optional attributes may annotate entities or edges without changing the relational structure.
  • 2.2. Structural Compatibility: A Unified Schema: The unified graph state space Gs constrains graphs to shared node identifiers, typed-edge conventions, and a common (u, r, v) format for cross-task comparability.Graphs from different modalities are mapped into this structured family, making recurring patterns such as hub-centered configurations comparable.
  • 2.3. Cross-task Reuse: Interleaved Role-based Training: Structural compatibility alone is insufficient, so interleaved role-based training reuses graphs across GENERATE and UNDERSTAND functions.Generation tasks construct or refine graphs, while understanding tasks operate on them for reasoning, prediction, or evaluation.
  • 2.3. Cross-task Reuse: Interleaved Role-based Training: Interleaving creates a graph trajectory G(0) → G(1) → · · · → G(N), allowing graphs to evolve across successive generation and understanding tasks.A generated graph can be reused as input to a subsequent understanding task rather than independently reconstructed.

3. Experiments

The experiments evaluate G-Substrate across four graph-learning settings and compare unified representations and role-based training with task-isolated and naive multi-task alternatives. Results show that shared structural alignment and cross-role reuse generally improve performance, with robustness to partial graph noise but sensitivity to fully corrupted structures.

  • Learning Settings and Tasks: The evaluation spans graph algorithmic reasoning, molecular graph description, scene graph generation, and event relation extraction across multiple domains and modalities.The tasks use accuracy, BLEU-4 and ROUGE-L, PCIs R@50, and F1-based metrics.
  • Main Results: On GAR, shortest-path accuracy rises from 41.46 to 48.59, while MGD reaches 51.53 BLEU-4 and 68.47 ROUGE-L.These comparisons are against G-Wiz for GAR and M-LLaMA for MGD.
  • Main Results: ERE F1 increases to 42.68, 40.91, and 25.15 on MA-T, MA-C, and HiE, while G-Substrate trails specialized systems on SGG and MA-S.On SGG, PGSG leads 26.9 to 25.38; on MA-S, LLMERE leads 54.30 to 52.20.
  • Schema–Training Interaction: Unified single-task training does not outperform naive single-task training, whereas unified multi-task training improves over its naive counterpart and benefits further from role-based reuse.The schema and interleaving mechanisms therefore show complementary effects when graphs are shared across tasks.
  • Effect of Schema Realization: Alternative schema serializations permit basic transfer but are less stable overall, with XML-style formats tending to underperform the proposed schema.The comparison uses identical multi-task training conditions and attributes the difference to attention on surface formatting versus relational semantics.
  • Robustness to Noisy Graph Extraction: Under 20% graph corruption, G-Substrate remains above clean NMT on MGD at 50.74 to 48.11 and ERE at 39.74 to 38.02, while complete corruption reverses gains.At 30% noise, three of four domains remain close to or above clean NMT; SGG is more sensitive.

4. Related Work

Related work shows that graphs are widely used across domains and that multi-task learning coordinates related tasks, but existing systems generally keep graph structure task-bound. G-Substrate instead treats graph structure as a persistent intermediate state shared across domains and modalities.

  • Graphs as a ubiquitous but task-bound tool: Graph-structured representations are standard across diverse domains, including perceptual and linguistic graph induction.Examples include scene graph generation from images and event graph extraction from text.
  • Graphs as a ubiquitous but task-bound tool: Existing systems typically construct graphs for a particular objective, optimize them within one pipeline, and discard them afterward.As a result, graph representations do not function as reusable state across heterogeneous learning contexts.
  • Multi-task learning: Multi-task learning coordinates learning across related tasks through shared training paradigms, unified architectures, and instruction- or prompt-based formulations.The related-work discussion situates G-Substrate within this broader coordination paradigm.
  • Graph as a unified substrate across modalities: G-Substrate reframes graph structure as a persistent intermediate state shared across domains and modalities rather than a task-bound interface.This design is intended to let structural knowledge accumulate and transfer across learning tasks.

5. Conclusion

G-Substrate addresses the task-isolated treatment of graph structure by combining a unified structural space with interleaved role-based training. Across heterogeneous settings, shared representations and role reuse produce the most consistent improvements, supporting persistent graph representations as a basis for structural reuse.

  • Conclusion: Graph structures are commonly optimized in isolated contexts and discarded, preventing them from serving as persistent intermediate representations.The paper attributes this limitation to a task-centric organization that treats intermediate structure as disposable.
  • Conclusion: G-Substrate combines a unified structural space for cross-task compatibility with interleaved role-based training that reuses the same graph across functional roles.The framework is designed around representation reuse across heterogeneous learning contexts.
  • Conclusion: The unified structural space improves results once multiple tasks share the representation, role-based interleaving amplifies those gains, and their combination yields the most consistent improvements.These findings support persistent graph representations as a driver of structural reuse across diverse learning contexts.

Impact Statement

The paper presents graph substrates as a reusable representation with potential benefits for data efficiency and generalization, while noting risks from cross-domain bias propagation and unanswered questions about heterogeneous supervision. Its structural analysis examines recurring motifs across four domains while allowing global scale to vary.

  • Potential impact: A reusable graph substrate may benefit data efficiency and generalization in systems operating over events, scenes, molecules, and algorithmic graphs.The statement identifies these benefits as potential rather than established outcomes of the impact analysis.
  • Risks and limitations: Reusing graph states can propagate systematic entity- or relation-representation biases across domains instead of keeping them task-local.The paper links responsible deployment to careful dataset composition, transparent graph construction, and cross-domain evaluation.
  • Risks and limitations: The work does not explicitly study how modality, domain, or role composition shapes representation formation, leaving supervision balancing as a future direction.It also proposes extending the representation-centric principle beyond graphs.
  • Structural analysis: Across four domains, the structural analysis treats each instance as relational tuples and measures coarse topology through AvgDeg, ASPL, TwoHop, and Star.The statistics are intended to establish recurring structural patterns rather than compare magnitudes or evaluate models.
  • Structural analysis: Local motifs such as two-hop dependencies and hub nodes recur across domains, whereas global structural scale varies in interpretable ways.Molecular graphs, for example, exhibit substantially larger ASPL consistent with chain-like or near-tree backbones.

A.3. Qualitative Evidence of Shared Structural Constraints Across Tasks

Qualitative comparisons show that recurring graph motifs across event and scene graphs preserve aligned constraint roles despite differing semantics. This supports using graph structure as a reusable intermediate substrate rather than a task-specific artifact.

  • Cross-domain interpretation: Motif identity is defined by topology rather than label semantics, so the comparison targets shared structural forms and constraint interpretations.The evidence does not claim semantic equivalence between labels or domains.
  • Two-hop chains: Two-hop chains use an intermediate node as a compositional bottleneck that constrains how two relations interact across event and scene graphs.The node’s meaning differs by domain, but its structural role remains invariant.
  • Hub / star motifs: Hub motifs use a shared anchor to coordinate multiple relations and enforce global consistency across dependent nodes.This role remains consistent across temporal, spatial, and part-of relations despite differing semantics.
  • Implication: Together, recurring motifs provide reusable relational organization that abstracts away task- and modality-specific semantics while preserving constraint-level meaning.This is the representation-centric role assigned to graph structure in G-Substrate.
  • Shared representation: One practical instantiation represents graphs as uniquely identified entities with typed, directed relations over ordered entity pairs, while optional attributes remain auxiliary.Structural identity is determined by relational connectivity, enabling the same form to support different task roles.
  • Shared representation: The structural form is reused across tasks, with task differences expressed through prompts, supervision, and evaluation rather than graph-structure changes.G-Substrate does not depend on one canonical schema, serialization format, or internal encoding.

D. Training Paradigm Definitions

The training paradigms isolate two design choices: whether graph states obey a unified structural schema and whether they are exposed to multiple functional roles. G-Substrate combines both choices, unlike the comparison baselines.

  • Shared configuration: All paradigms use the same backbone, optimizer, schedule, data mixture, and training budget, differing only in graph-state constraints and role exposure.This isolates representational constraints and training organization as the intended comparison dimensions.
  • Task-isolated baselines: Naive single-task training constructs, optimizes, and consumes task-specific graph states under one functional role without cross-role exposure.Each batch contains examples from a single task.
  • Task-isolated baselines: Unified single-task training applies the common schema but keeps tasks isolated, so graphs receive no cross-task reuse pressure.Learning signals remain task-specific despite shared structural admissibility constraints.
  • Multi-task baselines: Naive multi-task training jointly samples tasks but retains native graph formats and task-specific roles, without structural alignment or cross-role reuse.Shared parameter updates do not make graph states reusable across originating tasks.
  • Multi-task baselines: Unified multi-task training aligns graph representations through a common schema but does not expose graphs to different functional roles.It establishes structural compatibility without cross-role reuse.
  • Interleaved training: Naive multi-task plus interleave reuses graph states across roles while retaining heterogeneous task-specific formats and no unified admissibility constraint.It tests role interleaving without schema-level alignment.
  • Full framework: G-Substrate combines unified structural admissibility with explicit reuse of graph states under multiple functional roles during training.The combination pressures representations to remain compatible and reusable across heterogeneous contexts.

F. Detailed Experimental Results

Across detailed domain and task breakdowns, performance depends on combining structural alignment with cross-role reuse rather than on task mixing or schema standardization alone. The pattern persists in a transfer study using a different vision–language backbone.

  • Main detailed results: The full G-Substrate framework improves over naive and unified multi-task baselines, with the largest gains in multi-step relational composition tasks.Highlighted examples include shortest-path reasoning, rare scene-graph relations, and event substructure modeling.
  • Main detailed results: Unified schema alone can reduce performance in task-isolated settings but becomes beneficial under multi-task learning when graph states span multiple contexts.The detailed results therefore separate schema effects from the effects of cross-context reuse.
  • Main detailed results: Naive interleaving without schema-level alignment yields only limited and unstable improvements.The reported pattern attributes the stronger outcome to structurally admissible states being reused across heterogeneous roles, not task mixing alone.
  • Transfer robustness: The transfer study repeats selected comparisons with InternVL3 5-2B-HF under the same recipe, data composition, and evaluation protocol.It compares task-isolated training, naive multi-task learning, G-Substrate, and component ablations.
  • Transfer robustness: The alternative-backbone results reproduce the main pattern: schema-only alignment is not consistently better in isolation, while unified representations help under multi-task training.This mirrors the behavior observed with the primary backbone.
  • Transfer robustness: Consistency across two architecturally distinct vision–language backbones supports the framework’s generality beyond a specific model family.The authors attribute the improvements to how relational structure is represented and reused during training.

H.1. Tasks over Structured Graphs with LLMs and VLMs

Prior LLM, VLM, multi-task, and graph-foundation-model approaches generally treat graphs as task-bounded objects or coordinate learning through parameters and objectives. G-Substrate instead organizes intermediate graph states for structural compatibility and reuse across task roles.

  • Structured graph tasks: LLM- and VLM-based graph methods address algorithmic and structured-input tasks through serialization, prompting, or graph-aware tokenization tied to current objectives.Examples include shortest path, connectivity, traversal, and combinatorial queries.
  • Task-specific graph learning: Existing graph reasoning and generation systems generally optimize graph representations for one task rather than requiring compatibility with other tasks.Graphs function as task-specific inputs, outputs, or final prediction targets.
  • Multi-task and multi-modal learning: Multi-task and multi-modal methods primarily coordinate learning through shared parameters, losses, data scheduling, or curricula.Graph states are usually reused only implicitly through shared parameters rather than explicitly across functional roles.
  • Graph foundation models: Graph foundation models pursue generalization through large-scale pretraining, architectural unification, and broad task coverage.G-Substrate addresses a complementary axis centered on the organization of intermediate graph states.
  • G-Substrate perspective: G-Substrate explicitly enforces structural compatibility and cross-task reuse so graph states remain admissible under heterogeneous task roles.Its focus is intermediate-state generalization rather than parameter or architecture generalization alone.

I. Comparison with Gradient-Balancing Multi-Task Baselines

G-Substrate addresses representational sharing across heterogeneous tasks more effectively than gradient-balancing multi-task learning. Its advantage comes from explicit graph reuse rather than loss reweighting alone.

  • Gradient-balancing comparison: GradNorm improves NMT on GAR and MGD but hurts ERE by −2.93.Near-zero weighting of ERE after faster convergence illustrates a limitation of convergence-based reweighting under heterogeneous task difficulty.
  • Gradient-balancing comparison: G-Substrate outperforms NMT+GradNorm on all four domains without gradient balancing.This indicates that representational sharing is the dominant bottleneck in this setting.
  • Gradient-balancing comparison: Combining G-Substrate with GradNorm produces mixed effects: MGD +0.96, SGG +1.03, and ERE −1.98.Gradient reweighting can interfere with the balanced cross-role exposure used by G-Substrate.
  • Gradient-balancing comparison: Gradient balancing and explicit representation reuse address complementary but distinct bottlenecks.Gradient balancing is not a substitute for reusing relational structure across roles.

J. Robustness to Noisy Graph Extraction

G-Substrate remains effective under partial structural noise, with gradual degradation rather than catastrophic failure. Robustness varies by domain, and scene graph generation is especially sensitive to perturbations.

  • Noise robustness: Performance degrades gradually with no catastrophic failure as perturbed edges increase from 0% to 30%.The evaluation reports averaged accuracy for GAR, BLEU-4 for MGD, PCIs R@50 for SGG, and macro-averaged F1 for ERE.
  • Noise robustness: At 20% noise, G-Substrate exceeds clean NMT on MGD at 50.74 vs. 48.11 and ERE at 39.74 vs. 38.02.It remains competitive on GAR at 92.10 vs. 93.01 under the same noise level.
  • Noise robustness: SGG is more sensitive to noise and falls below clean NMT at all noise levels.The passage attributes this sensitivity partly to scene graphs being structurally compact, averaging 1.5 edges per relation.
  • Noise robustness: At 30% noise, three of four domains remain close to or above clean NMT levels.Partial noise therefore preserves useful performance across most evaluated domains.
Loading 2601.22384v2…