PaperlayerView exact revision on arXiv

Source-linked AI summary

TopoAgent: A Structure-Aware Perception-to-Reasoning Framework for Diagram-to-Graph Topology Extraction with Large Vision-Language Models

Bangwei Guo, Xujiang Zhao, Yanchi Liu, Wei Cheng, Shengyu Chen, Dongyue Li, Masaharu Morimoto, Takayuki Kuroda, Dimitris Metaxas, Haifeng Chen

arXiv:2608.28701v1cs.CV

TL;DR

Diagram-to-graph topology extraction remains challenging because it combines fine-grained perceptual grounding with globally consistent graph reasoning. The paper introduces the human-verified TopoBench-180 benchmark and TopoAgent, a staged structure-aware framework, which outperforms strong VLM and visual-reasoning baselines, especially on edge extraction.

  • Problem

    Diagram-to-graph topology extraction is underexplored and requires precise grounding plus globally consistent node–edge reasoning beyond query answering or coarse descriptions.

  • Method

    TopoAgent progressively constructs graphs through grounded perception, global structural inference, canonical node grounding, node-centric relation reasoning, and topological consistency enforcement.

  • Results

    TopoAgent consistently outperforms strong VLM baselines and recent visual reasoning frameworks on TopoBench-180, especially for edge extraction.

  • Takeaways & Limitations

    TopoBench-180 and TopoAgent establish a benchmark and structure-aware framework for topology-aware diagram understanding.

  • Takeaways & Limitations

    TopoBench-180 covers only Web-style and Network-style diagrams, so its scale and coverage remain limited relative to real-world technical diagram diversity.

Abstract

from arXiv · show

Diagram-to-graph topology extraction aims to extract a graph of entities and their connections from a structural diagram. This task remains challenging for current vision-language models because it requires both fine-grained perceptual grounding and topology-aware reasoning with global consistency. We present TopoBench-180, a human-verified benchmark for diagram-to-graph topology extraction, and TopoAgent, a structure-aware perception-to-reasoning framework for reliable topology extraction using large vision-language models. TopoBench-180 contains 180 structural diagrams spanning Web-style and Network-style categories, paired with canonical graph annotations. TopoAgent progressively extracts the target graph by combining grounded perception, global structural priors, canonical node inventory construction, node-centric local-to-global relation reasoning, and topological consistency enforcement. Experiments on TopoBench-180 show that TopoAgent outperforms strong vision-language model baselines and recent visual reasoning frameworks, especially on edge extraction. More broadly, this work fills an important gap in multimodal structured understanding by establishing a benchmark and framework for diagram-to-graph topology extraction. The benchmark and associated resources will be publicly released at https://huggingface.co/datasets/WayneGuo0011/TopoBench-180.

1 Introduction

Diagram-to-graph topology extraction remains underexplored because it demands precise visual grounding and globally consistent graph reasoning. The paper addresses this gap with TopoBench-180 and TopoAgent, a structure-aware framework that improves topology extraction.

  • Diagram-to-graph extraction requires structured graphs rather than query-specific answers or coarse textual descriptions.
  • Fine-grained grounding is difficult because labels, geometric primitives, and thin connectors are interleaved across multi-scale layouts.
  • Global reasoning must maintain consistent node–edge assignments because local errors can produce missing edges, hallucinated nodes, and inconsistent graphs.
  • TopoBench-180 provides a human-verified benchmark with canonical graph annotations for rigorous topology-extraction evaluation.
  • TopoAgent combines grounded region summaries with global attribute inference, canonical node grounding, node-centric relation reasoning, and topological consistency enforcement.
  • TopoAgent consistently outperforms strong VLM baselines and recent visual reasoning frameworks on TopoBench-180.

2 Related Work

Existing structured-visual benchmarks and reasoning methods address related capabilities but do not directly evaluate full diagram-to-graph topology extraction. The paper identifies precise grounding, consistent node–edge assignment, and graph-level constraints as jointly necessary.

  • AI2D, AI2D-RST, MMBench, ChartMuseum, and MMMU primarily evaluate question answering, caption generation, or general multimodal reasoning.
  • Spatial-understanding methods add bounding boxes, segmentation masks, or OCR regions, but these techniques address spatial cues rather than the complete topology-extraction task.
  • Chain-of-thought prompting and reinforcement learning improve structured reasoning paradigms, yet they are not designed for diagram topology extraction.
  • Topology extraction requires precise entity grounding, consistent node–edge assignment, and graph-level structural constraints to be handled jointly.

3 TopoBench-180

TopoBench-180 is built from diverse public and internal diagram sources through staged quality gating and human-guided canonicalization. The resulting benchmark contains 180 high-quality diagram–graph pairs spanning Web-style and Network-style diagrams.

  • TopoBench-180 combines existing public datasets, curated web collections, and internal diagram sources.
  • Quality gating uses content-based deduplication, VLM-driven pruning, and annotator cross-audit to retain structurally valid candidates.
  • Draft graph annotations are generated under a unified schema, then manually verified and refined through topology calibration and human-guided canonicalization.
  • The collection includes relational diagrams such as food webs and biological interaction graphs, alongside network or system diagrams with complex connectivity.
  • The initial pool contains over 6,000 raw images gathered from AI2D, Wikipedia, arXiv, Google Image Search, and internal collections.
  • The final benchmark contains 180 high-quality diagram–graph pairs, including 54 Web-style and 126 Network-style diagrams.

4 Method

TopoAgent progressively converts a structural diagram into a canonical topology graph through grounded perception, symbolic node construction, node-centric relation reasoning, and explicit consistency enforcement.

  • Problem formulation: TopoAgent represents diagram-to-graph extraction as constructing a canonical graph with diagram entities as nodes and connections as directed edges.Bidirectional or undirected connections are represented by both directed edge tuples.
  • Framework overview: The framework combines five stages: perception and grounding, global figure attribute inference, canonical node inventory construction, node-centric relation reasoning, and topological consistency enforcement.The stages progressively transform local visual evidence into a globally coherent graph.
  • Perception and grounding: Perception decomposes diagrams into candidate regions, aligns them with OCR text, and produces structured summaries containing labels, geometry, and connection primitives.Region decomposition uses initialization, splitting, and merging, while summaries provide grounded evidence for later topology reasoning.
  • Global figure attribute inference: Global attribute inference predicts diagram semantics, link directionality, and link density as structural priors for downstream relation synthesis.The predicted categories include web, network, or structural_diagram; one_direction or bi_direction; and single_link or multi_link.
  • Canonical node inventory construction: Canonical node inventory construction consolidates grounded visual and textual evidence into valid entities, while identifying discarded text regions and summarizing legends.This creates a stable symbolic scaffold that separates entity grounding from downstream relation reasoning.
  • Node-centric local-to-global relation reasoning: Node-centric reasoning decomposes full-graph extraction into N node-anchored subproblems, preserving global context while focusing locally on subtle connections.Aggregated node-local predictions form a draft edge set before topological consistency enforcement.
  • Topological consistency enforcement: The final consistency stage refines the draft graph under explicit constraints, including directionality consistency, without revisiting the raw image.This avoids additional visual hallucinations during verification and resolves unsupported reciprocal edges conservatively.

5 Experiments

Experiments evaluate TopoAgent against closed-source and open-source VLMs and visual reasoning frameworks on Web-style and Network-style diagrams. TopoAgent achieves the strongest topology extraction performance, with especially clear gains in edge-level metrics on Network-style diagrams.

  • Experimental setup: The evaluation compares TopoAgent with GPT-4o, GPT-4o-mini, multiple open-source VLMs, and recent visual reasoning frameworks.The baselines cover frontier general-purpose VLMs and structured reasoning systems.
  • Experimental setup: TopoBench-180 contains 54 Web-style and 126 Network-style diagrams, enabling evaluation of directed relation extraction and symmetric connectivity reasoning.Metrics include Node F1 and directed Edge Precision, Recall, and F1 after label normalization.
  • Main results: TopoAgent achieves the strongest topology extraction performance across both subsets, with the clearest gains appearing in edge-level metrics.Across methods, node extraction is generally easier than directed edge recovery.
  • Web-style diagrams: 95.6 Node F1 and 45.8 Edge F1 are achieved by TopoAgent on Web-style diagrams, including Edge F1 gains over Qwen2.5-VL-Instruct (72B) and Speculative Verdict.TopoAgent records the highest Edge Recall at 49.9, while Qwen2.5-VL-Instruct (72B) records the highest Edge Precision at 47.5.
  • Network-style diagrams: 58.1 Edge F1 is achieved by TopoAgent on Network-style diagrams, exceeding GPT-4o by 4.6 points, Chain-of-Region by 8.8 points, and Qwen2.5-VL-Instruct (72B) by 10.6 points.TopoAgent simultaneously achieves the highest precision and recall on this subset.
  • Interpretation: Network-style diagrams show larger gains, consistent with the stronger role of topological consistency enforcement in resolving ambiguous bidirectional or undirected connectivity.The reported overall pattern links perception and grounding with topology reasoning across direction-sensitive and consistency-demanding settings.

6 Ablation

Ablations show that every major TopoAgent component contributes to Edge F1 on both diagram subsets, while node-centric reasoning and final self-checking produce the largest drops when removed.

  • Component contributions: Removing any major component reduces Edge F1 on both Web-style and Network-style subsets.The evaluated components include perceptual summarization, global attribute inference, node-centric reasoning, and topological consistency enforcement.
  • Component contributions: Removing region-level perceptual summarization substantially degrades both Node F1 and Edge F1, underscoring the importance of localized perceptual evidence for node grounding.This remains necessary even for strong vision-language models.
  • Component contributions: Global figure attribute inference makes a smaller but consistent contribution by narrowing the reasoning space for downstream edge prediction.Its role is described as a coarse structural guide rather than a substitute for fine-grained grounding or relation reasoning.
  • Critical components: The largest performance drops occur when node-centric reasoning or final self-checking is removed.Without node-centric reasoning, Edge Recall declines sharply, especially on Web-style diagrams; removing self-check reduces both Node F1 and Edge F1.
  • Critical components: Self-checking improves graph-level coherence by filtering invalid predictions that local relation predictions alone cannot guarantee to be globally valid.The ablation supports complementary contributions from local grounding, relation coverage, and final graph consistency.

7 Conclusion

TopoAgent and TopoBench-180 jointly support structure-aware diagram-to-graph topology extraction. The staged design improves extraction over strong baselines, particularly for edges.

  • TopoAgent progressively extracts graph structure through grounded perception, structural priors, canonical node construction, local-to-global relation reasoning, and consistency enforcement.
  • TopoBench-180 provides the benchmark used to evaluate the framework on diagram-to-graph topology extraction.
  • TopoAgent improves topology extraction over strong vision-language model baselines and recent visual reasoning frameworks, especially on edge extraction.

8 Limitations

The paper identifies limitations in benchmark coverage, annotation scalability, model reliability, and inference efficiency. These constraints bound the diversity, growth, robustness, and practical deployment of the current framework.

  • Benchmark coverage: TopoBench-180 does not cover the full diversity of structural diagrams found in real-world technical documents.Its scale and coverage remain limited relative to the range of diagram formats encountered in practice.
  • Annotation cost: Topology annotation is labor-intensive because canonical graphs require exhaustive node, connector, and directionality verification.This limits how quickly larger and more diverse benchmarks can be constructed.
  • Model limitations: Prompting general-purpose vision-language models leaves TopoAgent vulnerable to hallucinated, missing, or incorrect edges.The stated failure modes arise when relations are not fully grounded or fine-grained perception fails.
  • Efficiency and scalability: The multi-stage pipeline improves extraction quality but increases inference complexity, latency, and computational cost.These costs are especially relevant for larger or more visually complex diagrams.

B.6 Dataset Complexity Analysis

TopoBench-180 has distinct Web-style and Network-style complexity profiles. Web-style diagrams are generally smaller and sparser, whereas Network-style diagrams are denser and pose greater completeness and grounding challenges.

  • TopoBench-180 complexity analysis reports node count, relation count, and unique-pair density across Web-style and Network-style diagrams.Figure 6 presents the corresponding distributions.
  • Unique-pair density is computed as |E|/(|V|(|V| −1)) for directed relations.
  • Web-style diagrams: Web-style diagrams generally contain fewer nodes and relations and have lower density.They remain sensitive to relation direction, local layout cues, and textual semantics.
  • Network-style diagrams: Network-style diagrams span wider node and relation counts and exhibit higher average density.Their density makes topology completeness and consistent node grounding more challenging.
  • The distributions indicate that diagram difficulty is not determined by node count alone, but also by edge density and structural connectivity.

C Implementation Details of TopoAgent

TopoAgent uses a staged perception pipeline that decomposes diagrams into grounded visual-textual regions before topology reasoning. Region initialization, splitting, merging, OCR grounding, and local/global summarization create evidence for subsequent graph extraction.

  • Perception and OCR grounding: The perception stage partitions dense diagrams into localized visual regions and pairs them with grounded textual evidence for node and relation reasoning.This creates an interface between low-level appearance and higher-level topology inference.
  • Region initialization: Initial regions are formed by Otsu-thresholded connected components from foreground and background maps, then merged with label offsets to avoid collisions.The combined map captures both dark-on-light and light-on-dark structures.
  • Region splitting: Region splitting separates composite shapes using structured geometric detectors or connectivity-based morphological operations that remove narrow connections.Structured splitting targets rectangles, ellipses, and line segments, while unstructured splitting uses distance transforms and elliptical opening.
  • Region merging: Region merging combines OCR-associated and visually related fragments, then applies agglomerative clustering until the region count reaches a predefined budget B.Structured merging uses OCR boxes and visual heuristics; unstructured merging uses centroid distances.
  • OCR grounding and summarization: Each final region is paired with overlapping or nearby OCR spans and optional geometric metadata, producing grounded units for summarization, node extraction, and relation reasoning.Local summaries describe only visible labels and structural cues, while global summaries provide figure-level node and legend context.
  • Topology reasoning inputs: The later reasoning setup classifies figure semantics, link directionality, and link density before constructing a canonical node inventory.These attributes provide global context for downstream topology extraction.

D Other Implementation Details

The implementation evaluates topology with normalized node matching and exact directed-edge matching under macro-averaged metrics. Experiments compare TopoAgent with VLM baselines and visual reasoning systems across model backbones and controlled evidence settings.

  • Evaluation: Topology extraction is evaluated with Node F1 and directed Edge Precision, Recall, and F1 using normalized node identities and exact endpoint-direction matches.Undirected or bidirectional gold connections are represented as two directed edges, and metrics are macro-averaged over diagrams.
  • Baselines: All methods use the same JSON schema, while baselines receive topology-aware structured prompts and are run through official APIs, released checkpoints, or official codebases.Unless otherwise specified, baseline temperature is 0 and inference is single-pass.
  • Backbone generalization: TopoAgent is instantiated with GPT-4o by default and additionally evaluated with Qwen2.5-VL-72B and Qwen2.5-VL-7B for backbone generalization.The framework is tested across Web-style and Network-style subsets.
  • Results: 43.7 Edge F1 is achieved by Qwen2.5-VL-7B with TopoAgent on the challenging Network-style subset, compared with 29.6 for the corresponding baseline.The reported gain is especially pronounced for the smaller model.
  • Controlled comparison: TopoAgent retains higher Edge Recall and Edge F1 than a GPT-4o single-pass baseline augmented with the same OCR results and region crops.The comparison attributes the remaining advantage to node-centric relation reasoning and global topology consistency mechanisms, not only additional perceptual evidence.

E.3 Oracle-Node Edge Extraction

Oracle-node analysis separates relation reasoning from node extraction by supplying gold node sets. TopoAgent remains stronger than GPT-4o in both standard and oracle-node settings, indicating gains beyond node grounding.

  • Oracle-node setup: Oracle-node evaluation provides each method with the gold node set and measures only edge extraction, removing node detection and name-matching errors as confounders.This isolates relation-prediction performance on Web-style and Network-style diagrams.
  • Oracle-node results: 49.5 Edge F1 is reached by GPT-4o on Web-style diagrams with oracle nodes, up from 32.8 under standard evaluation.The improvement shows that node errors substantially affect single-pass relation extraction.
  • Oracle-node results: 57.9 Edge F1 is reached by TopoAgent on Web-style diagrams with oracle nodes, up from 45.8 under standard evaluation.Both methods benefit from oracle nodes, but TopoAgent remains stronger.
  • Interpretation: TopoAgent consistently outperforms GPT-4o under both standard and oracle-node settings, supporting gains from stronger relation reasoning as well as node grounding.The analysis also identifies joint node grounding and relation prediction as challenging for single-pass inference.

E.4 Error-Type Analysis

Error analysis shows different failure patterns across Web-style and Network-style diagrams, while qualitative cases reveal strong performance on simple structures and degradation under clutter, crossings, ambiguity, and difficult text. The framework also incurs additional inference cost from its multi-stage design.

  • Error patterns: Web-style errors are relatively balanced across direction, hallucination, and missed-edge categories, whereas Network-style errors are dominated by missed edges and node-name mismatches.Missed edges account for 56.0% and node-name mismatches for 25.2% of Network-style errors.
  • Error patterns: The error distribution suggests that Web-style diagrams stress local relation interpretation, while Network-style diagrams stress topology completeness and node grounding.These patterns are consistent with the oracle-node analysis.
  • Inference efficiency: The multi-stage design introduces additional inference cost compared with single-pass inference, largely because of repeated node-centric reasoning.The paper suggests more efficient model routing as a possible way to reduce this overhead.
  • Qualitative cases: TopoAgent achieves Node F1 = 1.0 and Edge F1 = 1.0 on examples with few nodes, unique labels, simple structure, and limited irrelevant visual content.These are successful illustrative cases rather than aggregate benchmark results.
  • Qualitative cases: Edge F1 falls to 31.3 and 45.0 in challenging Web-style and Network-style examples containing many nodes, crossing or long arrows, ambiguous directionality, difficult text, or clutter.These factors make entity grounding and directed relation extraction more difficult.
Loading 2608.28701v1…