PaperlayerView exact revision on arXiv

Source-linked AI summary

Knossos and Ariadne: Benchmarking and Learning Complete Diagram Topology Extraction with Vision-Language Models

Bangwei Guo, Xujiang Zhao, Shengyu Chen, Yanchi Liu, Wei Cheng, Xi Zhu, Guoning Zhang, Dimitris N. Metaxas, Haifeng Chen

arXiv:2610.04721v1cs.CVcs.AI

TL;DR

Complete diagram-to-graph extraction remains underexplored because existing VLM tasks often assess selected entities or relations rather than the full topology. The paper introduces Knossos for supervised training and evaluation and Ariadne for decomposed node and edge prediction. Ariadne-Qwen reaches 91.6% Edge F1 on Knossos, and the paper identifies remaining synthetic-data and sequential-prediction limitations.

  • Problem

    Existing diagram-understanding evaluations often assess selected entities or relations rather than requiring extraction of the complete graph topology.

    Source: §1 · p. 1

  • Method

    The paper introduces Knossos, a symbolically generated benchmark with aligned complete-topology annotations, and Ariadne, which predicts node inventories followed by source-conditioned edges.

    Source: §1 · p. 2, §2.3 · p. 3

  • Results

    91.6% Edge F1 for Ariadne-Qwen versus 81.1% for GPT-5.6 on the 1,200-image Knossos test set, with the margin increasing from 2.0 points on simple diagrams to 12.2 points on difficult diagrams.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Knossos and Ariadne provide a practical framework for studying diagram-to-graph topology extraction, with source-conditioned decomposition particularly improving smaller task-adapted VLMs.

    Source: §5 · pp. 9–10

    Knossos is synthetic and uses finite domain-specific vocabularies and predefined graph grammars, leaving generalization to unseen entity types, relation semantics, and topology patterns open.

    Source: §5 · p. 9

Abstract

from arXiv · show

Structural diagrams are widely used to represent complex systems and relational information across scientific, engineering, procedural, and spatial domains. Recent vision-language models (VLMs) have become increasingly capable of recognizing diagram elements and reasoning about their content, while complete diagram topology extraction remains comparatively underexplored. In this paper, we study diagram-to-graph topology extraction: extracting all diagram entities and the complete relations among them. To enable large-scale supervised training and systematic evaluation of this task, we introduce Knossos, a benchmark of 19,200 diagrams across six diverse domains, with 245,179 nodes and 439,740 edges. Its symbolic generation process provides exact alignment between rendered diagrams and annotations of complete topology, relation types, and connector geometry. To address the modeling challenge of complete topology extraction, we also present Ariadne, a structured framework that decomposes the task into node inventory extraction and source-conditioned edge prediction. Extensive experiments show that training on Knossos substantially improves complete topology extraction in smaller open-source VLMs. Ariadne further improves over one-step extraction under matched supervision, demonstrating the additional benefit of structured decomposition. It achieves the highest average Edge F1 among the evaluated methods on Knossos, while both backbone variants also improve over their unadapted counterparts on the real-world external benchmark. Code and benchmark are available at https://github.com/bangwayne/knossos_Ariadne_Public.

1 INTRODUCTION

Existing VLM diagram tasks emphasize recognition and selected-relation reasoning, but do not require recovering every entity and relation. The paper studies complete diagram-to-graph extraction and introduces Knossos for supervision and Ariadne for structured prediction.

1 INTRODUCTION

  • Structural diagrams encode complex relational information across scientific, engineering, educational, and technical domains, and VLMs have advanced in recognizing diagram elements and reasoning about content.Prior work spans question answering, visual-graph reasoning, and specialized engineering-diagram analysis (Masry et al., 2022; Zhu et al., 2025; Zhao et al., 2025; Li et al., 2025a).Source: §1 · p. 1
  • Complete topology extraction requires recovering all diagram entities and relations, unlike evaluations that sample entities or relations and may miss omissions, reversals, or hallucinations.The extracted textual graph can then be analyzed directly for structure-centric question answering and reasoning.Source: §1 · pp. 1–2
  • Large-scale complete-topology supervision is scarce because exhaustively labeling entities and relations is demanding, while dense diagrams make complete graph generation difficult for VLMs.Longer graph serializations can become harder to generate consistently as diagram complexity grows.Source: §1 · p. 2
  • Knossos supplies large-scale supervised training and evaluation, while Ariadne extracts a node inventory before making source-group edge predictions assembled into complete topology.Knossos uses symbolic topology specifications to align diagrams with complete annotations by construction.Source: §1 · p. 2

2 TASK FORMULATION AND THE KNOSSOS BENCHMARK

The paper formulates diagram-to-graph extraction as recovering nodes and directed edges, including typed relations when defined. Knossos provides a synthetic, multi-domain benchmark with complete topology and geometry annotations, built through aligned symbolic generation and controlled variation.

2.1 DIAGRAM-TO-GRAPH TOPOLOGY EXTRACTION

  • A diagram graph comprises nodes with local identifiers and visible labels, plus directed source–target edges that may carry relation types and preserve multiple relations between the same endpoints.The edge set is a multiset, so distinct typed edges sharing endpoints remain represented.Source: §2.1 · p. 2

2.2 KNOSSOS: MULTI-DOMAIN SYNTHETIC DIAGRAM BENCHMARK

  • Knossos contains 19,200 diagrams across six domains, with 245,179 node instances, 439,740 edge instances, and complete graph, relation-type, and geometry annotations.The frozen release includes 18,000 training images and 1,200 held-out test images.Source: §2.2 · pp. 2–3, Figure 1 · p. 3

2.3 SYMBOLIC GENERATION PIPELINE

  • The generator fixes symbolic topology before assigning node positions and routing connectors, then renders the image and serializes annotations from the same finalized objects.This shared representation keeps rendered diagrams and annotations aligned by construction.Source: §2.3 · p. 3

2.4 STRUCTURAL AND VISUAL DIVERSITY

  • Parameterized domain-specific template families provide hierarchical, clustered, branching, layered, and spatial structures, with seeded controls varying graph size, density, depth, and connectivity.Harder samples generally include more nodes and edges, more layers and cross-links, and longer, more intersecting connector paths.Source: §2.4 · p. 4
  • Presentation varies through label-matched visual assets and seeded choices such as orientation, label casing, backgrounds, connector appearance, and legend placement.Visible labels are renderer-drawn, so this variation does not change node identity or graph topology.Source: §2.4 · p. 4

2.5 QUALITY CONTROL AND DATASET RELEASE

  • Structural validation and VLM-based visual screening filter samples before stratified selection forms the frozen release.The release has 3,000 training and 200 test examples per domain.Source: §2.5 · p. 4

3 ARIADNE

Ariadne separates node inventory prediction from source-conditioned edge prediction, merging group outputs into a complete graph. It also predicts geometric grounding for nodes and connectors without changing the graph topology.

3.1 NODE-TO-EDGE TOPOLOGY EXTRACTION

  • Ariadne partitions the predicted node inventory into source groups, predicts typed edges originating from each group, and deterministically merges the group-wise edge multisets.Each directed edge belongs to exactly one source group because it has a unique source.Source: §3.1 · pp. 4–5
  • Replacing one long graph serialization with shorter source-conditioned generations removes cross-group ordering ambiguity under the stated serialization setting.Under a position-sensitive decoding assumption, shorter generations may also reduce later-position error accumulation.Source: §3.1 · p. 5

3.2 TWO-STAGE TOPOLOGY PREDICTION

  • Stage 1 predicts node labels and bounding boxes without predicting edges or connectors, and the predicted inventory—not ground truth—is passed to Stage 2 at test time..Source: §3.2 · p. 5
  • Each Stage 2 query includes the diagram, all predicted nodes as candidate targets, and a group of active sources whose outgoing edges alone may be emitted.Relation types are supplied only when defined by the diagram legend; otherwise the type is left empty.Source: §3.2 · p. 5
  • Ariadne validates and merges source-group predictions while preserving direction, reciprocal representations for undirected or bidirectional connectors, and distinct parallel typed relations.An empty CHECK NODE block denotes a source with no outgoing edge.Source: §3.2 · p. 5

3.3 GEOMETRIC GROUNDING

  • Nodes receive predicted bounding boxes, while each fixed predicted edge receives a five-point trace sampled along its visible connector from source to target.Reciprocal edges use the same connector path in reverse order; traces provide auxiliary grounding and do not modify topology.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

Experiments evaluate node and edge extraction on Knossos and TopoBench-180, alongside ablations of staged extraction and diagram structure. Ariadne performs strongly on Knossos and improves edge extraction for smaller adapted models, while diagram complexity remains challenging.

4.1 EXPERIMENTAL SETUP

  • A frozen 1,200-image Knossos test set and 162-image TopoBench-180 subset are scored with macro-averaged Node and Edge F1; direct VLM baselines generate full graphs.The Knossos test set contains 200 images from each of six domains.Source: §4.1 · p. 6

4.2 NODE EXTRACTION

  • 97.8% Node F1: both Ariadne variants reach this Knossos macro-average, while node performance decreases on TopoBench-180.Strong open-source VLMs generally exceed 95% Node F1 on Knossos.Source: §4.2 · p. 6, Figure 3 · p. 7

4.3 EDGE EXTRACTION

  • Ariadne improves over its corresponding backbones on both TopoBench-180 subsets; on typed edges, Ariadne-Qwen scores 90.4% versus GPT-5.6’s 84.0%.Ariadne-GLM also reaches 88.9% Typed-Edge F1, 4.9 points above GPT-5.6.Source: §4.3 · p. 7
  • 91.6% vs. 46.1% Edge F1 for Qwen3-VL-8B and 88.8% vs. 38.9% for GLM-4.6V-Flash: Ariadne variants outperform their direct backbones on Knossos.Ariadne-Qwen also exceeds direct Qwen3-VL-32B (54.7%).Source: §4.3 · p. 7, Table 1 · p. 8

4.4 ANALYSIS AND ABLATIONS

  • Topology performance falls with increasing diagram difficulty; dense links, cross-links, and intersecting paths challenge endpoint assignment, while directionality is less of a bottleneck.Performance remains relatively stable across directed, bidirectional, and mixed settings.Source: §4.4 · pp. 8–9
  • 80.0% to 91.6% Edge F1: staged extraction improves adapted Ariadne-Qwen, while staging reduces GPT-5.6’s Edge F1 and changes Node F1 only marginally.The ablation suggests staged decomposition benefits smaller task-adapted models more than frontier models.Source: §4.4 · pp. 7–8, Table 2 · p. 9

5 DISCUSSION AND CONCLUSION

Ariadne-Qwen reaches 91.6% Edge F1 on Knossos, outperforming GPT-5.6, and improves over its direct open-source backbones on TopoBench-180, though stronger closed-source VLMs remain more competitive. Knossos and Ariadne address complete-topology training and extraction, while synthetic data, finite grammars, and sequential inference constrain scope.

5 DISCUSSION AND CONCLUSION

  • 91.6% Edge F1 for Ariadne-Qwen versus 81.1% for GPT-5.6 on the 1,200-image Knossos test set; the margin grows from 2.0 points on simple to 12.2 on difficult diagrams.On TopoBench-180, Ariadne substantially improves over its direct open-source backbones, while the strongest closed-source VLMs remain more competitive.Source: §5 · p. 9
  • Knossos remains synthetic and uses finite domain vocabularies and predefined graph grammars, leaving real-world diversity and generalization to unseen types, semantics, and topologies as open boundaries.Ariadne is sequential, so node errors can propagate; source grouping also requires multiple model calls per image, and gains with stronger backbones remain unstudied.Source: §5 · p. 9
  • Knossos provides large-scale training and controlled evaluation, while Ariadne's source-conditioned decomposition particularly improves topology extraction for smaller task-adapted VLMs.Together, they offer a practical framework for studying diagram-to-graph topology extraction.Source: §5 · pp. 9–10

A RELATED WORK

  • Prior diagram benchmarks commonly evaluate selected content through question answering or reasoning, while Knossos supplies complete-topology supervision and controlled evaluation across six domains.Ariadne extends structural prediction through supervised complete-topology training and source-group decomposition.Source: A RELATED WORK · p. 13

B DETAILED BENCHMARK CONSTRUCTION

Knossos construction fixes symbolic topology before assigning layout, connector routing, and visual assets, so graph annotations align with rendered diagrams. Its domain-specific vocabularies and grammars vary structures and difficulty, while asset pools diversify appearance without changing evaluation targets.

B.1 CANONICAL VOCABULARIES AND NODE INSTANCES

  • Each domain uses a version-controlled vocabulary whose entries separate stable labels and semantic roles from display names and rendering attributes; node instances retain exact labels and diagram-local IDs.The frozen release covers 310 domain-local canonical labels, with Table 4 defining its vocabulary and node-occurrence statistics.Source: §B.1 · pp. 13–14, Table 4 · p. 14

B.2 DOMAIN GRAPH GRAMMARS

  • Domain grammars use valid structural motifs and role constraints, with difficulty varying graph scale, relation density, and geometric ambiguity rather than sampling unconstrained graphs.Food Web, Network, and Workflow grammars use trophic tiers, network structures, and control-flow templates; Natural Process and Circuit use layered structures, while Map Route varies spatial routes and intersections.Source: §B.2 · pp. 15–16
  • Across domains, generation samples difficulty-conditioned grammar parameters and role slots, instantiates a structurally valid graph, then assigns layout, connector routes, and visual assets.Difficulty changes grammar scale, density, and geometric ambiguity; the Natural Process examples show stage and relation constraints and realized instance sizes.Source: §B.2 · p. 15, Figure 5 · p. 16

B.3 VISUAL ASSET PIPELINE

  • Visual assets are sampled only after symbolic graphs are fixed, so changing an asset cannot alter graph annotations or evaluation targets.Pools vary by domain; Workflow is rendered programmatically, and seeded domain-specific sampling supports reproducibility.Source: §B.3 · pp. 17–18, Figure 6 · p. 17

B.4 GEOMETRY AND ANNOTATION DERIVATION

The geometry pipeline derives connector annotations from finalized node boxes and routed paths, retaining both the visible connector and deterministic trace points. Human review, image screening, and fixed train/test allocation govern release construction.

B.4 GEOMETRY AND ANNOTATION DERIVATION

  • For each directed edge, the renderer intersects the center-to-center ray with source and target box boundaries, then routes a logical polyline using the domain's routing family.It stores both the logical route and rendered visible path because terminal clearance and arrow placement can shorten the drawn connector.Source: §B.4 · p. 18
  • Five trace points are sampled at equal arc-length intervals along each rendered connector in final-image coordinates, with reciprocal directions stored in reverse order.Node boxes, edges, relation types, connector geometry, and training targets are serialized from the same finalized objects rather than recovered from pixels or entered manually.Source: §B.4 · p. 18, Figure 7 · p. 19

B.5 QUALITY CONTROL AND RELEASE CONSTRUCTION

  • After human review of preview batches, GPT-4o screened rendered candidates for clarity and visibility while allowing interpretable dense layouts and crossings.Screening continued until quotas were filled; 19,200 of 19,223 evaluated candidates passed.Source: §B.5 · pp. 18–19
  • The release assigns 3,000 training and 200 test images per domain, totaling 18,000 training and 1,200 held-out test images with matching annotations.Released images use fixed landscape or portrait resolutions, and an audit found canvas dimensions matched the PNG resolutions.Source: §B.5 · p. 19

B.6 TOPOBENCH-180 EVALUATION SUBSET

The section evaluates Ariadne on an external diagram subset and analyzes how source grouping changes topology serialization and decoding. Its structural guarantees are unconditional, while its error-reduction result depends on a position-sensitive decoding assumption.

B.6 TOPOBENCH-180 EVALUATION SUBSET

  • The external TopoBench-180 evaluation uses 162 images—54 AI2D/Web and 108 Network diagrams—with 1,786 nodes and 3,063 ordered relations.The subset excludes 16 internal Network diagrams and two with unresolved edge references; exclusions reflect data availability and annotation integrity, not model performance.Source: §B.6 · p. 19

C.1 LOSSLESS DECOMPOSITION AND SERIALIZATION AMBIGUITY

  • Source grouping preserves the complete edge multiset because every directed edge belongs to exactly one source group.It changes the decoding representation without restricting which edge multisets can be represented.Source: §C.1 · p. 20
  • A one-shot sequence assigns probability to a particular reference interleaving, adding ordering-specific loss beyond the probability assigned to its equivalence class.Under a uniform distribution over the class, the added term is log N_interleave; grouped decoding removes cross-group interleaving choices but not within-group ordering choices, and does not guarantee lower total loss or graph-prediction error.Source: §C.1 · pp. 20–21

C.2 BOUND ON THE DECODING HORIZON

  • Source grouping strictly shortens the maximum edge-decoding horizon whenever at least two source groups contain edges, while preserving the complete edge set.It replaces one global generation with multiple shorter local generations.Source: §C.2 · p. 21

C.3 POSITION-SENSITIVE DECODING MODEL

  • Under a shared, nondecreasing position-dependent edge-error model, restarting decoding across source groups weakly reduces cumulative position-associated error.The model treats later decisions as no easier on average, but does not claim output position itself causes error.Source: §C.3 · p. 22

C.4 BOUND ON POSITION-DEPENDENT ERROR REDUCTION

The analysis bounds potential error reduction from restarting decoding across source groups, while distinguishing structural guarantees from conclusions that depend on assumptions about decoding errors.

C.4 BOUND ON POSITION-DEPENDENT ERROR REDUCTION

  • The quantitative reduction bound relies on bounded growth in expected error between successive decoding positions.The parameter bounds α and β constrain each per-position increase.Source: §C.4 · p. 22
  • The potential benefit grows with graph size and with how extensively edges are distributed across multiple decoding groups.For fixed total edge count, the bound indicates greater potential reduction when edges are distributed more evenly across groups; with one nonempty group, it gives zero reduction.Source: §C.4 · p. 23
  • Source grouping preserves topology and strictly shortens the maximum decoding horizon when multiple source groups are nonempty, independently of VLM error behavior.Error reduction is conditional: expected edge-level error must be nondecreasing with decoding position, and the quantitative bound additionally assumes bounded per-position error growth.Source: §C.4 · p. 23

D.1 RELATION-TYPE PREDICTION BY DOMAIN

On Knossos domains with explicit relation semantics, Ariadne variants outperform their corresponding direct backbones, with Ariadne-Qwen leading across all three domains. Results also vary by domain, particularly for closed-source models.

D.1 RELATION-TYPE PREDICTION BY DOMAIN

  • Ariadne-Qwen achieves the highest Typed-Edge F1 in each of the three domains with explicit relation semantics, and both Ariadne variants outperform their corresponding direct backbones.Ariadne-GLM is close behind; Circuit is most challenging for most methods, while Natural Process generally has the strongest performance.Source: §D.1 · pp. 23–24, Table 5 · p. 24
  • Several closed-source models vary substantially across domains, indicating that relation-type prediction remains sensitive to domain-specific semantics.Table 5 reports Typed-Edge F1 (%) for the three Knossos domains with explicit relation semantics.Source: §D.1 · p. 24, Table 5 · p. 24

D.2 BOUNDING-BOX AND CONNECTOR LOCALIZATION

Ariadne improves geometric grounding over GPT-5.6 on the evaluated Knossos subset, with gains across domains. Node extraction remains strong in-domain, though the external benchmark shows a different generalization pattern.

D.2 BOUNDING-BOX AND CONNECTOR LOCALIZATION

  • Ariadne-Qwen3 raises box mIoU from 81.81% to 96.59% and lowers connector localization error from 11.34 to 4.13 pixels versus GPT-5.6.Ariadne-GLM also improves bounding-box localization and substantially reduces connector localization error.Source: §D.2 · p. 24
  • Ariadne-Qwen3 achieves the highest box mIoU in all six domains and lower connector localization error than GPT-5.6 throughout.The largest connector-error reductions occur on Food Web and Circuit; Figure 8 shows node boxes and reference-conditioned connector predictions on two original test images, without manual coordinate adjustment.Source: §D.2 · p. 24, Figure 8 · p. 25, Table 7 · p. 25

D.3 NODE EXTRACTION BY DOMAIN

  • Both Ariadne variants achieve a 97.8% Knossos average Node F1, but general-purpose VLMs often retain stronger node-level generalization on TopoBench-180.Circuit is more challenging for many baselines, and the results suggest node-specific adaptation may specialize to the synthetic training distribution.Source: §D.3 · p. 25

D.4 PROTOCOL EFFICIENCY ABLATION

The ablation measures the inference-cost trade-off of staged decomposition against one-step generation on a 30-image Knossos slice. Staged inference increases calls and latency, with overhead growing as diagrams become more difficult.

D.4 PROTOCOL EFFICIENCY ABLATION

  • 5.23× as many model invocations; mean latency rises 28.9% for GLM-4.6V-Flash and 59.6% for GPT-5.6, while GPT-5.6 token usage rises 2.80×.Table 9 covers 30 images, with 10 each at simple, medium, and difficult levels.Source: Table 9 · p. 26, §D.4 · p. 26
  • The comparison pairs one-step and staged protocols within each backbone, but absolute latency should not be read as a local-hardware versus API-service benchmark.One-step uses one invocation per image; staged inference uses one node-prediction call and one call per active source group.Source: §D.4 · pp. 25–26
  • Overhead grows with diagram difficulty because larger node inventories require more source-group calls.Source: §D.4 · p. 26

E.1 TRAINING AND INFERENCE CONFIGURATION

The configuration trains separate Ariadne predictors and evaluates their staged inference alongside direct full-graph generation. The prompts distinguish node discovery, source-conditioned typed-edge prediction, and optional connector localization.

E.1 TRAINING AND INFERENCE CONFIGURATION

  • Ariadne predictors and same-backbone direct full-graph baselines use QLoRA training, with evaluation on frozen splits after training.Training uses four NVIDIA A100 GPUs; inference uses greedy decoding, and the node and edge predictors have 2,048-token budgets while the trace predictor has 1,024.Source: §E.1 · p. 27

E.2 PROMPTS FOR TOPOLOGY EXTRACTION

  • Ariadne first predicts visible node labels, identifiers, and boxes, then predicts typed edges for active source groups using the complete node inventory as target context.Relation direction follows ordered source-target pairs; reciprocal edges represent undirected or bidirectional connectors, while distinct types sharing endpoints remain separate.Source: §E.2 · pp. 27–29

E.3 INDEPENDENT GEOMETRIC GROUNDING PROMPT

  • The independently trained trace predictor localizes already-decided edges and cannot add, remove, redirect, or retype them.For geometric-grounding evaluation, it receives reference nodes and edges, isolating connector localization from upstream topology errors.Source: §E.3 · pp. 29–30

E.4 DIRECT FULL-GRAPH BASELINE PROMPT

  • Direct full-graph baselines generate the complete node inventory and edge set in one response, determining direction from visible arrowheads rather than layout or semantics.Source: §E.4 · pp. 30–31
Loading 2610.04721v1…