SciFlow-Bench: Evaluating Structure-Aware Scientific Diagram Generation via Inverse Parsing

1 Introduction

• Generated diagrams can look coherent while containing missing components, hallucinated labels, or inconsistencies between annotations and arrow directions.These errors undermine communication and challenge automated analysis and human interpretation.Source: §1 · p. 1
• Existing benchmarks often miss whether final rendered diagrams preserve directed structure: image metrics and subjective judgments overlook organization, while symbolic evaluations bypass pixel-generation failures.Visually polished diagrams can still contain reversed dependencies, missing steps, or unsupported connections.Source: §1 · pp. 1–2
• SciFlow-Bench pairs source framework figures with canonical graphs and inverse-parses generated images to measure structural recoverability rather than visual similarity alone.A hierarchical multi-agent system constructs reference graphs and predicted graphs within the round-trip evaluation protocol.Source: §1 · p. 2
• The benchmark evaluates code-driven, diffusion-based, and autoregressive models and finds structural correctness especially challenging for diagrams with complex topology.The reported results highlight a decoupling between visual fidelity and structural reasoning.Source: §1 · p. 2

2 Related Work

• Diffusion-based diagram generators can achieve strong visual fidelity yet struggle with long-range dependencies, spatial relations, and text-object binding.These weaknesses can produce missing components or incorrect connections.Source: §2 · p. 2
• Image-generation metrics assess realism or coarse semantic relevance, but remain largely insensitive to relational and topological structure.Existing scientific-image benchmarks still do not explicitly model directed and hierarchical dependencies needed for faithful interpretation or reconstruction.Source: §2 · p. 2
• SciFlow-Bench combines pixel-level evaluation, explicit graph topology, and deterministic structural metrics in a round-trip protocol.It compares graphs reconstructed from generated images with canonical graphs derived from source framework figures.Source: Figure 2 · p. 3
• Diagram benchmarks often focus on a single modality: symbolic-layout benchmarks bypass pixel generation, while perception benchmarks do not assess generated-diagram structural fidelity.VPEval evaluates diagram relations through visual question answering, but manual annotations constrain scalability and coverage.Source: §2 · p. 3

3.1 Problem Formulation and Overview

• The round-trip formulation constructs a canonical graph from a source framework figure, inverse-parses a generated image into a predicted graph, and compares them in graph space.The framework uses planning, perception, and structural reasoning to support graph construction and inverse parsing.Source: §3.1 · p. 3

3.2 Hierarchical Multi-Agent Pipeline

• The multi-agent pipeline extracts a method description, converts it into a shared structured visual prompt, and combines layout, segmentation, and OCR signals into grounded nodes.The Topology Coder and Graph Architect turn grounded entities into directed relations and structured graphs for dataset construction or evaluation.Source: §3.2 · pp. 3–4

3.3 Structure-Aware Evaluation

• Evaluation combines graph-level topology, text-level faithfulness, and image-level consistency, with graph-level correctness receiving the greatest weight in the leaderboard.Directed edges count as correct only when their endpoints match a valid canonical relation, penalizing missing, reversed, or unsupported connections.Source: §3.3 · p. 4

4.1 Data Collection

• The benchmark collects source framework figures from 2025 arXiv papers and preserves ambiguous or implicit relations without manual filtering, redrawing, or simplification.Each figure is paired with a canonical graph used as the structural reference for downstream evaluation.Source: §4.1 · pp. 4–5

4.2 Annotation Quality and Reliability

• On 100 human-verified source figures, HMAS achieves Node-F1 of 0.91 and Edge-F1 of 0.71, with lower edge-level performance reflecting the difficulty of recovering directed relations.The agreement metrics use exact identity consistency and are reported separately from benchmark evaluation metrics.Source: §4.2 · p. 5, Table 1 · p. 5

4.3 HMAS on Generated Diagrams

• HMAS maintains stable behavior across real and generated diagrams: Edge-F1 is 0.68 on Qwen-Image outputs and 0.70 on Gemini 3 Pro Image outputs, compared with 0.71 on real source figures.On structurally clean Graphviz diagrams, it achieves Node-F1 of 0.96 and Edge-F1 of 0.89.Source: §4.3 · pp. 5–6, Table 2 · p. 5

4.4 Benchmark Statistics and Coverage

• SciFlow-Bench contains 500 diagrams across five research areas, and over 70% exhibit non-linear structures, with node and edge counts showing long-tailed distributions.Structural difficulty is characterized by graph size and relational complexity.Source: §4.4 · p. 6, Figure 3 · p. 6

4.5 Comparison with Prior Benchmarks

• Compared with prior benchmarks, SciFlow-Bench unifies pixel-level evaluation, explicit graph topology, and deterministic metrics to assess structural recoverability directly from generated images.Code-centric benchmarks bypass pixel generation, while image-space benchmarks do not explicitly recover graph structure.Source: §4.5 · pp. 6–7, Table 3 · p. 6

5.1 Experimental Setup

• All generators share fixed source papers, structured prompts, and inverse parsing; Graphviz serves as a deterministic diagnostic reference rather than a direct pixel-generator competitor.The evaluated pixel-based models are treated as black boxes and assessed on their final rendered images.Source: §5.1 · p. 7

5.2.1 Structural Performance Regimes

• Vanilla diffusion models remain near zero on graph-level scores across difficulty levels despite moderate image-level relevance.The authors conclude that pixel-level distribution matching alone often fails to preserve directed dependencies or stable node identities.Source: §5.2.1 · p. 7
• Qwen-Image rises from 0.017 to 0.081 average graph-level performance over PixArt-Σ; Gemini 3 Pro Image leads overall and rises from 0.079 on Easy to 0.135 on Hard.The Qwen-Image improvement is reported across Medium and Hard subsets.Source: §5.2.1 · pp. 7–8

5.2.2 Visual Structural Dissociation

• Visual relevance often exceeds recoverable structure; diffusion models degrade with complexity, while less capable models lose connectivity as topology densifies.The paper reports that advanced systems such as Gemini 3 Pro Image improve on the Hard subset.Source: §5.2.2 · p. 8

5.2.3 Symbolic Generative Trade-offs

• Graphviz offers deterministic topology and strong text-level faithfulness but rigid layouts, whereas pixel generators look more polished while retaining logical uncertainty.SciFlow-Bench grounds pixel outputs back in graph space to expose that uncertainty.Source: §5.2.3 · p. 8

5.3 Ablation Study

• Removing the Text Spotter cuts node recall from 0.96 to 0.48, while the full parsing pipeline balances metrics using complementary perception signals.Removing the Shape Hunter also degrades structural completeness through under-segmentation and lower edge recall.Source: Table 5 · p. 8, §5.3 · p. 9
• Structured Prompts outperform Naive Prompts across representative generators, improving graph-level recovery and making structural evaluation better defined.The comparison includes multimodal, diffusion-based, and Graphviz systems, with other factors held fixed.Source: §5.3 · p. 9, Table 6 · p. 9

• SciFlow-Bench pairs source framework figures with canonical graphs and uses round-trip evaluation to prioritize structural recoverability over visual plausibility.The benchmark is intended as a scalable framework for diagnosing structural failures not visible to image-centric metrics.Source: §6 · p. 9
• Across layout-driven, open-source, and commercial models, preserving structural correctness remains difficult, particularly for diagrams with complex topology.The results also highlight decoupling between visual fidelity and structural reasoning in realistic settings.Source: §6 · p. 9
• The authors hope the benchmark supports treating structural recoverability as a core criterion and developing systems that reason about scientific structure.Source: §6 · p. 10

• The benchmark does not explicitly assess fine-grained aesthetics, including stylistic preferences or rendering quality.Source: Limitations · p. 10
• Its scope is framework-style scientific diagrams with explicit components and directed dependencies, not every diagram type.Source: Limitations · p. 10
• Broader diagram coverage and integrating aesthetic evaluation are left for future work.Source: Limitations · p. 10

• SciFlow-Bench is built from publicly available scientific papers and framework figures and involves no personal data or human subjects.Source: Ethical Considerations · p. 10
• Its stated purpose is research and evaluation of structural properties, not assessment of source-paper content or authorship.Source: Ethical Considerations · p. 10
• The authors do not foresee significant ethical risks and say results should be interpreted diagnostically.Source: Ethical Considerations · p. 10

A.1 Graph-Level Metric Implementation

• Graph-level scoring semantically matches predicted and reference nodes, then checks directed edges against valid canonical dependencies while allowing paths that skip intermediate nodes.Incorrect directions, unsupported endpoints, and hallucinated relations count as errors.Source: §A.1 · p. 13
• Graph-level scores emphasize relational structure: edge F1 receives 60% of the graph score, compared with 40% for node F1.All graph aggregation weights are fixed a priori and shared across evaluated models.Source: §A.1 · p. 13

A.2 Text-Level Metric Implementation

• Text-level scoring measures prompt-component coverage and faithfulness, plus global alignment of the predicted graph with the structured visual prompt.Coverage and faithfulness each contribute 30%; alignment contributes 40%.Source: §A.2 · pp. 13–14

A.3 Image-Level Metric Implementation

• Image-level scoring combines CLIP semantic consistency, inverted and normalized LPIPS similarity, and evaluator-rated visual-flow consistency.Semantic consistency and visual flow each contribute 40%; perceptual similarity contributes 20%.Source: §A.3 · p. 14

A.4 Overall Score Aggregation

• The leaderboard weights graph-level scores at 40% and text-level and image-level scores at 30% each, with weights fixed across models and domains.Prompts, thresholds, models, weights, and scripts are also fixed across experiments for reproducibility.Source: §A.4 · p. 14, §A.5 · p. 14

B.1 Design Objectives

• Human verification edits the automated graph rather than rebuilding it, preserving stable node and edge identifiers for exact comparison.The interface preloads extracted elements alongside the source figure and supports element-level inspection and edit auditing.Source: B Human Verification Interface · p. 15, §B.1 · p. 15, §B.2 · p. 15
• Annotators are instructed to make minimal corrections, removing unsupported elements and adding only missing, semantically necessary components.Constrained editing is intended to reduce subjective reinterpretation and keep verification consistent and efficient.Source: §B.1 · p. 15

B.3 Interaction Modes

• The synchronized graph interface offers select/delete and link-nodes modes, but does not allow free redrawing, arbitrary renaming, or wholesale restructuring.Added directed links are reserved for dependencies clearly evident in the source figure.Source: §B.3 · p. 15

B.4 Identity-Consistent Editing Protocol

• Annotators begin with every automatically extracted node and edge included, then exclude unsupported elements; additions and exclusions map to false negatives and false positives.Because edits preserve original identifiers, node- and edge-level precision and recall can be computed without post-hoc semantic matching.Source: §B.4 · pp. 15–16
• Identity-consistent editing makes annotation quality a direct measure of automated pipeline accuracy rather than subjective agreement between independent annotations.Inherited elements are blue, human-added elements magenta, and excluded elements muted for review.Source: §B.4 · p. 16, §B.5 · p. 16

B.6 Annotation Workflow and Quality Control

• Each figure is independently reviewed by a trained computer-science annotator using only visible diagram evidence, with ambiguous cases resolved conservatively.Shared guidelines cover unsupported or duplicated nodes, ambiguous relations, and missing dependencies; sessions log edits and annotation time.Source: §B.6 · p. 16

B.7 Qualitative Examples

• Qualitative examples and Table 1 agreement results indicate that HMAS recovers most structural components, with sparse, interpretable errors amenable to targeted correction.Discrepancies commonly involve weakly supported connections or missing relations in dense layouts, overlapping elements, or implicit arrow semantics.Source: §B.7 · p. 16

C Discrepancy Analysis

• The appendix presents representative mismatch categories to contextualize ambiguity in real scientific diagrams, not to enumerate every failure case.It notes that difficult structures can also challenge human readers without domain context.Source: C Discrepancy Analysis · pp. 16–17, Figure 5 · p. 17

C.1 Ambiguous or Implicit Relations

• Implicit or stylistic links can support multiple interpretations, so the pipeline favors avoiding unsupported directed edges over speculative inference.This ambiguity is especially common in high-level architectural diagrams.Source: §C.1 · p. 17

C.2 Text-Related Perception Errors

• OCR and grounding errors arise with vertical labels, small subscripts, dense text, and stylized fonts, potentially causing missed or incorrectly labeled nodes.Such discrepancies are described as sparse and localized rather than structural.Source: §C.2 · p. 17

C.3 Structurally Underspecified or Symbolic Nodes

• Abstract identifiers and symbolic placeholders may lack visual grounding, leading human annotators to reinterpret or reattach nodes that the parser leaves isolated or weakly connected.These cases expose the limits of diagram-only interpretation and motivate evaluation that tolerates multiple valid graph realizations.Source: §C.3 · pp. 17–18

C.4 Summary and Implications

• Across analyzed verification cases, mismatches are sparse and interpretable rather than cascading failures, with edge errors concentrated in structurally ambiguous regions.The analysis links this pattern to high node recall and lower edge-level agreement in Table 1.Source: §C.4 · p. 18

D.1 Generation Paradigms

• The qualitative comparison contrasts deterministic Graphviz layout, pixel-space Qwen-Image diffusion, and autoregressive Gemini 3 Pro Image under shared structured prompts.The Graphviz reference makes topology explicit; the two pixel-based systems are evaluated without manual prompt tuning.Source: §D.1 · pp. 18–19, Figure 6 · p. 18

D.2 Qualitative Examples

• Qwen-Image often appears locally coherent but can omit intermediate components or produce ambiguous arrows and unsupported connections, degrading graph recoverability.Gemini 3 Pro Image more often preserves interpretable layouts, arrow direction, and module grouping, particularly in complex multi-stage pipelines.Source: §D.2 · p. 19

D.3 Structural Recoverability under Inverse Parsing

• Polished diffusion outputs may still lack recoverable relational cues, while spatial separation, consistent arrows, and stable text-to-region alignment support higher graph-level scores.Common failures include merged components, ambiguous arrow geometry, and decorative elements mistaken for functional nodes.Source: §D.3 · p. 19

D.4 Discussion

• Controlled qualitative inspection complements quantitative metrics by showing why diagrams must be judged by recoverable structure, not appearance alone.The comparisons identify a trade-off between visual flexibility and structural determinism.Source: §D.4 · p. 19
• Layout-driven methods guarantee explicit topology but sacrifice stylistic freedom, whereas pixel-based models offer richer expression with variable structural recoverability.Stronger vision–language integration correlates with better recoverability among pixel-based systems, though substantial gaps remain.Source: §D.4 · p. 19