Source-linked AI summary
SciForma: Structure-Faithful Generation of Scientific Diagrams
Yuxuan Luo, Peng Zhang, Xinjie Zhang, Xun Guo, Zhouhui Lian, Yan Lu
TL;DR
Scientific diagram generators lack reliable ways to enforce correctness across components, arrows, and text. SciForma addresses this with structural verification and conjunctive preference optimization, and SciForma-9B surpasses reported baselines on SciFormaBench-2K and AIBench.
Problem
Existing open-source models lack mechanisms to independently verify and enforce structural correctness across components, directional relations, and textual annotations in methodology diagrams.
Method
SciForma decomposes diagrams into Component, Arrow, and Text axes, then uses structural inventories, curated data, M-DPO, and iterative verification-gated editing.
Results
On SciFormaBench-2K, SciForma-9B reaches 69.51, surpasses GPT-Image-1.5 and open-source baselines, and exceeds human-drawn originals on AIBench.
Takeaways & Limitations
SciForma demonstrates structure-faithful scientific diagram generation through explicit joint constraints over Component, Arrow, and Text axes.
Takeaways & Limitations
The evaluation-and-editing pipeline depends on a proprietary VLM, while 1024 px resolution limits fine-grained rendering in dense diagrams.
Abstract
from arXiv · showhide
Structural fidelity is essential to scientific methodology diagrams. To communicate research logic, these diagrams must faithfully render components, directional relations, and textual annotations. Since a single error, such as a reversed arrow or an unreadable equation, can invalidate the entire figure, structural fidelity is inherently conjunctive: correctness on one axis cannot compensate for failure on another. Current open-source models fail to satisfy this criterion. Supervised fine-tuning (SFT) learns plausible layouts but cannot reliably ensure structural correctness, while scalar reward-based post-training obscures which structural dimension has failed. To address this, we introduce SciForma, a framework for the structure faithful generation of scientific methodology diagrams. Specifically, SciForma decomposes diagram quality into three structural axes: Component, Arrow, and Text, guided by a structural inventory. Built on this foundation, we curate SciFormaData-700K for structured training and SciFormaBench-2K for logic-verified evaluation. To close the gap left by SFT, we develop Multi-Dimensional Conjunctive Preference Optimization (M-DPO), which enforces simultaneous correctness across all axes and adaptively routes gradients to the most deficient dimension in post-training. The same structural inventory also enables iterative editing at inference time to correct residual errors. This combination allows SciForma-9B to exceed all open-source baselines and GPT-Image-1.5 on both SciFormaBench-2K and AIBench, bringing open scientific diagram generation close to proprietary-level structural fidelity. Our code and data will be available at: https://github.com/microsoft/SciForma.
1 Introduction
SciForma frames scientific-diagram generation as a structural-fidelity problem requiring simultaneous correctness of components, arrows, and text. It introduces structured data and evaluation resources plus M-DPO to address limitations of SFT and scalar reward-based post-training.
- Motivation: Methodology diagrams must faithfully render every component, directional relation, and textual annotation because errors such as reversed arrows or unreadable equations can invalidate meaning.The paper contrasts this requirement with natural-image generation, where visual plausibility is the primary concern.
- Motivation: Current open-source models trail proprietary systems because SFT does not guarantee complete structural correctness and scalar post-training obscures failures across orthogonal structural axes.The introduction identifies a limitation in current training objectives as the source of this gap.
- SciForma: SciForma decomposes diagram quality into independently verifiable Components, Arrows, and Text, covering semantic modules, directional relations, and textual annotations.This decomposition follows established process-modeling standards and provides the framework’s central structural perspective.
- Contributions: The paper introduces SciFormaData-700K for structured generation and editing and SciFormaBench-2K for logic-verified evaluation.The dataset is topology-captioned, while the benchmark is designed for logic-verified evaluation.
- Contributions: M-DPO contrasts a global winner with axis-specific losers through a multi-way Bradley–Terry objective, enforcing conjunctive correctness while adaptively routing gradients.The method is presented as addressing the gap left by SFT.
- Results: SciForma-9B exceeds open-source baselines and GPT-Image-1.5 on SciFormaBench-2K, with its largest gains on the Arrow axis and Hard tier, and exceeds human-drawn originals on AIBench.AIBench results are described as having the largest margin on Topology; ablations report that M-DPO avoids scalar-method reward collapse and that decomposed pairs with a conjunctive objective are vital.
2 Related Work
Prior diagram-generation approaches trade precise control for limited expressiveness, while image-based scientific diagram generation remains challenging. Diffusion-model alignment methods use offline or online optimization, but existing approaches do not address independently verifiable structural axes under conjunctive constraints.
- Diagram generation: Code-based methods use TikZ, Mermaid/SVG, or domain-specific markup for precise control but produce simple or template-driven visualizations because expressiveness is constrained.These approaches synthesize diagrams through structured languages or specialized markup.
- Diagram generation: Image-based generation for scientific diagrams remains challenging.
- Diffusion alignment: Diffusion-model post-training alignment comprises offline DPO-style methods using pairwise or ranked comparisons and online policy-gradient or GRPO methods.The cited offline extensions include ranking and multi-candidate ranking, while online work improves sampling efficiency and training.
- Diffusion alignment: Existing alignment methods target smoothly co-varying perceptual axes, where scalar aggregation is often sufficient, rather than independently verifiable structural axes under conjunctive constraints.Neighbor GRPO couples multiple candidates but still collapses feedback into a single scalar.
3 Structural Inventory
SciForma defines structural fidelity through a three-part inventory of components, arrows, and text, making diagram correctness explicitly verifiable against a reference checklist. Evaluation compares outputs with this checklist and prioritizes per-axis diagnosis over aggregate averages.
- Structural primitives: Scientific diagrams encode meaning through typed entities, directed relations, and textual qualifiers, as established in UML activity diagrams and BPMN process models.
- Structural primitives: The structural inventory formalizes three independently verifiable primitives: Component(C), Arrow(A), and Text(T).Components represent entity identity and spatial layout; arrows represent information topology; text represents annotations.
- Checklist extraction: A VLM extracts component coordinates, arrow source-target pairs, and literal text strings into a JSON checklist defining the required contents of a faithful output.
- Axis evaluation: Each discrepancy is labeled critical (w_e=1.0) when missing or fundamentally wrong, or moderate (w_e=0.5) when present but defective.
- Axis evaluation: Per-axis scores remain primary for diagnosis, reward construction, and ablation analysis, while the per-axis average provides only a compact summary.
4 SciForma
SciForma combines a structurally annotated dataset and logic-verified benchmark with two-stage SFT, conjunctive M-DPO, and inventory-guided refinement to improve scientific diagram fidelity. Its design targets simultaneous correctness of components, arrows, and text rather than compensatory scalar scores.
- Data construction: 726K methodology diagrams from 593K arXiv LATEX sources support SciFormaData-700K through consensus filtering and pHash deduplication.The dataset comprises 656K generation pairs and 70K editing triplets with axis-decomposed captions covering Component, Arrow, and Text.
- Benchmark: SciFormaBench-2K evaluates structural fidelity with axis-level decomposition and element-level attribution across 2,000 human-verified candidates.The candidates are stratified as Simple (500), Medium (900), and Hard (600), averaging 10.9 components, 10.2 arrows, and 6.6 text labels; 56.8% require formula rendering.
- Training pipeline: Two-stage SFT adapts FLUX.2-klein-base-9B to diagram layout, text rendering, connectivity, generation, and localized editing across aspect-ratio buckets.Stage 1 uses the 656K training set at approximately 768 px, while Stage 2 co-trains generation samples and editing triplets at approximately 1024 px, producing SciForma-Base.
- M-DPO: M-DPO enforces conjunctive correctness by pairing one shared winner with axis-specific losers and optimizing a multi-way Bradley–Terry objective.It avoids the dilution of independent multi-axis losses: the loss approaches zero only when every axis-specific margin is positive, while adaptive weights concentrate gradients on deficient axes.
- Iterative refinement: An autonomous refinement loop uses GPT-5.4 and the structural inventory to localize, rank, and iteratively correct residual axis-specific defects in dense diagrams.The loop targets errors such as misrouted arrows and formula or text corruption under explicit verification gates.
5 Experiments
Experiments evaluate SciForma across structural fidelity, reader-oriented logic, agentic visualization, ablations, qualitative comparisons, and human preference. SciForma-9B improves over open-source baselines, surpasses GPT-Image-1.5 on core benchmarks, and benefits further from M-DPO and iterative refinement.
- SciFormaBench-2K: SciForma-9B raises FLUX.2-klein-base-9B’s SciFormaBench-2K score from 33.87 to 67.59, while M-DPO reaches 69.51 and surpasses GPT-Image-1.5.M-DPO’s largest gains target Arrow (+1.82) and Text (+3.16).
- SciFormaBench-2K: Iterative Refinement lifts SciForma-9B’s SciFormaBench-2K score to 72.40 (+2.89), showing that targeted inference-time repair addresses residual topological errors.Arrow gains most during refinement.
- AIBench: SciForma-9B reaches 70.29 on AIBench, edging out human-drawn originals at 70.09 and widening its lead over GPT-Image-1.5 by 8.67 points.The widest margin is on Topology (+6.19 over originals), while originals lead Component, 82.65 versus 77.53.
- PaperBanana: SciForma achieves 30.7% overall on PaperBanana, surpassing GPT-Image-1.5 and approaching Nano Banana Pro on Faithfulness, 34.5% versus 45.8%.Conciseness remains the main bottleneck, at 37.7% versus 80.7%.
- Ablations: M-DPO achieves a +1.92 average gain versus +0.37 for 30K additional SFT steps, using one-seventh the training steps while improving Text +3.16 and Arrow +1.82.Ablations attribute the advantage to axis-decomposed, conjunctive supervision rather than scalar or conflicting preference signals.
- User study: In a user study of 36 AI-specialized graduate students evaluating 30 samples, SciForma-9B outperforms Wan2.7-Image across all metrics and rivals Nano Banana Pro.The evaluated dimensions are Structure, Text, Visual Quality, and Overall Preference.
6 Discussion and Limitations
SciForma’s remaining limitations include resolution, evaluator openness, and potential drift in automatically generated structural inventories. The authors release the framework while advocating extensions, human verification, and in-the-loop editing to support responsible use.
- Limitations: SciForma’s 1024 px resolution limits fine-grained rendering in dense diagrams, while its evaluation-and-editing pipeline depends on a proprietary VLM.The authors identify higher-resolution training and open-source VLM evaluators as next opportunities.
- Limitations: Automatically generated structural inventories in agentic use can drift from user intent because they are not human-verified.Human-in-the-loop agentic frameworks are proposed as a next opportunity.
- Responsible Use: The authors release SciForma to advance open image-based diagram research and invite extensions of M-DPO’s conjunctive design to per-axis tasks.They also advocate human verification and in-the-loop editing of all generated outputs to help prevent misuse in paper fabrication.
7 Conclusion
SciForma makes scientific diagram generation structure-faithful by defining three verifiable quality axes—Component, Arrow, and Text—and enforcing their joint correctness with M-DPO. SciForma-9B approaches proprietary-level structural fidelity and aesthetics through explicit structural constraints.
- Conclusion: SciForma defines diagram quality through three verifiable axes: Component, Arrow, and Text.These axes provide the framework for evaluating structural fidelity.
- Conclusion: M-DPO enforces joint correctness across the Component, Arrow, and Text axes.The method addresses structural correctness as a conjunctive requirement.
- Conclusion: SciForma-9B approaches proprietary-level structural fidelity and aesthetics with explicit structural constraints.The experiments support the framework’s effectiveness for structure-faithful scientific diagram generation.
Supplementary Material Overview … A.1.4 Stage 4: Editing Triplet Construction.
The supplementary material details SciFormaData-700K’s construction, from LATEX-based source extraction and consensus filtering through structural captioning, stratification, and verified editing-triplet generation. It also documents the benchmark, theoretical analysis, implementation, ablations, qualitative results, user study, and prompt templates.
- Supplementary Material Overview: The supplementary material covers dataset and benchmark details, M-DPO theory, implementation, analyses, ablations, qualitative results, user studies, and prompt templates.
- A.1 Dataset Construction: SciFormaData-700K uses a four-stage construction pipeline, with full VLM-mediated prompt templates provided in Appendix I.
- A.1.1 Stage 1: Source Extraction.: LATEX sources from 593K arXiv papers across 17 cs.* categories are parsed directly to extract figures, captions, references, and macro-resolved metadata.
- A.1.2 Stage 2: Consensus Filtering.: Approximately 1.8M candidates are filtered for methodology diagrams, and dual-VLM agreement retains only figures independently classified as methodology targets.
- A.1.3 Stage 3: Captioning and Stratification.: Qwen3-VL produces Component, Arrow, and Text captions; complexity tiers are Low (13.6%), Medium (48.2%), and High (38.2%), with 2,000 samples held out and 656K generation pairs retained.
- A.1.4 Stage 4: Editing Triplet Construction.: 70K editing triplets are derived from a ∼252K high-quality subset using SAM3 segmentation, deterministic component or arrow edits, and visual and semantic validation.
- A.1.4 Stage 4: Editing Triplet Construction.: The construction funnel contains 658,000 post-filtering items, 2,000 held-out benchmark samples, 656,000 generation-training items, 70,000 editing triplets, and 726,000 total data volume.
A.2 Dataset Statistics and Analysis · B SciFormaBench-2K Details · B.1 Structural Inventory
SciFormaData-700K spans major computer-science methodology-diagram domains and exhibits substantial geometric, mathematical, and editing diversity. SciFormaBench-2K evaluates structural fidelity through a verifiable Components–Arrows–Text inventory and checklist-based image comparison.
- A.2 Dataset Statistics and Analysis: SciFormaData-700K draws source papers from 17 ArXiv Computer Science subcategories, covering major methodology-diagram areas.The listed categories include computer vision, machine learning, artificial intelligence, language, robotics, and other fields.
- A.2 Dataset Statistics and Analysis: Axis-decomposed captions have a substantial median length of 538, providing precise structural detail for reconstructing methodology logic from text.The passage connects these long prompts to reconstruction of methodology logic from text alone.
- A.2 Dataset Statistics and Analysis: 89.5% of diagrams are landscape, with a median aspect ratio of 2.03, reflecting geometric priors aligned with academic conventions.These statistics are reported as part of the dataset’s structural-diversity analysis.
- A.2 Dataset Statistics and Analysis: 25.9% of diagrams contain embedded mathematical formulae, while 2.6% are TikZ-rendered.Both figures characterize the dataset’s structural and rendering diversity.
- A.2 Dataset Statistics and Analysis: 71.4% of the 70K editing triplets involve 2–3 concurrent modifications, supporting diverse iterative-editing scenarios.The passage describes this distribution as providing a rigorous basis for editing.
- B SciFormaBench-2K Details: SciFormaBench-2K defines structural-inventory extraction and an error taxonomy for scoring, with Components, Arrows, and Text serving as verifiable primitives.The inventory is extracted through joint prompt-image analysis using GPT-5.4 or direct LATEX parsing, and a VLM compares ground-truth and generated images against the checklist.
- B.1 Structural Inventory: The inventory bypasses subjective aesthetic metrics by enabling reproducible assessment of component, arrow, and text discrepancies.During evaluation, a VLM performs a two-way comparison between ground-truth and generated images against the C/A/T checklist.
B.2 Evaluation Taxonomy … C.2 Adaptive Focus Property
The evaluation taxonomy weights structural errors by severity and verifies components, arrows, and text through focused axis-specific checks. M-DPO’s analysis shows that its gradient adaptively concentrates on failing axes, stops when all axes are satisfied, and reduces to standard DPO when only one axis remains.
- B.2 Evaluation Taxonomy: Critical errors receive w_e = 1.0, while moderate errors receive w_e = 0.5 to distinguish fundamental misrepresentations from defective but functional elements.The weighting captures nuanced flaws without penalizing the primary topological skeleton.
- B.2 Evaluation Taxonomy: Evaluation ignores non-topological traits and uses three axis-specific VLM calls to maximize checking focus and avoid context exhaustion.The protocol produces component-level feedback through structural-inventory verification and assigns error types and severity scores to discrepancies.
- C M-DPO Theoretical Analysis: The theoretical analysis derives adaptive gradient-focus dynamics for M-DPO and connects the objective to multi-way Bradley-Terry and InfoNCE formulations.The appendix provides the full derivation and formal connections.
- C.1 Gradient Derivation: Each gradient step decreases flow-matching loss for the winner and increases it for every axis-anchored loser, with each contribution weighted by w_d.The gradient acts simultaneously across the winner and axis-specific losers.
- C.2 Adaptive Focus Property: On a failing axis, Δ_d < 0 makes w_d large, whereas on a strongly satisfied axis, Δ_d ≫ 0 makes w_d approximately zero.The axis weights form a softmax distribution parameterized by the negated logits {−βΔ_d}.
- C.2 Adaptive Focus Property: When all axes are satisfied, M-DPO’s loss and gradient vanish, using the denominator’s +1 as a no-update anchor that prevents over-optimization.Training halts instead of redistributing updates to an arbitrary axis or continuing to change a converged model.
- C.2 Adaptive Focus Property: M-DPO concentrates weight on the bottleneck instead of diluting a single failing axis under the fixed 1/D weighting of the mean baseline.When D−1 axes are strongly satisfied, the mean objective can stall optimization of the remaining failing axis.
- C.2 Adaptive Focus Property: For D = 1, M-DPO exactly equals the standard DPO loss and therefore generalizes scalar DPO to conjunctive multi-axis optimization for D ≥ 2.The reduction follows from log(1+e^−x) = −logσ(x).
C.3 Multi-Way Bradley-Terry Derivation … E.3 Evaluation Robustness across Multiple Rounds
SciForma formalizes conjunctive structural preference learning, implements inventory-guided generation and editing, and validates its evaluation protocol through reflexivity, cross-evaluator agreement, and repeated-run stability. These analyses show that the method’s structural gains and rankings remain reliable across evaluators and scoring rounds.
- C.3 Multi-Way Bradley-Terry Derivation: M-DPO rewards a policy only when the winner defeats every axis-specific loser simultaneously, making conjunctive correctness explicit.The derivation uses a multi-way Bradley-Terry preference model with one winner and D axis-anchored losers.
- C.4 Connection to InfoNCE: M-DPO is equivalent to InfoNCE with D deliberately axis-anchored negatives, producing informative gradients for isolated structural deficiencies.Unlike i.i.d. InfoNCE negatives, these negatives are mined to target specific failure modes.
- D.2 Iterative Refinement Details: The closed-loop editor prioritizes text, module structure, and arrows, then applies localized edits or region redraws based on defect density and complexity.The critic uses a severity hierarchy, grid-anchored localization, and preserved surrounding context to target defects.
- D.2 Iterative Refinement Details: SciForma-9B generates up to K=3 candidate edits per region and rejects updates that reduce whole-image quality beyond ε=0.03.A global monotonic guard reverts edits that disrupt holistic layout or connectivity beyond the tolerance margin.
- E SciFormaBench-2K In-depth Analysis; E.1 Reflexivity Verification: GPT-5.4 recovers 99.94% of checklist elements on average, with every per-axis and per-tier score above 99.85%, supporting evaluator extraction reliability.Ground-truth diagrams were scored against their own human-verified structural inventories in the reflexivity analysis.
- E.2 Cross-Evaluator Consistency: SciForma-9B + Edit scores 61.05 versus 48.68 for FLUX.2-dev 32B, 49.76 for Qwen-Image-2512, and 59.32 for GPT-Image-1.5 under Qwen3-VL-8B-Instruct.Cross-evaluator system rankings show Pearson r=0.9945 and Spearman ρ=0.9824, while absolute scores shift with evaluator strictness.
- E.3 Evaluation Robustness across Multiple Rounds: Across two independent GPT-5.4 scoring passes, absolute score differences remain below 0.22 points, indicating stable and reliable judgments.The evaluation repeats the full SciFormaBench-2K protocol for a representative subset of models.
E.4 Independence of C, A, T Evaluation Axes … F.2 M-DPO Ablations
The paper validates that Component, Arrow, and Text scores capture largely independent structural failures, motivating conjunctive reward design, and then ablates editing supervision and M-DPO data-construction choices. Editing triplets substantially improve iterative repair across all axes.
- E.4 Independence of C, A, T Evaluation Axes: The independence analysis evaluates Component, Arrow, and Text axes across 7,391 valid samples from four diverse model types.The models include GPT-Image-2, Nano Banana Pro, Wan2.7-Image, and intermediate checkpoints.
- E.4 Independence of C, A, T Evaluation Axes: Component–Arrow scores are effectively orthogonal, with r< 0.02 and p> 0.05 across all evaluated models.This indicates that the two axes capture distinct failure modes rather than redundant properties.
- E.4 Independence of C, A, T Evaluation Axes: The remaining axis pairs show weak positive correlations of r≤0.26, with maximum shared variance strictly below 7%.The stable pattern holds across models and supports the rubric’s separation of structural properties.
- E.4 Independence of C, A, T Evaluation Axes: Because the axes are statistically independent, averaging their rewards can dilute gradients for a deficient dimension, such as misplaced arrows.A model with correct components but incorrect arrows may receive reward dominated by Component and Text terms, leaving little pressure to repair arrows.
- F More Ablations: The ablation section studies SFT Stage 2 editing data and M-DPO choices involving preference-pair curation, axis decomposition, and the KL penalty β.These experiments isolate modeling and training-design decisions across Sections F.1 and F.2.
- F.1 SFT Ablations: SciFormaData-700K includes 70K editing triplets, whose benefits are tested by comparing SFT checkpoints trained with or without them under the same iterative editing budget.The checkpoints are trained for 60K steps, and the triplets are intended for localized edits rather than full-image generation.
- F.1.1 Editing Triplet Supervision (Stage 2).: +2.96 average improvement results from including editing triplets during SFT after the same editing budget.Arrow improves by +2.44 and Text by +2.66, confirming that editing supervision enables localized repair during inference-time refinement.
- F.2 M-DPO Ablations: M-DPO preference optimization first ablates preference-pair construction by comparing four winner-source and rollout-diversity strategies on the same SFT checkpoint.This experiment isolates the data-construction strategy underlying subsequent M-DPO training.
F.2.1 Preference Pair Construction Strategy. … I Prompt Templates
The ablations identify long-short rollouts, triple-axis conjunctive optimization, and β=2000 as effective choices, while qualitative and user-study evidence shows strong structural fidelity alongside persistent failure modes. The paper also documents comparisons, refinement behavior, reproducibility procedures, and prompt-based data and evaluation pipelines.
- F.2.1 Preference Pair Construction Strategy.: Ground-truth winners reduce average performance by −1.02, while GPT1.5-selected winners still produce a net loss of −0.48.The distribution gap between model-generated TikZ and human-authored reference code limits the learnability of out-of-distribution preference signals.
- F.2.1 Preference Pair Construction Strategy.: The long-short rollout strategy supplies the essential preference data because alternatives yield degenerate or overly narrow contrastive signals.Without long-short rollouts, preference optimization does not improve over the SFT baseline.
- F.2.3 KL Divergence Penalty: All three β settings exceed SFT, while β=2000 achieves the highest weakest-axis gains: A +1.82 and T +3.16.Text is most sensitive to β; small values under-penalize weak-axis errors, whereas large values saturate the gate.
- G Comparison with Code-Based Baselines: Code-based baselines score 35.81 / 38.21 versus SciForma’s 69.51 Avg, with failures involving cluttered layouts, arrow endpoint mismatches, and rendering breakdowns.The comparison supports pixel-space generation with structural supervision over syntactic code synthesis for free-form scientific diagrams.
- H.1.1 Comparison with Proprietary Models.: SciForma-9B supports arbitrary aspect ratios, whereas GPT-Image-1.5 is limited to 1024×1024, 1024×1536, and 1536×1024 canvases.Fixed canvases can leave blank margins when diagram content does not match the selected proportions.
- H.1.2 Comparison with Open-Source Models.: Compared with open-source baselines, SciForma-9B avoids malformed shapes, misplaced or missing arrows, and dark backgrounds, though FLUX models offer more pleasing color schemes.SciForma’s main remaining aesthetic weakness is flat coloring compared with polished human-authored diagrams.
- H.2 Iterative Refinement and Alignment: Localized GPT-5.4-guided edits accurately redraw designated regions, while M-DPO reduces structural hallucinations, repairs arrow connections, and improves text layouts over SciForma-Base.These results highlight spatial control, iterative refinement, and improved alignment with ground truth.
- H.3 Failure Cases; H.4 User Study Details: Persistent failures include garbled rotated text, spurious or missing arrows in dense topologies, and bounding-box cropping artifacts during iterative editing.The paper proposes rotated-text data, graph-level topology constraints, and context-aware region selection as future remedies; the user study evaluates 50 prompts across Simple, Medium, and Hard difficulty.