PaperlayerView exact revision on arXiv

Source-linked AI summary

VFIG: Vectorizing Complex Figures in SVG with Vision-Language Models

Qijia He, Xunmei Liu, Hammaad Memon, Ziang Li, Zixian Ma, Jaemin Cho, Zhongzheng Ren, Daniel S Weld, Ranjay Krishna

arXiv:2603.24575v2cs.CVcs.AI

TL;DR

Raster-to-SVG reconstruction lacks complex diagram-centric data and evaluations that capture structural correctness. VFIG addresses these gaps with a 66K-pair dataset, a structure-aware benchmark, and a VLM trained by curriculum SFT plus rendering-aware RL, achieving the strongest open-source results and over 30% improvement over the best open-source baseline.

  • Problem

    Existing SVG datasets emphasize icons and decorative graphics, while pixel- or embedding-level benchmarks do not adequately measure structural correctness in complex diagram-centric figures.

  • Method

    VFIG combines VFIG-DATA, VFIG-BENCH and VFIG-BENCH-OOD with simple-to-complex SFT followed by RL using rendering-aware rewards for diagram structure.

  • Results

    VFIG achieves the strongest open-source performance, outperforming the best open-source VLM baseline by over 30%; on VFIG-BENCH, Gemini-Judge is 78.2% versus Claude-Sonnet-4.6 at 76.7%.

  • Takeaways & Limitations

    The paper provides diagram-centric training and evaluation resources that measure layout, connectivity, editability, and rendering quality beyond pixel similarity.

  • Takeaways & Limitations

    Rendered real-world targets preserve pixel alignment but can retain structural errors, especially in arrow connectivity; connectivity-aware sample filtering remains future work.

Abstract

from arXiv · show

Scalable Vector Graphics (SVG) are essential for technical illustration and digital design, offering resolution independence and semantic editability. In practice, original vector files are frequently lost, leaving only rasterized versions (e.g., PNG, JPEG) that resist modification, while manual reconstruction is prohibitively expensive. Progress on automating raster-to-SVG conversion has been bottlenecked by two gaps: existing SVG datasets are dominated by icons and decorative graphics that lack the complexity of professional diagrams, and existing benchmarks rely on pixel- or embedding-level similarity that fails to capture structural correctness (e.g., broken connectivity, misplaced arrows). We close both gaps with paired contributions targeting diagram-centric figures (e.g., model architectures, flowcharts, schematics). For training, we introduce VFIG-Data, the largest figure-to-SVG dataset of its kind at 66K pairs, combining real paper figures converted via a describe-and-generate pipeline with programmatic diagrams that supply noise-free supervision over arrow styles, fonts, and geometry. For evaluation, we introduce VFIG-Bench, a structure-aware evaluation suite, paired with VFIG-Bench-OOD, an out-of-distribution set of figures manually curated from highly cited arXiv papers. Beyond pixel and embedding similarity, our protocol reports rubric-based VLM-Judge scores and Elo ratings from pairwise human preference evaluation. Built on these contributions, VFIG is a VLM family trained with a simple-to-complex SFT curriculum followed by RL with rendering-aware rewards. VFIG achieves state-of-the-art open-source performance, outperforming the best open-source VLM baseline by over 30%, and matches Claude Sonnet 4.6 on VFIG-BENCH: Gemini-Judge 78.2% vs. 76.7% and GPT-Judge 87.5% vs. 87.4%. It remains slightly behind the strongest proprietary models GPT-5.2 and Gemini-3.

1 Introduction

VFIG targets raster-to-SVG reconstruction for complex diagram-centric figures, addressing gaps in training data and structure-aware evaluation. It combines a 66K-pair dataset, a multi-granularity benchmark, and a VLM trained with curriculum SFT and rendering-aware RL.

  • Motivation: SVG supports resolution-independent, semantically editable technical illustrations, but original vector sources are often unavailable for complex diagrams.Such diagrams contain nested layouts, heterogeneous primitives, precise alignments, and intricate connectivity.
  • Research gap: Existing figure-to-SVG methods and datasets focus mainly on icons or simple graphics, leaving performance on complex diagrams unclear.The relevant targets include model architectures, flowcharts, and hierarchical diagrams.
  • Contributions: 66K figure–SVG pairs form VFIG-DATA, combining real-world paper diagrams with procedurally generated diagrams filtered for vectorization suitability and editable primitives.The pipeline uses VLM-based filtering, describe-and-generate conversion, and code-level filtering.
  • Contributions: VFIG-BENCH evaluates visual fidelity, structural correctness, and holistic quality through pixel-level metrics, component-level matching, and image-level VLM judgments.The benchmark includes a held-out in-distribution split and an out-of-distribution benchmark.
  • Model: VFIG uses simple-to-complex supervised fine-tuning followed by reinforcement learning with rendering-aware rewards over alignment, grouping, connectivity, and layout.The curriculum progresses from primitive generation to multi-panel and hierarchical compositions.
  • Results: VFIG achieves the strongest open-source results and exceeds the best open-source VLM baseline by over 30%.On VFIG-BENCH, Gemini-Judge is 78.2% and GPT-Judge is 87.5%, versus Claude-Sonnet-4.6 at 76.7% and 87.4%.

2 Related Work

Prior SVG and diagram-generation research includes tracing, multimodal code synthesis, reasoning, feedback, and structured parsing methods. Existing datasets and evaluations remain limited for paired, diagram-centric raster-to-SVG reconstruction.

  • Prior methods: Earlier non-LLM approaches include tracing-based vectorization, differentiable rendering, and sequential decoder models.These methods are reviewed separately from the paper’s main LLM/VLM-related discussion.
  • LLM/VLM methods: Recent LLM/VLM methods generate SVG through multimodal code synthesis, semantic command tokens, unified token-coordinate modeling, and integrated understanding, editing, and generation.Other extensions use design rationales, rendering feedback, visual self-feedback, or internal visual guidance.
  • Datasets and evaluation: Existing SVG datasets largely emphasize icons, emojis, and general SVG tasks rather than diagram-centric figure-to-SVG conversion.Paper2Fig contains many arXiv figures but lacks paired SVG code, while related systems reconstruct or parse structured visual representations.
  • Positioning: VFIG introduces paired training and evaluation resources intended to fill the gap in complex diagram-centric figure-to-SVG generation.Its focus differs from datasets curated primarily for text-aware reconstruction or general SVG generation.

3 VFIG-DATA

VFIG-DATA is a 66K-pair dataset designed for diagram-centric figure-to-SVG generation, combining real paper diagrams, procedurally generated shapes and arrows, and academic SVG data. Its pipelines use describe-and-generate conversion and filtering for vectorization suitability, structural cleanliness, and editability.

  • Dataset overview: VFIG-DATA contains 66K filtered image–SVG pairs targeting model architectures, flowcharts, schematics, and other diagram-centric figures.It is positioned as a large-scale dataset purpose-built for structured diagram generation.
  • Data sources: The dataset combines real-world complex diagrams from scientific papers with programmatically generated diagrams containing diverse shapes, connectors, and spatial layouts.These form the Complex-Diagrams and Shapes-and-Arrows subsets.
  • Complex-diagram pipeline: Real paper figures are converted through a two-step VLM pipeline that first describes geometric elements, text, layout, and relationships, then generates SVG code from the image and description.The final training target uses the rendered generated SVG to align the SVG image with the input raster.
  • Limitation: Rendered targets guarantee pixel-level image–SVG alignment but do not eliminate structural errors, especially residual arrow-connectivity problems.The paper partially mitigates these issues with code filtering and noise-free Shapes-and-Arrows data, leaving connectivity-aware filtering for future work.
  • Procedural data: Procedural Shapes-and-Arrows data provides noise-free supervision for arrow styles, fonts, fill patterns, and geometric variations, with structured metadata for each visual attribute.Diagrams are synthesized from 19 layout templates with randomized shapes, arrows, fonts, and styles.
  • Filtering: Image filtering removes figures dominated by natural images, equations, plots, or tables, while code filtering removes path-dominated or structurally noisy SVG outputs.The filters target vectorization suitability, token efficiency, and downstream editability.
  • Training mixture: The training mixture also incorporates filtered SVG-Diagrams and Molmo2-Diagram data to strengthen primitive-aware generation.The mixture is summarized using structural complexity, element complexity, SVG cleanliness, and path dominance.

4 VFIG Model

VFIG maps figure images to structured SVG programs using staged supervised fine-tuning and reinforcement learning. Rendering-aware rubric rewards target presence, layout, connectivity, and details while penalizing invalid SVGs.

  • Model objective: Given a figure image, VFIG generates a structured SVG program that reconstructs visual content while preserving compositional structure.Training consists of supervised fine-tuning followed by reinforcement learning with rendering-aware rewards.
  • SFT curriculum: Simple-to-complex SFT first teaches primitive generation and basic layout understanding, then trains on complex diagrams for compositional reasoning.Direct SFT on complex diagrams is described as destabilizing because it combines low-level and high-level learning demands.
  • Rendering-aware RL: RL renders sampled SVGs with CairoSVG and scores them on Presence, Layout, Connectivity, and Details using a rubric-based VLM judge.The reward is the average of the four axis scores, and invalid SVGs receive zero reward.
  • Optimization: GRPO group-normalizes candidate rewards into advantages and updates the policy with KL regularization against the frozen SFT reference.This propagates structural credit jointly across the four rubric axes.
  • Reward design: The rubric judge correlates with human ratings at r=0.89 overall on 100 annotated examples, supporting its use beyond pixel metrics.The paper motivates this choice because misplaced arrows can have little effect on SSIM while changing diagram semantics.

5 Experiments

VFIG is evaluated across diagram-centric benchmarks using automatic, VLM-judge, code-quality, and human-preference metrics, with ablations isolating curriculum SFT and rendering-aware RL. It achieves the strongest open-source performance, generalizes to curated out-of-distribution figures, and narrows the gap to proprietary models.

  • Experimental setup: VFIG is evaluated on three diagram-centric benchmarks plus VFIG-BENCH-OOD, using pixel, embedding, rubric-based VLM-judge, code-quality, and human Elo metrics.VFIG-BENCH-OOD contains 198 manually curated figures, while Table 2 covers three in-distribution benchmarks.
  • Main results: VFIG achieves the best open-source score on every reported metric except SVG-Diagram SSIM, where Starvector-8B leads.The SFT+RL models obtain these results across the three benchmark datasets.
  • OOD generalization: 76.8 averaged VLM-Judge is achieved by VFIG on VFIG-BENCH-OOD, exceeding the best open-source baseline by over 30 points while remaining behind Gemini-3-Pro.Human Elo ranks VFIG above Claude-Sonnet-4.6, whereas the VLM-judge ranking reverses those two positions.
  • Ablations: Simple-to-complex curriculum SFT improves rendering and VLM-judge scores, while adding RL produces consistent gains across all metrics and benchmarks.On the curriculum ablation, SFT raises Render by +18.4 and Gemini/GPT scores by +2.9/+2.0; after RL, the curriculum retains SSIM and judge-score gains.

6 Conclusion and Limitations

VFIG advances editable diagram-centric figure-to-SVG generation through paired dataset and benchmark contributions, while documenting scope boundaries and remaining structural errors. The appendix details data construction, evaluation setup, and procedural choices supporting these contributions.

  • Contributions: VFIG-DATA combines real-world paper diagrams with programmatically generated shapes, connectors, and layouts for complex figure-to-SVG generation.The dataset construction includes image filtering, describe-and-generate conversion, and code-level filtering favoring semantic primitives.
  • Contributions: VFIG-BENCH evaluates layout, connectivity, editability, and rendering quality beyond pixel similarity.The broader benchmark suite includes held-out complex diagrams and manually curated evaluation figures, while the appendix describes complementary automatic and human evaluations.
  • Limitations: The approach remains limited by fine-grained connector and arrowhead errors and by stylistic priors inherited from Gemini-3-Pro-rendered real-world targets.The paper mitigates these issues with programmatic supervision, cross-family judging, and human Elo evaluation, but identifies human-authored ground truth as future work.
  • Dataset construction: The data pipeline collects figures from Paper2Fig and a large crawl of recent arXiv papers before filtering them into high-quality target diagrams.The crawl processed 259,073 candidates, with 52,698 of 105,494 non-plot candidates retained as high-quality diagrams.
  • Pipeline design: A two-stage describe-and-generate pipeline was selected after comparison with single-stage generation, with modest but consistent metric improvements and higher render success.The appendix reports approximately 0.02 higher SSIM from a base of about 0.65 and a 99% render success rate for the two-stage pipeline.
  • Programmatic data: Programmatic diagrams assign structured layouts, visual styles, and connection densities to provide controlled variation for shapes and arrows.Layouts use 19 templates with randomized jitter, while shapes receive randomized fill styles and diagrams sample directed-edge densities.

A.7 Rule-Based Benchmark for VFIG-DATA-Shapes-and-Arrows.

The rule-based benchmark scores generated SVGs against structured metadata by separately evaluating shapes and arrows across visual, geometric, semantic, and connectivity attributes. Scores are normalized to [0,1] and aggregated into shape, arrow, and overall composites.

  • Shape evaluation: Generated shapes are matched by label and scored across type, colors, styles, font, aspect ratio, and relative position.The shape composite aggregates nine visual attributes from the structured ground-truth metadata.
  • Arrow evaluation: Arrows are matched by endpoint proximity and scored for connectivity, arrowhead properties, curvature, and color.The arrow composite evaluates seven attributes, including source and destination correctness and arrowhead size.
  • Scoring: The benchmark aggregates shape and arrow scores into RS and RA, then averages them into R after applying extra-element penalties.Table 5 reports these composites over 500 programmatically generated samples.
  • Structural checks: The protocol explicitly measures endpoint correctness, occlusion-aware arrowhead presence, curvature, overlap violations, labels, and accepted SVG element types.These criteria target structural and semantic correctness rather than only rendered pixel similarity.

A.8 Standalone Rule-Based Evaluation and Benchmarks

Standalone rule-based evaluation extends shape-and-arrow benchmarking to external SVG datasets by extracting ground-truth attributes directly from SVG files. The method is useful but remains an approximation because markup-based extraction can misrepresent visual quality.

  • Evaluation protocol: Standalone SVG attribute extraction closely matches JSON-based evaluation, with mean absolute gaps of at most 0.003 for RS, 0.001 for RA, and 0.002 for R.This validation supports using extracted SVG attributes when structured metadata is unavailable.
  • External benchmarks: The standalone protocol benchmarks models on arXiv figures and Molmo-generated diagrams using unchanged metrics and weightings.Table 6 reports composite scores for these external benchmarks.
  • Limitations: Rule-based scores can under- or over-estimate quality because unconventional valid structures may be penalized and invisible or malformed elements may receive credit.The authors therefore interpret the scores as approximations rather than exact measures of fidelity.
  • Results: VFIG SFT+RL is the strongest open-source model across all three external benchmarks, leading on R, RS, and RA and approaching closed-source models on Molmo.Gemini-3-Pro and GPT-5.2 perform on par overall in the reported comparisons.
  • Per-metric results: Shape fidelity is harder than arrow fidelity, while fonts and fill styles remain consistent weak points across models.The reported analysis links low fill-style scores in fine-tuned models with a trade-off between structural accuracy and visual fidelity.

B.1 Training Setup

The training setup combines parameter-efficient supervised fine-tuning with reinforcement learning initialized from a simple-to-complex SFT checkpoint. Training and reward design target both SVG generation likelihood and rendered structural quality.

  • SFT: SFT uses Qwen3-VL-4B-Instruct with LoRA applied to language-model layers while freezing the vision encoder and projector.The configuration uses rank 64, an 8192-token sequence limit, three epochs, and bf16 precision.
  • RL: RL initializes from the simple-to-complex Qwen3-VL-4B SFT checkpoint and optimizes the policy with GRPO while keeping the vision tower frozen.RL was conducted only on the Qwen3-VL-4B backbone because of compute and time constraints.
  • Reward design: The rubric reward averages presence, layout, connectivity, and details scores from a VLM judge evaluating rendered SVGs against ground-truth images.The axes cover required elements, spatial arrangement, endpoint correctness, and text or styling fidelity.
  • Reward design: Pixel-level rewards combine Gemini judgment with image, edge-map, and SSIM similarity signals, including blurred Canny edges to reduce sensitivity to one-pixel shifts.The ablation compares this scalar reward variant with the rubric-based reward design.
  • Evaluation rationale: Rubric-based VLM judgment is motivated by pixel metrics assigning high similarity to structurally incorrect arrows or misplaced labels.On 100 annotated examples, Gemini judge scores correlate with human ratings at Pearson r=0.89 overall.
  • SFT: The simple-to-complex curriculum trains primitive-level generation before multi-panel and hierarchical compositions.The curriculum is evaluated alongside backbone and parameter-efficient fine-tuning choices in the ablation studies.

C.1 SFT Ablation

The SFT ablations show that backbone choice, curriculum design, adaptation capacity, and target modules materially affect diagram reconstruction quality. Qwen3-VL with simple-to-complex SFT provides the preferred balance for subsequent RL experiments.

  • Backbone: Qwen3-VL consistently outperforms earlier VLM backbones across datasets, especially on VisualSim and VLM-Judge semantic metrics.Increasing model size from 4B to 8B adds moderate gains across most metrics.
  • Curriculum: The simple-to-complex SFT curriculum improves rendering reliability and semantic alignment on compositionally complex datasets.It separates primitive-heavy diagram pretraining from realistic figure fine-tuning.
  • Parameter-efficient fine-tuning: LoRA rank 64 achieves the best overall performance on VFIG-Bench and Molmo2-Diagram, particularly for VisualSim and VLM-Judge.Lower ranks remain competitive on some datasets.
  • Target modules: Adapting the language model alone yields the strongest or most balanced results across datasets compared with also tuning the projector or vision encoder.The language component produces the structured SVG programs.

C.2 RL Ablation

The RL ablations identify reward composition, SFT initialization, and backbone size as important design choices. Full rendering-aware rewards and simple-to-complex initialization improve structural and semantic quality, while larger models trade code cleanliness for perceptual quality.

  • Reward design: The full reward achieves the strongest judge-based performance by jointly optimizing presence, layout, connectivity, and details.The relative importance of reward components varies across benchmarks.
  • Reward design: Pixel-level objectives can improve SSIM and LPIPS on some datasets without improving judge-based structural scores.This indicates that pixel reconstruction is not always aligned with structural correctness.
  • Reward design: Restricting RL to curated hard samples does not outperform the full reward across the three datasets.The results suggest that diverse training data benefits RL stability more than hard-example focus.
  • SFT initialization: Simple-to-complex SFT initialization yields stronger semantic and structural performance after RL than single-stage initialization on VFIG-Bench and SVG-Diagram.Both initializations benefit from RL, while simple-to-complex remains preferable for judge-based structural quality.
  • Backbone size: 8B improves VisualSim and VLM-Judge across datasets, whereas 4B often produces cleaner SVG code and remains competitive on rendering metrics.The comparison supports Qwen3-VL-4B with simple-to-complex SFT as a balanced default RL setting.

D Human Evaluation

The human evaluation compares VFIG and major proprietary and open-source systems through blinded pairwise judgments on held-out out-of-distribution figures. VFIG with SFT and RL ranks highly, strongly exceeds open-source baselines, and remains competitive with Claude Sonnet 4.6.

  • Evaluation setup: Five authors collected 4,158 paired annotations across 21 model pairs on VFIG-BENCH-OOD, whose images were excluded from training.Annotators compared two reconstructions against the ground-truth figure using four outcome labels.
  • Metrics: The protocol reports Elo rankings, quality-aware rates, and pairwise outcomes, treating both-good and both-bad cases as ties for Elo.Good Rate equals Win plus Both Good, while Decisive Win Rate excludes ties.
  • Results: VFIG (SFT+RL) ranks third in Elo, ahead of Claude-Sonnet-4.6 and other open-source baselines but below GPT-5.2 and Gemini-3-Pro.Its Elo score is 1577.5 versus 1562.4 for Claude-Sonnet-4.6.
  • Results: 46.0% win rate and 55.5% good rate for VFIG (SFT+RL) exceed VFIG (SFT) at 42.3% and 50.7%, respectively.The SFT+RL variant also far exceeds Qwen3-VL-4B and StarVector in good rate.
  • Results: VFIG (SFT+RL) wins 81.6% of trials against Qwen3-VL-4B and 91.4% against StarVector, while Claude-Sonnet-4.6 has a slightly higher win rate.Against VFIG (SFT), it wins 39.9% versus 25.3%, with a 61.2% decisive win rate excluding ties.

E Additional Qualitative Results

Qualitative examples show that VFIG generally preserves global diagram organization, layout, and major spatial relationships while improving validity and rendering robustness over open-source baselines. Performance nevertheless varies across benchmarks, and strong closed-source models retain higher visual fidelity.

  • General behavior: VFIG generally preserves high-level diagram organization and spatial relationships, producing outputs that often resemble the input figures visually.This strength is observed across the qualitative benchmark examples.
  • VFIG-Bench: VFIG substantially improves rendering success over base Qwen3-VL-4B-Instruct, which frequently produces invalid or nonrenderable SVG outputs.StarVector-8B and OmniSVG often produce repetition, noisy primitives, or rendering failures on complex scientific diagrams.
  • Remaining gap: Gemini3-Flash achieves very high visual fidelity on VFIG-Bench, leaving room for further improvement despite VFIG’s robustness gains.The qualitative examples therefore separate structural robustness from the strongest closed-source visual fidelity.
  • Cross-benchmark behavior: On Molmo2-Diagram and SVG-Diagram, VFIG more consistently produces valid SVGs and preserves global structure, layout, and major element relationships than open-source baselines.These benchmarks include stylized diagrams and other graphics beyond the complex scientific figures in VFIG-Bench.

F Failure Cases and Limitations

VFIG’s failures concentrate in fine-grained local structures, while broader generalization remains limited outside the structured diagram distributions emphasized during training.

  • Local visual fidelity remains VFIG’s main limitation, especially for thin lines, arrowheads, small annotations, and precise geometric structures.These errors can persist even when global layout and major object relationships are preserved.
  • 3D shapes and perspective-like objects are especially challenging, often producing incorrect geometry or missing structural elements.
  • VFIG struggles on stylized, sparse, logo-like, icon-like, object-drawing, and chart-like inputs, where it may simplify shapes or omit small components.These cases extend beyond canonical diagrams represented in the training distribution.
  • The reward design explores only limited pixel-level objectives, leaving open whether alternative pixel rewards or weighting ratios could improve fine-grained fidelity.
  • Reliance on Gemini as a visual judge and equal weighting of presence, layout, connectivity, and details may not fully align reward optimization with human preferences.Alternative component weights, prompts, or reward dimensions remain possible improvements.
  • Training data primarily targets complex diagram reconstruction, providing limited coverage of icons, sketches, logos, and other non-canonical graphics.The authors connect this mismatch to failures on Molmo2-Diagram and SVG-Diagram.
Loading 2603.24575v2…