PaperlayerView exact revision on arXiv

Source-linked AI summary

Chart2SVG: Editable SVG Generation from Raster Chart Images

Jinning Cui, Lu Chen, Haoyan Shi, Yue He, Chenglong Wang, Mengyu Zhou, Weidong Huang, Yunhai Wang

arXiv:2608.26544v1cs.LG

TL;DR

Static rasterization makes charts difficult to reuse because it removes underlying structure and data relationships. Chart2SVG reconstructs semantically structured, editable SVGs using a canonicalized 33K-sample corpus, chart-aware training, rendering-aware post-training, and a Chart Structure Graph; evaluations report strong reconstruction fidelity and editing utility, with scope limited by Beagle+’s emphasis on foundational marks.

  • Problem

    Raster chart images lose underlying structure and data relationships, making accurate reuse and modification difficult.

  • Method

    Chart2SVG combines the 33K-sample Beagle+ corpus, chart-specific semantic modeling, rendering-aware training, and a Chart Structure Graph for structured SVG recovery and coordinated editing.

  • Results

    Chart2SVG demonstrates strong reconstruction fidelity and practical editing utility across reconstruction benchmarks, ablations, and downstream editing tasks.

  • Takeaways & Limitations

    The reconstructed representation supports language-guided analysis, semantic editing, chart reuse with new data, multi-chart composition, and interactive repurposing.

  • Takeaways & Limitations

    Chart2SVG’s capabilities reflect Beagle+, whose data are dominated by rectangles, lines, and circles, with complex marks comprising only ≈5%.

Abstract

from arXiv · show

We present Chart2SVG, a multimodal large language model that converts static raster charts into structurally organized, semantically enriched SVGs that support programmatic editing. By incorporating chart-specific semantic tokens into a vision-language model, Chart2SVG captures both geometric primitives and their functional roles. To support robust structural recovery, we introduce Beagle+, a dataset of 33K canonicalized and structurally distilled chart samples. Our approach combines specialized training objectives with a rendering-aware post-training phase, producing SVGs that are both visually accurate and structurally consistent. To facilitate higher-level manipulations, we construct a Chart Structure Graph (CSG) that exposes visual dependencies, enabling tasks such as interactive exploration, chart repurposing, and layout reuse. Experiments show that Chart2SVG substantially outperforms baselines in reconstruction fidelity and downstream editing utility, advancing the development of intelligent and interactive visualization tools.

1 INTRODUCTION

Chart2SVG addresses the difficulty of reusing rasterized charts by reconstructing editable, semantically structured SVGs and exposing their dependencies for coordinated manipulation. The framework combines a canonicalized chart corpus, chart-aware modeling, and a structure graph, with evaluations showing strong reconstruction fidelity and practical editing utility.

  • Rasterization removes chart structure and data relationships, making reuse and modification difficult.
  • Existing vectorization and chart-to-code methods recover visual paths, structured data, or executable code but do not directly preserve original chart SVG structure.
  • Chart2SVG reconstructs structural SVG charts from raster inputs while capturing geometric primitives and their underlying semantics.
  • The Chart Structure Graph represents dependencies among semantic SVG components, enabling coordinated edits such as recoloring marks with corresponding legend updates.
  • Evaluations use reconstruction benchmarks, ablations, and downstream editing tasks to demonstrate strong fidelity and practical editing utility.
  • Beagle+ provides a 33K-sample canonicalized chart SVG corpus for training Chart2SVG on more regularized structural representations.

2 RELATED WORK

Prior research has recovered chart data, specifications, or generic vector graphics from raster images, but editable and visually faithful SVG reconstruction remains unresolved. Existing reuse systems also generally depend on structured vector sources, leaving raster-chart reuse insufficiently addressed.

  • Chart reverse engineering: Chart reverse-engineering systems recover chart structures, visual encodings, or underlying data using classification, mark extraction, OCR, learned detectors, and multimodal reasoning.
  • Chart reverse engineering: Existing methods typically produce tables or specifications rather than editable vector graphics, leaving faithful and reusable SVG reconstruction open.
  • Vector graphics generation: Classical and neural vectorization methods convert raster inputs into scalable vector representations, including SVG programs composed of geometric primitives and drawing commands.
  • Vector graphics generation: Multimodal SVG-generation models improve code generation and semantic representation, but prior work targets generic graphics without chart-specific structures such as axes, legends, and grouped marks.
  • Visualization reuse: Reuse research transforms existing visualizations into reusable templates, while systems such as DIVI and Mystique recover semantic components and layout relationships from SVG charts.
  • Visualization reuse: Most reuse systems assume structured vector sources and cannot operate on raster images.

3 CHART2SVG

Chart2SVG reconstructs raster charts as canonical, structured SVG sequences using a chart-oriented VLM and staged training. Its data preparation reduces representation variability, while rendering-aware optimization improves alignment between generated code and rendered charts.

  • Data Preparation: Chart2SVG canonicalizes heterogeneous SVGs while preserving hierarchical structure and primitive semantics for editable chart reconstruction.The pipeline standardizes canvas, styles, geometry, and structure rather than flattening SVGs into geometry-only representations.
  • Data Preparation: The curated Beagle+ dataset contains 33K valid examples after filtering SVG sequences exceeding the 8192-token context limit.Beagle+ is derived primarily from Beagle through the chart-oriented canonicalization pipeline.
  • Model and Training: The model predicts canonicalized SVG token sequences from raster images using a pretrained vision-language backbone and structured SVG tokenization.SVG-specific tokens cover structural containers, graphical primitives, visual attributes, and geometric operations.
  • Model and Training: Training combines supervised fine-tuning for SVG grammar and chart-to-vector mapping with rendering-aware reinforcement learning for perceptual and structural alignment.The second stage evaluates generated candidates using code validity and visual fidelity rewards, with KL regularization against a reference model.

4 SEMANTIC STRUCTURE-AWARE CHART MANIPULATION

Chart manipulation requires linking ambiguous SVG primitives to chart components and modeling dependencies among those components. Chart Structure Graphs provide a semantic interface where language-guided edits can propagate across data and layout relationships.

  • Motivation: Individual SVG primitives are ambiguous because identical shapes can represent different chart components, such as axes, gridlines, ticks, bars, legends, or backgrounds.Heuristic labeling methods address this component-identification problem before graph construction.
  • Motivation: Chart elements are tightly coupled, so modifying one component can require coordinated updates to legends, spacing, alignment, and other dependent elements.The paper identifies these interdependencies as a largely unaddressed challenge in chart manipulation.
  • CSG Construction: The CSG represents a labeled SVG as G = (V,E), exposing chart components, mapped attributes, and topological dependencies to the editing model.An LLM infers graph relationships under a predefined constraint schema rather than generating arbitrary edges.
  • Constraint Types: Data constraints preserve scale mappings, value-dependent geometry, and category-consistent visual encodings across marks, axes, and legends.Examples include rescaling axes when data ranges change and maintaining shared colors for elements in the same category.
  • Constraint Types: Spatial constraints maintain alignment, group spacing, stacking order, and overlap avoidance among chart elements.These rules govern layout relationships using positions, bounding boxes, and grouping relationships.
  • Language-Guided Manipulation: Graph-grounded editing targets existing chart vertices and propagates local changes through dependency edges to preserve semantic correctness and visual coherence.Bidirectional propagation can adjust axes, spacing, legends, and related elements after commands such as changing bar width or highlighting a category.

5 EVALUATION

Chart2SVG is evaluated with foreground-aware visual and structural metrics on in-distribution and out-of-distribution charts, alongside data-accuracy, qualitative, and editing studies. It achieves reliable reconstruction and measurable downstream editing gains, while cross-dataset generalization remains challenging.

  • Metrics and Baselines: Foreground-aware evaluation crops chart content before comparing rasterized predicted and ground-truth SVGs, reducing the influence of blank margins.Foreground pixels are identified by color difference from the estimated background using tol=5.
  • Metrics and Baselines: The evaluation reports PSNR, SSIM, LPIPS, Foreground IoU, Edge Consistency, and Failure Rate to measure fidelity, structure, chart overlap, critical edges, and renderability.The chart-specific metrics emphasize thin elements such as axes, ticks, gridlines, legends, and line marks.
  • Quantitative Results: Chart2SVG variants substantially outperform prior baselines on three Beagle+ subsets and achieve a zero failure rate on all in-distribution subsets.The results indicate improvements in both visual similarity and the basic validity of generated SVG charts.
  • Quantitative Results: On ChartBlocks, Chart2SVG-8B leads all metrics with 15.56 PSNR, 0.71 SSIM, and 0.26 LPIPS, while remaining strongest on structural metrics across Fusion and Plotly.On Plotly, Chart2SVG-4B leads SSIM and IoUfg, whereas Chart2SVG-8B leads LPIPS and edge consistency; both have zero failures.
  • Quantitative Results: On VisAnatomy, Chart2SVG-8B leads learning-based methods across all five metrics, with 0.78% failure rate, 12.28% higher PSNR than GPT-4o, and 24.48% higher edge consistency.It produces semantically structured, directly editable SVG output, while cross-dataset generalization remains challenging.
  • Quantitative Results: Chart2SVG-8B achieves bar-chart numerical accuracy of 0.1807 versus at most 0.1545 for baselines, while results vary across pie and line charts.The evaluation measures array-length matching and numerical accuracy over successfully matched data.
  • Ablation Studies: Ablations show that removing SVG grammar tokens lowers PSNR from 14.02 to 12.23 and raises failure rate from 0.52% to 14.81%.Removing GRPO causes larger drops, with PSNR falling from 14.02 to 6.81 and SSIM from 0.6951 to 0.4672.
  • Ablation Studies: Removing the foreground IoU reward raises failure rate from 0.52% to 9.87% and lowers IoUfg from 0.2640 to 0.1795.The foreground IoU term complements rendering feedback by preserving the spatial extent of chart content.

6 CASE STUDY

Chart2SVG supports language-driven chart exploration, semantic querying, repurposing, and layout reuse by reconstructing SVG structure and exposing dependencies through the CSG. These structured representations preserve numerical, geometric, and spatial relationships during downstream edits.

  • Interactive Exploration: Reconstructed SVGs let the LLM recover exact numerical values from charts, whereas raster-only inputs produce vague approximations and hallucinated ranges.
  • Interactive Exploration: Structured visual marks enable accurate conditional highlighting of bars whose actual values exceed expected values, unlike raster-based misidentification.
  • Interactive Exploration: CSG dependencies let rescaling update geometric paths while maintaining axis–mark consistency; without CSG, edits change labels but leave geometry unchanged.
  • Layout Reuse: CSG-guided layout reuse preserves x-axis alignment when a normalized stacked bar chart receives new data, whereas the baseline fails.
  • Chart Repurposing: CSG-guided repurposing converts a pie chart into a pictorial bar chart while preserving numerical fidelity; without CSG, generic geometries and hallucinated data appear.

7 DISCUSSION AND CONCLUSION

Chart2SVG transforms raster charts into semantically structured SVGs for programmatic and language-guided editing, combining rendering-aware training with a Chart Structure Graph. Its generalization remains limited by training data dominated by foundational marks and constrained chart complexity.

  • Chart2SVG combines specialized training objectives and rendering-aware post-training to produce visually accurate, structurally organized SVGs for direct manipulation.
  • The CSG makes latent visual dependencies explicit, supporting interactive data exploration and automated chart repurposing.
  • Generalizability: Complex marks comprise only ≈5% of Beagle+, while D3 charts and SVGs beyond 8192 tokens are excluded, limiting coverage of customized charts.
  • Generalizability: Generalization to irregular layouts, degraded screenshots, and unconventional mark types remains limited under the current training distribution.

A FULL DEFINITION OF SVG-SPECIFIC TOKENS

Chart2SVG extends the tokenizer with 115 SVG-specific tokens that represent chart hierarchy, primitives, geometric commands, and visual attributes. This tokenization compresses sequences and guides the model toward structured vector representations rather than unstructured XML text.

  • The tokenizer adds 115 SVG-specific special tokens to improve modeling of charts’ semantic and hierarchical structure.
  • Structural Containers & Anchors: Structural container and anchor tokens preserve hierarchical DOM groupings that support CSG extraction and layout reuse.
  • Graphical Primitives: Graphical primitive tokens represent bounded shapes, paths, and typography, guiding syntactically valid primitive components.
  • Geometric Opcodes: Geometric opcode tokens distinguish absolute and relative path commands for reconstructing customized shapes and continuous data boundaries.
  • Visual Channel & Geometry Attributes: Visual-channel and geometry-attribute tokens separate data-linked styling from spatial positioning through coordinates, colors, typography, and transformations.
  • Table 6 provides the complete list and descriptions of the 115 SVG-specific special tokens.

B PROMPT DESIGN FOR CHART STRUCTURE GRAPH CONSTRUCTION AND MANIPULATION

The prompting framework uses a two-stage process: extract a scene graph from parsed SVG structure, then apply that graph and the SVG code to analysis or editing requests. Editing prompts require dependency tracing, quantitative updates, and preservation of structural relationships.

  • Scene Graph Construction: The first prompting stage extracts a Scene Graph by inferring topological dependencies and structural hierarchies from the parsed SVG.
  • Chart Reasoning and Manipulation: The second stage uses the Scene Graph and raw SVG code to switch between numerical analysis and structure-aware visual editing.
  • Editing Constraints: The prompts enforce valid axis ranges and complete SVG output while preserving structural relationships.
  • Editing Constraints: Editing instructions require identifying affected elements, tracing layout_cascade dependencies, calculating new quantitative values, and applying edits in a consistent order.
  • Motivation: The customized infographic case illustrates why predefined plotting primitives struggle to preserve rose-shaped glyph geometry and visual style.

Reward Design

Chart2SVG combines foreground-aware overlap, normalized pixel fidelity, and structural similarity into a visual reward, then balances it with code validity. The design tolerates minor rendering offsets while prioritizing faithful chart reconstruction.

  • PSNR: PSNR is normalized by a 40 dB ceiling before aggregation, while standard unbounded decibel values remain reserved for reported benchmark results.For the models, PSNRnorm operates in [0.05,0.39], preventing it from dominating the combined reward.
  • SSIM: SSIM measures global structural similarity between Gaussian-smoothed grayscale images using means, variances, and cross-covariance.The formulation uses Gaussian smoothing with σ=1.5 and an absolute cross-covariance to keep the reward non-negative.
  • Combined Reward: The visual reward averages the three normalized terms uniformly before combining them with code-level reward weights λvis = 0.6 and λcode = 0.2.This weighting prioritizes faithful visual reconstruction while penalizing invalid or excessively verbose SVG outputs.
  • Design Rationale: Unlike general visual SVG reinforcement learning, Chart2SVG uses foreground-aware IoU because chart backgrounds could otherwise dominate perceptual similarity.The reward design is tailored to chart reconstruction rather than generic illustration synthesis.

D BASELINE SETTINGS AND COMPARATIVE ANALYSIS

The baselines are evaluated zero-shot under unified settings, while comparative analysis attributes their weaker chart reconstruction to domain mismatch and inefficient path representations. ChartCoder additionally faces limited robustness on diverse real-world styles, and precise numerical recovery remains an open field-wide problem.

  • Baseline Settings: All baselines use officially released checkpoints without Beagle+ or chart-specific fine-tuning and are evaluated zero-shot with a unified prompt.Inputs are resized to 512×512, and results average four independent runs per sample.
  • Comparative Analysis: Generic vector models lack chart-specific training and represent shapes mainly as Bézier-based <path> commands, producing fragmented outputs and high failure rates.Charts contain axes, ticks, gridlines, legends, and multi-series marks that exceed the complexity of typical icon datasets.
  • Comparative Analysis: ChartCoder’s synthetic Chart2Code-160K training data covers only Matplotlib and Seaborn, limiting robustness on real-world charts with diverse styles and custom layouts.The analysis reports failure to faithfully represent rose-shaped glyphs in a customized pictorial infographic.
  • Comparative Analysis: GPT-4o remains competitive but falls short of Chart2SVG on most metrics, particularly structural fidelity and failure rate.The passage attributes this difference to chart-specific training and rendering-aware optimization.
  • Scope and Limitations: No current chart-to-SVG or chart-to-code system embeds a dedicated robust mechanism for precise numerical recovery.The paper identifies OCR integration, either post hoc or architectural, as a key direction for future work.

4. Visual Channel & Geometry Attributes

Chart2SVG produces SVG code with a clear hierarchical structure and uniform color and coordinate representations, making the reconstructed chart directly editable programmatically.

  • Structural Comparison: Chart2SVG organizes grouped bar-chart SVG code hierarchically with uniform color and coordinate representations for direct programmatic editing.Compared with the shown alternatives, its output preserves clearer structural organization.
Loading 2608.26544v1…