Source-linked AI summary
ChartRevive: Reconstructing Data Visualizations from Chart Images Using MLLM
Yuki Ueno, Aditeya Pandey
TL;DR
Recovering both data and visual design from static chart images remains labor-intensive, and design extraction is less studied than data extraction. ChartRevive benchmarks five MLLMs across five chart types and combines extraction with interactive verification. Textual and categorical information is generally more reliable to extract than numeric and spatial information, while GPT-5.4 performs best overall among the evaluated models.
Problem
Recovering data and visual design from static chart images is a labor-intensive manual process, while prior work has focused more on data than design extraction.
Source: §1 · p. 1Method
Results
Textual and categorical information was generally extracted reliably, whereas numeric and spatial information remained challenging, and GPT-5.4 achieved the best overall performance among the evaluated models.
Source: §1 · p. 2Takeaways & Limitations
ChartRevive supports chart reconstruction through overlay-based verification, real-time rebuilding, and direct editing of extracted data and design properties.
Source: §7 · p. 4The benchmark uses synthetic charts that do not fully capture the visual artifacts, compression effects, and stylistic diversity of published charts.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Static chart images are widely used in scientific publications, business reports, and presentations, yet recovering both the underlying data and visual design from chart images remains a labor-intensive manual process, making them difficult to reuse. While prior work has primarily focused on data extraction, the extraction of visual design specifications, including colors, marker shapes, and axis configurations, remains underexplored. To identify a suitable model for chart reconstruction, we systematically benchmark five multimodal large language models (MLLMs) across five basic chart types on both data and design extraction tasks. Our evaluation shows that textual and categorical information can generally be extracted reliably, whereas numeric and spatial information remain challenging. Among the evaluated models, GPT-5.4 achieves the best overall performance and is adopted as the backbone of our system. Guided by these findings, we present ChartRevive, a mixed-initiative system that combines MLLM-based extraction with an interactive verification interface, supporting users to efficiently inspect, correct, and refine reconstructed charts through overlay-based verification and real-time rebuilding.
1 INTRODUCTION
Reconstructing static charts requires recovering both their data and visual design, but design extraction has received comparatively little attention. ChartRevive pairs MLLM extraction with interactive verification, guided by a benchmark showing that textual and categorical information is more reliable to extract than numeric and spatial information.
1 INTRODUCTION
- Static chart reuse requires recovering both underlying data and visual design, a labor-intensive process commonly performed manually.Design properties include colors, marker shapes, line styles, and axis configurations.Source: §1 · p. 1
- Prior chart-image research has focused primarily on data extraction, while visual design extraction supports consistent restyling and verification of reconstructed charts.Preserving original design matters when updating data across a collection of figures.Source: §1 · p. 1
- A formative study with professional designers identified difficult verification, cumbersome iterative refinement, and fragmented workflows in purely MLLM-based approaches.ChartRevive responds with MLLM-based extraction and an interactive interface for inspecting, correcting, and refining results.Source: §1 · pp. 1–2
- Textual and categorical information was generally extracted reliably, whereas numeric and spatial information remained challenging; GPT-5.4 performed best overall among the evaluated models.The authors selected GPT-5.4 as ChartRevive’s backbone.Source: §1 · p. 2
- The paper contributes a formative study, a benchmark of five commercial MLLMs across five chart types, and a mixed-initiative reconstruction system.The system integrates extraction with interactive verification.Source: §1 · p. 2
2 RELATED WORK
Prior chart-image research has addressed data extraction more extensively than visual design specification extraction. ChartRevive benchmarks both kinds of extraction and uses the findings to develop a system with human verification.
2 RELATED WORK
- Earlier data-extraction approaches paired OCR with chart-specific heuristics; WebPlotDigitizer requires point-by-point annotation, while ChartDetective is limited to vector graphics.DePlot converts chart images into linearized tables for LLM reasoning and assumes visible data labels.Source: §2 · p. 2
- ChartLlama, ChartReformer [17], ChartMimic, and OneChart support broader chart understanding, editing, or conversion tasks, but do not systematically assess design-property extraction reliability.The unassessed properties include colors, marker shapes, and spatial configurations.Source: §2 · p. 2
- ChartRevive provides a systematic benchmark of five commercial MLLMs for both data and design extraction across five basic chart types, then builds a mixed-initiative system with human verification.The authors identify design-property extraction as comparatively underexplored despite its role in faithful reconstruction and consistent restyling.Source: §2 · p. 2
3 FORMATIVE STUDY
A formative study with visualization designers surfaced verification, editability, and workflow needs in reconstructing charts from static images. The authors translated those findings into goals for accurate, editable, and efficient reconstruction.
3 FORMATIVE STUDY
- Three visualization designers who regularly reconstruct static charts for external data communication participated in 30-minute semi-structured interviews documented in interviewer notes.The study was conducted to understand reconstruction challenges and requirements.Source: §3 · p. 2
3.1 Current Challenges in Chart Reconstruction
- Participants reported using MLLMs for chart reconstruction but identified recurring challenges.The challenges concerned data verification, output editability, and iterative workflows.Source: §3.1 · p. 2
- Designers said extracted data still needs manual verification, including checks for inaccurate values, mismatched data points, and label or heading errors.They emphasized verification particularly for high-stakes medical and business decisions.Source: §3.1 · p. 2
- Participants found MLLM outputs difficult to edit when delivered as flat images without layers, and called for structured, component-based formats.Such formats support subsequent editing and reuse.Source: §3.1 · p. 2
- Designers described chart reconstruction as iterative, requiring repeated prompt refinement and output correction rather than a one-shot process.P3 reported that multiple prompts and refinements are often needed.Source: §3.1 · p. 2
3.2 Design Goals
- The study’s first design goal is accurate, verifiable data and design specifications that designers can validate side by side and correct with minimal manual effort.The authors derived three design goals from the study findings.Source: §3.2 · p. 2
- The second goal is a fully editable structured visualization, with layered, component-based representations such as Vega-Lite specifications.The output should allow refinement of both visual properties and underlying data.Source: §3.2 · p. 2
- The third goal is to make reconstruction substantially more efficient than the typical 2–4 hours of manual work while minimizing repetitive prompting and intervention.Verification and iterative refinement should remain available.Source: §3.2 · p. 2
4 BENCHMARK STUDY
The benchmark compares five commercial MLLMs on data and design extraction across five chart types. It finds that textual and categorical information is generally more reliably extracted than numeric and spatial information.
4.1 Evaluation Setup
- The evaluation covered five commercial MLLMs and five chart types, using a 3,250-chart Plotly dataset with manually verified samples across chart types.The five types were bar, line, scatter, pie, and box plots; the selected types represented approximately 78% (1881/2421) of charts in the surveyed corpus.Source: §4 · p. 2, §4.1 · pp. 2–3
- Data extraction was assessed for text, numerical values, and output format; design extraction covered visual attributes and chart structure, metadata, and text.Text used MCER, numerical values used Adaptive MAPE, and incorrect table formats were scored as complete extraction failures for both metrics.Source: §4.1 · p. 3
4.2 Results
- GPT-5.4 consistently performed best across design property categories, while quantitative visual properties remained challenging for most models.Categorical and text properties had error rates below 11.1% for all models except Nova Pro; only GPT-5.4 maintained consistently low error rates across all property types.Source: Figure 2 · p. 3, §4.2 · p. 3
- Text extraction was accurate across models, but numerical extraction varied by chart type: scatter plots reached 47.4%–99.6% Adaptive MAPE and line charts 9.2%–85.9%.Format errors, particularly on dashed or dotted line charts and dense scatter plots, were a major source of failure.Source: §4.2 · p. 3, Figure 3 · p. 4
5 SYSTEM DESIGN AND IMPLEMENTATION
CHARTREVIVE reconstructs charts from static images by extracting data and design specifications, then generating editable charts. Its interface supports verification and refinement through linked views, direct editing, and visual comparison.
5.1 System Architecture
- The system classifies chart type, extracts data and design specifications, generates a Plotly chart, and lets users verify and refine the result.Plotly was chosen for support of scientific chart elements and output compatible with JavaScript, Python, and R.Source: §5.1 · p. 4
5.2 MLLM-Driven Extraction
- Extraction uses two chart-specific MLLM calls, reusable shared and type-specific prompt blocks, typed JSON schemas, and programmatic corrections.Post-processing reshapes flattened grouped-bar data and expands axis bounds when extracted values fall outside the reported range.Source: §5.2 · p. 4
5.3 User Interface Features
- Overlay verification links table entries to markers on the source image, allowing users to drag markers and update data values in real time.Users first calibrate the overlay by aligning the axes.Source: §5.3 · p. 4
- Users can edit chart properties through linked panels and compare source and reconstructed charts with split, fade, or highlight views.Direct manipulation supports rapid editing without iterative chat-based prompts.Source: §5.3 · p. 4
6 DISCUSSION
The discussion identifies limitations in the evaluation’s user-study evidence, synthetic data, chart-type coverage, and extraction accuracy.
User Study Evaluation
- No formal user study has yet validated the system’s design goals or assessed its usability and efficiency against manual reconstruction.The authors identify evaluation with visualization designers as needed.Source: §6 · p. 4
Synthetic Data Limitations
- Synthetic charts omit some visual artifacts, compression effects, and stylistic diversity found in published charts, leaving real-world generalization to be evaluated.The benchmark uses charts generated from content derived from real-world examples.Source: §6 · p. 4
Chart Type Coverage
- Evaluation covers five basic chart types, leaving heatmaps, Kaplan–Meier curves, and composite figures outside the current scope.The framework is described as extensible to these additional visualizations.Source: §6 · p. 4
Model Adaptation for Chart Extraction
- MLLMs handle categorical or text design properties and text values better than numerical design properties and quantitative data values.Fine-tuning or chart-specific adaptation using the synthetic dataset is proposed as a direction for improving accuracy on challenging cases.Source: §6 · p. 4
7 CONCLUSION
CHARTREVIVE combines extraction of chart data and visual design with interactive verification and editing. Its benchmark finds textual and categorical information more reliably extracted than numeric and spatial information.
7 CONCLUSION
- CHARTREVIVE reconstructs charts from static images by extracting underlying data and visual design specifications.It combines MLLM-based extraction with overlay-based verification, real-time chart rebuilding, and direct editing of data and design properties.Source: §7 · p. 4
- The benchmark found that current models reliably extract textual and categorical information but continue to struggle with numeric and spatial information.The authors highlight the need for interactive verification.Source: §7 · p. 4
- The work presents combining automatic extraction with human verification as a way to support reliable and efficient chart reconstruction.Source: §7 · p. 4