PaperlayerView exact revision on arXiv

Source-linked AI summary

ChartArena: Benchmarking Chart Parsing across Languages, Scenarios, and Formats

Shangpin Peng, Gengluo Li, Xingyu Wan, Chengquan Zhang, Hao Feng, Binghong Wu, Huawen Shen, Weinong Wang, Ziyi Cai, Zhuotao Tian, Han Hu, Can Ma, Yu Zhou

arXiv:2606.01348v3cs.CV

TL;DR

Chart parsing lacks a unified benchmark spanning diverse chart types, output formats, languages, and realistic visual conditions. ChartArena addresses this with bilingual coverage, format-agnostic evaluation, and broad model analysis, finding that proprietary models lead overall while diagrammatic, radar, and hand-drawn cases remain difficult.

  • Problem

    Existing chart-parsing benchmarks narrowly cover chart types, use incompatible output formats, and rarely represent printed or hand-drawn images, leaving systematic evaluation incomplete.

  • Method

    ChartArena benchmarks eight chart families across three visual scenarios and two languages, normalizing heterogeneous outputs into canonical triple and directed-graph views for structure-aware evaluation.

  • Results

    Across 26 models, frontier proprietary systems lead overall, open-source systems remain competitive, diagrammatic structures expose gaps, and radar charts and hand-drawn conditions remain especially challenging.

  • Takeaways & Limitations

    ChartArena exposes capability gaps and provides a unified foundation for comparing future chart-parsing systems across chart families, formats, languages, and visual conditions.

  • Takeaways & Limitations

    The benchmark covers only single-page images and omits challenging families such as scatter plots, leaving multi-page aggregation and dense point localization for future extensions.

Abstract

from arXiv · show

Charts are a primary medium for conveying quantitative and relational information, yet systematically evaluating chart parsing models remains difficult. Existing benchmarks focus on narrow chart types and leave diagrammatic structures such as flowcharts and mind maps largely unaddressed, while models produce outputs in incompatible formats, and datasets rarely include the printed or hand-drawn images encountered in practice. To address these issues, we introduce ChartArena, a comprehensive bilingual benchmark covering eight chart families spanning both numeric charts and diagrammatic structures, each evaluated across three visual scenarios: digital renderings, printed photos, and hand-drawn photos. The dataset is built via a human-agent collaborative annotation pipeline with multi-stage human verification to ensure annotation reliability. To enable fair cross-model comparison, we further design a format-agnostic evaluation protocol that maps heterogeneous outputs into two canonical semantic spaces, a normalized triple view and a directed graph view, and scores them with structure-aware metrics. Through extensive evaluation of 26 leading MLLMs, we observe three consistent findings: (i) frontier proprietary models such as Gemini 3.1 Pro lead overall, yet the strongest open-source systems are rapidly closing the gap; (ii) document parsing models handle numeric charts reasonably but fall sharply behind on diagrammatic structures; and (iii) expert chart parsers remain limited to narrow chart families. Across all models, radar charts and hand-drawn scenarios stay especially challenging. These findings show that ChartArena exposes clear capability gaps and provides a unified foundation for future progress. ChartArena is publicly available at https://github.com/pspdada/ChartArena.

1 Introduction

ChartArena addresses fragmented chart-parsing evaluation by unifying eight chart families, three visual scenarios, and two languages, while normalizing heterogeneous outputs into shared semantic representations. Evaluation of 26 leading models reveals persistent gaps across model paradigms and chart structures.

  • Evaluation challenge: Existing chart parsers emit incompatible formats, including Markdown tables, JSON, CSV, Python, and SVG, making direct cross-model comparison difficult.This fragmentation contrasts with more unified evaluation standards for table and formula parsing.
  • ChartArena benchmark: ChartArena unifies eight numeric and diagrammatic chart families across digital, printed, and hand-drawn scenarios in Chinese and English.Its human-agent collaborative annotation pipeline is designed to ensure structural reliability.
  • Format-agnostic evaluation: The format-agnostic protocol deterministically maps heterogeneous outputs into normalized triple and directed graph views for structure-aware cross-model comparison.The protocol supports formats such as Markdown, JSON, CSV, Python, and Mermaid and is extensible to additional formats.
  • Comprehensive model analysis: Evaluation of 26 leading models finds proprietary systems lead overall, while the strongest open-source systems rapidly close the gap.The analysis also identifies capability differences between document parsing models and expert chart parsers.
  • Comprehensive model analysis: Document parsing models handle numeric charts better than diagrammatic structures, whereas expert chart parsers remain limited to narrow chart families.These findings are reported as key capability gaps revealed by ChartArena and its evaluation protocol.

2 Related Work

Chart parsing has evolved from OCR-and-geometry pipelines to end-to-end sequence generation with MLLMs, but evaluation remains fragmented by heterogeneous output formats. ChartArena addresses this fragmentation through comprehensive benchmark coverage across chart types, visual scenarios, and languages.

  • Evolution of chart parsing: Chart parsing shifted from modular OCR and heuristic-geometry pipelines to end-to-end sequence generation with MLLMs.Earlier cascaded systems suffered compounding errors and struggled with real-world visual noise.
  • Benchmark coverage: ChartArena provides comprehensive coverage across chart types, visual scenarios, and languages for realistic chart-parsing evaluation.The benchmark is presented as addressing the limited and fragmented coverage of existing benchmarks.
  • Output paradigms and evaluation of chart parsing: Existing chart-parsing methods serialize numeric-chart data into heterogeneous formats, including Markdown, SVG, Python code, HTML tables, CSV, and JSON.This diversity contributes to a lack of unified standards for chart parsing evaluation.

3 ChartArena Benchmark

ChartArena is a bilingual benchmark designed to evaluate chart parsing across eight chart families and three visual scenarios, including real-world printed and hand-drawn images. It combines broad data coverage with hybrid human-agent annotation and multi-stage verification for unified evaluation.

  • Data Collection: ChartArena deliberately over-samples printed and hand-drawn charts so evaluation is not dominated by easier digital renderings.Images are curated from public document corpora, web sources, and in-house collections spanning science, business, and education.
  • Annotation Pipeline: An MLLM generates coarse type-aligned annotations, after which human annotators refine them through multiple verification rounds.Numeric charts use Markdown tables, while diagrammatic charts use Mermaid graph descriptions to accelerate annotation.
  • Task Coverage: ChartArena covers eight numeric and diagrammatic chart families across digital renderings, printed photos, and hand-drawn photos, in English and Chinese.The chart families include bar, line, pie, radar, box plot, combination chart, flowchart, and mind map.
  • Task Coverage: 2,400 charts result from 50 samples per combination of eight chart types, three visual scenarios, and two languages.This design supports comprehensive and unified evaluation across the benchmark’s stated dimensions.

4 Format-Agnostic Evaluation Protocol

ChartArena uses a format-agnostic protocol that maps heterogeneous model outputs and references into canonical semantic representations before applying structure-aware scoring. Numeric charts use tolerance-aware triple matching, while diagrammatic charts use graph-based structural matching, with unified EM and mAP metrics enabling comparable evaluation.

  • 4 Format-Agnostic Evaluation Protocol: The protocol normalizes heterogeneous predictions and ground truths into shared canonical representations, then scores them with structure-aware metrics.This design addresses incompatible output formats and separates normalization from scoring.
  • 4.1 Format-Agnostic Normalization: Numeric charts are represented as semantic triples (header, entity, value), regardless of whether outputs use Markdown, CSV, JSON, Python, SVG, or HTML tables.The triple view captures axis-value relationships in a format-independent form.
  • 4.1 Format-Agnostic Normalization: Diagrammatic charts are represented as directed graphs with labeled nodes and directed labeled edges, abstracting differences among diagram languages.The graph view captures flowchart topology and mind-map hierarchy in one representation.
  • 4.2 Structure-Aware Scoring: The scoring backends produce per-sample similarities in [0, 1] and aggregate them into final metrics based on structural correctness rather than string or token overlap.Each canonical view has a dedicated scoring backend.
  • 4.2 Structure-Aware Scoring: Numeric chart scoring uses Intersection-over-Union (IoU), matching triples only when text keys and values satisfy tolerance conditions.Levenshtein distance handles text, while relative-error thresholds allow minor OCR or rounding errors without accepting genuinely wrong values.
  • 4.2 Structure-Aware Scoring: Diagrammatic chart scoring matches nodes and edges separately with the Hungarian algorithm, weighting edges more heavily because topological errors are more damaging.Mind maps use a tree-based variant that rewards partial recovery such as correct top-level branches.
  • 4.2 Structure-Aware Scoring: The protocol reports Exact Match (EM) for perfect recovery and mean Average Precision (mAP) for graded correctness across strict, slight, and high tolerance levels.mAPhigh is the primary metric unless otherwise stated because it balances robustness to matching leniency.

5 Experiments

ChartArena evaluates 26 models under standardized settings and shows that general-purpose MLLMs lead overall, while document parsers and expert chart parsers remain limited on diagrammatic structures and broader coverage. Its format-agnostic protocol produces stable scores for most common formats but exposes failures tied to SVG, PlantUML, and photograph-based visual noise.

  • Experimental setup: 26 models are evaluated across general-purpose MLLMs, document parsing MLLMs, and expert chart parsers using standardized inference settings and category-appropriate prompts.General-purpose MLLMs include open-source and proprietary systems, while document and expert models use official prompts and native output formats.
  • Main comparison: 59.2 EN / 73.2 ZH is Gemini 3.1 Pro’s highest overall average, while Kimi K2.5 leads open-source models at 54.8 EN / 68.1 ZH.Qwen3.5-35B-A3B follows at 54.1 EN / 65.5 ZH, and both are competitive with Seed-2.0 Pro at 54.0 EN / 70.5 ZH.
  • Main comparison: 41.4 EN / 52.8 ZH is HunyuanOCR’s overall numeric-chart performance, but its flowchart score falls to 39.9 EN / 35.9 ZH, trailing Gemini 3.1 Pro by 25.7 EN.The passage attributes diagrammatic difficulty to implicit nodes, relations, and hierarchies requiring broader world knowledge.
  • Main comparison: Expert chart parsers have narrow chart-family and language coverage, with ChartAst, ChartVLM, and TinyChart unable to handle flowcharts or mind maps.Only RRVF and MSRL produce non-trivial scores on those two diagrammatic families, while expert-model absolute performance remains low.
  • Output-format adaptability: 5.2 points is Seed-2.0 Pro’s mAPhigh spread across Code, Markdown, JSON, and CSV numeric-chart formats, whereas SVG drops to 40.0.Qwen3.5-35B-A3B ranges from 46.2 to 56.6 across those formats and drops to 39.9 on SVG.
  • Output-format adaptability: 3.7 points is Seed-2.0 Pro’s mAPhigh range across Mermaid, Cytoscape, Diagrams, and Graphviz flowchart formats, while PlantUML falls to 33.8.Qwen3.5-35B-A3B scores 29.0 on PlantUML; the protocol identifies these format-structure compatibility failures without penalizing syntactic choices generally.

6 Conclusion

ChartArena introduces a comprehensive bilingual benchmark and format-agnostic evaluation protocol spanning diverse chart families, visual scenarios, and output representations. Evaluation of 26 models reveals persistent gaps in diagrammatic chart parsing, despite strong proprietary performance and rapidly improving open-source systems.

  • Benchmark and protocol: ChartArena covers eight chart families across numeric and diagrammatic structures, three visual scenarios, and two languages.It is designed as a comprehensive benchmark for chart parsing.
  • Benchmark and protocol: Its normalization pipeline maps incompatible model outputs into canonical triple views and directed graph views for fair comparison.The protocol scores predictions in these canonical semantic representations.
  • Benchmark findings: Evaluation of 26 models finds frontier proprietary systems leading, while the strongest open-source systems remain highly competitive and are closing the gap.The comparison identifies this as a clear benchmark-wide finding.
  • Benchmark findings: Document parsing MLLMs handle numeric charts reasonably but fall sharply behind on diagrammatic structures requiring broader world knowledge.The result highlights a substantial capability difference between numeric and diagrammatic chart families.
  • Benchmark findings: Dedicated expert parsers show a fundamental coverage gap, with many unable to handle diagrammatic families such as flowcharts and mind maps.ChartArena is intended to expose such gaps and support more reliable, reproducible, general-purpose chart understanding.

Supplementary Material … A.2 Annotation Protocol and Human Effort

The supplementary material details ChartArena’s benchmark construction, evaluation procedures, extended analyses, case study, related work, and LLM usage statement. It especially documents broad real-world image sourcing across three scenarios and a human–agent annotation pipeline with format-specific verification and effort accounting.

  • Overview: The supplementary material covers benchmark details, evaluation details, extended experimental analysis, a further case study, related work, and an LLM usage statement.These topics are organized across Sections A–E and H.
  • A.1 Image Sources and Scenarios: ChartArena primarily collects images through web search and supplements under-represented categories with samples scanned, photographed, or hand-drawn by in-house annotators.This sourcing strategy aims to broaden real-world chart-image coverage rather than favor synthetic or templated inputs.
  • A.1 Image Sources and Scenarios: Collected images must contain readable text, real-world rather than placeholder data, and one complete chart without heavy occlusion or cropping.These requirements define the minimum image-quality bar for benchmark inclusion.
  • A.1 Image Sources and Scenarios: The benchmark includes digital renderings, printed photos, and hand-drawn photos to represent complementary visual scenarios.Printed captures reproduce interference such as uneven illumination, reflections, and moiré, while hand-drawn photos add irregular handwriting and layout variation.
  • A.2 Annotation Protocol and Human Effort: Table A.1 reports average annotation and review time per image alongside cumulative effort in person-days, using separate entries for annotation and quality review.Person-days are defined using 8 working hours per day.
  • A.2 Annotation Protocol and Human Effort: Annotation uses multiple MLLMs for candidate generation, cross-checking and merging, followed by human verification and correction of structural semantics and fine-grained content.The pipeline uses multi-round verification to improve ground-truth reliability.
  • A.2 Annotation Protocol and Human Effort: Numeric charts and mind maps are serialized as Markdown tables and nested unordered lists, respectively, under chart-specific guidelines and auxiliary reading aids.Guidelines cover column ordering, unit normalization, and nesting rules.
  • A.2 Annotation Protocol and Human Effort: Flowcharts are serialized as Mermaid code and rendered for checks of node sets, edge connectivity, and overall logical flow.Flowchart annotation is more demanding than numeric annotation because richer topology makes single mistakes consequential.

A.3 Benchmark Samples

ChartArena illustrates its three visual scenarios through representative samples pairing each original image with its ground-truth annotation. The examples include multi-series bar, line, pie, radar, and box plots, including hand-drawn and printed-photo cases.

  • Representative samples pair each original chart image with its ground-truth annotation to concretize ChartArena’s three visual scenarios.
  • The benchmark includes a representative multi-series bar chart sample.
  • The benchmark includes a representative line chart sample.
  • Hand-drawn examples include a pie chart with 12 labelled slices and a radar chart with overlapping rings.
  • A printed-photo scenario is represented by a box plot sample.

B Evaluation Details … C.1 Detailed Analysis under Different Visual Scenarios

ChartArena evaluates heterogeneous chart-parsing outputs through canonical triple, tree, and graph views with structure-aware scores, then analyzes robustness across visual scenarios. The protocol supports fair comparison, while experiments show substantial degradation for printed and especially hand-drawn images, particularly on diagrammatic charts.

  • B.1 Format-Agnostic Routing and Normalization: The evaluator routes outputs from formats such as Markdown, CSV, JSON, code, and SVG into canonical triple, tree, or graph representations based on task structure.Deterministic adapters extract semantic content while discarding presentation details.
  • B.2 Triple-Based Scoring for Numeric Charts: Numeric charts are scored as unordered (entity, header, value) triples using tolerance-aware IoU matching across strict, slight, and high settings.Table transposition therefore preserves the triple set, while deterministic preprocessing standardizes text and numeric fields without changing semantics.
  • B.4 Aggregation into Exact Match and mAP: Final reporting includes EM, mAP(t), and AP@θ(t), averaged over samples to produce per-chart-type and overall benchmark scores.EM measures strict exact recovery, whereas mAP aggregates performance across quality thresholds.
  • B.5 Evaluation Setup: Inference uses officially recommended prompts for specialized models and a unified chart-parsing template for general-purpose MLLMs, with target-specific output formats.Prompts require preserving labels, numerical precision, node text, hierarchy, connections, directions, and branch labels while ignoring irrelevant visual elements.
  • C Extended Experimental Analysis: The extended analysis examines robustness under digital rendering, printed photos, and hand-drawn photos, alongside ranking consistency and broader model coverage.These analyses are presented as complementary to the main experiments and orthogonal to the evaluation protocol.
  • C.1 Detailed Analysis under Different Visual Scenarios: Printed photos consistently reduce performance through camera noise, illumination variation, blur, perspective distortion, and printing artifacts, while hand-drawn photos add irregular strokes and ambiguous boundaries.Hand-drawn images are described as substantially more challenging because they also contain imperfect structures and inconsistent layouts.
  • C.1 Detailed Analysis under Different Visual Scenarios: Scenario-induced degradation is significantly larger for diagrammatic than numeric charts because diagrams require restoring node-link relations, directional connections, and hierarchical structure.Among general-purpose MLLMs, Gemini 3.1 Pro achieves the strongest overall performance across all scenarios.

C.2 Consistency Across Evaluation Metrics … E.3 Evaluation for Document and Chart Parsing

ChartArena’s metric analysis shows that model rankings remain highly stable across evaluation metrics, while metric severity differs between numeric and diagrammatic charts. Additional evaluations and case studies broaden comparisons and reveal persistent challenges in hand-drawn parsing, hallucination, and standardized chart-evaluation conventions.

  • C.2 Consistency Across Evaluation Metrics: Model rankings are highly consistent across metrics, with correlations to mAPhigh reaching 0.96 on numeric charts and 0.99 on diagrammatic charts.The reported correlations are 0.91 (EM), 0.93 (mAPstrict), and 0.96 (mAPslight) for numeric charts, versus 0.95, 0.99, and 0.99 for diagrammatic charts.
  • C.2 Consistency Across Evaluation Metrics: EM is harsher on diagrammatic charts, retaining 0.14 of mAPhigh versus 0.24 on numeric charts.The ratio of each metric to mAPhigh measures score decay as the criterion is tightened.
  • C.2 Consistency Across Evaluation Metrics: mAPhigh preserves model ordering while providing a wider dynamic range, making it suitable as the main metric alongside stricter complementary signals.The wider range is especially useful because many models’ EM scores are compressed near zero.
  • C.3 Additional Model Results: Additional experiments evaluate InternVL3.5 and Qwen3.5 size variants, Gemma 4-31B-IT, MiniCPM-V 4.5, and the Molmo series under the main experimental setup.The stated purpose is to improve transparency and broaden comparability across model scales.
  • D Further Case Study: Hand-drawn photos introduce substantial visual uncertainty, and Gemini 3.1 Pro can hallucinate when visual evidence is ambiguous.In the cited pie-chart case, the model replaces a visually present character with a more plausible alternative, causing parsing errors.
  • E.1 Large Language and Multimodal Models: MLLMs extend vision-language modeling by combining visual encoders, LLM backbones, cross-modal connectors, and visual instruction tuning.These systems support perception and reasoning over images, documents, and other modalities.
  • E.2 MLLMs for Chart Parsing: Modern chart parsing reframes brittle OCR-and-heuristic pipelines as end-to-end generation from chart images to structured serializations.Earlier cascaded systems accumulated errors across stages and degraded under real-world visual noise.
  • E.3 Evaluation for Document and Chart Parsing: Standardized output formats and metrics enable direct cross-model comparison in structured document parsing, motivating comparable conventions for chart parsing.Table and mathematical formula parsing are cited as related tasks that have largely converged on unified evaluation conventions.

F Limitations · G Broader Impact · H LLM Usage Statement

ChartArena remains limited to single-page images and excludes challenging families such as scatter plots, while supporting structured visual reasoning without identified broader societal concerns. LLMs contributed only auxiliary manuscript editing.

  • F Limitations: ChartArena currently focuses on single-page chart images rather than multi-page charts requiring cross-page aggregation.The omitted setting includes visual elements, legends, or continuation tables distributed across pages.
  • F Limitations: The benchmark excludes challenging chart families such as scatter plots.Scatter plots often require precise point localization and dense coordinate recovery.
  • F Limitations: These limitations leave multi-page aggregation and certain difficult chart families outside ChartArena’s current coverage.The passage identifies both omissions as remaining limitations despite the benchmark’s broad coverage.
  • G Broader Impact: The work aims to advance general chart parsing and evaluation through a unified benchmark and evaluation protocol.This design is intended to support more reliable assessment of multimodal models on structured visual reasoning tasks.
  • G Broader Impact: Chart understanding may support scientific analysis, business intelligence, and document accessibility.The paper presents these as application domains for the proposed benchmark and evaluation framework.
  • G Broader Impact: The authors identified no broader societal impacts warranting particular concern at this time.This statement accompanies the work’s intended contributions to chart parsing and structured visual reasoning assessment.
  • H LLM Usage Statement: LLMs were used only as auxiliary writing tools during manuscript preparation.Their role covered grammar correction, readability enhancement, and light wording refinement.
  • H LLM Usage Statement: LLM assistance was limited to improving language quality rather than performing substantive research tasks.The stated uses were grammar correction, readability enhancement, and light wording refinement.
Loading 2606.01348v3…