PaperlayerView exact revision on arXiv

Source-linked AI summary

ChartAB: A Benchmark for Chart Grounding & Dense Alignment

Aniruddh Bansal, Davit Soselia, Dang Nguyen, Tianyi Zhou

arXiv:2510.26781v2cs.CV

TL;DR

VLMs still struggle to perceive fine-grained chart details and compare charts, while existing benchmarks leave dense grounding and alignment insufficiently assessed. ChartAB evaluates chart data and attributes through structured grounding and a two-stage alignment pipeline. Its evaluations find persistent perceptual challenges and show the effectiveness of intermediate grounding for dense alignment.

  • Problem

    VLMs struggle to perceive fine-grained chart structures and compare charts, and existing benchmarks leave dense grounding and alignment insufficiently assessed.

    Source: §1 · pp. 1–2

  • Method

    ChartAB evaluates chart data and attributes across diverse charts using structured annotations and a two-stage pipeline that grounds information before aligning chart pairs.

    Source: §1 · p. 2, §3 · p. 4, §3.1 · p. 4

  • Results

    Evaluations find persistent perceptual bias, weak attribute understanding, and limited spatial reasoning, while intermediate grounding improves dense alignment.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The authors present grounding and alignment as essential foundations for robust chart understanding and downstream question answering.

    Source: §5 · p. 10

    The study evaluates general-purpose VLMs without fine-tuning and uses ChartX rather than more sophisticated real-world chart datasets to obtain precise ground truth.

    Source: §A.2 · p. 13

Abstract

from arXiv · show

Charts play an important role in visualization, reasoning, data analysis, and the exchange of ideas among humans. However, existing vision-language models (VLMs) still lack accurate perception of details and struggle to extract fine-grained structures from charts. Such limitations in chart grounding also hinder their ability to compare multiple charts and reason over them. In this paper, we introduce a novel "ChartAlign Benchmark (ChartAB)" to provide a comprehensive evaluation of VLMs in chart grounding tasks, i.e., extracting tabular data, localizing visualization elements, and recognizing various attributes from charts of diverse types and complexities. We design a JSON template to facilitate the calculation of evaluation metrics specifically tailored for each grounding task. By incorporating a novel two-stage inference workflow, the benchmark can further evaluate VLMs capability to align and compare elements/attributes across two charts. Our analysis of evaluations on several recent VLMs reveals new insights into their perception biases, weaknesses, robustness, and hallucinations in chart understanding. These findings highlight the fine-grained discrepancies among VLMs in chart understanding tasks and point to specific skills that need to be strengthened in current models.

1 INTRODUCTION

ChartAB targets fine-grained chart grounding and comparison across charts, addressing gaps in VLM chart perception and existing benchmarks. It combines structured chart extraction with a staged comparison pipeline and analyzes models’ weaknesses, biases, hallucinations, and robustness.

1 INTRODUCTION

  • Chart understanding requires accurate measurement and comparison of graphical components, chart structure and layout, and rich textual content, capabilities that remain challenging for VLMs.The introduction motivates these skills through chart use in finance, data science, media, biology, and other scientific domains.Source: §1 · p. 1
  • Existing chart benchmarks often emphasize simple question answering, while evaluation of fine-grained component grounding and alignment across chart layouts remains lacking.The introduction identifies cross-chart layout and component alignment as unexplored in prior work.Source: §1 · pp. 1–2
  • Its multi-stage pipeline grounds specified information in each chart before comparing the results, using structured JSON templates and metrics designed for symmetry and ambiguity.Similar chart pairs are created by changing one or a few data cells and/or a chart-generation attribute.Source: §1 · p. 2
  • Evaluations reveal VLM weaknesses in dense chart grounding and alignment, including biases and hallucinations around chart components and performance differences across models, chart types, and queried information.The analysis also assesses robustness under changes such as chart type and color scheme.Source: §1 · p. 2
  • ChartAB is presented as a comprehensive benchmark for evaluating VLM grounding and alignment of chart data and attributes across multiple images.The paper categorizes grounded information as data and visualization attributes, including colors, styles, legends, sizes, and positions.Source: §1 · p. 2

2 RELATED WORK

Related work spans chart-specialized and general-purpose VLMs, task-specific chart benchmarks, visual grounding methods, and multi-image reasoning benchmarks. The paper distinguishes ChartAB’s dense chart grounding and alignment focus from these prior approaches.

VLMs for Charts

  • Chart-specialized models show strong chart-benchmark performance, but task-specific instruction tuning and predefined routines are described as constraints on dense-level understanding.The related work contrasts these models with general-purpose multimodal models, including proprietary and open-source systems.Source: §2 · p. 2

Chart Understanding Benchmarks

  • Chart benchmarks cover question answering, summarization, and explanation generation; multi-task and newer QA benchmarks broaden the tasks, modalities, chart sources, and question types.Examples include ChartLlama, ChartX, CharXiv, SciGraphQA, MultiChartQA, and ChartQAPro.Source: §2 · pp. 2–3

Visual Grounding

  • Visual grounding work includes chart data extraction and plot-code grounding, while other methods address open-set object tracking and detailed descriptions of natural images.The passage identifies DePlot, StructChart, Grounded-SAM, BLIP-2, and LLaVA-Grounded as examples.Source: §2 · p. 3

Multi-Image Reasoning

  • Multi-image benchmarks span varied image domains and text-rich content, while MUIRBench is described as limited to coarse-level understanding of multi-chart diagrams.MMIR instead evaluates plotting-code correctness relative to chart images.Source: §2 · p. 3

3 ChartAB: CHART GROUNDING AND ALIGNMENT BENCHMARK

ChartAB benchmarks dense chart grounding, pairwise alignment, and robustness to appearance variation using structured, multi-stage evaluation. It builds on chart-level data and attribute annotations to test fine-grained extraction and comparison across diverse charts.

3 ChartAB: CHART GROUNDING AND ALIGNMENT BENCHMARK

  • ChartAB evaluates VLMs on dense chart grounding, pairwise alignment, and robustness to chart-appearance variation through a two-stage pipeline.Source: §3 · p. 4

3.1 DATASET TAXONOMY AND CONSTRUCTION

  • Built from ChartX, ChartAB annotates underlying data tables and visual attributes across nine simple and complex chart types.Source: §3.1 · p. 4

3.2 EVALUATION TASKS

  • 9,000+ instances cover grounding and alignment of data or visual attributes, with robustness pairs preserving data differences across five attribute-varied chart pairs.Data alignment includes pairs differing in one, two, or three cells.Source: Figure 2 · p. 4, §3.2 · p. 5
  • Grounding produces CSV data tables or JSON attribute representations, while dense alignment identifies local differences between similar charts.Source: §3.2 · p. 5

3.3 A TWO-STAGE EVALUATION PIPELINE

  • The two-stage pipeline grounds each chart's data or attributes before comparing the representations and outputting JSON differences, enabling end-to-end alignment evaluation.The authors state that the second stage mitigates grounding ambiguities and adds context compared with one-stage approaches.Source: §3.3 · p. 6, Figure 4 · p. 7

3.4 EVALUATION METRICS

  • Metrics are task-specific: SCRM evaluates CSV grounding precision, key-value scores evaluate alignment (legend alignment compares positions), and standard deviation measures robustness across variations.Source: §3.4 · p. 6
  • For downstream QA, a two-stage process extracts a chart's CSV before answering, and the study examines QA accuracy and its difference from one-stage QA against grounding and alignment quality.Source: §3.4 · pp. 6–7

4 EXPERIMENTS & ANALYSIS

Evaluations show that VLMs struggle with dense chart grounding and alignment, particularly for complex layouts, exact visual attributes, and spatial placement. Grounding and alignment performance also tracks downstream chart-question-answering performance.

4 EXPERIMENTS & ANALYSIS

  • The evaluation covers GPT-4o, four open-source VLM families, and chart-specialized models, but the specialized models fail to output the required JSON format.The authors attribute this failure to task-specific training collapsing general instruction-following capabilities.Source: §4 · p. 7
  • Dense data and color grounding/alignment is more challenging on complex charts than on simpler charts, while legend-position and text-style tasks are less affected.More components and irregular layouts make mapping visual encodings to colors and numeric values harder; legend positions and text styles have fewer options.Source: §4 · pp. 7–8
  • On Figure 5’s paired-chart tasks, most VLMs perform better on common line/bar charts; QWEN-2.5-VL outperforms GPT-4o on most chart types, while Llava-1.6 performs worse than most models.The figure compares data alignment on chart pairs differing by one cell and color alignment on fine-grained elements.Source: Figure 5 · p. 8
  • Most VLMs achieve below 20% accuracy on text-size and font-family grounding, except GPT-4o on font-family grounding; text-weight accuracy is about 60%.Models tend to rely on prior or relative sizes rather than identify exact text sizes in chart images.Source: §4 · p. 8
  • All evaluated models have median color-grounding L2 error above 50 in RGB space, and color-grounding performance is consistent with color-alignment performance.The authors link this weakness to difficulty understanding shades beyond common colors and to misalignment in color-conditioned reasoning tasks.Source: §4 · p. 8
  • Legend-position grounding shows strong model biases: Phi-3.5 favors upper-left, several 7–8B models favor upper-right, and GPT-4o has the least bias and best legend alignment.The reported 7–8B models are LLaVA-1.6, InternVL-2.5, and QWEN-2.5-VL.Source: §4 · p. 9, Figure 8 · p. 9
  • Failed QA tasks decrease and successful QA tasks increase with data-alignment score, while precise data grounding is associated with improved QA performance.The greatest reported gains from precise grounding are on bar/line charts and numbered bar/line charts.Source: §4 · p. 9, Figure 9 · p. 9
  • Most dense-alignment subtasks show a scaling law, but text-style alignment deviates amid more failures from InternVL-2.5’s incorrect JSON formatting.The authors attribute the exception to the greater complexity of the text-style JSON template.Source: §4 · p. 10

5 CONCLUSION

ChartAB evaluates fine-grained chart grounding, multi-chart alignment, and robustness across chart appearances. Its evaluations reveal persistent perception and spatial-reasoning challenges, and report benefits from intermediate grounding for alignment and downstream question answering.

5 CONCLUSION

  • ChartAB evaluates VLMs on fine-grained chart grounding, multi-chart dense alignment, and robustness to chart-appearance variations.The conclusion describes these capabilities as foundations for chart reasoning.Source: §5 · p. 10
  • Evaluations across diverse chart types reveal perceptual bias, weak attribute understanding, and limited spatial reasoning, especially on complex visual representations.Source: §5 · p. 10
  • Experiments with the two-stage pipeline show effectiveness of intermediate grounding for dense alignment, while grounding and alignment accuracy affect downstream question answering.Source: §5 · p. 10

A.1 LLM USAGE STATEMENT

LLMs served as general-purpose writing aids and literature-search assistants; the authors conducted the substantive research work.

A.1 LLM USAGE STATEMENT

  • LLMs were used as general-purpose writing aids, including polishing grammar and phrasing.Source: §A.1 · p. 13
  • LLMs also assisted with literature search.Source: §A.1 · p. 13
  • The authors conducted all substantive research ideation, experiments, and analysis.Source: §A.1 · p. 13

A.2 LIMITATIONS

The study evaluates general-purpose VLMs without assessing fine-tuning, and uses ChartX because its CSV data and plotting code support precise ground truth.

A.2 LIMITATIONS

  • The work evaluates VLMs but does not assess model fine-tuning.The authors note that fine-tuning might yield stronger results but diverges from their goal of studying general-purpose VLMs for dense-level understanding.Source: §A.2 · p. 13
  • ChartX was selected despite the availability of chart datasets with more sophisticated real-world examples.Source: §A.2 · p. 13
  • ChartX provides CSV data and plotting code, which the authors use to generate precise ground-truth values for dense grounding and alignment evaluation.Source: §A.2 · p. 13

A.3 DATASET CONSTRUCTION

ChartAB’s construction uses ChartX tables and plotting scripts to generate controlled data and attribute changes, then retains rendered chart pairs with labels for grounding and alignment.

A.3 DATASET CONSTRUCTION

  • Data pairs modify 1–3 uniquely valued cells, scale selected values by their column means, and update the table and script when matches are unambiguous.Charts are retained when execution successfully generates both images.Source: §A.3 · p. 13
  • The construction records chart pairs with grounding and alignment labels, while robustness sets use randomly assigned changes and diverse attribute values.The labels capture the relevant table or attribute differences between the paired charts.Source: §A.3 · pp. 13–14
  • ChartX supplies CSV tables and plotting code, enabling precise ground-truth generation and fine-grained changes across diverse chart types and domains.The construction perturbs plotting code to create data or attribute changes and renders the resulting charts.Source: §A.3 · p. 13
  • Attribute pairs perturb color arrays, legend positions, or text styles parsed from the plotting script.The described text-style fields include size, weight, and font family.Source: §A.3 · p. 14

A.4 A TWO-STAGE EVALUATION PIPELINE DETAILS

The pipeline uses natural-language zero-shot instructions and structured JSON outputs to evaluate chart-pair alignment, including robustness to visual variations.

A.4 A TWO-STAGE EVALUATION PIPELINE DETAILS

  • Natural-language instructions support zero-shot inference with minimal task-specific nuances, aiming for generalization across models.The pipeline constructs robustness examples with five visual variations per instance across color, legend, and text-style attributes.Source: §A.4 · pp. 14–15
  • Robustness-set construction modifies 1–3 table cells within a scaling range and generates five visual variations for each instance.The visual attributes varied are color, legend, and text style.Source: §A.4 · p. 15
  • For robustness examples, the pipeline applies each sampled visual variation to both charts and adds successfully rendered pairs to the set.The variations target color, legend, or text style.Source: §A.4 · p. 15
  • JSON outputs serialize element-specific grounding information and chart-pair differences, including the corresponding values of changed attributes.For data alignment, changed cells are identified by row and column headers and their values in each chart.Source: §A.4 · p. 15

A.5.1 DENSE ALIGNMENT

The section defines task-specific scores for chart-pair alignment and reports how performance varies with visual changes, model selection, and chart structure.

A.5.1 DENSE ALIGNMENT

  • Data, color, and text-style alignment use key-value elements, while legend alignment scores differences in legend positions.Key identification uses fuzzy matching for data and color, and exact matches for text-style keys; associated values are evaluated for precision.Source: §A.5.1 · pp. 15–16, Table 2 · p. 16

A.5.2 ROBUSTNESS

  • Robustness is the reciprocal of the standard deviation in data-alignment scores across visual variations, averaged over chart pairs by chart type.The evaluated variations are color, legend, and text style.Source: §A.5.2 · p. 17

A.6.1 VLM SELECTION

  • The evaluation covers several open-source VLM families and GPT-4o; TinyChart and ChartGemma fail to produce the required JSON format in the reported setup.GPT-4o was selected as the proprietary model based on CharXiv performance.Source: §A.6.1 · p. 17

A.6.2 ABLATIONS

  • The two-stage approach outperformed single-stage alignment on Phi-3.5 data alignment, and the authors report that intermediate grounding helps focus on localized relationships.The ablation compares stitched-chart and multi-image single-stage approaches with the proposed two-stage method.Source: Table 3 · p. 18, §A.6.2 · p. 18

A.7 ADDITIONAL FINDING & INSIGHTS

  • Robustness is best under color changes and worst under text-style changes; legend shifts can also alter positions and occlude chart elements.Distinguishable colors preserve data grounding, whereas text styles and legends carry information through ticks, labels, and legend items.Source: §A.7 · p. 19
  • 3D bars challenge scale and value measurement, rose charts suffer from fewer text cues, while numbered bars and lines usually perform better by displaying values.Rose-chart grid lines cannot be fully leveraged as non-text cues.Source: §A.7 · p. 20
  • Color alignment grounds each chart’s visual-encoding colors before comparing the pair and listing differing values in JSON.The pipeline separates per-chart grounding from the subsequent alignment stage.Source: Figure 15 · p. 21
  • Text-style and legend alignment likewise ground chart attributes first, then output JSON describing differences between the paired charts.Text-style grounding covers title, legend, axes labels, and axes ticks; legend grounding records legend position.Source: Figure 16 · p. 22, Figure 17 · p. 23
Loading 2510.26781v2…