Benchmark It Yourself (BIY): Preparing a Dataset and Benchmarking AI Models for Scatterplot-Related Tasks

1 INTRODUCTION

• Multimodal models have shown broad applicability, including in data analysis and visualization.The passage situates this work amid academic results and new products and services.Source: §1 · p. 1
• The expanding choices of models, prompts, and architectures make experimentation costly and time-consuming, while performance remains uncertain.Choosing among these options also requires expertise.Source: §1 · p. 1
• Scatterplot benchmarks lack representative, task-targeted coverage of clustering and outlier detection, despite scatterplots being a common chart type for exploring two numerical variables.The passage also notes concerns that existing benchmark results may be uncertain because benchmarks can be manipulated or require nuanced interpretation.Source: §1 · p. 1
• The paper contributes a synthetic annotated dataset and generation pipeline, evaluates ten proprietary models, and offers chart-design considerations for AI inputs.The stated applications include extracting insights from chart images when raw data are unavailable or extensively plotted, and using those insights in chart descriptions or multi-agent analysis and reporting.Source: §1 · p. 1

2.1 Datasets

• Existing chart datasets cover question answering, captioning, extraction, and hybrid combinations, while BIY adds scatterplot-oriented annotations for evaluating or training clustering and outlier detection.The authors also describe extending the dataset as easy, accurate, and negligible in cost.Source: §2.1 · pp. 1–2

2.2 Benchmarks and Experiments

• Reported chart benchmarks include ChartQA [69], ChartQAPro [68], CharXiv, ChartMuseum, MMMU [136], and MMMU-Pro, each with an expected prompt structure and at least one metric.The final prompt may be adjusted for the model.Source: §2.2 · p. 2
• No reviewed benchmark or experiment covers multimodal chart-image-and-text evaluation of scatterplots; PUB and ChartInsights [130] are the closest works.The paper focuses on scatterplots and undocumented tasks such as outlier detection, using three prompting strategies, new models and metrics, and a distinct chart-assembly methodology.Source: §2.2 · p. 2

3.1 Dataset

• The dataset contains 18,921 scatterplots from six generators, with annotations for cluster boxes, cluster centers, and outlier coordinates.It covers 17 chart specifications and data samples with zero to six clusters.Source: §3.1 · p. 2

3.2 Benchmark

• The benchmark tests five tasks on 1,725 scatterplots across 15 chart designs, using zero-, one-, and few-shot prompting.Few-shot examples were selected to represent the benchmarked data generators, including an extra example with outliers.Source: §3.2 · pp. 2–3

3.3 Results

• 90.49% Accuracy for few-shot Flash outlier counting contrasts with no model exceeding 25% Recall in cluster detection or identification.The authors also report that results are significantly better for counting than for precise detection and identification.Source: Figure 2 · p. 3, Figure 3 · p. 4, §3.4 · p. 4

3.3.1 Counting

• Few-shot Flash achieved 90.49% Accuracy for outlier counting, while few-shot o3 had the lowest cluster-counting MAE at 0.03.Across scatterplots, count predictions matched the number of localized clusters in 61.4% of cases and outliers in 57.7%; results on zero-answer plots also varied substantially.Source: Figure 2 · p. 3, §3.3.1 · p. 3

3.3.2 Detection and Identification

• No model exceeded 25% Recall for cluster detection or identification; few-shot Flash reached 65.01% Recall for outlier identification.Lowering the cluster-detection IoU threshold to 0.5 changed some results, but Recall remained low.Source: §3.3.2 · p. 3, Figure 3 · p. 4

3.3.3 Chart Design Performance

• Chart design had low overall explanatory power for cluster-counting accuracy, but some designs differed significantly from the default.Half opacity improved accuracy by 2.42 percentage points, while 16:9, 21:9, and random colors impaired it.Source: §3.3.3 · p. 4

3.4 Considerations

• The authors recommend few-shot prompting, caution against relying on the tested models for precise localization, and advise avoiding wide aspect ratios or random colors.They characterize chart design as a secondary factor and suggest investing in other components first.Source: §3.4 · p. 4

• The benchmark is a first step toward gauging chart-understanding capabilities and real-world applicability; planned analyses cover chart-design effects, cost-performance trade-offs, and invalid responses.Source: §4 · p. 4
• Planned benchmark extensions add generators, chart designs, internal data, prompting strategies, tasks, and proprietary and open models, plus a lightweight version for low-cost monitoring.Source: §4 · p. 4
• The authors plan to fine-tune small, open models and assess trade-offs of running less demanding models on their own infrastructure.Source: §4 · p. 4
• Future work includes template-based captioning for accessible scatterplots and evaluating AI-generated alt text; extension to other chart types comes later.Source: §4 · p. 4