ChartAgent: A Multimodal Agent for Visually Grounded Reasoning in Complex Chart Question Answering

1 Introduction

• Recent multimodal LLMs struggle on unannotated charts that require estimating values from graphical elements rather than reading explicit labels.Chart VQA performance degrades especially on charts without textual annotations of key values or labels.Source: §1 · p. 1
• ChartAgent decomposes queries into visual subtasks and interacts with chart images through specialized perception tools, drawing on human strategies for chart comprehension.Its multi-turn loop supports visual reasoning with occasional numerical subtasks.Source: §1 · p. 1
• The framework introduces chart-specialized visual capabilities for MLLMs, with interpretable tool outputs that support inspection and adaptive changes to tools or parameters.The authors describe it as the first demonstration of visually grounded chart reasoning through tool-augmented multimodal agents.Source: §1 · p. 2
• ChartAgent surpasses 30+ baselines by up to 16.07% absolute gain overall and 17.31% on unannotated, numerically intensive queries across ChartBench and ChartX.The benchmarks span 40+ chart types.Source: §1 · p. 2
• Analyses report effectiveness across chart types and complexity levels, plug-and-play gains across base MLLMs, and a failure-mode analysis.The introduction presents these findings as evidence of effectiveness and generalization.Source: §1 · p. 2

2.1 Chart Visual Question Answering

• Chart VQA progressed from synthetic datasets to realistic, diverse, numerically intensive benchmarks and chart-specific MLLMs, but precise interpretation of unannotated charts remains challenging.The paper frames its approach as addressing this visual-grounding gap with chart-specialized reasoning.Source: §2.1 · pp. 2–3

2.2 Multimodal LLMs and Visual Grounding

• Visual-grounding systems use structured tools or iterative visual prompts, and ChartAgent combines iterative reasoning, visual prompting, and modular vision tools for chart understanding.The cited examples include tool-using systems and methods that refine or annotate visual inputs.Source: §2.2 · p. 3

2.3 Agentic Frameworks

• Agentic systems organize perception, cognition, and action, while ChartAgent applies multimodal reasoning with modular chart-oriented vision tools.The related work describes ReAct as structuring interactions into iterative reasoning, action, and observation.Source: §2.3 · p. 3

3 ChartAgent: A Multimodal Agent for Visually Grounded Reasoning in Charts

• ChartAgent decomposes chart queries into visual subtasks and interacts with chart images through specialized tools whose interpretable outputs support adaptive refinement.The framework continues until it reaches a confident answer or exhausts its iteration limit.Source: §3 · p. 3

3.1 Visually Grounded Chart Reasoning

• The ReAct-style loop cycles through Thought, Action, and Observation, selecting visual or numerical subtasks, executing chart tools, and inspecting outputs to adjust tools or parameters.Tool examples include pie segmentation, bar isolation, legend detection, axis localization, and interpolation; unreliable tool-based reasoning can fall back to the base MLLM.Source: §3.1 · pp. 3–4

3.2 Chart Interaction and Manipulation

• The modular tool library treats chart elements as visual objects and combines universal tools with chart-specific tools to support diverse chart formats.Its primitives target shared components such as legends, axes, ticks, bar segments, and pie slices.Source: §3.2 · pp. 4–5

Chart Metadata Extraction and Orchestra-

• An orchestrator extracts chart metadata and routes annotated or qualitative queries directly to the base MLLM, while unannotated charts and numeric queries trigger iterative visual reasoning.For unannotated charts, metadata also informs chart-type-specific few-shot retrieval and tool parameterization; the system can fall back to direct MLLM reasoning when tool outputs remain insufficient.Source: Figure 2 · p. 4, Chart Metadata Extraction and Orchestra- · p. 5

4.1 Datasets

• ChartBench contains 3,800 chart–QA pairs across 9 categories and 42 subtypes, while ChartX contains 1,152 pairs across 18 chart types; both include substantial unannotated and numeric-QA subsets.The datasets also include relationship QA in ChartBench and value-comparison or global-perception QA in ChartX.Source: §4.1 · pp. 5–6

4.2 Baselines

• The comparison covers 42 baseline models, including proprietary, open-weight general-purpose, chart-specific, and concurrent models, evaluated with zero-shot and Chain-of-Thought prompting.The candidate set includes models released after or around the dataset release dates.Source: §4.2 · p. 6

4.3 Evaluation Metrics

• Accuracy is computed after GPT-4o standardizes responses and ground truth, using a strict 5% relative error tolerance for numeric answers and exact string matching for non-numeric answers.The authors report that LLM-as-a-Judge can inconsistently enforce numeric thresholds or miss small deviations.Source: §4.3 · p. 6

5.1 Performance

• ChartAgent reaches 71.39% overall accuracy on ChartBench (+16.07% over Phi-3 Vision) and 59.69% on ChartX (+2.83% over GPT-4o).On ChartBench, it also scores 60.81% on unannotated charts (+17.31% over Qwen2-VL) and 70.91% on numeric QA (+15.02% over Phi-3 Vision).Source: §5.1 · p. 6, Table 1 · p. 7
• ChartAgent’s largest ChartBench gains include Bar charts (up to +65%), Box charts (up to +69%), and Ring and Sector charts (up to +62%).On ChartX, the most substantial improvements are on Bubble, Ring, and Treemap charts.Source: §5.1 · p. 7, Table 2 · p. 8

5.2 Effectiveness of ChartAgent

• ChartAgent leads most complexity tiers on unannotated charts, with gains up to +21.2% on reasoning Easy tasks, but visually Hard charts remain challenging.Visually Hard charts score 17.9%; 3D, radar, and overlapping structures can obscure segment boundaries and axis references.Source: §5.2 · p. 8, Figure 4 · p. 8

Plug-and-Play Generalization Across MLLMs

• ChartAgent improves accuracy over its base MLLM on unannotated and numeric Chart VQA across GPT-4o, GPT-4o-mini, Claude 3 Haiku, and Pixtral.Absolute gains are +26.7%, +23.9%, +28.4%, and +12.2%, respectively.Source: Plug-and-Play Generalization Across… · p. 8, Figure 4 · p. 8

5.3 Additional Analysis

• Among 30 sampled ChartBench trajectories, visual self-verification recovered 70% of flagged unsatisfactory tool outputs, while unresolved tool failures contributed to a 15% overall error rate.Correct tool outputs needed no recovery in 50% of cases; of the remaining 50%, recovery failed 30% of the time.Source: Figure 3 · p. 7, §5.3 · p. 9
• ChartAgent outperforms ReAct without tools and ReAct with generic natural-image tools by +32.6% and +30.0% overall, respectively.On the unannotated-plus-numeric subset, the gains are +38.8% and +37.8%, respectively; all variants use GPT-4o and a 15-step limit.Source: Figure 4 · p. 8, §5.3 · p. 9
• In 30 sampled unannotated numeric-QA trajectories, fallback occurred below 10%, typically after inconsistent estimates or empty or missing OCR and detection outputs.The agent reverted to the base MLLM when it judged tool-based reasoning unreliable.Source: §5.3 · p. 9

5.4 Failure Mode Analysis

• Most failures are perception-driven, including OCR obstruction, poor contrast, occlusion, segmentation errors, overlapping series, and axis interpretation issues.Reasoning-related failures include incorrect tool choice, ambiguous queries, and duplicated labels.Source: §5.4 · p. 9

6 Conclusion

• The paper presents visually grounded reasoning through a multimodal chart agent as its central contribution.It describes ChartAgent as a novel agentic framework for chart reasoning.Source: §6 · p. 9
• ChartAgent uses a multi-turn, tool-augmented interaction loop for visually grounded chart reasoning, inspired by human iterative reasoning and annotation-based comprehension.The conclusion reports state-of-the-art performance on benchmarks spanning 40+ chart types and gains over 40+ baselines, particularly on unannotated charts and numeric QA.Source: §6 · p. 9
• Analyses report robustness across visual and reasoning complexity levels, plug-and-play generalization across MLLMs, and effectiveness of agent components.The conclusion also notes a failure mode analysis.Source: §6 · p. 9

Task Coverage and Context

• Evaluation is limited to single charts; multi-chart and slide-level scenarios remain future work, and visual in-context examples trade richer supervision against context length.The current task is question answering, with other tasks described as potential extensions.Source: §7 · p. 10

Computation and Latency

• Iterative reasoning, tool execution, and verification with large proprietary models add inference latency and cost, despite accuracy gains valued for precision-critical settings.The authors identify parallelization, smart routing, and caching as possible ways to reduce latency.Source: §7 · p. 10

Vision Tools and Query Handling

• The vision tools are manually designed but operate at the component level across 40+ chart types; on-the-fly tool construction and query clarification remain future work.The modular design can benefit from advances such as stronger OCR or segmentation models.Source: §7 · p. 10

Evaluation of Tool-Level Behavior

• No standard quantitative tool-level accuracy assessment is available because intermediate outputs lack ground-truth annotations; the paper instead reports end-task results and tool-use analyses.The authors provide tool-use statistics, error-propagation and recovery analyses, and qualitative trajectories; self-verification lets the agent assess tool sufficiency without manual heuristics.Source: §7 · p. 10

Enhancing Coverage for More Chart Types

• ChartAgent performs strongly on common real-world chart types, but harder 3D and radar plots remain targets for improvement because of depth distortion and radial coordinates.The authors propose exploring 2D projection correction and angle-to-numerical conversion for these formats.Source: §7 · p. 10

Broader Perspective

• The authors advise against automating critical chart- or image-related tasks with this framework or MLLM agents without human oversight, and limit accompanying resources to research purposes.Source: §7 · p. 10

Disclaimer

• J.P. Morgan disclaims warranties and liability for the document’s completeness, accuracy, or reliability, and says it is not investment research, advice, or a transaction recommendation.Source: Disclaimer · p. 11

A Annotated vs. Unannotated Charts

• Annotated charts provide textual value cues, while unannotated charts require values to be inferred from graphical features such as bar heights, pie-slice angles, or axis positions.Textual cues may also appear in chart titles, legends, or captions, and can let models extract information without complex visual reasoning.Source: A Annotated vs. Unannotated Charts · p. 16

A.1 Examples

• Figure 5 shows representative annotated and unannotated chart examples, including a bar chart with printed values contrasted with one requiring visual inference.Source: §A.1 · p. 16

A.2 How ChartAgent Handles Annotated vs. Unannotated Charts

• ChartAgent routes annotated charts directly to the base model, while unannotated charts trigger its full ReAct-style loop; a sampled subset yielded 100% classification accuracy.The base model already exceeds 90% accuracy on annotated charts, according to Table 1.Source: §A.2 · p. 16

B Related Work

• The related-work review is organized around chart VQA, multimodal LLMs and visual grounding, and agentic frameworks.Source: B Related Work · p. 16

B.1 Chart Visual Question Answering

• Recent chart VQA benchmarks broadened chart and query diversity, but precise numerical interpretation remains difficult on unannotated charts, motivating specialized visual grounding.Prior work found performance drops from annotated to unannotated charts, indicating reliance on OCR rather than visual reasoning; this approach adds chart-tailored modular vision tools.Source: §B.1 · pp. 16–17

B.2 General-Purpose Multimodal LLMs and Visual Grounding

• General-purpose MLLMs struggle with precise visual grounding, while ChartAgent combines iterative reasoning, dynamic visual prompting, and modular chart tools for adaptive replanning.Unlike fixed-plan tool approaches, its intermediate visualizations support inspection and decision-making at each reasoning step.Source: §B.2 · pp. 17–18

B.3 Agentic Frameworks

• ChartAgent adopts ReAct’s iterative multimodal reasoning and uses AutoGen to orchestrate interactions between the MLLM and chart-specific perception tools.The reviewed agentic frameworks span applications including software engineering, robotics, vision-language reasoning, and GUI navigation.Source: §B.3 · p. 18

C Datasets

• The evaluation focuses on visually grounded QA over unannotated charts using ChartBench and ChartX, benchmarks intended to test chart comprehension and data reliability.Most benchmark charts are unannotated, so answers cannot rely solely on OCR.Source: C Datasets · p. 18

C.1 ChartBench

• ChartBench spans 9 categories and 42 subcategories, with over 75% of images unannotated; the filtered test set contains 1,900 images and 3,800 QA pairs.Its tasks include numeric and relationship QA, and its chart types range from common plots to complex combinations.Source: §C.1 · pp. 18–19

C.2 ChartX

• ChartX covers 18 chart categories in 1,152 test image-question pairs, with more than 60% unannotated; its questions include numeric extraction and comparison or global-perception reasoning.Its challenging samples include occlusions, multi-axis plots, and chart elements difficult to distinguish from grid lines.Source: §C.2 · p. 19

C.3 Dataset Statistics

• Over 76% of ChartBench and over 61% of ChartX samples are unannotated; numeric QA exceeds 94% and is nearly 72%, respectively.The authors describe these dataset properties as creating visually demanding and numerically intensive testbeds.Source: §C.3 · p. 19

D Chart Types Supported in ChartAgent

• ChartBench covers 9 major categories and 38 subcategories, while ChartX contains 18 chart types in general, fine-grained, and domain-specific groups.Four ChartBench subtypes with corrupted or incorrect ground-truth labels are excluded.Source: D Chart Types Supported in ChartAgent · pp. 19–20, Figure 6 · p. 21

E Baselines

• Table 4 summarizes proprietary and open-weight baseline MLLMs, including general-purpose and chart-specific models and their architecture details.The table reports model versions or component architectures and, where applicable, vision-encoder and language-model parameter sizes.Source: E Baselines · p. 23, Table 4 · p. 23

F Taxonomy of Tools in ChartAgent

• ChartAgent’s tools combine universal chart primitives with chart-specific operations, supporting more than 40 chart types and extension through composing or adapting existing primitives.Universal tools cover elements such as axes and legends; chart-specific tools target structures such as pies, bars, and box plots.Source: F Taxonomy of Tools in ChartAgent · p. 25

F.1 Underlying Models Powering ChartAgent Tools

• SAM generates candidate chart-element masks that are filtered for chart-relevant segments, while Tesseract handles default OCR and EasyOCR is a fallback for complex or noisy charts.The segmentation tool targets elements such as pie slices, bars, and areas; OCR extracts axis ticks and legend text.Source: §F.1 · p. 25

F.2 Tool Outputs and Intermediate Visualizations for Self-Verification in ChartAgent

• ChartAgent inspects interpretable tool visualizations and can adjust parameters or switch tools when outputs appear incorrect or inconsistent.The authors describe this recover-and-retry loop as supporting visual self-verification.Source: §F.2 · p. 26, §F.3 · p. 26

F.3 Adaptive, Heuristic-Free Visual Self-Verification

• Tool-level accuracy is not evaluated directly because ground truth for intermediate outputs such as masks, ticks, or boxes typically does not exist.The paper therefore focuses on end-task performance while letting the agent interpret and adapt tool outputs.Source: §F.3 · p. 26

G.1 ChartAgent / G.2 Baselines

• ChartAgent uses GPT-4o as its primary reasoning model and averages 5–7 model calls per sample; baselines use deterministic settings and applicable models receive zero-shot and Chain-of-Thought prompts.The implementation allows up to 15 reasoning iterations as a safeguard; chart-based baselines that return tables receive a GPT-4o call to extract the final answer.Source: §G.1 · p. 26, §G.2 · p. 30

H Examples of Response Standardization for Accuracy Evaluation

• GPT-4o standardizes both responses and ground-truth answers before arithmetic correctness checks or string matching.The operations include converting units or scales and removing formatting differences.Source: H Examples of Response Standardization… · p. 30, (4) Standardizing Number Formats · p. 30

(4) Standardizing Number Formats

• Standardization handles scale and unit variants, including converting 2 days to 48 hours and stripping symbols such as $, %, K, and M.The examples also include scale abbreviations for thousands, millions, and billions.Source: H Examples of Response Standardization… · p. 30
• Over 97% of reviewed standardizations were accurate across 100 sampled examples per dataset.The remaining errors involved convoluted answers with multiple entangled values or ambiguous final quantities.Source: (4) Standardizing Number Formats · p. 30

I Analysis of Numerical Tolerance Choices in the Evaluation Metric

• The evaluation adopts a 5% relative-error threshold to balance strictness with visual ambiguity in chart-value reading.The paper notes that different application contexts may warrant different numerical tolerances.Source: I Analysis of Numerical Tolerance… · p. 30
• Six relative-error thresholds—0.1%, 1%, 3%, 5%, 10%, and 15%—are used to examine performance under differing precision requirements.The stratified analysis is intended to simulate varying risk sensitivity and precision requirements.Source: I Analysis of Numerical Tolerance… · p. 30
• Accuracy improves as relative-error tolerance widens, including at 10–15% settings; ChartAgent consistently achieves the highest accuracy across all thresholds, showing its advantage is not overly dependent on the standard 5% threshold.Source: I Analysis of Numerical Tolerance… · pp. 30–31

J Complexity Analysis

• Both datasets cover Easy, Medium, and Hard visual and reasoning complexity; fewer than 15% of charts are visually Hard, and ChartX has a larger Hard share.Reasoning tasks include substantial Medium and Hard cases; two annotators and the automatic labels disagreed on average 8% of reviewed examples.Source: J Complexity Analysis · p. 31

K.1.1 Agent Trajectories on Unannotated Charts and Numeric QA

• For unannotated numeric questions, the orchestrator triggers chart tools, retrieves chart-specific few-shot examples, and uses a multi-turn interaction to produce answers.The illustrated trajectories span multiple chart types, including pie, bar, line, area, combination, radial bar, and treemap charts.Source: §K.1.1 · p. 33, Figure 11 · p. 34, Figure 12 · p. 34

K.1.2 Agent Trajectories Demonstrating Visual Self-Verification in Action

• In illustrated cases, visual self-verification is followed by correction of the agent’s response.Figures 22–24 show trajectories in which verification was invoked and the response was subsequently corrected.Source: §K.1.2 · p. 39, Figure 22 · p. 39, Figure 23 · p. 40

K.1.3 Agent Trajectories on Annotated Charts

• For annotated charts, the orchestrator routes questions to direct GPT-4o reasoning, which produces the correct answer in the shown examples.The trajectories are presented as sample cases rather than a quantitative evaluation.Source: §K.1.3 · p. 41, Figure 25 · p. 41

K.1.4 Some Interesting Agent Trajectories

• When tools fail or cannot capture small points, ChartAgent may recognize the limitation and switch to visual judgment or direct reasoning.One example attributes interpolation failure to having only one y-axis value available.Source: §K.1.4 · p. 41

K.2 Representative Examples

• Qualitative comparison examples show improved performance across chart types on both ChartBench and ChartX.The examples compare ChartAgent with several baseline models.Source: §K.2 · p. 41, Figure 26 · p. 43

L.1 Performance by Chart Type

• Table 8 compares baseline accuracy on unannotated charts by chart type, but the supplied text gives no chart-specific scores.The caption identifies red as best and blue as second best.Source: Table 8 · p. 44, §L.1 · p. 44

L.2 Analysis of Tool Usage in ChartAgent

• Tool use varies by chart type: universal tools appear across nearly all types, chart-specific tools are used when structurally required, and combination charts show the most diverse use.Similar usage distributions for annotate_legend and get_marker_rgb suggest they are often used together in trajectories.Source: §L.2 · p. 45

L.3 Ablation Study

• Chart-specialized tools with iterative visual self-verification achieve state-of-the-art performance, particularly on unannotated charts and numeric QA.The ablation compares no tools, generic crop/zoom tools, and ChartAgent, using GPT-4o and visual self-verification across settings.Source: §L.3 · pp. 45–46, Table 10 · p. 46

L.4 Visual and Reasoning Complexity Analysis

• Table 11 stratifies unannotated-chart accuracy by visual chart complexity and reasoning complexity of the chart–QA pairs.The supplied passages describe the analysis but do not report its numerical outcomes.Source: §L.4 · p. 46, Table 11 · p. 47

L.5 Accuracy vs. LLM-as-a-Judge

• LLM-as-a-Judge often relaxes the 5% margin condition, inflating ChartBench accuracy relative to arithmetic evaluation that strictly enforces it.The comparison is reported for ChartBench.Source: §L.5 · p. 47, Table 12 · p. 47

L.6 Concurrent Works

• Because ChartBench and ChartX postdate GPT-4o’s October 1, 2023 knowledge cutoff, they were not part of that model’s training data.The datasets were released in December 2023 and February 2024, respectively.Source: §L.6 · p. 47, Table 13 · p. 48

L.6.1 Performance of Concurrent Works on Public Benchmarks

• Correct answers paired with incorrect reasoning or tool outputs in newer models raise the possibility of exposure to public benchmark data.The authors describe these observations as preliminary and curated a held-out internal dataset for more rigorous evaluation.Source: §L.6.1 · p. 48

L.6.2 Performance of Concurrent Works on the Internal Dataset

• +10.48% absolute accuracy gain over the second-best model (GPT-5) and a 5.72-point reduction in average absolute error relative to GPT-o3 on a curated internal dataset.The dataset contains 125 chart–QA pairs selected to reduce the likelihood that answers appeared in newer models’ training data.Source: §L.6.2 · p. 48, Table 15 · p. 49

L.7 Visual Self-Verification and Recovery Behavior

• In 30 sampled ChartBench trajectories, self-verification was triggered in half of cases, and 70% of those recoveries produced correct final answers.The remaining 30% of triggered cases failed to recover, contributing to a 15% overall error rate attributed to unresolved tool-level failures.Source: §L.7 · p. 48, Table 16 · p. 49

L.8 Fallback Analysis: When ChartAgent Reverts to the Base Model and Common Trigger Conditions

• Across selected ChartBench trajectories on unannotated charts and numeric QA, fallback to GPT-4o was under 10%; common triggers included inconsistent bar-height estimates, failed OCR detection, and unreliable line-chart outputs.Source: §L.8 · p. 49

• Overlays, poor contrast, occluded legends, indistinguishable elements, overlapping axes, and dense data-series overlap can obstruct OCR or visual segmentation.The examples span bars, box plots, radar plots, line charts, scatterplots, and filled regions.Source: M Details on Failure Mode Analysis · p. 50
• Complex axes, including 3D distortion or multiple scales, can make it difficult to map chart elements to the correct values.Source: M Details on Failure Mode Analysis · p. 51
• Reasoning failures include unit mismatches, wrong measurement-tool selection, ambiguous denominators, duplicated labels, and area-chart subtype confusion.Overlay and stacked area charts can look similar, and misclassification can lead to incorrect answer logic even when other steps are correct.Source: M Details on Failure Mode Analysis · p. 51
• Most failures are perception-driven, arising from chart-tool errors rather than complex reasoning or planning.The reported examples include OCR obstruction, low contrast, occlusion, invisible elements, segmentation difficulty, overlap, 3D distortion, and multiple-axis mapping.Source: M Details on Failure Mode Analysis · pp. 50–52, Figure 28 · p. 52

N.1 ChartAgent Prompts

• ChartAgent’s prompt suite covers reasoning, tool use, metadata extraction, and in-context learning, with examples selected to match the detected chart type.The appendix presents the prompts as an overall concatenated template and then as separate components.Source: §N.1 · p. 53
• The agent generates a thought and one notebook action at a time, reviews the tool output, and continues until it can provide an answer and terminate.The prompt directs it to use visual and numerical outputs, and to retry with adjusted parameters or another tool when results are uncertain.Source: §N.1 · pp. 53–54

N.1.1 System Prompt

• The system prompt casts the model as a multimodal chart-QA assistant that uses structured reasoning, tools when appropriate, and a defined answer format.It operates in a Python Jupyter environment and is instructed to execute one tool at a time, wait for results, and retry after errors or uncertainty.Source: §N.1.1 · p. 54

N.1.2 Chart Tool Definitions

• Legend tools crop and annotate legend entries, retrieve marker colors by text or label, and clean charts by removing titles or legends.The color-retrieval tool supports fuzzy text matching or numeric label lookup.Source: §N.1.2 · p. 55
• Segmentation labels image regions and filters small, duplicate, composite, or background-dominated masks to support isolating chart components.The tool returns a labeled image and a cleaned list of segmentation masks.Source: §N.1.2 · p. 56
• Axis tools detect tick values and pixel positions, while linear interpolation maps a pixel coordinate to its corresponding axis value.Axis localization uses Tesseract OCR and EasyOCR, with optional supplied tick labels to improve matching.Source: §N.1.2 · p. 56
• Quantitative tools compute chart-segment area by counting colored pixels, discrete visual elements, or selected masks, and perform arithmetic including percentages and ratios.The area tool is described as commonly used for pie charts and treemaps.Source: §N.1.2 · p. 57
• Bar tools locate bars by color or axis label, then map pixel bounds to values using OCR-localized axes, including left/right y-axes and horizontal x-axes.Source: §N.1.2 · p. 58
• Boxplot tools filter segments by color, axis label, or segmentation index, then compute statistics such as median, quartiles, range, or interquartile range.Source: §N.1.2 · p. 59
• For line, area, and scatter plots, edge-point detection filters chart segments by color, label, or mask and returns bounds perpendicular to the matched label.Source: §N.1.2 · p. 60
• Radial tools locate a segment, identify the chart center and outer circle, then estimate its value by scaling its maximum radius against an external reference value.The outer-circle reference defaults to 100.Source: §N.1.2 · p. 61

N.1.3 Chart Metadata Extraction

• Metadata extraction records chart type, title, legends, axes and ticks, annotation status, and visual structure to retrieve examples and parameterize later tool calls.Legend fields include hierarchical categories and whether the legend is embedded; axis fields cover labels and tick values, including right-y and radial axes.Source: §N.1.3 · p. 62

N.1.4 In Context Learning

• At inference, the prompt appends only in-context examples matching the detected chart type; when none exist for that type, it adds no examples.Source: §N.1.4 · p. 63
• The in-context prompt also instructs self-verification, recovery from errors, parameter changes, alternative tools, and fallback to the base multimodal model.Source: §N.1.4 · p. 63
• In the GDP example, the agent identifies India’s legend marker and color, cleans the chart image, then targets the India bar for 2020.It avoids cropping the legend when it is too close to an axis, to preserve axis content.Source: §N.1.4 · pp. 63–64
• The worked bar-chart example maps a detected bar to a value height of 4.2 and answers that India’s 2020 GDP growth rate is 4.2.Source: §N.1.4 · p. 64

N.2 Baseline Prompts

• Baseline comparisons use zero-shot and chain-of-thought prompts across proprietary and open-weight MLLMs, with ReAct for ablations and tabular QA for selected table-output baselines.Zero-shot asks for a direct answer without intermediate reasoning or tools; CoT requests stepwise reasoning before the answer.Source: §N.2 · p. 64, §N.2.1 · p. 64, §N.2.2 · p. 65

N.2.3 ReAct

• ReAct interleaves reasoning, actions such as tool calls or code, and observations until a final answer, isolating chart-specialized visual tools in ablations.Source: §N.2.3 · p. 65

N.2.4 Tabular Question-Answering

• For selected baselines that output structured tables, GPT-4o uses the extracted table and question to produce a concise answer.Source: §N.2.4 · p. 65

N.3.1 Accuracy

• Evaluation standardizes predictions and ground truth before checking numeric answers within a strict 5% relative-error tolerance and nonnumeric answers by string match.GPT-4o removes units and symbols and normalizes scales and number formats before the arithmetic comparison.Source: §N.3 · p. 66, §N.3.1 · p. 66

N.3.2 LLM-as-a-Judge

• GPT-Accuracy asks GPT-4o to judge prediction correctness directly, allowing 5% error for numeric answers, but may be imprecise for fine-grained numeric evaluation.Source: §N.3.2 · p. 67

N.4.1 Visual Complexity

• Visual complexity is rated Easy, Medium, or Hard from the chart image alone, considering visual elements, clutter, comparisons, and legibility, with ambiguous or layered charts tending toward Hard.Source: §N.4.1 · p. 68

N.4.2 Reasoning Complexity

• Reasoning complexity is rated Easy, Medium, or Hard from the chart-question pair, considering perception, cross-referencing, ambiguity, and numerical steps; direct reads are typically Easy, while multi-step comparisons are Hard.Two-step quantitative reasoning and arithmetic across extracted values are typically Medium.Source: §N.4.2 · pp. 68–69