PaperlayerView exact revision on arXiv

Source-linked AI summary

Charts as Epistemic Actions for Experience-Driven Agentic Data Analysis

Xinzhe Wang, Xudong Weng, Zang Li, Shaolei Zhang

arXiv:2610.30072v1cs.AI

TL;DR

Data agents’ intermediate reasoning is mostly text-based, while human analysts use charts to expose visual evidence. This paper probes agents’ chart use and proposes CHARTMIND, an experience-distillation method for representation choice. Across held-out benchmarks, it improves analytical accuracy and visual evidence quality over matched baselines and induces chart use in backbones with weak representation-selection habits.

  • Problem

    It is unclear whether agents should use charts as intermediate evidence and how they should choose charts versus text during analysis.

    Source: §1 · p. 1

  • Method

    CHARTMIND distils analytical-operation and representation-choice experience from conditioned trajectories, retaining updates only when a validation gate confirms held-out improvement.

    Source: §1 · p. 2

  • Results

    Across three held-out benchmarks and four LLM backbones, CHARTMIND raises analytical accuracy and visual evidence quality over matched baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A frozen experience package can make representation choice part of an agent’s analytical policy without changing model parameters.

    Source: §1 · p. 2

    Experience is distilled only from PROBEBENCH, and prior-result comparisons are limited to matched splits and protocols.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

Data agents built on ReAct-style loops now execute complex analytical workflows over structured data, yet their intermediate reasoning still runs almost entirely through text. Human analysts instead move between numbers, tables, and charts, reading visual evidence to expose trends and anomalies that text summaries drop. We ask whether charts can serve as internal epistemic actions inside a data agent's reasoning loop, and how to teach an agent to use them. We first build \probe{}, a suite of 200 controlled tasks under four matched interface conditions, and find a persistent gap between what agents can read and what they choose to use. Agents score as well from self-generated charts as from numeric output, yet plotting access alone does not translate into consistent use. Task slices show that charts lead on temporal, pattern, and anomaly reasoning, while text leads on precise reading and conditional verification. We then propose \method{}, which distils two families of experience from conditioned trajectory rollouts. One family records how to select and verify analytical operations, and the other records when to invoke text, charts, or both. A trajectory judge scores outcomes and labels intermediate steps, and a validation gate admits only experience that improves held-out performance. On three held-out benchmarks with four LLM backbones, \method{} raises analytical accuracy and visual evidence quality over matched baselines, and backbones that never drew charts begin using them, all without touching model parameters.

1 INTRODUCTION

The paper asks whether charts can serve as intermediate evidence in data-agent reasoning, motivated by human analysts’ use of charts to expose patterns and anomalies. It probes current chart use and proposes CHARTMIND, which distils experience to guide representation choice.

1 INTRODUCTION

  • The study focuses on how a data agent selects textual and chart evidence at intermediate analytical states.Human analysts move among numerical summaries, tables, and charts, using visual evidence to expose trends, distributions, and anomalies.Source: §1 · pp. 1–2
  • PROBEBENCH finds that charts sustain analysis, plotting access alone does not yield consistent use, and charts and text help on different task slices.Charts lead on temporal, pattern, and anomaly reasoning, while text leads on precise reading and conditional verification.Source: §1 · p. 2
  • CHARTMIND distils analytical-operation experience and representation-choice experience from conditioned trajectories, retaining packages only when held-out performance improves.A trajectory judge scores outcomes and labels intermediate decisions; the frozen package is read as external memory without changing model parameters.Source: §1 · p. 2
  • The authors construct PROBEBENCH with four matched interfaces and four backbones to expose the capability–use gap and measure complementary evidence roles.The suite quantifies where charts and text serve as complementary evidence.Source: §1 · p. 2
  • Across three held-out benchmarks and four LLM backbones, CHARTMIND raises analytical accuracy and visual evidence quality over matched baselines.Backbones with weak representation-selection habits begin using charts; ablations support contributions from rollout diversity, validation gating, and both experience types.Source: §1 · p. 2

2 CHARTS IN THE ANALYTICAL LOOP

PROBEBENCH probes whether charts carry analytical value for agents and how that value differs from text. Its findings show that charts can sustain analysis but that agents do not consistently choose them, and that the two representations help on different task slices.

2.1 PROBE DESIGN

  • PROBEBENCH comprises 200 tasks derived from InsightBench, with question-type and cognitive-load annotations for post-hoc slicing.A shared LLM judge scores answers against target insights on a normalized 0–100 scale; the annotations do not enter the agent context.Source: §2.1 · pp. 2–3
  • The probe compares four interfaces while holding the ReAct agent, model, initial context, interaction limit, and tool budget constant.The conditions are No-tool, Text-only, Chart-only, and Free-style.Source: §2.1 · p. 3

2.2 EMPIRICAL FINDINGS

  • Chart-only slightly outperforms Text-only, showing that self-generated charts sustain answers comparable in quality to those from numeric tables.Figure 1 reports judge scores under the four matched PROBEBENCH interfaces.Source: §2.2 · p. 3, Figure 1 · p. 3
  • Under Free-style, chart use varies sharply by backbone and does not track score, exposing a capability–use gap.Table 1 summarizes joint text-and-chart task counts across four backbones.Source: §2.2 · p. 3, Table 1 · p. 3
  • Charts lead on temporal change, pattern recognition, and anomaly detection; text leads on numeric lookup, information location, and conditional verification.Figure 2 slices results by question type and cognitive load and reports the Chart-only minus Text-only gap.Source: §2.2 · p. 3, Figure 2 · p. 4

3 LEARNING TO USE CHARTS THROUGH EXPERIENCE

CHARTMIND teaches chart use by keeping experience outside model parameters and consulting it within the analysis loop. It stores separate analytical and representation knowledge, distils both from conditioned rollouts, and validates updates.

3.1 STORING EXPERIENCE AS EDITABLE DOCUMENTS

  • The two-layer package stores analytical-operation rules separately from representation rules, which guide when to use text, charts, or both.The data agent retrieves relevant content through a read-only interface, while the builder can create, edit, and version documents.Source: §3.1 · p. 4

3.2 DISTILLING EXPERIENCE FROM CONDITIONED ROLLOUTS

  • Conditioned rollouts vary representation preferences while holding the agent, tools, and interaction budget fixed, including chart-preferred runs with diverse chart types.The rollout design supplies chart-led trajectories even for backbones that would not generate them independently.Source: §3.2 · p. 5
  • A trajectory judge scores final answers and labels key steps, helping the builder compare trajectories and distil transferable analytical and representational patterns.Step labels distinguish Helpful, Ineffective, and Harmful actions and separate analytical operations from representation choices.Source: §3.2 · p. 5
  • The validation gate accepts a proposed package only when validation performance improves; accepted experience is frozen for downstream use.The gate keeps validation performance monotonically nondecreasing and filters proposals that fit a single batch.Source: §3.2 · p. 6

3.3 CONSULTING EXPERIENCE INSIDE THE ANALYSIS LOOP

  • During analysis, the agent reads relevant frozen-package entries to decide whether intermediate evidence should be represented as text, charts, or both.Representation experience guides chart use when the analysis state calls for visual pattern recognition; the model parameters remain unchanged.Source: §3 · p. 4, §3.3 · p. 6

4 EXPERIMENTS

The experiments test CHARTMIND on three held-out benchmarks and analyze how its distillation and experience components contribute to results. Across matched backbones, it improves Overall scores, with ablations examining accuracy, visual evidence, and chart use.

4.1 BENCHMARKS

  • The evaluation covers open-ended multi-table reports, long-horizon 10-K exploration, and exact-answer tasks with spreadsheet, text, and image inputs.Each benchmark follows its official protocol.Source: §4.1 · p. 6

4.2 EXPERIMENTAL SETUP

  • Within each backbone, the baseline and CHARTMIND share the agent stack; experience is distilled using PROBEBENCH, while the three evaluation benchmarks are held out.A single package built with GPT-5.6 Sol is frozen before evaluation and reused across backbones.Source: §4.2 · pp. 6–7

4.3 MAIN RESULTS

  • Overall rose 1.71–5.46 points on DAComp-DA, 1.17–5.47 on DDR-Bench, and 0.10–2.02 on DSBench.The result passages report consistent gains across all four matched backbones on each benchmark.Source: Table 2 · p. 7, §4.3 · p. 7, Table 3 · p. 7
  • On DSBench, Task-level and Competition-level accuracy rise by 0.90 and 1.12 points on average, respectively, without trading off exact-answer correctness.The authors report gains despite strong baseline performance.Source: §4.3 · pp. 7–8, Table 4 · p. 7

4.4 ABLATION STUDIES AND MECHANISM ANALYSIS

  • One-step distillation raises Overall from 54.32 to 56.94; full CHARTMIND reaches 59.78 and exceeds one-step distillation by 1.87 points in Acc. and 8.86 points in Vis.The additional value of iterative refinement is particularly evident in visual evidence construction.Source: §4.4 · p. 8, Table 5 · p. 8
  • Unconstrained rollouts reduce Overall from 59.78 to 57.11, and removing the validation gate lowers it to 56.73; both variants remain above baseline.The larger drops in Vis. than Acc. point to the importance of representation-conditioned evidence and validation-based selection for representation-aware behavior.Source: §4.4 · p. 8, Figure 4 · p. 8
  • Analysis experience raises Acc. by 3.70 points, while representation experience raises Vis. by 13.09 points; combining them yields the best Acc., Vis., and Overall.The two experience families provide complementary rather than redundant guidance.Source: §4.4 · p. 8, Table 6 · p. 8
  • Across 30 candidate updates, the gate accepts eight and raises retained validation score from 72.50 to 85.42; CHARTMIND also increases plotting frequency on three of four backbones.The clearest shift is GPT-5.6 Luna’s move from rarely plotting under baseline to using charts during analysis; interaction depth rises modestly.Source: §4.4 · p. 8, Figure 4 · p. 8, Figure 5 · p. 9

5 RELATED WORK

Prior data-agent work keeps analysis text-centric and often treats charts as report outputs, while chart generation and understanding work typically assumes a fixed chart or plotting request. CHARTMIND instead treats charts as intermediate actions and learns representation choice as analysis unfolds.

Data agents

  • Data-agent systems cover diverse analytical workflows but keep the loop text-centric, treating charts mainly as report artefacts; CHARTMIND makes charts intermediate actions.Examples include DS-Agent, Data Interpreter, AIDE, DS-STAR, AutoKaggle, DeepAnalyze, and InsightBench.Source: §5 · p. 9

Chart generation and understanding

  • Chart-generation and chart-understanding methods improve production or reading, but assume a chart or plotting request is already fixed; CHARTMIND decides when one should exist.The cited work includes methods for rendered-feedback generation, chart-to-code, specialised chart reading, and derendering.Source: §5 · p. 9

Representation choice for reasoning

  • Representation-choice work studies text, tables, and visuals, while CHARTMIND learns state-dependent text or chart selection from labelled analytical trajectories.Related approaches include reasoning over existing charts, comparing input formats, routing table-query pairs, and building visual evidence during report writing.Source: §5 · p. 9

6 CONCLUSION

The paper treats representation choice as part of an agent’s analytical policy, studying charts as intermediate evidence rather than merely report output. PROBEBENCH motivates CHARTMIND, which improves held-out accuracy and visual evidence quality while inducing in-loop chart use without changing model parameters.

6 CONCLUSION

  • Across three held-out benchmarks and four LLM backbones, CHARTMIND improves analytical accuracy and visual evidence quality over matched baselines while inducing in-loop chart use without touching model parameters.The results position representation choice as part of the analytical policy rather than a cosmetic output stage.Source: §6 · p. 9
  • PROBEBENCH finds that agents can extract sufficient evidence from self-generated charts, but plotting access alone does not produce consistent chart use.Charts and text have complementary strengths: charts on temporal, pattern, and anomaly reasoning; text on precise reading and conditional verification.Source: §6 · p. 9
  • CHARTMIND distils analysis and representation experience from conditioned rollouts, scores outcomes, attributes intermediate steps, and retains only updates that pass a validation gate.The two experience families cover analytical operations and choices among text, charts, or both.Source: §6 · p. 9

A TOKEN COST

The section reports token-cost accounting for conditioned rollout collection, experience construction, and downstream evaluation with and without a frozen experience library.

Conditioned rollout cost

  • Table 7 reports train-split token costs for conditioned rollout collection across unconstrained, text-preferred, and chart-preferred policies.The chart-preferred total aggregates three runs per task; the other policies cover 174 tasks each.Source: Table 7 · p. 13, A TOKEN COST · p. 13

Experience distillation cost

  • Table 8 aggregates token costs for experience construction across 30 distillation iterations and 12 validation probes per iteration.The accounting includes the rollout corpus, builder, validation-probe agent, and probe judge.Source: Table 8 · p. 13, A TOKEN COST · p. 13

Downstream evaluation cost

  • Table 9 reports mean prompt, completion, and total tokens per completed downstream task for GPT-5.6 Luna, comparing agents without and with the frozen experience library.Benchmark judges and offline evaluators are excluded from the counts.Source: Table 9 · p. 13, A TOKEN COST · p. 13

B IMPLEMENTATION DETAILS AND CORE PROMPTS

This section details the prompts and procedures for implementing baseline and experience-enabled data agents, conditioned rollouts, trajectory judging, and experience-library construction, while grounding analysis in each task’s dataset and distilling reusable lessons. It also describes how experience packages are organized and updated, and how a held-out validation gate and analysis prompts are used to test candidate packages.

B.1 BASELINE DATA AGENTS

  • CHARTMIND keeps the baseline model, task input, ReAct loop, budget, and report procedure, adding only a read-only experience-library tool and catalogue.Agents must explore the task’s SQLite data before optionally loading a relevant experience, which they must apply using fresh queries.Source: §B.1 · p. 15

B.2 ROLLOUT BUNDLE

  • Construction tasks use five fixed-input rollouts—one unconstrained, one text-preferred, and three chart-preferred—with representation instructions as the only change.Later chart runs are instructed to avoid chart families used in prior runs.Source: §B.2 · p. 16
  • A post-answer judge scores semantic correctness from 0 to 4, normalizes the outcome as fscore = s/4, and attributes each trajectory step.It evaluates after answer generation and judges semantic equivalence rather than wording overlap.Source: §B.2 · p. 17

B.3 EXPERIENCE BUILD

  • Experience construction proceeds in two turns: first planning lessons without tools, then retrieving overlapping entries and applying the plan with repository tools.The builder receives compact judge reports in mini-batches, and repository changes must pass bundle validation.Source: §B.3 · pp. 18, 20
  • The experience builder maintains separate analysis and representation tracks, covering analytical operations and modality, plotting, chart design, and verification decisions.Analysis lessons include filters, aggregations, validation, and metric contracts; representation lessons include when to plot and chart-then-verify.Source: §B.3 · p. 18
  • The builder distills cross-task portable procedures from judge summaries and step attributions, not raw trajectories, memorized answers, or dataset-specific facts.It frames lessons from weak rollouts as preventive rules and lessons from strong rollouts as reusable procedures.Source: §B.3 · pp. 18–19
  • For representation lessons, the builder compares text and chart runs on the same task, using differences in step verdicts and analysis-versus-representation attribution.Data, filter, metric, denominator, or population errors route to analysis; modality, chart, label, axis, and visual-design issues route to representation.Source: §B.3 · p. 19
  • Experience files use analysis or representation categories, matching lowercase snake_case filenames, required name/description/category keys, and non-empty portable guidance.The description states the rule and when it applies in one line.Source: §B.3 · p. 20
  • Use edit_experience for small updates, write_experience for new or substantial rewrites, and bash only to rename or delete files in the skill directory..Source: §B.3 · p. 20
  • Before editing, retrieve all potentially overlapping experiences in the relevant category; refine existing guidance by default, and add only clearly missing ideas.Split bundled ideas or categories, delete duplicate or misleading entries, and leave adequately covered lessons unchanged.Source: §B.3 · p. 20
  • Apply every planned lesson not already covered well enough, with no batch limit on files or edits, then summarize files read, operations taken, and reasons..Source: §B.3 · p. 20
  • Candidate packages are evaluated on the same held-out probes by an unconstrained agent with read-only experience access; the trajectory judge’s mean normalized score feeds Equation 2.The judge prompt is identical to the trajectory-judge prompt.Source: §B.3 · p. 21
  • The validation agent analyzes only the supplied dataset, answers directly with key findings and concrete evidence, and must not mention tools, process, or reference answers.It may use execute_bash; each call is independent, and generated charts are returned as image files.Source: §B.3 · p. 21
  • The experience catalog lists each ID, category, name, and description; load an entry’s full text with read_experience before using it..Source: §B.3 · p. 21

C LEARNED EXPERIENCE CASES

The learned examples encode rules for defining ambiguous metrics, diagnosing elevated group averages, choosing representations for compact results, and validating charts against computed results.

Analysis experience case 1

  • For an ambiguous metric, define its entity, population, numerator and denominator, time basis, grouping, aggregation, and weighting before calculation.The rule also calls for checking field values, avoiding convenient proxies, and reconciling reported conclusions with the metric contract.Source: C LEARNED EXPERIENCE CASES · p. 22

Analysis experience case 2

  • To explain an elevated group average, separate category frequency, within-category intensity, and contribution to the total, then compare component mixes and like-for-like values.The example recommends recomputing after excluding or standardizing suspected high-value components when useful.Source: C LEARNED EXPERIENCE CASES · p. 22

Representation experience case 1

  • For a scalar, ranking, or small grouped summary, prefer concise text or a labeled table unless a chart adds material value.The rule reserves charts for cases where they improve perception of pattern, spread, trend, or scale.Source: C LEARNED EXPERIENCE CASES · p. 23

Representation experience case 2

  • After generating a chart, reconcile its data, population, metric, ordering, units, labels, scales, and claims with the supporting table before presenting it.If validation changes the data or ordering, recompute the source table and regenerate the chart.Source: C LEARNED EXPERIENCE CASES · p. 23
Loading 2610.30072v1…