PaperlayerView exact revision on arXiv

Source-linked AI summary

InsightChain: Optimized Chain-of-Insight Analytics for LLM-driven Data Visualization

Hanya Sun, Chen Zhang, Sheng Liang, Yongyue Zhang, Yong Liu

arXiv:2609.06438v1cs.CLcs.HC

TL;DR

Existing visualization approaches often map a query directly to a figure or code, leaving iterative analytical reasoning and evaluation of intermediate evidence under-addressed. InsightChain stages exploration through presentation, while VG-COPRO optimizes the executable pipeline and IPM evaluates its progression; reported experiments show advantages over included baselines and gains from VG-COPRO across both settings.

  • Problem

    Many visualization approaches use a one-shot query-to-output mapping, while multi-stage analysis lacks evaluation that checks whether intermediate hypotheses are grounded in executed evidence.

    Source: §1 · p. 1

  • Method

    InsightChain uses Explore, Focus, Test, and Present stages, with VG-COPRO adapting prompt optimization to the pipeline and its rendered outputs.

    Source: §1 · pp. 1–2, §3.4 · p. 5

  • Results

    InsightChain outperforms included step-matched baselines across task models, and VG-COPRO improves the chain in both in-domain and cross-domain settings.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The five IPM dimensions do not collapse into one factor, and the visual dimension captures information not covered by text-based dimensions.

    Source: §5.1 · p. 7

    The evaluation lacks a representative general-purpose data-analysis agent adapted with a visualization stage, so results do not establish an advantage over that broader class.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used for automated data visualization, yet existing approaches often frame visualization generation as a single-step mapping from user query to figure or code, overlooking the iterative analytical reasoning process of expert analysts. We present InsightChain, a four-stage visualization prompting pipeline (Explore--Focus--Test--Present) that emulates expert analytical workflows, together with VG-COPRO, a vision-guided automatic prompt optimization (APO) method adapted to jointly optimize such multi-stage, executable pipelines. To address the evaluation gap for complex data visualization, we introduce the Insight Progression Metric (IPM), a rubric combining four text-based dimensions with a vision-based dimension. We assess IPM through a 100-chain human pilot and an expanded 300-chain agent-based evaluation spanning all ten domains. Experiments on public datasets show that InsightChain consistently outperforms competing prompting baselines. Existing APO methods fail to yield consistent gains on this multi-stage task, whereas VG-COPRO improves performance in both in-domain and cross-domain settings.

1 Introduction

Existing visualization approaches often compress analysis into a single query-to-output step, unlike the iterative expert workflows of exploration, hypothesis formation, and encoding refinement. InsightChain, VG-COPRO, and IPM address this gap through staged prompting, pipeline-aware optimization, and evaluation of multi-step analysis.

Motivation

  • Single-step query-to-figure approaches bypass intermediate validation, unlike iterative expert workflows of exploring distributions, forming hypotheses, and refining encodings.The paper argues that compressing this reasoning can yield superficial visualizations lacking statistical rigor and deep insights in complex real-world data.Source: §1 · p. 1

Contributions

  • InsightChain structures open-ended visualization prompting into Explore, Focus, Test, and Present stages that emulate expert workflows.The pipeline is execution-grounded and links multi-stage reasoning with feedback from execution and rendered output.Source: §1 · pp. 1–2
  • VG-COPRO adapts automatic prompt optimization to multi-stage analysis by combining per-predictor search with proposal and vision-aware, pipeline-level evaluation.The paper studies how existing APO methods transfer to this executable pipeline and introduces VG-COPRO as a task-adapted optimizer.Source: §1 · pp. 1–2
  • IPM evaluates both the analytical trajectory and final visualization, and the paper examines its human agreement, repeatability, cross-family robustness, and performance across ten domains.The experiments also examine transfer to a text-to-image renderer.Source: §1 · pp. 1–2

2 Related Work

Related work covers executable visualization systems, analytical discovery and reporting tasks, visualization-quality metrics, and automatic prompt optimization. The paper distinguishes its open-ended, execution-grounded task and motivates evaluation and optimization that account for analytical progression and visual fidelity.

2.1 LLM-driven visualization

  • Existing visualization systems range from goal-then-code and query-conditioned generation to role-decomposed systems, but lack execution-grounded insight progression; InsightChain instead starts without predefined analytical intent (Zhang and Elhamod, 2025).The comparison measures practical performance in an open-ended setting rather than assuming identical task formulations.Source: §2.1 · p. 2

2.2 Insight discovery and analytical report generation

  • Insight discovery and report-generation benchmarks differ from InsightChain in inputs, outputs, and protocols, so their scores are not directly compared.InsightChain starts from a single table without predefined intent and requires executable exploration ending in a visualization.Source: §2.2 · p. 2

2.3 Visualization quality metrics

  • Specification matching can penalize valid alternative charts, caption similarity ignores chart accuracy and renderability, and vision-language metrics may miss data fidelity.IPM is designed to assess both intermediate analytical reasoning and visual fidelity.Source: §2.3 · p. 2

2.4 Automatic prompt optimization

  • Common APO paradigms are typically tested on text-only tasks and may optimize prompts in isolation using text-only signals, motivating VG-COPRO’s pipeline-aware, vision-guided design (Opsahl-Ong et al., 2024; Khattab et al., 2024; Yang et al., 2024; Guo et al., 2024; Wang et al., 2023).The cited paradigms include Bayesian search, coordinate ascent, LLM-as-optimizer, evolutionary search, and fixed structured-template scaffolds.Source: §2.4 · pp. 2–3

3 Methodology

InsightChain turns open-ended visualization into an iterative, execution-grounded four-stage workflow, while IPM evaluates its analytical progression and final visualization. VG-COPRO adapts prompt optimization to this multi-stage pipeline using full-pipeline evaluation, domain robustness, and visual feedback.

3.1 The InsightChain Pipeline

  • InsightChain treats open-ended analysis as four iterative stages, with executable programs, execution feedback, and prior-stage records informing later steps.The stages correspond to Explore, Focus, Test, and Present.Source: §3.1 · p. 3

3.2 Insight Progression Metric (IPM)

  • IPM combines four text-judged stage dimensions and a vision-judged visualization dimension into a weighted score for evaluating multi-step visualization.Visual Quality assesses whether the figure embodies the final finding; its score is capped at 3 for data misrepresentation or fundamental legibility failures.Source: §3.2 · pp. 3–4

3.3 Prompt Optimization Paradigms

  • Five representative APO methods are adapted to optimize one InsightChain stage at a time while holding the others fixed.Plan-and-Solve uses a fixed template once per predictor, serving as a no-search baseline.Source: §3.3 · p. 4

3.4 VG-COPRO: a vision-guided APO

  • VG-COPRO combines COPRO’s per-predictor loop with OPRO proposals and evaluates candidate prompts by executing the complete pipeline.Its modifications provide per-dimension critiques, use the minimum IPM across five in-domain folds as fitness, and include vision-judge assessment in the compile signal.Source: §3.4 · pp. 4–5

4 Experimental Setup

The experiments use 285 tabular datasets across ten domains, with five domains for compilation and in-domain testing and five held out for cross-domain testing. The setup also evaluates alternate judges and task models, and tests transfer to text-to-image rendering.

Datasets

  • 285 tabular datasets span 10 domains, with 150 datasets from five domains used for compilation and in-domain testing and 135 datasets from five held-out domains used for cross-domain testing.91.9% of datasets come from real public sources, with a small synthetic-template fallback for underrepresented demographics and energy domains.Source: §4 · p. 5

Evaluation models and metrics

  • Gemini 3 Flash provides primary IPM judgments and task-model runs, with GPT-4.1 and Qwen-VL-Max re-scoring a stratified 25-chain-per-condition subsample and GPT-4.1 used for cross-model replication.Generated code runs in a sandbox with a 60-second timeout and up to two retries; auxiliary metrics use OpenCLIP, SacreBLEU, and LIDA.Source: §4 · p. 5

APO compilation

  • APO methods share a 20-example validation pool and three compile iterations per predictor; VG-COPRO’s five leave-one-domain-out folds use only the five training domains.The optimizer does not access the five cross-domain test domains.Source: §4 · p. 5

Text-to-Image transferability

  • Text-to-image transfer is tested by rendering the Present stage with gemini-3.1-flash-image-preview while keeping the preceding stages, datasets, IPM weights, and judges unchanged.Source: §4 · p. 5

5 Results

InsightChain outperforms single-shot baselines, and VG-COPRO improves the pipeline’s IPM performance across in-domain and cross-domain tests. The paper also validates IPM and finds that common APO methods do not consistently improve this multi-stage task.

5.1 IPM Validation

  • IPM distinguishes controlled quality bands, separates adjacent levels except MEDIUM-A from MEDIUM-B, and agrees with human overall-quality ratings (Spearman ρ = 0.88).The aggregate IPM yields F(3, 96) = 153.1 with p < 10−36; each dimension separates bands at p < 10−4.Source: §5.1 · p. 6, Figure 2 · p. 6
  • IPM dimensions show reliability and robustness while retaining distinct diagnostic information; merging Focus and Hypothesis leaves all nine method rankings unchanged (Kendall’s τ = 1.0).All dimensions have Krippendorff’s α ≥ 0.85; text–vision correlations are below 0.15, and rankings remain stable under weight changes. Cross-family evaluators preserve non-negative per-condition Kendall’s τ.Source: §5.1 · pp. 6–7

5.2 Chain vs. Baselines

  • InsightChain consistently outperforms the single-shot baseline across task models, with small positive cross-domain transfer and favorable judge-free code metrics.The code metrics favor InsightChain over the single-shot baseline (d ≈ 0.4, p < 10−5).Source: §5.2 · p. 7, Table 2 · p. 9

5.3 APO on InsightChain

  • Existing APO methods show no consistent cross-domain gains, and score-blind proposers can disrupt the reasoning chain; the MALARIA_INC example contrasts their outputs with VG-COPRO’s clearer annotated plot.The non-chain example contradicts its title’s 37% with an annotated 54.9%, while pre-APO InsightChain has cluttered layout and VG-COPRO produces a dual-trend plot.Source: §5.3 · p. 7, Figure 3 · p. 8, Table 1 · p. 8

5.4 VG-COPRO on InsightChain

  • VG-COPRO improves IPM over pre-APO on both splits (p1 < 10−5, d > 1), exceeds OPRO by more than one point, and remains positive across runs.Cross-domain gains were +1.16, +0.44, and +0.91 points (mean +0.84 ± 0.37); in-domain gains averaged +0.86 ± 0.42.Source: §5.4 · p. 8, Table 1 · p. 8, Table 3 · p. 9
  • Each VG-COPRO modification significantly improves both splits; Mod 2 has the largest mean Δ, while the full combination has the largest paired Cohen’s d.Stage diagnostics identify visual quality as weakest for Focus, Test, and Present, and hypothesis rigor as weakest for Explore; the feedback steers prompts toward insight-bearing annotations and suitable encodings.Source: §5.4 · p. 9, Table 3 · p. 9

5.5 Generalization to Text-to-Image Models

  • InsightChain’s performance improvement with a text-to-image Present stage is similar to Table 1, suggesting its benefits are not specific to code-based rendering.The T2I comparison uses the same four-prompt single-shot architecture and reports paired-test p1 < 10−14.Source: §5.5 · p. 9, Table 4 · p. 9

6 Conclusion

The paper presents InsightChain, an execution-grounded four-stage pipeline, alongside IPM and VG-COPRO. Its experiments report advantages over included baselines and gains from VG-COPRO in both evaluation settings.

6 Conclusion

  • InsightChain structures open-ended insight discovery and visualization as Explore, Focus, Test, and Present stages grounded in execution.The paper pairs this pipeline with IPM and the vision-guided optimizer VG-COPRO.Source: §6 · p. 9
  • InsightChain outperforms the included step-matched baselines across task models, while VG-COPRO improves the chain in both in-domain and cross-domain settings.The optimized pipeline also retains its advantage with text-to-image rendering.Source: §6 · p. 9
  • IPM separates controlled quality levels and agrees with human judgments, while judge-free indicators support deeper analysis.Source: §6 · p. 9

Limitations

The authors identify limits in human validation of IPM, baseline coverage, and the diversity of prompt-optimization conditions. The validation-table caption also reports a calibration difference between vision- and text-judged dimensions.

Limitations

  • Full five-dimensional human ratings are limited to the 100-chain pilot.Source: Limitations · p. 9
  • The 300-chain agent-based evaluation supports repeatability and broad-regime separation, but does not fully validate individual dimensions or fine-grained, context-dependent insight quality.Source: Limitations · p. 10
  • The results establish an advantage over included executable-visualization and prompting baselines, not the broader class of agentic exploratory-data-analysis systems.The Gemini-row single-shot and LIDA-style baselines are in-house reproductions, while the GPT-4.1 row directly invokes the installed LIDA package.Source: Limitations · p. 10
  • Five training domains, a 20-task validation pool, and three compilation runs limit optimization-condition diversity; positive unseen-domain and run-level gains do not fully rule out prompt-level overfitting.Source: Limitations · p. 10

A.1 IPM Validation Tables

  • In the pilot table, vision-judged Visual Quality is calibrated to human ratings (Δ = −0.04), while text dimensions have a +1.2-to-+2.2 strictness offset that does not affect rank.Source: Table 5 · p. 13

A.2 IPM Expanded Validation and Close-Pair Evaluation

The expanded validation examines agent agreement, repeatability, broad quality-regime separation, and close-pair discrimination. Supporting materials describe the audit, metric and optimizer comparisons, stage instructions, and VG-COPRO algorithm.

A.2 IPM Expanded Validation and Close-Pair Evaluation

  • Agent mean ratings agree with IPM (Pearson r = 0.88, Spearman ρ = 0.89, MAE = 0.93), and practical meaningfulness rises from 0% for RANDOM/LOW outputs to 95.4% for HIGH outputs.The evaluation covers 300 chains across ten domains, with five dimension scores and independent overall-quality ratings.Source: §A.2 · p. 13
  • Across 508 complete chains, mean off-diagonal inter-dimension correlation is r̄ = 0.19 and the maximum is r = 0.50 for Focus Relevance–Hypothesis Rigor; removing Visual Quality raises Cronbach’s α from 0.39 to 0.44.Source: Figure 4 · p. 14
  • IPM reliably distinguishes outputs when |∆IPM| > 0.8; smaller differences behave as ties.The blind comparison covered 80 pairs, excluding pairs without a rendered Present-stage figure.Source: §A.2 · pp. 13–14
  • The expert audit checks factual correctness against execution evidence and treats its 37 deliberately enriched disagreement cases as unsuitable for population-level agreement estimates.It separately audits effect-size reporting and practical meaningfulness.Source: §A.2 · p. 14

A.3 Metric Comparison

  • Table 8 compares visualization metrics on multi-stage scoring, image use, faithfulness gating, insight awareness, and whether a gold chart specification is required.Source: §A.3 · p. 14, Table 8 · p. 15

A.4 APO Design-Space Summary

  • Table 9 organizes five APO paradigms by search algorithm and search target; Plan-and-Solve serves as a structured-template control without learnable parameters.Source: §A.4 · p. 14, Table 9 · p. 15

A.5 Stage Instructions and VG-COPRO Algorithm

  • The four seed instructions progress from broad exploration to a testable hypothesis, statistical testing, and a polished annotated visualization; Algorithm 1 incorporates VG-COPRO’s three modifications.Source: §A.5 · pp. 14–15

A.6 Optimization Dynamics

Text-only optimization gains do not consistently transfer to held-out multimodal evaluation, whereas vision-aware optimization improves cross-domain IPM. The section also traces stage-level optimization behavior and illustrates a complete healthcare analysis chain.

A.6 Optimization Dynamics

  • OPRO’s validation gains are 0.25, 0.13, 0.14, and 0.30 points for Explore, Focus, Test, and Present, while other variants improve in only some stages.Prompt improves in two stages and is unchanged in the other two; diagnostic-only and vision-only variants improve in three and two stages, respectively.Source: §A.6 · p. 15
  • Across complete cross-domain ratings, five published APO methods score 0.33–0.87 points lower on INSIGHT COMMUNICATION and 0.56–1.19 lower on VISUAL QUALITY than pre-APO InsightChain; vision-aware Mod 3 improves held-out cross-domain IPM by 1.23 points.For OPRO, text-only validation gains in all four stages coincide with a cross-domain full-IPM mean decline from 7.38 before APO to 7.27 after optimization.Source: §A.6 · pp. 15–16

A.7 Complete Qualitative Analysis Chains

  • The qualitative examples cover one in-domain and two held-out cross-domain tasks, including both supported and refuted hypotheses, with stage outputs and decision-relevant statistics retained.Source: §A.7 · p. 16

A.7.1 Healthcare: Mental-Health Survey

  • The healthcare example narrows its analysis from several initial relationships to a hypothesis about hybrid work and productivity, then tests that hypothesis.Source: §A.7.1 · p. 16
  • In the mental-health survey chain, tests found no significant work-arrangement difference in productivity variance or the stress–productivity relationship; all correlations were below |0.10|.The reported variance test has p = 0.3048, and Fisher’s z test has p = 0.2581.Source: §A.7.1 · p. 16

A.7.2 Transportation: Parking Tickets

The transportation chain tests whether Police fines exceed PPA fines, finding a higher Police mean while treating enforcement portfolios as an interpretation rather than a causal estimate. The appendix also presents a held-out ecommerce chain and materials for annotation and prompt-quality regimes.

A.7.2 Transportation: Parking Tickets

  • The transportation dataset contains 50,000 rows and received a composite IPM score of 9.10.Its five-dimension IPM tuple was (8, 9, 10, 9, 10).Source: §A.7.2 · p. 16
  • Police fines averaged US$60.99 versus US$41.81 for the PPA, a 45.87% increase; the chain interprets the difference through enforcement portfolios, not as a causal estimate.The Welch test reported p < 2.2 × 10−16, Cohen’s d = 0.53, and a 95% confidence interval of [US$17.69, US$20.67] for the difference; Figure 7 presents the final visualization.Source: §A.7.2 · pp. 16–17, Figure 7 · p. 18

A.7.3 Ecommerce: Google Play Reviews

  • For Google Play reviews, one-star ratings had 136.9% higher mean thumbs-up counts than five-star ratings, but the pre-set 200% threshold was not met.The difference was statistically significant, but the small effect size and maximum of 397 votes support the chain’s caveat that rare, highly visible complaints drive the mean difference; the jitter plot retains the near-zero mass.Source: §A.7.3 · p. 17, Figure 8 · p. 19

A.8 Annotator Handbook

  • The handbook scores four dimensions from text artifacts and visual quality from the rendered figure, alongside overall quality and three categorical judgments.Annotators assign five independent 1–10 dimension scores; automatic scores remain hidden until human judgments are complete.Source: §A.8 · p. 18

A.9 Quality-Regime System Prompts

  • The quality-regime examples range from LOW one-line tasks and a content-broken random-permutation control to HIGH prompts for thorough exploration and rigorous testing.MEDIUM-A and MEDIUM-B share a prompt set but use different random seeds; the regime prompts cover all four stages.Source: §A.9 · p. 24

A.10 Dataset Tables

The appendix catalogs benchmark and dataset properties, then breaks down domain-level performance. It also situates InsightChain among work on research agents and insight-to-visualization systems.

A.10 Dataset Tables

  • Table 10 compares InsightChain with prior benchmarks, Table 11 reports dataset source composition, and Table 12 gives split sizes for compilation, candidate selection, and testing.The cross-domain test set is identified as the central testbed for the transfer claim; synthetic templates are limited to demographics and energy where real public coverage was thinnest after filtering.Source: §A.10 · p. 24, Table 10 · p. 25, Table 11 · p. 25

A.11 Per-Domain Breakdown

  • InsightChain outperforms both step-matched baselines in every in-domain and held-out cross-domain setting; gains over single-shot range from +0.44 on CLIMATE to +1.85 on SPORTS in-domain.Cross-domain improvements range from +0.77 to +1.16.Source: §A.11 · p. 25, Figure 18 · p. 25

B Extended Related Work

B.1 Deep Research and Data-Analysis Agents

  • InsightChain passes computed findings between executable analysis stages, rather than only dialogue history, and evaluates both the analytical path and rendered figure.The related work contrasts this with agents that structure or reuse intermediate evidence and systems focused on hypothesis search or text–chart reports.Source: §B.1 · pp. 25–26

B.2 Insight Mining and Insight-to-Visualization

  • InsightChain grounds free-form exploration, hypothesis selection, testing, and presentation in execution, while IPM assesses whether the final figure communicates the validated finding.This is contrasted with systems that enumerate predefined patterns or evaluate open-ended insight generation.Source: §B.2 · p. 26

B.3 LLM-as-Judge Evaluation of Visualizations and Multi-Step Reasoning

The related-work discussion distinguishes IPM from chart-validity evaluation and describes VG-COPRO as adapting prompt optimization to a visually evaluated, multi-stage analysis pipeline.

B.3 LLM-as-Judge Evaluation of Visualizations and Multi-Step Reasoning

  • IPM evaluates whether a final chart faithfully communicates a finding established by executable stages, while separately assessing four stages with rubric dimensions.The paper reports human agreement, repeatability, cross-family robustness, and a content-broken RANDOM-PERMUTATION control.Source: §B.3 · p. 26

B.4 Automatic Prompt Optimization

  • VG-COPRO combines per-predictor prompt search and OPRO-style proposals with dimension-specific critique of the executed pipeline and a leave-one-domain-out worst-case selection objective.The discussion contrasts this approach with text-only multi-module prompt optimization and identifies adapting feedback and selection—not prompt search itself—as the novelty.Source: §B.4 · p. 26

Related papers

Loading 2609.06438v1…