PaperlayerView exact revision on arXiv

Source-linked AI summary

SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agents

Kuangshi Ai, Haichao Miao, Kaiyuan Tang, Nathaniel Gorski, Jianxin Sun, Guoxi Liu, Helgi I. Ingolfsson, David Lenz, Hanqi Guo, Hongfeng Yu, Teja Leburu, Michael Molash, Bei Wang, Tom Peterka, Chaoli Wang, Shusen Liu

arXiv:2603.29139v4cs.AIcs.GRcs.HC

TL;DR

SciVis agents lack a principled, reproducible benchmark for realistic multi-step scientific visualization tasks. SciVisAgentBench addresses this gap with an expert-grounded taxonomy and multimodal outcome-centric evaluation, then uses the framework to compare representative agents and reveal capability gaps. The benchmark indicates that specialized agents are efficient and reliable in predefined pipelines, while general-purpose coding agents are more adaptable, and domain-specific augmentation may combine these strengths.

  • Problem

    The community lacks a principled, reproducible framework for evaluating SciVis agents on realistic, multi-step scientific data analysis and visualization tasks.

  • Method

    SciVisAgentBench combines a four-dimensional taxonomy, 108 expert-curated cases, and outcome-centric evaluation using LLM judging, deterministic evaluators, and expert-defined ground truths.

  • Results

    Specialized tool-integrated agents excel in efficiency and reliability within predefined pipelines, while general-purpose coding agents show superior adaptability and overall success across diverse domains.

  • Takeaways & Limitations

    The benchmark provides a standardized yardstick and diagnostic tool for systematic comparison, capability-gap analysis, and architectural choices in SciVis agents.

  • Takeaways & Limitations

    The current benchmark focuses on structured workflows with verifiable outcomes and does not yet address open-ended exploration, long-horizon discovery, or multiple equally valid visualization designs.

Abstract

from arXiv · show

Recent advances in large language models (LLMs) have enabled agentic systems to translate natural-language intent into executable scientific visualization (SciVis) tasks. Despite rapid progress, the community lacks a principled and reproducible benchmark for evaluating these emerging SciVis agents in realistic, multi-step analysis settings. We present SciVisAgentBench, a comprehensive and extensible benchmark for evaluating scientific data analysis and visualization agents. Our benchmark is grounded in a structured taxonomy spanning four dimensions: application domain, data type, complexity level, and visualization operation. It currently comprises 108 expert-crafted cases covering diverse SciVis scenarios. To enable reliable assessment, we introduce a multimodal outcome-centric evaluation pipeline that combines LLM-based judging with deterministic evaluators, including image-based metrics, code checkers, rule-based verifiers, and case-specific evaluators. We also conduct a validity study with 12 SciVis experts to examine the agreement between human and LLM judges. Using this framework, we evaluate representative SciVis agents and general-purpose coding agents to establish initial baselines and reveal capability gaps. SciVisAgentBench is designed as a living benchmark to support systematic comparison, diagnose failure modes, and drive progress in agentic SciVis. The benchmark is available at https://scivisagentbench.github.io/.

1 INTRODUCTION

SciVisAgentBench addresses the lack of reproducible evaluation for complex, multi-step scientific visualization agents by combining expert-grounded benchmark cases with multimodal validation. It focuses on structured, verifiable outcomes while supporting extensibility and future evaluation of more open-ended workflows.

  • Motivation: Existing visualization benchmarks mainly cover short-horizon tasks, 2D charts, or declarative plotting rather than high-dimensional, multi-step SciVis workflows.They therefore do not capture view-dependent semantics, complex pipelines, or structured scientific analysis.
  • Motivation: SciVis evaluation is difficult because tasks involve multiple operations, admit multiple valid visual outcomes, and intertwine correctness with domain expertise.These challenges make ground-truth specification and objective assessment difficult.
  • Benchmark: SciVisAgentBench provides an expert-curated, extensible benchmark of 108 cases spanning canonical visualization areas and varying difficulty levels.The framework is designed for new agents, domain-specific tasks, datasets, and evaluators to be added over time.
  • Evaluation: The benchmark combines LLM judging with expert-authored rubrics and examines agreement, reliability, and robustness across human and multimodal-LLM evaluation.The validation study compares independent expert and MLLM assessments and tests prompt and presentation variations.
  • Scope: The current benchmark emphasizes structured tasks with explicit, verifiable outcomes as a foundational step toward evaluating open-ended, long-horizon scientific analysis.This scope reflects the reproducibility difficulty of broader settings with multiple valid outcomes.
  • Contribution: The authors release datasets, tasks, ground-truth artifacts, and evaluation rubrics across multiple domains and difficulty levels.The release is intended to support systematic evaluation and continued community development.

2 RELATED WORK

Prior agent and visualization benchmarks establish useful infrastructure but inadequately represent scientific visualization’s iterative, perceptual, tool-mediated, and domain-specific workflows. This fragmentation motivates a comprehensive framework for evaluating end-to-end SciVis agents.

  • General agent benchmarks: General-purpose agent benchmarks evaluate multi-turn task completion but treat visualization generically and omit its exploratory, iterative, and perceptual characteristics.They provide infrastructure without SciVis-specific evaluation coverage.
  • Multimodal evaluation: Multimodal and system-level benchmarks highlight visual reasoning and native-environment evaluation, but do not directly assess scientific visualization workflows.Their settings provide methodological context rather than comprehensive SciVis coverage.
  • Agentic visualization systems: Agentic visualization systems increasingly support visual interpretation, iterative rendering control, script validation, multi-agent coordination, and 3D editing.These systems demonstrate diverse agent responsibilities and integration strategies.
  • Open challenges: Existing SciVis systems nevertheless leave open challenges in reliability, repeatability, and systematic comparison.These challenges motivate a dedicated benchmark for agents operating over scientific data, tools, and workflows.
  • Visualization evaluation: Visualization-specific methods assess readability, code executability, correctness, perceptual quality, and iterative refinement beyond generic code metrics.However, these efforts remain focused on narrower visualization or plotting settings.
  • Adjacent benchmarks: Adjacent NL2VIS, data-science, and remote-sensing benchmarks acknowledge exploratory analysis but stop short of complex end-to-end SciVis workflows.They emphasize query translation, code generation, or tool augmentation rather than scientific rendering pipelines and interaction.

3 BENCHMARK GOALS AND DESIGN REQUIREMENTS

SciVisAgentBench is designed around real-world fidelity, multidimensional assessment, reproducibility, and extensible growth. Its current evaluation prioritizes outcome quality while preserving infrastructure for future process-oriented analysis.

  • Real-world fidelity: The benchmark grounds tasks in authentic SciVis tools, pipelines, data formats, and interactive environments rather than synthetic proxies.Examples include ParaView, napari, and VMD.
  • Multidimensional evaluation: Assessment considers outcome quality, process behavior, and efficiency, with outcome correctness currently primary and execution cost measured through time and token usage.Process-level analysis is deferred because trajectory assessment remains difficult to reproduce reliably.
  • Reproducibility: Explicit visualization outcomes and recorded artifacts support deterministic reruns, systematic agent comparison, and post hoc failure inspection.Additional states or scripts are recorded when supported by the underlying toolchain.
  • Extensibility: Modular task specifications, evaluator interfaces, and agent abstractions allow new datasets, tasks, tools, and metrics to be added without restructuring the benchmark.The contribution pipeline and public repository support versioning, access, and reproducibility.
  • Growth: Forward-looking cases are welcome even when current agents cannot solve them, allowing the benchmark to surface open challenges and guide future research.Such cases must remain well-defined and achievable by human experts.

4 OVERVIEW OF SCIVISAGENTBENCH

SciVisAgentBench organizes scientific visualization cases across domains, data types, complexity levels, and visualization operations while remaining agnostic to agent implementation. Its cases and interfaces support realistic, compositional, and cross-paradigm evaluation.

  • Taxonomy: The taxonomy characterizes scientific domains, dataset structures, analytical operations, and workflow complexity rather than visualization appearance alone.This reflects SciVis as a process supporting knowledge discovery and sensemaking.
  • Application domains: The benchmark covers domains including astronomy, medical science, biology, physics, earth system science, mathematics, and chemistry.These domains are intended to reflect the breadth of real-world visualization practice.
  • Data types: Datasets include scalar, vector, tensor, multivariate, and time-varying data, with some downsampling used to control evaluation costs while preserving analytical characteristics.Data structure influences visualization design and analysis strategy.
  • Complexity levels: Complexity is organized into operations, tasks, and workflows, with reported benchmark cases restricted to task- and workflow-level entries.Operations function as compositional building blocks within those higher-level cases.
  • Visualization operations: Each case is annotated with one or more visualization operations spanning enabling actions and higher-level scientific analysis objectives.The taxonomy includes both low-level controls and operations such as scientific insight derivation.
  • Benchmark resources: The benchmark provides taxonomy annotations, task descriptions, rubrics, ground-truth visualizations, and agent results through its project resources.These materials support inspection and reproducible evaluation of benchmark cases.
  • Agent interfaces: Agent interfaces abstract away architecture and interaction style, accommodating multimodal and multi-agent systems alongside other LLM-based agents.This supports consistent comparison across different implementation paradigms.

5 EVALUATION FRAMEWORK

SciVisAgentBench organizes evaluation around outcome, process, and efficiency dimensions, using structured tasks and multiple evaluators to support reproducible comparison. The framework prioritizes verifiable outcomes while treating open-ended tasks and trajectory-level assessment as current boundaries.

  • Evaluation taxonomy: The taxonomy evaluates SciVis agents through outcome, process, and efficiency dimensions.This organization supports comparison across heterogeneous architectures and large-scale reproducible benchmarking.
  • Outcome-based evaluation: Outcome evaluation treats agents as black boxes and checks whether visualizations satisfy task constraints and semantic requirements.Tasks currently require a single explicit visualization result to reduce ambiguity from nonunique valid solutions.
  • Process-based evaluation: Process evaluation remains incomplete because reliable ground-truth trajectories are difficult to establish when multiple action sequences can produce the same visualization.The benchmark records trajectories, tool interactions, intermediate artifacts, and outcomes for future trajectory analysis, but trajectory-level evaluation is currently impractical for complex path-dependent tasks.
  • Task definition and structuring: Tasks are expert-authored, selected for real-world relevance and clear objectives, and paired with ground-truth artifacts and evaluation rubrics.Rubrics score outcome quality, while additional artifacts can include executable scripts or visualization-engine states.
  • Task definition and structuring: Complex workflows are decomposed into smaller checkpoints with explicit intermediate outcomes, enabling reliable evaluation without unstable process-based scoring.Success on subtasks does not guarantee end-to-end workflow completion.
  • Deterministic evaluators: Outcome assessment combines MLLM rubric scoring with deterministic signals from rule-based evaluators, image metrics, and code checkers.Image-based assessment includes PSNR, SSIM, and LPIPS; scaled variants incorporate the completion rate r = Npass/Ntotal.

6 VALIDITY STUDY

SciVisAgentBench evaluates validity through taxonomy-grounded task design and outcome-focused studies of human and LLM judgments. The study finds strong LLM–human alignment and high judgment stability, while identifying reduced agreement for subjective, higher-level visual interpretation.

  • 6.1 Task Validity: Task validity is grounded in a taxonomy spanning application domains, data types, complexity levels, and visualization operations, with cases annotated to intended capabilities.This coupling is intended to ensure that benchmark success requires the capability each task is designed to evaluate.
  • 6.1 Task Validity: Benchmark coverage is broad but intentionally non-uniform, with fewer tensor-field, geometric, and topological transformation cases reflecting practical frequency and expert-driven construction.The benchmark also excludes sampling and resolution-control operations because they often function as preprocessing and can add computational overhead without directly probing agentic reasoning.
  • 6.2 Outcome Validity: Human-LLM alignment was assessed using 12 SciVis experts, 21 vision-based cases, 65 rubric ratings, and correlations, MAE, and RMSE between averaged human and model judgments.The evaluation used the same task information, ground truth, agent output, rubrics, and scoring guidelines for experts and the LLM judge.
  • 6.2 Outcome Validity: Two human rating outliers were excluded after leave-one-out analysis, improving inter-rater agreement and defining the filtered human evaluations used for subsequent alignment analysis.The initial analysis found that removing either of two experts shifted reliability from insufficient toward tentative conclusions under standard Krippendorff’s α interpretations.
  • 6.2 Outcome Validity: All three LLM judges showed strong positive correlation with human judgments; Gemini-3.1-Pro had the highest correlation, while Claude-Opus-4.6 had the lowest RMSE and closest score distribution.These results support approximation of expert evaluation in the evaluated benchmark setting, without establishing equivalence for open-ended tasks.
  • 6.2 Outcome Validity: All LLM judges achieved Sstable > 0.92 under prompt and presentation perturbations, with Claude-Opus-4.6 achieving the highest stability.Agreement was strongest for explicit, visually grounded criteria and more discrepant for subjective quality judgments or multi-step reasoning over visual evidence.

7 BASELINES AND BENCHMARK RESULTS

The benchmark evaluates specialized SciVis agents and general-purpose coding agents across multiple task suites using complementary LLM-based, deterministic, and image-based measures. General-purpose agents often achieve stronger results, while specialized pipelines can be more efficient on well-defined tasks.

  • Baseline Agents and Setup: The benchmark covers ParaView, molecular, bioimage, topology, and object-identification suites, with agents evaluated only where their supported environments are compatible.General-purpose coding agents were additionally tested across toolchains using minimal access to the underlying visualization engines.
  • Baseline Agents and Setup: Claude-Opus-4.6 was selected as the evaluation judge because it aligned best with SciVis experts and produced stable judgments across repeated trials.Results report mean and standard deviation for overall score and task completion rate across three repetitions.
  • Benchmark Results and Analysis: General-purpose coding agents outperform most specialized SciVis agents across most task suites, while Opus-4.6 judging yields slightly lower means and smaller variances than GPT-5.2 judging.Completion rate measures runs without explicit execution errors, whereas pass metrics require valid outputs such as saved results and non-empty visualizations.
  • Benchmark Results and Analysis: The evaluation combines expert-authored rubrics with task-specific criteria, but vision-based LLM judges remain imperfect for subjective assessment and higher-level scientific interpretation.Representative examples connect numerical scores to failures such as missing streamlines, poor isosurfaces, and mismatched color mappings.
  • Benchmark Results and Analysis: Image-based evaluation on ParaView tasks is consistent with LLM-based evaluation: Claude Code and Codex achieve better visual similarity than specialized baselines.The reported measures are scaled variants of PSNR, SSIM, and LPIPS and require controlled ground-truth renderings.
  • Agent Choices and Tradeoffs: General-purpose agents typically use more tokens and longer execution, whereas MCP-based agents reduce usage through predefined tool pipelines.General-purpose agents need additional environment exploration, code generation, and iterative debugging, while MCP agents can be efficient on predictable tasks.
  • Agent Choices and Tradeoffs: MCP-based agents can outperform general-purpose agents on well-defined domains: BioImage-Agent performs better on bioimage tasks while using significantly fewer tokens.MCP agents may struggle when tasks exceed their predefined capabilities, while general-purpose agents can misuse APIs or capture incorrect outputs.

8 CALL FOR PARTICIPATION

SciVisAgentBench is presented as a starting point for community expansion rather than complete coverage of SciVis. The authors invite contributions that broaden domains and datasets and deepen evaluation toward long-horizon scientific workflows.

  • Expanding Benchmark Breadth: The current benchmark covers diverse domains and data types but underrepresents important areas of SciVis, motivating expansion with new datasets, domains, and scenarios.The stated goal is broader coverage that better reflects real-world practice.
  • Deepening Complexity: Future development should emphasize long-horizon workflows combining environment setup, iterative analysis, visualization design, and result reporting.The authors point to end-to-end workflows comparable to an IEEE VIS contest as a target for increased complexity.
  • Community Contributions: Researchers and practitioners can submit datasets, cases, and evaluation scenarios through the benchmark infrastructure using provided templates and taxonomy.Planned workshops, tutorials, and other community activities are intended to support consistent and reproducible contributions.

9 CONCLUSIONS AND FUTURE WORK

SciVisAgentBench establishes a 108-case, taxonomy-driven benchmark for reproducible evaluation of agentic scientific visualization. Its validated judging framework supports scalable assessment, while baseline results expose complementary strengths between specialized and general-purpose agents and motivate continued expansion toward longer-horizon workflows.

  • SciVisAgentBench comprises 108 cases organized by application domain, data type, complexity level, and visualization operation.
  • The validity study with 12 SciVis experts found that state-of-the-art MLLMs can approximate human expert judgment, including a Pearson correlation of 0.806 for Claude-Opus-4.6.
  • Specialized agents excel in efficiency and reliability within predefined pipelines, whereas general-purpose coding agents show greater adaptability and overall success across diverse domains.
  • SciVisAgentBench functions both as a standardized progress yardstick and as a diagnostic tool for identifying capability gaps and guiding architectural choices.
  • Future work includes expanding underrepresented coverage, extending evaluation to long-horizon workflows, and supporting continuous benchmark evolution through community contributions.

A BENCHMARK TAXONOMY

SciVisAgentBench uses a four-dimensional taxonomy to represent scientific visualization cases: application domain, data type, complexity, and visualization operation. The taxonomy captures both workflow scope and the analytical, rendering, and scientific actions performed.

  • The taxonomy categorizes cases by application domain, data type, complexity level, and visualization operation.
  • It covers eight application domains, including Astronomy, Biology, Chemistry, Earth System Science, Mathematics, Medical Science, Physics, and Others.
  • The benchmark spans five data types: Scalar field, Vector field, Tensor field, Multivariate, and Time-varying.
  • Complexity is divided into operations, tasks, and workflows, with benchmark cases reported only at task and workflow levels.
  • The operation taxonomy includes actions such as data restructuring, feature identification, field computation, geometric transformation, glyph placement, chart generation, insight derivation, slicing, temporal processing, camera control, and volume rendering.
  • The complete benchmark contains 108 cases annotated with their domains, data types, complexity levels, and visualization operations.

C ADDITIONAL RESULTS

The additional results section documents the benchmark’s broad case inventory and reporting structure across task suites. Tables specify case metadata, repeated-trial performance summaries, and qualitative outcome comparisons, with topology results evaluated separately by rules.

  • Table 1 lists all 108 benchmark cases with their domains, data types, complexity levels, and visualization operations.
  • The listed cases span workflows and tasks involving temporal processing, glyph placement, field computation, feature identification, volume rendering, and other operations.
  • Table 2 reports performance across four task suites using GPT-5.2 as the LLM judge, with scores and completion rates averaged over three repeated trials.
  • Topology visualization results are excluded from the GPT-5.2-judged table because they use rule-based evaluation rather than an LLM judge.
  • Table 3 reports Claude Code performance on the bioimage visualization suite under different backbone and skill settings.
  • Table 4 compares expert-defined ground truths with outputs from ParaView-MCP, ChatVis, Claude Code, and Codex using vision-based scores that form only part of the final benchmark scores.

D VISION-BASED OUTCOME EVALUATION EXAMPLES WITH LLM JUDGES

The paper supplements aggregate benchmark scores with qualitative comparisons of expert-defined visualizations and agent outputs. Vision-based LLM-judge scores are interpreted alongside goal-level rubrics and deterministic evaluation components rather than treated as complete benchmark results.

  • Three qualitative examples compare expert-defined ground truths with outputs from different agents and report vision-based LLM-judge scores with goal-level rubric breakdowns.
  • Vision-based scores are not final benchmark scores; final scores also incorporate compliance checks, token usage, execution time, multiple-choice answers, and case-specific validators.
  • The qualitative examples connect score differences to concrete errors such as missing flow structures, incorrect chart construction, weak boundary extraction, and mismatched visual encodings.
  • Each vision-based score sums per-goal rubric scores, with each goal scored from 0 to 10 and normalized by the maximum applicable total.

E FURTHER ANALYSIS ON CODING AGENTS WITH SKILLS

The follow-up analysis shows that reusable, domain-specific skills can reduce tool-use friction and improve coding-agent performance on bioimage visualization tasks. Napari skills increased scores while substantially reducing token consumption, with stronger models solving tasks more efficiently.

  • Motivation: Coding agents often encounter tool-setup and tool-use friction, motivating lightweight skills with working code snippets and tool-specific guidance.The main benchmark omitted additional tool instructions to preserve an unbiased baseline.
  • Skill development: The bioimage follow-up developed a reusable napari skill by inspecting and distilling capabilities from a napari MCP agent without exposing benchmark tasks.The skill also captured practical implementation details and version-specific constraints.
  • Results: Skills improved Claude Code performance for both Sonnet-4.5 and Opus-4.6 while reducing token consumption by roughly half.The direct comparison used GPT-5.2 as judge and evaluated Claude Code with and without the skill.
  • Results: Opus-4.6 consistently achieved stronger results with fewer tokens, suggesting stronger models solved the tasks more efficiently through fewer interaction steps.Table 6 reports mean±std values across three repeated trials.
  • Results: Claude Code with the napari skill scored 70.9 versus 57.6 without it, a 23% improvement with approximately 50% lower token consumption.These results support pairing flexible reasoning engines with modular, domain-specific tool-use knowledge.
Loading 2603.29139v4…