PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Agent Discovery and Resource-Aware Autonomous Exploration of Scientific Datasets

Aashish Panta, Hugo Lee, Giorgio Scorzelli, Kyongsik Yun, Valerio Pascucci

arXiv:2608.22045v1cs.HCcs.AI

TL;DR

Large scientific datasets are difficult to discover, access, and explore because researchers must navigate repository conventions, multiresolution data, and visualization settings. WebVisus combines catalog-grounded multi-agent discovery with constrained autonomous exploration and resource-aware progressive access. The system demonstrates bounded visual exploration across remote datasets while adapting representations to computational constraints.

  • Problem

    Researchers face difficulty discovering and exploring large scientific datasets because repository organization, data formats, multiresolution structures, and visualization parameters require specialized knowledge.

  • Method

    WebVisus uses a constrained multi-agent workflow that discovers datasets from natural-language goals, explores visualizations iteratively, and adapts resolution, spatial extent, access mode, and timestep to available resources.

  • Results

    All five evaluated workflows satisfied their success criteria without intermediate user actions, using an average of 7.8 of 12 iterations and approximately 108 seconds end to end.

  • Takeaways & Limitations

    WebVisus supports autonomous visual exploration of remote multiresolution datasets without full downloads or manual tuning of low-level visualization parameters.

  • Takeaways & Limitations

    The system remains bounded by explicit registered interface actions and resource limits, with researchers retaining the ability to inspect, interrupt, or override execution.

Abstract

from arXiv · show

Modern scientific facilities and instruments generate datasets at scales that are difficult for individual researchers to discover, access, and explore. Although many datasets are publicly available, using them often requires familiarity with repository organization, data formats, multiresolution structures, and visualization parameters. We present WebVisus, a constrained and resource-aware multi-agent system for discovering and autonomously exploring remote, multiresolution scientific datasets. Given a natural-language research question, WebVisus identifies the user's intent and launches an autonomous exploration agent that examines slices, volumes, and timesteps while adapting data resolution and retrieval quality to available client memory and computational resources. This design supports progressive exploration without complete dataset downloads or manual configuration of low-level visualization parameters using natural languages. We report the system architecture, constrained agent protocol, resource-aware access mechanism, and case studies evaluating autonomous visual exploration and resource-aware agentic access across scientific datasets.

I. INTRODUCTION

WebVisus addresses the gap between open-ended scientific goals and grounded dataset discovery, feasible remote access, and iterative visual investigation. It combines catalog-grounded multi-agent discovery with bounded autonomous exploration and resource-aware data retrieval.

  • Motivation: Existing tools typically address catalog search, visualization, natural-language specification, or workflow control after dataset selection and access are resolved.Researchers may still need repository knowledge, dataset identification, variable and timestep selection, and visualization configuration.
  • WebVisus: WebVisus interprets a natural-language goal, searches approximately 1,400 datasets, identifies suitable collections with metadata-grounded tools, and opens selected data through a constrained action interface.The workflow can add new datasets on-the-fly.
  • WebVisus: The system autonomously explores slices, volumes, isovalues, viewing configurations, and timesteps while adapting resolution and transfer size to memory and latency constraints.It reasons about the scientific value of a view and the feasibility of retrieving and rendering it.
  • Contributions: WebVisus provides a bounded exploration agent that observes analysis state, selects visualization operations, evaluates resulting views, and records actions through an inspectable execution protocol.This design makes autonomous exploration inspectable rather than unrestricted.
  • Contributions: Its resource-aware mechanism estimates data-request cost and adapts resolution, access mode, spatial extent, and transfer size to runtime constraints.This mechanism supports exploration when datasets are too large to download or place directly within a model context.

II. RELATED WORK

Prior systems support natural-language visualization, autonomous scientific workflows, and progressive access, but commonly assume that dataset selection and access are already settled. WebVisus extends this workflow from metadata-grounded discovery through resource-feasible remote access and bounded visual exploration.

  • LLM-Based Tools for Scientific Data Visualization: Scientific visualization systems increasingly use language models for natural-language interaction, tool use, visualization design, and visual analysis.The related work includes systems for analytical specifications, transfer-function optimization, dimensionality-reduction analysis, and scientific scene editing.
  • LLM-Based Tools for Scientific Data Visualization: Scientific datasets pose additional challenges because they are often volumetric, time varying, and too large to include directly in a model context.These constraints motivate approaches that operate through visualization systems and remote data access.
  • WebVisus: WebVisus uses a closed-loop workflow in which the agent observes the visualization, selects subsequent actions, and adjusts resolution and transfer size under computational constraints.The workflow supports remote exploration when the complete dataset cannot be downloaded on the client.
  • Agentic AI for HPC and Autonomous Science: Existing agentic systems demonstrate autonomous exploration and scientific visualization, but largely begin after dataset selection and access have been resolved.This leaves the transition from open-ended goals to grounded data access outside much of the prior workflow.
  • WebVisus: WebVisus extends agentic workflows to metadata-grounded discovery, resource-feasible remote access, and bounded visual exploration.Its architecture connects these stages across remote multiresolution data services and controlled application components.

A. Catalog-Grounded Multi-Agent Runtime

WebVisus uses three specialized agents to translate natural-language objectives into catalog-grounded and executable visualization actions. The runtime maintains state, validates commands, and iteratively revises plans as exploration proceeds.

  • Catalog-Grounded Discovery: The catalog snapshot indexes approximately 1,400 datasets across more than 40 groups and supports semantic matching of conversational requests to repository data.Indexed collections include climate, biomedical, materials, microscopy, and other scientific datasets reaching terabytes in size.
  • Three-Agent Runtime: Interactive assistance and autonomous exploration share an Orchestrator, Catalog Agent, and Executor.The Orchestrator interprets the objective and delegates catalog and visualization tasks to the specialized agents.
  • Three-Agent Runtime: The Catalog Agent searches statistics and metadata, recommends relevant datasets, resolves dataset locations, and returns ranked matches with supporting metadata.It serves as the specialist for dataset discovery and repository grounding.
  • Three-Agent Runtime: The Executor converts the plan, catalog results, and current interface state into typed actions such as dataset selection, access-mode choice, timestep changes, and visualization updates.Intermediate state updates can trigger another round of Orchestrator reasoning when the current representation is insufficient.
  • Constrained Action Protocol: Agent execution is restricted to registered commands, while invalid, unregistered, or resource-violating operations are rejected deterministically.The model cannot directly manipulate the DOM, invoke a shell, access the operating system, or construct arbitrary data-service requests.
  • Constrained Action Protocol: Each request can include the selected dataset, access mode, timestep, quality level, active visualization nodes, and optionally a rendered-view snapshot.Successful and failed operations are appended to the interaction transcript.

C. Resource-Aware Progressive Access

WebVisus uses multiresolution array layouts to estimate decoded memory before loading data and select the finest representation within a prescribed budget.

  • Resolution Representation: WebVisus selects representations using an axis-refinement string whose symbols refine the X, Y, or Z spatial dimensions.The string belongs to {0, 1, 2}∗, with n0(s), n1(s), and n2(s) counting refinements along each axis.
  • Memory Estimation: The system estimates decoded memory from the refinement counts and bytes per sample before loading a candidate representation.For b bytes per sample, the footprint is determined by the product of axis refinement factors and b.
  • Budget-Constrained Selection: A refinement example produces an array shape of (4096, 4096, 512), illustrating how refinement choices determine spatial dimensions.For float32 data, the resulting representation can exceed a 4 GiB memory budget, motivating selection of a shorter refinement string.
  • Budget-Constrained Selection: Given memory budget B, WebVisus selects the finest candidate refinement string whose estimated footprint remains within budget.This avoids requiring the user or agent to guess which representation is feasible.

2) Agentic Resource Control:

WebVisus performs resource-aware control at both request planning and browser delivery, adapting data representations to active memory, transfer, and rendering constraints.

  • Agentic Resource Control: The Executor estimates request cost and adapts quality, spatial extent, access mode, and timestep to fit the active budget.If needed, it lowers quality, restricts the region, selects a slice instead of a volume, or limits temporal access.
  • Agentic Resource Control: A second resource check limits browser transfer and display size after the server has checked server-memory requirements.Oversized results can be reduced in resolution, restricted spatially, or returned as slices to reduce transfer time, browser memory use, and rendering cost.

D. Visualization Pipeline

The Playground turns retrieved scalar fields and metadata into interactive two- or three-dimensional visualizations using client-side plotting and WebGL pipelines.

  • Visualization Pipeline: Two-dimensional arrays use Plotly.js, while three-dimensional arrays use a client-side WebGL pipeline implemented with Three.js.A ParaView-style organization supports volume sources, orthogonal slices, isovolume bands, transfer functions, opacity, and camera controls.
  • Visualization Pipeline: Visualization operations reuse the cached texture, allowing display parameters to change without additional data retrieval.Supported changes include slice positions, colormaps, opacity, isovalue ranges, and camera settings.

E. Autonomous Exploration Agent

WebVisus autonomously explores remote scientific datasets through a constrained Observe-Plan-Act-Evaluate loop that adapts views and data access without intermediate user configuration.

  • Autonomous Exploration Agent: The agent combines catalog grounding, progressive access, and constrained visualization control to pursue a natural-language scientific objective.It resolves a relevant dataset, retrieves an initial budget-compliant representation, and determines subsequent operations autonomously.
  • Autonomous Exploration Agent: Each iteration supplies the agent with visualization state, array statistics, selected parameters, recent operations, rendered views, and catalog metadata.The agent evaluates the current state and proposes a small set of actions toward the objective.
  • Adaptive Exploration: The agent can change axes, switch slice and volume views, move through time, adjust transfer functions, or reduce representations when views are uninformative or costly.A client-side backoff mechanism lowers retrieval settings when operations time out or exceed expected duration.
  • Run Completion: Execution ends when the objective is complete, the step limit is reached, or the user stops the run, after which WebVisus generates a grounded exploration summary.The transcript records observations, plans, actions, execution results, and resource decisions.
  • Evaluation: The evaluation includes autonomous improvement of a remote tomography visualization and progressive-access latency measurements across six datasets.The first study uses a fixed resource budget and constrained action space; the second examines which data scales support closed-loop exploration.
  • Tomography Case Study: A 12-iteration tomography run began from an approximately 45×64×64 progressive representation and required no subsequent user interface changes.The agent received visualization and state information at each step while increasing resolution as needed.

2) Autonomous Exploration Behavior:

WebVisus autonomously refined an initially uninformative visualization by testing slices, isovalues, opacity, viewing directions, and representations. The resulting exploration exposed a cylindrical internal feature while adapting access to resource constraints and producing an auditable summary.

  • The agent rejected a nearly uniform initial rendering as inconclusive and evaluated alternative slices, isovalues, opacity settings, viewing directions, and representations.
  • By step 8, combining orthogonal slices with an isovolume revealed spatial variation that was difficult to distinguish initially.
  • Further adjustments to opacity, isovalue ranges, and slice locations exposed a cylindrical internal feature in the final view.
  • The study produced a traceable summary of visited states and parameter changes without user control of intermediate visualization parameters.
  • The observations remain exploratory and require domain validation rather than constituting validated scientific conclusions.
  • The controller used measured latency and estimated memory cost to guide subsequent access decisions, with larger transfers introducing dataset-dependent delays.

4) Result:

The access study measured cloud retrieval and local loading across six scientific datasets as progressive request sizes increased. Small representations supported rapid repeated observations, whereas larger transfers caused sharply rising, dataset-dependent delays.

  • At 1MB, tested datasets loaded in approximately 0.1-0.2 s, allowing repeated retrieval without dominating inference, action execution, and rendering time.
  • At 256MB, access times ranged from approximately 10 to 47s across the evaluated datasets.
  • Requests of 1 GB and 2 GB required tens to hundreds of seconds, with 2GB retrieval reaching approximately 387s for the 2kbit1 micro CT dataset.
  • Smaller representations supported repeated observation and action, while larger transfers could stall autonomous execution.
  • The measurements indicate that the largest feasible request may be unsuitable for iterative analysis all the time.

3) Adaptive Agent Policy:

WebVisus treats transfer size and resolution as part of the agent’s action policy, selecting representations under memory and latency budgets and recording decisions through a bounded protocol. Across five workflows, it completed autonomous exploration despite some rejected commands.

  • Adaptive Agent Policy: Before retrieval, the system considers logical dimensions, field type, timestep range, memory budget, and transfer preset.
  • Adaptive Agent Policy: The agent retains sufficient representations, requests higher resolution or new samples for ambiguous views, and reduces size or quality when latency or memory exceeds policy limits.
  • Results: All five workflows satisfied success criteria without intermediate user actions, averaging 7.8 of 12 iterations and approximately 108 seconds end to end.
  • Results: Across five workflows, 175 of 203 proposed actions executed successfully, corresponding to 86.20%, while 28 actions were rejected.
  • Results: Rejections commonly occurred when the agent modified a parameter before creating the associated visualization node, but every workflow recovered and completed.
  • Results: The generated summaries provide exploratory first-contact assessments rather than validated scientific conclusions.

3) Implications for Autonomous Science:

WebVisus extends autonomous visualization by making data access part of agent reasoning, jointly considering useful information and retrieval cost. The system supports inspectable, resource-bounded exploration, but remains limited to exploratory analysis without established scientific correctness or repeatability.

  • Implications for Autonomous Science: A fixed high-resolution input can introduce long delays and exceed memory limits, while a permanently coarse policy may fail to reveal relevant features.
  • Implications for Autonomous Science: The agent selects resolution, spatial extent, access mode, and timestep by considering both the visual result and the cost of obtaining it.
  • Implications for Autonomous Science: A progressive representation retrieved in approximately 0.16 s enabled inspection of multiple axes and isovalue adjustments without downloading the approximately 11 GB logical volume locally.
  • Implications for Autonomous Science: Registered interface actions let researchers inspect, interrupt, or override each step of autonomous exploration.
  • Implications for Autonomous Science: WebVisus estimates decoded memory footprint before retrieval and adapts requests to network, graphics-memory, and server-side read constraints.
  • Limitations: The current evaluation does not establish validated scientific measurements, repeatability, scientific correctness, or agreement with domain experts.
Loading 2608.22045v1…