Source-linked AI summary
MIVAIS: A Study Environment for Multi-Agent Mixed-Initiative Visual Analytics Applications
Tobias Stähle, Simon Schneider, Rita Sevastjanova, Mennatallah El-Assady
TL;DR
MIVAIS addresses engineering complexities in mixed-initiative visual analytics by providing a research framework that streamlines development and evaluation. Its infrastructure supports modular replication and rapid adaptation, while its environment supports configurable user studies.
Problem
Increasing agent complexity creates a need for toolkits that abstract engineering complexities in mixed-initiative visual analytics.
Method
MIVAIS combines a research framework for agent behavior, feedback, and multimodal provenance with infrastructure supporting communication among agents.
Results
MIVAIS systems can be implemented on the same modular Infrastructure, and agents could be quickly adapted in an existing system.
Takeaways & Limitations
MIVAIS allows developers and researchers to build modular web-based mixed-initiative visual analytics systems and configure multi-agent user studies.
Takeaways & Limitations
MIVAIS's current design operates under inherent conceptual and technical limits.
Abstract
from arXiv · showhide
Mixed-initiative Visual Analytics (VA) systems empower human users by interleaving human intuition with software agents and their machine intelligence. However, the development and rigorous evaluation of such systems remain constrained by engineering overhead. Developers must, e.g., implement complex, low-level state synchronization to manage asynchronous agent behaviors, while researchers struggle to capture the multimodal provenance required to study and evaluate human-AI collaboration. We present MIVAIS, a dual-layered research platform designed to abstract the structural complexities of mixed-initiative VA. First, it contributes a computational Infrastructure that standardizes human-software agent interaction, state synchronization, and communication between the agents. Second, it provides a declarative Study Environment that automatically logs multimodal human-AI telemetry - including application/system state, screen capture, audio, and additional sensor data - enabling seamless, in-situ user studies and post-session analysis. We technically validate our infrastructure by replicating three state-of-the-art systems (Podium, Voyager 2, and ProactiveVA). Furthermore, we evaluate the framework's expressiveness and efficiency through expert case studies with HCI and VA researchers, demonstrating how MIVAIS effectively lowers the barrier to prototyping and evaluating intelligent, co-adaptive interfaces.
MENNATALLAH EL-ASSADY, ETH Zürich, Switzerland
MIVAIS combines a modular infrastructure for mixed-initiative VA systems with a study environment for configuring experiments, collecting multimodal data, and conducting post-analysis.
- The MIVAIS Infrastructure supports modular mixed-initiative VA systems in which software and human agents collaborate.
1 Introduction
Mixed-initiative VA research is hindered by engineering complexity and insufficient support for capturing synchronized, multimodal human-AI collaboration data. MIVAIS addresses these gaps with a modular infrastructure and integrated study environment, validated through system replications and expert case studies.
- Motivation and gap: Mixed-initiative VA systems require synchronization across asynchronous human and software-agent updates, creating substantial implementation complexity.
- Motivation and gap: Existing structural toolkits are limited, leaving HCI researchers to build bespoke middleware instead of studying phenomena such as trust and cognitive load.
- Motivation and gap: Evaluating multi-agent collaboration requires synchronized provenance covering application state, agent behavior, visual stimuli, reasoning audio, and inter-agent communication.
- MIVAIS contributions: MIVAIS provides a modular infrastructure with APIs for environment observation, interaction, state synchronization, agent communication, and action provenance.
- MIVAIS contributions: Its integrated Study Environment declaratively configures studies and automatically synchronizes agent logs with screen and audio captures for post-hoc analysis.
- Evaluation: MIVAIS was demonstrated by replicating Podium, Voyager 2, and ProactiveVA and evaluated through expert case studies with 3 HCI and VA researchers.
2 Background & Related Work
Prior work established mixed-initiative VA applications, agent-centered models, and declarative study environments, but these approaches generally do not support fully interactive multi-agent evaluation.
- Mixed-initiative VA: Mixed-initiative VA research introduced software agents to support humans in complex analytical tasks while balancing automation with human steering.
- Agent-centered models: Agent-based models represent mixed-initiative VA through agents, infrastructure, and the VA environment, with dimensions including observations, communication, actions, and logic.
- Study environments: HCI study tools support reusable experiment modules, interaction capture, and extensions such as eye tracking.
- Study environments: ReVISit provides declarative web-based visualization studies, session replay, interaction tracking, and study sharing for replication.
- Research gap: Existing study frameworks target single-participant sessions with isolated visual stimuli and therefore do not meet the requirements of fully interactive mixed-initiative VA systems.
- MIVAIS relation to prior work: MIVAIS adopts modular study configuration and session replay while extending these approaches toward mixed-initiative VA evaluation.
3 Design Goals and Requirements
MIVAIS’s design goals combine complete provenance and flexible study configuration with modular multi-agent infrastructure, synchronized state, collaborative setups, and advanced data collection.
- Provenance and study analysis: The design goals require logging state updates, agent communications, and environment interactions so sessions can be reconstructed, replayed, and analyzed post hoc.
- Flexible study design: Study configurations should minimize setup burden while supporting reusable tasks, questionnaires, randomization, attention checks, and invalid-operation handling.
- Replication and analysis: The Study Environment should support sharing and extending studies, session replay, linked data exploration, and access to collected data for further analysis.
- Modular agent infrastructure: The Infrastructure should decouple agents from the VA environment and standardize how human and software agents observe states, act, communicate, and log behavior.
- Roles and permissions: Declarative roles and permissions should govern agent observations, actions, and communications while allowing auditable runtime updates.
- State synchronization: Runtime synchronization must propagate environment changes across data, visual interfaces, models, and agents, while enabling agents to follow interaction trajectories.
- Collaborative studies and sensing: The Study Environment should accommodate collaborative scenarios with varying numbers of collaborators and collect biometric data for later stress and cognitive-load analysis.
4 The MIVAIS Architecture
MIVAIS models mixed-initiative VA systems as modular environments in which human and software agents interact through a shared World State, governed permissions, and communication channels. Its infrastructure also supports provenance logging and declarative study telemetry for evaluating these systems.
- Architecture: MIVAIS decomposes mixed-initiative VA systems into modular infrastructure and agents, supporting standardized design, configuration, comparison, and evaluation.
- World State: The World State centrally represents the VA environment, allowing agents to observe and update shared state while accommodating asynchronous operation and dynamic participation.
- Agents: Agents are modeled uniformly as entities that perceive the environment, derive decisions, and act by updating permitted parts of the World State.
- Communication: Communication Channels support distributed, centralized, or hierarchical agent networks, with text-based messages and infrastructure-level logging of exchanged messages.
- Permissions: The Agent Registry and Permission Guard define and validate which agents can access or update World State components and communication channels.
- Study Support: MIVAIS abstracts state synchronization, communication, and permission management while capturing multimodal data aligned on a shared timeline for study analysis.
5 Technical Evaluation
MIVAIS was technically evaluated by reimplementing three mixed-initiative VA systems and examining how the infrastructure supports distinct agent configurations and coordination patterns. Replication shaped a guarded selective-update mechanism and demonstrated applicability across heterogeneous systems.
- Replicated Systems: The evaluation reimplemented Podium, Voyager 2, and ProactiveVA on the MIVAIS Infrastructure across ranking, data exploration, and proactive assistance scenarios.
- Podium: Podium uses human and software agents with distinct permissions to update rankings, priorities, and attribute weights in a shared World State.
- Voyager 2: The Voyager 2 variant separates recommendation and insight generation between two software agents with different permissions and communication responsibilities.
- ProactiveVA: ProactiveVA was implemented with five agents to demonstrate agent-to-agent communication and decentralized coordination, without claiming that this decomposition is superior.
- Infrastructure Validation: The replications showed that heterogeneous systems can share one modular infrastructure and led to a guarded, selective state-update scheme instead of unrestricted broadcasting.
- Efficiency: MIVAIS enables agents to be implemented and tested independently of the VA interface and study setup, while study subtasks can be tested without full-study walkthroughs.
6 User Evaluation: Expert Case Studies
Expert case studies examined MIVAIS through three researchers who implemented or retrofitted mixed-initiative VA systems and pilot studies. Experts found the agent-centric abstraction learnable and expressive, while study configuration and iteration became faster, though the small self-assessment pilots do not quantify analysis savings.
- Methodology: Three HCI or VA researchers implemented or retrofitted systems, configured pilot studies, and provided formative feedback after two weeks of use.
- Case Studies: The case studies covered UserRAG, TopoAlign, and PleaSQLarify across deployed study modes and session replay analysis.
- Experience: All experts found the agent-centric abstraction intuitive and quick to learn, and reported no expressiveness ceiling for their intended collaboration behaviors.
- Experience: Experts used familiar tools and stacks without toolchain changes, although learning the declarative DSL grammar and separating study and task files created friction.
- Development Workflow: The agent decomposition served as a design lens, helping experts reason about roles and abilities and add another agent during development.
- Study Iteration: Complete study setups were prepared within hours, and copied tasks supported ablations and additional metrics without editing the VA application.
- Limitations: Because the findings are self-assessments from small pilots, they report perceived effort reductions but do not quantify analysis savings.
7 Discussion
MIVAIS supports modular development and low-effort deployment of mixed-initiative VA systems and studies, while collecting diverse study data. Its current limitations include desktop-first scope, incomplete post-analysis integration, browser compatibility constraints, and adoption overhead.
- Framework capabilities: MIVAIS enables developers to build web-based mixed-initiative VA systems through a modular Infrastructure and configure multi-agent user studies.The framework supports deploying studies according to researchers’ requirements.
- Evaluation: Expert evaluation found the Infrastructure flexible and modular enough to implement new systems or retrofit existing ones, while researchers could deploy studies with low effort.Researchers also valued the diversity of collectable data and the option to add external streams such as biometric sensors.
- Limitations: Adopting MIVAIS requires aligning application architecture with its theoretical concepts and maintaining separate mental models for application state and study setup.This can add development overhead and cognitive demand, despite enabling quicker study setup and execution.
- Limitations: Post-session analysis currently visualizes data streams separately, does not join streams, and does not overlay eye-tracking information on captured screen replay.Such enrichment is out of scope for the current work, although the core implementation can be adapted.
- Limitations: MIVAIS is currently limited to desktop web-based applications, lacks VR/AR and mobile support, and has browser-dependent features such as Web Bluetooth heart-rate collection.A generic Sensor API provides a workaround for additional sensor data, while broader browser support remains future work.
- Future study uses: MIVAIS supports flexible study design by allowing researchers to manipulate VA environment states and agents without heavy modification of the underlying system.Agent permissions and roles can be adapted to compare wizard agents with real agents.
- Future study uses: Its collected data can support human-behavior capture and behavioral cloning into software agents, with learned behavior tested in the same or another environment.The framework also supports sharing locally hosted or deployed systems and studies through its modular structure.
8 Conclusion
MIVAIS combines a modular Infrastructure with a Study Environment for configuring multi-agent experiments and collecting data for post-session analysis. The authors report that it supports diverse mixed-initiative VA applications and enables rapid, comprehensive study deployment.
- MIVAIS provides a modular Infrastructure and Study Environment for building mixed-initiative VA systems and configuring, deploying, and running multi-agent experiments.The environment supports collection of data for post-session analysis of collaboration, interaction, and user behavior.
- The Infrastructure architecture supports building diverse mixed-initiative VA applications.
- Once a system is implemented, researchers can configure, deploy, and run studies in a short time.
- MIVAIS supports a comprehensive set of data collection methods for mixing qualitative and quantitative studies.
- The framework is available as open source software for community use.
GenAI Usage Disclosure
GenAI was partially used as a coding assistant during MIVAIS implementation, while the authors conducted the architecture design, expert case study protocols, analysis, and manuscript writing.
- GenAI was partially used as a coding assistant during MIVAIS implementation.
- The authors conducted all system architecture design, expert case study protocols, analysis, and manuscript writing.
A Usage Scenario
The usage scenario follows MIVAIS from conceptualizing agents and the VA interface through backend orchestration, frontend integration, study deployment, analysis, and replication. Researchers define system structure and permissions, run controlled studies, review multimodal trajectories, and share applications, setups, and session replays.
- System Conceptualization: The workflow begins by defining the data model, machine-learning components, VA interface, agents, capabilities, and action spaces.
- Backend Orchestration: The developer implements the conceptual draft by defining the World State, configuring agents and permissions, and setting event listeners and subscriptions.
- Backend Orchestration: Software-agent logic is implemented through the MIVAIS base-agent class, which operationalizes observations, actions, and inter-agent communication.
- Frontend Integration: The visual interface connects to the backend through the MIVAIS Context Provider and standardized WebSocket API, with user identities resolved at runtime.
- Study Deployment and Post-Session Analysis: Researchers configure study sequences and task parameters, run controlled lab studies, and analyze multimodal interaction trajectories and inter-agent communication.
- Sharing Application and Study Setup for Replication: Researchers can open-source applications, share study setups for replication, and provide public session replays for reviewers and stakeholders.
B Semi-Structured Interview Protocol
The semi-structured interview protocol gathers expert perspectives on building, adapting, evaluating, and integrating MIVAIS-based mixed-initiative VA systems and studies. Questions address expressiveness, synchronization, development effort, usability, logging, and suitability.
- Study setup: The protocol first establishes participants’ backgrounds and asks them to describe the MIVAIS system, purpose, collaboration scenario, and implementation process.
- Implementation experience: Participants assess which implementation aspects were straightforward or difficult, including concrete moments of confusion or frustration.
- Expressiveness: Questions examine how MIVAIS handles state synchronization, interaction patterns, collaboration behaviors, design constraints, creativity, and evolving study requirements.
- Efficiency and usability: The protocol compares MIVAIS with conventional development regarding time, effort, API learning, tool integration, project reuse, and features that accelerate or hinder work.
- Logging and reflection: Participants evaluate logged data for analyzing application state and user-agent interactions, compare logging effort and data quality, and discuss reuse, alternatives, limitations, and desired changes.
C.3 Task Configuration
Task configuration supports reusable or inline task specifications, multiple task types, ordering controls, and state initialization or updates written through the Study Gateway at task start.
- Task specification: Task configuration supports reusable fragments, inline task objects, filename-default identifiers, and task types including information, choice, rating, input, and VA interaction tasks.
- Task state: The configuration distinguishes full replacement scenarios from state deltas, with both written through the Study Gateway when a task starts.
- Task semantics: A task may specify a default variant, while ground_truth applies only to answerable task types.