Source-linked AI summary
MultiVis-Agent: A Multi-Agent Framework with Logic Rules for Reliable and Comprehensive Cross-Modal Data Visualization
Jinwei Lu, Yuanfeng Song, Chen Zhang, Raymond Chi-Wing Wong
TL;DR
Automated visualization systems struggle with multimodal inputs, iterative refinement, and reliability. MultiVis-Agent combines coordinated specialized agents with four mathematical logic-rule layers, formalizes four task scenarios, and introduces MultiVis-Bench; experiments report stronger visualization quality and reliability than baselines and no-logic-rule variants.
Problem
Existing visualization systems inadequately support multimodal inputs and iterative refinement, while LLM-based agents face failures, error propagation, parameter violations, and nontermination.
Method
MultiVis-Agent combines centralized multi-agent orchestration with four logic-rule layers that constrain parameters, classify tasks, recover errors, and guarantee loop termination.
Results
MultiVis-Agent achieves 75.63% visualization quality on challenging tasks and reports higher reliability than baselines and the same framework without logic rules.
Takeaways & Limitations
MultiVis-Agent and MultiVis-Bench provide a framework for evaluating and generating multimodal, iterative visualizations with improved system stability.
Takeaways & Limitations
The formulation indicates that different MultiVis scenarios may require distinct model capabilities or architectural components rather than one uniform solution.
Abstract
from arXiv · showhide
Real-world visualization tasks involve complex, multi-modal requirements that extend beyond simple text-to-chart generation, requiring reference images, code examples, and iterative refinement. Current systems exhibit fundamental limitations: single-modality input, one-shot generation, and rigid workflows. While LLM-based approaches show potential for these complex requirements, they introduce reliability challenges including catastrophic failures and infinite loop susceptibility. To address this gap, we propose MultiVis-Agent, a logic rule-enhanced multi-agent framework for reliable multi-modal and multi-scenario visualization generation. Our approach introduces a four-layer logic rule framework that provides mathematical guarantees for system reliability while maintaining flexibility. Unlike traditional rule-based systems, our logic rules are mathematical constraints that guide LLM reasoning rather than replacing it. We formalize the MultiVis task spanning four scenarios from basic generation to iterative refinement, and develop MultiVis-Bench, a benchmark with over 1,000 cases for multi-modal visualization evaluation. Extensive experiments demonstrate that our approach achieves 75.63% visualization score on challenging tasks, significantly outperforming baselines (57.54-62.79%), with task completion rates of 99.58% and code execution success rates of 94.56% (vs. 74.48% and 65.10% without logic rules), successfully addressing both complexity and reliability challenges in automated visualization generation.
1 Introduction
Real-world visualization requires multimodal inputs and iterative refinement, but existing systems remain limited and unreliable. MultiVis-Agent addresses these gaps with coordinated agents and four logic-rule layers for safer, more flexible execution.
- System limitations: Existing systems lack multimodal input and iterative refinement, limiting support for reference images, code examples, and evolving visualization requirements.These limitations distinguish practical visualization workflows from simple text-to-visualization generation.
- Reliability crisis: LLM-based visualization agents face catastrophic failures, uncontrolled error propagation, parameter violations, and nonterminating iterative loops.These reliability problems motivate explicit constraints and recovery mechanisms.
- Proposed framework: MultiVis-Agent uses a centralized Coordinator Agent to orchestrate specialized database, visualization, and validation agents according to task context.The architecture supports multimodal fusion and state-aware refinement.
- Logic-rule design: Its four logic layers constrain parameters, classify tasks deterministically, recover from errors, and guarantee loop termination while guiding rather than replacing LLM reasoning.The rules are presented as mathematical constraints that preserve flexibility for complex visualization tasks.
- Empirical motivation: 75.63% visualization quality is reported for challenging tasks, while MultiVis-Agent’s logic-rule framework improves task completion and code execution over versions without logic rules.The supplied introduction reports the visualization-quality result and contrasts logic-rule and no-logic-rule reliability metrics.
2 MultiVis: A Comprehensive Task for Visualization Generation
MultiVis extends traditional text-to-visualization into a unified task family covering multimodal generation and iterative refinement. Its explicit scenario structure supports systematic development and evaluation beyond one-size-fits-all workflows.
- Task rationale: MultiVis explicitly supports multimodal inputs, stages of generation and refinement, structured scenario categorization, and progression from basic to complex cases.These properties provide a systematic basis for developing and evaluating adaptive visualization systems.
- Task formulation: MultiVis defines a mapping from combinations of natural-language, database, image, and code inputs to executable visualization code.The formulation represents each scenario as a specialized function over its input modalities.
- Scenario coverage: The framework covers four scenarios: Basic Generation, Image-Referenced Generation, Code-Referenced Generation, and Iterative Refinement.These scenarios extend traditional text-and-database generation across the visualization lifecycle.
- Task rationale: Different scenarios likely require distinct model capabilities or architectural components rather than a single uniform solution.The formulation highlights image-style interpretation, code adaptation, and precise modification as scenario-specific challenges.
3 MultiVis-Bench: Benchmark of MultiVis
MultiVis-Bench evaluates multimodal, iterative visualization across four scenarios using executable code rather than intermediate representations. It combines human-led construction with LLM assistance and quality validation to support broad lifecycle coverage.
- Motivation and scope: MultiVis-Bench evaluates multimodal inputs, iterative refinement, and directly executable visualization code, addressing gaps in benchmarks limited to basic text-to-visualization or intermediate representations.Its scope spans the full visualization lifecycle from basic generation to refinement.
- Construction: The benchmark uses a human-led, LLM-assisted construction process with expert review averaging 2.5 rounds per example.This process targets quality, diversity, and representativeness across the four scenarios.
- Quality assurance: Quality assurance checks technical correctness, semantic faithfulness, and perceptual effectiveness through expert checklists and automated controls.Validation covers executable code, query-visualization alignment, reference adherence, clarity, and chart appropriateness.
- Benchmark composition: 1,202 cases cover Basic Generation, Image-Referenced Generation, Code-Referenced Generation, and Iterative Refinement across 127 chart types and 141 complex SQLite databases.The benchmark contains 306, 109, 233, and 554 examples for the four scenarios respectively.
- Benchmark distinctions: The benchmark combines text, database, image, and code inputs with executable Altair Python outputs and comprehensive scenario coverage.This distinguishes it from prior benchmarks centered on text-only inputs and non-executable formats such as VQL or Vega-Lite.
4 MultiVis-Agent: System Implementation
MultiVis-Agent implements visualization generation and refinement through a coordinated workflow of specialized agents. Its operational process combines data retrieval, code generation, evaluation, and refinement around a central coordinator.
- Operational workflow: The framework’s workflow cycles through data retrieval, visualization generation, evaluation, and refinement under Coordinator Agent orchestration.The workflow is illustrated as an operational example of the system’s collaborative process.
4.1 Design Philosophy and Architecture Overview
MultiVis-Agent combines flexible multi-agent specialization with centralized coordination to address complex visualization tasks. Its centralized design maintains consistency, integrates validation feedback, and supports iterative refinement and recovery.
- Design Philosophy: The approach combines logic rule-enhanced reliability with the flexibility of multi-agent behavior for complex multi-modal scenarios.This design responds to the rigidity of traditional pipelines and instability of pure LLM-based agents.
- Architecture Overview: MultiVis-Agent decomposes complex visualization tasks into specialized sub-problems while a central Coordinator manages the overall process.The architecture combines Database & Query, Visualization Implementation, and Validation & Evaluation agents with centralized decision making.
- Design Advantages: Centralized coordination maintains global consistency across visual elements and integrates holistic validation feedback during generation and refinement.The architecture also supports state management, history tracking, error isolation, and recovery.
4.2 Concrete Execution Example
The execution workflow uses three specialized agents to retrieve data, generate or modify visualization code, and evaluate results. Their coordinated interfaces support iterative visualization refinement.
- Execution Workflow: Three specialized agents cover data retrieval, visualization implementation, and validation and evaluation in the execution workflow.The Coordinator orchestrates these agents through cycles of retrieval, generation, evaluation, and refinement.
- Database & Query Agent: The Database & Query Agent explores available tables and relationships before iteratively formulating executable SQL queries.Its tools include list_tables, get_table, get_foreign_keys, find_fields, and execute_sql.
- Visualization Implementation Agent: The Visualization Implementation Agent generates initial Altair code and modifies it during iterative refinement using reference examples and code execution.Its interfaces are generate_visualization_code and modify_visualization_code.
- Validation & Evaluation Agent: The Validation & Evaluation Agent assesses code correctness and perceptual effectiveness, returning structured feedback for subsequent refinement.It evaluates rendered visualizations through Altair and Matplotlib execution interfaces.
4.3 Logic rule-enhanced Agent Architecture
The logic rule-enhanced architecture constrains agent behavior with formal rules for task routing, safe execution, recovery, and termination while preserving LLM reasoning. The framework presents theoretical guarantees for parameter safety, bounded recovery, and finite execution.
- Logic Rule-Enhanced Architecture: MultiVis-Agent uses mathematical constraints to guide LLM decisions without replacing the flexibility of LLM reasoning.The framework differs from static if-then systems by formally constraining behavior while retaining generative adaptability.
- Four-Layer Framework: Four hierarchical rule layers govern coordination, tool execution, error handling, and iteration control across the agent architecture.CR rules govern coordination, TE and EH rules constrain tool execution and error handling, and RC rules manage iteration and termination.
- Coordination Rules: Task classification deterministically prioritizes existing code, Python references, image references, and basic text as D≻C≻B≻A.When multiple input types coexist, the highest-priority category determines routing.
- Coordination Rules: Deterministic evaluation mapping returns completion after successful validation or prioritized modification feedback when critical issues remain.The decision function maps evaluation results to system actions and supports consistent behavior.
- Tool Execution Rules: Parameter constraints, code standardization, and reference processing enforce safer and more consistent tool execution.The TE rules validate parameter ranges, standardize code environments, and enforce reference utilization.
- Theoretical Guarantees: Theoretical results establish parameter safety, bounded error recovery, and termination within T_max=10 iterations.The reliability argument composes these properties into a formal reliability guarantee, and the paper frames the theorems as supporting practical deployment.
- Reliability Guarantees: The framework is designed to degrade gracefully and recover systematically rather than fail catastrophically when errors occur.This behavior is presented as a consequence of constrained decisions and structured recovery mechanisms.
4.4 Multi-Agent Coordination Framework
The coordination framework manages multimodal visualization tasks through deterministic routing, bounded iterative execution, and specialized-agent reasoning. It combines state updates, tool calls, error handling, and validation feedback in a closed-loop process.
- Dynamic Coordination: The Coordinator dynamically routes tasks among specialized agents while applying formal rules for classification, prerequisite validation, and recovery.This constrained coordination is intended to make decisions consistent rather than relying solely on LLM reasoning.
- Coordination Loop: The main coordination loop initializes state, classifies the task, repeatedly invokes an agent and tool, updates state, and returns the final visualization code.Execution continues while t<T_max and the task is incomplete.
- Agent Reasoning: Specialized agents use Thought-Action-Observation cycles in which LLM-generated reasoning is followed by rule validation before actions execute.TE and EH rules constrain parameters, standardize environments, and handle errors during agent execution.
- Multimodal Integration: The framework integrates natural-language, database, image, and reference-code inputs through modality-specific processing mechanisms.Reference images are processed with vision-language capabilities, while reference code is analyzed for reusable patterns.
- Iterative Refinement: Validation analyzes both generated code and rendered output, and its structured feedback drives further refinement through the logic-rule framework.This creates a closed loop connecting generation, perceptual assessment, and correction.
5 Evaluation Metrics
The evaluation framework combines structural code assessment with perceptual visualization assessment into a unified score. This dual-layer design measures both technical correctness and human-perceived effectiveness.
- High-level perceptual assessment: Each perceptual score is a weighted sum of six dimensions covering chart appropriateness, layout, textual elements, data representation, styling, and global clarity.The weights sum to 100 points, with chart appropriateness, textual elements, data representation, and visual styling each receiving 20 points.
- Assessment procedure: The perceptual layer uses rendered charts, structured VLM prompts, detailed rubrics, and normalized scores to support reproducible visual-quality assessment.The method is designed to capture effectiveness that conventional code-execution metrics may miss.
- Dual-layer evaluation: The framework evaluates visualizations through low-level structural metrics and high-level perceptual metrics, then combines both into a holistic assessment.Structural analysis compares generated code across six dimensions, while perceptual analysis scores rendered visualizations across six weighted dimensions.
- Low-level structural assessment: Each structural score is a weighted sum of six normalized dimensions covering chart type, data mapping, encoding, interaction, configuration, and transformation correctness.The dimensions assess both visualization specifications and preprocessing operations.
- Score integration: A unified visualization score combines structural and perceptual scores using an integration factor that was set to α=0.5 for equal weighting.The framework permits different emphasis on technical correctness versus perceptual quality through α.
6 Experiments
Experiments evaluate MultiVis-Agent across four visualization scenarios using MultiVis-Bench and compare it with direct prompting and sequential workflow baselines. The framework achieves its strongest gains in generation and multimodal tasks, while logic rules substantially improve reliability.
- Overall results: 74.18% average Visualization Score in Scenarios A–C surpassed Instructing LLM at 63.37% and LLM Workflow at 64.41%.The complete MultiVis-Agent achieved the strongest overall generation performance in Table 2; nvAgent scored 30.30–33.33%.
- Basic generation: 65.95% structural and 80.02% perceptual scores in Basic Generation exceeded LLM Workflow’s 45.93% and 68.52%.The coordinated validation loop particularly improved technical correctness in encoding and configuration dimensions.
- Multi-modal generation: 75.63% in Image-Referenced Generation and 76.58% in Code-Referenced Generation exceeded LLM Workflow by 12.84% and 13.64%, respectively.These results show the largest advantages in multimodal scenarios requiring reference interpretation.
- Iterative refinement: In Iterative Refinement, LLM Workflow reached 74.69% versus MultiVis-Agent’s 71.52%, although MultiVis-Agent retained advantages in high-level perceptual aspects.The only GEN variant also reached 76.44% versus 71.52%, suggesting focused refinement can benefit from simpler processing when executable code already exists.
- Ablation analysis: Removing logic rules reduced performance by 21.75% in Basic Generation, 31.70% in Image-Referenced Generation, and 17.58% in Code-Referenced Generation.The ablation results indicate that formal behavioral constraints are especially important for multimodal inputs.
- Reliability analysis: Logic rules raised task completion to 98.68–99.58% and code execution success to 94.56–97.10%, improving reliability by 20.46–25.10% and 29.46–32.53%.The gains are attributed to systematic task classification, parameter-boundary enforcement, and error handling.
7 Related Work
Visualization generation has progressed from rigid rule-based and learning-based systems to LLM-driven and agent-based approaches, but existing methods remain limited in multimodal handling and reliability. MultiVis-Agent combines a four-scenario task formulation, a dedicated benchmark, and centralized logic-rule-enhanced coordination.
- Evolution of visualization systems: Earlier rule-based systems offered interpretability but struggled with query ambiguity and flexibility, while learning-based systems treated visualization as sequence translation.Modern LLM-driven systems extend flexibility but introduce reliability concerns for complex workflows.
- MultiVis formulation: MultiVis extends traditional Text-to-Vis to four scenarios covering multimodal inputs and iterative refinement, enabling systematic development and evaluation.The formulation unifies diverse generation and refinement stages within one task framework.
- Benchmark: MultiVis-Bench contains over 1,000 cases designed to evaluate images, code, iterative refinement, and directly executable visualization code.The benchmark was constructed through a human-led process with LLM assistance.
- Agent-based systems: Existing agent-based visualization approaches use specialized agents but often lack robust error handling and effective fusion of multimodal inputs.These limitations motivate tighter coordination and formal reliability mechanisms.
- Proposed framework: MultiVis-Agent uses centralized coordination and mathematical constraints to guide LLM reasoning while preserving flexibility, with formal support for error recovery and termination.Its four-layer logic rules address parameter boundaries, task classification, error recovery, and loop termination.
8 Conclusion
MultiVis-Agent combines logic-rule-enhanced multi-agent architecture, formal robustness foundations, task and benchmark formulation, and empirical validation to improve reliability in automated visualization. The approach is presented as a basis for production deployment through systematic error recovery and stronger execution and completion performance.
- MultiVis-Agent establishes a logic rule-enhanced multi-agent design with formal robustness foundations, MultiVis task and benchmark formulation, and experimental validation.The framework is presented as a new paradigm for robust agent-system design.
- 98.7% task completion and 94.2% code execution success outperform baseline ranges of 85.6-89.2% and 71.3-78.5%, respectively.The reported comparison also includes 96.8% error recovery performance versus frequent system failures in baseline approaches.
- Systematic error recovery enables reliable production deployment, improving on heuristic methods that can fail catastrophically when errors occur.The conclusion frames error recovery as the mechanism supporting this deployment-oriented reliability advantage.