PaperlayerView exact revision on arXiv

Source-linked AI summary

A Multi-Agent Platform for Automated Enterprise Analytics and Insight Generation

Manoj N M, Vijayakrishna S, Manjunath Srinivas, Rohit Pahan

arXiv:2608.18740v1cs.AI

TL;DR

Enterprise analytics often requires labor-intensive SQL or specialized business-intelligence tools, limiting conversational access to actionable insights. This paper introduces a sequential multi-agent platform that decomposes queries into specialized planning, retrieval, analysis, and aggregation steps, achieving 95.3% functional accuracy across 300 test cases spanning synthetic and production enterprise datasets.

  • Problem

    Enterprise data analysis remains time-consuming and labor-intensive because data is distributed across heterogeneous systems and traditional methods require manual SQL or complex business-intelligence tools.

  • Method

    The platform uses a sequential multi-agent architecture that decomposes complex queries into discrete planning, retrieval, analysis, and insight-aggregation steps handled by specialized agents.

  • Results

    95.3% functional accuracy was achieved across 300 test cases spanning synthetic and production enterprise datasets, alongside 4.52/5.0 response quality and a 93.0% hallucination-free rate.

  • Takeaways & Limitations

    Cross-model evaluation across four LLM backends supports the generalizability of the pipeline architecture, while human evaluation supports the reliability of the LLM-as-a-Judge methodology.

  • Takeaways & Limitations

    Enterprise evaluation covers a single production domain, so generalizability to other verticals requires further study.

Abstract

from arXiv · show

This paper proposes a multi-agent framework built on CrewAI [1] for conversational business intelligence. Five specialized AI agents operate in a sequential pipeline to process natural language queries, retrieve and analyze data, generate visualizations via the Model Context Protocol (MCP) [2], and deliver actionable insights. The platform features a defense-in-depth security architecture for multi-tenant data isolation and a query parameterization mechanism for transforming conversational insights into reusable dashboard components. Evaluation across 300 end-to-end test cases spanning synthetic and production enterprise datasets demonstrates 95.3% functional accuracy, a mean response latency of 24 seconds, and a response quality score of 4.52/5.0 as assessed by an LLM-as-a-Judge framework, with a 93.0% hallucination-free rate, representing a 22.6 percentage point accuracy improvement and 20.2% quality gain over a single-agent baseline. Cross-model evaluation across four LLM backends and human expert validation confirm architectural generalizability and evaluator reliability. An ablation study confirms that the Data Analysis and Report Aggregation agents are the primary drivers of output quality.

1 Introduction

The paper addresses the time-consuming, labor-intensive nature of enterprise insight generation by proposing a conversational multi-agent platform that decomposes complex queries into specialized analytical steps.

  • Problem: Enterprise data is distributed across heterogeneous storage systems, while traditional analysis requires manual SQL or complex business intelligence tools.These methods make generating actionable insights time-consuming and labor-intensive.
  • Contribution: The proposed platform assigns planning, data retrieval, analysis, and insight aggregation to specialized agents with domain-specific tools.This decomposes complex user queries into discrete steps for automated data interaction.
  • Contribution: The paper introduces an evaluation framework combining automated functional testing with LLM-based quality assessment.The framework is intended to validate the reliability of multi-agent analytical solutions.

2 Literature Review

The literature reflects a shift from single-agent analytical tools toward multi-agent systems for complex reasoning and autonomous execution. Prior work addresses agent architectures, enterprise Text-to-SQL reliability, dashboard generation, visualization, and security, while this system integrates these capabilities into full chart generation and analytics.

  • Literature scope: Recent LLM advances motivate the broader transition from single-agent analytical tools to dynamic multi-agent systems capable of complex reasoning and autonomous task execution.The related work is organized into two thematic areas.
  • Multi-agent architectures: Multi-agent research defines Star, Ring, Graph, and Bus architectures and emphasizes standardized collaboration protocols, role-based workflows, and hierarchical planning.MetaGPT encodes operating procedures into prompts, while AgentOrchestra uses the Tool-Environment-Agent protocol for hierarchical planning.
  • Enterprise analytics: Enterprise analytics studies combine retrieval-augmented generation, security layers, dialogue refinement, chain-of-thought reasoning, and SQL validation to improve Text-to-SQL reliability.SiriusBI reports 93–96% SQL accuracy through multi-round dialogue refinement across Tencent’s business sectors.
  • Dashboard generation: Data-to-Dashboard and Insight Agents demonstrate modular or hierarchical systems that generate grounded insights and visualizations while balancing accuracy with low latency.Insight Agents combine powerful LLMs with lightweight routing models for e-commerce deployment.
  • Visualization: Visualization research covers chart recommendation and iterative scientific plotting through in-context learning, multimodal feedback, and multi-agent collaboration.The proposed system extends beyond recommendation to full chart generation while also managing data access, query parameterization, and multi-tenant security.

3 System Architecture

The platform uses a five-tier microservices architecture with a sequential five-agent CrewAI pipeline for conversational analytics. Query parameterization and MCP-based visualization extend analysis into reusable dashboard components and dynamically generated charts.

  • Layered Architecture: The layered architecture comprises UI, API, Agent, Data, and MCP tiers supporting interaction, routing, orchestration, storage, and visualization.The Agent Layer is powered by CrewAI, while the MCP Layer integrates Antvis/mcp-server-chart.
  • Sequential Agent Pipeline: Five sequential agents retrieve authorized data, analyze patterns and anomalies, aggregate multilingual reports, suggest follow-up questions, and configure interactive charts.The Data Retrieval Agent generates schema-aware read-only SQL; the Data Analysis Agent generates charts through MCP.
  • Query Parameterization: Query Parameterization converts ad-hoc queries into reusable dashboard components through parameterized filters, schema introspection, cost estimation, materialization recommendations, and drill-down dimensions.It extracts literal WHERE-clause values and derives drill-down dimensions from SELECT and GROUP BY clauses.
  • MCP Visualization: MCP operates through an independent server-adapter microservice that exposes chart tools, supports runtime tool and schema discovery, and returns cloud storage URLs.Available chart types include bar, line, pie, scatter, funnel, treemap, and sankey.

4 Methodology

The methodology adopts a sequential CrewAI multi-agent pipeline with specialized stages for predictable execution and independent testability. It combines cumulative context flow, defense-in-depth security, and iteratively refined agent prompts.

  • Architecture Selection: A sequential multi-agent pipeline was selected over monolithic and hierarchical alternatives to provide separation of concerns, predictable execution, independent testing, and graceful degradation.The alternatives were associated with cognitive overload, poor error isolation, unpredictability, and communication overhead.
  • Pipeline Execution: The CrewAI sequential process performs validation, retrieval, analysis with MCP chart generation, report aggregation, follow-up generation, chart configuration, and post-processing.Agents execute in predefined order, while post-processing restores cached query results and assembles the final response.
  • Pipeline Execution: Each agent receives cumulative outputs from preceding agents through context injection, while dashed and solid boxes distinguish non-agent phases from autonomous LLM stages.This execution-flow distinction is specified in Figure 2.
  • Security Architecture: Defense-in-depth security uses input validation, query guardrails, multi-tenant isolation, and output validation to constrain unsafe queries and protect sensitive data.Controls include SQL-injection sanitization, blocked destructive operations, table-level access control, tenant-specific credentials, sensitive-data scanning, and sanitized error messages.
  • Prompt Development: Agent prompts were iteratively refined through role and goal definition, guardrail and anti-hallucination integration, edge-case handling, and language and formatting specifications.The refinement process targeted both behavioral safeguards and response presentation.

5 Evaluation

The platform is evaluated across functional accuracy, response latency, and output quality using 300 test cases and complementary automated, LLM-based, and human assessment methods.

  • Evaluation dimensions: 300 test cases span functional accuracy, response latency, and output quality.The evaluation covers three dimensions of platform performance.
  • Evaluation methods: Automated functional testing assesses the platform’s functional accuracy.
  • Evaluation methods: An LLM-as-a-Judge quality framework and human expert validation assess output quality.

5.1 Experimental Setup

The evaluation uses synthetic and production BigQuery datasets, 300 categorized test cases, and structured LLM-based assessment with human scoring. It compares the multi-agent system against a single-agent baseline and tests generalizability across four LLM backends.

  • Datasets: The evaluation uses synthetic e-commerce and production competitive-pricing datasets on Google BigQuery.The synthetic dataset contains 12,500 records across four relational tables, while the production dataset spans two e-commerce platforms and approximately 569,000 overlapping products.
  • Test Cases: 300 test cases span eight categories, including aggregations, time-series analysis, joins, complex analytics, multi-turn conversations, and security guardrails.Of the 260 analytical cases, 100 are independently scored by human domain experts; each analytical test validates SQL generation, execution correctness, result retrieval, and structured insight presence.
  • Evaluation Protocol: 260 analytical responses are scored by Claude Sonnet 4.5 across five 1–5 Likert dimensions, with additional hallucination detection against raw query results.Structured rubrics with explicit per-dimension criteria are used to mitigate evaluator bias and require grounded assessments.
  • Baselines and Generalizability: The comparison uses a same-model GPT-4.1 single-agent baseline with BigQuery tooling, while omitting regex-based input validation and selectively removing agents for ablation.The full 300-test evaluation is replicated across GPT-4.1, Claude Sonnet 4, Gemini 2.5 Flash, and self-hosted Llama 3.1 70B.

5.2 Results

The multi-agent system delivers high functional accuracy, quality, security, and hallucination-free performance, outperforming a single-agent baseline despite higher latency. Ablation, cross-model, human-evaluation, and parallelism results further support the architecture’s effectiveness and robustness.

  • Core performance: 95.3% accuracy (286/300 tests) was achieved, including interception of all 40 adversarial inputs; failures concentrated in complex analytics and multi-turn conversations.Complex failures involved semantically incorrect SQL over enterprise schemas and cross-turn comparisons exceeding the reasoning window.
  • Core performance: 23.8 seconds overall mean latency remained within the 120-second target, with complex analytics averaging 38.6s and multi-turn queries averaging 16.2s.Complex analytics latency reflected window functions and CTEs, while multi-turn queries leveraged prior context.
  • Quality and safety: 4.52/5.0 overall quality and a 93.0% hallucination-free rate were recorded, with hallucinations concentrated in multilingual enterprise data and complex cross-platform joins.Relevance scored 4.85 and language quality 4.94.
  • Baseline comparison: 22.6 percentage point accuracy improvement over the single-agent baseline was achieved, while quality rose 20.2% and hallucination-free performance improved from 72.8% to 93.0%.The multi-agent system achieved 95.3% vs. 72.7% accuracy and 4.52 vs. 3.76 quality; latency was higher at 23.8s vs. 15.2s.
  • Ablation study: Removing the Data Analysis Agent reduced quality by –1.14 points, while removing Report Aggregation reduced it by –0.66 points; accuracy remained 95.3% across configurations.Follow-up Questions and Chart Configuration removals had negligible impact (–0.02).
  • Robustness and validation: GPT-4.1 achieved the highest accuracy (95.3%) and quality (4.52), while Gemini 2.5 Flash reached 17.4s latency and Llama 3.1 70B achieved 79.7% accuracy.Human scores correlated with LLM-judge scores (Pearson r = 0.89, p < 0.001), and selective parallelism reduced latency from 23.8s to 20.1s without changing quality or accuracy.

5.3 Discussion

The discussion finds that the multi-agent system combines strong SQL accuracy and hallucination control with measurable gains over a single-agent baseline. These gains incur higher latency and token cost, although selective parallelism reduces the latency gap while preserving output quality.

  • Performance and Grounding: 93.5% SQL accuracy (243/260) and a 93.0% hallucination-free rate demonstrate effective analytical-query performance and grounding across enterprise-scale, multilingual schemas.Each agent operates on verified predecessor outputs, supporting the reported hallucination-free rate.
  • Baseline Comparison: 22.6 percentage point accuracy improvement and 20.2% quality gain over the baseline incur an 8.6-second latency cost per query.Selective parallelism reduces the gap to 4.9 seconds (20.1s vs. 15.2s) while maintaining full output quality.
  • Efficiency and Cost: 30,800 tokens per query versus 10,200 for the single-agent baseline produce approximately $0.08 versus $0.03 per query, a 2.7× cost factor.The higher cost is associated with 22.6 percentage points higher accuracy and 20.2 percentage points higher hallucination-free rate.

5.4 Limitations

The evaluation has limited generalizability and validation breadth: it covers one enterprise domain, samples only part of analytical responses for human evaluation, and evaluates four—but not all major—LLM families.

  • Evaluation Scope: The enterprise evaluation covers a single domain, so generalizability to healthcare, logistics, and other verticals requires further study.The validated production dataset is from competitive pricing.
  • Evaluation Scope: Human evaluation covers 100 of 260 analytical responses, limiting annotation breadth.Broader human annotation would further strengthen validity.
  • Evaluation Scope: Cross-model evaluation spans four LLMs but does not cover all major model families.The paper identifies broader model-family coverage as an outstanding limitation.

6 Conclusion

The multi-agent platform delivers strong conversational business intelligence performance across diverse enterprise datasets, with cross-model and human validation supporting its generalizability and evaluation methodology. Its security and latency results indicate deployment readiness, while future work targets broader evaluation, adaptive behavior, caching, and cost-efficient model routing.

  • Core results: 95.3% functional accuracy, 4.52/5.0 response quality, and 93.0% hallucination-free rate were achieved across 300 synthetic and production enterprise test cases.These results summarize the platform’s end-to-end performance for conversational business intelligence.
  • Validation and generalizability: 79.7% for open-source Llama 3.1 70B surpassed the single-agent GPT-4.1 baseline of 72.7%, supporting architectural generalizability across four LLM backends.Human evaluation of 100 analytical responses also validated the LLM-as-a-Judge methodology with r = 0.89 and κ = 0.82.
  • Deployment readiness: 15.5% latency reduction was achieved through selective parallelism without quality loss, while the defense-in-depth security architecture blocked all adversarial inputs.Together, these findings support readiness for multi-tenant enterprise deployment.
  • Future work: Future work will broaden longitudinal human-in-the-loop evaluation, develop feedback-driven adaptive agents, investigate semantic caching, and assess heterogeneous ensembles with cost-efficient routing.The proposed routing reserves high-capability models for critical analytical tasks while assigning auxiliary agents more cost-efficient models.
Loading 2608.18740v1…