PaperlayerView exact revision on arXiv

Source-linked AI summary

BI-Agent and BI-Bench: Towards Automating End-to-End Business Intelligence

Chuxuan Hu, Yeye He, Penny Zhou, Wee Hyong Tok, Daniel Kang, Surajit Chaudhuri

arXiv:2609.20886v1cs.LGcs.AIcs.CLcs.DB

TL;DR

BI users must prepare raw tables before analysis, and it remains an open question whether LLMs can automate this end-to-end workflow. The paper builds BI-Bench from real BI projects and develops BI-Agent with data-management tools and domain-specific post-training. BI-Agent tools improve accuracy by up to 40 percentage points, while post-training enables a small model to match or exceed much larger models at up to 50× lower cost.

  • Problem

    BI workflows require table selection, transformation, and joins before analysis, while existing benchmarks do not specifically target end-to-end challenges in real BI workflows.

    Source: §1 · pp. 1–2

  • Method

    The authors construct BI-Bench from real BI projects and develop BI-Agent, combining data-management tools with post-training on synthesized trajectories.

    Source: §1 · pp. 2–3, §5.1 · p. 7, §5.2 · p. 10

  • Results

    BI-Agent’s tools improve accuracy by up to 40 percentage points, and post-training enables Qwen3-8B to match or exceed larger models at up to 50× lower cost.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Combining tool use with domain-specific post-training produces synergistic gains on end-to-end BI tasks.

    Source: §1 · p. 3

    The work targets one-off, low-touch analysis over raw tables where no curated data foundation exists, not replacement of mature enterprise BI systems.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Business intelligence (BI) is a cornerstone of enterprise decision-making and is widely used by enterprise users in software such as Power BI and Tableau. In traditional BI workflows, users need to prepare data by (1) identifying relevant tables, (2) performing data transformations, and (3) building join relationships, before they can (4) answer their business questions. These steps can be complex and time-consuming, making BI challenging. Given the strong capabilities of large language models (LLMs) in working with data, we study their ability to answer BI questions end-to-end, without requiring users to manually perform the tedious preparation steps. To do this, we harvest a large collection of real-world BI projects from public sources, and manually extract pairs of (questions, ground-truth answers) from real user dashboards. The resulting benchmark, BI-Bench, is the first benchmark to systematically study LLMs' ability on end-to-end BI. We find that even frontier LLMs perform poorly on BI-Bench, with less than 50% accuracy. To address their limitations, we design a tool-augmented BI-Agent that decomposes BI workflows into subtasks on structured data, such as search, join, and transform, and orchestrates specialized data management methods across BI stages. Furthermore, we develop a post-training framework that synthesizes training trajectories from real BI projects, enabling BI-Agent to be further post-trained using both supervised fine-tuning (SFT) and reinforcement learning (RL). BI-Agent achieves substantial accuracy gains of up to 40 percentage points with vanilla LLMs, and post-trained BI-Agent yields gains of up to 30 points. Our results highlight the importance of combining tool-augmented reasoning with domain-specific post-training in complex BI workflows, and point to promising directions for future research.

1 INTRODUCTION

End-to-end BI requires users to select, transform, and join raw tables before analysis, but those steps challenge non-technical users. This paper introduces BI-Bench to evaluate LLMs on real BI workflows and BI-Agent, which combines data-management tools with post-training.

1 INTRODUCTION

  • BI workflows require identifying relevant tables, transforming data, and defining joins before users can perform analysis, tasks that challenge users without database or programming expertise.Source: §1 · pp. 1–2
  • BI-Bench is built from real-world BI projects and manually curated business questions with ground-truth answers, providing a benchmark for end-to-end BI workflows.Source: §1 · pp. 2–3
  • Frontier reasoning and chat-based LLMs still fail on over 50% of BI-Bench queries using SQL, especially when predicting joins and transformations over complex schemas.Source: §1 · p. 3
  • BI-Agent integrates data-management solutions as tools that LLMs can invoke in a tool-call loop to address BI workflow challenges.Source: §1 · p. 3
  • A post-training framework synthesizes training trajectories from real BI projects to improve models through supervised fine-tuning and reinforcement learning.Source: §1 · p. 3
  • BI-Agent’s tools improve accuracy by up to 40 percentage points, while post-training lets Qwen3-8B match or exceed larger models at up to 50× lower cost.Experiments cover 24 models and systems; Figure 1 compares quality and cost on BI-Bench using SQL.Source: Figure 1 · p. 1, §1 · p. 3

2 RELATED WORK

Prior data-analysis benchmarks do not capture the full preparation demands of real BI workflows, and no benchmark specifically targets the BI domain. BI-Bench addresses this gap with real dashboard questions, complementing work on individual workflow tools and data-analysis agents.

Business Intelligence

  • BI dashboards remain difficult to construct end-to-end because raw-data transformation and join definition are significant pain points for enterprise users.Tableau and Power BI offer drag-and-drop interfaces, but these workflows remain complex.Source: §2 · p. 3

Data analysis tasks and benchmarks

  • BI-Bench targets real BI workflows that existing NL2SQL benchmarks miss because they assume cleaned, structured, analysis-ready tables.It is constructed from real business questions extracted from dashboards created by users in the wild.Source: §2 · p. 3
  • No existing benchmark specifically targets the BI domain, whose tasks involve messy raw data and complex schemas uncommon in other domains.Examples include spreadsheet exports requiring structural transformations and star, snowflake, or constellation schemas.Source: §2 · p. 3

Methods to optimize BI workflows

  • Data-management research addresses individual BI challenges such as joining and transforming, and this work investigates using those algorithms as agentic tools for end-to-end BI.Source: §2 · p. 3

Data analysis agents

  • Existing data-analysis agents perform strongly on traditional analysis tasks but fall short on end-to-end BI workflows requiring complex data manipulations.Source: §2 · p. 3

3 PROBLEM: END-TO-END BI

End-to-end BI automates the path from raw tables to an answer, including data search, transformation, joins, and analysis. The problem definition focuses on producing a correct result table for a query, not reproducing a dashboard visualization.

3.1 Preliminaries

  • End-to-end BI requires automatically searching, transforming, and joining raw tables before analysis can answer a business question.Finding relevant tables is burdensome when source projects contain many unfamiliar tables.Source: §3.1 · p. 4
  • Table-reshaping operations such as pivot and unpivot convert non-relational layouts into relational tables suitable for analysis.A spreadsheet pivot table with months as column headers makes aggregation and joining on month difficult; relationalizing it makes both straightforward.Source: §3.1 · p. 4, Figure 3 · p. 5
  • LLMs often struggle to recognize and correctly execute table-reshaping transformations, risking downstream analysis failures.The passage attributes this to difficulty understanding table structure and generating correct reshaping code.Source: §3.1 · p. 4
  • Join prediction is difficult in large BI projects because cryptic surrogate keys and limited prompt-visible rows can obscure reliable value overlap.Classical methods use full-table statistics and global schema patterns such as star, snowflake, or constellation structures.Source: §3.1 · pp. 4–5
  • After preparation, BI analysis applies filtering, grouping, aggregation, and custom calculations to produce a result table using SQL or Python.Source: §3.1 · p. 5

3.2 Problem Definition

  • The formal problem takes a natural-language query and raw input tables, then requires search, transformation, joining, and analysis to produce a correct result table.Source: §3.2 · p. 5
  • The scope is one-off, low-touch analysis over fresh or external raw tables, not replacement of mature enterprise BI systems or reproduction of a particular visualization.The work evaluates correctness of the result table rather than a matching dashboard visualization.Source: §3.2 · p. 5

4 BI-BENCH: END-TO-END BI BENCHMARK

BI-Bench is built from real Power BI projects and manually curated dashboard questions with ground-truth result tables. Its evaluation shows that frontier models and strong NL2SQL systems struggle on these end-to-end BI tasks.

Dataset Preprocessing

  • BI-Bench contains 100 query–result-table cases curated from more than 3K public Power BI project files.The projects provide raw data, user-created transformations and joins, and dashboards.Source: §4 · pp. 5–6

Test Case Generation and Validation

  • The benchmark draws queries from clear, English user-dashboard visualizations and checks that each paraphrased question matches the visualization’s computation logic.Curators inspect visualization logic and outputs to restore implicit filters omitted from titles.Source: §4 · p. 6
  • Constructing BI-Bench took over 400 person-hours, with the 100-case cap balancing manual curation effort and statistical reliability.Source: §4 · p. 6
  • When simplified LLM checks fail, they trigger extensive manual review rather than excluding cases based on model solvability.The review repairs extraction errors, discards ambiguous or mismatched cases, and retains verified cases that models cannot solve.Source: §4 · p. 6

Ground-truth augmentation

  • Each test case can include multiple semantically valid ground-truth tables, reviewed by two domain experts, to accommodate valid schema variations.The alternatives are based on query semantics and visualization logic, not model outputs.Source: §4 · p. 6

Benchmark Analysis

  • 48.2% SQL accuracy is the best BI-Bench score reported for o4-mini; leading open-source BIRD and Spider 2.0-lite systems reach only 6.0–17.3% and 23.8–26.3% SQL accuracy, respectively.One test query requires reasoning over 38 tables.Source: §4 · p. 7
  • BI-Bench spans domains including sales, finance, and education, with sports and transportation among the areas grouped as “other.”The paper describes this domain diversity as reflecting the range of analyses BI users perform in practice.Source: §4 · p. 7

5 BI-AGENT: AI AGENT FOR END-TO-END BI

BI-Agent combines LLM reasoning with specialized data-management tools in an iterative tool-call loop, then post-trains models using automatically synthesized BI tasks. Its design targets table selection, reshaping, and joining, while training data and procedures are constructed to support supervised fine-tuning and reinforcement learning.

5 BI-AGENT: AI AGENT FOR END-TO-END BI

  • BI-Agent combines LLM reasoning, traditional data-management methods, tool use, and post-training to automate end-to-end BI.The authors say specialized tools can offload difficult data challenges from LLMs and improve overall model success rates.Source: §5 · p. 7, §5.1 · p. 8

5.1 Architecture: reasoning and tool-call loop

  • The orchestrator invokes specialized search, transform, and join tools alongside a coding tool, which interact with raw data in an iterative reasoning and tool-call loop.The architecture supports both vanilla and post-trained LLMs as orchestrators.Source: §5.1 · p. 7, Figure 5 · p. 8
  • Transform predicts reshaping operations and returns original and transformed tables, while join predicts relationships for snowflake-like schemas.The transform tool considers transpose, pivot, and unpivot; the join tool returns predicted joins for input tables.Source: §5.1 · p. 7
  • Search narrows attention by retaining plausibly relevant tables while pruning only those clearly irrelevant to the query.The conservative strategy is intended to reduce confusion from semantically related but irrelevant tables.Source: §5.1 · p. 8
  • The tool set is explicitly non-exhaustive and can be extended with additional data-management primitives in the same tool-call loop.Source: §5.1 · p. 8

5.2 Post-training models for end-to-end BI

  • Leakage controls exclude BI-Bench projects and projects with at least 5 rows of partial table-content overlap; with column-header-token Jaccard, 0.73% of training transforms are more similar to a test transform than its BI-Bench nearest neighbor.Source: §5.2 · pp. 8–9
  • Training tasks start from connected project sub-schemas materialized as wide tables, then synthesize queries and results before rebuilding raw inputs with irrelevant tables and occasional inverse reshaping.The pipeline also generates trajectories that take the raw inputs to the target result table.Source: §5.2 · p. 9
  • SFT trajectories are retained only after exact result-table verification, whereas RL explores trajectories against target tables using graded correctness and syntax-error rewards.RL uses tasks for which GPT-4o generated correct trajectories during SFT as rollout tasks.Source: §5.2 · p. 10

6 EXPERIMENTS

The experiments evaluate how BI-Agent’s tools and post-training affect accuracy across BI-Bench and an unseen NL2SQL benchmark, while analyzing errors, interaction efficiency, token use, latency, and trajectory changes under SFT and RL.

6.1 Experiment Setup

  • The experiments compare 24 models across tool/no-tool and SQL/Python settings, evaluating accuracy by table shape and cell values.The tool condition adds BI-Agent’s search, transform, and join tools to a generic coding tool; results are averaged over 10 runs, with paired significance tests and bootstrap confidence intervals.Source: §6.1 · pp. 10–11

6.2 Main Results: Quality and Costs

  • BI-Agent tools increase accuracy by 14 percentage points in SQL and 11 in Python, with statistically significant gains in 19 of 20 cases.GPT-OSS-120B accuracy rises by over 8× in both languages; its median model calls increase from 3 to 9 in SQL and from 2 to 5 in Python as it interrogates data rather than guessing. Across post-trained variants, Qwen3-8B-RL costs $0.19 for the entire benchmark versus over $10 for o4-mini and GPT-4o.Source: §6.2 · pp. 11–12, Table 3 · p. 12
  • With tools and RL post-training, Qwen3-8B gains 29.8 percentage points in SQL, with all overall post-training gains significant at p<0.001.The post-trained model’s $0.19 BI-Bench execution cost is reported as 54× more cost-effective than o4-mini and GPT-4o while delivering similar performance. Top BIRD NL2SQL models score under 10% on BI-Bench, while ktx and Databao Agent score around 25%.Source: Table 4 · p. 12, §6.2 · p. 12

6.3 Sensitivity Analysis

  • Removing either dense partial credit or syntax-error penalties reduces RL accuracy below the SFT initialization.In the Python-with-tools setting, full-reward RL reaches 25.1% accuracy versus 22.6% for SFT, while the three reward ablations score 20.7%, 19.1%, and 20.6%.Source: §6.3 · p. 13
  • Accuracy improves monotonically as tools are added, and tool and post-training gains grow with task complexity.For GPT-4o, gains rise from 4.1 to 4.6 to 5.2 percentage points in Python and from 1.8 to 3.7 to 5.1 in SQL as one, two, and three tools are enabled. On tasks with more than two joins, Python without tools, SFT improves accuracy by 4.1× and RL by 5×, versus 1.5× and 1.8× on tasks with at most two joins.Source: §6.3 · p. 13, Figure 7 · p. 14, Figure 9 · p. 14

6.4 Error Analysis

  • Across all large models, tools reduce transformation errors by 54.2% and table-selection errors by 40.9%, with transformation errors showing the larger reduction.For Qwen3-8B-RL, table-selection errors decrease by 67% and transformation errors by 61%. Comparing Qwen3-8B without tools with Qwen3-8B-RL with tools, table-selection errors decrease by 67% and transformation errors by 50%.Source: §6.4 · pp. 13–14

6.5 Generalizability Test

  • On unseen Spider 2.0 tasks, tools improve results over no-tool settings, SFT beats the base model, and RL further improves over SFT.Combining tools with SFT+RL yields a 20.5× quality improvement over the base Qwen3-8B model’s accuracy, despite post-training on different BI tasks.Source: Table 5 · p. 15, §6.5 · p. 15

6.6 Cost and latency benefits

  • Tools reduce average interaction rounds from 7.2 to 6.3 overall, replacing longer reasoning or coding processes with structured operations.The reduction is larger in SQL, from 8.4 to 6.6 rounds, than in Python, from 6.1 to 6.0; the paper attributes the smaller Python gain to native data-manipulation operations for complex transformations.Source: Figure 11 · p. 15, §6.6 · p. 15
  • With tools, correct executions average 6.3 rounds versus 6.2 without tools, while incorrect executions average 6.5 versus 7.5 without tools.Tools add debugging and verification steps that can correct intermediate mistakes, while helping prevent prolonged failures through faster recovery or earlier termination.Source: §6.6 · p. 15
  • Across Python and SQL, correct executions average 6.4 rounds, compared with 7.1 rounds for incorrect executions.Incorrect trajectories require more debugging, retries, or compensatory steps before termination.Source: §6.6 · p. 15
  • Output tokens decrease by 25% for Python and 16% for SQL with tools, alongside fewer interaction rounds and lower user-observed latency.The passage attributes reduced user-observed latency and model response time to these changes.Source: §6.6 · p. 15
  • Average interaction rounds initially increase after SFT, then decrease following RL as rollout-based feedback refines action strategies and reduces unnecessary steps.SFT on traces generated by large models encourages more active tool use and explicit intermediate reasoning, producing longer trajectories.Source: §6.6 · p. 15

7 CONCLUSIONS AND FUTURE WORK

The paper takes a first step toward answering ad-hoc BI questions with LLMs, introducing BI-Bench and BI-Agent. It reports substantial improvements from data-management tools and model post-training.

7 CONCLUSIONS AND FUTURE WORK

  • BI-Bench is the first benchmark to systematically evaluate LLMs using BI workflows.It is built from real queries extracted from real BI projects.Source: §7 · p. 15
  • BI-Agent is an agentic framework for end-to-end BI that combines data-management tools with model post-training.The paper reports that these lead to substantial improvements.Source: §7 · p. 15
  • The authors present LLM-driven BI as a promising direction and hope the work serves as a springboard for future research.Source: §7 · p. 15

Related papers

Loading 2609.20886v1…