Source-linked AI summary
DI-Bench: Systematically Generating In-Domain Data Intelligence Benchmarks for Enterprise Agents
Jiangyun Zhang, Kristen Surrao, Torpong Nitayanont, Yupei Zhang, Roopali Singh, Zhiyu Chen, Julia Huang, Zhou Tang, Shayan Ali Akbar, Omar Alonso, Erwin Cornejo, Yuan Li, Yi Zhang
TL;DR
Enterprise-agent benchmarks rarely test the integration of retrieved business knowledge with analytical computation, and constructing such benchmarks manually is costly. DI-Bench generates validated tasks by linking databases and business documents, computing gold answers programmatically, and generating questions from those answers. Across four models, average accuracy falls to 32% when retrieved rules must modify computation, revealing a persistent knowledge-to-computation gap.
Problem
Public benchmarks rarely evaluate whether enterprise agents can combine retrieved business knowledge with analytical computation, despite the importance of such tasks.
Method
DI-Bench builds an artifact linkage graph over enterprise data and documents, computes gold answers through query execution, then generates and validates natural-language tasks.
Results
Average accuracy drops from 82% on Knowledge tasks to 32% on Knowledge + Analytics + Rule tasks across four evaluated models.
Takeaways & Limitations
The benchmark indicates that retrieving business knowledge is less difficult than correctly incorporating it into downstream computation.
Takeaways & Limitations
The demonstration uses synthetic business-rule documents and metric catalogs, so the benchmark approximates rather than reproduces any specific organization’s business processes.
Abstract
from arXiv · showhide
Evaluating enterprise agents on domain-specific benchmarks is critical, yet public benchmarks rarely evaluate whether agents can integrate business knowledge with analytical computation, and constructing such benchmarks manually is costly. We present DI-Bench, a pipeline for generating realistic benchmarks for data intelligence (DI), the practice of extracting insights from large volumes of enterprise data. To emulate realistic DI tasks that require both computation and knowledge retrieval, DI-Bench builds an artifact linkage graph over data tables, dimensions, metrics, and documents to form questions involving structured data and associated knowledge. Ground truth answers are derived via query execution, followed by LLM question generation and validation. Applied to two public datasets, the pipeline produces a 731-task benchmark covering knowledge retrieval, analytical computation, and rule-grounded reasoning. To show the discriminatory capability and difficulty of the benchmark, we evaluate four models, revealing a substantial finding: models achieve only 32% accuracy when doing computational tasks where retrieved business rules modify the computation.
1 Introduction
Existing benchmarks rarely test whether agents can retrieve business knowledge and incorporate it into analytical computation, while manual construction is costly. DI-Bench addresses this gap with an end-to-end pipeline that generates validated, domain-specific benchmarks from structured data and business knowledge.
- Motivation: Correctly answering enterprise questions can require retrieving service-level rules, querying relevant tables, and applying those rules during computation.For resolution-time commitments, premium-customer targets, paused clocks, and weekend exclusions affect the eligible calculation.
- Motivation: Existing benchmarks cover only parts of the sequence spanning knowledge retrieval, database querying, business-rule understanding, and analytical computation.Text-to-SQL benchmarks provide required knowledge inline, while other benchmarks name the documents needed for each task.
- Motivation: Multi-hop text benchmarks structurally test evidence retrieval but do not address database-linked rules whose inclusion must change a computed answer.Analytical tasks require verifying answer effects, not only evidence necessity.
- Contribution: DI-Bench links structured databases with unstructured business knowledge and uses programmatic execution and automated validation to produce deterministic ground truth.The pipeline is designed to reduce manual benchmark-construction effort while retaining realistic enterprise artifacts.
- Contribution: 731 validated tasks across Brazilian e-commerce and Czech retail banking require retrieval, computation, and business-rule reasoning.The benchmark spans nine analysis types and supports empirical evaluation of enterprise agents.
2 Problem Setting
DI-Bench targets business questions whose required information is scattered across relational data and rule documents. Its setting assumes a known-schema database and a knowledge base containing computation-relevant definitions, thresholds, and overrides.
- Setting: DI-Bench studies questions requiring both relational query execution and retrieval-augmented generation over documents containing computation-relevant rules.The running example combines ticket histories, customer-plan data, and service-level rules.
- Inputs: The assumed inputs are a relational database with a known schema and a knowledge base of business definitions, thresholds, conditional overrides, and metric formulas.The knowledge base also includes sample SQL templates, while the demonstration documents are LLM-generated augmentations.
- Core challenge: Information scattering makes each task depend on multiple artifacts rather than a single complete source.A metric catalog may define CSAT while a separate rule document excludes neutral reviews from the denominator.
- Core challenge: Ignoring a computation-relevant rule can shift the resulting answer by several percentage points.The CSAT example illustrates why retrieving the separate denominator rule matters analytically.
3 Methodology
DI-Bench generates tasks by linking database artifacts and business documents, executing parameterized computations for gold answers, generating questions backward from those answers, and validating the resulting pairs. Its artifact linkage graph and counterfactual checks are designed to ensure that retrieved rules genuinely affect computation.
- Pipeline: The pipeline constructs an artifact linkage graph, samples task subgraphs, executes queries for gold answers, generates natural-language questions, and validates task quality.The stages are organized as graph construction, subgraph sampling, gold-answer computation, answer-conditioned question generation, and validation.
- Artifact Linkage Graph: The artifact linkage graph connects tables, metrics, rules, and dimensions through typed dependency edges.The rule→metric edge records which business documents modify metric computations.
- Artifact Linkage Graph: Candidate rule links are checked counterfactually to determine whether including a rule changes the SQL or answer.This filtering reduces spurious associations, while imperfect extraction primarily affects coverage rather than correctness.
- Task generation: Connected subgraphs fix the metric, data sources, dimensions, and applicable rules, then yield multiple tasks by varying filters, time windows, and analysis types.Enumeration and sampling respect rule applicability and metric dependencies, while MMR promotes diversity across selected subgraphs.
- Gold answers: DI-Bench computes deterministic gold answers by applying rule-specific SQL modifications to parameterized metric queries and executing them on the database.Post-processing converts raw query results into structured answers for nine supported analysis types.
- Question generation and validation: An LLM generates stakeholder-style questions from the known metric, answer, analysis type, and required documents, after which candidates undergo automated and LLM-based validation.Admission requires passing eight binary quality dimensions, with phrasing-only failures eligible for rewriting and rescoring.
4 Experiments
DI-Bench evaluates agents on 731 tasks from two public domains and finds that performance declines sharply as tasks add analytical computation and rule modification. The benchmark also exposes task-type difficulty and failure modes centered on applying retrieved rules rather than merely finding them.
- Benchmark and evaluation: DI-Bench contains 731 tasks: 449 from Brazilian e-commerce and 282 from Czech retail banking.The two domains differ in schema shape, terminology, and rule structure, while their metric and rule documents are fictional.
- Model performance: Opus achieves 57.9% overall accuracy, followed by DeepSeek V3.2 at 50.9%, Haiku at 49.0%, and Qwen3-80B at 47.9%.The Brazilian–Czech accuracy gap remains approximately 10 percentage points across models.
- Knowledge-to-Computation Gap: Average accuracy falls from 82% on Knowledge tasks to 50% on Knowledge + Analytics and 32% on Knowledge + Analytics + Rule.The task groups contain 137, 419, and 175 tasks respectively.
- Task difficulty: Accuracy ranges from 43–81% on directional or binary judgments, while ranking and point-query tasks remain difficult because every result row must match exactly.Counterfactual and decomposition tasks receive minimum-coverage augmentation but remain descriptive because their cells are small.
- Failure Analysis: Manual inspection finds agents either omit retrieved rules from SQL or acknowledge them while executing the unchanged naive query.These errors show that the benchmark primarily measures rule application during computation rather than rule location.
5 Ablation and Robustness Analyses
Matched variants show that the main difficulty is applying retrieved business rules during computation, not merely retrieving those rules. Validator checks indicate strong but imperfect agreement with human judgments and similar question quality across generators.
- Rule application: 20.4 points: removing the rule requirement raises average accuracy from 32.5% to 52.9%, while directly providing the rule changes accuracy by only 0.2 points.The matched variants hold the metric, dimension, time window, analysis type, and answer format fixed.
- Rule application: 15.8 points: within threshold_crossing, accuracy rises from 62.1% to 77.9% when the rule requirement is removed.This task-type result is consistent with rule application being a bottleneck beyond retrieval alone.
- Generator and LLM judge sensitivity: 79.9% versus 79.6%: Qwen3-80B- and DeepSeek-generated questions receive similar mean quality scores from an independent Claude Sonnet 4.6 judge.The comparison uses 87 valid paired examples with the same task skeletons.
- Generator and LLM judge sensitivity: 94–96% precision and 79–93% recall: four validator judges show strong but incomplete agreement against a balanced human-labeled sample.The checks bound but do not eliminate the shared-model confound.
6 Conclusion
DI-Bench is a transferable pipeline for generating in-domain benchmarks that combine knowledge retrieval with analytical computation and use programmatic execution for deterministic ground truth. Across two public domains, its 731-task benchmark exposes a large gap between retrieving business knowledge and correctly incorporating it into computation.
- Conclusion: DI-Bench links structured databases with unstructured business documents and computes gold answers through programmatic execution to produce realistic tasks with deterministic ground truth.The pipeline is designed for data-intelligence tasks requiring both knowledge retrieval and analytical computation.
- Conclusion: 731 tasks across two public domains reveal a knowledge-to-computation gap in all four evaluated models.The benchmark is demonstrated on Brazilian e-commerce and Czech retail banking.
- Conclusion: 32% accuracy when retrieved business knowledge must be incorporated into computation, compared with 82% on retrieval-only tasks.The paper identifies grounding retrieved knowledge in downstream tasks as the major bottleneck within this evaluation.
- Conclusion: DI-Bench provides a scalable framework for evaluating agentic capability as business logic, metrics, and data evolve.This consequence is stated within the scope of the demonstrated benchmark pipeline.
Limitations
The benchmark’s transferability and realism are bounded by its limited domain coverage and synthetic business artifacts. Exact-match scoring and small analysis-type cells also constrain interpretation of some results.
- Scope: The pipeline is demonstrated on only two domains, so broader validation across additional industries is needed to strengthen the transferability claim.The demonstrated domains are e-commerce and banking.
- Data realism: Synthetic rule documents and metric catalogs approximate enterprise analytical pipelines but may not capture production documentation’s ambiguity, inconsistency, evolving governance, or cross-team dependencies.The artifacts are grounded in schemas but are not artifacts from a real enterprise.
- Evaluation: Exact-match scoring awards no partial credit for correct reasoning with minor computational errors.This scoring choice can make small computational mistakes count as fully incorrect.
- Evaluation: Small analysis-type cells remain too limited for strong inferential comparisons, despite minimum coverage of at least 10 counterfactual, decomposition, and significance-testing tasks per domain.The benchmark includes these minimum counts through augmentation, but the resulting cells remain small.
- Benchmark design: Each analysis type composes primitive operators such as Retrieve, Filter, Aggregate, Compare, Rank, Temporal, and Test.The supplied passages define this representation but do not state an additional limitation beyond the benchmark’s design scope.
C Ablation: Impact of Artifact Linkage Graph
The artifact linkage graph is evaluated by testing whether its rule pairings change computed answers and by describing the quality-diversity selection and validation pipeline. Graph-selected rules are substantially more discriminative than semantic or random alternatives, while MMR and validation support varied, executable tasks.
- Rule-pairing ablation: Graph-selected rules capture answer-changing dependencies more reliably than topical similarity because typed rule→metric edges encode computational effects.Semantic similarity offers only a small gain over random pairing, showing that related wording need not imply a changed metric value.
- Quality and diversity selection: MMR selects candidate subgraphs by balancing composite quality against diversity from already selected subgraphs.The quality score combines five independent scorers, while the distance is Jaccard distance over metrics, tables, dimensions, and rules.
- Quality and diversity selection: With λ = 0.3, selection favors quality over diversity, followed by a coverage pass that adds missing metrics or rule documents.When targeting 100 questions, approximately 93% of metric–rule links are retained, with losses concentrated among metrics having many associated rules.
- Validation: Tasks are admitted only after deterministic checks and LLM judging verify structural integrity, answer completeness, objectivity, leakage, and naturalness.Programmatic checks exclude defective tasks before LLM evaluation, and the LLM judge scores eight dimensions with binary pass/fail decisions.
E.4 Generator and Judge Sensitivity
Generator-swap results show similar overall judged question quality for Qwen3-80B and DeepSeek, but dimension-level differences remain. Human experts solve most sampled tasks while agents perform substantially worse, supporting benchmark discrimination rather than universal task impossibility.
- Generator sensitivity: 79.9% versus 79.6% mean quality scores were assigned to Qwen3-80B- and DeepSeek-generated questions by an independent Claude Sonnet 4.6 judge.The comparison used fixed skeletons and SQL-computed gold answers on 87 valid paired examples.
- Generator sensitivity: SQL correctness decreases from 51% to 34%, while format alignment increases from 86% to 99% across the generator swap.Individual quality dimensions therefore move in both directions despite similar mean scores.
- Human reference point: Business-intelligence experts solve 13 of 15 sampled tasks, or 87%, whereas agents achieve substantially lower accuracy on the same samples.The human reference covers Knowledge, Knowledge + Analytics, and Knowledge + Analytics + Rule tiers.
G Benchmark Construction Time
DI-Bench’s pipeline shifts benchmark construction from extensive manual authoring to shorter human verification. For the 731-task benchmark, the estimated effort falls from about 420 hours to about 22 hours.
- Manual construction: About 420 hours are estimated for fully manual construction of a 731-task benchmark.The estimate is based on roughly 35 minutes per task, including subgraph creation, SQL work, rule-modified SQL verification, phrasing, answer formatting, and final checks.
- Pipeline-assisted construction: Pipeline-based verification takes roughly 22 hours for the entire benchmark, or about 1.8 minutes per task on average.Verification is the remaining human effort after automated generation and validation.
I Example Tasks
The benchmark includes diverse analytical tasks spanning retrieval, computation, rankings, trends, thresholds, statistical tests, and business-rule application. Across 731 tasks, models perform strongly on easier task types but fall below 35% on several harder analytical types.
- Task Coverage: The examples cover nine analysis types, including point queries, rankings, trends, threshold comparisons, statistical significance, decomposition, and counterfactual reasoning.The benchmark combines structured questions, required rules, and gold answers across Brazilian e-commerce and Czech retail banking.
- Validation: The validation process uses human and LLM checks, with admitted tasks achieving a 99% average human-verified pass rate while only 25% of candidates are admitted.The validator is stricter than human reviewers, trading candidate yield for high precision.
- Rule-Grounded Analysis: Representative tasks require rule-aware computations such as applying transaction filters, service-level thresholds, delivery exclusions, or statistical procedures.Examples include summing only transactions with type = ‘PRIJEM’, comparing ship-out time with a 60-hour target, and applying Welch’s t-test.
- Evaluation Results: 731 tasks were evaluated across models and analysis types, with all four models exceeding 75% on knowledge retrieval and 60% on threshold crossing and trend analysis.These results are reported for the combined Brazilian and Czech benchmark.
- Evaluation Results: All four models fell below 35% on ranking, point query, counterfactual, and decomposition tasks.On point query, Opus reached 34%, compared with 18% for Qwen3-80B and 22% for Haiku.
K Detailed Failure Cases
Detailed failures show that agents can retrieve governing rules and produce roughly correct base SQL yet still mishandle rule-induced adjustments. These failures explain why rule-aware analytical tasks provide the benchmark’s strongest discrimination.
- Failure Pattern: The failure analysis isolates rule application as the bottleneck: retrieval succeeds, base SQL is roughly right, but adjustments are dropped or misapplied.The recurring adjustments include grace periods, privacy suppression, and counterfactual rule swaps.
- Case 1 — Threshold Crossing: A one-business-day SLA grace period changed DF’s order defect rate from 16.40% to 14.46%, flipping the result from PASS to FAIL against a 15.0% threshold.Opus counted deliveries after the estimated date rather than after the estimated date plus one day.
- Case 2 — Ranking: For HHI ranking, Opus computed the base concentration metric but failed to apply confidentiality suppression for categories supported by fewer than three unique sellers.The expected ranking retained only categories satisfying the post-hoc privacy rule, whereas Opus returned unsuppressed alternatives.
- Case 3 — Counterfactual: The counterfactual case requires parallel computation of On-Time Delivery Rate with and without the grace period for each customer state.The example compares strict and with-grace values, making the rule swap part of the requested output.
- Pipeline and Reproduction: DI-Bench’s implementation combines graph construction, candidate scoring, SQL records, validation inputs, and executable stages, but exact task wording is not fully reproducible because LLM-backed stages use nondeterministic decoding.The released configuration records model identifiers and decoding parameters; non-LLM randomized stages use seed 42.