Source-linked AI summary
Evaluating Enterprise Analytics Agents: An End-to-End, Trace-Backed Methodology
Teja Venkat Kolli, Sang Su Lee, Xueying Yan, Jessie Chen, Chi Cheng, Kartik Ravisankar, Shishir Dash, Vijay Anand Raghavan
TL;DR
Enterprise analytics agents require evaluation beyond SQL correctness because plausible answers can use wrong sources, miss analytical requirements, or vary across runs. The paper presents an end-to-end, trace-backed methodology using golden answers, repeated runs, validity checks, and abstention-aware scoring. In the case study, the higher-capability configuration answered more often and used real data more frequently, but still exhibited source, decomposition, efficiency, and consistency failures.
Problem
Final-answer or SQL-only evaluation does not adequately assess analytics agents that interpret business intent, select sources, execute tools, and produce decision-relevant answers.
Method
The paper evaluates agents with human-written golden analytical contracts, repeated isolated runs, runtime traces, run-validity checks, and tiered abstention-aware scoring.
Results
The higher-capability configuration reduced early refusal from 73% to 0% and increased real-data answers from 21% to 73%, while still producing source, decomposition, efficiency, and interpretation failures.
Takeaways & Limitations
Trace-backed, end-to-end evaluation exposes failures that final-answer evaluation hides and provides a measurement surface for improving analytics-agent reliability.
Takeaways & Limitations
The case study uses internal data and agents, compares only two configurations, and estimates consistency from three repetitions per question.
Abstract
from arXiv · showhide
Enterprise analytics agents are not only text-to-SQL systems. They interpret business intent and choose metric definitions. They select data sources, execute tools, inspect results, and produce natural-language answers. Those answers may influence operational, financial, or executive decisions. Grading final answers hides where these agents fail. A plausible answer can use the wrong source of truth. It can skip a required decomposition, claim causality without support, or change its table interpretation across repeated runs. We present an end-to-end evaluation methodology for analytics agents. The methodology grades agent behavior across three families: semantic understanding, execution quality, and reliability. Grading uses question banks with human-written golden answers, repeated runs, and runtime traces. Each run first passes a run-validity check, then receives tiered, abstention-aware scores that feed a decision framework rather than a release gate. We instantiate the methodology on a controlled internal analytics agent at a large online marketplace. The case study uses 50 analytics questions, two anonymous model configurations, and three randomized repetitions per configuration, yielding 300 traces. The higher-capability configuration reduced early refusal from 73% to 0% and increased real-data answers from 21% to 73%. However, it also exhausted the tool-round budget on 16% of runs. It overran the schema-exploration budget on 77% of traces. It changed its table interpretation on 41 of 50 questions. On finance questions with structured golden answers, source table use and escalation improved, but canonical decomposition remained weak in both configurations. These results show why trust in analytics agents requires end-to-end, trace-backed evaluation rather than SQL correctness or final answer quality alone.
1 Introduction
Enterprise analytics agents require workflow-level evaluation because plausible answers can still use wrong sources, skip analytical requirements, or make unsupported claims. The paper proposes an end-to-end methodology covering these failures and demonstrates them in a 300-trace case study.
- Plausible final answers can still reflect wrong sources, date windows, business-rule filters, missing decomposition, or unsupported causal claims.
- SQL correctness alone is insufficient because trustworthy evaluation must inspect how answers were produced and whether repeated runs remain stable.
- The methodology is positioned as a measurement foundation for decision-grade analytics-agent governance, including decisions to abstain, escalate, or seek human review.
- The paper proposes workflow-level analytics-agent evaluation covering interpretation, source selection, SQL and tool execution, analytical method, grounding, confidence, and repeatability.
- The case study shows that a higher-capability configuration can answer more questions while still failing source selection, decomposition, grounding, efficiency, and cross-run consistency.
2 Why Analytics-Agent Evaluation Is Different
Analytics-agent evaluation differs from standard text-to-SQL because agents must resolve business-specific semantics, execute tools correctly, reason about insights, and remain repeatable. The methodology therefore combines analytics-domain checks with trajectory-aware and cross-run evaluation.
- Enterprise analytics agents must reason about metric definitions, source hierarchies, table grain, time windows, and the user’s decision context beyond returning an expected SQL answer.
- Evaluation spans question interpretation, SQL and data sourcing, insight reasoning, and output quality across semantic understanding, execution quality, and reliability.
- Trajectory-aware evaluation detects wrong tool arguments, skipped prerequisite lookups, schema loops, failed recovery, and missing escalation that final answers can hide.
- Analytics-specific checks map generic tool failures to domain failures such as wrong source, grain, time window, missing decomposition, and unsupported guidance.
- Repeatability testing is required because different tables, numbers, or conclusions across repetitions undermine reliability for high-stakes analytics questions.
3 End-to-End Evaluation Methodology
The methodology evaluates analytics agents through representative question banks, human-written analytical contracts, repeated isolated runs, and runtime traces. These stages create evidence for locating failures rather than judging final answers alone.
- The evaluation runs randomized, repeated trials with human-written golden answers and runtime traces before applying run-validity checks and tiered scoring.
- Question banks are sampled from realistic workflows and tagged by business area, question type, source tier, ambiguity, and decision risk.
- Golden answers define an analytical contract including required or prohibited sources, windows, filters, decomposition steps, numeric bands, expected terms, and disallowed claims.
- Isolated sessions, fixed-seed question randomization, and repetition make stochastic behavior observable and test whether interpretations remain consistent.
- Runtime traces record questions, metadata, tool calls, queries, outputs, errors, retries, latency, tool-round counts, and final answers to localize failures.
3.5 Run Validity Before Scoring
The methodology checks run validity before interpreting quality scores, separates primary from diagnostic signals, and treats abstention as an explicit outcome. It is a decision framework requiring human calibration rather than a validated release policy.
- Run-validity checks prevent broken environments, authentication failures, and absent real data from being mistaken for agent behavior.
- Primary signals assess the analytical contract, while diagnostic signals explain changes without requiring every diagnostic to become a release gate.
- Scorers return yes, partial, no, or unknown; unknown is an abstention reported separately through coverage rather than treated as failure.
- The methodology is a decision framework rather than a validated release policy, so thresholds require domain-expert calibration before blocking deployment decisions.
4 Analytics-Specific Failure Taxonomy
The failure taxonomy combines analytics-specific correctness with tool-agent process failures. It also distinguishes guidance, workflow enforcement, and trajectory evaluation as complementary safeguards.
- It maps each failure mode to workflow-oriented repairs in context, tools, workflow enforcement, or review policy.
- The taxonomy covers question interpretation, semantic and source selection, analytical method, insight reasoning, execution, grounding, and repeatability.
- Semantic guidance, workflow enforcement, and trajectory evaluation are all needed because they respectively describe, enforce, and verify expected behavior.
5 Case Study
The case study evaluates two model configurations of a controlled enterprise analytics agent using repeated traces and a taxonomy-based decision framework. Its finance scorecard provides diagnostic evidence rather than a complete benchmark.
- The study evaluates two model configurations of an internal marketplace analytics agent using the same tools, prompt scaffold, and runner.The configurations are anonymized as Fast and Reasoning.
- 50 questions were run three times per configuration, yielding 300 total traces across five internal business areas.
- Finance provides the only complete structured-golden scorecard, with 30 traces per configuration; the other areas lack equivalent golden coverage.
6 Results
Across 300 traces, the higher-capability configuration answered more often but still exhibited source-selection, decomposition, trace-only, and cross-run reliability failures. Trace-backed evaluation exposed these contract violations beyond final-answer quality alone.
- 6.1 Run Validity and Answering Behavior: 0% versus 73% early refusal and 73% versus 21% real-data answers show that the Reasoning configuration answered substantially more often.It also hit the maximum tool-round budget on 16% of runs, shifting the dominant failure mode toward over-exploration and unstable interpretation.
- 6.2 Finance Golden Answers: 50% required-table use and weak decomposition show that improved answering did not reliably satisfy the expected analytical method.Table 4’s finance scorecard reports improved attempts, escalation, and source-table discovery, but required-table use reached only 50%.
- 6.3 Correct Number, Wrong Analytical Contract: A plausible finance answer with revenue within about 0.3% of the golden answer still used a non-canonical source and skipped decomposition on 90% of Reasoning traces.Final-answer grading would treat these traces as completed answers, whereas trace-backed scoring identifies analytical-contract failures.
- 6.4 Trace-Only Risks: The Reasoning configuration produced more implausible numeric claims and hard contradictions, while lower numeric-claim rates for Fast partly reflected answering fewer numerical questions.These trace-only diagnostics do not require complete golden answers, and silence is not reliability.
- 6.5 Cross-Run Consistency: On 41 of 50 questions, the Reasoning configuration changed its table set or query interpretation across repetitions.This interpretation instability means repeated runs can yield different coherent answers without revealing that the underlying source of truth changed.
7 From Evaluation to Intervention
The paper argues that model upgrades alone do not remove analytics-agent failures, so evaluation should connect observed failure surfaces to enforceable workflow interventions. Its governance primitives support traceability, abstention, tiered scoring, stability, and version comparison without constituting a deployed governance program.
- 7 From Evaluation to Intervention: Model upgrades changed the failure surface but did not remove source-of-truth errors, weak decomposition, schema overrun, contradictions, or interpretation drift.The Reasoning configuration answered more questions and produced more grounded outputs while retaining these failures.
- 7 From Evaluation to Intervention: Analytics agents need semantic context paired with enforcement through metric glossaries, source hierarchies, canonical routing, decomposition templates, validation checks, and observable escalation.Access to definitions or canonical patterns alone is insufficient; lookup, validation, and escalation must be testable in the workflow.
- 7 From Evaluation to Intervention: Interventions should be decomposed into measurable components so evaluation can show whether a change improves source selection, answer completeness, or another specific layer.When decomposition is not possible, per-layer scoring can still identify what changed.
- 7.1 Governance for Business Decisions: The methodology defines governance primitives for traceability, abstention and escalation, tiered scoring, run stability, and version comparison.These primitives are grounded in runtime traces, abstention-aware scoring, isolated repeated runs, and a trace-backed contract.
- 7.1 Governance for Business Decisions: These governance primitives make evaluation-based governance possible but do not constitute a deployed governance program.The paper explicitly distinguishes measurement foundations from operational governance.
8 Related Work
Prior work evaluates tool use, trajectories, repeated task success, and text-to-SQL, but enterprise analytics also requires domain-specific source selection, metric definitions, and analytical contracts.
- Prior agent benchmarks establish the value of interleaved reasoning, tool use, and repeated task consistency beyond simple question answering.
- Judge-based scoring can support nuanced interpretation and insight evaluation, but the methodology relies primarily on deterministic trace-backed signals and requires calibration against domain experts.
- Enterprise analytics evaluation adds source-of-truth selection, local metric definitions, and domain-practice explanations beyond public text-to-SQL and trajectory benchmarks.
- A workflow-level evaluation ties runtime trajectories to analytical contracts and maps analytics-specific failures to concrete repair targets.
9 Limitations and Future Work
The study’s evidence is constrained by its internal single-company setting, limited model and repetition coverage, and nonportable case-study measurements. Future work targets calibrated decision thresholds, semantic context instrumentation, and cross-domain validation.
- 9 Limitations and Future Work: The study is limited to one company, two anonymized model configurations, and no deployed semantic-layer evaluation, constraining generalization and attribution.
- 9 Limitations and Future Work: Three repetitions per question provide lower-bound interpretation-instability counts rather than precise estimates of run-to-run stability.
- 9 Limitations and Future Work: Latency and execution-efficiency figures reflect the specific warehouse and execution environment rather than intrinsic or portable model properties.
- 9 Limitations and Future Work: Authoring and reviewing each structured golden answer took a few hours of a domain-familiar data scientist, motivating calibrated automation of mechanical checks.
- 9 Limitations and Future Work: Future work will calibrate decision thresholds with domain experts, instrument semantic context components, and test the failure taxonomy on another domain.
- 9 Limitations and Future Work: The specific case study cannot be directly reproduced because its question bank, agent system, and warehouse are internal, although the methodology is independently reproducible.
10 Conclusion
The paper argues that analytics agents must be evaluated across their analytical workflows, using traces and repeated trials rather than final answers alone. Its case study shows that greater answer coverage can coexist with source, decomposition, efficiency, contradiction, and interpretation-drift failures.
- The methodology combines question banks, human golden answers, repeated runs, runtime traces, run-validity checks, tiered scoring, abstention-aware decisions, and cross-run consistency analysis.
- Trace-backed evaluation exposes source-of-truth errors, weak decomposition, schema overruns, contradictions, and interpretation drift that final-answer evaluation hides.