Fail Loudly: An Auditable Runtime for Agentic Data Analysis

Motivation

• Among 580 zero-scored DS-STAR runs, 405 (69.8%) produced normal-looking but incorrect outputs without reporting failure.One example is an unkeyed merge of two Top-5 rankings that succeeds but produces incorrect row correspondence.Source: §1 · p. 1, Figure 1 · p. 2
• Existing approaches leave room for operation-level checks that connect analytical choices to relevant source observations and their coverage.A complete column-value enumeration can support rejecting an absent filter value; omission from a partial preview cannot.Source: §1 · p. 2

Contribution

• RADAR links evidence-preserving exploration, typed execution, and runtime validation so conflicts can prompt rejection, diagnostic feedback, or bounded repair.The runtime checks arguments, execution results, and output requirements against available evidence, while the agent retains responsibility for interpreting the task and revising its choices.Source: §1 · p. 2

Results

• On KramaBench, RADAR scores 0.723 with Full Input and 0.747 with Oracle Input, versus strongest-baseline scores of 0.532 and 0.580, respectively.Across 1,054 tasks, silent errors fall from 405 for DS-STAR to 289 for RADAR (28.6%); DA-Code and DABStep margins are 6.5 and 19.5 points, respectively.Source: §1 · pp. 2–3

2 Evidence-Preserving Exploration

• Given a natural-language query, RADAR returns retrieved evidence Eq and selected source scope Xq to initialize analysis.Evidence records observed content; source scope retains files for further analysis, and later operations can add observations.Source: §2 · p. 3, §2.2 · p. 4

2.1 Source Map Construction

• Source Cards record each source unit’s path and position, observed structure and representative content, and whether inspection was complete or budget-limited.Units such as workbook sheets are profiled separately; related files or units can be grouped for retrieval and selection.Source: §2.1 · p. 3

2.2 Query-Aware Exploration

• BM25 ranks source groups for query and evidence-need views, and reciprocal-rank contributions across fields prioritize groups without comparing field-specific BM25 scores directly.The LLM decomposes the query into evidence needs, and the rank contributions use an offset κ.Source: §2.2 · p. 4
• The selector chooses candidate groups using paths, observed fields and values, previews, and matched sections; RADAR then retrieves relevant records or passages with source locations.Inspection reports identify presented portions, inspected sources, and content omitted because of inspection limits.Source: §2.2 · p. 4

3.1 Typed Operators and Execution Records

• Typed operators expose analytical choices, while execution records link each invocation’s arguments and declared inputs to its resulting observations.The eight operator types cover source discovery, data access, inspection, computation, and answer submission; a Query declares its input tables and names its output.Source: §3.1 · p. 4, Table 1 · p. 5

3.2 Validation Checks

• Checks compare operation arguments with observations of referenced inputs, and are skipped when the SQL structure or required evidence is unsupported.If no check rejects the invocation, it proceeds to execution; the agent remains responsible for deciding whether the analysis answers the question.Source: §3.2 · p. 5
• Validation checks reject unsupported field references and unkeyed joins between multirow inputs, and answer checks verify declared result types and applicable artifact requirements.A literal filter is rejected for an absent value only when Probe supplies a complete enumeration; an omission from a partial preview is insufficient evidence.Source: §3.2 · p. 5

3.3 Diagnostic Feedback and Bounded Repair

• After rejection or suspicious execution, RADAR returns supporting observations and revision hints; bounded internal repair can address SQL execution failures and failed or suspicious data loads.A successful empty-result query may receive diagnostics, but the agent decides whether the result is valid or should be revised; SQL repair must preserve analytical intent and data sources.Source: §3.3 · pp. 5–6

4.1 Experimental Setup

• Evaluation spans three benchmarks and six agent baselines, using shared raw files, a common model at temperature 0, and up to 25 decision rounds.KramaBench includes Full Input and Oracle Input settings; DA-Code and DABStep cover 500 and 450 tasks, respectively.Source: §4.1 · p. 6

4.2 Main Results

• RADAR achieves the highest Overall scores across all three benchmarks; on DABStep Hard tasks, it scores 47.1% versus 22.8% for DS-STAR.Hard tasks account for 84% of DABStep, outweighing a 5.5-point decrease on Easy tasks.Source: §4.2 · p. 6

4.5 Efficiency and Backbone Robustness

• On KramaBench, RADAR costs less and runs faster than setting-specific strongest baselines; it also outperforms baselines with GPT-5.6 Luna (0.738) and GPT-4.1-mini (0.460).Table 2 reports end-to-end scores and resource use across backbones.Source: Table 2 · p. 7, §4.5 · p. 8

4.3 Effect on Silent Errors

• Across 1,054 tasks, RADAR reduces silent errors from 405 to 289 (28.6%) and non-deliveries from 84 to 24; its combined error rate is 29.7%, versus 46.4%–83.0% for baselines.Figure 3 compares non-delivery and silent errors across systems on each benchmark.Source: §4.3 · p. 6, Figure 3 · p. 7
• Among substantive deliveries, silent-error rates fall from 41.8% for DS-STAR to 28.1% for RADAR; transformation errors fall from 12.4% to 3.7%.Scope or selection remains the most frequent category, decreasing from 17.6% to 15.6%.Source: §4.3 · p. 6, Figure 4 · p. 8

4.4 Component Analysis

• Removing evidence-preserving exploration has the largest performance effect on KramaBench, while removing runtime validation causes the largest decrease on DABStep; across 1,054 tasks, silent errors rise from 289 to 409 and 377, respectively.Source: §4.4 · p. 8

Data-science agents

• Data-science agents address end-to-end analysis, heterogeneous data, visualization, insight discovery, and general workflows, while fine-tuned methods may require retraining or may not apply to proprietary APIs.These adaptation constraints motivate RADAR’s training-free design.Source: §5 · pp. 8–9
• File analyzers and partition agents can leave later choices without source location, scope, or coverage; RADAR preserves that context in evidence.Source: §5 · p. 9

Runtimes for AI-driven analytics

• Analytics runtimes use explicit operators for pipeline construction and execution, while RADAR’s typed operators expose agent-selected operations, dependencies, and intermediate states for validation.Source: §5 · p. 9

• RADAR preserves source locations, observation scope, and coverage, linking typed operations to runtime observations so conflicts can prompt diagnostic feedback and revision.Source: §6 · p. 9
• Across KramaBench, DA-Code, and DABStep, RADAR achieves the highest Overall scores among evaluated agents and 28.6% fewer observed silent-error cases than DS-STAR.Source: §6 · p. 9
• Runtime validation remains limited by available evidence; broader validation coverage and cost-based optimization are future directions.Source: §6 · p. 9