PaperlayerView exact revision on arXiv

Source-linked AI summary

DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces

Boyan Li, Zhuowen Liang, Yupeng Xie, Xiaotian Lin, Tianqi Luo, Xinyu Liu, Yizhang Zhu, Zhangyang Peng, Yuan Li, Zhengxuan Zhang, Jiayi Zhang, Nan Tang, Guoliang Li, Yuyu Luo

arXiv:2608.03451v1cs.AI

TL;DR

DataSpace addresses the lack of a unified benchmark for discovering and reconciling heterogeneous evidence while producing complete, verifiable tabular answers. It constructs such tasks with DataSpace-Builder and deterministic evaluation, finding that the best accuracy is 66.34% and that multimodal integration and joins remain bottlenecks.

  • Problem

    Existing benchmarks do not jointly evaluate heterogeneous evidence discovery, complete tabular outputs, and deterministic evaluation for workspace analytics.

  • Method

    DataSpace-Builder creates cross-language heterogeneous workspaces from executable Text-to-SQL resources with expert review, paired with semantics-aware deterministic tabular-result evaluation.

  • Results

    The best controlled backbone reaches 66.34% Task Accuracy, while harness choice spans 30.98% to 46.34% and multimodal tasks and joins consistently reduce accuracy.

  • Takeaways & Limitations

    DataSpace remains unsaturated and identifies multimodal evidence integration, joins, and exact output materialization as key reliability challenges.

Abstract

from arXiv · show

Data agents enable natural-language analytics over organizational workspaces, where relevant evidence may be scattered across databases, structured files, long documents, and multimedia. Existing benchmarks largely isolate structured querying, retrieval, or open-ended analysis, leaving heterogeneous evidence discovery, complete tabular outputs, and deterministic evaluation insufficiently unified. We introduce DataSpace, a benchmark in which data agents produce verifiable tabular results from task-local heterogeneous workspaces. It contains 410 cross-language tasks and 7,439 artifacts totaling 15.01 GB across CSV, JSON, SQLite, Markdown, PDF, and video. DataSpace also served as the official evaluation benchmark for the KDD Cup 2026 Data Agents for Complex Data Analysis competition. Each agent receives only a question and workspace and returns the complete requested tabular result. We construct DataSpace with DataSpace-Builder, an execution-grounded framework comprising cross-language transformation, constraint-aware relational sampling, modality routing and artifact rendering, and human review and task repair by 11 domain experts. A deterministic evaluator performs header-invariant column alignment, type- and precision-aware normalization, and order-aware row comparison. Across six recently released frontier multimodal models and five widely used agent harnesses, the best accuracy reaches 66.34%, while harness choice creates a 15.36-point spread with the backbone fixed. Multimodal evidence integration and joins consistently reduce accuracy across all six backbones. These results show that DataSpace remains unsaturated and identify key challenges for improving data-agent reliability.

1 Introduction

DataSpace targets verifiable analytics when evidence is distributed across heterogeneous, multilingual workspaces rather than a single clean table. It introduces an execution-grounded benchmark and evaluation framework for complete tabular answers, with empirical results showing substantial remaining difficulty.

  • Motivation: Data agents must discover, align, compute over, and combine evidence across databases, files, documents, media, and languages.The setting includes relevant and irrelevant artifacts, requiring workspace inspection and multi-step analysis.
  • Related gap: Existing benchmarks test structured querying or unstructured retrieval separately, without unifying heterogeneous evidence discovery, complete outputs, and deterministic evaluation.Structured benchmarks often preselect the database, while unstructured benchmarks emphasize retrieval, grounding, and synthesis.
  • Benchmark: DataSpace contains 410 cross-language tasks and 7,439 artifacts across CSV, JSON, SQLite, Markdown, PDF, and video, each paired with a complete tabular reference answer.Tasks use self-contained workspaces and may contain Chinese and English across questions and artifacts.
  • Construction: DataSpace-Builder transforms executable Text-to-SQL resources through cross-language transformation, relational sampling, modality routing, artifact rendering, and expert review.Eleven domain experts cross-review and repair disputed question, workspace, reference-answer, and evaluation-semantics cases.
  • Evaluation: The benchmark evaluates complete tabular results with deterministic semantics-aware comparison that tolerates equivalent representations while rejecting incomplete or erroneous outputs.The evaluator aligns columns despite header wording or order and normalizes values by type and precision.
  • Findings: The best agent reaches 66.34% accuracy, harness choice creates a 15.36-point spread, and multimodal evidence integration and joins consistently reduce performance.These results establish substantial room for improving reliable data agents over heterogeneous workspaces.

2 Related Work

Prior benchmarks provide strong but separated tests of structured reasoning, long-document understanding, and open-ended data analysis. DataSpace is positioned against this landscape by targeting evidence discovery and reconciliation across heterogeneous artifacts with complete, deterministic tabular evaluation.

  • Structured-data benchmarks: Structured-data benchmarks test table question answering and Text-to-SQL, often with deterministic result evaluation over known tables or databases.Spider, BIRD, EHRSQL, and BULL cover cross-domain, large-scale, clinical, and financial analytics.
  • Unstructured-data benchmarks: Unstructured-data benchmarks emphasize retrieval, grounding, and answer synthesis over multi-hop, retrieval-augmented, financial, or long-document inputs.They provide rich evidence settings but do not primarily evaluate full workspace-level tabular analytics.
  • Benchmark gap: Data-agent benchmarks move toward open-ended workspaces, but prior settings had not unified evidence discovery, cross-artifact integration, and complete tabular-result semantics.The comparison distinguishes artifact coverage, workspace demands, and answer/evaluation semantics.

3 Benchmark Overview and Task Formulation

DataSpace treats each task as workspace-level analytics: an agent receives a question and task-local workspace, interacts with heterogeneous artifacts, and must return the complete typed table. The benchmark spans diverse domains and languages while requiring discovery, alignment, computation, and exact result materialization.

  • 3 Benchmark Overview and Task Formulation: DataSpace frames an agent as a workspace solver that receives an analytical question and task-local workspace contents.Artifact contents are acquired through interaction rather than being supplied as a preselected input.
  • 3.1 Task Formulation: The workspace includes structured and semi-structured artifacts, with available tools covering inspection, parsing, SQL, code execution, document extraction, and video understanding.The agent selects tools through its policy and receives observations from tool execution.
  • 3.1 Task Formulation: The terminal action serializes the predicted tabular result as a CSV answer.The benchmark-side record separately stores the reference result and compact evaluation configuration.
  • 3.1 Task Formulation: A task is correct only when the complete prediction matches the reference under configured types, numeric rules, and row-order semantics.Missing or extra rows and columns make a prediction incorrect even if some returned values match.
  • 3.2 Benchmark Scope: The benchmark contains 410 tasks across financial, macroeconomic, and healthcare analytics, using six artifact formats in 13 modality combinations.Questions and artifacts may mix Chinese and English, and answer tables range from one cell to thousands of rows.
  • 3.2 Benchmark Scope: Every sampled source table remains represented, including records or fields encoded in long documents and conditions, intermediate values, or answers supplied by video.This preserves downstream analytical roles for non-tabular artifacts.
  • 3.2 Benchmark Scope: Tasks jointly exercise workspace discovery, semantic alignment, relational computation, and complete tabular-result materialization.The computation includes filtering, joining, aggregation, ranking, and temporal reasoning.

4 Benchmark Construction and Evaluation

DataSpace-Builder transforms executable Text-to-SQL resources into reviewed heterogeneous workspaces, while a deterministic evaluator verifies complete tabular outputs using task-specific semantics.

  • Construction: DataSpace-Builder uses four stages: Cross-Language Transformation, Constraint-Aware Relational Sampling, Modality Routing & Artifact Rendering, and Human Review & Task Repair.
  • Construction: Cross-language transformation jointly migrates the question, database state, and executable SQL, using consistency-aware replacement maps for linked schema and value groups.
  • Construction: Constraint-aware sampling creates task-local relational instances, executes the transformed SQL, and accepts samples only when relationships and results remain valid and nondegenerate.
  • Construction: Base routing renders sampled tables as CSV, JSON, SQLite, Markdown, or PDF, while video is added separately through query-conditioned evidence rendering.
  • Evaluation: Human review freezes task-specific column types, precision, units, null handling, and row-order requirements after independent verification by domain experts.
  • Evaluation: The evaluator aligns predicted columns independent of headers, normalizes values by configured semantics, preserves row associations, and compares ordered sequences or unordered multisets as required.

5 Benchmark Statistics and Analysis

DataSpace spans 410 tasks and diverse artifacts, with verified solution paths requiring selective evidence discovery and increasingly compositional analytical operations.

  • Evidence: CSV appears in every workspace, but verified solution paths use CSV in only 58 tasks and SQLite in 177, showing that availability does not imply necessity.
  • Evidence: Long documents provide required evidence in 135 tasks, while video is required in 97 of 189 video-enabled workspaces.
  • Operations: Projection and filtering are required most often, followed by ordering, aggregation, and joins across the benchmark.Their task-level prevalences are 338, 321, 209, 146, and 113 respectively.
  • Operations: Tasks combine two to eleven nontrivial operation types, with a median of five and 248 tasks combining at least five types.

6 Experiments

Controlled experiments show substantial variation across backbones and harnesses, with multimodal integration and joins consistently degrading task accuracy. Efficiency trade-offs and failure analysis further indicate that exact output materialization remains a central challenge.

  • Overall effectiveness: 15.36 points separate harness accuracies when MiMo-V2.5 is fixed, showing that harness design affects end-to-end task completion.Accuracy ranges from 30.98% for Smolagents to 46.34% for Grok Build.
  • Overall effectiveness: 66.34% is the highest backbone Task Accuracy, yet 76 tasks are missed by all six backbones.Grok 4.5 leads, while the oracle union solves 81.46% of tasks.
  • Efficiency and trade-offs: 74.2% fewer tokens, 50.3% fewer actions, and 39.2% less wall-clock time give GPT-5.6 Sol near-top accuracy despite trailing Grok 4.5 by 1.71 points.GPT-5.6 Sol reaches 64.63% versus Grok 4.5 at 66.34%; monetary efficiency follows a different frontier.
  • Performance across Task Characteristics: 1.8–14.0 points separate multimodal from single-modal tasks for every backbone, while joins reduce accuracy by 9.7–19.8 points.Language configuration and answer shape instead show backbone-dependent effects; the reported gaps are descriptive rather than causal.
  • Failure Analysis: 56.6% of Grok 4.5’s audited failures involve target-result misunderstanding or faulty column projection, whereas only 5 of 13 no-submission outcomes are pure termination failures.Answer materialization accounts for 71 of 136 failures, while only three failures arise from selecting the wrong evidence source.

7 Conclusion

DataSpace is a benchmark for verifiable analytics over cross-language, heterogeneous workspaces with complete tabular outputs. Its construction and evaluation framework expose persistent bottlenecks in multimodal evidence integration and joins, while the best accuracy reaches 66.34%.

  • DataSpace contains 410 tasks for verifiable analytics over cross-language, heterogeneous workspaces with complete tabular outputs.
  • DataSpace-Builder transforms executable Text-to-SQL instances into multimodal tasks with expert review and semantics-aware tabular-result evaluation.
  • 66.34% is the best accuracy across six backbones and five harnesses, while multimodal evidence integration and joins remain key bottlenecks.

A Additional Benchmark Details

The appendix specifies a deterministic, validated cross-language transformation that preserves database, workload, and question semantics. It uses clustered terminology mappings, protected-value handling, and bounded validation before producing accepted task artifacts.

  • Transformation contract: DataSpace-Builder transforms executable Text-to-SQL inputs into cross-language databases, questions, SQL, and a materialized replacement map.The transformation enforces referential, workload, execution, and question-fidelity invariants.
  • Mapping construction: Column clusters combine foreign-key links, shared names, and substantial value overlap to maintain consistent entity translations.Clusters merge only when overlap exceeds a threshold and includes at least a minimum number of distinct values.
  • Mapping construction: Protected values exclude nulls, identifiers, codes, URLs, dates, numbers, and other non-localizable forms from translation.Remaining values are deduplicated by cluster and divided into bounded prompt chunks while retaining provenance.
  • Mapping representation: The materialized map stores scoped table, column, and value mappings for deterministic downstream rewriting.Keys retain database, table, and column-cluster scope.
  • Validation: Validation checks mapping coverage, uniqueness, protected identities, and response completeness before assembly.Invalid or incomplete model outputs are retried with validator errors, while unresolved conflicts are repaired manually or rejected.

A.1.4 Deterministic Database Migration.

Deterministic migration rewrites database contents, metadata, SQL, and translated questions while preserving execution and analytical intent. Validation executes the rewritten workload and checks semantic alignment before acceptance.

  • Database migration: Database migration applies scoped value updates, column renames, table renames, and metadata rewrites inside a transaction.Dependent definitions and affected constraints are reconstructed against translated names before commit.
  • SQL rewriting: SQL rewriting resolves scopes and replaces tables, columns, and literals only in their typed contexts before reparsing.Protection prevents substitutions inside keywords, functions, constants, and partial identifiers; ambiguous references are marked for repair.
  • Question alignment: Question translation preserves entities, predicates, aggregation, ordering, limits, units, and temporal boundaries without modifying database or SQL artifacts.SQL provides construction-time alignment context rather than copied natural-language output.
  • Validation: Validation executes rewritten SQL on the translated database and compares its result with the translated source result while preserving multiplicity and specified order.Question validation separately checks answerability and analytical-intent preservation.
  • Acceptance: Failed artifacts are regenerated or repaired against the same map, while cases that cannot achieve execution alignment and semantic fidelity are discarded.Accepted artifacts retain transformed data, question, SQL, mappings, synchronized metadata, and validation status.

A.5 Human Review & Task Repair Details

Human review evaluates complete task versions for intent, evidence, answer uniqueness, gold correctness, and evaluation suitability. Independent review, evidence-based adjudication, and versioned repair determine which tasks are released.

  • Review scope: Experts review complete question–workspace–reference-result task versions after structural and artifact-level checks.The protocol determines whether each assembled task has clear intent, sufficient evidence, a unique answer, and suitable semantics.
  • Review staffing: Eleven domain experts review each task, with at least two reviewers assigned independently.Reviewer membership is fixed once an independent answer has been submitted.
  • Information boundaries: During independent solving, reviewers see the question and workspace but not source SQL, gold results, peer submissions, or discussion history.Peer judgments unlock only after both review stages are locked.
  • Review stages: Reviewers submit rectangular answers with evidence references, then verify gold results and independently configure column types, precision, units, and row-order requirements.The configuration is derived from the question and verified gold rather than model predictions.
  • Consensus: Consensus requires every reviewer to mark the gold as Match, select Pass, and submit the same canonical configuration.Disagreement over type, precision, or ordering triggers adjudication even when values match.
  • Repair and release: Adjudication cites conflicting workspace evidence and applies minimal, versioned repairs to affected task components.Administrators record final accept-or-remove decisions, while append-only audit trails preserve review and repair history.

A.6 Evaluation Protocol Details

The evaluation protocol freezes per-task semantics and compares complete tables after normalization. It supports header-invariant alignment, typed value comparison, configurable numeric precision and units, and optional row-order sensitivity.

  • Frozen configuration: Each task configuration records schema version, task identifier, ordering requirements, and one auditable entry for every reference column.Reference names support auditing but are never compared against prediction headers.
  • Frozen configuration: Task 120 normalizes region, period-end date, and per-capita GDP as text, datetime, and one-decimal-place number in order-sensitive rows.The example demonstrates per-column semantic typing and precision configuration.
  • Canonicalization: Text comparison trims whitespace and applies Unicode NFC normalization while remaining case sensitive; empty text cells become null.Null matches require null on both sides.
  • Canonicalization: Numbers use finite base-10 parsing, with integer, decimal-place, or significant-digit modes compared after round-half-up normalization.Canonical decimal representations must match after the configured precision is applied.
  • Units: Configured numeric units distinguish plain values, percentage points, and fractions, with percent signs accepted only when enabled.Under percentage_points, 3.5% denotes 3.5; under fraction, it denotes 0.035.
  • Dates and Booleans: Dates and datetimes use ISO syntax, timezone-aware values convert to UTC, and Boolean tokens are normalized across English and Chinese forms.A date column accepts midnight datetimes but rejects nonmidnight times.

A.6.3 Joint Alignment and Tabular-Result Matching.

This section specifies deterministic matching of complete tabular predictions and documents the controlled comparison setting for models and harnesses. It also frames reproducibility and comparison boundaries for specialized systems.

  • Tabular-result matching: Predictions must match the reference shape, after which the evaluator tests one-to-one column mappings using reference-column canonicalization rules.Prediction headers are required for CSV validity but are excluded from scoring.
  • Tabular-result matching: Order-sensitive tasks require identical canonical row sequences, whereas order-insensitive tasks compare row multisets while preserving duplicate multiplicity.Rows are reassembled after column mapping before comparison.
  • Controlled comparisons: The harness comparison fixes the endpoint to xiaomi/mimo-v2.5 and maps all harness-exposed model roles to that same backbone with fallback disabled.This isolates harness differences while retaining each harness’s pinned implementation.
  • Agent harness: DataSpace-Agent alternates model responses with bash, image viewing, and answer-submission actions until completion or a resource limit.Its prompt supplies the task, tool interfaces, workspace location, and tabular-output contract without task-specific source selection or analytical operators.
  • Compatibility and reproducibility: Specialized systems are included only when reproducibly runnable over complete task directories and compatible with multimodal input and tabular-output requirements.Reimplementing closed systems or rewriting output paths would introduce choices absent from the originals and prevent faithful comparison.

A.8 Additional Statistics and Results

Additional analyses examine efficiency, task characteristics, and human-confirmed failure causes under the benchmark’s controlled evaluation setup. They also document the benchmark’s competition deployment and released evaluation materials.

  • Backbone efficiency: For five of six backbones, incorrect tasks consume 1.2–3.2 times as many mean tokens as correct tasks and require more tool actions.The reported efficiency statistics are per-task averages over all 410 tasks.
  • Backbone efficiency: GPT is the exception, showing nearly identical resource use for correct and incorrect tasks; longer exploration often reflects failure to converge rather than solved work.This interpretation is stated for the efficiency comparison.
  • Trace-level root-cause audit: The root-cause audit assigns one primary cause to each failed Grok 4.5 task, selecting the earliest persistent divergence that determines the result or prevents submission.Later-corrected exploratory errors are excluded as primary causes.
  • Trace-level root-cause audit: A human researcher reviewed every proposed cause for the 136 audited failures against the trace, workspace evidence, submitted prediction, reference, and verified solution.The audit used GPT-5.6 Sol to propose categories and counterfactual corrections before human confirmation or revision.
  • Competition deployment: DataSpace served as the official evaluation benchmark for the KDD Cup 2026 Data Agents for Complex Data Analysis competition.The paper’s release instead uses the finalized semantics-aware evaluation protocol described in Section 4.5.
Loading 2608.03451v1…