PaperlayerView exact revision on arXiv

Source-linked AI summary

Data World: Scaling Data Environments from Scratch

Zhanli Li, Zeyuan Xu, Chaoxu Pang, Zeyuan Liu, Anyang Ji, Te Sun, Kongcheng Zhang, Wenbo Xu, Xubin Yue, Shipeng Luo, Junjie Zhao, Yingfan Hua, Jingqing Ruan, Zhongda Su, Feng Hong, Zhizhao Zeng, Ke Zeng, Xunliang Cai, Hanjia Ye

arXiv:2610.21656v1cs.AI

TL;DR

Data-agent training is difficult to scale when real workflows are costly or restricted and open-ended findings lack known generative truth for verification. Data World specifies mechanisms before generating files, then builds audited tasks and rubrics; frontier-model evaluation reveals a computation–insight gap, and RL-zero training improves the base model across held-out and external benchmarks.

  • Problem

    Data-agent training needs diverse environments and feedback, but real workflows are costly to acquire, privacy- and IP-restricted, and difficult to verify without the underlying data-generating process.

    Source: §1 · pp. 1–2

  • Method

    Data World specifies generative mechanisms, synthesizes canonical states and analyst-visible files, audits target recoverability, and grounds tasks and rubrics in the generative record.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Across ten frontier models, DataWorld-Bench reveals a computation–insight gap, while RL-zero training yields substantial gains over the base on held-out worlds and three external benchmarks.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    Mechanism-grounded synthesis expands task supply without requiring curators to rediscover each targeted finding from generated files, providing evaluation criteria and training signals as a complement to real-world data.

    Source: §5 · pp. 8–9

    Human preference-based realism curation does not establish that synthetic tasks represent real user requests, particularly their ambiguity, implicit context, and evolving goals.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Data agents use code to process heterogeneous files and derive analytical insights. Scaling their training faces costly environment curation, privacy and intellectual-property restrictions, and difficulty verifying open-ended findings without known generative truth. Our core insight is to specify the world behind the files before generating the files themselves. We introduce Data World, a multi-agent engine that synthesizes environments, tasks, and rubrics from explicit mechanisms. Independent audits check target recoverability from observed files; rubrics support evaluation and RL rewards. Human preference-based curation for question realism selects 100 tasks across 50 held-out worlds and 16 domains for DataWorld-Bench. The engine also yields 2,135 training tasks from 1,037 disjoint worlds. Evaluation of ten frontier models shows that the benchmark remains challenging, particularly for insight discovery, with a consistent computation-insight gap. RL-zero training on the synthesized tasks produces DataWorld-35B-A3B from Qwen3.6-35B-A3B without trajectory distillation. Under matched evaluation, it improves over its base by 7.06 points on DataWorld-Bench, 14.28 on DAComp, 20.90 on DABstep-Hard, and 12.13 on JobBench, a broader occupational benchmark.

1 Introduction

Data World addresses the cost and verification challenges of scaling data-agent training by specifying the mechanisms behind files before generating observations. Its audited environments ground analytical tasks and rubrics, supporting both benchmark evaluation and RL training.

1 Introduction

  • Scaling data-agent training is constrained by costly real-workflow acquisition, privacy and intellectual-property restrictions, and difficulty verifying open-ended findings without known generative processes.Exploration-based rubrics can omit valid explanations that were not discovered during curation.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • Data World specifies mechanisms before generating files, then uses a canonical generative state and independent audits to check whether target phenomena are recoverable from imperfect observations.Rubrics grounded in the generative record support evaluation and training rewards.Source: §1 · p. 2
  • The engine produces DataWorld-Bench and a world-disjoint training set, and fine-grained rubrics reveal a computation–insight gap across ten frontier models.The benchmark contains 100 tasks from 50 held-out worlds across 16 domains; the training set contains 2,135 tasks.Source: §1 · p. 2
  • RL-zero training on synthesized tasks, without stronger-model trajectory distillation, yields substantial gains over the base model on held-out worlds and three external benchmarks.The resulting model is DataWorld-35B-A3B.Source: §1 · p. 2

2 Related Work

Prior work spans data-agent environments, insight-discovery benchmarks, task synthesis, rubric-based rewards, and synthetic environments. Data World differs by grounding analytical targets in specified generative relationships across imperfect, interdependent files.

2 Related Work

  • Data-agent environments draw on competitions, scientific and government datasets, and enterprise records; benchmarks cover text-to-SQL, code-driven analysis, and spreadsheet workflows.The related work also includes multi-table government and enterprise spreadsheet settings.Source: §2 · pp. 2–3
  • Insight-discovery benchmarks score extracted insights, but often use available-source environments and references from expert annotation or model-based exploration.Data World complements them with mechanism-grounded criteria and audits of target recoverability.Source: §2 · p. 3
  • Task-synthesis work explores real data to propose tasks or mine insights, while rubric-derived rewards use LLM judges for open-ended RL.Data World instead grounds rubrics in a generative record and independent audit results, while retaining possible biases in mechanism design, rubric selection, and judging.Source: §2 · p. 3
  • Synthetic-environment pipelines build simulated tools, distill environments from trajectories, or assemble them from web-found interfaces; synthetic-data methods also learn from existing databases or simulate records.Data World targets terminal-based analysis of interdependent files and audits analytical findings through specified generative relationships.Source: §2 · p. 3

3 Method

Data World specifies generative mechanisms before producing files, then synthesizes and audits worlds before generating tasks. Its pipeline links world specification, environment generation, auditing, task synthesis, and agent evaluation.

3 Method

  • Data World connects mechanism-based environment synthesis to checkable targets and agent evaluation using observed files.Figure 2 depicts the pipeline from world specification and generation through auditing, task synthesis, and agent evaluation.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Formalization and Overview

  • A synthesized world is formalized as a dependency graph, canonical state, analyst-visible data environment, and set of recomputable phenomena.The pipeline synthesizes and audits worlds before generating tasks; a separate curation step selects benchmark examples.Source: §3.1 · p. 3

3.2 Environment Synthesis

  • A world contract fixes entities, mechanisms, invariants, observation systems, and deliverables before layer-wise state generation and projection into imperfect observed files.An independent auditor checks edge relationships, invariants, and whether specified phenomena remain recoverable from the observed environment.Source: §3.2 · pp. 4–5

3.3 Task Synthesis

  • Tasks target nodes along cause routes through incoming edges or consequence routes through outgoing edges, with depth-stratified maximal-degree targets supporting structural coverage.Each task-rubric pair is grounded in the generative record and distinguishes data computation from insight discovery; evaluators score reports item by item.Source: §3.3 · p. 5

3.4 Benchmark and Training-Set Curation

  • 1,087 of 1,302 candidate worlds were retained, yielding 2,135 RL tasks from 1,037 training worlds and 100 test tasks from 50 held-out worlds across 16 domains.Test worlds are excluded from training and must pass every audit check; training worlds have a 95.98% DAG-edge pass rate.Source: §3.4 · p. 5

4 Experiments

The experiments test what DataWorld-Bench reveals about frontier agents and whether RL on synthesized tasks improves performance on held-out worlds and external benchmarks. DataWorld-35B-A3B improves over its base across all four benchmarks, while the frontier-model results show a computation–insight gap.

4.1 Experimental Setup

  • The evaluation uses a shared mini-swe-agent harness and rubric judge to assess benchmark performance; the experiments examine both existing agents and training on separate synthesized worlds.The study asks whether DataWorld-Bench reveals computational and insight-discovery capabilities and whether training transfers to held-out worlds and external benchmarks.Source: §4.1 · p. 6, §4 · p. 6

4.2 DataWorld-Bench Results

  • 74.43 overall and 67.95 in insight discovery are the best scores, and every model shows a 4.87–21.35-point computation–insight gap.The authors caution that criterion coverage and scoring differences may contribute to the rubric-defined gap.Source: §4.2 · p. 6
  • Interaction cost does not uniformly predict scores, and negative rubric flags distinguish incorrect work from successful recovery of targeted findings.Longer interaction alone is insufficient for stronger analysis; cross-model comparisons do not isolate additional inference effects.Source: §4.2 · p. 6

4.3 Validating Training Utility

  • RL-zero training uses the model’s own rollouts and rubric rewards, without task-solving demonstrations or stronger-model trajectory distillation.The base model itself serves as the rubric judge to reduce reward-scoring costs.Source: §4.3 · p. 7
  • 7.06 points on DataWorld-Bench, 14.28 on DAComp, 20.90 on DABstep-Hard, and 12.13 on JobBench: DataWorld-35B-A3B gains over its base across all four benchmarks.The comparisons use matched evaluation; JobBench covers 35 occupations and is the least task-aligned suite.Source: §4.3 · p. 8, Table 3 · p. 8
  • MMLU-Pro accuracy remains 84.73%, LiveCodeBench-Hard accuracy falls from 71.14% to 70.29%, and SimpleQA-Verified F1 rises from 22.20 to 24.00.The paper interprets these checks as preserving general language and coding capabilities without noticeably disturbing factual world knowledge.Source: §4.3 · p. 8, Table 4 · p. 8

5 Conclusion

Data World separates canonical generative truth from analyst-visible files to provide scalable, mechanism-grounded supervision. Its training and evaluation results extend across unseen worlds, external analysis tasks, and occupational workflows.

5 Conclusion

  • Gains extend to unseen worlds, external data-analysis tasks, and broader occupational workflows, without task-solving demonstrations from stronger models.The framework is presented as a scalable complement to real-world data, providing evaluation criteria and training signals.Source: §5 · pp. 8–9
  • Data World specifies generative mechanisms before synthesizing observations and audits whether analytical targets can be recovered from them.Separating canonical truth from analyst-visible files grounds supervision in known mechanisms while agents recover findings from observations alone.Source: §5 · p. 8
  • The engine yields DataWorld-Bench for measuring the computation–insight gap and a world-disjoint task set for RL-zero training.The joint synthesis expands task supply without requiring curators to rediscover each targeted finding from generated files.Source: §5 · pp. 8–9

Limitations

The authors identify limits in component-level attribution, the realism of synthetic tasks as proxies for user requests, and the coverage and consistency of rubric-based grading.

Limitations

  • The synthesis components are tightly coupled and regenerating worlds and tasks is costly, limiting exhaustive ablations and controlled pipeline variants.The results therefore support the framework as a whole rather than isolate each component; the paper also does not compare synthesized with real-world data for effectiveness.Source: Limitations · p. 9
  • Human preference-based realism curation does not establish that synthetic tasks represent real user requests, especially their ambiguity, implicit context, and evolving goals.Source: Limitations · p. 9
  • Mechanism-grounded rubrics may omit valid analytical findings, while LLM-based grading may introduce judge bias and inconsistent scores across judging models.Audited generative truth verifies specified criteria but does not guarantee appropriate credit for alternative insights.Source: Limitations · p. 9

Ethical Considerations

The authors caution that audits and human curation do not establish suitability for real-world decisions. They call for content review before release and independent validation and human oversight in consequential domains.

Ethical Considerations

  • Page constraints and limited knowledge led the authors to cite and discuss only the closest related work known to them.Source: Ethical Considerations · p. 9
  • Audits verify specified relationships and recoverability, while human curation assesses question realism; neither establishes suitability for real-world decisions.Source: Ethical Considerations · p. 9
  • Outputs should still be reviewed for sensitive or proprietary content before release, and consequential-domain use requires independent validation and human oversight.Source: Ethical Considerations · p. 9

A Appendix

The appendix outlines theoretical analysis, model and benchmark details, RL training configuration, and the evaluation protocol.

A Appendix

  • The appendix provides an intuitive theoretical analysis of rubric coverage.Source: A Appendix · p. 12
  • It introduces the evaluated models and describes training and test distributions alongside additional benchmark analyses.Source: A Appendix · p. 12
  • It documents the RL training configuration and evaluation protocol.Source: A Appendix · p. 12

A.1 Intuitive Theoretical Analysis

The analysis compares valid findings recoverable from observed files with those covered by rubric items, including rubrics written with access to Data World’s generative record. Specification access can improve coverage under an idealized matched-budget assumption, but hidden-state facts still require recoverability checks.

A.1 Intuitive Theoretical Analysis

  • The overlap |S ∩ R| counts valid recoverable findings whose meanings match rubric criteria, excluding negative penalty items.Robs uses observed files alone; RDW also uses the Data World generative record.Source: §A.1 · p. 12
  • Specification access can strictly improve valid-finding coverage within the same rubric-item budget, but does not guarantee better actual LLM-generated rubrics.The idealized writer can reproduce the file-only strategy, ignore unhelpful information, and choose the strategy with greatest valid-finding coverage.Source: §A.1 · pp. 12–13
  • A finding known only from the hidden state is not necessarily recoverable from observed files and therefore may fall outside the valid-finding set.The question-writing agent can use the original world specification, generated state, and audit artifacts to identify candidate findings and verification evidence.Source: §A.1 · pp. 12–13

A.2 Evaluated Models

The section introduces ten evaluated models and says their descriptions draw on public technical reports, model cards, and system cards. All use the same lightweight bash-only mini-swe-agent harness, which alternates model reasoning, shell execution, and command feedback.

A.2 Evaluated Models

  • All ten models use a bash-only mini-swe-agent harness with a ReAct-style loop of reasoning, shell commands, and command-output feedback.The model descriptions summarize public technical reports, model cards, and system cards.Source: §A.2 · p. 13
  • DeepSeek-V4.1-Flash is described as a 552B Causal Encoder-Decoder with 8B prefill and 16B decode activated parameters and a 1M-token context.Its global always-in-HBM KV cache is reported at 890 bytes per token, about 4× smaller than DeepSeek-V4-Flash and 437× smaller than DeepSeek-V1.Source: §A.2 · p. 13
  • Gemini-3.8-Flash is reported to score 90.8% on Terminal-Bench 2.1, versus 81.6% for its 3.7 Flash predecessor.The passage also reports a lead on Harvey’s Legal Agent Benchmark and notes that no formal technical report has been published.Source: §A.2 · p. 13
  • The listed model set includes Qwen-3.8-27B, MiniMax-M3, GPT-6-Astra, GPT-6.1-Sol, Qwen-3.8-Max-0902, and Kimi-K3, described variously as multimodal, reasoning-focused, efficiency-oriented, or open-weight models.Source: §A.2 · pp. 13–14

A.3.1 Training and Test Distributions

The appendix describes task distributions and case studies, then reports per-domain and cause-versus-consequence analyses. It also details training and paired evaluation procedures, including the scope and scoring of the external benchmarks.

A.3.1 Training and Test Distributions

  • The held-out test split covers taxonomy categories approximately uniformly, while the training set is more unevenly distributed.Figure 3 normalizes percentages within each split and describes coverage, not task difficulty.Source: §A.3.1 · p. 14

A.3.2 Task Case Studies

  • Three task case studies illustrate heterogeneous files and analytical routes, including public-safety deployment, class-capacity planning, and production yield with downstream availability.The case studies are task inputs, not model solutions or evidence of successful completion.Source: §A.3.2 · pp. 14–15

A.3.3 Per-Domain Results

  • Bioinformatics & omics (+7.70), operations, supply chain & manufacturing (+7.25), and energy, utilities & IoT (+6.21) have the largest positive domain offsets; transportation & mobility (−11.44), environment, climate & geospatial (−8.50), and education & learning analytics (−6.22) have the largest negative offsets.Offsets remove each model’s mean before averaging across ten models; domains contain only 6–8 tasks, so they indicate broad patterns rather than reliable within-domain rankings.Source: §A.3.3 · p. 16, Table 5 · p. 16

A.3.4 Cause versus Consequence Tasks

  • No model’s cause-versus-consequence score difference reaches the stated minimum detectable difference, so the table provides no evidence of a systematic route difficulty difference.The per-model differences range from −3.28 to +5.03 points; this does not establish equal difficulty or rank models.Source: §A.3.4 · p. 16, Table 6 · p. 16

A.4.1 Training Data and Software

  • DataWorld-35B-A3B uses 2,135 tasks from 1,037 worlds disjoint from test worlds and is trained from the Qwen3.6-35B-A3B base without task-specific supervised fine-tuning.Training uses GSPO with group size eight and rubric-derived rewards scored by the base model itself; the rollout stack and evaluation harness are distinct.Source: §A.4.1 · p. 16, §A.4.3 · p. 17

A.5 RL Evaluation Protocol

  • The paired evaluation compares DataWorld-35B-A3B with its base on four benchmarks probing professional reasoning, open-ended data science, verifiable tabular reasoning, and analysis in executable worlds.The benchmarks are JobBench, DAComp, DABstep-Hard, and DataWorld-Bench.Source: §A.5 · p. 17

A.5.1 Shared Evaluation and Scoring

  • Table 3 comparisons share a mini-swe-agent harness, budget chain, and scoring procedure, while benchmark scoring differs between rubric-judged scores and DABstep-Hard’s binary correctness checks.DAComp and DataWorld-Bench use a single Gemini-3.8-Flash judging pass; DataWorld-Bench normalizes net rubric points to [0, 1], while DAComp retains native scoring.Source: §A.5.1 · p. 17

A.5.2 Paired Statistical Analysis

  • Paired analysis matches tasks by identical name and covers 200 DAComp, 378 DABstep-Hard, and 100 DataWorld-Bench pairs, with no dropped tasks.The reported MDE is calculated from per-task differences; JobBench’s 65-task main set spans 35 occupations, and aggregate gains do not establish improvement within every occupation.Source: §A.5.2 · p. 17, §A.5.3 · p. 17

Related papers

Loading 2610.21656v1…