Source-linked AI summary
DataFlow-Harness: A Grounded Code-Agent Platform for Constructing Editable LLM Data Pipelines
Runming He, Zhen Hao Wong, Hao Liang, Zimo Meng, Chengyu Shen, Xiaochen Ma, Wentao Zhang
TL;DR
Data-processing coding agents often produce disposable scripts instead of persistent, editable platform workflows. DataFlow-Harness grounds agents in live platform state while incrementally constructing visual DAGs, achieving pass rates close to script-generation baselines with lower measured cost and latency.
Problem
Production workflows require persistent, visualizable, editable, reusable pipeline assets grounded in platform semantics rather than disposable generated scripts.
Method
DataFlow-Harness guides agents to construct platform-native workflows using procedural Skills, live MCP grounding, and synchronized conversational and visual authoring.
Results
Across a 12-task benchmark, DataFlow-Harness achieves pass rates close to script-generation baselines while measured construction cost and latency are lower.
Takeaways & Limitations
Live platform grounding can produce governable, persistent workflow artifacts with observed reliability close to script-generation baselines and lower measured construction cost and latency.
Takeaways & Limitations
Procedural guidance does not resolve downstream numerical-constraint failures and may reduce flexibility when multiple execution strategies are valid.
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly used to automate data-processing workflows, yet coding agents typically produce scripts that are not automatically materialized as persistent, editable platform artifacts. We call this disconnect the \textit{NL2Pipeline gap}. To bridge it, we introduce \textsc{DataFlow-Harness}, a platform that guides an LLM agent to construct platform-native directed acyclic graphs (DAGs) through typed, incremental mutations rather than free-form scripts. The platform combines \textsc{DataFlow-Skills} for procedural guidance, a Model Context Protocol (MCP) layer that exposes the live operator registry and current pipeline state, and \textsc{DataFlow-WebUI}, which synchronizes conversational authoring with a visual DAG editor. On a 12-task data-engineering benchmark, \textsc{DataFlow-Harness} achieves a 93.3\% observed end-to-end pass rate. Relative to Vanilla Claude Code, it reduces measured monetary cost by 72.5\% and generation latency by 49.9\%; its observed pass rate is within 0.9 percentage points of the Context-Aware Claude Code baseline while its cost is 42.8\% lower. Per-task analysis indicates that Skills are most useful when construction depends on implicit procedural knowledge. These results show that live platform grounding can produce persistent, editable workflow artifacts with an observed reliability close to script-generation baselines and with lower measured construction cost and latency.
1 Introduction
DataFlow-Harness addresses the NL2Pipeline gap by guiding agents to construct persistent, platform-native workflow artifacts rather than disposable scripts. It combines procedural Skills, live MCP grounding, typed incremental mutations, validation, and synchronized conversational and visual authoring, achieving pass rates close to script-generation baselines with lower measured token usage, cost, and latency.
- Approach: DataFlow-Harness guides agents to construct platform-native workflows through typed, incremental mutations instead of directly generating scripts.This design targets disposable code outputs that are difficult to audit graphically and may hallucinate dependencies.
- Approach: Its three components are DataFlow-Skills, live MCP access to the operator registry and workflow state, and synchronized conversational and visual authoring.Skills encode operator-selection patterns, schema dependencies, and assembly procedures.
- Results: The observed pass rate is close to script-generation baselines, while measured token usage, cost, and latency are lower.The evaluation also includes a per-task Skills ablation and two controlled case studies of downstream training utility.
- Problem: The NL2Pipeline gap is the disconnect between natural-language workflow intent and persistent, platform-native artifacts that remain inspectable and editable.Production workflows must also be reusable and compatible with platform governance mechanisms.
2 Related Work
Prior work spans code-synthesis agents, data-centric and composable data systems, and structured workflow generation. DataFlow-Harness builds on these directions by targeting interactive, stateful, agent-assisted construction of editable pipelines within a live data-engineering platform.
- Code synthesis and agent systems: Code-synthesis research progressed from pretrained models to feedback-driven autonomous agents and repository-level editing, while MCP unified agent tool interfaces.Examples include Codex, StarCoder, Reflexion, Self-Debug, SWE-agent, and Claude Code.
- Data-centric and composable systems: Data-centric AI systems operationalize governance and large-scale curation, while DataFlow and DSPy represent LLM operations as composable components in formal execution graphs.Data-Juicer and DCLM exemplify extensible operator-based curation systems.
- DataFlow-Harness distinction: DataFlow-Harness builds on DataFlow by focusing on agent-assisted pipeline construction rather than pipeline execution.Its architecture synchronizes a shared pipeline representation across the agent runtime and DataFlow-WebUI, with Skills guiding construction and validation checking DAG structure and schema compatibility.
- Structured workflow generation: Structured-workflow systems generate reusable workflows or validated workflow DAGs and encode domain knowledge as reusable Skills.AutoFlow and Balis et al. establish the value of structured workflow generation.
- DataFlow-Harness distinction: DataFlow-Harness addresses the complementary problem of interactive, stateful authoring inside a live data-engineering platform.This positions the work beyond standalone structured-workflow generation toward platform-integrated pipeline editing.
3 System Architecture
DataFlow-Harness organizes workflow synthesis around four components, with the Data Pipeline Backend as the authoritative state store and typed MCP-mediated mutations connecting agent reasoning to persistent pipeline artifacts. Its synchronized conversational and visual interfaces, structural validation, and procedural DataFlow-Skills jointly support grounded workflow construction.
- Four components organize workflow synthesis: DataFlow-WebUI, the MCP Tools Layer, the Data Pipeline Backend, and DataFlow-Skills.
- The Data Pipeline Backend is the authoritative source of truth, receiving validated MCP mutations and synchronizing state across conversational, visual, and programmatic interfaces.DataFlow-Skills shapes agent reasoning without directly modifying pipeline state.
- Typed mutations add or remove operators, update parameters, and connect edges; commits require an acyclic graph and compatible adjacent schemas, but these checks do not ensure semantic correctness.
- DataFlow-WebUI synchronizes natural-language authoring with a visual DAG editor, allowing users to inspect, edit, relink, or reconfigure workflows while manual changes immediately update backend state.
- DataFlow-Skills supplies procedural blueprints and compositional constraints covering schema inference, operator selection, parameter configuration, serving verification, and compatibility conventions.The MCP Tools Layer exposes live operator metadata and workflow state, while Skills provide recommended construction strategies.
4 Experiments
DataFlow-Harness produces platform-native DAGs with reliability close to script-generation baselines while substantially reducing construction cost and latency. Its procedural guidance is most beneficial for ambiguous, knowledge-intensive workflows and can improve downstream data utility.
- Workflow synthesis effectiveness: 93.3% end-to-end success is within 0.9 percentage points of Context-Aware CC and 10.0 percentage points above MCP-only.The results indicate reliability approaching script-generation baselines without claiming statistical equivalence.
- Efficiency: 72.5% lower monetary cost and 49.9% lower generation latency versus Vanilla CC accompany a higher end-to-end success rate.Cost falls from $0.950 to $0.261, while latency falls from 190.7s to 95.5s.
- Efficiency: 42.8% lower cost and 17.6% lower latency versus Context-Aware CC yield nearly identical end-to-end performance.Lower token consumption drives these efficiency gains: DataFlow-Harness reduces total token consumption by 25.5% compared with MCP-only.
- Skills ablation: Procedural guidance most improves QA-generation and language-processing tasks, where MCP-only struggles to infer task-specific procedures from operator descriptions.For simple transformation and filtering tasks, both methods achieve perfect success, indicating limited benefit when routing is straightforward.
- Skills ablation: Procedural guidance does not resolve failures caused by downstream numerical constraints, and it can reduce flexibility when multiple execution strategies are valid.Both methods exhibit the same failure rate on multi-field scoring tasks; differences on tasks 2a, 2b, and 3a are small and occasionally favor MCP-only.
- End-to-end outcome evaluation: 51.6 versus 49.9 average accuracy after one epoch and 55.7 versus 54.5 after two show higher downstream utility from DataFlow-Harness-generated data.At one epoch, AIME24@32 improves from 25.1 to 35.9 and AIME25@32 from 21.6 to 34.5.
5 Conclusion
DataFlow-Harness addresses the NL2Pipeline gap by combining procedural Skills, live MCP grounding, typed mutations, structural validation, and synchronized conversational and visual editing. On a 12-task benchmark, it achieves pass rates close to script-generation baselines with lower measured construction cost and latency, while the evaluation leaves important limitations and broader workflow-governance claims unresolved.
- Contribution: DataFlow-Harness addresses the NL2Pipeline gap through Skills, live MCP grounding, typed mutations, structural validation, and synchronized conversational and visual editing.These mechanisms support construction of platform-native workflow artifacts rather than only free-form scripts.
- Results: On a 12-task benchmark, the platform’s observed end-to-end pass rate is close to script-generation baselines, with lower measured construction cost and latency.The per-task ablation also identifies where procedural guidance is most useful.
- Limitations: The evaluation covers one coding-agent and model family and a relatively small, platform-specific benchmark.The available ablation does not isolate every component, and schema validation cannot guarantee semantic correctness.
- Limitations: Broader workflow-governance claims require direct evaluation of persistence, reuse, provenance, concurrent editing, and recovery.The downstream-utility results cover two case studies without multiple independently authored pipelines and training seeds.