PaperlayerView exact revision on arXiv

Source-linked AI summary

kRAIG: A Natural Language-Driven Agent for Automated DataOps Pipeline Generation

Rohan Siva, Kai Cheung, Lichi Li, Ganesh Sundaram

arXiv:2603.20311v1cs.SEcs.AIcs.CL

TL;DR

Early machine-learning data preparation requires substantial manual effort, while natural-language pipeline generation must address underspecified intent. kRAIG combines intent clarification, retrieval-augmented pipeline generation, and validation; on ELT Bench, it improves extraction, loading, and transformation performance compared with established agents.

  • Problem

    Ingestion, transformation, and validation require substantial manual effort to build and maintain before model training.

    Source: §1 · p. 1

  • Method

    kRAIG translates natural-language requirements into data pipelines, using explicit intent clarification and retrieval-augmented examples to guide generation.

    Source: §1 · p. 1, §2 · p. 2, Conclusion · p. 7

  • Results

    On ELT Bench, kRAIG significantly improves extraction, loading, and transformation performance compared with established agents.

    Source: §6.4 · p. 7

  • Takeaways & Limitations

    Explicit questioning reduces task ambiguity and prevents cascading failures caused by underspecified instructions.

    Source: Conclusion · p. 7

    A comprehensive evaluation across the full ELT-Bench task suite remains future work.

    Source: Future Work · p. 8

Abstract

from arXiv · show

Modern machine learning systems rely on complex data engineering workflows to extract, transform, and load (ELT) data into production pipelines. However, constructing these pipelines remains time-consuming and requires substantial expertise in data infrastructure and orchestration frameworks. Recent advances in large language model (LLM) agents offer a potential path toward automating these workflows, but existing approaches struggle with under-specified user intent, unreliable tool generation, and limited guarantees of executable outputs. We introduce kRAIG, an AI agent that translates natural language specifications into production-ready Kubeflow Pipelines (KFP). To resolve ambiguity in user intent, we propose ReQuesAct (Reason, Question, Act), an interaction framework that explicitly clarifies intent prior to pipeline synthesis. The system orchestrates end-to-end data movement from diverse sources and generates task-specific transformation components through a retrieval-augmented tool synthesis process. To ensure data quality and safety, kRAIG incorporates LLM-based validation stages that verify pipeline integrity prior to execution. Our framework achieves a 3x improvement in extraction and loading success and a 25 percent increase in transformation accuracy compared to state-of-the-art agentic baselines. These improvements demonstrate that structured agent workflows with explicit intent clarification and validation significantly enhance the reliability and executability of automated data engineering pipelines.

1 Introduction

Early ML pipeline stages require substantial manual effort, while modern workflows span increasingly complex data systems. kRAIG translates natural-language data requirements into executable pipelines, and ReQuesAct adds intent clarification to the agent loop.

1 Introduction

  • Ingestion, transformation, and validation require substantial manual effort to build and maintain before model training.Source: §1 · p. 1
  • kRAIG provides a conversational interface that translates natural-language data integration and transformation requirements into executable pipelines.It is built on Kubeflow Pipelines and generates containerized components.Source: §1 · p. 1
  • Unlike broad agentic systems, kRAIG specializes in composable data-engineering pipelines, initially targeting representative enterprise workflows and a subset of tools and tasks.The paper states this specialization aims to reduce manual preparation while preserving robustness and scalability.Source: §1 · pp. 1–2

1.1 Methodology Comparison

  • ReQuesAct adds an explicit intent-clarifying question to the reasoning-and-action loop because reason-then-act agents can behave unstably on underspecified or execution-heavy tasks.Source: §1.1 · p. 2

2 Task understanding

kRAIG uses multiturn dialogue to clarify user intent and derive a structured pipeline specification. Once intent is sufficient, retrieval-augmented examples support pipeline generation and few-shot generalization.

2 Task understanding

  • kRAIG uses multiturn dialogue to clarify intent and extract the structured specification needed to synthesize a data pipeline.Source: §2 · p. 2

2.1 Interface Example

2 Task understanding

  • After intent is sufficient, kRAIG retrieves relevant pipeline examples and emphasizes few-shot generalization from a small representative set.The authors report that these examples enabled completion of their internal tasks.Source: §2 · p. 2

3 Dynamic tool discovery and on-demand tool synthesis

After interpreting the task, kRAIG retrieves relevant tool specifications for pipeline generation and can synthesize task-specific tools when existing capabilities do not suffice.

  • After task interpretation, kRAIG retrieves relevant tools and supplies their interfaces, parameters, and constraints as context for pipeline generation.Source: §3 · p. 3, Figure 3 · p. 3
  • Because transformation steps vary widely, kRAIG can synthesize task-specific tools when its existing capabilities are insufficient.Automated synthesis may compromise data integrity through truncation or deletion.Source: §3 · p. 3
  • Inference-time steering with example pipelines biases tool selection and synthesis toward valid transformations and reduces deviation from expected pipeline structure.Source: §3 · p. 3

4 Safeguards

kRAIG combines post-synthesis pipeline validation with constrained tool use to reduce the likelihood of data-integrity compromises. The paper also illustrates a generated pipeline running in Kubeflow.

4 Safeguards

  • After synthesis, an LLM validation stage checks for destructive operations and rejects or sanitizes pipelines that violate safety constraints.Examples include DROP TABLE and bulk deletions.Source: §4 · p. 3
  • For high-risk operations, kRAIG restricts use to curated tools with constraints such as read-only access or scoped writes.The agent must compose workflows from these controlled tools rather than arbitrarily constructing or invoking low-level operations.Source: §4 · pp. 3–4
  • Together, kRAIG’s safeguards reduce the likelihood that generated or executed pipelines compromise data integrity.Source: §4 · p. 4

4.1 Kubeflow Pipeline Example

5 Analysis and Data Visualization

kRAIG adds visualization and aggregation-table capabilities to support auditing of pipeline transformations. Its standard components include extraction, transformation, migration, and basic validation.

5 Analysis and Data Visualization

  • kRAIG typically generates extraction, transformation, migration, and basic-validation components, including checks comparing inserted and extracted row counts.Data extraction must be performed separately before downstream augmentation and visualization.Source: §5 · p. 4
  • Generated tables and visualizations let users audit intermediate results and verify correctness without manual checks.Extraction is a separate step before downstream augmentation and visualization.Source: §5 · p. 4

5.1 Example of visualizations

6 Evaluation

The evaluation examines prompting and model variants, repeated-generation stability, and ELT Bench performance. kRAIG achieves strong benchmark results, with performance varying by model, prompt structure, and task complexity.

6.1 Model Variant Performance

  • With NL prompting alone, both models achieve 0% SC and 0% SPC; with NL + Exp, Claude 3.7 reaches 91.67% SC and 100% SPC with two edits.Nova Premier reaches 58% SC and 33% SPC with NL + Exp.Source: §6.1 · p. 5
  • With NL + Exp + Tools, Claude 3.7 achieves 100% SC and 100% SPC with zero edits, while Nova Premier reaches 77% SC and 66% SPC with more edits.The results indicate a capability–complexity tradeoff: optional tooling helps Claude 3.7 but raises the error rate for the less capable model.Source: §6.1 · p. 5

6.2 Pipeline Generation Variance and Reproducibility

  • Across most data sources, repeated generations have average similarity of 0.96–0.99; transformation-heavy GitHub tasks show greater variation but remain executable.Pipelines were generated 20 times per source using identical prompts; transformation-heavy tasks produced more unique versions, while standardized Hugging Face tasks showed particularly high similarity and low variance.Source: §6.2 · p. 5

6.3 ELT Bench Evaluation

  • 75% SRDEL and 25% SRDT: kRAIG outperforms all baselines on ELT Bench, particularly on transformations where prior methods largely fail.The evaluation compares kRAIG with SWE-Agent, Spider-Agent, and a modified Spider-Agent variant.Source: Table 3 · p. 6, §6.3 · pp. 6–7, §6.4 · p. 7
  • ELT Bench results primarily measure load and transformation performance because the benchmark does not capture kRAIG’s broader extraction capabilities.The evaluation uses eight randomly selected tasks of evenly distributed difficulty, the same subset for the agents, and each baseline’s strongest single model configuration.Source: §6.3 · p. 6
  • The modified Spider-Agent adds setup and structural details to its examples; it reaches 25% SRDEL and 12.5% SRDT, while original Spider-Agent reaches 25% SRDEL and 0% SRDT.SWE-Agent achieves 12.5% SRDEL, with SRDT near zero across most configurations.Source: §6.3 · pp. 6–7
  • Clarifying missing execution assumptions is identified as a contributor to kRAIG’s results, preventing cascading failures in workflows with unspecified infrastructure or schema details.The cited examples include MongoDB workflows where deployment type or expected schema is unspecified.Source: §6.3 · p. 7

6.4 Discussion

  • Internal applications included CVE security-data cleaning and Hugging Face image preprocessing across diverse sources, transformations, and destinations.The paper also concludes that kRAIG reduces human intervention and supports complex transformations across data-engineering tasks.Source: §6.4 · p. 7

Limitations

kRAIG is more reliable on extraction and loading than on complex transformations, and its extraction performance can be challenged by highly heterogeneous or dispersed sources. Complex tasks may also proceed without sufficient clarification.

  • kRAIG performs more reliably on extraction and loading than on complex transformations, which require more nuanced reasoning and domain-specific logic.Source: Limitations · p. 7
  • Many heterogeneous or geographically dispersed sources can make extraction harder; complex tasks may also proceed without sufficient clarification, producing incomplete or suboptimal pipelines.These limitations align with reported challenges in current LLMs’ ability to identify underspecified requirements and generate clarifying questions (Li et al., 2025).Source: Limitations · p. 7

Conclusion

The conclusion highlights explicit questioning and context distillation as contributions to addressing ambiguity and complex workflows, while describing kRAIG’s role in automating repetitive internal tasks.

Conclusion

  • ReQuesAct’s explicit questioning reduces ambiguity and cascading failures; Spider-Agent trajectories show that extra reasoning alone cannot resolve local–cloud schema inconsistencies.Traditional ReAct approaches are also described as more susceptible to ambiguity and inefficient reasoning paths.Source: Conclusion · p. 7
  • Context distillation improves success on complex workflows, particularly when instructions approach model context limits.Source: Conclusion · p. 7
  • kRAIG is designed to automate repetitive operations and reduce manual effort while supporting researchers’ internal data-analysis tasks on Kubeflow.Source: Conclusion · p. 8

Future Work

Future work targets broader evaluation, stronger questioning, more capable transformation and visualization workflows, and improved safeguards for high-stakes environments.

Future Work

  • A full ELT-Bench evaluation and expanded capabilities are planned, while maintaining security and governance constraints for internal deployment.Source: Future Work · p. 8

6.5 Expansion of Questioning Ability

  • Questioning remains a reduced-performance area; planned improvements target missing information and clarifying questions, while workflow structure supports reasonable performance on standard ELT tasks.The paper attributes standard-task performance to few-shot generalization, retrieval-augmented examples, and constrained tool use.Source: §6.5 · p. 8

6.6 Enhancements on Data Visualization and Analytics

  • Improved complex-transformation support is intended to enable kRAIG to generate, audit, and analyze intermediate and final datasets for end-to-end visual summaries.Source: §6.6 · p. 8

6.7 Guardrail Improvement

  • Current LLM-based validation and static patterns are insufficient for high-stakes production under jailbreak or sophisticated prompt-injection risks; formal verification and sandboxed symbolic execution are proposed.Source: §6.7 · p. 8

Ethical Considerations

The ethical discussion identifies risks from pipeline access and write operations, describes preventive design measures, and notes that the evaluation does not comprehensively test security.

Ethical Considerations

  • Incorrect generations or misuse of pipelines accessing sensitive sources and performing writes could disclose private data, create non-compliant datasets, damage data, or disrupt operations.Source: Ethical Considerations · p. 8
  • kRAIG uses static validation to reject destructive operations and curated vetted tools to reduce unsafe behavior, alongside recommended governance controls and human review for high-risk changes.Source: Ethical Considerations · p. 8
  • The evaluation measures executability and benchmark performance, not comprehensive security or adversarial red-teaming; stronger safety evaluation and write-operation policy enforcement remain future work.Source: Ethical Considerations · p. 8

Use of AI Assistants

The authors report using AI assistants for editing and code debugging, while taking responsibility for the work’s content.

Use of AI Assistants

  • AI assistants were used for editing assistance and code debugging; the authors take full responsibility for the content.Source: Use of AI Assistants · p. 9
Loading 2603.20311v1…