Source-linked AI summary
DataWeave: Deploying Human-LLM Analytics for Exploratory Structured Data Analysis
Raquib Bin Yousuf, Harith Laxman, Vitaliy Shkremetko, Eunice Son, Shambhavi Verma, Brian O'Leary, Venketesh Subramony, Sylvain Nazef, Jacquelyn Elias, Ron Coddington, Chris Contakes, Michael Riley, Naren Ramakrishnan
TL;DR
Exploratory analysis of structured data is difficult when schemas vary, semantics are domain-specific, and analysts’ questions evolve, while existing question-answering or autonomous approaches may not support iterative, inspectable work. DataWeave combines human-guided interaction, schema grounding, analytical planning, and query execution for journalist analysis of IPEDS. The paper reports that deployment experience shaped its architecture and that its design choices can improve correctness while maintaining practical latency and cost.
Problem
Journalists exploring large datasets must navigate complex schemas and evolving questions, while question-answering and autonomous systems may not fit exploratory workflows or expose assumptions for correction.
Source: §1 · p. 1Method
DataWeave uses a human-in-the-loop conversational workflow combining objective clarification, year-specific schema grounding, analytical planning, and schema-grounded query execution over IPEDS.
Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 4Results
Takeaways & Limitations
The paper distills design principles for balancing agent flexibility with structured grounding, human oversight, and deployment compatibility in reliable LLM analytics.
Source: §7 · p. 8Agent-mediated orchestration increased analytical flexibility but made schema exploration and computational cost more dependent on the agent’s reasoning path.
Source: §4.4 · p. 5
Abstract
from arXiv · showhide
Data journalism, the practice of using data analysis to surface newsworthy stories, depends increasingly on the ability of reporters and investigative journalists to uncover trends, disparities, and accountability narratives. In practice, exploring large structured datasets remains slow and brittle: journalists must navigate hundreds of variables across many datasets over years, understand data coding conventions, and write non-trivial analysis code while hypotheses evolve. Although LLMs are often touted as "ask in English, get SQL/answers," real newsroom workflows expose recurring failures, e.g., schema mismatches and drift, misread domain semantics and units, and silent assumptions. We present DataWeave, a system that addresses these needs by combining conversational interaction, schema grounding, analytical planning, and executable query generation to support exploratory analysis over structured data. Rather than treating LLMs as autonomous answer engines, DataWeave frames them as interactive partners whose outputs can be inspected, corrected, and steered as hypotheses shift. We present a case study with professional journalists using our system to analyze the U.S. Department of Education's Integrated Postsecondary Education Data System (IPEDS), a high-stakes public dataset with substantial domain semantics and frequent schema updates. We also report how deployment experience and iterative refinement shaped the current DataWeave architecture and its analytical workflow. Our findings distill design principles and deployment lessons for trustworthy human-LLM collaboration in structured data analysis.
1 Introduction
Journalists must explore complex, evolving datasets while questions and hypotheses change, but question-answering and fully autonomous approaches can obscure assumptions or miss the needs of exploratory work. DataWeave combines schema-grounded analysis with human oversight and inspectable artifacts to support iterative investigation.
1 Introduction
- Structured-data exploration requires navigating many variables and tables, interpreting cryptic schemas, and writing code as questions evolve, creating friction for domain experts with limited time or technical expertise.Journalists may have deep contextual knowledge without the time or expertise for low-level data wrangling.Source: §1 · p. 1
- Question-answering and fully autonomous analytics poorly fit exploratory work because analysts may not know the right question in advance and autonomous execution can obscure assumptions needed to trust or correct results.The paper notes that such approaches may work in controlled settings, while real-world analysis often requires evolving questions and visible intermediate steps.Source: §1 · p. 1
- IPEDS presents a demanding setting: its datasets cover thousands of institutions, schemas vary across years and tables, semantics are domain-specific, and missing or inconsistent values are common.Journalists asked for support exploring trends, testing hypotheses, inspecting intermediate results, and correcting errors as understanding evolves.Source: §1 · p. 1
- DataWeave combines intent interpretation, schema grounding, analytical planning, query formulation, and bounded execution in a human-in-the-loop workflow that keeps users in control of goals and interpretation.It preserves intermediate traces for inspection and refinement and supports cross-table joins, longitudinal comparisons, and subgroup analyses.Source: §1 · p. 2
- The authors report that exposing intermediate artifacts makes recurring failures—including schema drift, silent aggregation errors, and misaligned temporal assumptions—readily correctable, and that deployment refinement shaped the current architecture.Their deployment analysis and retrospective IPEDS case study examine iterative sensemaking while preserving editorial control and accountability.Source: §1 · p. 2
2 Related Work
Prior systems support natural-language querying, analytical artifact generation, visualization, and iterative specification, while related Text-to-SQL work generally targets well-posed questions and relatively stable schemas. DataWeave instead emphasizes collaborative exploratory analysis over heterogeneous, evolving public datasets, with visible intermediate steps and human oversight.
2 Related Work
- Related structured-data systems include open-domain table reasoning, autonomous insight generation, visualization interfaces, mixed-initiative transformation tools, code assistants, and multi-agent frameworks.These approaches span natural-language interfaces, program synthesis, and visualization support.Source: §2 · p. 2
- Text-to-SQL systems translate natural-language questions into executable SQL and typically assume well-posed questions and relatively stable schemas, aiming to produce a correct executable query.The cited discussion identifies schema linking, prompt design, and execution-guided generation as areas of improvement.Source: §2 · pp. 2–3
- DataWeave targets high-stakes exploratory analysis over heterogeneous, evolving public datasets, emphasizing objective-conditioned grounding, analytical planning, constrained execution, and persistent traces for collaborative human oversight.Questions may be underspecified, schemas may drift across releases, and analysts may refine their goals iteratively.Source: §2 · p. 3
3 System Architecture
DataWeave coordinates intent interpretation, schema grounding, analytical planning, query execution, recovery, and answer synthesis through an agent-mediated workflow. Its persistent interface and per-user state support continuing analysis across turns and sessions.
3 System Architecture
- The agent-mediated architecture coordinates clarification, schema grounding, analytical planning, query formulation, execution and recovery, and answer synthesis with persistent analytical traces.The workflow combines LLM-based intent interpretation and planning with schema-grounded query formulation and deterministic execution over IPEDS data.Source: §3 · p. 3
3.1 Front-End and Session Management
- The conversational interface maintains per-user sessions, showing clarification requests, analytical activity, results, and prior logs while preserving context for follow-up questions.Per-user storage retains prior turns, outputs, errors, and user-approved preferences.Source: §3.1 · pp. 3–4
3.2 Data and Metadata Representation
DataWeave represents IPEDS schemas as year-specific metadata and retrieves relevant information on demand rather than assuming a fixed global schema. It clarifies ambiguous or incomplete objectives before grounding and analysis proceed.
3.2 Data and Metadata Representation
- IPEDS table families are represented with year-specific relational tables and metadata catalogs, and table and column names are normalized during ingestion.Catalogs describe tables, variables, data types, and categorical value sets; the representation supports reasoning about schema changes over time.Source: §3.2 · p. 4
3.3 Intent Interpretation and Clarification
- When an objective has an ambiguous abbreviation, missing temporal scope, or other required information, DataWeave requests clarification before querying the data.The clarified objective then guides schema grounding and subsequent analysis.Source: §3.3 · p. 4
3.4 Schema Grounding
DataWeave retrieves schema information conditioned on the clarified objective, then uses that grounded context to form a constrained analytical plan.
3.4 Schema Grounding
- Objective-conditioned retrieval identifies candidate tables and relevant variable descriptions, including allowed categorical values when needed.This supplies compact schema context without including the complete database schema in the model prompt.Source: §3.4 · p. 4
3.5 Analytical Planning
- The analytical plan specifies tables, variables, filters, temporal scope, and operations needed to answer the question.Source: §3.5 · p. 4
- Planning uses retrieved schema and coded-variable meanings to constrain subsequent query formulation rather than relying on model memory.Source: §3.5 · p. 4
3.6 Query Formulation
DataWeave formulates queries from a grounded analytical plan, executes them under read-only and bounded constraints, and preserves results and interaction traces for inspection and continuation.
3.6 Query Formulation
- DataWeave formulates read-only SQL queries to retrieve, filter, join, and aggregate data according to the analytical plan.Source: §3.6 · p. 4
- Query formulation derives table and variable names from retrieved metadata and follows year-specific structures, variable semantics, and categorical codes.Source: §3.6 · p. 4
3.7 Execution and Recovery
- Execution is restricted to retrieval operations and bounded result sets, returning either query results or explicit errors.Source: §3.7 · p. 4
- When a query fails, execution feedback supports reformulation using the original objective, grounded schema context, and observed error.Source: §3.7 · p. 4
- Recovery remains bounded, and queries, results, and errors are retained in the analytical trace.Source: §3.7 · p. 4
3.8 Answer Synthesis and Trace Persistence
- The final response is synthesized from successful query results and retains provenance such as collection year and source tables.Source: §3.8 · p. 4
- The analytical trace preserves the objective, tool interactions, queries, returned results, and execution errors.Source: §3.8 · p. 4
- Persisted traces let users inspect prior steps, continue with follow-up questions, and resume exploratory workflows across sessions.Source: §3.8 · p. 4
3.9 Design Principles behind DataWeave
Deployment, evaluation, and iterative refinement informed design principles presented as potential takeaways for reliable human-LLM analytics on complex, heterogeneous structured datasets.
3.9 Design Principles behind DataWeave
- Deployment, evaluation, and iterative refinement of DataWeave informed the authors’ design principles.Source: §3.9 · p. 4
- The principles are offered as potential takeaways for designing reliable human-LLM analytics systems.Source: §3.9 · p. 4
- The stated target setting is complex, heterogeneous structured datasets.Source: §3.9 · p. 4
4 Deployment Experience and System Evolution
Deployment with journalists and analysts produced repeated, iterative use and informed a shift from explicit stages to agent-mediated orchestration. The deployment analysis also describes schema use and operational trade-offs of that architecture.
4.1 Deployment and Iterative Use
- 298 analysis sessions comprised 797 analytical steps from 18 users; the median user completed eight sessions, and 12 users completed at least five.Sessions often included follow-up questions and refinements rather than a single query.Source: Table 2 · p. 4, §4.1 · p. 5
4.2 Schema Grounding and Planning Behavior
- Core identifiers and enrollment variables appeared most often in executed analyses, while planners generally identified more candidate columns than execution used.The planner-to-execution pattern suggests conservative schema coverage followed by task-specific selection; column use tended to increase in later steps of longer sessions.Source: §4.2 · p. 5, Table 3 · p. 5
4.3 From Staged to Agent-Mediated Orchestration
- DataWeave retained its core analytical functions as coordination shifted from application-defined stages to agents that dynamically sequence schema discovery, planning, query formulation, and execution feedback.Source: §4.3 · p. 5
- The final architecture preserves the original analytical structure while allowing the execution path to adapt more directly to the task.Source: §4.3 · p. 5
4 Deployment Experience and System Evolution
- Across analytical steps, each step received one dominant intent and could receive multiple analytical dimension tags for additional properties.Source: Figure 3 · p. 5, §4.1 · p. 5
4.4 Operational Implications
- Agent-mediated orchestration increased flexibility but made schema exploration and computational cost more dependent on the agent’s reasoning path.This reinforced the need for structured schema access, explicit planning, constrained execution, and persistent traces; flexibility and integration compatibility remain separate deployment considerations.Source: §4.4 · p. 5
5 Evaluation
The evaluation examines answer correctness, operational behavior, and investigative utility, using a benchmark of structured IPEDS questions and a retrospective reconstruction of journalistic analysis. Across evaluated configurations, agentic systems substantially outperform the earlier prototype on correctness, while the case study shows DataWeave supporting sequential investigative analysis.
5 Evaluation
- The planner proposes broader column sets than execution uses, while column usage often expands in later steps of longer sessions.These patterns suggest conservative schema coverage followed by task-specific selection as investigations deepen.Source: Figure 5 · p. 6, Figure 6 · p. 6
5.1 Evaluation Setup
- The benchmark covers 400 questions derived from 45 published questions across 12 subject areas and 20 collection years; the prototype baseline covers 100 recent-year questions.Accuracy is measured within 1% relative error, or within 0.5 percentage points for percentage-valued answers.Source: §5.1 · p. 6
5.2 RQ1: Answer Correctness
- Accuracy is 70.0% for Agentic Exploration and 72.8% for Agentic RAG, versus 7.0% for the baseline.Figure 7 compares the prototype on a 100-question subset from the five most recent years with the agentic configurations on the full 400-question benchmark.Source: Figure 7 · p. 6, §5.2 · p. 7
5.3 RQ2: Operational Behavior
- Mean tool calls are 8.80 for Agentic Exploration and 9.02 for Agentic RAG, with median latency of 18.5 s and 18.0 s, respectively.Mean cost per question is $0.0601 for the baseline, $0.0330 for Agentic Exploration, and $0.0065 for Agentic RAG; costs are operational, not strictly normalized.Source: §5.3 · p. 7, Figure 8 · p. 7
5.4 RQ3: Investigative Utility in Story Construction
- The retrospective case study reproduces key reported statistics and supports a sequential workflow from signal detection through segmentation, validation, and outlier discovery.The reconstructed workflow follows a 2021 Chronicle of Higher Education article on graduate enrollment trends.Source: §5.4 · p. 7
6 Lessons Learned
Deployment with professional journalists yielded lessons about collaboration, transparency, agent delegation, and organizational fit. DataWeave supported journalists’ control over interpretation and verification, while agent-mediated flexibility increased the need for structured grounding, constrained execution, and persistent traces.
6 Lessons Learned
- Journalists used DataWeave collaboratively, inspecting queries and intermediate outputs to verify results and retain control over interpretation.Agent-mediated orchestration offers flexibility but can broaden and make schema exploration less predictable, reinforcing the need for grounding, constrained execution, and persistent traces.Source: §6 · pp. 7–8
- The retrospective case study links computed outputs to article claims and tables, illustrating how conversational analysis can support journalistic verification and story development.Source: Figure 9 · p. 8
- Analytical capability, reasoning flexibility, efficiency, and integration compatibility are separate considerations in whether a system fits an organization’s workflows.Source: §6 · p. 8
7 Conclusion
DataWeave’s conversational, query-centric workflows supported iterative sensemaking while preserving transparency and user control. The paper concludes that its design choices can improve correctness while maintaining practical latency and cost, and identifies extensions to additional domains as future work.
7 Conclusion
- Conversational, query-centric workflows can support iterative sensemaking while preserving transparency and user control.Source: §7 · p. 8
- The reported design choices can substantially improve correctness while maintaining practical latency and cost.Source: §7 · p. 8
- Future work will extend DataWeave to additional domains and explore guided autonomy while preserving human control.Source: §7 · p. 8