PaperlayerView exact revision on arXiv

Source-linked AI summary

MUSE: An Interactive Meta-Agent for Understanding and Steering LLM-powered Data Science Systems

Wei-Hao Chen, Weixi Tong, Yuan Tian, Chenglong Wang, Tianyi Zhang

arXiv:2608.16181v1cs.HCcs.AI

TL;DR

Agentic data science systems remain difficult for users to interpret, diagnose, and steer when workflows produce unexpected outputs. MUSE restructures execution traces and supports contextual inspection, verification, and repair; a 15-participant study found faster task completion and greater confidence.

  • Problem

    Users lack accessible support for judging agentic data science workflows and correcting erroneous steps beyond overwhelming low-level traces and free-form repair instructions.

  • Method

    MUSE restructures execution logs into semantic levels and supports contextual monitoring, verification, questioning, feedback, and scaffolded workflow repair.

  • Results

    Participants using MUSE completed tasks in 17 minutes on average, reducing task completion time by 35% and 9% compared with the other two conditions, while reporting greater confidence.

  • Takeaways & Limitations

    MUSE provides more accessible ways for users to inspect and intervene in agent-generated data science workflows.

  • Takeaways & Limitations

    The controlled study primarily reflects support for less-experienced users, and MUSE’s predefined warning heuristics may miss errors encountered in practice.

Abstract

from arXiv · show

Recent advances in large language models have enabled a new class of agentic data science systems that allow users to complete complex data science workflows through natural language. Although these systems can significantly reduce manual effort, it remains difficult to diagnose their behavior and steer the reasoning process when failures or unexpected outputs occur. We present MUSE, an interactive meta-agent that enhances user understanding and control of agentic data science systems by (1) dynamically restructuring low-level execution traces into multiple semantic levels that support navigation from high-level overviews to low-level implementation details; (2) enabling users to reference specific workflow steps in context to ask grounded questions, provide feedback, and revise problematic steps without manually locating relevant execution history; and (3) supporting mixed-initiative steering by surfacing suspicious steps for inspection, scaffolding the repair process, and translating user repair intent into contextualized instructions for the underlying agent. In a between-subjects study (n = 15), MUSE improved task efficiency and increased users' confidence in understanding and steering agentic data science workflows.

1 Introduction

Agentic data science systems make sophisticated workflows accessible through natural language but offer limited transparency for understanding and steering their reasoning. MUSE addresses this gap by restructuring execution traces, supporting grounded repair and feedback, surfacing suspicious steps, and scaffolding verification, improving task efficiency and confidence in a 15-participant study.

  • Motivation: Agentic systems orchestrate specialized agents for data cleaning, feature engineering, and visualization from a few natural-language prompts, broadening access to sophisticated data science tasks.They support users with limited expertise in completing complex workflows.
  • Motivation: Users must sift through low-level execution traces to determine what went wrong, while existing observability tools primarily target professional agent developers.Exposed information includes agent messages, internal states, and tool invocations.
  • Motivation: A formative study with 7 participants found that debugging was difficult because raw logs lacked structure for judging workflow progress, and repair was also difficult.The central challenge was assessing whether the overall workflow made sense rather than merely viewing system logs.
  • MUSE: MUSE restructures execution logs across semantic levels and supports grounded step reference, suspicious-step detection, scaffolded repair, and workflow-grounded verification.Users can move between high-level workflow overviews and low-level implementation traces while revising problematic steps and checking intermediate results.
  • Evaluation: 17 minutes on average: MUSE users completed tasks 35% and 9% faster than the other two conditions and reported greater confidence in the results.The evaluation was a between-subjects user study with 15 participants.

2 Related Work

Prior work supports exploratory data science, natural-language interaction, execution-trace observability, and interactive debugging, but users still face difficulty interpreting agent behavior and formulating workflow-level revisions. MUSE addresses this gap by restructuring low-level execution information and supporting more agentic verification and refinement across end-to-end data science workflows.

  • Interactive Data Science Systems: Data science workflows are exploratory and iterative, spanning stages such as cleaning, exploration, modeling, and evaluation.HCI researchers have developed interactive tools to support these workflow stages.
  • Interactive Data Science Systems: LLM-based systems shifted interaction toward intent-based outcome specification, enabling analysts to express analytical goals in natural language and iteratively receive generated charts and transformations.The related work describes this shift as moving from specifying what to do toward expressing desired results.
  • Interactive Debugging and Observability: Existing LLM-powered systems and debugging tools help users perform data science tasks or refine generated code, but verbose, low-level execution traces widen the gulf of evaluation.Prior systems focus on verifying generated data-science code or editing execution histories rather than understanding the agent behavior trajectory.
  • MUSE’s Distinction: MUSE differs by generating verification plans and scripts, proactively detecting potential errors, recommending revisions, and supporting sophisticated end-to-end data science workflows.The cited comparison contrasts MUSE with prior systems focused on data queries and data analysis tasks.
  • Agent Observability and Debugging: Agent observability tools provide hierarchical traces, graph-based workflow summaries, dashboards, or execution-level inspection, primarily for developers building and debugging agent systems.These tools make system behavior easier to inspect but do not primarily target users’ workflow-level understanding.
  • MUSE’s Distinction: MUSE restructures low-level execution logs to reduce users’ need to translate scattered traces or per-agent activities into an overall data-science workflow and manually formulate revisions.Existing tools center on execution-level inspection, message-level intervention, or agent-centric explanation.

3 Formative Study

A formative study with 7 data scientists identified four user needs for interacting with LLM-powered data science tools, motivating MUSE’s design rationale. MUSE addresses these needs through semantic representations, runtime warnings, contextual step-based interaction, and mixed-initiative control.

  • 3 Formative Study: 7 data scientists with AI coding-tool experience participated in the formative study, which identified 4 major user needs.The study procedure is described in Appendix A.1, with participant backgrounds reported in Table 1.
  • 3.1 User Needs: Participants needed clearer understanding of system behavior because low-level traces did not convey the overall data science workflow.They struggled to map exposed execution details onto their mental models, including distinguishing data cleaning from model training.
  • 3.1 User Needs: Participants wanted to interrupt, redirect, or verify long-running execution instead of waiting until the workflow finished.They reported being unable to interact with the model during lengthy steps and sometimes seeing only a spinning indicator.
  • 3.2 Design Rationale: MUSE translates unstructured logs into five semantic levels, allowing users to navigate from high-level summaries to detailed execution evidence.Semantic sliders support moving from an overview to lower-level implementation details.
  • 3.2 Design Rationale: MUSE monitors the underlying data science agent at runtime and attaches warnings to suspicious workflow stages to localize diagnostic attention.The warnings direct users toward steps most likely to contain useful diagnostic information.
  • 3.2 Design Rationale: Users can drag workflow steps into the chatbox, enabling MUSE to resolve relevant code and logs and contextualize requests for actionable repairs.Each step across semantic levels links to its underlying evidence, so users can express repairs at the workflow-step level.
  • 3.2 Design Rationale: MUSE supports mixed-initiative control by inspecting execution traces in parallel, allowing users to ask questions and understand long-running steps without stopping the agent.It also scaffolds verification and repair through context-grounded repair options and targeted verification questions.

4 System Design

MUSE organizes its meta-agent functionality into four layers that transform execution traces into navigable semantic representations, surface suspicious steps, support grounded verification, and enable contextualized workflow repair. These layers collectively connect overview-to-detail understanding with localized warnings, in-context questions, independent checks, and user-directed revisions.

  • System architecture: MUSE comprises Summarization, Monitoring, Verification, and Steering layers for restructuring traces, highlighting problematic steps, verifying intermediate results, and revising workflow steps.The Summarization Layer structures raw logs; the Monitoring Layer surfaces potential problems; the Verification Layer supports in-situ checks; and the Steering Layer enables feedback and repair.
  • Summarization Layer: The Summarization Layer captures execution traces, groups them into semantic chunks, and transforms each chunk into a multi-level semantic representation.Chunk boundaries follow explicit step-completion markers, synchronizing representations with stages the agent has completed rather than inferred boundaries from free-form logs.
  • Summarization Layer: Five abstraction levels let users navigate from a data-science stage title to detailed code, commands, and tool outputs.L1 gives the stage title, L2 its summary, L3 ordered sub-steps, L4 detailed actions and parameters, and L5 underlying traces; MUSE generates this representation in real time.
  • Monitoring Layer: The Monitoring Layer detects suspicious behaviors and places localized warning badges on corresponding workflow steps, with explanations and follow-up repair actions.Heuristics target reasoning–outcome mismatches, incorrect data processing, execution errors, and other agent misbehavior; warning panels support both understanding and repair.
  • Verification Layer: Users can anchor questions to workflow steps, while Help me verify combines selected outputs with file and session context to generate sandbox verification plans.Anchored questions retrieve relevant reports, scripts, artifacts, and sandbox files as prompt evidence, and verification occurs independently without instructing the underlying data-science agent.
  • Steering Layer: The Steering Layer scaffolds revision with warning explanations, repair suggestions, referenced code or data locations, and contextualized repair requests.Users can inspect referenced files, choose a suggested repair plan, and revise the affected workflow step without manually diagnosing the issue.

5 USAGE SCENARIO

The usage scenario shows how MUSE helps Johnny understand and steer a demand-forecasting workflow through multi-level traces, grounded questions, warning explanations, revisions, and result verification. Across the workflow, MUSE supports progressively detailed inspection, interpretation refinement, targeted repair, and confidence in the final recommendation.

  • Understanding workflow traces: MUSE presents each pipeline stage through semantic levels, allowing Johnny to expand step cards from concise summaries to detailed logs.Johnny adjusts the semantic slider from L1 to L5 while inspecting the same workflow stage.
  • Understanding workflow traces: Johnny drags an EDA summary into context and learns that annual seasonality varies across store-item combinations, including weaker or noisier signals.The overall dataset shows clear annual seasonality, but the pattern is uneven across groups.
  • Steering workflow execution: MUSE surfaces a Feature Engineering warning, explains how global dropna() may remove useful early-history rows, and links Johnny to the relevant script line.Johnny confirms that the script calls dropna() and asks MUSE to explain the operation before revising the step.
  • Steering workflow execution: MUSE offers three revision options, and Johnny selects “Keep rows with partial history” because it best matches the warning’s explanation.The selected option reflects Johnny’s judgment that some rows may remain useful without sufficient history for every feature.
  • Verifying results: 0.9296: Johnny verifies the recommended Random Forest R2 score by selecting “Help me verify,” choosing recomputation, and confirming that MUSE’s script reproduces the value.MUSE explains what R2 measures before generating the verification script, assuring Johnny that the reported metric was computed correctly.

6 USER STUDY DESIGN

The study evaluated MUSE in a counterbalanced between-subjects experiment with 15 participants, three interaction conditions, six Kaggle data-science tasks, and two Claude models. Participants completed two prompted tasks under their assigned condition, with 30 minutes allotted per task.

  • Participants: 15 participants from diverse academic backgrounds took part in the between-subjects user study.Backgrounds included Business Analytics, Computer Science, Finance, Architecture, Statistics, and several engineering and science fields.
  • Conditions: The study compared performance across three conditions differing in abstraction and interaction support, including a raw-log interface exposing execution traces and chat feedback.Raw logs included code edits, reasoning traces, intermediate plans, tool calls, and runtime messages.
  • Tasks: Six representative Kaggle data-science tasks were selected to resemble real-world scenarios rather than potentially memorized classic tasks.The authors identified Kaggle as a primary benchmark source for state-of-the-art data-science agents and noted concerns about unrealistic, memorized tasks such as Titanic survival prediction.
  • Models and assignment: The study included Claude Haiku 4.5 and Claude Opus 4.6, with conditions, tasks, and models counterbalanced across participants.Participants were not told which model they were using.
  • Procedure: After an introduction and tutorial, participants completed two assigned-condition tasks, with 30 minutes allotted per task, and crafted their own prompts from dataset and prediction-task descriptions.The study procedure, measures, data analysis, and interview protocol are documented in Appendices B.2–B.5.

7 USER STUDY RESULTS

MUSE improved task success, completion time, confidence, and users’ perceived ability to understand, inspect, verify, and steer agentic data science workflows. Its warnings and semantic representations redirected monitoring toward localized, actionable inspection, while users’ responses varied by warning type and behavioral profile.

  • Task Performance: 90% success in Conditions B and C versus 50% in Condition A, while Condition C had the shortest completion time at 17 minutes.Condition C’s completion time was 35% lower than Condition A and 9% lower than Condition B; the ANOVA difference was significant (F(2, 12) = 4.71, p = .031, η2 = .44).
  • Confidence: Participants reported the highest confidence in Condition C, with confidence differences across conditions statistically significant (p = .0265).The passage reports means of 4.20, 2.60, and 5.80 across the three conditions but does not specify their order.
  • User Experience: Significant condition differences occurred for Hurried, Effort, and Frustration, but not for Demand or Performance.The reported ANOVA p-values were .0165, .0298, and .0428 for Hurried, Effort, and Frustration, versus .1516 and .1196 for Demand and Performance.
  • Warning Interaction: Participants resolved substantially more warnings in Condition C than Condition B, despite 39 warnings detected in B and 36 in C.Warnings were generated using the same detection method in Conditions B and C, with differences attributed to task assignment and randomness in LLM inference.
  • Warning Analysis: The most frequent mistakes were Unverified Schema Assumptions (56%), Missing Data Not Inspected (16%), and Single-Split Evaluation (12%).Haiku triggered 27% more warnings than Opus, while participants could not reliably identify whether they were interacting with the weaker or stronger model.
  • Perceived Understanding and Control: MUSE users found workflows easier to understand, suspicious steps easier to identify, verification easier, results more trustworthy, and agent steering easier than users in the other conditions.The corresponding ANOVA results were significant for understanding (p = .008), identifying suspicious steps (p < .001), verification and trust (both p < .001), and steering (p < .001), with localized warnings and contextualized scaffolds supporting these activities.

8 Discussion

The discussion highlights that agentic data science systems often produce seemingly complete results with silent errors, motivating contextual explanations and more flexible abstraction designs. It also notes that MUSE’s evidence is limited by the controlled study population and reliance on predefined warning heuristics.

  • System limitations: Agentic data science systems rarely produced syntax errors or crashes but often generated seemingly complete results containing silent errors.These errors included questionable assumptions, inappropriate data processing, and logic problems.
  • System limitations: Future systems should provide contextual information explaining proposed actions, affected data, and potential downstream effects.The discussion presents this information as a response to users’ difficulty interpreting potentially problematic system behavior.
  • Abstraction design: MUSE’s multi-level semantic abstraction hierarchy addresses verbose, unstructured agent outputs, but users may require customizable abstraction structures.The discussion suggests flexible mechanisms, including malleable user interfaces, as a direction for future work.
  • Study limitations: MUSE’s controlled study used six data science tasks and included many nonprofessional developers, so its findings primarily reflect less-experienced users.The discussion cautions that the setting may not capture real-world data science complexity or establish effectiveness for expert practitioners.
  • Study limitations: MUSE’s monitoring layer relies on predefined warning heuristics, limiting how it currently identifies suspicious behavior.This reliance is identified as a limitation of the monitoring layer.

9 Conclusion · A Formative Study · B User Study

MUSE helps users understand and steer LLM-powered data science agents by restructuring execution traces and supporting contextualized inspection and revision. The formative study examined participants’ interactions with two agentic systems across representative data science tasks using think-aloud sessions and interviews.

  • 9 Conclusion: MUSE restructures low-level execution traces into multi-level semantic representations for more accessible workflow inspection and intervention.It also supports contextualized questioning, verification, and in-situ revision of agent-generated workflows.
  • 9 Conclusion: MUSE is designed to help users understand and steer LLM-powered data science agents.The study results suggest improvements in task efficiency and user confidence in understanding and steering workflows.
  • A.1 Formative Study Procedure: Participants completed two data science tasks using two LLM-powered systems while thinking aloud, followed by a semi-structured interview.Each session lasted approximately 60 minutes.
  • A.1 Formative Study Procedure: The formative study introduced MLE-STAR for machine learning workflows and Cursor as a general-purpose AI coding assistant.Participants used both systems on Kaggle-based data science tasks involving California Housing Prices and Optiver Realized Volatility Prediction.
  • A.1 Formative Study Procedure: During task execution, participants verbalized their reasoning, expectations, and difficulties while interacting with the systems.Follow-up interviews probed their experiences after task completion.
  • A.1 Formative Study Procedure: Interviews examined participants’ workflows, helpful and frustrating interactions, interpretations of intermediate outputs, and desired understanding or control.Sessions were screen-recorded and transcribed for inductive thematic analysis by two researchers who independently coded and discussed the data.

B.1 User Study Tasks · B.2 User Study Procedure · B.3 User Study Measures

The user study evaluated MUSE across six diverse data science tasks with counterbalanced, think-aloud sessions conducted under assigned interface conditions. It measured task performance, subjective confidence and workflow understanding, and logged participants’ feature interactions.

  • B.1 User Study Tasks: Six Kaggle tasks covered regression, classification, time-series forecasting, and recommendation across varied domains, modeling objectives, and data characteristics.The tasks were categorized into three difficulty levels to represent different workflow complexities.
  • B.2 User Study Procedure: Each participant completed an individual 80-minute study session after an interface tutorial and a description of the study’s comparison conditions.The proposed system’s condition was not disclosed to reduce demand characteristics.
  • B.2 User Study Procedure: Participants completed two assigned-condition tasks in counterbalanced order while crafting their own prompts and thinking aloud.Each task provided dataset and prediction-task descriptions rather than requiring participants to copy the descriptions.
  • B.2 User Study Procedure: A task counted as successful when the participant produced a runnable pipeline generating acceptable target-variable predictions within 30 minutes.The study also recorded whether each task was completed within the time limit.
  • B.2 User Study Procedure: After both tasks, participants completed a post-task survey and a brief semi-structured interview, receiving a $50 Amazon gift card.These activities followed task completion in the study procedure.
  • B.3 User Study Measures: Performance measures included task completion time, task success, and the number of task restarts.Task success required an acceptable-prediction runnable pipeline within 30 minutes.
  • B.3 User Study Measures: Subjective measures assessed confidence and workflow understanding through five statements covering ease of understanding, error location, intermediate-result verification, output trust, and execution control.The statements were labeled S1 through S5.
  • B.3 User Study Measures: Behavioral measures logged interface interactions, including adjusting semantic levels, inspecting warnings, and asking contextualized questions.The logs were used to analyze feature usage under each condition.

B.4 User Study Data Analysis · B.5 User Study Interview Questions

The study combined quantitative comparisons with inductive qualitative analysis to evaluate participants’ performance, ratings, and experiences. Semi-structured interviews examined understanding, diagnosis, verification, revision, and perceived control, including concrete difficulties encountered during use.

  • B.4 User Study Data Analysis: ANOVA compared performance and subjective ratings across experimental conditions.The analyses also reported descriptive statistics and p-values.
  • B.4 User Study Data Analysis: The quantitative analyses reported descriptive statistics and p-values across all measures.
  • B.4 User Study Data Analysis: Qualitative data were analyzed inductively across think-aloud data, post-task responses, interview transcripts, and interaction traces.
  • B.4 User Study Data Analysis: Two researchers independently reviewed the qualitative materials and iteratively developed a shared codebook.
  • B.4 User Study Data Analysis: Researchers resolved disagreements through discussion and refined the final qualitative themes through consensus.
  • B.5 User Study Interview Questions: After each condition, participants completed a semi-structured interview.
  • B.5 User Study Interview Questions: The interviews asked about understanding, diagnosis, verification, revision, and control of the system’s behavior.
  • B.5 User Study Interview Questions: Follow-up probes asked about concrete difficulties, including not knowing where to look or what information to use.

C Underlying System Architecture … C.8 Skill: step-complete

The underlying system is a skill-based, staged data science pipeline orchestrated by ds-pipeline, with MUSE operating as a read-only meta-agent that can return control for revisions. Its skills coordinate fixed-stage execution, artifact-based progress checking, focused analysis, and explicit phase-completion signaling.

  • C Underlying System Architecture: ds-pipeline instantiates a Claude Code agent with custom skills and orchestrates stage execution through ds-progress-checker.The control loop executes or dispatches a stage, emits a completion signal, and invokes the progress checker to determine the next action.
  • C.1 Relationship Between MUSE and ds-pipeline: MUSE runs alongside rather than replacing ds-pipeline, reading sandbox artifacts to answer questions and handing control back when execution must continue or change.ds-pipeline advances the workflow and writes artifacts, while MUSE provides read-only interpretation and control handoff.
  • C.2 Skill: ds-pipeline: The orchestrator sequences seven fixed stages from objective and dataset understanding through data-issue detection, cleaning, EDA, feature engineering, baseline modeling, and final reporting.ds-progress-checker controls stage transitions and resume logic.
  • C.3 Skill: ds-cleaning: ds-cleaning supports detection-only or issue-driven cleaning, producing issue reports or cleaned data within the session sandbox under fixed-input and single-execution rules.Detection covers missingness, type inconsistencies, duplicates, and simple outliers; cleaning applies focused actions from the issue report.
  • C.4 Skill: ds-eda: ds-eda generates a controlled set of 5–8 plots in standard mode, inspects them, writes a concise report, and marks the EDA step complete.The plots cover major univariate, bivariate, and correlation findings, with max_plots=8.
  • C.5 Skill: ds-feature-engineering: ds-feature-engineering prepares model-ready data through minimal, EDA-supported transformations, applying required encoding, justified scaling, and limited derived features.It saves the engineered dataset, writes a concise report, and marks the step complete.
  • C.6 Skill: ds-modeling: ds-modeling performs fast baseline checks with at most two untuned models, a single train/test split, task-appropriate metrics, and saved metrics and report artifacts.The skill runs in baseline_fast mode with max_models=2 and tuning=off.
  • C.7 Skill: ds-progress-checker; C.8 Skill: step-complete: ds-progress-checker audits sandbox artifacts to classify stages as done, missing, or needing retry, while step-complete emits a single human-readable marker for major completed phases.The checker respects stage dependencies and selects whether to run a skill or finish; step-complete is not used after every tool call.

D System Implementation · D.1 Prompt Design

MUSE combines a React/Vite/TypeScript frontend with a FastAPI backend for real-time structured agent-event streaming. Its prompt design includes templates for semantic translation, warning detection, contextualized referencing, revision handoffs, verification options, and verification execution.

  • D System Implementation: MUSE’s frontend uses React, Vite, TypeScript, Tailwind CSS, React Markdown, React Syntax Highlighter, and Recharts.The backend uses FastAPI with RESTful APIs and a persistent WebSocket endpoint, while agent execution is orchestrated through the Claude Agent SDK and built-in tools.
  • D.1 Prompt Design: The semantic chunk translation process is specified by a dedicated prompt template.Table 5 documents the prompt template used for semantic chunk translation.
  • D.1 Prompt Design: MUSE uses a dedicated prompt for warning detection and documents detailed warning heuristics.These are described in Table 6 and Table 11, respectively.
  • D.1 Prompt Design: A prompt template supports contextualized step referencing in MUSE.Table 7 documents the template used by MUSE for contextualized step referencing.
  • D.1 Prompt Design: MUSE constructs contextualized revision handoff requests using a dedicated template.Table 8 documents the template used to construct these requests.
  • D.1 Prompt Design: MUSE uses prompt templates to generate scaffolded verification options and execute verification.Table 9 covers scaffolded verification options, while Table 10 covers verification execution.
  • D.1 Prompt Design: The paper includes an example of a multi-level semantic representation for a data cleaning stage.Figure 12 illustrates this representation in the prompt-design material.
Loading 2608.16181v1…