PaperlayerView exact revision on arXiv

Source-linked AI summary

DS-Lighting: Making Agent Harnesses Explicit for Data-Science Automation

Fan Liu, Hao Liu

arXiv:2608.28590v1cs.AI

TL;DR

Data-science agents can fail despite capable models because existing harnesses leave task representation, execution state, artifact constraints, and evaluation implicit. DS-Lighting makes the harness explicit through four reusable layers, executable operator programs, and a shared benchmark interface; experiments report improved robustness, reproducibility, comparability, and reliability across agents and benchmarks.

  • Problem

    Existing data-science agents leave harness assumptions about task representation, execution state, validation, and evaluation implicit, complicating reproducible comparison and performance attribution.

  • Method

    DS-Lighting decomposes the harness into data, workflow, execution, and evaluation layers, represents agents as operator programs, and evaluates them under a shared benchmark task interface, sandboxed runtime, and metric protocol.

  • Results

    Experiments across agents, harnesses, ablations, models, and failure modes show that explicit harness design improves reproducibility, comparability, and end-to-end reliability.

  • Takeaways & Limitations

    DS-Lighting improves robustness across agents and benchmarks, reduces avoidable execution and evaluation failures, and helps smaller backbone models approach stronger ones.

  • Takeaways & Limitations

    DS-Lighting does not cover the full range of industrial workflows and results may vary with dependencies, resources, APIs, or sandbox configuration.

Abstract

from arXiv · show

Large Language Model (LLM) agents have shown promise for automating data-science workflows, yet their end-to-end performance depends critically on the agent harness that represents tasks, manages execution state, constrains output artifacts, and provides evaluation feedback. Existing data-science agents often leave this harness implicit, making results difficult to reproduce, compare, and attribute across heterogeneous tasks. We introduce DS-Lighting, a unified harness toolkit that makes harness design explicit for data-science automation. DS-Lighting decomposes the harness into four reusable layers: data, workflow, execution, and evaluation, and represents diverse agents as executable operator programs that support both predefined pipelines and adaptive search. We further integrate multiple open-source data-science benchmarks into an MLE-Bench-style task format, enabling controlled comparison under a shared task interface, sandboxed runtime, and metric protocol. Experiments across agents, harnesses, models, and ablations show that explicit harness design improves reproducibility, comparability, and reliability, while reducing avoidable system-level failures in end-to-end data-science workflows. Our code is available at https://github.com/usail-hkust/dslighting

1 Introduction

Data-science agent performance depends on harness design as well as model capability, but existing systems leave harness assumptions implicit. DS-Lighting makes the harness explicit through reusable layers and shared evaluation, improving reproducibility, comparability, and reliability.

  • Harness design, not only model choice, determines end-to-end data-science agent performance.Heterogeneous inputs, long-horizon execution, and structured outputs can cause failures despite plausible reasoning or executable code.
  • Existing agents embed different assumptions about task representation, execution state, intermediate validation, and output evaluation.These implicit assumptions make performance differences difficult to attribute to models, workflows, or harness choices.
  • Data-science harnesses must unify fixed pipelines, adaptive search, and multi-agent workflows through a common interface.They must also ground inputs, maintain consistent code and file states, and enforce task-specific output constraints.
  • DS-Lighting makes harness design explicit through reusable data, workflow, execution, and evaluation layers.The toolkit represents agents as executable operator programs and supports predefined pipelines and adaptive search.
  • DS-Lighting converts multiple open-source benchmarks into an MLE-Bench-style format for controlled comparison under shared interfaces, runtime, and metrics.Experiments span agents, harnesses, ablations, models, and failure modes.

2 Related Work

Prior work spans data-science benchmarks, fixed and adaptive agent workflows, and general-purpose harness infrastructure. These efforts provide relevant components but do not directly target unified harnesses for data-science workflows.

  • Data-science benchmarks cover exploratory analysis, code generation, interactive refinement, and end-to-end modeling tasks.Examples include DA-Code for fine-grained code-generation operations and DSEval for interactive refinement.
  • LLM data-science agents generally follow fixed workflows or adaptive search.Fixed systems use predefined stages, while adaptive systems search over candidate solutions or actions.
  • General agent benchmarks and frameworks emphasize tool use, stateful interaction, validation, orchestration, logging, and reproducible comparison.Examples include WebArena, AppWorld, OSWorld, HAL, AutoHarness, and SafeHarness.
  • Existing harness efforts are not designed for data-science workflows.They therefore do not directly provide the task-specific integration needed for data-science automation.

3 Preliminaries

DS-Lighting formalizes data-science tasks, harnesses, and workflows as interacting components. Workflows select operators using fixed procedures or adaptive state-dependent policies, while execution produces observations that update the state.

  • A data-science task is represented as x = (q, D, A), combining an objective, data assets, and auxiliary requirements.The output y is a required artifact such as a report, prediction file, or model.
  • The agent harness manages task context, workflow execution, feedback, and artifact evaluation around four responsibilities.These responsibilities are constructing data context, instantiating workflows, executing actions, and evaluating artifacts.
  • An agent workflow organizes reusable operators and a controller across multiple execution steps.Operators can inspect data, generate code, train models, validate results, and submit outputs; the controller determines their composition and invocation.
  • Fixed workflows follow predefined operator sequences, whereas adaptive workflows select operators from intermediate observations.Adaptive behavior can retry code generation after execution errors or switch models after weak validation scores.
  • At step t, the controller selects an operator from the current state, the model generates an action, and execution returns an observation for subsequent state updates.The final output is produced and evaluated after the iterative loop.

4 DS-Lighting: A Layered Agent Harness

DS-Lighting implements a layered harness that turns raw task inputs into grounded operator workflows, controlled runtime interactions, and evaluated outputs. Its shared abstractions support fixed and adaptive execution while preserving state, constraints, and feedback.

  • Data Layer: The data layer converts raw task descriptions and assets into structured task contracts for downstream workflows.Data Analyzer and Task Handler reduce ambiguity about goals, files, schemas, and output requirements.
  • Workflow Layer: The workflow layer separates reusable atomic capabilities from controllers that schedule them during execution.This supports stable operator sequences as well as dynamic selection for exploration, refinement, and recovery.
  • Execution Layer: The execution layer runs workflow actions in a controlled environment and preserves intermediate results and feedback across steps.For concurrent workflows, scheduling coordinates ready actions under CPU/GPU and LLM-concurrency limits.
  • Layered Harness: The four layers provide a common harness interface spanning task contracts, operator programs, sandboxed execution, artifact production, and feedback.Figure 2 summarizes this progression from data-layer task construction through evaluation-layer validation.
  • Evaluation Layer: The evaluation layer validates intermediate artifacts during execution and computes final task-level scores after completion.Structured feedback can expose missing artifacts, schema mismatches, execution errors, or invalid submissions before final evaluation.

5 Experiments

The experiments evaluate heterogeneous agents and harnesses under shared interfaces, runtimes, and metrics, finding that explicit harness mechanisms improve reliability and benchmark performance. Additional analyses show that workflow style, model choice, and failure-mode control each shape outcomes.

  • Unified Evaluation of Data-Science Agents: DS-Lighting reproduces seven agents spanning fixed manual workflows and adaptive search workflows under a common task interface, execution environment, and evaluator.
  • Unified Evaluation of Data-Science Agents: Fixed-workflow agents perform reliably on standardized analyses, while search-based agents often reach higher peaks on selected task types but less uniformly.Data Interpreter reaches 0.91550 on Correlation Analysis and 0.88640 on Summary Statistics; DSAgent reaches 0.88890 and 0.91110 on those tasks under DeepSeek-V3.1-Terminus.
  • Comparison with Agent-Harness Frameworks: DS-Lighting is best or tied-best across all four benchmark comparisons with existing harness frameworks under matched task, model, execution, and evaluator settings.It matches VanillaHarness at 0.8521 on DABench and improves over the strongest baseline on DACode, MosciBench, and ScienceAgentBench.
  • Ablation Study of System-Level Mechanisms: Removing execution governance causes the largest ablation decline, reducing macro-average accuracy to 0.2714 and DABench accuracy from 0.8521 to 0.2724.The ablation retains the benchmark inputs, base agent, execution environment, and final evaluator while removing one mechanism at a time.
  • Ablation Study of System-Level Mechanisms: Removing data grounding lowers macro-average accuracy from 0.5167 to 0.3721, while removing evaluation alignment lowers it to 0.3971.The reported drops are especially large on DACode, MosciBench, and ScienceAgentBench, linking grounding and evaluator-aligned feedback to task fidelity and repair.
  • Further Analysis: Harness design can outweigh backbone scale, and DS-Lighting reduces noncapability failures so remaining errors concentrate more on intrinsic model limitations.Figure 4 reports DS-Lighting with Qwen surpassing LangChain with GPT-5.5; Figure 5 analyzes grounding, runtime-control, and evaluation-alignment failures.

6 Conclusion

The conclusion presents DS-Lighting as a unified toolkit that makes data context, workflow execution, and evaluation explicit across heterogeneous data-science agents. Experiments indicate improved robustness, fewer avoidable system failures, and closer performance between smaller and stronger backbone models.

  • DS-Lighting uses reusable data, workflow, execution, and evaluation layers to develop and evaluate heterogeneous data-science agents through a common interface.
  • Experiments show improved robustness across agents and benchmarks, reduced avoidable execution and evaluation failures, and smaller models approaching stronger ones.

Limitations

The evaluation does not cover the full range of real-world industrial data-science workflows and remains dependent on executable environments, libraries, evaluators, and sandbox configurations.

  • DS-Lighting does not cover industrial settings such as private databases, streaming data, human-in-the-loop review, or domain-specific governance.The authors also note that dependency versions, resource limits, API availability, and sandbox configuration may affect results.

A Usage Examples

DS-Lighting provides a consistent API for running single tasks, switching predefined workflows, defining custom agents, and scaling configurations to benchmark suites.

  • Single-task execution: A single-task run loads a task contract, executes a selected workflow, and collects the required artifacts.The same interface accepts either a loaded task object or a benchmark task id.
  • Predefined workflows: Predefined workflows such as aide, react, dsagent, and autokaggle can be compared by changing only the workflow argument.The model, task id, and benchmark runner remain fixed.
  • Custom workflows: Custom workflows implement solve(...), call operators, and write the required artifact to output_path for compatibility with the evaluator.Custom logic follows the same contract as predefined agents.
  • Benchmark evaluation: The same workflow configuration scales from individual tasks to benchmark suites through the DSBenchmark runner and official evaluation protocol.Figures 6–9 depict the progression from single-task execution to predefined and custom agents and benchmark runs.

B Task Grounding and Benchmark Normalization

DS-Lighting normalizes heterogeneous benchmarks into shared task contracts and represents workflows as executable operator programs with common runtime and evaluation interfaces.

  • Task normalization: Each normalized task contains descriptions, preparation and grading scripts, public agent-visible artifacts, and private evaluation references.The layout separates public inputs from hidden references reserved for grading.
  • Task contract: The task contract x = (q, D, A) specifies the objective and metrics, public data context, and required output artifact constraints.D includes prepared data and analyzer summaries; A defines filename, format, kind, sample submission, and grading mode.
  • Task normalization: Normalization prevents private-answer leakage, removes benchmark-specific I/O assumptions, and enables comparison through one task contract and evaluation path.The shared path supports controlled comparison across workflows.
  • Operator programs: An operator program acts on the fixed task contract while workflow control logic determines how the agent proceeds over time.The separation keeps task representation fixed while allowing agents to differ in control logic.
  • Operator programs: The operator library includes capabilities such as analysis, planning, code generation, execution, validation, review, and memory summarization, while state stores intermediate outputs and memory.The controller selects an operator and applies it to the current workflow state.
  • Workflow types: Manual workflows follow fixed operator schedules, whereas search-based workflows select operators adaptively to branch, refine, recover, revisit decisions, or terminate.Examples include debugging failed execution, refining weak results, and submitting after successful validation.
  • Execution interface: Both workflow types use structured operator inputs and outputs, sandboxed side effects, and output-contract checks before grading.This common interface supports grounding, runtime control, validation, and benchmark evaluation.

D Dataset Details

The unified suite covers 947 tasks across data analysis, code generation, multimodal scientific discovery, scientific program generation, and Kaggle-style modeling.

  • Data analysis: DABench evaluates closed-form analytical questions over tabular datasets using structured CSV answers and exact-match or tolerance-based grading.Its tasks cover statistics, correlations, distributions, feature engineering, preprocessing, outliers, and basic machine learning.
  • Code generation: DACode evaluates executable code and artifact production across data manipulation, interpretation, visualization, databases, and machine learning.Outputs include CSV, JSON, database, plot, and prediction files graded automatically by task-specific evaluators.
  • Scientific discovery: MoSciBench contains 88 multimodal scientific discovery tasks spanning six domains and modalities including time series, tables, images, spectra, molecular structures, genotype matrices, and text.The benchmark targets end-to-end scientific analysis.
  • Scientific program generation: ScienceAgentBench contains 102 data-driven scientific discovery tasks extracted from 44 papers across bioinformatics, chemoinformatics, geoscience, and cognitive neuroscience.Each task requires code execution to produce a required output artifact.
  • Data modeling: The suite also includes Kaggle-style predictive tasks across tabular, time-series, sequential-signal, and image-based medical data.The competition datasets are curated to represent heterogeneous predictive tasks across multiple modalities.

E Implementation Details

DS-Lighting enables controlled framework comparisons by using identical task information and evaluation metrics, while implementing workflow steps as explicit operators with controlled side effects.

  • Framework comparison: VanillaHarness, LangChain, AutoGen, OpenHands, and DS-Lighting are reproduced under the same data-science benchmark adapter.The adapter builds a unified contract with original descriptions, public files, sample artifacts when available, and required outputs.
  • Framework comparison: DS-Lighting’s gains are attributable to runtime design rather than access to extra benchmark supervision.The comparison isolates framework-level execution behavior under identical task information and evaluation metrics.
  • Operator implementation: Each workflow step is an asynchronous operator with declared inputs and outputs, controlled side effects, and explicit data bindings between dependencies.Operators can be composed sequentially, in parallel, conditionally, or through a dynamic DAG runtime.

F Additional Experimental Results

Experiments under DS-Lighting reveal large efficiency differences across agent workflows and show that harness implementation itself materially affects runtime. Competition-task results are task-dependent, while a shared structured task contract supports controlled comparisons.

  • Token Consumption: 5.1K tokens per task makes React the most token-efficient agent, versus 123.1K for AutoKaggle.AutoKaggle uses 24.21× more tokens than React, while React reduces consumption 51.2% relative to DeepAnalyze.
  • Token Consumption: Input tokens account for 82.1%–92.5% of total usage across agents, with React using 86.0% input and 14.0% output tokens.Task descriptions, intermediate observations, execution histories, and tool feedback dominate token cost.
  • Runtime Comparison across Agents: 0.69 minutes per task makes React fastest, while AutoKaggle takes 22.62 minutes, producing a 32.93× runtime gap.Interpreter-style workflows are faster; search, candidate generation, evaluation, and refinement increase runtime overhead.
  • Runtime Comparison across Frameworks: 642.7 versus 236.9 total minutes gives DS-Lighting ReAct a 2.71× speedup over LangChain ReAct across five complete model runs.The speedup appears on four of five models, reaching 5.49× on GPT-5.5; the exception involved frequent execution errors and early termination.
  • Kaggle-Style Competition Tasks: AIDE performs better on five of nine valid competition datasets, while Data Interpreter performs better on four.The comparison indicates task-dependent performance rather than a single framework dominating all competition-style settings.
  • Structured Task Contract: Each benchmark instance is represented as x = (q, D, A), giving compared harnesses the same rendered task contract and public feedback.q specifies the objective, D describes public data artifacts and analyzer summaries, and A specifies the required answer artifact and evaluation interface.
Loading 2608.28590v1…