DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?

Motivation

• DSBench targets limitations in existing benchmarks, including brief single-modality instructions, incomplete code-focused evaluation, constrained environments, and tool-dependent settings unlike real-life data science.Real-world tasks instead require lengthy multimodal reasoning, extensive data manipulation, and tool-irrelevant, data-centric problem solving.Source: §1 · pp. 1–2

Benchmark design

• DSBench contains 466 data analysis tasks and 74 data modeling tasks drawn from ModelOff and Kaggle competitions.Analysis tasks require answering financial questions, while modeling tasks require training on provided data and predicting test results.Source: §1 · pp. 2–3
• The benchmark incorporates long contexts, multimodal task backgrounds, large data files, multi-table structures, and end-to-end data modeling.Table 1 compares these characteristics with existing agent benchmarks across data, modality, evaluation, environment, and task dimensions.Source: §1 · p. 3, Table 1 · p. 3

Evaluation

• DSBench evaluates data analysis primarily with accuracy and introduces Relative Performance Gap to normalize heterogeneous data-modeling metrics.The benchmark assesses representative LLMs, LVLMs, and agents on both task categories.Source: §1 · p. 3
• The best-performing agent achieves only 34.12% accuracy on data analysis tasks and 34.74% Relative Performance Gap on data modeling tasks.These results support the benchmark’s conclusion that current approaches struggle with most realistic data science tasks.Source: §1 · p. 3

2.1.1 DATA COLLECTION

• 466 data analysis tasks were collected from 38 Modeloff challenges, while 74 data modeling competitions were selected from Kaggle.Source: §2.1.1 · p. 4, §2.2.1 · p. 5

2.1.2 TASK FORMULATION

• Data analysis agents receive task introductions, data files, and questions, with correctness assessed by semantic answer matching at task and competition levels.Source: §2.1.2 · p. 4

2.1.3 FEATURES

• Data analysis challenges combine images, Excel files, tables, long descriptions, and multi-table reasoning, while allowing tools such as Python or Excel.Source: §2.1.3 · pp. 4–5
• The benchmark supports comparison among pure LLMs/LVLMs and data science agents by permitting diverse tool-use strategies and minimal task constraints.Source: §2.1.3 · p. 5, §2.2.3 · p. 6

2.2.2 TASK FORMULATION

• Data modeling agents receive competition descriptions, training and testing data, and submission templates, then generate prediction files through an end-to-end modeling process.Source: §2.2.2 · p. 5, §2.2.3 · p. 6
• Data modeling evaluation measures both bug-free submission generation and competition-specific predictive performance, normalized across tasks with Relative Performance Gap.Source: §2.2.2 · p. 6

2.2.3 FEATURES

• Data modeling tasks include long contexts averaging 79,187 words and data, and execution-based evaluation spans 18 competition metrics.Source: §2.2.3 · p. 6, Figure 3 · p. 6

3.2.1 DATA ANALYSIS TASKS

• AutoGen improves over vanilla GPT-3.5, GPT-4, GPT-4o, and GPT-4o mini on data analysis, but its interaction mechanism increases time and cost; GPT-4o performs best among vanilla baselines while costing more than GPT-3.5.Even the most advanced agent remains far behind human performance.Source: §3.2.1 · pp. 7–8
• Models with similar overall data-analysis accuracy can solve different instances: GPT-3.5 solves 64.21% of the instances solved by Llama3-8b despite resolving 79 versus 61 instances overall.Figure 4 presents baseline accuracy across all data analysis challenges.Source: §3.2.1 · p. 8, Figure 4 · p. 8
• Data analysis performance declines as total input context length increases, as shown for AutoGen with GPT-4o and similarly observed in other models.The authors associate longer contexts with more complex backgrounds and larger data files requiring analysis.Source: §3.2.1 · p. 8, Figure 5 · p. 8
• Data analysis difficulty increases across release years, while human participants spend an average of 18.5 minutes per question.The authors attribute increasing difficulty to advances in data technology and the resulting complexity of questions.Source: §3.2.1 · p. 8, Table 5 · p. 8

3.2.2 DATA MODELING TASKS

• AutoGen with GPT-4o and GPT-4 generally improves data modeling capability through code execution and multi-turn revision, while AutoGen with GPT-3.5 and GPT-4o tends to finish faster than corresponding Interpreter systems.Interpreter with GPT-3.5 has twice AutoGen’s task success rate but a similar RPG, indicating weaker performance on the tasks it resolves.Source: §3.2.2 · p. 9, Table 6 · p. 9
• Data modeling completion shows no clear relationship with input length, suggesting that attribute clarity and specificity matter more than total context size.Figure 6 compares AutoGen with GPT-4o across input-length partitions.Source: §3.2.2 · pp. 9–10, Figure 6 · p. 9
• Data modeling performance varies little across most release-time slots, with completion rates of 50.00%, 53.33%, 25.00%, and 51.61% across the reported periods.The authors conclude that modeling ability does not significantly correlate with task-creation year.Source: Table 7 · p. 9, §3.2.2 · p. 10

LLMs/LVLMs as Agents

• Prior agents expanded from simulated textual environments to systems using code interpreters, browsers, and office tools, but evaluating real-world agent performance remains challenging.Examples include ReAct, AppAgent, OS-Copilot, and SWE-agent.Source: §4 · p. 10

Evaluations of LLMs/LVLMs

• Existing evaluations cover language, vision, safety, faithfulness, instruction following, and agent tasks, whereas this work evaluates both models and their interactive environments in real-world data science.The cited benchmarks include OSWorld, Spider2, DS-1000, SWE-Bench, and others.Source: §4 · p. 10

5 CONCLUSION

• DSBench contains 466 data analysis tasks and 74 data modeling tasks drawn from ModelOff and Kaggle competitions, providing a benchmark grounded in practical data-science environments.The authors present this authentic context as a resource for advancing data-science models.Source: §5 · p. 10

A EXPERIMENTAL SETUPS

• The evaluation reports accuracy for data analysis, RPG for data modeling, and costs and average resolution time when applicable.Open-source models run on a 4 × NVIDIA A100 GPU server.Source: A EXPERIMENTAL SETUPS · p. 16

B HUMAN PERFORMANCE FOR DATA ANALYSIS TASKS

• 30.41% accuracy-rate gap separated the most advanced agent from humans, which achieved 33.65% accuracy on the sampled tasks.The strongest agent therefore remained far from resolving all sampled tasks.Source: B HUMAN PERFORMANCE FOR DATA ANALYSIS… · p. 16
• Baseline performance was similar between the original and sampled testing data, with LLaVA performing worst on both.Source: B HUMAN PERFORMANCE FOR DATA ANALYSIS… · p. 16

C PROMPTS FORMAT

• Data-analysis prompts combine an introduction containing text, tables, and images with workbook content and explicit questions.LLMs and LVLMs receive the question and data as prompt content, while the template requires an answer to the question.Source: C PROMPTS FORMAT · p. 17
• Data-modeling prompts instruct the model to act as a data scientist and return predicted results as a CSV using training, test, and sample-submission files.Source: C PROMPTS FORMAT · p. 17
• The semantic comparison prompt evaluates whether a prediction matches the true answer and restricts output to True or False.Source: C PROMPTS FORMAT · pp. 16–17

D METRICS OF DATA MODELING TASKS

• 18 metrics are used to evaluate the data-modeling tasks, with their competition counts distributed across Figure 7.Source: D METRICS OF DATA MODELING TASKS · p. 18, Figure 7 · p. 18
• Figure 7 categorizes Kaggle competitions by evaluation metric, covering metrics from Accuracy and ROC to error, correlation, ranking, and logarithmic-loss measures.Source: Figure 7 · p. 18

E MORE BASELINE DETAILS

• Data-analysis LLMs and LVLMs receive task introductions concatenated with data files converted to text using Pandas.Source: E MORE BASELINE DETAILS · p. 18
• AutoGen receives task introductions and local file paths, allowing it to access data through a local code-execution environment.Source: E MORE BASELINE DETAILS · p. 18
• For data modeling, Code Interpreter accesses uploaded files through the OpenAI Assistant API, while AutoGen uses local file paths.Source: E MORE BASELINE DETAILS · p. 18
• The original competition submission file supplies the baseline performance used to compute RPG.Source: E MORE BASELINE DETAILS · p. 18

F HUMAN EVALUATION FOR SEMANTICS COMPARISON FUNCTION

• 100 sampled GPT-3.5 predictions were judged by people using the question, predicted answer, and ground-truth answer.Source: F HUMAN EVALUATION FOR SEMANTICS… · p. 18
• 100% human-evaluation accuracy was reported for the semantic comparison function.The authors describe this result as showing the function’s effectiveness.Source: F HUMAN EVALUATION FOR SEMANTICS… · p. 18
• The evaluation tests whether the function correctly determines if generated answers are right or wrong.Source: F HUMAN EVALUATION FOR SEMANTICS… · p. 18

• Figure 8 shows a Kaggle competition example.Source: G KAGGLE EXAMPLE · p. 19
• The example is used to illustrate a data modeling task context.Source: G KAGGLE EXAMPLE · p. 19
• The passage identifies the visual but does not report a comparison or outcome.Source: G KAGGLE EXAMPLE · p. 19

• Agents may fail to identify and retrieve the data required to complete a task.Source: H ERROR ANALYSIS · p. 19
• Incorrect approaches or formulas reveal deficiencies in problem-solving strategy.Source: H ERROR ANALYSIS · p. 19

I.1 QUALITATIVE ANALYSIS

• GPT-4o misinterpreted district codes as voter identifiers, incorrectly concluded every district had 10 voters, and failed to identify the smallest district.The example concerned counting voters across nine districts in an Excelstan election task.Source: Figure 9 · p. 20, §I.1 · pp. 21, 24, Table 9 · p. 22
• The qualitative set includes both resolved and unresolved data-analysis examples, showing that GPT-4o and AutoGen vary substantially across task instances.Tables 9 and 12 document unresolved cases, while Tables 10 and 11 document resolved cases.Source: §I.1 · p. 21, Table 9 · p. 22, Table 10 · p. 25
• AutoGen with GPT-4o successfully solved a Monte Carlo dice task, reporting a 73.36% winning probability and the >70% range.The procedure simulated 5,000 outcomes with up to 50 rolls per game.Source: §I.1 · pp. 29–30, Table 11 · p. 29
• AutoGen with GPT-4o failed to complete a workbook-based electricity-usage analysis because the generated Python script terminated with an execution error.The task required loading the Usage sheet, filtering February records, and calculating mean hourly usage.Source: Table 12 · p. 32, §I.1 · pp. 33, 34, 36

Initial diagnosis

• The initial script failed with `KeyError: ’Date’` because the `Usage` sheet contained one poorly structured column rather than a separate date field.The inspected column was `’ 3 PM Mon 24th-Mar-2014___0.384 kwh ’`.Source: §I.1 · p. 34

Date parsing

• A first cleaning attempt then failed because strings such as `Wed 16th-Jul-2014` did not match the specified `%d-%b-%Y` date format.The proposed correction removed weekday names and ordinal suffixes before date conversion.Source: §I.1 · p. 36

Intermediate result

• The corrected script produced an average February electricity usage of 0.720 kWh, but the filtered data contained only one entry, so the result matched none of the provided options.The sole displayed record was `2014-02-20` with usage `0.72`.Source: §I.1 · pp. 37, 39

Robust parsing

• Inspection of the raw data revealed inconsistent formatting, including extra spaces, underscores, and varied date and time representations, motivating a more robust parser.Examples included `3 PM Mon 24th-Mar-2014___0.384 kwh`, `__8PM Thu 20-Mar-2014____1.523 kwh`, and `_1AM Friday 19th-Dec-2014___0.209 kwh`.Source: §I.1 · p. 39

Electricity-usage analysis

• The calculated average electricity usage per hour in February was 0.855 kWh, which did not match any provided options: 0.760, 0.784, 0.808, or 0.833 kWh.Source: §I.1 · p. 41
• The agent checked data completeness and parsing accuracy by printing February entries after cleaning, parsing, filtering, and dropping missing rows.The inspection supported the conclusion that the data contained 34 correctly parsed February entries, although the expected options remained unexplained.Source: §I.1 · pp. 44–45
• 0.055 kWh was obtained by dividing February’s 29.083 kWh total across 22 days and then by 24 hours, so it also failed to match the options.This alternative daily-average interpretation was substantially below the provided choices.Source: §I.1 · p. 47

Data-modeling task

• AutoGen with GPT-4o generated code for a data-modeling task but did not execute it to produce the submission file.This illustrates an instruction-following limitation when completion requires both code generation and execution.Source: Table 13 · p. 48

COVID-19 forecasting task

• The COVID-19 forecasting example required cumulative case and fatality predictions by location and future date, evaluated using the mean column-wise RMSLE over two columns.The challenge’s stated goal was to identify factors affecting transmission rather than primarily maximize forecast accuracy.Source: §I.1 · pp. 48–49

COVID forecasting

• For COVID forecasting, the generated workflow trained separate linear regression models for confirmed cases and fatalities, predicted both targets, prevented negative predictions, and wrote a CSV submission.The sample is identified as an AotoGen-with-GPT-4 data-modeling task that generates a submission file.Source: §I.1 · p. 50, Table 14 · p. 52

Essay scoring

• The initial essay-scoring approach converted essay text with TF-IDF and trained a RandomForestRegressor, but execution failed with a timeout.The analysis attributed the timeout to the data size and computational cost of TF-IDF transformation and Random Forest training.Source: §I.1 · p. 54
• The revised approach reduced computational load by simplifying the model to linear regression while retaining TF-IDF-based text features and rounded score predictions.The stated goal was to reduce feature space and use a less computationally intensive model.Source: §I.1 · p. 55
• The revised essay-scoring script completed successfully and saved the submission file to ./learning-agency-lab-automated-essay-scoring-2.csv.The successful run followed an earlier timeout and used a computationally lighter linear regression approach.Source: §I.1 · pp. 54–55