PaperlayerView exact revision on arXiv

Source-linked AI summary

DSEffi-Bench: Demystifying Large Language Models' Capability in Efficient Data Science Code Generation

Zhihao Gong, Junzhe Yu, Dong Huang, Zeyu Sun, Jie M. Zhang, Dan Hao

arXiv:2608.30248v1cs.SE

TL;DR

Existing DS code-generation benchmarks largely equate quality with correctness despite large runtime differences among correct solutions. DSEffi-Bench introduces a stress-tested, human-validated efficiency benchmark and evaluates 16 models, finding that correctness does not characterize efficiency and that taxonomy-guided diagnostics can support improvement.

  • Problem

    Existing DS code-generation benchmarks evaluate functional correctness while leaving execution efficiency and large runtime differences among correct solutions unexamined.

  • Method

    DSEffi-Bench builds 1,000 human-validated instances from Stack Overflow posts and evaluates correctness and efficiency across 16 models using stress-testing harnesses.

  • Results

    Correctness and efficiency are not aligned, while 79.1% of efficiency deficits have domain-specific root causes and diagnostics yield up to +14.7% efficiency gains.

  • Takeaways & Limitations

    Execution efficiency should be evaluated as a distinct capability in DS code generation, because library-specific patterns differentiate efficient solutions from merely correct ones.

  • Takeaways & Limitations

    Runtime measurements depend on hardware resources and software versions, so the evaluation uses a controlled environment.

Abstract

from arXiv · show

Current data science (DS) code generation benchmarks equate correctness with quality, overlooking execution time differences that span orders of magnitude between correct solutions. We introduce DSEffi-Bench, the first benchmark specifically targeting execution efficiency in LLM-generated DS code, comprising 1,000 instances across 10+ DS libraries with stress-testing harnesses and human-validated references. Evaluating 16 models across 3 tiers, we find that correctness alone fails to characterize efficiency: GPT-5.4 leads in correctness (Pass, 66.9\%) but its efficiency score (B$|$P, 71.7\%) nearly matches GPT-5.4-mini (71.6\%), which solves 47 fewer tasks; Kimi-K2.5 ranks lowest in correctness among frontier models (40.2\%) yet achieves the highest efficiency score (73.6\%) across all 16 models. A human-annotated five-category taxonomy reveals that 79.1\% of efficiency deficits extend beyond algorithmic complexity to domain-specific root causes, with distinct failure profiles across model tiers and libraries. Two exploratory experiments provide initial evidence that these diagnostics can guide improvement, yielding up to +14.7\% efficiency gains via taxonomy-guided optimization and approaching Claude-Opus-4.6 Best@3 in efficiency at 13.0$\times$ lower cost via library-conditioned routing.

1 Introduction

DSEffi-Bench addresses the gap between functional correctness and execution efficiency in LLM-generated data science code, where equivalent solutions can differ greatly in runtime. Evaluations show that correctness and efficiency are not aligned, while domain-specific diagnostics can guide improvement.

  • Motivation: Functionally equivalent data science implementations can differ by an order of magnitude in execution time because of library-specific API choices.For example, groupby().apply() incurs Python-level callback overhead compared with groupby().cummax().
  • Benchmark: DSEffi-Bench contains 1,000 human-validated instances with evaluation harnesses covering correctness and stress tests.The benchmark is built from efficiency-oriented Stack Overflow posts through a quality-gated pipeline.
  • Findings: 66.9% Pass and 71.7% B|P make GPT-5.4 the correctness leader while nearly tying GPT-5.4-mini at 71.6% B|P despite solving 47 fewer tasks.This comparison illustrates that correctness does not determine efficiency among otherwise related models.
  • Findings: 40.2% Pass and 73.6% B|P make Kimi-K2.5 the lowest-correctness frontier model but the highest-efficiency model across all 16 models.The result further demonstrates rank inversion between correctness and efficiency.
  • Diagnostics and remedies: 79.1% of efficiency deficits extend beyond algorithmic complexity to domain-specific root causes, with distinct profiles across model tiers and libraries.Taxonomy-guided prompting improves efficiency by up to +14.7%, while library-conditioned routing approaches Claude-Opus-4.6 Best@3 at 13.0× lower cost.

2 Related Work

Prior efficiency benchmarks largely target algorithmic, implementation-level, or repository optimization, while data science benchmarks emphasize execution-based correctness and task breadth. DSEffi-Bench positions execution efficiency in DS code as a distinct, domain-specific dimension orthogonal to correctness.

  • Efficiency evaluation: Existing code-efficiency benchmarks focus mainly on competitive programming algorithms, implementation optimization, or repository-level patches.Examples include EffiBench, Mercury, COFFE, ENAMEL, EvalPerf, SWE-Perf, and GSO.
  • DS benchmarks: DS code-generation benchmarks such as ExeDS, DS-1000, and DSCodeBench expand execution-based testing and task coverage across DS libraries.Related efforts also scale toward end-to-end data-science pipelines.
  • Position: DSEffi-Bench is presented as the first benchmark targeting execution efficiency in LLM-generated DS code.Its efficiency dimension is treated as orthogonal to functional correctness.
  • Position: 79.1% of measured efficiency deficits are localized to library-specific root causes, including domain patterns such as loop-to-vectorization gaps.The paper distinguishes these patterns from those studied in competitive-programming benchmarks.

3 Benchmark Construction

DSEffi-Bench is constructed from authentic Stack Overflow efficiency questions through a four-stage, quality-gated pipeline that converts filtered posts into validated benchmark instances. The resulting schema combines a problem description, evaluation harness, and human-validated reference.

  • Pipeline: 1,000 instances are derived from real Stack Overflow posts to preserve authentic data-science optimization intent.The pipeline comprises post collection, relevance and feasibility filtering, instance transcription, and reference-solution validation.
  • Benchmark Schema: Each benchmark instance contains a problem description, an evaluation harness, and a human-validated reference solution.The harness defines correctness and stress-scale conditions, generates deterministic test inputs, and validates candidate outputs.
  • Efficiency-oriented Post Collection: 17,257 seed posts are retrieved by requiring a data-science library tag and efficiency keyword to co-occur in qualifying posts.Posts must also meet vote, view, and accepted-answer thresholds.
  • Relevance and Feasibility Filtering: LLM-as-Judge filtering reduces the seed set to 4,360 through sequential question-relevance and answer-feasibility checks.The filters assess DS relevance, efficiency orientation, self-contained detail, and reproducible solutions.

Appendix A.4.

The appendix details human validation and the evaluation protocol used to distinguish correctness from execution efficiency. It also specifies log-space efficiency scoring, model sampling, timing, and benchmark coverage.

  • Human Validation: 50.3% of reference solutions use fundamentally different strategies, 32.2% refine the accepted answer, and 17.5% reproduce it faithfully.Annotators revise benchmark artifacts when descriptions, generators, or validators are ambiguous or incomplete.
  • Human Validation: 1,247 candidates produce 1,000 validated instances after quality-based rejection of ambiguous tasks or unclear efficiency advantages.Two annotators compare reference solutions and may select, refine, or hybridize strategies.
  • Evaluation Metrics: Pass requires every generated test case to satisfy the output validator, with failures and timeouts receiving a score of 0.Pass is the fraction of tasks passed completely.
  • Evaluation Metrics: 14.5× median slowest-to-fastest runtime spread motivates log-space Beyond normalization, which preserves discrimination among correct solutions.The 90th-percentile spread is 522.0×, and log-space normalization yields a median score of 0.74.
  • Evaluation Metrics: Beyond|Pass reports mean efficiency among solved tasks, separating execution efficiency from the fraction of tasks solved.The protocol reports Pass, Beyond, and Beyond|Pass at the model level.
  • Model Evaluation: 16 models are grouped into Closed Frontier, Open Frontier, and Open Coder tiers and contribute up to four samples per task.The resulting efficiency spectrum averages 29.4 solutions per task.
  • Measurement Protocol: Execution times use one warm-up and three timed runs, averaged across test inputs in a controlled benchmark environment.The input generator produces an average of 38.8 test cases per task, including correctness and stress cases.

5 Evaluation

Evaluation across 16 models shows that correctness and execution efficiency are distinct capabilities, with model rankings varying across pairwise comparisons and libraries. A five-category diagnosis identifies domain- and tier-specific efficiency gaps, while exploratory optimization and routing experiments improve efficiency within the benchmark.

  • 5.1 Overall Evaluation: Correctness and efficiency are partially decoupled: GPT-5.4 leads Pass at 66.9% but has B|P 71.7%, nearly tied with GPT-5.4-mini at 71.6%.GPT-5.4-mini solves 47 fewer tasks, while Kimi-K2.5 has the lowest frontier-model Pass rate at 40.2% but the highest B|P at 73.6%.
  • 5.1 Overall Evaluation: Kimi-K2.5 remains highly efficient in pairwise comparisons, with a 39.5% win rate and 23.1% loss rate against representative models.On 131 tasks solved by all six representative models, Kimi-K2.5 ranks first with B|P 76.7%.
  • 5.1 Overall Evaluation: Efficiency strengths rotate across libraries: GPT-5.4 leads SciPy at 82.3%, GPT-5.4-mini leads PyTorch at 60.9%, and DeepSeek-V4-Flash leads Polars at 72.1%.GPT-5.4 leads GPT-5.4-mini on SciPy by 15.0% but trails it on PyTorch by 7.3%, indicating library-specific efficiency knowledge.
  • 5.2 Efficiency Gap Diagnosis: The taxonomy identifies five efficiency-gap categories, with OPU most frequent at 23.3% and OSM most severe at a median 14.4× slowdown.CRC and L2V have median slowdowns of 11.6× and 10.6×, respectively.
  • 5.2 Efficiency Gap Diagnosis: Model tiers show different blind spots: L2V rises from 7.3% in Closed Frontier models to 29.6% in Open Coder models, while OSM falls from 29.3% to 9.3%.Open Coder models often use explicit loops where vectorized calls exist, whereas Closed Frontier models use heavier computation stacks than required and show a 27.2% OPU compositional bias.
  • 5.3 Exploratory Experiments: Taxonomy-guided optimization raises B|P by +12.9% for DeepSeek-V4-Flash and +14.7% for Kimi-K2.5, while library-conditioned routing reaches B|P 74.1 versus Claude-Opus-4.6 Best@3 at 74.6.Lib-Routing costs $0.004 per task, 13.0× less than Claude-Opus-4.6 Best@3.

Limitations

The benchmark’s scope and measurements are constrained by its Python focus, execution environment, and public Stack Overflow provenance.

  • The evaluation focuses on Python-based data science coding, excluding languages such as R and MATLAB whose efficiency patterns may differ.
  • Runtime measurements depend on hardware resources and software versions, so candidate solutions are evaluated in one controlled environment.
  • DSEffi-Bench derives tasks from public Stack Overflow posts, creating potential exposure to source content during model training.

A.6 Human-Validated Reference Solution

Human annotators construct reference solutions by evaluating accepted-answer strategies and selecting, refining, or replacing them when appropriate. The resulting references provide grounded baselines but are not guaranteed to be fastest on every task.

  • Annotation process: Annotators compare differing solutions and may select the more efficient one or compose a hybrid combining their best elements.
  • Reference solution relationships: 50.3% of reference solutions use an alternative strategy, 32.2% refine the accepted answer, and 17.5% faithfully adopt it.
  • Reference construction: 82.5% of references refine or replace the accepted answer’s strategy, so most are not direct implementations of the source answer.
  • Reference scope: The human-validated reference is not necessarily fastest for every task, although it serves as a correctness and efficiency baseline grounded in community knowledge and annotator expertise.

A.7 Metric Robustness Analysis

The study adopts log-scale normalization because task-level execution-time spans are highly variable, while model rankings remain robust across normalization strategies.

  • Normalization choice: The adopted log-scale spectrum score assigns equal score intervals to equal multiplicative speedups, preserving efficiency differences across wide runtime ranges.
  • Ranking robustness: Rankings are robust across normalization strategies, with Spearman correlations of ρ = 0.994 for linear versus log-scale and ρ = 0.961 for rank-based versus log-scale.
  • Runtime variation: The median maximum-to-minimum execution-time ratio is 14.5× across tasks with multiple correct solutions, reaching 522.0× at the 90th percentile.
  • Normalization choice: Linear normalization yields a median spectrum score of 0.93 among correct LLM solutions, whereas log-scale normalization yields 0.74.

B.1 Library Breakdown Analysis

Efficiency and correctness vary substantially by library, showing that model performance depends on library-specific optimization demands rather than a single aggregate ranking.

  • Per-library results: GPT-5.4 leads NumPy correctness at 76.8%, while Kimi-K2.5 leads NumPy B|P at 79.6% despite solving fewer than half the tasks.
  • Per-library results: Kimi-K2.5 leads Pandas B|P at 74.0%, while GPT-5.4-mini leads PyTorch correctness at 73.5%.
  • Per-library results: Polars has the sharpest efficiency divergence: DeepSeek-V4-Flash reaches 72.1% B|P, compared with 41.1% for DeepSeek-V3.2.
  • Pairwise efficiency: Pairwise comparisons show GPT-5.4 has the highest average win rate at 45.9%, while Kimi-K2.5 follows with 43.6%.

B.3.3 Model-Level Optimization Profiles

Model efficiency weaknesses shift across capability levels, while fast-side strengths and recurring overhead patterns reveal distinct optimization profiles.

  • Weakness profiles: 48.3% of DS-Coder-V2-Lite’s slow pairs are Loop-to-Vectorization, compared with 45.5% Overhead-Aware Stack Matching for GPT-5.4.The dominant weakness migrates from basic vectorization toward over-engineering as capability increases.
  • Weakness profiles: 45.5% of GPT-5.4’s slow pairs are Overhead-Aware Stack Matching, involving heavy imports, unnecessary validation, or defensive wrappers around vectorized code.GPT-5.4 nevertheless has the lowest median slowdown at 3.0×.
  • Weakness profiles: Data Layout Optimization dominates slow pairs for DeepSeek-V4-Flash (47.3%), Qwen3-Coder-Next (31.8%), and Qwen3.5-397B (31.4%).The profile treats data-layout awareness as distinct from algorithmic or API knowledge.
  • Strength profiles: Kimi-K2.5’s fast-side profile is Optimized Primitive Utilization-heavy (29.5%) and has the lowest Overhead-Aware Stack Matching share (5.1%).Its wins are consistent with direct, lightweight code paths.
  • Cross-model pattern: 49 of 868 pairs (5.6%) contain Numba or JIT decorators in the slow code, with median slowdown 1,131.0× versus 6.5× for non-JIT pairs.On small-to-medium DS inputs, JIT overhead can exceed the computation it targets.

B.3.4 Case Study

Case studies show that DS efficiency gains come from replacing slow implementation patterns with optimized primitives, reduced complexity, vectorization, better data layout, or lower-overhead stacks.

  • Optimized Primitive: 389.4× separates a triple-nested Python loop from scipy.spatial.distance.cdist for pairwise distances.cdist performs the identical computation in compiled C, eliminating Python-level iteration.
  • Complexity-Reducing: 183.8× separates per-matrix np.linalg.det calls from the closed-form determinant ad − bc for batched 2×2 matrices.The closed form uses four element-wise array operations and is fully vectorized across the batch.
  • Loop-to-Vectorization: 167.4× separates per-cell np.sum calls in a nested loop from nine-slice vectorization for 3×3 neighborhood sums.The vectorized version sums shifted array views in one compiled NumPy expression.
  • Data Layout: 95.5× separates DataFrame-based row replication from pre-allocated NumPy buffers that construct the DataFrame once.The DataFrame approach triggers repeated materialization and internal copies, whereas the NumPy approach scatters original rows into pre-allocated buffers.
  • Stack Matching: 218.4× separates identical tensor computation with GPT-5.4’s function-level import and dead branch from a leaner implementation.The algorithm is unchanged; repeated module lookup and dummy-tensor construction account for the overhead.

B.4 Memorization Analysis

The memorization analysis is a sensitivity check using temporal splits, while the supplied additional passages describe optimization experiments and reference implementation patterns.

  • Temporal split: 904 tasks precede the June 2024 cutoff and 96 follow it, so the post-cutoff analysis is interpreted as a sensitivity check rather than conclusive evidence against memorization.The limited post-cutoff subset constrains the strength of the conclusion.
  • Results: Pass rates drop by a mean −3.5% on post-cutoff tasks.The supplied result reports a slight correctness decline after the cutoff.
  • Results: B|P does not decline systematically: 8 of 16 models score higher on post-cutoff tasks, with rankings preserved at Spearman ρ = 0.87 for B|P and 0.95 for Pass.These results are consistent with the main correctness–efficiency decoupling.
  • Caveat: The analysis does not eliminate contamination concerns because all source posts remain public and the post-cutoff subset is relatively small.This is the stated scope boundary for interpreting the temporal comparison.
  • Optimization experiments: Effi-Learner iteratively profiles code, prompts the base model to optimize hotspot lines, and accepts only faster versions that retain correctness.Taxonomy-guided prompting augments this loop with category prescriptions and code examples.
  • Optimization experiments: Taxonomy-guided optimization produces library- and model-dependent gains, with the largest reported improvements of +23.0% for DeepSeek-V4-Flash on Others and +28.6% for Kimi-K2.5 on Others.PyTorch is the only library where the prescription occasionally underperforms vanilla Effi-Learner.

C.1 Evaluation Protocol

The evaluation uses controlled hardware and software, multiple correctness and stress tests, calibrated timing, and 16 models across three accessibility tiers.

  • Environment: The controlled environment uses dual Intel Xeon Platinum 8468V processors, 512 GB DDR5 RAM, four NVIDIA H200 NVL GPUs, and fixed library versions.The server runs Ubuntu 22.04 LTS and has no concurrent workloads during evaluation.
  • Test generation: Each task averages 26.8 correctness cases and 12.0 stress cases, for 38.8 total cases.These derive from 5.4 correctness sub-domains and 2.4 stress sub-domains with five unique tests per domain.
  • Test generation: N=5 test inputs exceed 95% line and branch coverage while limiting evaluation cost.Increasing from N=5 to N=7 adds only +0.9% line and +1.2% branch coverage.
  • Timing: Increasing timed runs from M=1 to M=3 reduces CV by 3.24% at W=1, while increasing warm-ups from W=1 to W=5 reduces CV by only 0.77% at M=3.The calibration therefore prioritizes timed repetitions over additional warm-ups.
  • Models: The evaluation covers 16 models grouped into Closed Frontier, Open Frontier, and Open Coder tiers.Closed models use proprietary APIs; open-weight models are served via vLLM.
Loading 2608.30248v1…