Source-linked AI summary
ORCA: Evaluating LLMs on Data Science Code Translation
Xiaolong Li, Jinyang Li, Bowen Qin, Ge Qu, Nan Huo, Xiaohan Xu, Shipei Lin, Reynold Cheng
TL;DR
Cross-library code translation must preserve functional behavior, including across multi-stage projects. ORCA benchmarks grounding- and project-level translation with executable tests, and its experiments find persistent challenges for current LLMs.
Problem
DSCT must preserve syntactic correctness and semantic integrity across libraries, with project translation also requiring consistency across dependent files and stages.
Source: §1 · p. 2Method
ORCA provides 1,600 grounding-level tasks across three domains and 200 complete-project tasks across seven task types, with reference translations and executable tests.
Source: §1 · p. 2Results
DSCT remains challenging: Claude-Opus-4.6 achieves success rates of 56.92% on ORCA-MAIN and 33.67% on ORCA-PROJECT.
Source: §1 · p. 2Takeaways & Limitations
Current systems still fall short of the reliability required for practical DSCT, with directional asymmetry, data complexity, and cross-stage mismatches recurring as difficulty sources.
Source: §7 · p. 9Evaluating the 1,600 grounding-level and 200 project-level tasks has a modest carbon footprint, with large proprietary-model evaluations requiring significant energy.
Source: L Limitations · p. 27
Abstract
from arXiv · showhide
Data Science Code Translation (DSCT) is the process of converting code between data science libraries while preserving functional equivalence and enabling interoperability across data science ecosystems. While Large Language Models (LLMs) have demonstrated considerable progress in Data Science Code Generation (DSCG), their performance in DSCT remains insufficiently studied. To address this gap, we introduce ORCA, a comprehensive benchmark with two complementary settings: ORCA-MAIN, which comprises 1,600 carefully curated grounding-level tasks across 3 representative domains: Data Querying, Data Manipulation, and Deep Learning; and ORCA-PROJECT, which contains 200 translation tasks over complete data science projects across 7 data science task types. Each task is accompanied by annotated reference translations and test cases for validating functional equivalence. We further incorporate a multi-stage quality verification process that thoroughly verifies task correctness and test case robustness. Experimental results demonstrate challenges in DSCT, with even frontier LLMs showing limited performance. Specifically, Claude-Opus-4.6 achieves a success rate of 56.92% on ORCA-MAIN and 33.67% on ORCA-PROJECT, indicating considerable room for improvement in DSCT. We also observe a clear directional preference in DSCT, where translation is consistently easier when the source code expresses the task through more explicit, fine-grained operations. Motivated by this, we propose an intent-augmented method, in which the model first infers source-code intent and then uses it as additional context for translation, achieving average absolute success-rate gains of 4.80% and 5.33% on ORCA-MAIN and ORCA-PROJECT, respectively.
1 Introduction
Data science code translation must preserve behavior across libraries, including semantic details and consistency across project stages. ORCA evaluates this task at grounding and project levels, and its experiments show that current LLMs remain challenged.
1 Introduction
- Data scientists need code translation for environment migration, evolving library ecosystems, and modernization of projects using outdated or unmaintained libraries.Examples include rewriting Pandas analysis in SQL for production-database workflows and coordinating migrations across whole projects.Source: §1 · pp. 1–2
- DSCT requires both syntactic correctness and semantic integrity, while project translation must remain consistent across interdependent files and workflow stages.Semantic integrity includes preserving transformation logic, indexing behavior, and computational results.Source: §1 · p. 2
- ORCA comprises 1,600 grounding-level tasks in three domains and 200 project-level tasks across seven task types, with reference translations and executable tests.A multi-stage quality-verification process checks reference correctness and test-case validity.Source: §1 · p. 2
- Claude-Opus-4.6 achieves success rates of 56.92% on ORCA-MAIN and 33.67% on ORCA-PROJECT, while intent-augmented translation improves success at both levels.The reported absolute gains are 4.80% on ORCA-MAIN and 5.33% on ORCA-PROJECT.Source: §1 · p. 2
- Analyses identify translation direction, data complexity, and cross-stage data mismatches as factors shaping DSCT difficulty.The paper also reports that making source-code intent explicit can improve translation success at both levels.Source: §1 · p. 2
2 Preliminaries
DSCT translates code between data-science libraries while preserving functional equivalence. The section distinguishes snippet-level and project-level tasks and defines the success-rate evaluation metric.
2 Preliminaries
- DSCT converts code between data-science libraries while preserving functional equivalence to the source code.Tasks pair the code with input data and a translation requirement specifying source and target settings.Source: §2 · pp. 2–3
- Grounding-level tasks translate snippets between one library pair, whereas project-level tasks cover interdependent files and stages that may use multiple libraries.Source: §2 · p. 3
- Success Rate (SR) is Pass@1: the percentage of evaluation tasks whose translated code meets all testing criteria.Source: §2 · p. 3
3 Benchmark Construction
ORCA is constructed through source collection, task curation and reference translation, followed by executable testing and independent quality verification.
3.1 ORCA-MAIN
- ORCA-MAIN draws data-querying tasks from BIRD-SQL and LiveSQLBench and data-manipulation and deep-learning snippets from Stack Overflow, with inputs prepared for execution.Annotators construct missing input data for some Stack Overflow tasks, and each task specifies a source–target library direction.Source: §3.1 · pp. 3–4
- ORCA-MAIN curation standardizes library versions, transforms operations to reduce direct replication, adds underrepresented operations, and retains tasks implementable in both libraries.Reference translations reimplement source functionality rather than translating line by line; executable suites test both libraries.Source: §3.1 · p. 4
3.2 ORCA-PROJECT
- ORCA-PROJECT sources complete workflows from GitHub and Kaggle, then pairs each task with local-database inputs and a skeleton requirement specifying project structure and interface constraints.The requirement supports execution and verification despite project-level interface and dependency mismatches.Source: §3.2 · p. 4
- Project curation adapts workflows to compatible local databases, reconstructs queries and dependencies, and configures Docker environments to fix versions and services.Source: §3.2 · pp. 4–5
- Reference projects reimplement the same workflow with selected target libraries, while paired tests check both individual stages and cross-stage consistency and final outputs.If no target library supports a required function, annotators may compose multiple APIs; unsupported projects are filtered out.Source: §3.2 · p. 5
3.3 Multi-Stage Quality Verification
- An independent second annotator tests suite robustness and code reliability; after adjudication, inconsistent tasks are excluded, yielding 1,600 ORCA-MAIN and 200 ORCA-PROJECT tasks.Pre-adjudication agreement is 93.2% for grounding-level tasks and 89.7% for project-level tasks.Source: §3.3 · p. 5
4 Dataset Statistics
The benchmark composition is summarized by ORCA-MAIN operation groups and ORCA-PROJECT task categories; Table 1 is identified as reporting key statistics.
4 Dataset Statistics
- ORCA-MAIN groups data-querying tasks into ranking, aggregation, counting, and match-based queries; data manipulation into creation, transformation, computation, and indexing; and deep learning into five operation types.The deep-learning types are initialization, training, optimization, evaluation, and inference.Source: §4 · p. 5, Figure 3 · p. 6
- ORCA-PROJECT covers classification, regression, clustering, anomaly detection, classic reinforcement learning, recommendation, and dimensionality reduction.Source: §4 · p. 5
- Table 1 reports key statistics for ORCA-MAIN and ORCA-PROJECT, while Figure 3 summarizes their composition.Source: §4 · p. 5, Table 1 · p. 5, Figure 3 · p. 6
5 Experiment
The experiments evaluate LLMs on grounding-level and project-level translation, finding limited performance, directional and project-complexity challenges, and gains from adding inferred code intent.
5.1 Experimental Setup
- Models receive source code, input data, and translation requirements; project tasks use a ReAct-style agent with file I/O, code generation, and execution tools.Each task runs three times per model, with mean success rate and standard deviations reported.Source: §5.1 · p. 6
5.2 ORCA-MAIN Results
- Claude-Opus-4.6 reaches 56.92% overall on ORCA-MAIN and 33.67% SR on ORCA-PROJECT, while project-level translation lowers performance for all evaluated models.Table 2 reports ORCA-MAIN performance; Figure 5 shows ORCA-PROJECT baseline performance.Source: Table 2 · p. 6, §5.2 · p. 7, §5.4 · p. 7
5.3 Directional Preferences in DSCT
- Translation success varies by direction: Pandas→PostgreSQL, NumPy→Pandas, and TensorFlow→PyTorch outperform their reverse directions.The authors hypothesize that explicit, fine-grained source operations are easier to translate; the directional gap is most pronounced for Pandas→NumPy.Source: §5.3 · p. 7, Figure 4 · p. 7
5.4 ORCA-PROJECT Results & Analysis
- Across four analyzed models, average SR falls from 35.23% to 18.38% as project data complexity increases.Classification-task average SR also declines, from 36.25% to 15.91%, between simple and complex data.Source: §5.4 · p. 8
- Thirty-eight percent of analyzed project-level failures involve cross-stage data mismatches, including incompatible dtypes, column names, shapes, or value ordering.Mitigation requires preserving data consistency across sequential operations and stages.Source: §5.4 · p. 8
5.5 Augmenting DSCT with Code Intent
- Intent-augmented translation adds inferred natural-language intent as context; for projects, it combines module-level intent and data flow into project-level guidance.The approach improves average SR by 4.80% on ORCA-MAIN and 5.33% on ORCA-PROJECT across five models; CoT-Intent results vary by model.Source: §5.5 · pp. 8–9, Table 3 · p. 9
6 Related Work
Prior code-translation benchmarks emphasize programming-language migration, while data-science benchmarks emphasize generation and agentic analysis. Neither directly evaluates broad, execution-grounded translation across data-science libraries and projects.
Code Translation Benchmarks
- General code-translation benchmarks cover programming-language migration but do not capture data-centric semantics, library-specific operations, or workflow dependencies characteristic of DSCT.Earlier rule-based and aligned-data machine-learning approaches faced limitations in handling complex constructs or data availability and quality.Source: §6 · p. 9
Data Science Code Benchmarks
- Data-science code benchmarks evaluate generation or agentic analysis, not whether existing implementations translate across libraries while preserving functional equivalence.CodeTransOcean includes a deep-learning translation subset, but realistic execution-grounded evaluation across diverse tasks and project-level settings remains underexplored.Source: §6 · p. 9
7 Conclusion
The conclusion presents ORCA as a grounding- and project-level benchmark for data-science code translation and summarizes current challenges and intent-based gains.
7 Conclusion
- ORCA evaluates data-science code translation at grounding and project levels, using executable test suites to verify functional equivalence across library pairs.The authors identify directional asymmetry, data complexity, and cross-stage data mismatches as recurring difficulties; intent-based translation improves performance, but systems remain short of practical reliability.Source: §7 · p. 9
A.1 Data Statistics
ORCA combines benchmark coverage, executable validation, and annotation procedures to examine code translation across data-science libraries, projects, and translation directions.
A.1 Data Statistics
- ORCA combines bidirectional, execution-grounded, data-science-specific cross-library translation with both grounding-level and project-level settings.Compared with prior DSCT resources, it covers more tasks, longer and more complex solutions, and more diverse real-world data sources.Source: §A.1 · p. 14, Table 4 · p. 14
B. Annotation and Evaluation Protocol
- Annotators completed structured training and qualification; construction and verification required approximately 1,400 person-hours across the benchmark.Six annotators built the benchmark and three senior experts oversaw verification.Source: §B.1 · p. 15, §B.5 · p. 16
B.2 Data Collection
- ORCA-MAIN draws DQ tasks from SQL benchmarks and DM/DL tasks from Stack Overflow, while ORCA-PROJECT selects complete workflows from GitHub and Kaggle and migrates inputs to value-equivalent formats.Project candidates are screened for popularity and completeness, including a workflow from data loading through modeling and evaluation.Source: §B.2 · pp. 15–16, §B.3 · p. 16
B.4 Test Case Design
- Grounding-level tests check relational outputs and data or model behavior; project tests also check cross-stage consistency, reproducibility, and full-pipeline execution.Translation pairs are iteratively refined and retained only when both implementations pass their associated tests.Source: §B.4 · p. 16
C Case Studies
- Intent-augmented variants succeed on representative DQ, DM, and DL cases where baseline prompting fails, with CoT-Intent succeeding on DQ and Intent on DM and DL.The cases stress output ordering and formatting, indexing and aligned computation, and model structure and training-state semantics, respectively.Source: C Case Studies · p. 17
D Pass@k Cost Trade-off
- <3% marginal SR gain from Pass@k on ORCA-MAIN comes with almost triple the inference cost for the evaluated Gemma-3-27B and DeepSeek-R1 models.Pass@k uses three independently sampled outputs and counts a task successful if any output passes execution-based evaluation.Source: Table 7 · p. 18, D Pass@k Cost Trade-off · p. 18
E Dialect-to-Dialect SQL Translation
- 53.22% average SR in dialect-to-dialect SQL translation versus 19.06% for Pandas↔PostgreSQL, a 34.15-point difference across the same seven models.The comparison macro-averages each model’s mean across translation directions.Source: Table 8 · p. 18, E Dialect-to-Dialect SQL Translation · p. 18
- 8.02 versus 12.98 average bidirectional gap points to smaller directional asymmetry for SQL dialects than for Pandas↔PostgreSQL.The reported reduction is 4.96 percentage points.Source: Table 8 · p. 18, E Dialect-to-Dialect SQL Translation · p. 18
F ORCA-PROJECT Baseline Results
The project-level baseline evaluation reports results for a representative subset of models, reflecting the higher cost of evaluating this setting.
F ORCA-PROJECT Baseline Results
- A representative subset of models was evaluated because project-level agentic evaluation is substantially more costly.Source: F ORCA-PROJECT Baseline Results · p. 18
- Table 9 reports baseline ORCA-PROJECT success rates as mean ± standard deviation over three runs, with the best results marked in bold.The supplied passages do not provide the table’s cell values.Source: Table 9 · p. 19
G ORCA-PROJECT Difficulty by Data Complexity
The ORCA-PROJECT analysis associates higher data complexity with lower success rates across tasks and within Classification.
G ORCA-PROJECT Difficulty by Data Complexity
- 35.23% in the Low bin versus 18.38% in the High bin: average SR decreases as data complexity increases across ORCA-PROJECT tasks.The complexity score multiplies the number of distinct input sources by the number of input fields.Source: Table 10 · p. 19, G ORCA-PROJECT Difficulty by Data… · p. 19
- 36.25% in the Low bin versus 15.91% in the High bin: Classification tasks show the same average-SR decline with increasing data complexity.Source: Table 11 · p. 19, G ORCA-PROJECT Difficulty by Data… · p. 19
H ORCA-MAIN Baseline Detailed Results
The additional analyses examine directional performance, translation difficulty, model and prompt choices, and execution feedback across ORCA’s settings.
H ORCA-MAIN Baseline Detailed Results
- Across models, Pandas→PostgreSQL, NumPy→Pandas, and TensorFlow→PyTorch consistently outperform their reverse directions.Table 12 reports mean SR across three runs for each direction across DQ, DM, and DL.Source: Table 12 · p. 20, H ORCA-MAIN Baseline Detailed Results · p. 20
I.1 Positioning and Coverage of DSCT
- DSCT is a data-centric specialization of code translation: ORCA-MAIN tests operations, while ORCA-PROJECT tests semantic consistency across complete workflows.ORCA-MAIN covers five libraries selected for adoption, workflow-stage coverage, and bidirectional executable evaluation; project tasks broaden coverage to 21 libraries and seven categories.Source: §I.1 · pp. 20–21
I.1.1 Direct and non-direct operation correspondence
- The easier DQ and DM directions have lower non-direct operation rates; in DL, high direct-correspondence rates coexist with lower PyTorch→TensorFlow SR.The DL direction gap is consistent with TensorFlow-specific program and execution structure discussed in the paper.Source: §I.1.1 · p. 21, Table 13 · p. 21
I.2 Task Granularity and Structural Complexity
- SR decreases with structural complexity in every domain, with a sharper decline in DL amid additional shape, layout, dtype, device, and framework-state constraints.ORCA-MAIN complexity groups are defined separately by domain-specific operations and dependencies.Source: §I.2 · pp. 21–22, Table 15 · p. 22
I.3.1 GPT-family model coverage
- GPT-5.5 with xhigh reasoning matches or exceeds Claude-Opus-4.6 in all six ORCA-MAIN directions and performs better on ORCA-PROJECT, while failures and directional preferences remain.Source: Table 16 · p. 22, §I.3.1 · p. 22
I.3.2 Functional-equivalence and library-version prompts
- Detailed functional-equivalence criteria improve overall SR for both tested models, whereas version prompts have mixed results and do not explain most failures.The criteria specify behavior to preserve, not how to implement it in the target library.Source: §I.3.2 · pp. 22–23, Table 17 · p. 23
I.3.3 Execution-feedback agent on ORCA-MAIN
- Execution feedback improves every domain; DQ benefits most on Pandas→PostgreSQL, while DM gains less because visible-value matches can still violate structural requirements.The ORCA-PROJECT ReAct-style agent was also evaluated on ORCA-MAIN, making results conditional on the inference configuration and harness.Source: §I.3 · p. 22, Table 18 · p. 23, §I.3.3 · p. 23
I.3.4 Frontier coding-agent evaluation
Claude Code improves translation performance at both task granularities, but substantial failures remain, particularly when validation misses source–target mismatches and projects must preserve consistency across stages. The analysis also documents domain-specific failure patterns and the benchmark’s construction and evaluation artifacts.
I.3.4 Frontier coding-agent evaluation
- Claude Code consistently improves both granularities by reconstructing source behavior, translating, executing source and candidate implementations, and consulting documentation.Across 300 ORCA-MAIN tasks, it executes 1,358 Bash commands and accesses documentation in 292 trajectories; on ORCA-PROJECT, all 100 trajectories inspect the source implementation.Source: §I.3.4 · p. 24
- Approximately 30× and 3.5× the corresponding baseline costs accompany the ORCA-MAIN and ORCA-PROJECT gains, respectively.Agentic performance also reflects the harness, tool interfaces, exploration strategy, and turn budget.Source: §I.3.4 · p. 24
I.4.1 Failures across prompt settings
- DQ failures center on query logic (63.5%) and numeric, type, or null behavior (33.8%); DM and DL failures center on representation and model-interface problems, respectively.DM failures concentrate on shape or layout (37.8%) and output or schema interfaces (36.6%); DL failures concentrate on model interfaces (34.3%) and incomplete implementations (31.9%).Source: §I.4.1 · p. 24
- 44.0% of 600 matched model-task pairs fail consistently across all three prompts, and 90.9% of those pairs retain the same fine-grained failure type.The Detailed-Criteria prompt rescues 39 baseline failures but loses 26 passes; the Library-Version prompt rescues 26 failures and loses 26 passes.Source: §I.4.1 · p. 24
I.4.2 Claude Code failures and validation behavior
- Claude Code reduces absolute failures in all three categories, but remaining errors retain domain-specific patterns: DQ logic failures, DM data/interface mismatches, and DL data/interface or incomplete-implementation failures.Among remaining failures, 26 of 31 DQ cases involve logic or behavior, 50 of 55 DM cases involve data or interface mismatches, and 31 of 56 DL cases involve data or interface mismatches.Source: §I.4.2 · p. 25
- Among 142 failed ORCA-MAIN tasks, 91 execute only the candidate; among 74 failed ORCA-PROJECT tasks, 73 do so, while 35 cross-stage failures remain.The analysis recommends equivalence-aware validation: run source and target on matched inputs and compare relevant outputs and producer-consumer contracts.Source: §I.4.2 · p. 25
I.4.3 Project-level failure taxonomy
- Cross-stage project failures most often involve column or schema mismatches and train–inference misalignment, while local-data failures often involve field extraction and missing-value handling.Subtype labels are multi-label, so their distributions need not sum to 100%.Source: §I.4.3 · p. 26
I.5 Construction Automation and Analysis Artifacts
- Expert annotators construct reference translations and executable tests, with independent checks and senior-expert adjudication when needed; project instances use reconstructed local database inputs.The process required approximately 1,400 person-hours, and released test-level outcomes identify which properties remain incorrect in near-passing translations.Source: §I.5 · p. 26
K Reproducibility Statement
The paper reports its evaluation settings and provides prompt templates for grounding-level and project-level translation. It also states the benchmark’s evaluation scope, ethical considerations, and intended uses.
K Reproducibility Statement
- Each task is run three times per model, with mean Success Rate and standard deviation reported; temperature is 0.0 and top_p is 1.0.The reproducibility statement also specifies SDKs and inference platforms, and reports approximately 200 local GPU hours and USD 1,000 in API costs.Source: K Reproducibility Statement · p. 27
L Limitations
- The benchmark covers 1,600 grounding-level tasks and 200 project-level tasks, and comprehensive evaluations of large proprietary models require significant energy.The authors describe the resulting carbon footprint as modest and identify more efficient evaluation protocols as a future direction.Source: L Limitations · p. 27
M Ethics Statement
- The benchmark uses publicly available data, avoids sensitive or personally identifiable information, and relies on inference without additional model training.The paper also reports reviewing data for politically sensitive or ethically concerning material.Source: M Ethics Statement · p. 27
N Broader Impact
- ORCA is intended to help identify DSCT failure modes, guide safer tooling, and reduce manual migration effort across libraries and back ends.The paper characterizes negative social-impact risk as relatively low because tasks concern code translation between established libraries.Source: N Broader Impact · p. 28
Baseline Prompt
- The baseline prompt asks for equivalent Pandas code from a PostgreSQL query, supplies schema and code, and requires table loading and a result variable.It requests code only and shows conversion of the final dataframe to a list of tuples.Source: Baseline Prompt · p. 29
Intent Prompt (Step 1: Intent Acquisition)
- The intent prompt elicits schema use and query logic, while the CoT-intent variant requests a stepwise account of the data and operations.Both ask for specificity about queried data and how it is used, while avoiding specific SQL function names where possible.Source: Intent Prompt (Step 1: Intent… · p. 30, CoT-Intent Prompt (Step 1: Intent… · p. 31
Shared Prompt (Step 2: Translate with Intent)
- The shared translation prompt retains the PostgreSQL-to-Pandas task requirements and adds the previously generated intent as context.The prompt still requests Pandas code only.Source: Shared Prompt (Step 2: Translate with… · p. 32
Action Generation Prompt
- Project-level prompts support file reading, writing, execution, and submission, while hierarchical intent prompts describe pipeline stages, per-file roles, and cross-stage mappings.The thought prompt focuses on the current bottleneck, data-flow consistency, and a useful next step.Source: Action Generation Prompt · p. 33, Hierarchical Intent Generation Prompt · p. 35