PaperlayerView exact revision on arXiv

Source-linked AI summary

CODA-BENCH: Can Code Agents Handle Data-Intensive Tasks?

Yuxin Zhang, Ju Fan, Meihao Fan, Shaolei Zhang, Xiaoyong Du

arXiv:2606.15300v1cs.AIcs.CL

TL;DR

Existing benchmarks isolate code or data skills, missing the combined challenge of discovering relevant files and executing code in realistic data-intensive environments. CODA-BENCH evaluates this integration and finds that top-performing agents achieve only 61.1% execution accuracy, showing substantial room for improvement.

  • Problem

    Existing benchmarks typically evaluate code intelligence or data intelligence in isolation, leaving limited evidence about agents’ ability to integrate both in large, realistic file systems.

  • Method

    CODA-BENCH uses Kaggle-derived data networks and verifiable tasks requiring agents to discover relevant files among distractors and write code to produce answers.

  • Results

    61.1% execution accuracy is achieved by the top-performing agents on CODA-BENCH, while agents still struggle to autonomously identify relevant data sources in large, unstructured datasets.

  • Takeaways & Limitations

    CODA-BENCH provides a foundation for evaluating integrated code and data intelligence, while current agents leave substantial room for improvement on data-intensive tasks.

  • Takeaways & Limitations

    CODA-BENCH is domain-biased toward curated, notebook-based Kaggle workflows and may not represent messier data or enterprise collaboration and production pipelines.

Abstract

from arXiv · show

Advanced agents are increasingly demonstrating the potential to operate as autonomous engineers, creating a growing demand for evaluation benchmarks that capture the complexity of real-world development. Such environments typically involve both complex code and large-scale data (i.e., file system). However, existing benchmarks usually evaluate code-centric or data-centric capabilities in isolation, leaving a clear gap with real development scenarios. In this paper, we bridge this gap by introducing CODA-BENCH, the first benchmark to jointly evaluate code and data intelligence in a data-intensive environment. We construct a data-intensive Linux sandbox based on the Kaggle ecosystem (containing hundreds of datasets), where agents must actively explore complex file hierarchies to identify relevant resources and generate code for data-driven analytical tasks. CODA-BENCH comprises 1,009 tasks spanning 31 communities, with each task environment containing an average of 980 files, simulating realistic data scale and noise. Evaluations of advanced agents reveal that even top-performing systems struggle to effectively integrate data discovery with code execution, achieving a success rate of only 61.1%. These results highlight a substantial gap in current agentic capabilities for data-intensive tasks and point to promising directions for future research.

1. Introduction

CODA-BENCH addresses the gap between code-centric and data-centric evaluation by jointly testing agents’ ability to discover relevant files and execute code in data-intensive environments. It introduces a Kaggle-based benchmark with 1,009 tasks and 980-file environments, where top agents achieve 61.1% execution accuracy, falling to 49.6% on CODA-HARD.

  • Motivation: Real-world autonomous agents must navigate directory hierarchies, identify relevant files among hundreds of candidates, and perform appropriate operations using both code and data intelligence.This capability extends beyond solving isolated algorithmic problems and requires agents to interact with large-scale file-system data.
  • Limitations of Existing Benchmarks: Existing benchmarks isolate code correctness from data processing, overlooking the coupled challenge of discovering and accessing large-scale data in shell-based environments.This separation creates a gap between benchmark performance and real-world utility, where data is rarely presented directly to agents.
  • CODA-BENCH: CODA-BENCH is the first benchmark to jointly evaluate agents’ code and data intelligence in a data-intensive environment.The benchmark leverages the Kaggle ecosystem to construct realistic environments rather than relying on trivially distinguishable random files or unscalable manual curation.
  • Results: 61.1% execution accuracy is achieved on CODA-BENCH, while accuracy falls to 49.6% on the more challenging CODA-HARD subset.Evaluations of Codex CLI, Claude Code, and Openhands reveal substantial limitations in autonomously completing data-intensive tasks.

2. Related Work

Related work has developed along two largely separate tracks: code-centric benchmarks have advanced toward realistic software-development settings, while data-centric benchmarks have focused on agents’ ability to understand and manipulate data.

  • Code-centric Benchmarks: Code-centric benchmarks have progressed from simple function-level tasks toward realistic software-development settings.Early efforts primarily measured functional correctness by executing unit tests on generated programs.
  • Data-centric Benchmarks: Data-centric benchmarks evaluate capabilities needed to understand and manipulate data.Early benchmarks primarily evaluated LLMs’ abilities to understand structured data.

3. Benchmark Construction

CODA-BENCH jointly evaluates code and data intelligence by placing agents in realistic, semantically noisy data environments and deriving verifiable tasks from human-authored Kaggle solutions. Its scalable construction combines graph-based dataset relationships, solution anchors, and adversarial task evolution to preserve both realism and difficulty.

  • Benchmark objective: CODA-BENCH is designed to test whether agents can discover relevant data and complete complex tasks through code in data-intensive environments.The benchmark addresses realistic development scenarios requiring both code intelligence and data intelligence.
  • Data environment construction: 21,122 Kaggle datasets are modeled through notebook co-occurrences into a relational graph, whose communities define semantically coherent evaluation environments.The graph represents datasets as nodes connected when they co-occur in notebooks; Leiden partitioning uses γ = 1.0 to form domain-specific communities.
  • Data environment construction: 980 data instances per environment include semantically similar distractors across CSV, JSON, Parquet, image, and PDF formats, with sizes from 20.3 MB to 45.4 GB.Distractors come from the same community as the target data, preventing superficial keyword or format-based filtering.
  • Task construction: Solution-based back-construction derives tasks from verified Kaggle notebook outputs called solution anchors, including statistics, rankings, correlations, and aggregations.Static analysis identifies candidate anchors, dynamic verification reconstructs their solution paths, and human annotators review generated questions for ambiguity.
  • Adversarial task evolution: State-of-the-art LLMs serve as both generator and discriminator in an adversarial evolution process that increases task difficulty while retaining solvability.The generator modifies tasks iteratively, while an ensemble of K discriminator models evaluates solve rates and successful or failed trajectories guide further refinement.

4. CODA-BENCH

CODA-BENCH evaluates autonomous agents on data-intensive analysis by requiring file-system discovery and code-based answer generation in realistic Linux sandboxes. It jointly measures data intelligence and code intelligence across diverse tasks and formats, including a harder subset that stresses multi-file discovery and non-trivial programs.

  • Task setup: Agents start from a sandbox root with only a natural-language objective and must autonomously discover relevant data without file-location, filename, or schema information.Each environment contains hundreds of data files, creating a realistic exploration-and-discovery setting.
  • Task setup: Each task requires identifying target files among semantically similar distractors, understanding diverse file structures, and writing code to derive the ground-truth answer.Tasks are formalized as T = (q, F, a∗), with F containing target and distractor files.
  • Evaluation metrics: CODA-BENCH measures Discovery Accuracy for locating all required data and Execution Accuracy for producing correct normalized answers through code.Together, the metrics separate data discovery from end-to-end completion requiring both discovery and correct execution.
  • Benchmark composition: 1,009 tasks span 31 communities and environments ranging from 10 to 8,158 files across CSV, JSON, Parquet, PDF, and image formats.The benchmark also includes 119 CODA-HARD tasks designed to challenge data and code intelligence simultaneously.
  • Benchmark composition: CODA-HARD requires discovering at least two target files and solving reference tasks exceeding 30 effective lines of code.These criteria create tasks requiring multi-source information integration and non-trivial programs.
  • Benchmark positioning: Unlike prior benchmarks supplying only oracle files, CODA-BENCH introduces large-scale, relevant yet uncurated data and jointly evaluates data and code intelligence.The design targets a more realistic assessment of discovering, selecting, and exploiting useful information in data-intensive environments.

5. Evaluation

CODA-BENCH evaluation shows that leading coding agents struggle to combine data discovery with code execution, especially when tasks require multi-file coordination. Performance also depends on model–agent alignment, with native and framework-based systems exhibiting different accuracy–cost trade-offs.

  • Main results: 61.1% EA is achieved by Mini-SWE-Agent with GPT-5.5, followed by OpenHands with GPT-5.5 at 59.7%.These are the highest execution-accuracy results among evaluated systems.
  • Main results: Agents struggle to autonomously identify relevant data sources within large, unstructured datasets despite strong coding capabilities on isolated benchmarks.Discovery Accuracy evaluates an agent’s ability to locate relevant resources.
  • Model–framework alignment: GPT-family models gain a 0.8% point EA improvement with Mini-SWE-agent over its native CLI, while Claude-family models perform better natively at 51.9% versus 49.3% EA.The results indicate that model–framework pairing affects performance.
  • CODA-HARD: 49.6% EA is achieved by Mini-SWE-Agent on CODA-HARD, while all models experience substantial degradation compared with the full benchmark.CODA-HARD requires coordination across multiple files and complex data-processing pipelines.
  • Cost efficiency: 53.8% EA at $0.11 per task is achieved by Claude Code with Sonnet-4.6, representing the most favorable native-CLI cost–performance ratio.Codex CLI with GPT-5.5 achieves 60.3% EA at $1.3 per task.

6. Analysis

The analysis shows that data discovery is a major source of difficulty, while code generation and reasoning remain challenging even when relevant files are provided. Environmental scale, framework efficiency, model capability, and failure profiles all substantially shape agent performance.

  • Environmental Characteristics: ρ = 0.466 for signal-to-noise ratio and ρ = −0.461 for data volume, with low-SNR communities performing worse and volumes above 3GB showing degradation.Several communities above 8GB drop to near-zero accuracy, indicating bottlenecks from large-scale data reading and exploration.
  • Data Discovery and Code Generation: 27.7 points: Claude Code (Sonnet-4.6) improves from 45.4% to 73.1% with oracle data, while OpenHands (GPT-5.5) improves by 24.4 points from 44.5% to 68.9%.The ablation confirms that discovering relevant files among thousands of candidates accounts for a substantial share of task difficulty.
  • Data Discovery and Code Generation: 71.0% average accuracy with oracle context leaves a 29.0% failure rate from code generation challenges including heterogeneous-schema integration, semantic ambiguity, and multi-step reasoning.These challenges persist even when the correct files are explicitly provided.
  • Interaction Efficiency: 60.3% EA in 6.8 rounds: Codex CLI is nearly as accurate as Mini-SWE-agent at 61.1% EA in 32.5 rounds and OpenHands at 59.7% EA in 18.1 rounds.Frameworks using the same underlying model can require nearly 5× different numbers of interaction rounds for comparable accuracy.
  • Failure Attribution: 44.0% of GPT-5.5 failures are code-generation errors and 33.0% are data-discovery errors, whereas Kimi-K2.6 has 40.7% discovery and 34.7% code-generation errors.Kimi-K2.6 also has more execution errors than GPT-5.5: 12.6% versus 6.5%.

7. Conclusion … B.1. Graph Construction and Community Detection

CODA-BENCH jointly evaluates agents’ code and data intelligence in Kaggle-based, data-intensive environments, where evaluations reveal substantial challenges. Its construction uses a co-occurrence graph of Kaggle datasets and notebooks, applies Leiden community detection, and curates 31 communities for practical analysis tasks while verifying dataset licensing.

  • 7. Conclusion: CODA-BENCH is introduced as the first benchmark jointly evaluating agents’ code and data intelligence.It leverages the Kaggle ecosystem to construct verifiable tasks and data-intensive environments.
  • 7. Conclusion: Evaluations reveal that current agents face significant challenges solving complex problems in data-intensive settings.The findings highlight substantial room for improvement in current agentic capabilities.
  • A. Ethical Statement: CODA-BENCH uses Kaggle datasets and notebooks distributed under open licenses permitting academic research and redistribution.The listed licenses include Creative Commons variants and Open Data Commons licenses.
  • A. Ethical Statement: The dataset licensing terms were carefully verified to ensure compliance with each dataset’s usage requirements.The paper states that personally identifiable information was excluded, although the supplied passage is truncated after this point.
  • B.1. Graph Construction and Community Detection: The co-occurrence graph represents Kaggle datasets as nodes and connects datasets appearing together in at least one notebook.Edge weights count notebook-based co-occurrences, capturing real-world dataset usage patterns.
  • B.1. Graph Construction and Community Detection: 529,739 files across all datasets include formats such as CSV, Excel, JSON, Parquet, images, and PDFs.Each dataset node may contain one or more individual data files.
  • B.1. Graph Construction and Community Detection: 323 communities were identified with the Leiden algorithm at resolution γ = 1.0, achieving modularity 0.711.The reported modularity indicates strong community structure, with within-community co-occurrences exceeding cross-community co-occurrences.
  • B.1. Graph Construction and Community Detection: 31 communities containing 829 datasets were selected for their relevance to practical data analysis tasks.The selection targets diverse real-world data science scenarios while preserving coherent thematic groupings.

B.2. Community Analysis

CODA-BENCH spans 31 communities across diverse data-science domains, with network structure showing clear domain-based clustering. Classic machine-learning benchmarks form the largest community, while a separate COVID-19 and global-geography community illustrates event-driven dataset ecosystem evolution.

  • Community coverage: CODA-BENCH covers 31 communities spanning foundational machine learning, healthcare analytics, entertainment recommendation, and geospatial pandemic analysis.The sampled communities demonstrate broad coverage across data-science domains.
  • Community structure: The community network exhibits clear clustering patterns corresponding to different data-science domains.Node colors indicate community membership, node sizes reflect notebook usage frequency, and edge widths represent co-occurrence strength.
  • Community 0: Classic ML Benchmarks: 154 datasets make Community 0, Classic ML Benchmarks, the largest and most interconnected community.It is anchored by foundational datasets including Iris Species, Pima Indians Diabetes Database, and Credit Card Fraud Detection.
  • Community 2: COVID-19 & Global Geography: 88 datasets make Community 2, COVID-19 & Global Geography, a community integrating pandemic statistics with geographical, demographic, and socioeconomic indicators.Its emergence reflects the surge in pandemic-related data science during 2020–2021 and supports comparative country-level analysis.

C. Example of Benchmark Construction … Final Task (Human Verification)

CODA-BENCH constructs tasks by anchoring questions in verifiable Kaggle results, adversarially evolving them through model evaluation, and applying human verification. The example shows that removing domain-specific hints increased difficulty while preserving a deterministic, authentic, and accepted final task.

  • C. Example of Benchmark Construction; Pseudocode and Workflow for Adversarial Evolution: Four advanced LLMs form the discriminator pool, with three sampled as discriminators and the remaining model generating each evolution strategy.The rotation mechanism is intended to prevent evolved tasks from exploiting one model’s weaknesses.
  • C. Example of Benchmark Construction; Step 1: Solution Anchor Identification; Step 2: Initial Question Generation (Iteration 0); Step 5: Human Verification and Finalization: The construction pipeline extracts numerical solution anchors, generates an initial question, iteratively increases difficulty, validates solve-rate degradation, and performs human verification.The five stages are solution-anchor identification, question generation, iterative evolution, difficulty validation, and final quality checking.
  • Source: Kaggle Notebook Solution; Step 1: Solution Anchor Identification: The Netflix notebook anchor reports five leading TV-show categories: International TV Shows, 1199; TV Dramas, 704; TV Comedies, 525; Crime TV Shows, 427; and Kids’ TV, 414.Re-execution confirms deterministic reproducibility with ϵ < 10−6.
  • Step 2: Initial Question Generation (Iteration 0); V0: Initial Question (LLM Generated): 100% solve rate at V0 made the initial question too easy, as all three sampled discriminators answered it correctly.The generator identified redundant context and an explicit hint revealing the splitting approach.
  • Step 3: First Evolution Iteration; V1: First Iteration (Remove Redundancy): 100% solve rate at V1 persisted after removing redundancy, so evolution continued because column-name and domain hints still exposed the solution context.Human review approved the V1 question as unambiguous and non-lossy.
  • Step 4: Second Evolution Iteration; V2: Second Iteration (Generalize Terms): 66.7% solve rate at V2 showed increased difficulty after generalizing “listed categories” to “assigned genres” and removing “title” from the count description.Two of three newly sampled discriminators were correct, and further generalization was judged ambiguous.
  • Step 5: Human Verification and Finalization; Final Task (Human Verification): The final task asks for the top five most frequently assigned genres for TV shows and their counts, with the anchored answer preserved exactly.It was accepted after 2 evolution iterations and 9 total evaluations.

D. Tasks Illustration

CODA-BENCH tasks combine a natural-language analytical question with an intensive data environment, requiring agents to discover relevant files, implement an analytical solution, and produce an answer. The representative task illustrates this process through a noisy multi-dataset file hierarchy and detailed metadata for a relevant dataset.

  • Task specification: Each task provides a natural-language analytical objective and an intensive data environment for the agent to analyze.The agent must autonomously explore the environment, determine an appropriate analytical approach, implement it in code, and produce the final answer.
  • Task specification: The example question asks how many rows represent cancellations and what percentage of all rows they constitute.
  • Data environment: The provided hierarchy contains datasets for traffic, online retail, credit cards, fraud detection, and ecommerce, totaling 441 files across 154.
  • Data environment: The relevant OnlineRetail.csv file is 45.6 MB with 541,909 rows, 8 columns, and Quantity listed among its relevant columns.
  • Data environment: The task environment has a signal-to-noise ratio of 0.002 and high distractor similarity.

E. Evaluation Sandbox … 6. Security (3 Levels)

CODA-BENCH uses a reproducible Docker sandbox with standardized resources, agent configurations, and security controls, then illustrates capabilities and failure modes through case studies and prompt-driven task construction. The cases show that correct data discovery and processing enable success, whereas dataset selection and missing-value semantics can produce incorrect results.

  • E. Evaluation Sandbox; E.1. Evaluation Environment: The benchmark provides a production-grade, reproducible evaluation environment that mirrors real-world development while maintaining experimental control.Docker containers isolate standardized evaluation runs and reduce system-level confounding factors.
  • E.1. Evaluation Environment: Each sandbox includes Python 3.11, curated data-science and file-format libraries, essential system tools, and a standardized workspace containing task instructions, data, and results.Agents may install additional packages within the environment.
  • E.1. Evaluation Environment: 4GB memory, 2 CPU cores, a 600-second timeout, and read-only data directories constrain execution and prevent destructive data modification.These limits simulate practical computational constraints and prevent agents from circumventing task requirements through data changes.
  • E.2. Agent Configurations: The evaluation covers commercial CLI tools Claude Code (v2.1.150) and Codex CLI (v2.3.1), plus OpenHands v1.7 and mini-swe-agent v2.0.0.Commercial tools use default configurations, while the open-source frameworks use temperature 0.
  • F. Case Study; Success Case: Qatar; 125,000.0 was correctly produced after environment exploration, dataset discovery, sorting verification, reproducible scripting, and proper output formatting.The success factors were correct dataset identification, accurate data processing, and proper output formatting.
  • Failure Case; Dataset Discovery Error: 10 competitions matched the requested generative-AI keywords, but selecting an incomplete dataset yielded an agent answer of 2.The selected dataset ended in September 2022, missing nine months of the requested range through June 2023; meta_kaggle produced the correct answer.
  • Failure Case; Data Processing Semantic Error: A missing-value semantic error biased BTX estimates and produced factor 2 instead of 3, while the O3 trend analysis remained correct and classified as Stable.Using min_count=1 treated missing pollutant measurements as zero; NaN propagation was the correct implementation.

H. Limitations and Future Directions

CoDA-Bench is limited by its Kaggle-centered scope, curated data, and potential contamination from public datasets. The authors argue that its data-discovery challenge and construction principles can generalize, motivating broader domains, newer datasets, and private data sources.

  • Current Scope and Domain Bias: Kaggle emphasis on exploratory analysis and predictive modeling may underrepresent ETL pipelines and real-time analytics, while its curated datasets differ from messier real-world repositories.The benchmark also reflects single-user Kaggle workflows, as stated in the passage.
  • Future Directions: CoDA-Bench targets a generalizable challenge—discovering relevant data in large, noisy environments—and uses domain-agnostic construction principles.These principles are community construction via co-occurrence analysis, solution-based back-construction with verifiable outputs, and adversarial evolution via model-based difficulty control.
  • Contamination Risk: Public Kaggle datasets create contamination risk because models may recognize dataset schemas or common patterns despite tasks requiring precise computation.Planned mitigations include periodically releasing versions using recent post-training-cutoff datasets and analyses, and expanding to private-domain enterprise data.
Loading 2606.15300v1…