Collaborative Evolving Strategy for Automatic Data-Centric Development

• Data-driven discovery is presented as an emerging paradigm that accelerates scientific advances, while LLM-based AI agents make reducing its impediments more attainable.Source: §1 · p. 1
• AD2 asks agents to prioritize candidate data-development methods, prepare engineering implementations, and execute them to obtain accurate results.Source: §1 · p. 2
• AD2 requires resource-aware scheduling across many candidates and implementation of complex research tasks that demand domain-specific knowledge learned through practice and effectively reused.Source: §1 · pp. 2–3
• The paper argues that scheduling and implementation should evolve cooperatively through practice: implementation feedback improves schedules, while sensible schedules accelerate implementation learning.Source: §1 · p. 3
• Co-STEER couples a scheduling agent that learns dependencies and implementation feedback with an implementation agent that builds transferable practical knowledge, while sharing status and feedback.Source: §1 · p. 3
• Implementation correctness rose from 0.454 to 0.646, a 44.5% improvement over previous SOTA methods; scheduling assistance boosted performance by 24% on average across different settings.Source: §1 · p. 3

2.1 Agent Workflows

• Unlike planning methods that decompose tasks into steps, Co-STEER prioritizes candidate steps, a planning aspect that receives little attention.Source: §2.1 · p. 3
• Co-STEER uses feedback from accumulated real-experiment experience in a knowledge base, with a mechanism for transferring that knowledge across tasks.Prior work may use textbook knowledge or no knowledge base.Source: §2.1 · pp. 3–4
• Rather than merely calling existing tool APIs or learning to use existing tools, Co-STEER combines tools to build new ones and uses practical experience to support tool learning.Source: §2.1 · p. 4

2.2 Agents in Related Scenarios

• Prior work [5] shows LLM potential in scientific research through case studies with human evaluation, while presents a systematic approach to idea formulation; this work studies the less-explored development problem, especially data-centric development.Source: §2.2 · p. 4
• Unlike agents for feature engineering [14], hyper-parameter tuning [46], or broader machine-learning tasks [15], this work addresses prioritizing candidate methods and preparing engineering implementations for data-centric development.Source: §2.2 · p. 4
• Unlike software-development agents such as GitHub Copilot and AutoDev [34], this work addresses both prioritization and implementation in scientific data-driven development, using an evolving knowledge base.The role of an evolving knowledge base in software-development agents is not clearly discussed.Source: §2.2 · p. 4

3.1 Problem Formulation

• AD² agents seek to complete as many valuable code-implemented tasks as possible within limited trials, with execution results evaluated against ground truth.The formulation values both the number of completed tasks and their quality.Source: §3.1 · p. 4

3.2 Overall Design

• Co-STEER pairs scheduling and implementation agents that evolve collaboratively: implementation feedback improves schedules, while optimized schedules support implementation learning.The scheduling agent iteratively refines task order, while the implementation agent builds knowledge transferable across tasks.Source: §3.2 · p. 5

3.3 Scheduling Agent

• The scheduler weighs task complexity and dependencies, uses guided CoT to map a justified task DAG, and revises priorities from implementation feedback such as repeated failures.The workflow encourages consideration of additional factors that may affect scheduling.Source: §3.3 · pp. 5–6

3.4 Implementation Agent

• Because AD² tasks require domain-specific knowledge beyond standard coding, the implementation agent grows a practical knowledge base from experience to support other tasks.The agent also incorporates general implementation capabilities used in previous work.Source: §3.4 · p. 6

3.4.1 Knowledge Base Design

• Its knowledge base stores successful tasks with trial-and-feedback traces, then retrieves corrective steps by feedback similarity—not task similarity—using embeddings to transfer fixes across complex tasks.Retrieved experience can identify a solution to a current error even when the prior task differs.Source: §3.4.1 · pp. 6–7

3.4.2 Feedback Design

• Feedback combines LLM and Python-execution checks with expert-ground-truth comparisons of solutions and execution metrics, including correlation and value accuracy; expert-crafted examples warm-start ongoing unsupervised learning.Source: §3.4.2 · p. 7

4.1 Datasets

• The experiments use RD2Bench, with 27 implementable and 13 mistakenly extracted unimplementable financial factors across fundamental, high-frequency, and price-volume categories.The factors are categorized as easy, medium, or hard, and their implementations require different data sources; the domain has many published dataset-development methods [17].Source: §4.1 · p. 7

4.2 Experimental Settings

• Implementation baselines are Few-shot, CoT [37], Reflexion [32], Self-Debugging [8], and Self-Planning [16].The methods use task examples, step-by-step reasoning, introspection, code-feedback correction, or autonomous planning, respectively.Source: §4.2 · p. 8
• Implementation is evaluated using average execution, average format, average correlation, and maximum correlation.Execution scores count errors as zero; format checks expected output structure, while correlation metrics compare generated factor series with ground truth.Source: §4.2 · p. 8

4.3 Results of Method Implementation (RQ1)

• Across 27 test cases, Co-STEER significantly outperforms baseline workflows on all four implementation metrics.The authors attribute the gains to dynamically expanded knowledge and retrieval of practical experience, unlike baselines that use only current-attempt feedback.Source: §4.3 · p. 8, Table 2 · p. 8

4.4 Overall Results of AD2 (RQ2)

• Co-STEER with an evolving scheduler significantly outperforms Co-STEER with a random scheduler, showing that scheduling and implementation order affect overall AD^2 performance.The agent learns which methods are hard to implement and whether one method’s implementation can facilitate another’s.Source: §4.4 · pp. 8–9, Table 3 · p. 9
• Co-STEER’s performance improves with more computational resources under both random and evolving schedulers, while baseline self-correction models do not benefit from additional resources.Co-STEER accumulates and retrieves more practical knowledge as resources increase; both success and failure experiences contribute to its evolution.Source: §4.4 · p. 9, Table 3 · p. 9

4.5 Visualization of the Evolution of Co-STEER

• Co-STEER’s implementation ability improves at each evolution step and gradually approaches a boundary set by the foundation model’s intelligence.The paper visualizes this progression in Figure 3.Source: §4.5 · p. 9, Figure 3 · p. 9
• Baseline models without access to knowledge show only marginal gains from repeated self-correction, and a model without knowledge or correction attempts improves merely by observing environmental feedback.Source: §4.5 · p. 9

5 Conclusion

• Co-STEER significantly outperforms previous state-of-the-art methods in experiments.The experiments evaluate its evolvable scheduling and implementation capabilities.Source: §5 · p. 10
• The paper identifies AD2 as an overlooked scenario with challenges in method scheduling and implementation.It presents AD2 as having the potential to advance AI research.Source: §5 · p. 10
• Co-STEER is an LLM-based autonomous agent designed to address AD2’s scheduling and implementation challenges.It builds domain knowledge through a co-evolving strategy and strengthens both skills by accumulating and retrieving domain-specific experience.Source: §5 · p. 10

A Broader Impacts

• Co-STEER could let developers focus on creative tasks, reduce workforce needs through automation, require updated worker education, and risk perpetuating bias in sensitive decisions.The passage identifies these as potential economic and ethical implications.Source: A Broader Impacts · p. 14

B Limitation

• Its resource and customization requirements may constrain use in resource-limited settings or domains without extensive adaptation, though falling LLM token prices could ease this limitation.The paper identifies lower token prices as a possible way to address the limitation in the future.Source: B Limitation · p. 14

C.1 Scheduling Agent Prompt

• Figure 4 depicts the scheduling agent learning to schedule based on task information.Source: Figure 4 · p. 14
• Figure 4 also depicts practical feedback as a basis for the scheduling agent’s learning.Source: Figure 4 · p. 14

C.2 Scheduling Agent Response

• Figure 5’s scheduling response considers task complexity and task dependency.Source: Figure 5 · p. 15
• The Figure 5 response considers multiple factors to prioritize tasks.Source: Figure 5 · p. 15

C.3 Latest attempt with corresponding feedback

• The passage states that a figure is shown in section 6.Source: §C.3 · p. 15

C.4 Retrieved similar correct implementation and error

• The passage indicates that Figure 7 is shown.Source: §C.4 · p. 15

• Tables 4–8 cover few-shot, CoT, Reflexion, self-debugging, and self-planning results.Source: Table 4 · p. 15, Table 5 · p. 16, Table 6 · p. 16
• Table 9 reports results for Co-STEER, the paper’s proposed method.Source: Table 9 · p. 19
• Table 10 reports mean model-performance results in the AD2 scenario.Source: Table 10 · p. 19
• Tables 11–14 report results for Random (5), Random (10), Random (15), and Random (20) with evolving.Source: Table 11 · p. 20, Table 12 · p. 20, Table 13 · p. 20
• Tables 15–18 report Scheduler configurations with evolving for 5, 10, 15, and 20 methods.Source: Table 15 · p. 21, Table 16 · p. 21, Table 17 · p. 22
• Table 19 reports co-evolving model performance in AD2, with the parenthetical number indicating how many methods the scheduler selected for implementation.Source: Table 19 · p. 23