EvoCast: Reliable Autonomous Research Agents for Iterative Forecasting Architecture Evolution

1 Introduction

• Real forecasting tasks differ in periodicity, drift, variable interaction, horizon, and noise, so developing a suitable architecture requires extensive expert effort.That effort includes model selection, mechanism diagnosis, architecture modification, code integration, and controlled evaluation.Source: §1 · p. 1
• EvoCast combines baseline diagnosis, evidence-grounded research directions, bounded implementation, canonical evaluation, and promotion in a fixed-protocol loop.LLM agents generate hypotheses and implement code; deterministic authorities control source validity, evaluation, promotion, and state updates.Source: §1 · p. 2
• EvoCast is evaluated for implementation reliability and efficiency, forecasting utility across three real-world cases, and the role of cognition-authority separation.The evaluation asks whether it improves implementation success and agent-side token/time cost and develops architectures that outperform selected baselines and strong models.Source: §1 · p. 2

Automated Forecasting Architecture Development

• Forecasting architectures encode different assumptions, and their behavior is dataset-dependent under a common evaluation protocol [38] [48] [65] [84] [50].AutoML, hyperparameter optimization, and neural architecture search typically operate within specified templates or operator spaces [7].Source: §2 · p. 2

Autonomous Research Agents and Controlled Experimentation

• EvoCast evolves executable source modules from a task-specific repository baseline using executed mechanism diagnosis and a canonical evaluator.It connects research-direction generation with deterministic boundary validation, promotion, and evidence write-back under one protocol.Source: §2 · p. 2

LLMs and Agentic Systems for Time Series

• LLM-assisted forecasting systems automate data understanding, planning, validation, reporting, or iterative architecture development [49] [68] [78].EvoCast adds an executable task baseline, executed baseline ablations, persistent evidence, and deterministic promotion authority.Source: §2 · p. 2

Autonomous Research Agents and Controlled Experimentation

• Autonomous research systems connect proposals, code editing, and experimental feedback, with related work emphasizing standardized environments, code-space search, and execution feedback [16] [21] [26] [72].Long-horizon systems also emphasize task contracts, consistent evaluation, execution infrastructure, and cross-round evidence management [9] [12] [22] [67] [85].Source: §2 · p. 2

3.1 Task Setting

• The task protocol fixes the dataset, target, lookback, horizon, forecasting mode, evaluation metric, and research budget throughout one run.The same protocol is reused for baseline selection, diagnosis ablations, candidates, and final promotion checks.Source: §3.1 · pp. 2–3
• The repository-level task requires executable architecture modifications and a validated current-best architecture with supporting research evidence.The task runs on heterogeneous forecasting repositories under one stable comparison surface.Source: §3.1 · p. 3

3.2 Research State and Round Outcome

• The research state aggregates the incumbent model, evidence bundle, and round history, including diagnoses, accepted results, negatives, unstable gains, repairs, and failures.The history preserves the executed trajectory for later planning and state updates.Source: §3.2 · p. 3
• Rounds are classified as accepted, rejected, unstable, or failed according to boundary validation, canonical metrics, improvement, and seed stability.Each outcome contributes auditable evidence to a unified state-update interface.Source: §3.2 · p. 3

4.1 Overview of the EvoCast Research Loop

• EvoCast transitions from research state S_r to S_r+1 by generating one direction, executing one bounded candidate, and recording its validated outcome.The workflow comprises task grounding, baseline establishment, baseline diagnosis, formal research, and final reporting.Source: §4.1 · p. 3

4.2 Task Grounding and Baseline Establishment

• Task grounding freezes the dataset, target, lookback, horizon, forecasting mode, and metric as protocol P, then adds computed dataset characteristics to task evidence.The selected executable baseline becomes the single formal comparison anchor, and the evaluator reuses the same split, metric, training policy, and evaluation path.Source: §4.2 · pp. 3–4

4.3 Baseline Diagnosis for Mechanism Evidence

• EvoCast extracts mechanisms from the active baseline path and executes ablations under the same protocol to create mechanism-level evidence for later planning.Degradation supports preserving a function, improvement suggests redundancy or task mismatch, and small changes indicate a weak or unresolved mechanism.Source: §4.3 · p. 4

4.4 Formal Iterative Research with Cognition-Authority Separation

• The planner uses task, model, diagnosis, prior-result, and failure evidence to identify unresolved needs and select a feasible, nonredundant module-level hypothesis.Each direction specifies its target mechanism, insertion point, testable hypothesis, expected information-flow change, failure to avoid, and expected knowledge gain.Source: §4.4 · p. 4
• Candidate construction is bounded to an allowed model-source region while task semantics, training policy, evaluation interface, and public model contract remain unchanged.Program authorities validate boundaries, run canonical evaluation, and promote only candidates satisfying the fixed comparison rule.Source: §4.4 · p. 4, §4.5 · p. 5

4.5 Validated Outputs and Evidence-Carrying State Update

• Accepted candidates replace the incumbent; rejected, unstable, and failed candidates preserve it while recording their outcome-specific evidence and failure information.Later planning rereads the accumulated records, while the model state changes only after promotion.Source: §4.5 · p. 5
• At termination, EvoCast freezes the incumbent, metrics, round outcomes, gate decisions, and failures for a report based only on those facts.The output bundle includes the validated model, reproducible source and configuration, resource usage, and an auditable report.Source: §4.5 · p. 5

5.1 Research Questions

• The experiments ask whether EvoCast improves implementation reliability, develops useful architectures, and benefits from its orchestration design, using a shared benchmark path and controlled implementation tasks.RQ1 uses 30 tasks spanning 10 backbones and local edits, cross-module integration, and novel module insertion; the RQ2 cases are Air Quality, Melbourne Pedestrian T33, and Steel Industry.Source: §5.1 · p. 5, §5.2 · p. 5, §5.3 · p. 5

5.3 RQ1: Implementation Reliability

• EvoCast has the highest implementation success while maintaining perfect fidelity and strongest repair performance, with greater time and cost efficiency than the other agent baselines.Direct Independent Edit is cheapest but has the lowest success rate; other agent baselines outperform it but remain behind EvoCast.Source: §5.3 · p. 5, Table 1 · p. 6

5.4 RQ2: End-to-End Research Utility

• Across three real forecasting tasks, EvoCast reduces error versus the selected baseline on all three, achieves the lowest MSE on each, and the lowest MAE on two.Results use one unchanged evaluator, task protocol, and final-artifact evaluation rule.Source: §5.4 · p. 6
• EvoCast has the highest valid-round ratio on all three tasks and lower total tokens and API cost than R&D-Agent, while its resource comparison with AIDE varies by task.Against AIDE, EvoCast uses more tokens and API cost on Air Quality and Steel Industry, and more tokens but lower API cost on Melbourne T33; relative wall-clock time varies by task.Source: §5.4 · pp. 6–7, Table 4 · p. 7
• The same workflow retains different active mechanisms across tasks: trend anchoring and horizon refinement for Air Quality, periodic memory and calendar retrieval for Melbourne, and recency-aware refinement for Steel Industry.The retained structures differ across the three tasks despite following the same protocol.Source: §5.4 · p. 7, Figure 4 · p. 8

5.5 RQ3: Orchestration and Authority Design

• The full loop achieves the lowest final MSE; A4 has the weakest quality and highest resource cost, while ablations also weaken stability, budget efficiency, or later-round progress.The best-refresh trajectory shows the full chain entering a stronger error region earlier and continuing to improve, while weakened chains stagnate earlier or make limited gains.Source: §5.5 · pp. 7–8, Table 5 · p. 8

6 Conclusion

• EvoCast develops a task-specific baseline, accumulates executable research evidence across rounds, and delivers an architecture tailored to the target scenario.Accepted changes are tied to executable source code, canonical evaluation, and traceable state updates, making the final result reusable and practically deployable.Source: §6 · pp. 8–9
• One fixed workflow develops suitable time-series models for different real-world scenarios, illustrating the value of executable, auditable, scenario-adaptive research artifacts.Figure 5 tracks best-refreshing events and whether ablation chains continue to improve in later rounds.Source: Figure 5 · p. 8, §6 · p. 9

A.1 Deep Time-Series Forecasting Architectures

• Forecasting architectures span decomposition, frequency, patch-level, cross-variate, linear, dense-encoding, and time-frequency hybrid families, each encoding different structural assumptions.TSLib provides a reusable benchmark library covering forecasting, imputation, anomaly detection, and classification.Source: §A.1 · p. 12
• Dataset-dependent architecture behavior makes baseline selection, controlled comparison, and architecture modification central to forecasting research.EvoCast organizes baseline establishment, diagnosis, and architecture evolution within one fixed-protocol research loop.Source: §A.1 · p. 12

A.2 Automated Forecasting Model and Architecture Development

• AutoML and neural architecture search automate selection and configuration within explicit, predefined operator spaces or architecture templates.LLM-assisted approaches such as NADER, AgentNAS, and GenAutoML extend this toward dynamic or open-ended architecture construction.Source: §A.2 · p. 12
• EvoCast modifies executable modules inside a fixed forecasting repository, grounding bounded source changes in an established task baseline, active mechanisms, and reusable evidence state.Each round outcome is written into that state for subsequent architecture directions.Source: §A.2 · p. 12

A.3 LLMs and Agentic Systems for Time Series

• Time-series agents span foundation models in forecasting or reasoning workflows and agents that automate task-level forecasting processes.Examples include Time-LLM, TimeCMA, Time-VLM, TS-Agent, Nexus, and TimeSeriesScientist.Source: §A.3 · p. 12
• Compared with SEA-TS and GenAutoML, EvoCast combines executable task-specific baselines, executed diagnosis, persistent evidence, and deterministic promotion authority outside the LLM.This supports fixed-protocol, repository-level architecture evolution across rounds.Source: §A.3 · p. 12

• Prior autonomous research systems span idea generation, experimentation, verification, and writing, including MLAgentBench, The AI Scientist, InternAgent, and R&D-Agent [16] [19] [42] [70] [72].Source: §A.4 · p. 12
• EvoCast applies executed ablations to active baseline mechanisms so diagnostic outcomes become task-specific evidence for later research planning.Source: §A.4 · p. 12
• ResearchLoop, AIRA2, FML-bench, ResearchGym, and Arbor emphasize task contracts, consistent evaluation, execution infrastructure, or cross-round evidence management [9] [12] [22] [67] [85].Source: §A.4 · pp. 12–13

B.1 Detailed Workflow

• Program authorities—not agent self-reports—validate candidate scope, run canonical evaluation, apply promotion rules, and update model state, while LLM agents generate research intent, code, and narrative.The editable boundary excludes the dataset, split, metric, training schedule, evaluator, and promotion logic; protocol-invalid candidates are excluded from formal comparison.Source: §B.1 · p. 13

B.2 Fixed Task Protocol and Task Grounding

• Each run fixes its dataset, targets, lookback, horizon, forecasting mode, metric, and budgets, then characterizes data features as planning evidence rather than direct architecture choices.Characterization follows TFB [50] and includes correlation, transitions, distribution shift, seasonality, trend, stationarity, and short- and long-term distribution differences.Source: §B.2 · p. 14

B.3 Baseline Establishment and Diagnosis

• EvoCast diagnoses the active baseline through runnable, source-grounded ablations under the shared protocol, admitting only valid evaluations as evidence and classifying outcomes as essential, harmful or redundant, uncertain, or failed mechanisms.Clear degradation indicates an essential mechanism; clear improvement identifies a replacement or restructuring target, while small changes remain weak or uncertain evidence.Source: §B.3 · p. 14

B.4 Evidence-Grounded Research Direction Selection

• The Research Planner combines task, model, scientific, and engineering evidence to select one source-grounded module-level direction with a testable hypothesis, implementation risks, and expected knowledge gain.It generates research intent but cannot implement code, change the protocol, evaluate candidates, or update the current best model.Source: §B.4 · pp. 15–16

B.5 Candidate Construction and Program Authorities

• The Builder implements the selected direction within fixed interface and shape constraints; a deterministic Promotion Gate requires valid source and runtime checks plus seed-stable improvement before replacing the current best.Single-run gains, valid rejections, and unstable improvements remain evidence, while only stable gains authorize promotion.Source: §B.5 · p. 16

B.6 Iterative State Update and Final Reporting

• Each round permits repair within the same research direction and protocol, then records accepted candidates, valid rejections, unstable improvements, or engineering failures so later planning can use the updated evidence.Repair is limited to restoring executability, such as fixing import, shape, interface, or live-path activation issues; it does not change the research goal or modification scope.Source: §B.5 · p. 16, §B.6 · p. 17
• Accepted candidates replace the incumbent; rejections and unstable improvements update evidence; engineering failures update failure memory, while invalid protocol results are excluded from formal evidence.Accepted candidates provide positive evidence, rejections negative evidence, and unstable improvements weak positive evidence; failures also preserve repair traces.Source: §B.6 · p. 18
• The next-round planner reads updated evidence, repair history, failure memory, and baseline diagnosis to guide planning and reduce repeated failure paths and unnecessary trial-and-error.Source: §B.6 · p. 18
• At termination, the system freezes the current-best model and research facts for a narrative-only HTML report and outputs the validated model, reproducible source and configuration, token usage, and report.The Report Narrative Agent cannot modify metrics, promotion decisions, model state, or evidence state.Source: §B.6 · p. 18

C.1 Evaluation Protocol

• Baseline, ablation, and candidate evaluations use TFB’s canonical benchmark path, and the editable model pool comes from the same repository family.Source: §C.1 · p. 18

C.2 Agent Model and Pricing

• Planner, builder, and reporting calls use MiniMax-M3; experiments use standard-tier pricing with input capped at 512K tokens and distinguish cached from uncached input.Source: §C.2 · pp. 18–19

C.3 Budget Ledger and Alignment

• Each workflow is capped at 20 formal improvement rounds and three repairs per round, while resource totals accumulate from workflow start through termination.Source: §C.3 · p. 19

C.4 Supported Baseline Pool

• EvoCast establishes baselines from a pool of 52 forecasting architectures spanning linear, Transformer, MLP, CNN, GNN, and other families.Source: §C.4 · p. 19

C.5 Execution Environment

• Experiments run locally on Ubuntu 22.04 with Python 3.12.7, PyTorch 2.7.1+cu128, CUDA 12.8, NumPy 1.26.4, pandas 2.2.2, and one NVIDIA GeForce GPU.Source: §C.5 · p. 19

• The 30-task suite crosses 10 backbones with three predefined idea-complexity levels; compared methods receive the same repository snapshot, idea, edit boundaries, and validation entry.Source: §D.1 · p. 19
• The comparison spans one-shot editing, lightweight software-engineering agents, search-based engineering, execution-grounded research workflows, and split-role research-and-development pipelines.Source: §D.1 · p. 19
• 96.7% of tasks pass the canonical build/smoke pipeline for EvoCast, versus 90.0% for mini-SWE-agent, 86.7% for AutoResearch, and 83.3% for AIDE (2025); EvoCast also has the second-lowest average token, cost, and time.Source: §D.1 · p. 20
• EvoCast remains stable from Level 1 to Level 2 and drops only modestly at Level 3, while Direct Independent Edit deteriorates rapidly beyond local modifications.Source: Figure D.1 · p. 20, §D.1 · p. 20
• Direct edits fail completely on four backbones, while agent baselines show distinct weak-backbone clusters, so similar overall scores can mask robustness differences.Source: §D.1 · pp. 20–21
• Across backbones, EvoCast has the narrowest failure profile and most uniform behavior; its advantage reflects consistency rather than isolated wins on easier targets.Source: §D.1 · p. 21

D.2 RQ2 Additional Results

• The RQ2 comparison fixes task definitions, budgets, and evaluation metrics in advance and reevaluates final deliverables under the shared TFB evaluator.Chronological splits use a 7:1:2 train-validation-test ratio, with training-only normalization reused for validation and testing.Source: §D.2 · pp. 21, 24
• The three tasks cover multivariate air-quality forecasting, single-series pedestrian counts, and multivariate-to-univariate industrial energy forecasting.The dataset descriptions identify Air Quality, Melbourne Pedestrian T33, and Steel Industry as the respective task sources.Source: §D.2 · p. 21
• EvoCast reduces selected-baseline error on all three tasks, achieving the lowest MSE on each and the lowest MAE on two; AIDE has the lowest MAE on Melbourne Pedestrian T33.Agent values are from final selected programs evaluated with fixed seed 2027.Source: §D.2 · p. 22
• EvoCast achieves lower MSE and MAE than all listed general-purpose forecasting models on all three tasks.The models and tasks are compared under the same TFB protocol.Source: §D.2 · p. 22, Table D.3 · p. 23
• EvoCast uses far fewer tokens and lower API cost than R&D-Agent on all three tasks, but relative to AIDE its token and cost trade-offs vary by task.Against AIDE, EvoCast uses more tokens and API cost on Air Quality and Steel Industry, and more tokens but lower API cost on Melbourne T33; relative wall-clock time also varies by task.Source: §D.2 · p. 23
• EvoCast maintains high round validity across the three tasks, and Figure D.3 traces distinct multi-round progress: Air Quality improves later, Melbourne earlier, and Steel Industry more steadily.The figure marks accepted rounds and shows available seed-level stability information.Source: §D.2 · pp. 23–24, Figure D.3 · p. 24
• On Air Quality, Figure D.2 shows EvoCast in a lower, steadier error band; AIDE and R&D-Agent fluctuate more, lose more rounds to unusable outcomes, and converge to different artifact types.AIDE lands on a Ridge feature-engineering solution and R&D-Agent on a GRU/LSTM ensemble, while EvoCast continues deep forecasting architecture development.Source: Figure D.2 · p. 23, §D.2 · p. 23
• The retained live-path modules differ by task: Air Quality adds trend anchoring and horizon-shape refinement, Melbourne adds periodic and calendar-conditioned mechanisms, and Steel Industry adds recency-aware and output-side corrections.The task architectures extend FreTS for Air Quality and PatchMLP for Melbourne Pedestrian T33 and Steel Industry.Source: §D.2 · pp. 23–24

D.3 RQ3 Additional Results

• The ablations use Melbourne Pedestrian T33 with PatchMLP, lookback 168, horizon 24, 20 research rounds, and a repair budget of three, removing one component at a time.The full workflow is the reference condition for all variants.Source: §D.3 · p. 24
• The full EvoCast loop achieves the lowest final MSE while keeping runtime, token usage, and API cost controlled; every ablation weakens this balance.Final MSE uses the retained artifact and is averaged over three seeds; failed ablations retain the baseline value.Source: §D.3 · p. 25
• Removing dataset characterization yields 0.173547, removing baseline diagnosis yields 0.173473 with 2,794,140 tokens and 1.4694 USD cost, and removing evidence-grounded planning yields 0.173250 despite 20/20 valid rounds.The ablations respectively remove task-profile input, executed baseline mechanism evidence, and structured accumulated evidence for direction selection.Source: §D.3 · pp. 24–25
• Removing authority-controlled execution leaves the retained artifact at baseline MSE 0.185001 while consuming 10.80 hours, 25,321,252 tokens, and 7.8792 USD; removing iterative evidence update yields 0.178489 with 15/20 valid rounds.The first ablation permits edits beyond the mechanism-linked source regions; the second prevents later results and failure records from informing planning.Source: §D.3 · p. 25
• Figure 5 shows the full system continuing to improve its cumulative best value in the later half, while A5 plateaus earlier and A1–A3 finish higher.The full system uses dataset characterization, baseline diagnosis, evidence-grounded planning, authority-controlled execution, and iterative evidence update.Source: §D.3 · pp. 24–25

E Code and Usage

• The `python -m evocast` entry opens a wizard for task definition, budget, baseline strategy, and research start conditions, then runs the configured workflow under one fixed protocol.Build mode supports faster end-to-end reliability checks with reduced training, validation, and testing budgets.Source: E Code and Usage · pp. 25–26
• The interface combines live task, stage, metric, time, and token monitoring with round status and raw logs; HTML reports and model packs support offline review and verification.The model pack exports the retained model, configuration, and metric records.Source: Figure E.1 · p. 26, E Code and Usage · p. 26