PaperlayerView exact revision on arXiv

Source-linked AI summary

ROAR: Unifying Runs across Heterogeneous AI-Driven Research Systems

Leo Y. Lin, Vishakha Ramani, Z. Berkay Celik, Paul Castro, Marquita Ellis

arXiv:2610.03966v2cs.AI

TL;DR

ADRS run data is fragmented across teams and formats, with no systematic way to aggregate and analyze it across domains. ROAR unifies outputs through a relational schema and parser layer, then uses pooled runs to study ADRS behavior and configure runs. Its analyses show how pooled data can reveal problem-landscape and strategy patterns and inform budget allocation.

  • Problem

    No systematic way exists to aggregate, share, and analyze cross-domain ADRS data, despite potential practical benefits from pooled runs.

    Source: §1 · p. 1

  • Method

    ROAR uses a unified relational schema and parser layer to normalize heterogeneous ADRS outputs, preserve lineage and temporal structure, and support analytics across differing evaluators and problems.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Analyses of a dataset with over 900 runs revealed basin structures, stopping and context-composition insights, and a data-driven budget-allocation policy.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that the same compute may be better allocated across several shorter, replicated runs, which also sample distinct landscape regions.

    Source: §5 · p. 9

    The replay evaluation’s gain direction was positive in 11 of 12 configurations, but per-configuration intervals are anti-conservative and the policy-gain size remains uncertain.

    Source: §J.1 · p. 24

Abstract

from arXiv · show

Each run of an AI-driven research system (ADRS) is an expensive search over a vast solution space, and dependable evaluation requires many runs, making run data both costly to produce and valuable to retain for large-scale analysis. Yet this data remains fragmented: teams operate in isolation, ADRS frameworks emit results in different formats, and no shared infrastructure exists to aggregate or compare runs across problems and systems. We present ROAR, a solution for systematically unifying and analyzing heterogeneous ADRS outputs. ROAR addresses two challenges: reconciling heterogeneous ADRS outputs and enabling analytics across runs with different objectives and scoring functions. We achieve this through a relational schema and parsing layer that normalize heterogeneous ADRS outputs while preserving data lineage and temporal structure, and accommodating new systems without requiring schema modifications. From building a corpus of more than 900 runs from multiple ADRS, we show how pooled data can reveal properties of problem landscapes that are difficult to observe. Consistent with prior work, runs with identical configurations may converge to different scores. We find that many runs realize most gains early, and that the effectiveness of different strategies for incorporating prior solutions into the search process varies across problems. We further show that the pooled corpus is actionable and not merely analytical by using ROAR to configure ADRS runs. Together, these results illustrate how pooled ADRS data can expose problem-dependent structure in search behavior that is difficult to detect from any single system, team, or benchmark. Such cross-cutting insights are difficult to obtain while runs remain siloed; ROAR is the first infrastructure designed to unify them.

1 INTRODUCTION

ROAR addresses fragmented ADRS run data with shared infrastructure for heterogeneous outputs and cross-problem analytics. Its pooled corpus supports analyses of problem landscapes, ADRS strategies, and budget allocation.

1 INTRODUCTION

  • ADRS run data is fragmented across teams and formats, with no systematic way to aggregate, share, and analyze it across domains.Different formats and semantics complicate extraction, while differing assessors and problems make pooled analytics challenging.Source: §1 · pp. 1–2
  • ROAR combines a unified relational schema and parser layer to aggregate heterogeneous ADRS outputs while preserving lineage and temporal structure.The architecture supports new systems without schema changes and analytics across runs with differing evaluators and problems.Source: §1 · p. 2
  • The contributed dataset contains over 900 runs spanning 7 problems, 14 generator models, and 6 optimization mechanisms across multiple ADRS frameworks.Contributors include independent researchers and frameworks such as OpenEvolve, SkyDiscover, and GEPA.Source: §1 · p. 2
  • Exploratory analyses use pooled runs to study multimodal basin structures, stopping heuristics, context composition, and data-driven budget allocation.These analyses demonstrate ROAR’s utility across four research questions.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

ADRS span early systems and open-source frameworks, but prior behavior analyses rely primarily on authors’ own runs. Their varied output formats are not reconciled by generic experiment trackers.

2 BACKGROUND AND RELATED WORK

  • ADRS range from early systems FunSearch and AlphaEvolve to open-source frameworks including OpenEvolve, GEPA, and SkyDiscover.The cited examples include Romera-Paredes et al. (2024), Novikov et al. (2025), Sharma (2025), Agrawal et al. (2026), and Liu et al. (2026b).Source: §2 · p. 2
  • Prior analyses of ADRS behavior rely primarily on runs generated by the authors themselves.The passage cites Cheng et al. (2025b;a), Ellis & Castro (2026), and Gupta et al. (2026).Source: §2 · p. 2
  • Frameworks record run histories in different formats that generic experiment trackers such as MLflow do not reconcile.The passage cites Zaharia et al. (2018) for MLflow.Source: §2 · p. 2

3 METHOD

ROAR combines a relational schema, parser-based ingestion, cross-run analytics, and a procedure for deriving run configurations from pooled campaigns. Its design accommodates heterogeneous systems and evaluators.

3 METHOD

  • ROAR comprises a schema, repository and parser layer, analytics pipelines for differing graders and problems, and a procedure that derives run configurations from pooled campaigns.The first three components support cross-framework analysis; the fourth is evaluated for budget allocation.Source: §3 · p. 3

3.1 DATABASE SCHEMA DESIGN

  • The six-table relational schema models systems, campaigns, candidates, candidate lineage, flexible measurements, and raw artifacts.A parser converts native outputs to a common intermediate representation, allowing new systems without database-layer changes and validating types at parse time.Source: §3.1 · p. 3

3.2 AI-DRIVEN EVOLVING REPOSITORY

  • Known formats use deterministic parsers, while coding agents can write one-off parsers for unfamiliar outputs; both paths validate through the same Pydantic schema layer.The agent workflow uploads the structured file alongside raw data.Source: §3.2 · p. 4

3.3 ANALYTICAL PIPELINE

  • ROAR analyzes runs through absolute score comparability, ordinal improvement events, and assessor-agnostic metrics derived from stored artifacts.These tiers support distributional analyses, stagnation analysis, and artifact-based diversity analysis, respectively.Source: §3.3 · p. 4

3.4 USING ROAR FOR BUDGET ALLOCATION

  • A practitioner supplies a problem, generator, mechanism, and budget; ROAR returns a policy that stops stagnant campaigns and reallocates freed iterations to fresh campaigns.The policy reports the best score achieved across campaigns, evaluated against a single full-budget campaign without stopping.Source: §3.4 · p. 4
  • ROAR fits policies by grid-searching stopping thresholds and campaign counts, replaying candidate policies against stored campaigns, and selecting the highest estimated expected score.The grid uses 9 log-spaced stopping-threshold values and log-spaced campaign counts; fitting costs at most NB + TB^2 score comparisons.Source: §3.4 · p. 4

4 EMPIRICAL ANALYSIS

ROAR’s pooled corpus supports analyses of basin structure, stagnation, context composition, and budget allocation across ADRS campaigns. The analyses reveal problem-dependent search patterns and test whether pooled history can improve run configuration at equal compute.

4.1 EXPERIMENTAL SETUP

  • The corpus contains 928 campaigns and 48,210 candidates; analyses focus on 2D Knapsack and Polynomino Packing, which have the most comprehensive coverage.RQ1–RQ3 analyze pooled-corpus dynamics and prediction; RQ4 applies the corpus to budget allocation.Source: §4.1 · p. 5, §4 · p. 5

4.2 BASIN STRUCTURE (RQ1)

  • 12 of 24 qualifying configurations show confirmed multimodality across two problems, whose score distributions reveal distinct basin geometries.Polynomino scores span a continuous range, while Knapsack runs tend toward failure or near-ceiling outcomes; the authors conclude that replication is needed to assess a configuration’s reliability.Source: §4.2 · pp. 5–6

4.3 STAGNATION HEURISTICS (RQ2)

  • Observed escape rates remain far below the rule-of-3’s 63% ceiling, typically under 15% and around 4% at longer stagnation lengths.The mixed-effects estimate is an odds ratio of 0.91 per 10-step increase (95% CI [0.86, 0.96]); the decline is significant after accounting for run-level differences.Source: §4.3 · pp. 6–7, Figure 2 · p. 7
  • Escape rates vary by problem: Knapsack is consistently 4–9%, Polynomino 13–14%, and CUDA Attention’s 40% spike at k ≥30 is inconclusive.Mechanism differences are not clearly distinguishable; the overlapping confidence intervals suggest stagnation behavior is more influenced by problem landscape than mechanism choice.Source: §4.3 · p. 7

4.4 DIVERSITY AND CONTEXT COMPOSITION (RQ3)

  • Early diversity weakly predicts final score on Polynomino Packing (ρ = 0.15, p = 0.011), is absent on Knapsack, and is inconclusive on CUDA Attention.Some EvoX configurations even reverse the Polynomino direction (ρ = −0.24, p = 0.030).Source: §4.4 · p. 7
  • Model–mechanism combinations add heterogeneity: some show significant positive effects and others significant negative effects beyond aggregate patterns.Source: §4.4 · p. 8
  • Context-composition signals differ by problem: worse-scoring examples associate with improvement on Knapsack and Polynomino, while other factor directions differ.Knapsack also shows negative associations for score diversity and better-context ratio; Polynomino’s significant factors are all positive. CUDA Attention associates better-context ratio negatively and code diversity positively, but two factors show no significant relationship.Source: §4.4 · p. 8

4.5 BUDGET ALLOCATION (RQ4)

  • +18.9 points of final score [+11.6, +25.6] on Polynomino Packing across 115 campaigns in 9 configurations at identical budget.Bounded 2D Knapsack gained +16.9 [−1.0, +33.6], not significant; gains trended positive in 11 of 12 tested configurations, with fitted kstop varying from 2 to 22.Source: Table 1 · p. 9, §4.5 · p. 9

5 CONCLUSION

ROAR unifies heterogeneous ADRS run data and uses pooled history to analyze behavior and guide run configuration. The conclusion presents this as a way to derive design-time guidance, while noting that broader coverage depends on community contributions.

5 CONCLUSION

  • A fixed-compute early-stopping and replication policy fitted from pooled history improved Polynomino Packing final score by +18.9 points and gained in 11 of 12 configurations.The conclusion also recommends allocating compute across shorter replicated runs, which can sample distinct landscape regions.Source: §5 · p. 9
  • ROAR combines a unified relational schema and parser layer to aggregate heterogeneous ADRS outputs while preserving lineage and temporal structure.The conclusion describes it as the first general data aggregation infrastructure for ADRS.Source: §5 · p. 9
  • Because the value of solution diversity and context composition varies by problem, exploration and exploitation should be tuned to landscape characteristics rather than fixed a priori.The conclusion links pooled results to this design-time guidance.Source: §5 · p. 9
  • The corpus covers more than 900 runs across 7 problems and 14 models, but remains limited relative to the space of ADRS configurations and tasks.The conclusion says expanding coverage will require community adoption.Source: §5 · p. 9

A EXTENDED RELATED WORK

  • Early ADRS such as FunSearch and AlphaEvolve placed language models inside evolutionary loops, while later frameworks also adapt search parameters or algorithms during runtime.Examples of the early systems include FunSearch and AlphaEvolve (Novikov et al., 2025); later examples include AdaEvolve, EvoX (Liu et al., 2026a), and SkyDiscover.Source: A EXTENDED RELATED WORK · p. 12
  • Prior ADRS behavior studies often rely on authors’ controlled runs, while different systems record histories differently and external studies may use different evaluation procedures and metrics.ROAR addresses this aggregation gap with analytical paradigms for heterogeneous runs; generic experiment tracking leaves naming conventions and semantic abstractions to individual teams.Source: A EXTENDED RELATED WORK · p. 12

B DATASET DETAILS

The dataset appendix uses Tables 2–4 to summarize corpus scale and composition, problem coverage, and generator models with their providers.

B DATASET DETAILS

  • Table 2 summarizes dataset statistics, while Table 3 breaks coverage down by research problem.The supplied passages identify the tables’ scope but do not provide their cell values.Source: B DATASET DETAILS · p. 13, Table 2 · p. 13
  • The three combinatorial problems dominate the corpus; CUDA kernel problems provide signal on continuous landscapes, and single-campaign problems are excluded from statistical analyses.Source: Table 3 · p. 13
  • Table 4 lists the generator models and their providers.Source: B DATASET DETAILS · p. 13, Table 4 · p. 13

C ADRS DATABASE SCHEMA

ROAR represents ADRS data in a six-table relational schema and ingests native outputs through parsers. The appendix describes its entities, processing setup, and candidate embedding procedure.

C ADRS DATABASE SCHEMA

C.1 SYSTEMS; C.2 CAMPAIGNS

C.3 CANDIDATES; C.4 CANDIDATE EDGES

  • Candidates are proposed solutions indexed by campaign and iteration; typed edges record candidate relationships and enable lineage tracking and graph visualizations.The schema distinguishes parent links from context links.Source: §C.3 · p. 14, §C.4 · p. 15, Table 8 · p. 15

C.5 MEASUREMENTS; C.6 ARTIFACTS

  • Measurements store evaluation outcomes, metrics, and observations as named values associated with candidates, while artifacts reference files whose metadata supports tracking and integrity verification.Source: §C.5 · p. 15, §C.6 · p. 15, Table 9 · p. 15

C.7 RELATIONSHIPS

  • The schema links systems to campaigns, campaigns to candidates and artifacts, and candidates to measurements and optional artifacts.A system can produce many campaigns, and a campaign can contain many candidates.Source: §C.7 · p. 15

D SYSTEM PERFORMANCE; D.1 HARDWARE CONFIGURATION

  • Processing performance is reported for parsing, database insertion, and embedding generation on 914 campaigns drawn from 59.4 GB of raw files.The processing setup used 8 parallel workers for parsing and insertion; embedding generation ran on an NVIDIA H100 node.Source: D SYSTEM PERFORMANCE · p. 16, §D.1 · p. 16

D.2 EMBEDDING PROCEDURE

  • Source code is split into overlapping 2048-token chunks with 256-token overlap, embedded independently, then averaged into one vector per candidate.The chunking addresses programs that exceed the embedding model’s context window; vectors are stored in the database via pgvector.Source: §D.2 · p. 16

D.3 PERFORMANCE RESULTS

The performance-results section points to a table comparing throughput across processing stages, followed by descriptions of three benchmark problems.

D.3 PERFORMANCE RESULTS

  • Table 11 summarizes throughput for parsing, database insertion, and embedding generation.The supplied passages do not state the table’s throughput values.Source: §D.3 · p. 16, Table 11 · p. 16

E PROBLEM DESCRIPTIONS

  • Bounded 2D Knapsack packs integer-dimension items without rotation or overlap to maximize value under a weight-capacity constraint.Source: E PROBLEM DESCRIPTIONS · p. 16
  • Polynomino Packing places non-overlapping polyomino pieces on a fixed grid to maximize grid coverage.Source: E PROBLEM DESCRIPTIONS · p. 16
  • CUDA Attention Kernel Optimization tunes a bf16 scaled-dot-product attention kernel to maximize runtime performance over a reference on NVIDIA H100.Source: E PROBLEM DESCRIPTIONS · p. 16

F ADDITIONAL RQ1 FIGURES

The appendix presents configuration-level basin-structure figures for Polynomino Packing and Bounded 2D Knapsack, extending the Polynomino set shown in the main text.

F ADDITIONAL RQ1 FIGURES

  • The Polynomino figures show the complete configuration set, whereas the main-text Figure 1 shows a subset.Source: F ADDITIONAL RQ1 FIGURES · p. 16
  • 7 of 14 Polynomino Packing configurations are confirmed bimodal, 6 are inconclusive, and 1 is unimodal.The figures include configurations with at least 10 runs; 16 configurations with fewer runs are omitted.Source: Figure 4 · p. 17
  • For Bounded 2D Knapsack, 5 of 10 configurations are confirmed bimodal, 3 are inconclusive, and 2 are deterministic.Each violin represents runs from one configuration.Source: Figure 5 · p. 17

G ADDITIONAL RQ2 FIGURES

The appendix adds visualizations of stagnation-escape behavior and defines code diversity as a measure of variation among candidate implementations.

G ADDITIONAL RQ2 FIGURES

H.1 CODE DIVERSITY

  • Candidate code is embedded with jina-embeddings-v2-base-code into 768-dimensional vectors.Each candidate's source code is mapped to an embedding.Source: §H.1 · p. 18
  • Campaign-level code diversity is the average pairwise cosine distance among candidate embeddings, ranging from 0 for identical candidates to a theoretical maximum of 2.In practice, code embeddings fall below 1.Source: §H.1 · pp. 18–19

H.2 EARLY DIVERSITY AND FINAL SCORE

The appendix examines early diversity, score-tier diversity, and context-composition factors, finding that diversity–score patterns vary by problem and configuration.

H.2 EARLY DIVERSITY AND FINAL SCORE

  • ρ = 0.15, p = 0.011 for Polynomino Packing; Bounded 2D Knapsack shows no relationship (ρ = −0.00, p = 0.943), and CUDA Attention is inconclusive (ρ = −0.50, p = 0.667).Diversity is measured among candidates from the first 25% of iterations.Source: Figure 8 · p. 19, §H.2 · p. 19
  • The positive Polynomino direction holds for most subgroups, but EvoX is significantly negative (ρ = −0.24, p = 0.030) and gpt-5.4 trends negative (ρ = −0.41, p = 0.051).The authors note that predictive direction may vary with configuration.Source: §H.2 · p. 19

H.3 WITHIN-DECILE DIVERSITY

  • Neither problem shows a significant monotonic relationship between score decile and code diversity; higher-scoring solutions show no evidence of narrower code-space convergence.Knapsack: ρ = −0.06, p = 0.87; Polynomino: ρ = 0.08, p = 0.83.Source: §H.3 · p. 22, Figure 13 · p. 22

I.1 FACTOR DEFINITIONS

  • The RQ3 context-composition factors include score diversity and code diversity within a candidate's context.The passage identifies these as score and Dcode, respectively.Source: §I.1 · p. 22

I.2 FACTOR IMPORTANCE BY MODEL AND MECHANISM

  • For Bounded 2D Knapsack, factor effects are negative across most model–mechanism combinations, with Gemini-2.5-Flash an outlier; Polynomino Packing shows an overall more positive trend.The figures report GLMM log-odds per standard deviation; hatched cells are not significant at p ≥ .05 by Wald test.Source: Figure 14 · p. 23, Figure 15 · p. 23

J.1 LIMITATIONS OF THE REPLAY EVALUATION

The replay evaluation has limits in how it estimates policy gains: same-campaign replay likely inflates gains, and the configuration-level intervals are anti-conservative.

J.1 LIMITATIONS OF THE REPLAY EVALUATION

  • Fitting and replaying from the same campaigns likely makes the gain estimate optimistic; half-split fitting yields smaller gains, positive in 10 of 12 configurations.Held-out campaigns are used to score the comparison with current practice.Source: §J.1 · p. 23
  • Configurations were included only with at least 8 campaigns of 50 or more iterations and best-score standard deviation σs ≥ 2 on the 0–100 scale.The cutoff was operational rather than tuned; five of 17 configurations with enough campaigns were excluded.Source: §J.1 · pp. 23–24
  • The 12 tested configurations are not enough to pin down gain size, and small out-of-bag sets make per-configuration bootstrap intervals anti-conservative.The authors treat blue interval coloring as suggestive, relying instead on positive gains in 11 of 12 configurations and problem-level intervals pooling 115 and 37 campaigns.Source: §J.1 · p. 24, Figure 16 · p. 24
Loading 2610.03966v2…