PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoResearch at Production Scale: Failure Modes and a Multi-Agent Framework

Aparajith Chandran, Juwon Kim, Saurav Jha, Pablo Castells, Florian Hottier

arXiv:2609.30541v1cs.LGcs.IR

TL;DR

Production recommendation research requires extensive experimentation, but costly runs, competing criteria, and long campaigns strain the original AutoResearch paradigm. The paper studies two independently developed systems over twelve weeks and proposes cost-sensitive scaffolding for recurring failure modes. It reports improvements over hand-tuned baselines and an autonomous catalog-coverage expansion.

  • Problem

    Production-scale AutoResearch combines costly iterations, competing evaluation criteria, and campaigns spanning jobs beyond a single context, conditions whose behavior had not been reported.

    Source: §I · p. 1

  • Method

    Across two production book-recommendation systems, the paper analyzes recurring failure modes and maps them to prevention, durable memory, and stagnation-triggered redirection, scaled to iteration cost.

    Source: §I · p. 2, §VI · p. 8

  • Results

    1.82× Recall@6 lift on System A and 2.1× lift on System B’s custom evaluation metric over hand-tuned baselines; System A catalog coverage scaled up 5.8×.

    Source: §I · p. 2

  • Takeaways & Limitations

    Across the two systems, scaffolding differed with iteration cost: expensive iterations benefited from pre-execution checks, while cheap ones used post-execution reversion.

    Source: §VI · p. 8

    Both systems come from one organization and one domain, so generalization of the failure modes and cost-dependence principle remains unestablished.

    Source: §VI · p. 9

Abstract

from arXiv · show

Optimizing embedding systems for production recommendation pipelines demands systematic exploration that consumes disproportionate engineering effort at scale. We apply Andrej Karpathy's AutoResearch paradigm -- a large language model that iteratively edits a training script and retains modifications that improve a held-out scalar metric -- to automate this exploration. We report on twelve weeks of running this paradigm at production scale, where iterations consume hours of multi-GPU compute, evaluation involves competing criteria, and campaigns span weeks across many training jobs. Across two independently developed representation-learning systems for a book recommendation pipeline, we ran 220+ experiments and observed five recurring failure modes absent from the original setting: infrastructure fragility, agent memory decay, search-direction stagnation, iteration-cost asymmetry, and metric fixation. We contribute a three-principle scaffolding design -- prevent, persist, redirect -- that maps each failure mode to a structural remedy and whose instantiation scales with iteration cost. The framework produced a 1.82x Recall@6 lift and a 2.1x coherence lift over hand-tuned baselines, and the agent autonomously designed a text-only fallback that expanded catalog coverage by 5.8x. The two systems span nearly three orders of magnitude in per-iteration cost yet exhibit the same failure modes, suggesting these are structural properties of production-scale autonomous research rather than artifacts of either application.

I. INTRODUCTION

AutoResearch automates code-level experimentation, but production settings challenge its assumptions through costly iterations, competing criteria, and campaigns spanning many jobs. Across two production book-recommendation systems, the paper identifies recurring failure modes and proposes scaffolding whose design depends on iteration cost.

I. INTRODUCTION

  • AutoResearch iteratively edits training code and retains changes that improve a held-out scalar metric, unlike AutoML approaches limited to predefined search spaces.The original paradigm demonstrated code-level search in single-GPU, five-minute language-modeling experiments.Source: §I · p. 1
  • Production-scale campaigns combine hours-long multi-GPU iterations, competing evaluation criteria, and weeks of jobs, conditions not previously reported for AutoResearch.The study reports twelve weeks across two independently developed systems; results beat hand-tuned baselines by 1.82× Recall@6 and 2.1× on System B’s custom metric.Source: §I · pp. 1–2
  • The agent also designed a text-only fallback that expanded System A catalog coverage by 5.8× without human direction.This autonomous coverage scale-up is reported separately from the baseline metric lifts.Source: §I · p. 2
  • The paper identifies five production-scale failure modes and proposes prevent, persist, and redirect scaffolding, with its overhead scaled to iteration cost.The paper frames its contribution as a production-grounded failure taxonomy and cost-dependence principle, not a new algorithm.Source: §I · p. 2

II. PRODUCTION SETTING

The production setting comprises two independently adapted AutoResearch systems for book recommendation, with different representations, optimization targets, and iteration costs. System A produces retrieval embeddings; System B assigns hierarchical Semantic IDs.

A. System A: Embedding Generation

  • System A generates retrieval embeddings for 98% of the active catalog; Recall@6 measures how often six nearest neighbors include a held-out book’s actual next purchase.The hand-tuned baseline reached 3.79% Recall@6 after four engineer-weeks of manual work.Source: A. System A: Embedding Generation · p. 2

B. System B: Configuration-Level Semantic ID Assignment

  • System B encodes book embeddings as hierarchical Semantic IDs through residual quantization, with each code level grouping books into progressively finer clusters.Weighted coherence combines within-cluster similarity across three levels as 0.5 × L1 + 0.3 × L2 + 0.2 × L3; its hand-tuned baseline scored 0.348.Source: B. System B: Configuration-Level… · pp. 2–3

II. PRODUCTION SETTING

  • The systems share standard AutoResearch components but differ in adaptation, scale, and cost regime; their per-iteration costs span nearly three orders of magnitude.The system-comparison table identifies the cost contrast as approximately 760×.Source: §II · p. 2, Table I · p. 3

C. Cost Profile and Campaign Scale

The twelve-week campaign covered more than 220 experiments across two systems whose per-iteration costs differ sharply. Four cost- and horizon-driven failure modes appeared in both systems, including infrastructure failures and repeated rediscovery of failed configurations.

C. Cost Profile and Campaign Scale

  • System A iterations cost roughly 760× System B iterations, across a campaign of twelve weeks and 220+ experiments.The cost difference combines instance hourly price, GPU count, and wall-clock duration.Source: C. Cost Profile and Campaign Scale · p. 3

III. FAILURE MODES AT PRODUCTION SCALE

  • Four cost- and horizon-driven failure modes appeared independently in both systems, while metric fixation was specific to multi-criteria System B.The campaign comprised 60+ System A iterations across six runs and 150+ System B iterations across 15 runs.Source: §III · p. 3, C. Cost Profile and Campaign Scale · p. 3

A. Infrastructure Fragility

  • Infrastructure fragility wasted compute through environmental and coordination failures, including a multi-GPU bug that ran 70M text encodings on one GPU for over 10 hours.After deployment, the Code Fixer caught recurrences of the same distributed-training misconfiguration before hardware execution.Source: A. Infrastructure Fragility · p. 3

B. Agent Memory Decay

  • Limited context and absent durable memory led agents to repeat failed experiments, including System B’s 100-epoch configuration at least six times.System A also reintroduced a prohibited file-system operation six times across four iterations.Source: B. Agent Memory Decay · p. 3

C. Search-Direction Stagnation

At production scale, agents can persist in exhausted search directions, undervalue costly or consequential experiments, and optimize metrics without questioning their suitability. These problems differ by cause and evaluation setting.

C. Search-Direction Stagnation

  • The agent repeatedly made marginal changes near local optima, including five System B iterations within ±0.008 weighted coherence.Without external redirection, System A would have continued fusing pre-compressed 128d features indefinitely.Source: C. Search-Direction Stagnation · p. 3

D. Iteration-Cost Asymmetry

  • The agent did not weigh expected information gain against compute cost, treating a 15-hour run like a 30-minute ablation.In System B, ten consecutive behavior-loss iterations brought negligible genre-purity improvement; a separate parameter change dropped weighted coherence from 0.562 to 0.503.Source: D. Iteration-Cost Asymmetry · pp. 3–4

E. Metric Fixation

  • System B’s metric fixation let codebook-size increases improve weighted coherence while producing unusable singleton-heavy deep levels.Weighted coherence, genre purity, and median cluster size were never simultaneously met in 15 runs across 150+ iterations; Table II indicates metric fixation requires multi-criteria evaluation and was specific to System B.Source: E. Metric Fixation · p. 4, Table II · p. 4
  • The five modes arise when production-scale work breaks assumptions of cheap iterations, simple metrics, short sessions, and negligible failure costs.The table aggregates failure-mode incidence across the campaign.Source: E. Metric Fixation · p. 4, Table II · p. 4

IV. FRAMEWORK DESIGN

The framework organizes production-scale AutoResearch scaffolding around preventing failures, redirecting stalled search, and preserving knowledge across iterations and runs. System A implements these principles through a three-agent architecture whose safeguards and roles reflect the high cost of its iterations.

IV. FRAMEWORK DESIGN

  • Prevention targets infrastructure fragility and iteration-cost asymmetry, redirection targets search-direction stagnation, and persistence targets memory decay; metric fixation is only partly mitigated.The same principles apply across both systems, but their implementations differ with iteration cost.Source: §IV · p. 4, Table III · p. 4
  • Pre-execution gates are justified when failed iterations waste hours of compute, while post-execution reversion suffices when iterations are cheap enough to treat as disposable hypotheses.The framework’s emphasis on each principle should vary with iteration cost.Source: §IV · p. 4

A. Architecture

  • System A assigns script generation to the Researcher, pre-execution code repair to the Code Fixer, and stagnation-triggered strategic direction to the Criticizer.The evaluator and program.md are immutable to the agents.Source: A. Architecture · pp. 4–5, Figure 1 · p. 5

B. Prevention: Code Fixer

The Code Fixer repairs latent infrastructure faults before execution, while the Criticizer redirects search when progress stagnates. A representation-focused directive produced the campaign’s largest single attributed improvement, and later iterations reached a 1.82× lift over the hand-tuned baseline.

B. Prevention: Code Fixer

  • In its first deployment, the Code Fixer corrected 14 latent bugs in one pass, and no later iteration was lost to the agent-coordination layer.The DDP wrapper bug alone would have wasted 10+ hours running an 8-GPU job on a single device.Source: B. Prevention: Code Fixer · p. 5, Table IV · p. 5

C. Redirection: Criticizer

  • The Criticizer fires after five iterations without more than 8% relative gain; three iterations triggered too often, while eight came too late.Its directive is cleared on the next Keep, so it acts as a stagnation breaker rather than accumulating authority across the campaign.Source: C. Redirection: Criticizer · p. 5
  • +50 bps followed the Criticizer’s input-representation directive, and the campaign later peaked at 6.90% Recall@6 at iter-12, a 1.82× lift over the hand-tuned baseline.The directive led to multi-hop neighborhood profiles and moved Recall@6 from 5.48% to 5.98% in three iterations.Source: C. Redirection: Criticizer · p. 5, Figure 2 · p. 6
  • A human-driven Phase 3→4 pivot redirected System B from content-only to behavioral+content embeddings, producing its best result, WC = 0.735.The paper reports paradigm shifts as externally redirected: automated for cheap-to-emit directives and human-driven for cross-run programs.Source: C. Redirection: Criticizer · p. 6

D. Persistence: Cross-Job Memory

Durable cross-job memory reduced—but did not eliminate—repeated rediscovery, while metric fixation remained unresolved and limited hands-free operation.

D. Persistence: Cross-Job Memory

  • Durable cross-job program.md storage let new jobs inherit accumulated discoveries, but the agent still retested prohibited 100-epoch training at least six times.In System B, the human researcher maintained the program file between runs; the mechanism reduced memory decay frequency but did not eliminate it.Source: D. Persistence: Cross-Job Memory · p. 6
  • Automatic revert triggered on 4 of 21 System B iterations after OOM-causing configurations, but its low cost did not translate to System A’s hundreds-of-dollars iterations.The approach was cheap in System B, where iterations cost on the order of dollars, and impractical in System A.Source: D. Persistence: Cross-Job Memory · p. 6

E. Metric Fixation: Partial Mitigation

  • Multi-metric reporting, downstream usability context, and human QA reduced metric fixation’s severity, but the agent never questioned its target.Because the agent could modify training parameters but not evaluation criteria, the system could not run completely hands-free; whether current LLM capabilities can solve this remained open.Source: E. Metric Fixation: Partial Mitigation · p. 6

A. System A: Recall@6 Trajectory

  • 1.82× relative lift over the 3.79% hand-tuned baseline was achieved by full three-agent code-level AutoResearch on the peak evaluation date.The across-date mean retained a 1.63× lift over the hand-tuned baseline.Source: A. System A: Recall@6 Trajectory · p. 6
  • The agent discovered a text-only fallback that expanded catalog coverage by 5.8×, while the study remained offline and online pilots were underway.Progressive discoveries included cross-attention, sentence-transformer fine-tuning, hard-negative contrastive learning, a temperature curriculum, multi-hop neighborhood profiles, and dual mean+top-K aggregation.Source: A. System A: Recall@6 Trajectory · pp. 6–7
  • System B’s campaign best followed a human-driven Phase 3→4 pivot from content-only to behavioral+content embeddings, after early architectural choices locked in.The reported weighted coherence was 0.735.Source: Figure 3 · p. 7
  • The gain exceeded the ceiling of configuration-level and single-agent search only as additional agents were introduced.Researcher-only configurations plateaued around the hand-tuned baseline; adding the Reviewer/Fixer and then Criticizer raised the lift to 1.45× and 1.82×.Source: A. System A: Recall@6 Trajectory · p. 7

B. System A: Autonomous Coverage Scale-Up

The agent found a catalog-coverage gap and autonomously added a text-only fallback, expanding retrieval coverage beyond the behavioral graph’s limited item set.

B. System A: Autonomous Coverage Scale-Up

  • The co-purchase graph covered only ∼17% of items, and the hand-tuned baseline omitted the remaining catalog items from its FAISS index.The coverage gap was discovered during an iteration focused on an unrelated loss-function experiment.Source: B. System A: Autonomous Coverage… · p. 7
  • 5.8× coverage expansion came from a text-only fallback the agent proposed and implemented without human direction.Items absent from the behavioral graph received text-derived embeddings projected into the same 128d space via a jointly trained small MLP; a pre-execution review caught a DDP wrapper bug before the successful iteration.Source: B. System A: Autonomous Coverage… · p. 7
  • Scope expansion differs from improving Recall@6, and it depended on pipeline statistics exposing the gap and a Code Fixer catching the fallback’s infrastructure bug.Without the coverage signal, the agent would have continued optimizing on the ∼17% subset.Source: B. System A: Autonomous Coverage… · pp. 7–8

C. System B: Phased Progression

System B’s agent improved weighted coherence within a phase, but a human-driven shift to behavioral+content embeddings produced the campaign best after content-only progress regressed.

C. System B: Phased Progression

  • System B showed the same redirection pattern as System A, but its cross-run paradigm pivot was issued by a human rather than an automated Criticizer.The content-only approach reached a representational ceiling, and the genre-purity constraint prevented recovery through codebook scaling alone.Source: C. System B: Phased Progression · p. 8
  • Weighted coherence reached the campaign best of 0.735 after the human-driven Phase 3→4 pivot to behavioral+content embeddings.The agent’s largest autonomous single-phase gain was 0.348→0.691 in Phase 1; Phase 3 regressed from 0.692 to 0.633.Source: C. System B: Phased Progression · p. 8
  • The three usability criteria were never simultaneously met in 150+ iterations, indicating a structural coherence-density tradeoff rather than a tuning problem.This finding freed the researcher to design a deterministic post-processing solution outside the model; about 65% of iterations were productive.Source: C. System B: Phased Progression · p. 8

D. Cross-System Findings

Across two systems with iteration costs differing by nearly three orders of magnitude, the same failure modes appeared, and validation checks supported the reported gains. The discussion relates scaffolding and human direction to iteration cost, practical value, unresolved limitations, and potential applications beyond AutoResearch.

D. Cross-System Findings

  • The two systems showed the same cost- and horizon-driven failure modes despite nearly three orders of magnitude between their per-iteration costs.Metric fixation appeared wherever evaluation was multi-criteria; incremental search reached local optima, while redirection reframed the problem.Source: D. Cross-System Findings · p. 8

E. Validation Protocol

  • System A’s iter-12 gain held across 11 purchase dates, both evaluators were immutable to agents, and System B’s early architectural choices were retested at larger scale.The System A peak was 6.90% and the mean was 6.18% ± 0.28 pp.Source: E. Validation Protocol · p. 8

VI. DISCUSSION

  • AutoResearch delivered throughput, systematic coverage, and autonomous problem-solving, while shifting human effort toward evaluation design and strategic pivots.The authors estimate savings of approximately 8 engineer-weeks; System B’s iterations established a structural coherence–genre-purity tradeoff, freeing the human to design post-processing.Source: §VI · pp. 8–9
  • Scaffolding differed with iteration cost: the high-cost system used a pre-execution Code Fixer, while the low-cost system used post-execution automatic reversion.The authors present this two-system observation as a parameter to consider, not a proven design principle.Source: §VI · p. 8
  • Recommendations include finding a meaningful, self-contained experiment loop, persisting findings across jobs, redirecting only at stagnation, and treating metric fixation as unsolved.For costly iterations, the authors recommend a semantic gate; they also recommend multi-metric reporting and human QA rather than expecting agents to question metrics.Source: §VI · p. 9
  • The conclusions are bounded by observational System A ablations, one organization and domain, and offline metrics not validated against actual customer outcomes.The metrics were chosen for these systems and have an established correlation with pipeline online metrics; online pilots were underway.Source: §VI · p. 9
  • Open problems include cost-aware planning, parallel exploration, communication between system agents, and teaching agents to assess whether their metric is appropriate.The authors identify metric self-evaluation as the most fundamental problem because current LLMs do not reliably perform this meta-reasoning.Source: §VI · p. 9
  • Prevent, persist, and redirect are proposed for other long-horizon agent loops, with their relative weight scaled to action irreversibility, session length, and search-space dimensionality.The discussion illustrates the principles with agentic software engineering and automated scientific experimentation.Source: §VI · p. 9

VII. CONCLUSION

At production scale, the framework supports a collaboration in which agents map within-model limits and humans design around them. The authors describe the framework’s role and the information available for replication, while noting that the training data are proprietary.

VII. CONCLUSION

  • The framework combines pre-execution prevention, cross-job persistence, and stagnation-triggered redirection to make agent–human collaboration viable when iterations cost hours and hundreds of dollars.The agent maps within-model limits while humans design around them; the paper says AutoResearch replaces researchers’ weekends, not researchers.Source: §VII · pp. 9–10
  • In System A, an explicit-neighborhood directive drove the largest improvement attributed to one directive, while System B’s structural tradeoff led the human to design post-processing.The Researcher could not reach the System A paradigm shift through incremental code search, and the System B solution was one the agent could not have proposed.Source: §VII · p. 9
  • The paper describes its orchestrator and evaluator protocol, but proprietary training data cannot be released; it provides schema and evaluation details to support replication on other behavioral datasets.A reproducibility checklist is submitted alongside the manuscript.Source: REPRODUCIBILITY · p. 10

Related papers

Loading 2609.30541v1…