Source-linked AI summary
Self-Evolving Recommendation System: End-To-End Autonomous Model Optimization With LLM Agents
Haochen Wang, Yi Wu, Daryl Chang, Li Wei, Lukasz Heldt
TL;DR
Large-scale recommendation optimization must bridge proxy training objectives and delayed, semantically complex user satisfaction while reducing reliance on manual experimentation. The paper uses Gemini-based LLM agents to generate, train, and deploy model changes through fast offline discovery and slower online validation. Production deployments at YouTube report successful autonomous evolution in experimentation velocity and model performance.
Problem
Recommendation optimization requires designing sophisticated optimizers, architectures, and rewards, but training proxies do not directly capture delayed, sparse, and semantically complex user satisfaction.
Method
A self-evolving system uses specialized LLM MLE agents to analyze logs, generate hypotheses, implement model changes, train candidates, and validate them through offline proxies and online metrics.
Results
The system achieved successful production launches at YouTube and demonstrated autonomous discovery of structural and semantic recommendation changes.
Takeaways & Limitations
LLMs grounded with appropriate context and tools can support structural and semantic innovation in industrial recommendation systems.
Takeaways & Limitations
Proxy-loss values are comparable only when the loss definition remains invariant, and the system's stated online objective is subject to explicit metric and cost constraints.
Abstract
from arXiv · showhide
Optimizing large-scale machine learning systems, such as recommendation models for global video platforms, requires navigating a massive hyperparameter search space and, more critically, designing sophisticated optimizers, architectures, and reward functions to capture nuanced user behaviors. Achieving substantial improvements in these areas is a non-trivial task, traditionally relying on extensive manual iterations to test new hypotheses. We propose a self-evolving system that leverages Large Language Models (LLMs), specifically those from Google's Gemini family, to autonomously generate, train, and deploy high-performing, complex model changes within an end-to-end automated workflow. The self-evolving system consists of an Offline Agent (Fast Loop) that performs high-throughput hypothesis generation to optimize for proxy metrics, and an Online Agent (Slow Loop) that validates candidates against delayed north star business metrics in live production. Our agents act as specialized Machine Learning Engineers (MLEs): they exhibit deep reasoning capabilities, discovering novel improvements in optimization algorithms and model architecture, and formulating innovative reward functions that target long-term user engagement. The effectiveness of this approach is demonstrated through several successful production launches at YouTube, confirming that autonomous, LLM-driven evolution can surpass traditional engineering workflows in both development velocity and model performance.
1 Introduction
Large-scale recommendation systems must optimize long-term user satisfaction despite semantic reward-design challenges that exceed conventional AutoML. The paper introduces an autonomous LLM-agent framework that evolves recommendation models through end-to-end MLE workflows.
- Recommendation systems are increasingly framed as reinforcement-learning problems that balance immediate gratification with delayed rewards such as retention and content exploration.
- The central bottleneck is aligning differentiable training proxies with delayed, sparse, non-differentiable, and semantically complex user satisfaction.
- Traditional AutoML tunes numerical hyperparameters but cannot invent reward logic, interpret experiment results, or design novel interaction layers.
- The proposed YouTube system uses LLM agents as expert MLEs that read production code, modify neural architectures, and formulate reward-function logic autonomously.
- The contributions include a hierarchical autonomous MLE framework, semantic discovery of architectures and multi-objective rewards, and faster experimentation with metric gains.
2 Related Work
Prior optimization methods automate parameter selection or search within predefined spaces, while newer LLM agents generate and refine solutions using reasoning and tools. Recommendation reward design remains harder because real-world satisfaction lacks a clear oracle and arrives through delayed, noisy interactions.
- Automated machine-learning methods typically tune continuous hyperparameters within predefined ranges but lack semantic understanding of why configurations succeed or fail.
- Neural architecture and optimizer search can remix predefined operations but cannot invent new modules, refactor bottlenecks, or add unprogrammed logic.
- LLM optimization methods iteratively refine solutions from natural-language descriptions, supported by advanced reasoning and long-context understanding.
- Agent frameworks interleave reasoning with tool execution, and scientist agents write executable code for open-ended problem solving.
- Recommendation reward engineering lacks a clear oracle for user satisfaction, unlike robotics or simulations with immediate feedback.
- Real-world interactions arrive over Θ(days) or Θ(weeks), requiring reward designs to align offline proxy analyses with eventual deployment validation.
3 Problem Formulation
The paper formulates recommendation optimization as a bi-level problem: train ranking models on proxy rewards, then optimize system configuration for online north-star metrics under constraints. An MLE agent automates the traditionally human process of refining optimizers, architectures, and reward definitions.
- 3 Problem Formulation: The ranking task is formulated as bi-level optimization for maximizing expected long-term user satisfaction.
- 3 Problem Formulation: Direct optimization of online satisfaction is intractable because feedback is sparse, delayed, and noisy.
- 3.1 The Lower Level: Model Training: At the lower level, ranking models optimize an engineered cumulative-reward proxy objective using training data.
- 3.1 The Lower Level: Model Training: The deployed value-based RL model estimates long-term ranking-action value with Qθ(s,a) and trains using SGD on a differentiable proxy loss.
- 3.1 The Lower Level: Model Training: The meta-configuration Φ includes the optimizer, model architecture, and reward definition that determine lower-level training.
- 3.2 The Upper Level: Optimizing North Star Metrics: Improving offline loss does not guarantee better north-star metrics because the proxy-reward mapping to user satisfaction is not guaranteed.
- 3.2 The Upper Level: Optimizing North Star Metrics: The upper-level search optimizes Φ against expensive, noisy online feedback while respecting system constraints such as training cost.
- 3.2 The Upper Level: Optimizing North Star Metrics: The framework automates human research by iteratively refining learning-rate rules, ranking-network structure, and reward logic balancing engagement signals.
4 The Self-Evolving System
The system separates autonomous recommendation-model evolution into synchronized fast and slow reasoning loops sharing an Experiment Journal. Specialized LLM personas generate, score, rank, and validate candidate changes while human engineers provide the initial idea and review final metrics.
- The Experiment Journal stores configurations, offline scores, historical trials, and available online metrics for both agents.
- Offline Agent: The Offline Agent rapidly generates candidates, schedules training or analysis, and minimizes proxy metrics in historically promising regions.It runs daily with frequent wakeups and uses specialized personas, tools, and historical metrics to guide proposals.
- Online Agent: The Online Agent ranks candidates using north star metrics, promotes promising configurations to live experiments, and terminates unpromising trials.This slow loop reserves expensive online traffic for candidates with demonstrated potential.
- Specialized personas: Specialized personas divide recommendation-model design into optimizer, architecture, and reward tasks to avoid the hallucinations observed in monolithic prompts at ~400k tokens.The personas use task-specific tools and objectives, including compute_loss, run_sql_query, and compute_eval.
- Scoring and reward engineering: Optimizer and architecture personas compare candidates with invariant loss, whereas reward engineering uses loss-independent proxies because changing rewards alters the optimization landscape.Reward discovery combines large-scale user-log analysis with candidate evaluation.
- The dual-agent filtration funnel limits human involvement to presenting an initial research idea and reviewing experiment metrics.
5 Deployment and Results
The system was deployed across critical YouTube surfaces and evaluated through offline validation followed by online A/B experimentation. Candidates reaching the slow loop improved north star metrics, while launches outperformed many traditional manual launches.
- The framework was deployed across several critical YouTube surfaces and compared against human-engineered baselines.
- Offline validation tested whether agents found candidates with lower loss or stronger correlation with key signals, followed by online A/B experiments against north star metrics.The production model is a deep-neural-network reinforcement-learning system for video ranking, with training typically requiring Θ(hours).
- 64% of traditional YouTube-level launches and 73% of surface-level launches were outperformed on average by agent-generated improvements.
5.2 Evaluation of the Optimizer and Architecture Components: Loss Optimization
The Offline Agent improved optimizer and architecture components by minimizing offline loss and produced changes that transferred to live traffic. It also reduced training latency while preserving convergence and business metrics.
- Switching from legacy Adagrad to RMSprop with tuned parameters significantly reduced offline loss and improved live traffic.The agent searched for an effective optimizer without requiring a predefined list of available optimizer names.
- System optimization: Training time improved by 8× without sacrificing business metrics after iterative reductions of 4× and 2× in training latency.The reductions came from adjusting batch sizes, training epochs, and optimizer hyperparameters without degrading convergence.
- Architecture refinement: After exploring hundreds of topology candidates, the agent proposed a Gated Path architecture that introduced a multiplicative gate and produced robust deployment gains.
5.3 Evaluation of the Reward Component: Semantic Alignment via Signal Correlation
The reward component required loss-independent reasoning about semantic user behavior and conflicting business objectives. The agent synthesized a new engagement signal and tuned existing reward hyperparameters through online experimentation.
- Reward optimization uses a loss-independent proxy because reward changes alter the optimization landscape and make direct loss comparisons ill-defined.
- Semantic discovery: The agent synthesized a reward function containing a novel signal for active content engagement through iterative analysis of user interaction patterns.The reward search spans Θ(100) signals and combinations, including watch time, survey responses, and retention metrics.
- Hyperparameter tuning: The agent tuned four reward hyperparameters in two weeks after manual attempts over several months failed to improve both YouTube-level and surface-level metrics.This exploration used only the Online Agent’s slow loop and no offline metrics.
5.4 LLM Performance and Ablation Studies
The ablations show that discovery quality depends on model reasoning capability, expert MLE framing, and comprehensive ranked experiment history. The evaluation uses normalized offline loss, where more-negative scores are better.
- Evaluation scope: The ablation benchmarks examine model selection, persona framing, and context management as determinants of discovery performance.These dimensions are presented as sources of sensitivity in the discovery process.
- Evaluation setup: The study evaluates optimizer discovery across six independent runs, with 70 ideas explored per run and lower normalized z-scores representing superior performance.The proxy loss remains comparable when the loss definition is invariant.
- Model size and reasoning: Gemini 2.5 Pro consistently achieves lower normalized loss than Gemini 2.5 Flash, indicating stronger performance for algorithmic discovery.Results are averaged over six independent runs exploring 70 ideas each.
- Persona framing: Expert MLE persona framing improves the relevance and depth of proposed configurations compared with removing the expert identity.The comparison treats persona framing as an explicit ablation factor.
- Context management: Full, loss-sorted experiment history performs better than no history, restricted top-k history, or timestamp-ordered history.The findings identify comprehensive ranked context as important for iterative discovery.
5.5 Efficiency and Costs
The system improves experimental efficiency by separating fast offline discovery from slower online validation. Its token costs were modest, while model compute and memory costs remain comparable to human-driven workflows.
- Velocity: Decoupling offline discovery from online validation removes repetitive manual work and shortens the Idea-to-Data cycle.The resulting velocity enables more launches than previously possible.
- Costs: Approximately $20,000 in LLM token costs accrued over six months, described as a minimal fraction of a full-time MLE cost.This figure covers the reported six-month period.
- Costs: Per-model compute and memory costs are identical for human-driven and agentic workflows, so remaining infrastructure costs scale with models tested.The comparison isolates infrastructure scaling from LLM token costs.
5.6 Lessons Learned
Production deployment exposed practical prompting and initialization requirements for reliable autonomous discovery. Delta-based configuration generation, explicit diversity instructions, and warm starts address distinct failure modes.
- Scope: The deployment lessons span production stability concerns and the reasoning capabilities unlocked by an agentic system.The lessons are organized as practical considerations for operating the self-evolving system.
- Production stability: Generating configuration deltas against the production file improves proposal validity compared with requesting complete configuration files.Complete-file generation often caused omissions and hallucinations.
- Exploration strategy: Explicitly prompting for exploration, exploitation, and innovation prevents the agent from collapsing into minor incremental hyperparameter tuning.Without this instruction, proposed changes could remain narrowly incremental.
- Initialization: Warm-starting the Offline Agent with strong prior configurations is important because proposal quality depends on Experiment Journal density.Without past trials, the agent tends to propose generic textbook improvements.
6 Conclusion
The paper presents a production-deployed LLM framework that autonomously evolves recommendation systems through fast offline discovery and slower online validation. It reports structural and semantic innovation, compressed experimentation, and a future role for MLEs centered on guardrails and long-term direction.
- Framework: The framework uses an Offline Agent for cheap proxy signals and an Online Agent for delayed north star business metrics.This decoupling supports autonomous recommendation-model development at production scale.
- Contribution: LLMs grounded with context and tools discover structural and semantic recommendation-system changes beyond repetitive engineering mechanics.The paper reports successful deployment at scale on a large video platform.
- Impact: Automating code generation, compilation, and experiment orchestration compresses the Idea-to-Data cycle and increases experimental throughput.The increased throughput allows exploration of configuration-space regions human engineers lack bandwidth to investigate.
- Future role of MLEs: The envisioned MLE role shifts toward strategic guardrails, ethical constraints, and long-term system vision as autonomous modeling improvements proceed.The paper frames this as a future direction rather than a completed operational result.