Source-linked AI summary
Agentic ML Exploration (A-MLE) for Ads Ranking
Erwin Gao, Vinodh Kumar Sunkara, Jingyi Guan, Qinjin Jia, Hangjun Xu, Xiang Ji, Sherman Wong, Surya Teja Chavali, Pratik Vaishnavi, Aryan Pandhi, Xiaoyu Deng, Zhaodong Wang, Samarth Inani, Fan Yang, Jakob Moberg, Zoe Zu, Nicolas Bievre, Sami Khenissi, Amit Jaspal, Ehsan Fakharizadi, Srinidhi Viswanathan, Dorothy Sun, Abishek Vanam, Sneha Iyer, Sheela Yadawad, Wenjie Chen, Gaby Nahum, Junhua Gu, Peter Chu, Yucheng Liu, Xin Zhao, Vitor Cid, Chaorong Chen, Vijay Pappu, Ashwin Kumar, Wenlin Chen, Ben Schulte, Deepak Chandra, Ritwik Tewari
TL;DR
Industrial ads ranking is slowed by heterogeneous models and costly, human-driven iteration, which limits exploration across the portfolio. A-MLE uses a five-stage LLM-agent workflow with shared knowledge, sandboxed execution, and human checkpoints, and reports meaningful improvements on a majority of models, especially the long tail.
Problem
Heterogeneous ranking models and multi-week human iteration limit the number of model × technique combinations explored, leaving long-tail models with little attention.
Method
A-MLE orchestrates hypothesis generation, exploration strategy, experiment execution, result analysis, and shared knowledge through one LLM agent with human checkpoints.
Results
A-MLE produced meaningful relative improvements on a majority of models, with the largest gains on models that historically received the least senior attention.
Takeaways & Limitations
A-MLE operationalizes agentic exploration as a force multiplier for industrial ML engineers, particularly for the long tail of models.
Takeaways & Limitations
Reliability remains constrained by orchestration failures including hallucinated APIs, baseline drift, infrastructure fragility, and over-confident triage.
Abstract
from arXiv · showhide
Modern industrial ads ranking stacks are increasingly bottlenecked not by model capacity or training compute, but by the throughput of human ML iteration - the cycles of research, implementation, training, debugging, evaluation, and launch required to surface a single statistically significant improvement. A typical ranking stack contains numerous differentiated models with heterogeneous data, architectures, and infrastructure constraints, and each cycle takes days to weeks of senior engineer attention per model. As a result, techniques that have proven effective on one model diffuse into others slowly and unevenly, leaving substantial recoverable signal unexplored. We present Agentic ML Exploration (A-MLE), an autonomous LLM-agent system that systematically explores ML techniques across a portfolio of ads ranking models. A-MLE decomposes ML iteration into five stages involving hypothesis generation, exploration strategy, experiment execution, result analysis and shared knowledge substrate which are orchestrated by a single agent that invokes domain-specific skills and agentic workflows against a sandboxed execution layer, with human-in-the-loop checkpoints at each stage boundary. We deploy A-MLE across a representative set of large-scale ads ranking models and evaluate it along a tiered capability framework (tool availability, autonomous workflow execution, and open-ended exploration). We further report a controlled cross-LLM study using a fixed agent loop, which surfaces qualitative differences in execution reliability and exploration aggressiveness across the Claude Sonnet, Gemini, and GPT families. We discuss failure modes and the design choices that govern reliability. Our findings suggest that agentic exploration is a practical force multiplier for ML engineers in industrial recommenders, especially for the long tail of models that rarely receive expert attention.
1 Introduction
Industrial ads ranking portfolios contain heterogeneous models, making technique transfer costly while human ML iteration remains slow. A-MLE addresses this bottleneck with an agent that autonomously explores techniques across models and coordinates the work through shared infrastructure and human review.
- Industrial ads ranking portfolios span differentiated objectives, surfaces, data, architectures, and infrastructure constraints, making proven-technique transfer costly.
- Days-to-weeks iteration cycles and finite engineering capacity restrict exploration to a small subset of model × technique combinations, leaving long-tail models underserved.
- A-MLE uses an LLM agent with domain-specific skills, code execution, and training and evaluation access to explore techniques across ranking models.
- The system decomposes ML exploration into hypothesis generation, planning, execution, result analysis, and shared knowledge, with engineers reviewing at defined checkpoints.
- The paper evaluates A-MLE across a representative model portfolio using a tiered capability framework and reports a concrete improvement headline, cross-LLM comparison, and failure modes.
2 Related Work
Prior automation work targets bounded search spaces, code-generation tasks, or academic benchmarks. A-MLE instead addresses open-ended, infrastructure-constrained exploration across costly industrial ranking portfolios.
- AutoML and Neural Architecture Search: AutoML and neural architecture search automate bounded search spaces around defined objectives, whereas A-MLE permits open-ended code-level changes and a composite industrial objective.
- LLM Agents and Tool Use: A-MLE extends tool-using LLM agents into ML engineering, where adaptive training, failure recovery, evaluation, and candidate selection surround code changes.
- ML Engineering Automation: Unlike data-science agents focused on Kaggle-style benchmarks, A-MLE targets many industry-scale models with large datasets, costly iterations, and statistical launch criteria.
- Recommendation and Ads Ranking: A-MLE treats ads-ranking techniques as a search space and selects which technique to try on which model from the current evidence state.
3 The Manual ML Iteration Bottleneck
Manual ML exploration is a multi-phase process whose cumulative duration limits how many model–technique pairs engineers can investigate. Its stages are constrained by expertise, code complexity, infrastructure variability, and noisy evaluation.
- A canonical manual iteration includes ideation, candidate prioritization, implementation, training and recovery, evaluation triage, and proposal preparation.
- Candidate prioritization allocates limited training compute across hypotheses and determines how many variants each hypothesis receives.
- Implementation and execution require adapting architecture and training code, launching and monitoring jobs, and intervening when shared infrastructure causes failures.
- Evaluation triage compares offline metrics with a rolling baseline, decomposes segment-level gains, and determines whether a candidate merits promotion.
- Multi-week end-to-end iterations limit the number of model–technique pairs explored, while infrastructure stochasticity, metric variance, and baseline drift constrain reliable decision-making.
4 System Design
A-MLE organizes ML exploration as a five-stage workflow operated by one tool-using agent over a shared skill library and sandbox. The design combines adaptive experimentation, statistical evaluation, cross-model knowledge reuse, and human checkpoints.
- System Overview: A-MLE runs hypothesis generation, exploration strategy, experiment execution, result analysis, and shared-substrate stages for each model, objective, and compute session.
- Hypothesis Generation: Hypotheses use current training configuration, baseline metrics, and attempt history, then receive novelty and feasibility scores from an LLM critic.
- Exploration Strategy: The workflow interleaves isolated exploration with exploitation of promising candidates while exposing compute-versus-aggressiveness trade-offs at a user checkpoint.
- Experiment Execution: The agent validates sandboxed code, submits and monitors training, distinguishes infrastructure errors from divergence, and retries or repairs runs when needed.
- Result Analysis: Result analysis tests candidates against a rolling baseline, segments metrics, reruns high-variance experiments, and produces a leaderboard for further rounds or proposal assembly.
- Shared Substrate and Human Checkpoints: A shared per-technique and per-model substrate transfers accumulated knowledge across sessions, while human checkpoints can approve, modify, or terminate each stage.
5 Experimental Setup
The evaluation uses a representative portfolio of ads ranking models and a three-tier capability framework, with rolling and methodological baselines plus throughput, reliability, acceptance, and coverage metrics.
- Models: The model portfolio spans click, conversion, and view objectives, multiple surfaces, and deep cross network, deep interest network, and multi-tower architectures.
- Tiered capability framework: The benchmark uses three tiers: tool availability, autonomous workflow execution, and open-ended exploration.L1 tests focused access to model operations; L2 tests multi-step workflow operation; L3 tests improvement discovery under compute constraints.
- Baselines: The rolling baseline is the current reference training configuration, while manual and semi-automated baselines represent engineer-led and script-assisted iteration.
- Metrics: Evaluation metrics include completed iterations per engineer-week, training success, proposal acceptance, and technique coverage.Completed iterations end in either a documented proposal or a documented null result.
- Reporting: Performance results are reported as relative improvements against the relevant baseline and assessed for statistical significance using the rolling-baseline framework.
6 Results, Analysis, and Discussion
A-MLE improves iteration throughput and proposal quality while showing that domain-specific skills are crucial for tool-level operations; its results include successful execution, stronger review acceptance, and a headline exploration gain.
- End-to-end throughput: Multiple times the productivity in completed iterations per engineer-week was delivered by A-MLE versus the baseline.The semi-automated baseline achieved a smaller throughput improvement and weaker offline impact, with longer convergence cycles.
- Training reliability: A-MLE-triggered training runs meaningfully surpassed the baseline in successful completion after automated debugging and bounded retries.A small minority still required human attention when recovery exceeded the retry limit.
- Proposal quality: A-MLE-authored proposals passed human review at a much higher rate than the baseline.Reviewers highlighted statistical analysis, negative-result documentation, and segment-level decomposition.
- Tier-1 capability bench: The domain-equipped agent achieved 68% overall L1 accuracy versus 16% for the generic ML agent and 8% for the generic LLM.On job-config modification, the domain-equipped agent solved all questions while generic configurations scored below 40%.
6.5 Tier-2 Workflow Execution
Tier-2 evaluation tests whether A-MLE can operate asynchronous, failure-prone workflows, while tier-3 evaluation measures open-ended improvement and cross-LLM differences under a fixed loop.
- Tier-2 Workflow Execution: Four tier-2 tasks require baseline refresh, variance testing, configuration-change experiments, and end-to-end workflow handling.The tasks involve submitting training workflows, monitoring asynchronous completions, recovering from failures, and summarizing results.
- Tier-2 Workflow Execution: The domain-equipped A-MLE configuration completes all four tier-2 tasks end-to-end with high reliability.Its differentiators include waiting for external events, distinguishing infrastructure failures from training divergence, and reliable summarization.
- Tier-3 Exploration Outcomes: A-MLE delivered measurable offline improvements on a majority of evaluated models at the open-ended exploration tier.
- Tier-3 Exploration Outcomes: +2.557% relative improvement on the regression objective was obtained with neutral training throughput (+0.42% QPS).This multi-source result exceeded the single-hypothesis baseline.
- Cross-LLM Comparison: With the agent loop, skills, and prompts fixed, the cross-LLM study compares L2 task completeness and L3 exploration outcomes across Claude Sonnet, Gemini, and GPT families.
- Cross-LLM Comparison: Sonnet ≥3.5, Gemini 2.5, and GPT-5 achieved high-90s average L2 task completeness, while model families differed in exploration aggressiveness and prompt sensitivity at L3.Under stressful prompts, Sonnet 4.0 reached approximately 2.6×10−2 rMSE, while GPT-5 became more conservative.
6.8 Agent-Surfaced Techniques
A-MLE surfaced technique families across the model portfolio, with strongest results from transferring validated techniques to structurally similar models. Reliability remained limited by baseline changes and recurring execution and evaluation failure modes.
- Agent-Surfaced Techniques: Technique transfer produced A-MLE’s strongest results by applying techniques validated on one model to structurally similar models.The agent struggled when models had recently undergone non-trivial baseline changes because its hypotheses reflected prior model versions.
- Agent-Surfaced Techniques: Table 2 counts technique-family attempts and candidates that passed human statistical-significance gating without rework.The table buckets model counts as many (≥8), several (3–7), or few (≤2).
- Failure Modes: A-MLE exhibited five recurring failure modes spanning hallucinated APIs, baseline drift, infrastructure fragility, over-confident triage, and LLM-specific workflow failures.Pre-flight checks, retries, automatic reruns, and orchestration mitigations reduced some failures but did not eliminate all of them.
- Discussion: Reliability depended more on the orchestration harness than on the underlying model’s reasoning capability.The relevant harness components were skill-library coverage, statistical evaluation rigor, and resilience to infrastructure noise.
7 Conclusion and Future Work
A-MLE reframes industrial ML progress around increasing human-iteration throughput and implements that view through a five-stage agent operating over shared skills and a sandbox. Across evaluation, it produced meaningful relative improvements on most models, with the largest gains on the long tail, while future work targets resilient execution, stronger hypotheses, and broader exploration.
- Conclusion: A-MLE reframes the industrial ML bottleneck as human-iteration throughput rather than the ceiling of any single model.Its implementation uses a five-stage agent over a shared skill library and execution sandbox.
- Conclusion: A-MLE produced meaningful relative improvements on a majority of models, with the largest gains on models that historically received the least senior attention.This conclusion identifies the long tail as the primary area of improvement.
- Future Work: Future work targets resilient execution, richer domain-specific skills, deeper architectural understanding, and expansion from breadth-oriented exploration.The stated directions include distinguishing infrastructure noise from genuine divergence and extending exploration beyond rapidly scaling proven techniques.
A Skill Library Excerpt
The skill library exposes structured procedures covering codebase navigation, training configuration, launch and monitoring, evaluation, and proposal authoring. These skills connect agent decisions to validated edits, monitored runs, statistical analysis, and launch recommendations.
- Skill Categories: Codebase-navigation skills locate model implementations, training entry points, active feature groups, and recently modified files.These procedures help the agent identify where an exploration should be implemented.
- Skill Categories: Training-configuration skills read current settings, propose structured edits, and validate changes against schemas.The skills expose typed procedures with structured input and output schemas.
- Skill Categories: Launch-and-monitoring skills estimate compute, submit training runs, poll status, retrieve logs, and distinguish infrastructure errors from training divergence.This supports execution under shared compute and infrastructure failures.
- Skill Categories: Evaluation skills compute statistical significance against a rolling baseline and decompose results by segment.Proposal-authoring skills then assemble experiment design, results, confidence intervals, and recommendations.
B Phase Boundary Conventions
A-MLE places human review checkpoints at each stage boundary, from hypothesis approval through proposal revision and routing. Reviewers can substitute hypotheses, confirm compute plans and analyses, and revise proposals before standard review.
- Phase Boundary Conventions: After hypothesis generation, a reviewer approves the proposed hypothesis or substitutes one.This checkpoint preserves human control over the direction of exploration.
- Phase Boundary Conventions: After exploration strategy, a reviewer confirms the planned experiment set and compute allocation.The checkpoint occurs before execution begins.
- Phase Boundary Conventions: After experiment execution, a reviewer confirms run results and statistical analysis before proposal authoring.This places human review between empirical evaluation and recommendation drafting.
- Phase Boundary Conventions: After proposal authoring, a reviewer revises the proposal as needed and routes it through the standard review process.The final checkpoint governs the handoff from agent-authored proposal to established review.