Source-linked AI summary
AutoLR: Automating the Path from Research to Launch Review in Industrial Recommender Systems
Qi Zhang, Yanlin Chen, Wenchao Xiao
TL;DR
Industrial recommender development remains difficult to coordinate across research, implementation, evaluation, and launch decisions, especially during long-running experimentation. AutoLR addresses this gap with an evidence-governed harness combining expert proposal review, budgeted selection, layered knowledge, and deterministic control. In the studied deployment, it completed 1,586 evaluations across two scenarios, supported nine production Launch Review records, and ran routine iterations at RMB 3–4 per iteration, while online testing and rollout remained human-gated.
Problem
Long-running recommender experimentation requires coordinated handoffs and explicit governance that LLM agents alone do not provide.
Method
AutoLR combines multi-expert proposal review, evidence-weighted trial selection, layered experiment memory, LLM reasoning and code generation, and deterministic controllers for execution, verification, guardrails, and persistent state.
Results
1,586 unique completed evaluations across two recommendation scenarios and nine production Launch Review records were documented, while routine iterations ran at an observed LLM API cost of RMB 3–4 per iteration.
Takeaways & Limitations
Governed automation can operate from research through offline candidate packaging, but online A/B admission and full-traffic rollout remain human-gated in the studied deployment.
Takeaways & Limitations
The study does not estimate component-level causal effects or a false-promotion rate and evaluates one application with two recommendation scenarios.
Abstract
from arXiv · showhide
Improving an industrial recommender is an iterative research-and-engineering process rather than a direct path from idea to deployment. In \textbf{DASHEN, NetEase's gaming-community app}, algorithm engineers typically identify promising directions from research papers, technical reports, and prior production experiments; reproduce or adapt the underlying methods; implement them in the production codebase; and evaluate the resulting models through training and offline experiments. Promising candidates are then advanced to online A/B tests, and those demonstrating robust gains are submitted to Launch Review---the internal gate for full-traffic rollout. Large language models (LLMs) can assist with individual stages of this workflow, but the overall process remains human-dependent without a harness that can reliably coordinate them across long-running, often multi-day experimental cycles. We present \textbf{AutoLR}, initially built as \textbf{Auto Launch Review} and later extended upstream into an autonomous research-to-launch harness. AutoLR combines three system mechanisms: a \textbf{multi-expert council} that debates and adversarially reviews proposals; a \textbf{deterministic evidence-weighted exploration--exploitation selector} that allocates a limited trial budget across candidate directions and uses Council reranking; and a layered knowledge system that combines external research, production-system knowledge, and DASHEN-specific domain knowledge---such as game communities, player characteristics, and content-interaction patterns---with posterior evidence from configurations, patches, logs, failures, and offline outcomes. LLM agents perform semantic reasoning and code generation, while deterministic controllers retain authority over execution, metric extraction, guardrails, and persistent state transitions.
1 Introduction
AutoLR addresses the human-dependent handoffs that constrain long-running industrial recommender experimentation by automating the path from research through offline candidate packaging while retaining human control over online testing and rollout. Its governed harness combines lifecycle automation, expert proposal review, evidence-weighted trial allocation, layered knowledge, and persistent experiment memory.
- Motivation: Industrial recommender improvement is constrained by repeated handoffs among research, code, training, logs, metrics, and business decisions, not by a single workflow step.As systems mature, expected gains shrink while plausible ideas multiply, leaving human attention, engineering time, and lesson preservation as limiting resources.
- AutoLR: AutoLR autonomously retrieves evidence, generates and debates hypotheses, selects directions, modifies and verifies code, trains and evaluates models, and packages promising candidates.The system is designed to reach Launch Review readiness rather than automate the review meeting itself.
- Governance: Human authority remains over online A/B admission and full-traffic rollout, while later production results feed subsequent research cycles.In the studied deployment, autonomy ends at offline evaluation and candidate packaging.
- Decision process: A role-specialized council generates, debates, and adversarially reviews candidate directions, while a deterministic evidence-weighted selector allocates limited trials using prior outcomes and council reranking.This combines semantic proposal review with bounded exploration–exploitation decisions.
- Knowledge: AutoLR grounds agents in external research, production-system facts, and DASHEN domain knowledge while accumulating configurations, patches, logs, failures, offline results, and production outcomes.The layered knowledge and experiment-memory system preserves lessons across iterations.
- Evidence: A path-aware audit identified 1,586 unique completed evaluations across two DASHEN recommendation scenarios, with nine production Launch Review records documenting online outcomes for engineer-selected candidates.The scenarios are the standard single- and two-column feed and the immersive-video feed.
2 Related Work
Related work establishes open-ended LLM experimentation and industrial recommendation loops, but AutoLR emphasizes governed, longitudinal experimentation rather than claiming a uniquely long pipeline or unique authority split. The paper also frames adaptive evaluation and mutable baselines as reliability risks requiring explicit state and machine-enforced controls.
- From AutoML to autonomous engineering: Classical AutoML searches predefined spaces, whereas LLM agents can generate and implement new ranking-system directions beyond parameter selection.Examples include new sequence encoders, task relationships, sample-weighting rules, and reward formulations.
- Governance: Industrial systems add governance requirements: they must control context, tools, experiment state, recovery, memory, metrics, retries, and persistent-state authority.A capable foundation model alone does not determine which proposal to trial or when evidence justifies a state change.
- Positioning: AutoLR overlaps with prior systems in experiment memory, direction selection, and deterministic promotion gates, so it does not claim a uniquely long automation pipeline or unique division of authority.Its distinction is a Launch-Review-centered, longitudinal production study of an evidence-governed experimentation loop.
- Adaptive evaluation: Repeatedly selecting hypotheses after inspecting the same held-out data can overfit evaluation sets and invalidate ordinary statistical conclusions.Reusable holdouts, the Ladder, always-valid inference, and online false-discovery control are cited as mitigation approaches.
- Selection bias: Selecting winners from many noisy trials creates selection bias, especially when gains are comparable to variation from seeds, benchmark noise, or baseline tuning.The paper relates this winner’s-curse effect to industrial online controlled experiments.
- Mutable baseline: The KEEP ratchet makes evaluation errors persistent because accepted candidates become baselines for later trials and can create apparent progress that does not persist.A KEEP changes the repository artifact used for subsequent experiments, not merely a reported score.
- Promotion controls: AutoLR combines a primary optimization score with independent protected-metric guardrails, allowing a candidate to be rejected despite ranking highly on the primary score.This non-compensatory rule mirrors human Launch Review veto semantics.
3 AutoLR System
AutoLR coordinates research-to-offline-packaging through council-based proposal generation, deterministic direction selection, repository-grounded execution, and evidence-controlled state transitions. LLMs supply reasoning and code proposals, while deterministic controllers govern execution, metrics, guardrails, and persistent state.
- System lifecycle: AutoLR retrieves evidence, proposes and reviews directions, selects experiments, modifies the repository, runs training and evaluation, diagnoses outcomes, and packages promising candidates.Admission to online A/B testing and full-traffic rollout remains human-gated.
- Authority model: LLM agents handle research, hypothesis generation, debate, repository understanding, code generation, and diagnosis, while deterministic controllers govern execution, validation, metrics, guardrails, and state transitions.Authoritative transitions require machine-validated artifacts rather than raw LLM outputs.
- Multi-Expert Proposal Council: The Multi-Expert Proposal Council independently generates hypotheses, reconciles mechanisms, adversarially reviews survivors, and emits candidates with falsifiable hypotheses, edit surfaces, activation checks, and protected metrics.Hard feasibility constraints reject infeasible proposals regardless of novelty or expected gain.
- Evidence-Weighted Direction Selection: Direction selection constructs a diversity-aware frozen candidate window, applies advisory council reranking, and executes only the top-ranked admitted candidate before training.Operator-prioritized candidates retain precedence, and incomplete rankings are repaired by the controller.
- Evidence-Weighted Direction Selection: A deterministic memory gate scores candidates using historical success, expected improvement, novelty, feasibility, stability, confidence, and risk before GPU training.The selector uses a Beta(1, 1) pseudocount as a scoring feature, not posterior sampling; outcomes and failure diagnostics update later rounds.
- Evidence-Weighted Direction Selection: AutoLR is characterized as a deterministic evidence-weighted selector with diversity constraints and council reranking, not as Thompson sampling or a Bayesian bandit.The integrated production corpus does not establish superiority over random, greedy, or direct LLM-based routing under a matched trial budget.
- Offline Promotion and Launch-Review Handoff: KEEP rewrites the working repository, PACK retains candidate lineage, and DISCARD rejects candidates under a decision contract requiring complete scalar and guardrail evidence.A one-run threshold crossing is not a confirmed effect, and missing historical payload is not compliance.
- Offline Promotion and Launch-Review Handoff: The audited controller used a fixed KEEP floor when noise calibration was disabled, while a non-promotable CALIBRATION_REQUIRED state is proposed as a future safeguard.The proposed safeguard is not implemented in the audited controller.
4 Production Evaluation
The production evaluation audits AutoLR across two DASHEN recommendation scenarios using deduplicated experiment records and session-level uncertainty handling. It frames the analysis around operational scale, reported online movements, and reliability risks, while treating historical LLM-cost changes as non-causally attributable.
- Evaluation Scope: The audited corpus covers the single- and two-column feed and the immersive-video feed, both evaluated with fixed multi-objective offline protocols.Within sessions, AutoLR may modify designated model and configuration code while training and evaluation remain fixed.
- Corpus Construction: 4,250 non-empty ledger records yielded 4,247 parseable records, 3,289 canonical experiment records, and 1,586 completed evaluations after path-aware deduplication and filtering.The final corpus contains completed non-baseline evaluations.
- Statistical Treatment: Uncertainty intervals resample whole experiment sessions within each recommendation scenario because evaluations sharing data windows, trunks, and history are correlated.Individual evaluation rows are therefore not treated as independent samples.
- Research Questions: The evaluation asks how AutoLR supported routine iteration, what online movements appeared in engineer-selected Launch Review packages, and what risks arise when one-run offline results change later baselines.These correspond to RQ1, RQ2, and RQ3.
- Operational Analysis: Historical LLM-cost analysis cannot causally attribute changes to a single factor because harness version, model family, task mix, and accumulated experiment memory changed together.It is reported as an operational observation rather than a research question.
5 Results
AutoLR operated at production scale across two recommendation scenarios, completing routine offline iteration and advancing candidates into online Launch Review. The audit also exposes promotion reliability risks, including incomplete calibration, mutable baselines, and heterogeneous evidence.
- Operating scale: AutoLR completed 1,586 evaluations across the single- and two-column feed and immersive-video feed, with deterministic services controlling the train–infer–evaluate path.Routine proposal execution, repository-grounded implementation, training, offline evaluation, and experiment-state updates were handled within the workflow.
- Operating scale: Engineers retained authority over online A/B admission and full-traffic rollout while AutoLR handled routine research, implementation, training, and offline evaluation.Human responsibilities shifted toward objectives, evaluation contracts, protected metrics, code boundaries, governance, and ambiguous failures.
- Operating scale: Structured failures and recovery events, together with 1,586 completed evaluations, support sustained experimentation with explicit failure handling but do not establish component-level superiority.Recorded failures include infeasible proposals, inactive mechanisms, smoke-test failures, timeouts, and training crashes.
- Launch Review outcomes: Nine production Launch Review records show AutoLR-generated candidates progressed beyond offline evaluation into online A/B testing and production launch decisions.The records cover the two primary recommendation scenarios and provide production evidence without implying pooled or immutable offline-to-online linkage.
- Launch Review outcomes: +5.75% content-consumption penetration rate, +10.83% total content-consumption time, and +5.55% total valid content views were the arithmetic sums of reported relative lifts across nine records.These descriptive sums are not pooled treatment effects; metrics not separately disclosed were treated as flat.
- Broader deployment: AutoLR transferred to search, user-growth, and commercialization workflows, with positive online results reported for search and user-growth and online validation ongoing for commercialization.The paper states that commercialization had completed initial exploration and produced clear offline AUC improvements.
- Evidence scope: The audit reconstructed 1,586 completed evaluations from 4,250 physical ledger records, while nine heterogeneous Launch Review records were not pooled and lacked immutable offline-to-online-to-review linkage.Of the completed evaluations, 1,583 retained numeric scalar deltas.
- Promotion reliability: The replayed reference floor of 10^-3 was descriptive rather than an audited operative threshold, and its crossing rates were selective relative to the realized candidate distribution.The IQR did not establish statistical significance, an optimal threshold, or the historical threshold for evaluations without provenance.
6 Discussion and Limitations
AutoLR argues that durable autonomy depends on governed experiment lifecycles rather than autonomous agents alone. Its evidence supports operational viability, while leaving component effects, false-promotion rates, and broader transfer effectiveness unresolved.
- Implications: Deterministic evidence contracts, guardrails, recovery policies, and state transitions let semantic components evolve without granting them authority over persistent state.The proposed design treats the governed experiment lifecycle as the durable unit of autonomy.
- Limitations: The production corpus supports operational viability of the integrated design but does not isolate the causal contribution of individual mechanisms.This limits attribution of observed behavior to specific AutoLR components.
- Implications: Persistent promotion should require meaningful improvement, protected-metric compliance, and protocol-matched confirmation; unavailable calibration evidence should cause promotion to fail closed.PACK can retain promising candidates without changing the trunk, while online A/B admission and full-traffic rollout remain human-gated.
- Limitations: The study demonstrates sustained production experimentation and identifies mutable-baseline risk, but does not estimate component-level causal effects or a false-promotion rate.The archive also lacks matched-budget ablations, broad fixed-snapshot repeats, complete threshold and guardrail provenance, and immutable offline-to-online lineage.
7 Conclusion
AutoLR automates the upstream research-to-review path while keeping execution, verification, guardrails, and persistent offline state under deterministic control. Its deployment audit indicates operational viability, but component-level causal contributions remain unisolated and long-running operation exposes risks from non-reproducible promotions.
- System scope: AutoLR automates evidence retrieval, proposal generation, direction selection, implementation, training, offline evaluation, and candidate packaging before human-gated online admission and rollout.LLM agents provide semantic reasoning and code generation, while deterministic controllers retain authority over execution, verification, guardrails, and persistent offline state.
- Operational evidence: 1,586 unique completed evaluations were recovered across the single- and two-column feed and immersive-video feed, with nine production Launch Review records reporting positive online movements.The records concern engineer-selected candidate packages.
- Operational evidence: Routine AutoLR iterations migrated to a mixed DeepSeek-V4-Pro/Flash stack at an observed LLM API cost of RMB 3–4 per iteration.The reported cost excludes model-…
- Evidence boundary: The audit demonstrates integrated-lifecycle viability but does not isolate the causal contribution of individual AutoLR components.
- Reliability boundary: The KEEP ratchet can let a non-reproducible promotion propagate into later comparisons, creating apparent progress that may not persist.Because each KEEP changes the baseline for subsequent experiments, autonomous systems require conservative state transitions and explicit human authority over production deployment.