Source-linked AI summary
Decoding ML Decision: An Agentic Reasoning Framework for Large-Scale Ranking System
Longfei Yun, Yihan Wu, Haoran Liu, Xiaoxuan Liu, Ziyun Xu, Yi Wang, Yang Xia, Pengfei Wang, Mingze Gao, Yunxiang Wang, Changfan Chen, Wenjie Fu, Hong Yan, Junfeng Pan
TL;DR
Large-scale ranking optimization is hindered by the difficulty of converting ambiguous product intent into executable, verifiable hypotheses and by the gap between offline optimality and production feasibility. GEARS coordinates programmatic candidate search, specialized ranking knowledge, and deterministic validation, achieving strong benchmark and production results while remaining subject to the scope of structured, verifiable instructions.
Problem
Large-scale ranking systems face an engineering-context constraint: translating product intent into auditable hypotheses, while static optimization can produce offline policies that are brittle or undeployable in production.
Method
GEARS uses an LLM-based controller to coordinate programmatic filtering, domain-grounded interpretation through Specialized Agent Skills, and deterministic validation or governance.
Results
GEARS achieves 0.94 nDCG@1 on 100 structured policy-selection instructions and delivers consistent metric improvements across nine production surfaces, reducing engineering cycles from weeks to days.
Takeaways & Limitations
The framework supports ranking-policy discovery by combining algorithmic candidate search with contextual ranking knowledge and production-oriented validation.
Takeaways & Limitations
The benchmark measures the full pipeline on structured instructions with verifiable answers rather than open-ended reasoning.
Abstract
from arXiv · showhide
Modern large-scale ranking systems operate within a sophisticated landscape of competing objectives, operational constraints, and evolving product requirements. Progress in this domain is increasingly bottlenecked by the engineering context constraint: the arduous process of translating ambiguous product intent into reasonable, executable, verifiable hypotheses, rather than by modeling techniques alone. We present GEARS (Generative Engine for Agentic Ranking Systems), a framework that reframes ranking optimization as an autonomous discovery process within a programmable experimentation environment. Rather than treating optimization as static model selection, GEARS leverages Specialized Agent Skills to encapsulate ranking expert knowledge into reusable reasoning capabilities, enabling operators to steer systems via high-level intent vibe personalization. Furthermore, to ensure production reliability, the framework incorporates validation hooks to enforce statistical robustness and filter out brittle policies that overfit short-term signals. Experimental validation across diverse product surfaces demonstrates that GEARS consistently identifies superior, near-Pareto-efficient policies by synergizing algorithmic signals with deep ranking context while maintaining rigorous deployment stability.
1 Introduction
GEARS addresses the engineering-context and deployment gaps in large-scale ranking optimization by coordinating programmatic search, domain expertise, and deterministic validation. Across production-scale evaluation, it identifies strong policies while reducing engineering effort and maintaining deployment reliability.
- Large-scale ranking progress is constrained by translating product intuition into auditable, executable hypotheses amid conflicting metrics and complex architectures.
- The deployment gap arises when statistically promising offline policies fail to account for operational constraints or feature instability, making them brittle or undeployable.
- GEARS decomposes ranking optimization into programmatic filtering, domain-grounded interpretation, and deterministic governance under an LLM-based controller.
- Specialized Agent Skills externalize ranking expertise into reusable, auditable capabilities covering feature definitions, stability diagnostics, and trade-off explanations.Skills are loaded lazily to bound context length and mitigate context rot in long sessions.
- 0.94 nDCG@1 was achieved on 100 structured policy-selection instructions derived from 20 billion-user-scale online experiments, outperforming prompting baselines and code-execution methods.
- Deployment across nine production surfaces delivered consistent metric improvements, reduced the engineering cycle from weeks to days, and kept annual LLM cost under $1,500.
2 Related Works
Related work spans context-engineered LLM reasoning, uplift modeling, and adaptive experimentation. These approaches provide complementary capabilities, but ranking deployment requires responsiveness to changing behavior and production constraints beyond static offline optimization.
- Context Engineering: Context Engineering treats LLM input as a dynamically structured assembly of informational components, while Tool-Integrated Reasoning turns LLMs into world interactors.
- Uplift Modeling and HTE: Uplift modeling estimates heterogeneous treatment effects using meta-learners, trees, or neural networks to identify user segments responsive to interventions.These methods differ in modeling strategy, interpretability, and ability to capture complex relationships between user features and interventions.
- Uplift Modeling and HTE: Existing uplift approaches often rely on offline features and static user profiles, limiting responsiveness to real-time behavioral shifts and dynamic intervention strategies.
- Adaptive Experimentation: Adaptive Experimentation uses sequential decision-making to balance exploration and exploitation while dynamically searching configurations or policies.Multi-Armed Bandits and Bayesian Optimization can search vast parameter spaces more rapidly than traditional A/B testing cycles.
3 Preliminary
The preliminary framework models personalized ranking as segment-level treatment assignment over multiple metrics. GAS constructs candidate policies through quantile-based segmentation and searches for Pareto-efficient trade-offs using weighted objectives.
- GAS evaluates heterogeneous treatment effects for treatments relative to a control across multiple performance metrics.
- GAS defines user segments through quantile-based splits over ranking features, including individual and binary segmentation strategies.
- These segmentation strategies create a large combinatorial space of candidate policies across cohorts and treatments.
- A policy maps user segments to actions and is represented as a set of segment-action pairs forming a population partition.
- GAS searches weight vectors over multiple metrics, maximizing scalarized rewards to obtain the induced Pareto policy set.
4 The GEARS System
GEARS connects intent-conditioned candidate generation, domain-grounded interpretation, and deterministic lifecycle governance to address the deployment gap in ranking optimization. Its tolerance filtering preserves near-Pareto candidates, while skills and validation hooks support contextual assessment and production readiness.
- The deployment gap is the disconnect between statistically optimal policies and policies feasible for production.
- GEARS decomposes optimization into programmatic candidate filtering, specialized-agent interpretation, and deterministic validation coordinated by an LLM controller.
- Tolerance-based Pareto filtering retains near-Pareto policies within uncertainty margins δm(π) = γ · σm(π), including candidates that may offer better stability or interpretability.
- Programmatic pre-filtering translates intent into executable operations over candidate tables, applying deterministic constraints before language-model reasoning.
- Specialized Agent Skills provide on-demand ranking knowledge for feature understanding, stability auditing, and policy interpretation.
- Deterministic governance hooks check feature stability and statistical significance before candidate policies are promoted.
- In the production trace, programmatic checks reduce 29 candidates to 16, feature auditing finds ϕshift = 8% below a 15% threshold, and governance confirms significant Metric 1 gain with neutral Metric 2.
- The resulting policy retains 76% of the naive baseline’s Metric 1 gain while fully eliminating the guardrail violation.
5 Experiments
GEARS is evaluated on structured policy selection, component ablations, feature stability, and deployment-scale outcomes. The experiments show strong selection reliability, substantial dependence on programmatic filtering, and measurable gains across diverse surfaces.
- Experimental Setup: The benchmark uses 100 structured instructions derived from 20 large-scale online experiments, with deterministic ground-truth rankings for objective evaluation.Instructions cover joint optimization, constrained optimization, trade-off analysis, efficiency optimization, and single-metric optimization.
- Structured Policy Selection: GEARS consistently outperforms five baselines across most ranking and decision-oriented metrics under multi-metric policy constraints.Evaluation covers nDCG@k, precision, global rank correlation, recall, and Top-1 performance.
- Component Analysis: Removing code execution reduces nDCG@1 from 0.94 to 0.40, identifying programmatic filtering as the primary contributor on structured selection tasks.These tasks map directly to tabular operations such as filtering, sorting, and ranking.
- Component Analysis: Removing specialized skills reduces nDCG@1 from 0.94 to 0.87, while skills contribute substantially more when production instructions are less structured and feature context is critical.On structured tasks, the skills mainly improve interpretation of domain-specific selection criteria.
- Stability Governance: The stability benchmark identifies unavoidable baseline drift and filters unstable high-lift features before deployment.Feature set S shifted by 6% over six months, while Feature 4 showed 50% shift under quantile cuts; GEARS automatically disqualified such unstable features.
- Broad Adoption and Real-World Impact: Across diverse experimental surfaces, GEARS delivers measurable metric improvements and reduces the engineering cycle from 1–2 weeks to 1–2 days.GEARS operates offline, with an approximate cost of $3.75 per run and $1,500 annually at 400 experiments per year.
6 Practical Evaluation
GEARS addresses multi-objective recommendation trade-offs by discovering cohort-specific policies and validating them against competing and guardrail metrics. In the case study, its recommendations improve Metric 2 while preserving Metric 1 neutrality and reveal differentiated treatment responses across user groups.
- 6.1 Complex Trade-off Optimization in Large-Scale Recommendation: GEARS replaces manual slice-and-dice analyses with agentic exploration of high-dimensional cohort space and candidate segment proposals.The workflow begins from a natural-language question about the trade-off between Metric 1 and Metric 2.
- 6.1 Complex Trade-off Optimization in Large-Scale Recommendation: Candidate segments are validated against both primary objectives and guardrail metrics before policies are recommended.This constrains the output to cohort-specific policies meeting statistical and neutrality requirements.
- 6.1 Complex Trade-off Optimization in Large-Scale Recommendation: All three top-ranked GEARS policies significantly improve Metric 2 while maintaining neutrality on Metric 1.Table 5 defines neutrality as a 95% confidence interval that includes zero.
- 6.1 Complex Trade-off Optimization in Large-Scale Recommendation: Different user types respond differently to content-mix changes, with active and less active users favoring different treatments.The case study uses these differentiated responses to support targeted rather than uniform policies.
- 6.1 Complex Trade-off Optimization in Large-Scale Recommendation: Deploying the cohort-targeted policy lifts the prioritized metric significantly while keeping the competing metric neutral.GEARS also automates a previously multi-week expert-driven discovery process and explains the decision rationale.