PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoRSE: Evaluating LLM Agents as Recommendation Engineers for Self-Evolving Recommender Systems

Rui Tian, Yi Fang, Yejin Hua, Du Wang, Yuzhe Wang, Minxuan Shan, Zirui Ren, Xinyu Lin, Chongming Gao, Wenjie Wang, Fuli Feng

arXiv:2610.37228v1cs.IR

TL;DR

RSE-Agents lack a unified evaluation framework and systematic analysis of how they achieve recommendation gains. AutoRSE evaluates agents across six industrial-grade tasks and analyzes their trajectories; agents improve performance on five tasks, but idea generation remains a key bottleneck.

  • Problem

    Existing RSE-Agent systems demonstrate feasibility in proprietary pipelines, but provide neither general quantitative comparison nor systematic understanding of how agents gain performance.

    Source: §1 · p. 1

  • Method

    AutoRSE benchmarks frontier LLM agents across six industrial-grade recommendation tasks and analyzes their optimization trajectories and final algorithmic novelty.

    Source: §1 · p. 1, §2.3 · p. 4

  • Results

    Across five of six tasks, RSE-Agents improve strong baselines by an average relative 16.1%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Idea-injection experiments show that research- or expert-sourced ideas outperform self-generated ideas by 13.35%, identifying idea generation as a key bottleneck.

    Source: §1 · p. 2

    Current agents mostly gain through engineering refinements, while genuine algorithmic innovations remain rare.

    Source: §1 · p. 2

Abstract

from arXiv · show

Recommender Systems Engineer Agents (RSE-Agents), which use LLM-based agents to autonomously improve recommendation algorithms, are emerging as a promising paradigm. However, their capabilities and limitations remain poorly understood due to the lack of a unified, standardized evaluation framework and systematic trajectory-level analysis. To address this gap, we introduce AutoRSE, an evaluation framework spanning six representative industrial-grade recommendation tasks, and conduct a large-scale empirical study of three frontier LLMs across three widely used agent harnesses. Our results show that RSE-Agents consistently enhance the performance of backbones on five of the six tasks, with an average relative improvement of 16.1%, indicating that they can already autonomously improve recommendation performance. However, trajectory-level analysis also reveals that current RSE-Agents function primarily as engineering optimizers rather than algorithmic improvers: their gains are largely driven by hyperparameter tuning, training strategies, and model ensembling, while genuine algorithmic innovations are rare. Since the largest gains in recommender systems often stem from algorithmic advances, this behavior underscores a fundamental limitation of current RSE-Agents. To probe this limitation, we perform idea-injection experiments and show that agents equipped with innovative ideas from frontier research or human experts achieve substantially larger improvements than those relying on self-generated ideas. These results suggest that idea generation is a central bottleneck of current RSE-Agents, highlights a key direction for the future recommendation system research. Our implementation is publicly available on GitHub: https://github.com/Ray-Tiann/AutoRSE.

1 Introduction

This paper introduces a unified evaluation framework and trajectory-level analysis to assess how RSE-Agents improve recommendation systems. Agents improve performance across most tasks, but their gains mostly come from engineering refinements; idea-injection experiments identify idea generation as a key bottleneck.

1 Introduction

  • Existing systems lack a general framework for comparing RSE-Agents or a systematic account of how they achieve performance gains.The resulting open questions concern unified, realistic evaluation and current agents’ capability boundaries and bottlenecks.Source: §1 · p. 1
  • AutoRSE evaluates agents on six industrial-grade recommendation tasks using reproduced state-of-the-art baselines, a one-GPU sandbox, and a 12-hour optimization budget.The study tests three frontier LLMs with Claude Code, Codex, and Terminus-2.Source: §1 · p. 1
  • Across five of six tasks, RSE-Agents improve strong baselines by an average relative 16.1%.The study evaluates three frontier LLMs across three widely used agent harnesses.Source: §1 · pp. 1–2
  • Current agents act mainly as engineering optimizers: gains come from tuning, training strategies, and ensembling, while genuine algorithmic innovations remain rare.The paper notes that the largest historical recommender-system gains typically come from algorithmic advances.Source: §1 · p. 2
  • Idea-injected agents outperform those relying on self-generated ideas by 13.35%, identifying idea generation as a key bottleneck.The injected ideas come from frontier research or human experts; implementation and experimental execution are not identified as the bottleneck.Source: §1 · p. 2
  • The paper contributes a unified benchmark, trajectory-level capability analysis, idea-injection experiments, and publicly released code and agent trajectories.A live leaderboard supports open submissions for continued evaluation.Source: §1 · p. 2

2 Method

AutoRSE formulates recommendation-engineering tasks as iterative code-improvement under a computational budget, then evaluates final predictions and analyzes optimization trajectories. Its framework spans six representative tasks and classifies both iteration strategies and final algorithmic novelty.

2.1 RSE Task Formulation

  • Each RSE task instance comprises a task description, baseline codebase, dataset, and computational budget; test labels remain hidden until final evaluation.The description specifies the objective, primary metric, and submission requirements, while the dataset includes training, validation, and unlabeled test inputs.Source: §2.1 · p. 2
  • Agents iteratively explore a baseline, propose and implement changes, run experiments, and use feedback to retain, revise, or discard modifications until the budget expires.The process yields the best-validated strategy and its test predictions.Source: §2.1 · p. 2

2.2 The AutoRSE Evaluation Framework

  • AutoRSE instantiates six representative industrial-grade tasks, covering generative recommendation and click-through-rate prediction with distinct modeling strategies.The suite includes sequential modeling, semantic-ID generation, joint search–recommendation, feature interaction, multi-head token mixing, and unified sequence modeling.Source: §2.2 · p. 2
  • Agents work in a sandboxed one-GPU environment with local coding tools, no internet access, and a fixed 12-hour budget; final predictions are scored against hidden test labels.AutoRSE reports both the final test score and relative improvement over the initial baseline.Source: §2.2 · p. 3, Figure 2 · p. 3

2.3 Trajectory-level Analysis

  • Trajectory analysis segments runs into experiments, identifies their optimization strategies, and assesses the novelty of each final submission.The six strategies range from parameter tuning, ensembling, and training changes to feature modeling, architecture changes, and objective-function changes.Source: §2.3 · pp. 3–4
  • Three LLM judges rate final changes from baseline on a 0–4 ordinal novelty scale, from routine tuning to transformational innovation.Intermediate levels distinguish recombining existing modules, using standard components, and integrating new mechanisms with existing modules.Source: §2.3 · p. 4

3 Experiments

The experiments test whether agents can improve recommendation systems, examine how they optimize, and assess whether idea guidance can further strengthen performance.

3.1.1 Evaluated RSE-Agents.

  • Three frontier LLMs paired with three harnesses yield seven compatible combinations, evaluated in three trials under a 12-hour, single-H200 budget.Source: §3.1.1 · p. 4, §3.1.2 · p. 4

3.2.1 Main Results.

  • Agents beat baselines in 35 of 42 settings, averaging 16.1% relative improvement across six tasks; five of seven agents underperform on IntSR.Source: §3.2.1 · pp. 4–5, Table 2 · p. 5

3.2.2 Optimization Dynamics over Time.

  • Agents typically surpass baselines within 10%–25% of the budget and plateau after 50%, while runs diverge and weaker agents struggle on IntSR.Source: §3.2.2 · p. 5, Figure 3 · p. 6

3.3.1 Distribution of Optimization Strategies.

  • Engineering tactics account for 91.7% of 4,425 experiments—ensembling 59.0% and parameter tuning 26.2%—while algorithmic innovation accounts for 8.3%.Source: §3.3.1 · pp. 5–6, Figure 4 · p. 6

3.3.2 Case Study: Algorithmic Innovation vs. Hyperparameter Tun-

  • Final algorithms average 2.42/4 novelty, and novelty correlates with relative test gains (Pearson r=0.79).Source: Figure 5 · p. 6, §3.3.3 · p. 7
  • On HSTU, feature fusion yields validation NDCG@10 0.07216 (+6.6% vs. baseline), above tuning and warmup's 0.06997 (+3.4%).Source: §3.3.2 · p. 6

3.4.1 Experimental Setup.

  • The ideation comparison tests standard optimization, GenSelect self-generation, and an external idea library distilled from research papers and competition solutions.Source: Figure 6 · p. 7, §3.4.1 · p. 7

3.4.2 Results and Analysis.

  • Injected ideas beat Standard and Self-generated across six tasks, averaging score gains of 14.23% and 13.35%, while self-generation degrades IntSR (−8.08%).Source: §3.4.2 · p. 7

4 Related Work

Prior recommendation automation and LLM-agent systems motivate a shared benchmark and trajectory analysis for evaluating recommender engineering agents. AutoRSE situates that evaluation within broader long-horizon engineering-agent research.

4.1 Recommender System Automation

  • Earlier recommender automation optimized configurations or components within predefined search spaces, leaving research hypotheses and broader experimental workflows largely to human engineers.Examples include model and hyperparameter search, feature-interaction selection, and interaction-architecture search.Source: §4.1 · p. 7
  • AgentX and Self-Evolving Recommendation System demonstrate iterative recommendation research, but their proprietary pipelines leave cross-agent comparisons and strategy attribution open.Their workflows include proposal generation, implementation, experimentation, and, in one system, validation against delayed business metrics.Source: §4.1 · p. 8
  • AutoRSE addresses these gaps with six tasks, standardized environments, budgets, and hidden-test evaluation, plus trajectory analysis of agent changes and outcomes.Its idea-injection experiment and selection module support exploring substantive algorithmic improvements before costly implementation and training.Source: §4.1 · p. 8

4.2 LLM Agents for Engineering Tasks

  • LLM-agent research spans software maintenance, iterative machine-learning experimentation, and autonomous scientific workflows involving literature, experiments, and reporting.Examples include SWE-bench, MLAgentBench, The AI Scientist, and AI-Researcher.Source: §4.2 · p. 8
  • The RSE setting tests sustained optimization of strong recommendation baselines under fixed computation, requiring agents to form hypotheses, implement changes, and evaluate them over long horizons.Costly training and noisy validation make unsuccessful directions expensive; trajectory analysis examines whether changes go beyond routine tuning and engineering heuristics.Source: §4.2 · p. 8

5 Conclusion

AutoRSE evaluates recommender systems engineering agents and finds that they improve recommendation performance while rarely developing substantive algorithmic innovations. Idea-injection experiments point to idea generation as a key bottleneck, while the appendix specifies the trial and assessment procedures.

5 Conclusion

  • AutoRSE experiments find that agents improve recommendation performance but mostly use ensembling and parameter tuning; injected frontier-research and expert ideas yield larger gains than self-generated ideas.The findings identify idea generation as a key bottleneck and motivate stronger ideation capabilities.Source: §5 · p. 8

A.1 Task Launch Prompt

  • The task-launch prompt fixes the baseline, data and evaluation protocols, public-validation selection, hidden-test reward, submission requirements, and restrictions on private labels and test inputs.Runs also use assigned GPUs and a fixed seed, and must preserve a valid best-so-far submission through the research and finalization deadlines.Source: §A.1 · p. 10

A.2 Execution Control Prompts

  • Execution-control prompts resume research with evidence-driven experiments and validation comparisons, then request finalization when the research phase ends.Agents must continue useful work rather than merely summarize while time remains, and preserve the best submission.Source: §A.2 · p. 10

A.3 Idea Generation and Selection

  • Idea-generation rounds require eight fresh ideas, with two or three selections frozen before implementation; each selected experiment is executed and recorded separately.Ideas draw only on task materials and trial memory, and experiments use the original seed, model, resources, and public-validation contract.Source: §A.3 · p. 10

B.1 Dataset Details

The appendix describes the benchmark’s varied data domains and formats and documents how baselines were reproduced and checked. Officially released baselines were matched or closely aligned with their published results.

B.1 Dataset Details

  • The six benchmark datasets span recommendation and click-log settings, including chronological Amazon Beauty splits, 100-candidate Kindle splits, and large-scale KuaiSAR v2, Criteo-5M, and MovieLens Latest data.Reported examples include 22,363 users and 12,101 items for Amazon Beauty, 1.46M KuaiSAR v2 test samples, and 4M/500K/500K Criteo-5M train/validation/test splits.Source: §B.1 · p. 11

B.2 Baseline Reproduction and Correctness Verification

  • Baseline implementations were validated against original literature, official code, or trusted repositories; reproduced WuKong and HSTU scores matched or closely aligned with published results.The appendix also reports checks for convergence, metric trajectories, competitive performance, and deterministic evaluation scripts.Source: §B.2 · p. 11, §B.2.1 · p. 11

B.2.2 Community-Guided Reproduction and Dataset Migration.

For baselines without official releases, the appendix describes community-guided implementations and an independent IntSR reproduction, followed by correctness and evaluation checks.

B.2.2 Community-Guided Reproduction and Dataset Migration.

  • For TIGER, OneTrans, and RankMixer, executable baselines were built from reputable open-source projects and migrated to academic datasets while preserving core algorithmic mechanics.Examples include migration to KuaiSAR v2 and Criteo-5M.Source: §B.2.2 · p. 11

B.2.3 Best-Effort Independent Reproduction for Unreleased Base-

  • With no public official or community implementation for IntSR, the authors independently reproduced it from the paper’s architecture and joint search-and-recommendation objectives, adapting UniSAR to Amazon Kindle.Source: §B.2.3 · p. 11

B.2.4 Correctness and Evaluation Protocol Verification.

  • Migrated and independently reproduced baselines were checked for stable convergence, valid metric trajectories, and competitive performance; evaluation scripts were deterministic.Examples of tracked metrics include NDCG@10, AUC, and LogLoss.Source: §B.2.4 · p. 11

C Additional Experimental Results and Details

The section reports how categorized modification actions are distributed across model–harness combinations.

C.1 Overall Distribution of Agent’s Change

  • Table 4 pools categorized modification actions across tasks and trials within each model–harness combination, so counts reflect recorded research activity rather than runs.Percentages use the categorized actions as their denominator.Source: §C.1 · p. 11

C.2 Segment Split Details

The section describes reconstructing structured action records from agent trajectories and defines the scope of the improvement-strategy distribution in Table 4.

C.2 Segment Split Details

  • Table 4 reports improvement-strategy percentages over categorized modification actions pooled across tasks and trials for each model–harness combination.N denotes categorized actions; percentages may not total 100% because of rounding.Source: Table 4 · p. 12
  • Action reconstruction links tool-call anchors to intervening code edits and gathers outputs from the tool call and subsequent interaction window to extract unvalidated metric candidates.The process restores the context of each trial.Source: §C.2 · pp. 11–12

C.3 Complete results for all six tasks

Figure 7 presents optimization trajectories across the six recommendation tasks.

C.3 Complete results for all six tasks

  • Figure 7 shows optimization trajectories across all six recommendation tasks.Source: Figure 7 · p. 12

C.4 Case Studies

The case studies describe concrete implementations of agent guidance and report validation comparisons for content fusion and an added residual stage.

C.4 Case Studies

  • The HSTU implementation standardizes 768-dimensional item content embeddings, projects them without bias, and adds them to learned item-ID embeddings shared by sequence lookup and full-catalog scoring.Content-fused models are retained in the final ensemble.Source: §C.4 · p. 12
  • Validation NDCG@10 is 0.078877 with content fusion versus 0.068080 without it, but the runs use early-stopping patience of 20 and 25, respectively.Source: §C.4 · p. 12
  • Adding a residual feedforward stage changes NDCG@10 from 0.080149 to 0.078877, with a paired-user bootstrap 95% confidence interval of [−0.002898, 0.000220].The passage reports no clear incremental benefit from the added component in this configuration.Source: §C.4 · p. 12
  • IntSR training expands four negatives per example toward 128 candidates using in-batch positives while retaining cross-entropy and masking target duplicates and observed-history negatives.Full batches of 128 need no additional popularity sampling; the isolated contribution to final performance is not measured.Source: §C.4 · p. 12

Related papers

Loading 2610.37228v1…