PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-EvolveRec: Self-Evolving Recommender Systems with LLM-based Directional Feedback

Sein Kim, Sangwu Park, Hongseok Kang, Wonjoong Kim, Jimin Seo, Yeonjun In, Kanghoon Yoon, Hyunsik Jeon, Chanyoung Park

arXiv:2602.12612v2cs.IRcs.AI

TL;DR

Existing NAS methods use fixed human-defined search spaces, while LLM-driven code evolution uses open-ended programs but mainly relies on scalar metrics that provide limited diagnostic guidance. Self-EvolveRec combines qualitative User Simulator critiques, quantitative Model Diagnosis Tool verification, and co-evolving diagnostics to guide recommender code evolution. Experiments report superiority over NAS and LLM-driven baselines in recommendation performance and user satisfaction.

  • Problem

    Fixed NAS spaces constrain innovation to predefined operators, while scalar-metric LLM code evolution lacks qualitative explanations and directional guidance for recommender failures.

  • Method

    Self-EvolveRec combines a User Simulator, a Model Diagnosis Tool, and Diagnosis Tool–Model Co-Evolution in a directional feedback loop for evolving recommendation code.

  • Results

    Self-EvolveRec significantly outperforms NAS and LLM-driven evolutionary baselines in recommendation performance and user satisfaction.

  • Takeaways & Limitations

    Directional qualitative and quantitative feedback supports more informed evolution of recommender pipelines as their architectures change.

  • Takeaways & Limitations

    The framework employs a fixed user simulator for efficiency while separately investigating co-evolution of the simulator with the codebase.

Abstract

from arXiv · show

Traditional methods for automating recommender system design, such as Neural Architecture Search (NAS), are often constrained by a fixed search space defined by human priors, limiting innovation to pre-defined operators. While recent LLM-driven code evolution frameworks shift fixed search space target to open-ended program spaces, they primarily rely on scalar metrics (e.g., NDCG, Hit Ratio) that fail to provide qualitative insights into model failures or directional guidance for improvement. To address this, we propose Self-EvolveRec, a novel framework that establishes a directional feedback loop by integrating a User Simulator for qualitative critiques and a Model Diagnosis Tool for quantitative internal verification. Furthermore, we introduce a Diagnosis Tool - Model Co-Evolution strategy to ensure that evaluation criteria dynamically adapt as the recommendation architecture evolves. Extensive experiments demonstrate that Self-EvolveRec significantly outperforms state-of-the-art NAS and LLM-driven code evolution baselines in both recommendation performance and user satisfaction. Our code is available at https://github.com/Sein-Kim/self_evolverec.

1 Introduction

Self-EvolveRec addresses the limits of fixed NAS search spaces and scalar-metric code evolution by using directional feedback to diagnose and improve recommendation pipelines. It combines qualitative user critiques, quantitative model verification, and co-evolving diagnostics, and reports gains over NAS and LLM-driven baselines.

  • Motivation: Fixed NAS search spaces limit optimization to human-defined operators and struggle to jointly improve non-architectural pipeline components.These components include loss functions and data processing logic.
  • Motivation: LLM-driven code evolution enables open-ended program optimization beyond predefined design spaces, but existing methods primarily rely on scalar metrics.Such methods have been applied from isolated functions to entire codebases.
  • Motivation: Scalar metrics cannot explain multifaceted recommendation failures, such as popularity bias, insufficient diversity, or missing short-term interests.Directional feedback is therefore needed to diagnose failure causes and guide improvement.
  • Contribution: Self-EvolveRec integrates a User Simulator for qualitative critiques with a Model Diagnosis Tool for quantitative verification of structural deficiencies.The combination can connect user-perceived failures with internal model properties such as embedding geometry.
  • Contribution: Diagnosis Tool–Model Co-Evolution updates diagnostics as the recommendation pipeline changes and supports metrics for newly emerging user critiques.The framework reports improvements over NAS and LLM-driven baselines in recommendation performance, user satisfaction, and algorithmic logic quality.

2 Problem Definition

The paper formulates recommender code evolution over an open-ended program space, starting from a functional seed pipeline and seeking an improved codebase through iterative optimization. The codebase includes architecture, data processing, loss computation, and parameter updates.

  • Dataset: The dataset comprises users, items, interactions, timestamps, ratings, reviews, and item side information.User histories are chronologically ordered by interaction time.
  • Optimization Goal: A codebase B represents the entire recommendation pipeline within an open-ended program space.The seed codebase includes a recommender architecture, data loaders, and an optimization loop.
  • Optimization Goal: The seed pipeline can include an architecture such as NCF, processing for interactions and attributes, loss computation, and parameter updates.These elements define the starting point for subsequent code evolution.
  • Optimization Goal: The objective is to evolve the seed codebase into an optimal codebase maximizing a standard recommendation metric within T iterations.The paper formulates this task as a bi-level optimization problem involving code-defined loss minimization and evaluation.

3 Proposed Framework: Self-EvolveRec

Self-EvolveRec uses directional feedback and an iterative evolutionary pipeline to refine recommender code. User critiques identify behavioral problems, diagnosis probes verify internal deficiencies, and co-evolving diagnostics remain compatible with changing codebases.

  • Framework Overview: Self-EvolveRec combines Directional Feedback Generation, an Evolution Pipeline, and Diagnosis Tool–Model Co-Evolution.The framework integrates qualitative and quantitative guidance before directed code modifications.
  • User Simulator: The User Simulator evaluates recommendation lists through diverse virtual-user personas and produces natural-language critiques of specific deficiencies.Personas combine interaction histories with Activity, Conformity, and Diversity traits.
  • Model Diagnosis Tool: The diagnosis tool directly accesses model parameters and data loaders to probe structural or behavioral deficiencies.This complements user feedback by detecting issues invisible to perception, such as embedding collapse.
  • Model Diagnosis Tool: Embedding-collapse diagnosis uses mean pairwise cosine similarity among sampled item embeddings, where high similarity indicates representation degeneration.The probe targets representations that lose discriminative power by collapsing into a narrow subspace.
  • Evolution Pipeline: The evolutionary workflow evaluates a parent codebase, generates user and diagnosis reports, plans targeted retrieval, and applies directed refinement.An archive stores codebases and feedback to support population-based exploration and cached evaluation results.
  • Diagnosis Tool–Model Co-Evolution: Diagnosis Tool–Model Co-Evolution adapts diagnostics to new components and translates qualitative critiques into measurable metrics.For example, a diversity critique can lead to a tailored top-k diversity metric.

4 Experiments

Experiments evaluate Self-EvolveRec across datasets, baselines, user-satisfaction measures, ablations, initialization settings, and simulator reliability. The results show consistent recommendation gains, improved codebase quality from directional feedback, adaptability across starting points, and benefits from co-evolving diagnostic tools.

  • Evaluation setup: The evaluation uses four datasets, leave-last-out testing, and NDCG@5 and HR@5 measured on real held-out interactions.The ranking metrics do not involve an LLM-based component, providing simulator-independent evidence of recommendation improvements.
  • Overall performance: Self-EvolveRec consistently outperforms AlphaEvolve and DeepEvolve across all datasets and seed models.It also significantly surpasses NAS-based baselines including AutoFIS and NASRec.
  • User satisfaction: Self-EvolveRec’s user-satisfaction analysis uses Agent4Rec and PUB with disjoint evaluation users and distinct persona schemes from the optimization-time simulator.This setup is designed to reduce concerns about a simulator-to-simulator evaluation loop.
  • Ablation studies: Introducing either SIM or DIAG improves performance over scalar-metric-only evolution, with SIM producing the larger increase across datasets.The complete framework is represented by Row (6) in the ablation study.
  • Extreme initialization: Starting from a random recommender, Self-EvolveRec reaches peak performance at iteration 8 on CDs and iteration 11 on Electronics.The extreme-initialization experiments also assess an optimized Ensemble starting point.
  • Planning and retrieval: Removing feedback-aware planning preserves comparable recommendation performance but substantially reduces Creativity and Explicitness.The ablation attributes recommendation performance primarily to identifying failure modes through directional feedback.

5 Conclusion

Self-EvolveRec automates recommender-system design with directional feedback from qualitative user critiques and quantitative diagnosis, adapting its criteria as pipelines evolve. The paper reports gains in recommendation accuracy and multifaceted user satisfaction, with robust adaptability across initialization settings.

  • 5 Conclusion: Self-EvolveRec combines a User Simulator’s qualitative critiques with a Model Diagnosis Tool’s quantitative verification in an LLM-driven code-evolution framework.Its Diagnosis Tool–Model Co-Evolution strategy dynamically adapts diagnostic criteria to structural changes in the codebase.
  • 5 Conclusion: Self-EvolveRec significantly outperforms state-of-the-art NAS and LLM-driven evolutionary baselines in recommendation accuracy and multifaceted user satisfaction.The conclusion also reports robust adaptability from scratch scenarios to complex ensembles.
  • 5 Conclusion: The framework evaluates simulated users through Activity, Conformity, and Diversity traits, each divided into LOW, MID, and HIGH levels.These traits represent engagement level, mainstream adherence, and interest breadth, respectively.

D Implementation Details

The implementation fixes evolution and training budgets across methods while using GPT-5-mini for simulation and a shared recommender configuration for fair comparison.

  • D Implementation Details: All LLM-driven evolutionary frameworks use a maximum of 21 evolution iterations.NAS baselines use five search epochs for AutoFIS and one for NASRec.
  • D Implementation Details: The User Simulator uses GPT-5-mini with 20 sampled users, processed under the reported experimental configuration.The experiments run on a single NVIDIA GeForce A6000 GPU with 48GB memory.
  • D Implementation Details: Recommender models share embedding dimension 50, batch size 128, learning rate 0.001, and a maximum of 300 training epochs.These settings also apply to the NAS retraining phase.
  • D Implementation Details: Because AlphaEvolve’s official code was unavailable, the study uses OpenEvolve as its baseline implementation, following prior work.OpenEvolve is identified as an open-source implementation.

E Datasets

The dataset section reports post-preprocessing dataset statistics and organizes comparisons across NAS baselines, LLM-driven code-evolution baselines, and seed recommender architectures.

  • E Datasets: Table 7 presents the statistics of the datasets after preprocessing.The supplied passage provides the table description but not the individual dataset values.
  • E Datasets: NAS baselines include AutoFIS, NASRec, AutoLossGen, and DNS-Rec, covering feature interactions, full architectures, loss functions, and compact sequential models.Their stated search targets span interactions, architectures, losses, and data-aware or resource-aware sequential designs.
  • E Datasets: LLM-driven code-evolution baselines include AlphaEvolve and DeepEvolve.AlphaEvolve evolves algorithmic codebases, while DeepEvolve adds retrieval-augmented generation with external scientific knowledge.
  • E Datasets: The seed recommender models are NCF, NGCF, SASRec, and MoRec, representing neural collaborative filtering, graph-based, sequential, and multimodal approaches.MoRec uses pretrained encoders such as SBERT for textual item features and employs SASRec as its backbone.

G Detailed Implementation of the Efficient Variant

The efficient variant replaces post-training diagnosis with zero-cost representation and gradient signals, adapts simulation through code-to-intent summaries, and is evaluated for LLM robustness and execution cost.

  • G Detailed Implementation of the Efficient Variant: The efficient variant defines zero-cost signals from user, positive-item, and interaction representations obtained in one mini-batch forward pass.It uses effective ranks, top-1 singular-value share, and mean parameter gradient norm.
  • G Detailed Implementation of the Efficient Variant: The proxy Mzero combines effective ranks, top-1 singular-value share, and gradient magnitude as a surrogate for standard recommendation evaluation.The supplied formulation states Mzero = i_e + u_e − s_v − | log g + 3|.
  • G Detailed Implementation of the Efficient Variant: DIAG(0) uses effective ranks as representation-diversity proxies, s_v for rank-collapse detection, and g for trainability assessment.This avoids standard post-training probes that require fully trained model parameters.
  • G Detailed Implementation of the Efficient Variant: The efficient SIM summarizes the current codebase into architectural intent and combines that intent with zero-cost signals for persona-based assessment.The original SIM instead requires explicit recommendation lists from a fully trained model.
  • G Detailed Implementation of the Efficient Variant: With Claude Opus 4.6, Self-EvolveRec consistently outperforms AlphaEvolve and DeepEvolve, with negligible performance difference from its GPT-based variant.The authors interpret this as evidence that the diagnostic evolutionary paradigm is robust across LLM backbones.

H.2 Efficiency Analysis

Self-EvolveRec spends more time per evolution iteration but reaches peak performance in fewer iterations, improving overall time to peak in several settings.

  • Approximately 19 minutes per iteration exceeds AlphaEvolve’s 6 minutes and DeepEvolve’s 14 minutes.
  • Self-EvolveRec reaches peak performance in around 8 to 11 iterations, compared with 13 to 19 iterations for baselines or failure to surpass the initial model.
  • 3h 27m versus DeepEvolve’s 4h 19m is the reported peak-performance time for the random recommender on Electronics.

H.2.1 Time Efficiency Analysis.

Efficiency analyses examine iteration cost, user-sample size, literature size, token budgets, zero-cost proxies, and persona schemes, showing robust or improved performance under several practical variations.

  • User Sample Size: Performance improves substantially from 1 to 3 sampled users and stabilizes around 5 users.
  • Literature Size: Self-EvolveRec maintains robust performance across different literature sizes K used in Planning.
  • Fixed Token Budget: Self-EvolveRec consistently outperforms AlphaEvolve and DeepEvolve under a fixed 5M-token evolutionary budget.
  • Zero-Cost Proxy: The zero-cost proxy variant achieves comparable recommendation performance with marginal degradation while reducing overhead by up to 12×.On MoRec/CDs, iteration time decreases from 12m 08s to 1m 02s.

H.3 Robustness to Persona Scheme

Self-EvolveRec remains effective when the User Simulator’s default persona traits are replaced with Big Five personality traits.

  • Self-EvolveRec achieves comparable recommendation performance under both default and Big Five persona schemes.
  • The default scheme uses Activity, Conformity, and Diversity traits.
  • The alternative scheme uses Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism.

H.4 Hallucinations and Groundedness of the User Simulator

The User Simulator’s feedback is largely grounded in observed user behavior, but its critiques can over-generalize semantic patterns and require DIAG for technical calibration.

  • Hallucination Analysis: No analyzed simulation feedback contradicted actual user histories or recommendation lists.
  • Interest Groundedness: IG_Precision measures groundedness of inferred interests, whereas IG_Recall measures coverage of verifiable interests.
  • Interest Groundedness: High IG_Precision on CDs and Electronics indicates that nearly all inferred interests are grounded in explicit positive evidence.
  • Interest Groundedness: IG_Recall covers a substantial portion of interests verifiable from users’ histories.
  • Complementary Interaction of SIM and DIAG: The simulator may label repeated items as severe redundancy even when DIAG finds near-zero duplicate exposure and geometrically distinct items.
  • Complementary Interaction of SIM and DIAG: DIAG bridges user-centric simulator perceptions with the model’s actual architectural progress.

H.5.2 Additional Case Study: Evolutionary Trajectory.

The case study tracks Self-EvolveRec from a Random Recommender through successive code changes, linking identified failures to algorithmic modifications and performance progress. Compared with baseline evolution, directional feedback supports a more robust trajectory.

  • Case Study Setup: The trajectory begins with a Random Recommender and compares evolutionary paths with performance across iterations.Figure 9 presents causal alignment between directional feedback and evolved code changes alongside iteration-level performance.
  • Evolutionary Trajectory: Self-EvolveRec links algorithmic improvements to identified failures during code evolution.The study reports a structured path in which RDIAG flagged deficiencies before subsequent modifications.
  • Baseline Comparisons: AlphaEvolve’s “Popularity Bias and Global Bias” update degraded HR from 0.2405 to 0.1969 before being removed.The later addition of “User Bias” illustrates the trial-and-error adjustments required without diagnostic guidance.
  • Baseline Comparisons: DeepEvolve improved with a “Two-Stage Pipeline” at iteration 6 but failed to improve at iteration 9 after an incompatible curriculum-learning strategy for LM negatives.This case demonstrates that a successful intermediate change did not ensure continued progress.
  • Overall Findings: Baselines showed unstable or delayed progress, whereas Self-EvolveRec maintained a robust evolutionary trajectory enabled by directional feedback.The comparison attributes baseline instability to reliance on scalar metrics without diagnostic guidance.
Loading 2602.12612v2…