PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Successor Accuracy: State Retention for Recursive Self-Improvement in Recommendation

Jinfeng Xu, Zheyu Chen, Ziyue Peng, Zheng Lin, Wenhao Yuan, Jian Chen, Shujie Li, Edith Ngai

arXiv:2610.07105v1cs.IRcs.AI

TL;DR

Successor-only evaluation in Rec-RSI may miss useful ranking behavior retained across an update. The paper measures cross-generation complementarity and uses label-free rank separation to choose a pair family; preferences vary by architecture, and selected families beat direct successors in 34/36 held-out trajectories.

  • Problem

    Successor-only evaluation can obscure which relevant items support a score and whether a successor preserves useful predecessor behavior, leaving open what should persist across an update.

    Source: §1 · p. 1

  • Method

    The paper measures cross-generation advantage with a matched contrast between cross- and within-generation pairs and uses rank separation to select a family without outcome labels.

    Source: §1 · p. 1

  • Results

    Cross-generation pairing favors GRU4Rec and SASRec, while FMLP shifts from within-generation preference to cross-generation after a second update; selected families beat direct successors in 34/36 held-out trajectories.

    Source: §1 · p. 1, §4.2 · p. 3, §4.4 · p. 4

  • Takeaways & Limitations

    Rec-RSI progress may reside in relations between generations as well as in the latest model, making relational state retention a distinct design problem.

    Source: §6 · p. 4

    Rank separation diagnoses how an encoder responds to the current update but is not a reliable selector among seeds of the same encoder.

    Source: §4.3 · p. 4

Abstract

from arXiv · show

Recommendation recursive self-improvement (Rec-RSI) feeds recommender outputs into subsequent training. Evaluating each round solely through its latest model assumes that the successor consolidates the update, although pre- and post-update models may retain complementary ranking decisions. We term this \emph{distributed progress} and quantify it using cross-generation advantage (CGA), a marginally matched contrast between cross- and within-generation model pairs. A rank-separation statistic, label-free at selection time, predicts which family to retain. Across four datasets and three sequential recommendation encoders, the preferred retention regime varies by architecture: cross-generation pairing benefits GRU4Rec and SASRec, whereas FMLP initially favors within-generation pairing and shifts toward cross-generation pairing after a second update. Rank separation selects the stronger family in 12/12 first-update and 5/6 second-update dataset-encoder settings; on held-out tests, the selected family outperforms the direct successor in 34/36 trajectories. Five transfer mechanisms do not consistently reproduce these gains in one model. These findings establish state retention as a distinct Rec-RSI problem: progress may reside in relations between generations as well as in the latest model. Code is available at \href{https://github.com/Jinfeng-Xu/RecRSI}{https://github.com/Jinfeng-Xu/RecRSI}.

1 Introduction

In Rec-RSI, successor-only evaluation can miss distinct ranking behavior and complementary behavior retained across an update. The paper frames progress as relational, measures it with CGA, and proposes rank separation to select which pair family to retain.

1 Introduction

  • Successor-only evaluation attributes progress to the latest model when Recall or NDCG rises, although similar utility can reflect different relevant items and successors may not preserve useful predecessor behavior.The retained state governs the next round, raising the question of what should persist across an update.Source: §1 · p. 1
  • Distributed progress describes stage-structured complementarity across an update, where cross-generation pairs may yield greater utility than pairs within either generation.The joint behavior may not reside in any individual model.Source: §1 · p. 1
  • CGA contrasts cross- and within-generation pairs under matched marginal exposure and equal inference cost, while rank separation selects a family using earlier-window top-k overlap.Selection requires no outcome labels.Source: §1 · p. 1
  • 34/36 held-out trajectories favor the selected pair family over the direct successor, motivating retention beyond the latest model.Rank separation selects the family without outcome labels at selection time.Source: §1 · p. 1

2 Relational State across Updates

The paper tests whether a single successor consolidates gains from Rec-RSI and defines relational measures for progress across updates. It also introduces a label-free ranking statistic for selecting which pair family to retain.

2.1 Single-Model Consolidation

  • A transfer operator consolidates each source configuration into one successor, and the difference between source and successor gains measures gain left outside that model.The study instantiates the operator with five consolidation routes.Source: §2.1 · p. 2

2.2 Cross-Generation Advantage

  • CGA compares cross-generation pairs with marginally matched within-generation pairs, so an additive decomposition cancels main effects and leaves the interaction contrast.Positive CGA means cross-generation pairing yields higher utility; the measure covers the update as a whole, including genealogical and non-genealogical pairs.Source: §2.2 · p. 2

2.3 Rank Separation for State Selection

  • Rank separation compares top-k overlap across update stages with overlap within a generation to diagnose whether the update separates rankings more than seed variation.Positive Sep20 indicates greater disagreement near the recommendation boundary across adjacent generations, and it can be computed without relevance labels.Source: §2.3 · p. 2

3 Evaluation Protocol

The evaluation covers four recommendation datasets and three sequence encoders, with protocols for first and second updates and comparisons against several single-model consolidation routes.

Datasets and encoders

  • The evaluation uses Amazon Toys, Amazon Beauty, Yelp, and Amazon Sports with GRU4Rec, SASRec, and FMLP, and three independently seeded trajectories per dataset–encoder setting.The cohorts contain 9,011, 10,980, 17,168, and 17,297 users, respectively.Source: §3 · p. 2

First-update protocol

  • In the first-update protocol, pre-update P and B and post-update A and A′ are trained for 10,000 updates, with A and A′ sharing generated data but using different seeds.Only the selection window chooses checkpoints; Sports is evaluated prospectively, and original validation and test splits are reserved for final evaluation.Source: §3 · p. 2

Held-out evaluation

  • Table 1 reports original held-out-test three-trajectory means, with W and C denoting within- and cross-generation families and CGA reported as NDCG@20 (×10−3) plus positive trajectories.The caption says bolding follows unrounded values.Source: Table 1 · p. 3

Second-update extension

  • The second-update extension trains C and C′ on the fit prefix plus D1 and D2, creating 18 trajectories across GRU4Rec and FMLP.A generates D2; the two new seeds use the same 10,000-update budget.Source: §3 · p. 3

Consolidation baselines and training

  • The consolidation study compares five routes: hard generated-data retraining, full-catalog KL distillation, top-50 listwise distillation, complementary item targets, and linear parameter interpolation.Each route has a matched comparison baseline, and effect intervals use 20,000 dataset-cluster bootstrap replicates.Source: §3 · p. 3

4 Empirical Results

Across four datasets and three encoders, pair-family preferences vary by architecture and update round, while rank separation predicts retention choices. On held-out tests, selected families outperform the direct successor in 34/36 trajectories, and single-model consolidation remains incomplete.

4.1 Distributed Progress after Consolidation

  • The direct post-update model A exceeds parent P in 26/36 held-out trajectories, but this does not rule out progress distributed across models.Figure 2 compares ΔNDCG@20 against route-specific comparators across domains.Source: §4.1 · p. 3, Figure 2 · p. 3
  • Five consolidation routes recover different portions of progress, with listwise transfer reducing mean source-to-successor loss from 2.313 × 10^-3 to 1.375 × 10^-3 NDCG@20.Generated-data retraining changes sign across domains; full-catalog, listwise, and complementary-target transfer beat controls in 1/4, 2/4, and 1/4 settings, while interpolation never beats its stronger endpoint in ten trajectories.Source: §4.1 · p. 3

4.2 Encoder-Dependent Pair Utility

  • Cross-generation pairing improves GRU4Rec and SASRec across datasets at both cutoffs and metrics, while FMLP favors within-generation pairing at k=20.FMLP’s k=10 differences remain near zero, and its NDCG@20 preference agrees with the confirmation window in all 12 settings.Source: §4.2 · p. 3

4.3 Rank Separation as a Retention Signal

  • Rank separation matches the CGA sign in all 12 first-update settings; the indicator selects the stronger family in 12/12 settings and succeeds in 32/36 trajectories.Across the 12 settings, Spearman ρ=0.916; the selected family beats the snapshot control in 35/36 trajectories by 2.065 × 10^-3 NDCG@20.Source: §4.3 · p. 3, §4.4 · p. 4
  • Top-20 separation correlates with CGA at 0.850, versus 0.500 for Jensen–Shannon and 0.270 for total-variation full-catalog separation.The predictive signal is near the ranking boundary, and its association is primarily between architectures rather than seeds within an architecture.Source: §4.3 · p. 3

4.4 Generalization across Datasets and Update Rounds

  • The reported average NDCG@20 gain is 10.6% relative, with a dataset-cluster interval of [1.380, 2.613] × 10^-3.Retaining the pair adds one stored model and one forward pass.Source: §4.4 · p. 4
  • After the second update, GRU4Rec retains positive CGA, while FMLP’s pooled CGA shifts from −0.187 × 10^-3 to +0.035 × 10^-3.Rank separation selects 5/6 setting-level regimes and beats the G2 snapshot control in 17/18 trajectories; realized CGA remains setting-dependent in 5/6 settings.Source: §4.4 · p. 4

5 Related Work

Related work approaches recursive improvement, consolidation, and diversity from several angles. This paper distinguishes update-boundary complementarity through a marginally matched comparison of cross- and within-generation pairs.

5 Related Work

  • RSIR generates interactions and retrains successors under fidelity control [14], while self-evolving systems revise objectives, code, and pipelines [6] [12].Source: §5 · p. 4
  • RISE uses policy distillation for recursive improvement [9]; recommender distillation transfers scores or ranks [4] [10], and EMKD combines sequential encoders [2].Hydra and FLAME preserve ensemble diversity in one network [11], whereas CGA diagnoses complementarity aligned with an update boundary under matched marginals.Source: §5 · p. 4
  • Continual recommendation balances stability and plasticity [13], ReLoop and meta-learning carry errors or experience forward [1] [15], and snapshot ensembles retain an optimization path [8].CGA instead targets update-boundary complementarity through marginally matched cross- and within-generation pairs.Source: §5 · p. 4

6 Conclusion

Rec-RSI progress may reside across generations rather than in the latest model alone, making relational state retention a design problem for self-improving recommenders.

6 Conclusion

  • CGA exposes progress distributed across an update rather than consolidated in the latest model.The conclusion frames this as relational state retention.Source: §6 · p. 4
  • Rank separation identifies the stronger pair family at selection time.Source: §6 · p. 4
  • Retention regimes vary across architectures and rounds, alongside incomplete single-model consolidation.Source: §6 · p. 4

Related papers

Loading 2610.07105v1…