PaperlayerView exact revision on arXiv

Source-linked AI summary

How Can Recommendation Feedback Evolve Agent Memory?

Shanwen Mao, Mingming Li, Hao Zhang, Zhiheng Li, Yige Wang, Penghua Yu, Junxiong Zhu

arXiv:2609.37544v2cs.AI

TL;DR

Recommendation outcomes can guide agent memory, but delayed, context-dependent feedback makes it hard to assess which experiences contributed. TIDE evolves a capacity-constrained external memory through credit assignment, and the paper reports offline and online gains alongside generalization to delayed-label tasks.

  • Problem

    Recommendation feedback is delayed, depends on delivery context, and does not directly identify which content decisions or memories produced an outcome.

    Source: §1 · p. 2

  • Method

    TIDE uses temporal, semantic, and responsibility credit to evolve a capacity-constrained external memory while keeping the generator parameters frozen.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    TIDE reports strongest overall performance among evaluated memory methods, +7.75 offline MEG, and online relative uplifts of 5.10% in UCTR and 4.79% in activation rate.

    Source: §4.1 · p. 8, §5 · p. 9

  • Takeaways & Limitations

    The reported results support TIDE as an approach to continual agent adaptation through feedback-driven, traceable memory evolution.

    Source: §5 · p. 9

    The studied setting keeps generator parameters frozen while continually updating external memory.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Content-generation agents continuously receive impressions, clicks, conversions, and negative feedback from recommendation systems, providing real-world outcome signals for memory evolution. However, these signals are delayed and noisy, confounded by audience composition, placement, and recommendation policies, and may result from the combined influence of multiple memories, making accurate attribution difficult. Existing methods rely primarily on immediate feedback or semantic retrieval and therefore struggle to reliably translate recommendation outcomes into memory fitness. To address this challenge, we propose TIDE (Trajectory-Informed Directed Memory Evolution), an external memory evolution framework driven by delayed recommendation feedback. We further introduce Memory Evolution Gain (MEG), which measures the utility improvement of evolved memory over a no memory baseline on strictly future tasks. TIDE treats memory as a capacity-constrained population of experiences: temporal and semantic credit assignment estimates contextual fitness, while responsibility credit distributes outcome signals according to the memories referenced during generation. These signals are then used to reinforce, crossover, mutate, or evict memories. On an e-commerce membership marketing content-generation agent, TIDE achieves a +7.75-percentage-point MEG in offline temporal replay and significantly improves both unique click-through rate (UCTR) and activation rate in an online A/B test. On a delayed-label benchmark, TIDE achieves the lowest mean absolute error (MAE) and root mean squared error (RMSE) and the highest MEG among the compared methods, demonstrating its effectiveness.

1 INTRODUCTION

Recommendation outcomes offer real-world feedback for adapting content-generation agents, but delayed, context-dependent signals make it difficult to assess which memories contributed. TIDE uses credit assignment to evolve external memory, and the paper reports gains in offline and online evaluation.

1 INTRODUCTION

  • Recommendation feedback includes clicks, conversions, dwell time, and negative responses, which align more closely with business objectives than static human annotations.These signals create a loop from generation through recommendation and delivery to subsequent generation.Source: §1 · p. 2
  • Delayed outcomes depend on audience, context, and recommendation policy, and do not directly identify which content decisions or past experiences produced them.The paper defines MEG as the utility gain of evolved memory over a reference memory on strictly future tasks.Source: §1 · p. 2
  • TIDE estimates contextual memory fitness through temporal and semantic credit, then uses generation-time memory-use records for responsibility credit.It updates memories through reinforcement, crossover, mutation, or eviction under a fixed capacity constraint.Source: §1 · p. 2
  • The paper frames recommendation-feedback-driven memory evolution as fitness evaluation and multi-level credit assignment under delayed, context-dependent outcomes.Its stated contributions include MEG for measuring effects on strictly future tasks and directed, capacity-constrained memory updates.Source: §1 · p. 2
  • +7.75 percentage-point MEG offline, with significant online improvements in UCTR and activation rate, are reported for TIDE on e-commerce marketing generation.The evaluation includes offline temporal replay and a real-world online A/B test.Source: §1 · p. 2

2 RELATED WORK

Prior work studies both parameter updates from feedback and memory-based reuse or revision of experience. The paper positions TIDE as addressing the joint problem of delayed outcomes, delivery context, and responsibility across memories.

2 RELATED WORK

  • Parameter-based feedback methods primarily produce scores, rewards, or parameter updates, while this work focuses on traceable, individually revisable external memories.The cited examples include content generation, click feedback, and reward decomposition approaches.Source: §2 · p. 3
  • Existing memory methods use trajectories, interaction histories, engagement, or explicit corrections, but do not jointly model delayed-outcome reliability, delivery context, and responsibility across memories.TIDE integrates outcome reliability, content-level evidence, and memory-level responsibility in a feedback-to-memory credit-assignment pipeline.Source: §2 · p. 3

3 METHOD

TIDE evaluates memory updates by their utility on strictly future tasks, using delayed lifecycle and contextual evidence to assign fitness to memories that contributed to generation. It then revises and selects memories under a fixed capacity.

3.1 PROBLEM FORMULATION

  • MEG compares average utility on the next H tasks under a memory-update policy with a static or append-only reference policy.The update value is determined by strictly future tasks, which are unavailable when the update is made.Source: §3.1 · p. 3

3.2 TEMPORAL CREDIT: RELIABILITY OF DELAYED OUTCOMES

  • Temporal credit waits for a mature, continuous, comparable outcome lifecycle and summarizes stage-wise exposure, engagement, and conversion rather than relying on an arbitrary snapshot.The summary retains individual objectives and their uncertainty instead of forcing potentially conflicting metrics into one scalar.Source: §3.2 · p. 4

3.3 SEMANTIC CREDIT: CONTEXTUALIZING OUTCOME EVIDENCE

  • Semantic credit compares lifecycle outcomes with matched controls and delivery context to estimate contextual fitness and identify content decisions supported by online evidence.Insufficient controls, unstable estimates, or unresolved counterexamples make semantic evidence insufficient for attribution.Source: §3.3 · pp. 4–5

3.4 RESPONSIBILITY CREDIT AND MEMORY EVOLUTION

  • Responsibility credit assigns fitness only to memories actually used, linked to the evidenced decision, and shown by counterfactual replay to change that decision when removed.Retrieved memories that did not affect the decision receive no credit.Source: §3.4 · pp. 5–6
  • TIDE turns conditional or contradictory evidence into narrowed, revised, or split candidates, then governs and selects memories within the maximum capacity B.Complementary memories may be combined, while confirmed novel evidence can create a parentless provisional candidate; selection includes validity filtering, deduplication, conflict resolution, and expiration.Source: §3.4 · p. 6

4 EXPERIMENTS

Experiments evaluate TIDE on e-commerce content generation and delayed-label tasks, addressing end-to-end performance, feedback interpretation, responsibility attribution, and generalization.

4.1 MAIN RESULTS (RQ1)

  • +7.75 MEG on qwen3.5-27b makes TIDE the strongest memory method, with 750 inference tokens per request; it remains strongest among memory methods on Qwen3-4B.Combined with SFT, TIDE yields MEG +6.71 with 0.118 additional PFLOPs beyond SFT.Source: Table 1 · p. 7, §4.1 · p. 7
  • 5.10% UCTR and 4.79% activation-rate uplifts were statistically significant (both p < 0.001) in the one-week online comparison.The comparison kept the product pool, placement, and delivery mechanism fixed.Source: Table 2 · p. 7, §4.1 · pp. 7–8

4.2 ALIGNING ONLINE OUTCOMES WITH FEEDBACK MEMORY (RQ2)

  • TIDE achieves the highest CVR rank correlation (0.116) and CTR pairwise accuracy (0.604), plus the largest Future MEG (+0.180), highest win rate (60.7%), and lowest negative-transfer rate (25.0%).BERT leads the other two ranking metrics but requires parameter adaptation.Source: Figure 3 · p. 8, §4.2 · p. 8

4.3 FROM ALIGNED FEEDBACK TO MEMORY EVOLUTION (RQ3)

  • 0.591 Responsibility F1 and 0.400 Correct Edit follow removal of counterfactual replay; Bad-Memory Survival rises to 0.417 and Future MEG falls to −0.0031.Removing usage traces reduces Responsibility F1 to 0.708 and Future MEG to +0.0344.Source: §4.3 · pp. 8–9
  • Responsibility F1 of 0.817, Correct Edit rate of 0.650, zero Bad-Memory Survival, and Future MEG of +0.0750 give TIDE the strongest overall performance.The evaluation used 20 controlled episodes with five injected noise types; TIDE combined memory-use traces with leave-one-out counterfactual replay.Source: §4.3 · p. 8, Figure 4 · p. 9

4.4 GENERALIZATION TO DELAYED-LABEL TASKS (RQ4)

  • 0.531 MAE and 0.895 RMSE are TIDE's lowest results, versus strongest non-TIDE results of 0.572 and 1.073; it also achieves the highest MEG@δ=5 (+0.0365).Among adaptive methods, ∆delay is +0.0019, and user-level negative transfer ties Mem0 at 16.7%.Source: §4.4 · p. 9

5 CONCLUSION

The conclusion presents TIDE as a feedback-driven approach to evolving external memory under delayed, context-dependent outcomes.

5 CONCLUSION

  • TIDE combines temporal, semantic, and responsibility credit to evolve capacity-constrained external memory while keeping the generator frozen.The paper introduces MEG to measure utility improvement on strictly future tasks.Source: §5 · p. 9
  • +7.75 offline MEG and online uplifts of 5.10% in UCTR and 4.79% in activation rate are reported for e-commerce marketing generation.The online gains are described as associated with TIDE-evolved materials.Source: §5 · p. 9
  • MemoryCD results demonstrate generalization to delayed-label tasks, alongside analyses of feedback alignment and responsible editing.Table 3 presents results on the MemoryCD delayed-label benchmark.Source: §5 · p. 9, Table 3 · p. 10

AI USE DISCLOSURE

The disclosure describes generative AI assistance with manuscript clarity and software development, with author review and validation.

AI USE DISCLOSURE

  • Generative AI tools assisted with manuscript clarity, grammar, and readability.The authors reviewed and revised all AI-assisted text.Source: AI USE DISCLOSURE · p. 10
  • AI tools also assisted software development through code completion, debugging, and refactoring.The authors manually inspected and validated AI-assisted code through testing and experimental verification.Source: AI USE DISCLOSURE · p. 10
  • The authors take full responsibility for the work's correctness, originality, claims, results, and final content.Source: AI USE DISCLOSURE · p. 10

A ADDITIONAL EXPERIMENTAL CONFIGURATION

The experiments use chronological cutoffs and controlled comparisons to evaluate memory evolution, online outcomes, responsibility attribution, and delayed-label prediction. Future evaluation excludes information used to construct memory or train models.

Main offline experiment

  • The main offline experiment tests on 50 frozen future requests, excluding samples used for memory construction or parameter training and computing MEG against No Memory.Methods sharing a backbone use matched task order, feedback budget, generation constraints, candidate sets, and test samples; memory methods also share initial memory, retriever, retrieval count, and capacity.Source: A ADDITIONAL EXPERIMENTAL CONFIGURATION · p. 15

Online A/B experiment

  • The one-week randomized online A/B test compares TIDE-generated materials with the existing material pool while holding the base model, product pool, audience, placement, candidate count, and delivery mechanism fixed.The AA bucket checks traffic-allocation stability and is excluded from treatment-effect estimates; reported outcomes are relative UCTR and Activation Rate lifts.Source: A ADDITIONAL EXPERIMENTAL CONFIGURATION · p. 15

Online–offline alignment, predictive baselines, and feedback memory

  • The ranking experiment evaluates 28 equally weighted channel–benefit tasks using temporally available evidence, with TIDE’s two-layer feedback memory comprising experience cards and traceable source cases.The CVR ranking evaluation excludes five tasks with identical candidate outcomes; CTR retains all 28 tasks.Source: A ADDITIONAL EXPERIMENTAL CONFIGURATION · pp. 15–16

Responsibility attribution and memory editing

  • Responsibility and editing are tested on 20 controlled requests with five injected noise types, using ground-truth responsible memories and intended editing operations.Counterfactual analysis uses deterministic leave-one-memory-out replay; an edit is triggered only when removal produces a positive utility change.Source: A ADDITIONAL EXPERIMENTAL CONFIGURATION · p. 16

Public delayed-label benchmark

  • The delayed-label benchmark adapts MemoryCD by releasing labels five task steps after prediction, evolving memory on the first 80% of interactions, then freezing it for the final 20%.Labels from the final test segment never update memory; the protocol controls label-release timing rather than claiming MemoryCD natively contains delayed labels.Source: A ADDITIONAL EXPERIMENTAL CONFIGURATION · p. 16

B MULTI-ROUND MEMORY EVOLUTION AND CONVERGENCE

Across ten feedback rounds, TIDE achieves the strongest final utility while keeping memory bounded, unlike the growing Full History and Append-only baselines. Its evictions preferentially target injected noisy memories.

Results

  • TIDE reaches its strongest final utility while maintaining a bounded 80-item population, whereas Full History and Append-only grow throughout replay.The trajectories cover ten rounds on 12 fixed tasks; Figure 5 plots mean normalized utility with task-bootstrap 95% confidence intervals and active-memory tokens.Source: Figure 5 · p. 17, B MULTI-ROUND MEMORY EVOLUTION AND… · p. 17
  • TIDE’s advantage is concentrated in Compliance, Format, and Efficiency; its final batch-level Diversity is comparable to other methods rather than uniformly higher.The experiment also tracks generation and judge token consumption alongside utility, Joint Pass, component scores, and memory tokens.Source: B MULTI-ROUND MEMORY EVOLUTION AND… · p. 17, Figure 7 · p. 18
  • After ten feedback rounds, TIDE achieved the highest final Mean Utility, 0.809 (95% task-bootstrap CI [0.698, 0.909]), and a Joint Pass rate of 0.617.Mean Utility averages normalized compliance, format, and efficiency scores; confidence intervals use task-level bootstrap over 12 fixed tasks.Source: B MULTI-ROUND MEMORY EVOLUTION AND… · p. 17, Table 4 · p. 18
  • 2,121 active-memory tokens give TIDE the best utility, 20.1% fewer than Full History’s 2,653 tokens and 20.4% fewer than Append-only’s 2,666 tokens.Its performance gain therefore does not depend on unbounded historical accumulation.Source: B MULTI-ROUND MEMORY EVOLUTION AND… · p. 19
  • 9 of TIDE’s 10 EVICT operations target injected noisy memories (90.0%), versus 29 of 120 Random Replacement evictions (24.2%).The difference is significant under a one-sided Fisher exact test (p = 5.81 × 10−5).Source: Figure 8 · p. 18, B MULTI-ROUND MEMORY EVOLUTION AND… · p. 19

C CREDIT AND MEMORY-OPERATOR ABLATIONS

The ablations identify responsibility credit as the most consequential component: removing it produces the largest utility decline and the only confidence interval excluding zero. Removing each credit signal or editing operator also lowers Mean Utility.

Experimental setup

  • REVISE mutates an individual memory, MERGE consolidates complementary memories, and EVICT removes low-fitness memories during capacity-constrained selection.Without EVICT, candidates are not admitted once capacity is reached, and no incumbent is removed.Source: C CREDIT AND MEMORY-OPERATOR ABLATIONS · p. 19

Results

  • −0.149 is the paired Mean Utility difference when responsibility credit is removed, with a 95% CI of [−0.262, −0.044]; it is the only ablation interval excluding zero.Mean Utility falls from 0.852 to 0.703, while Joint Pass falls from 0.658 to 0.292.Source: C CREDIT AND MEMORY-OPERATOR ABLATIONS · p. 19, Table 5 · p. 19, Figure 9 · p. 20
  • Removing temporal credit, semantic credit, REVISE, MERGE, or EVICT reduces Mean Utility by 0.006, 0.015, 0.012, 0.040, and 0.030, respectively.The ablations share the initial memory, feedback order, capacity, 12 tasks, and seed 13; differences are measured relative to Full TIDE.Source: C CREDIT AND MEMORY-OPERATOR ABLATIONS · p. 19

D FITNESS–DIVERSITY DYNAMICS AND POPULATION HEALTH

At Round 10, TIDE combines higher utility and lower memory cost with a larger effective population. Its lower embedding dispersion is interpreted alongside functional diversity because the material space is tightly constrained.

Results

  • 0.809 Mean Utility exceeds Full History (0.655), Append-only (0.579), and Random Replacement (0.663), while TIDE uses 2,121 tokens and keeps 80 active items.Its Effective Population Size is 60.80, compared with 29.19–33.86 for the three baselines.Source: D FITNESS–DIVERSITY DYNAMICS AND… · p. 20, Table 6 · p. 20
  • Figure 10 presents quality–diversity trajectories over 11 snapshots on 12 fixed tasks, with replay round shown by color and active-memory tokens by marker size.TIDE attains the highest final utility with fewer memory tokens; the figure interprets embedding dispersion jointly with Effective Population Size and batch-level Diversity.Source: Figure 10 · p. 21
  • At Round 10, TIDE retains a larger effective population than the baselines while keeping its active population bounded at 80 items without unbounded token growth.Figure 11 reports Near-duplicate Rate, Effective Population Size, active population size, and Memory Churn across replay rounds.Source: Figure 11 · p. 21
  • TIDE’s Round-10 batch-level Diversity score is 0.500, comparable to Full History (0.510) and Append-only (0.490), and higher than Random Replacement (0.438).Its embedding-based Semantic Diversity is 0.174 and Near-duplicate Rate is 0.131; these are surface-level diagnostics, not complete measures of functional diversity.Source: Table 6 · p. 20, D FITNESS–DIVERSITY DYNAMICS AND… · p. 22

E ADAPTATION AND RECOVERY AFTER A DISTRIBUTION SHIFT

Under a controlled distribution shift, TIDE maintains higher post-shift utility and recovers faster than the compared memory methods, including when feedback is delayed. Its memory operators use attributed evidence and contextual safeguards to update a capacity-constrained memory population.

E ADAPTATION AND RECOVERY AFTER A DISTRIBUTION SHIFT

  • The controlled shift uses 12 stratified tasks and seed 13, switching channel–benefit composition from task distribution A to B.Feedback uses four evaluation dimensions, not real CTR/CVR or historical lifecycle labels.Source: E ADAPTATION AND RECOVERY AFTER A… · p. 22, Figure 12 · p. 22
  • TIDE recovers to 95% of pre-shift utility in 0.33 rounds on average, faster than Random Replacement, Append-only, Frozen Memory, and Full History.Its Immediate Drop is 0.034, the smallest among the methods.Source: E ADAPTATION AND RECOVERY AFTER A… · p. 22
  • Post-shift Utility AUC reaches 0.833, the highest among methods, and mean MEG relative to Frozen Memory is +0.074 (95% CI [0.024, 0.121]).Table 7 covers adaptation and recovery after the controlled distribution shift.Source: E ADAPTATION AND RECOVERY AFTER A… · p. 22, Table 7 · p. 23
  • With one- or two-round feedback delays, TIDE retains Post-shift AUC values of 0.788 and 0.791, above Frozen Memory’s 0.759.TIDE also recovers fastest while retaining a stale-memory survival rate of 0.996.Source: E ADAPTATION AND RECOVERY AFTER A… · p. 23

F MEMORY-OPERATOR PROMPT TEMPLATES

  • The memory operators reinforce, merge, revise, or assess memories for eviction using attributed evidence, context conditions, and provenance safeguards.Merging is rejected for incompatible contexts or unsupported shared evidence; eviction assessment considers reliability, usefulness, responsibility, redundancy, conflict, and recency.Source: F MEMORY-OPERATOR PROMPT TEMPLATES · p. 25
Loading 2609.37544v2…