Source-linked AI summary
Learning What to Share and What to Personalize: Hierarchical Strategy Co-Evolution for Agent Memory
Yupeng Han, Shuochen Liu, Kai Zhang, Ze Liu, Zhihong Pan, Xianquan Wang
TL;DR
Memory systems need to manage evolving conversational context, but existing strategies are often static and user-agnostic. HiPS co-evolves universal and personalized rules from on-policy evidence, with experiments reporting consistent gains and domain-dependent component importance.
Problem
Existing memory strategies are commonly static and user-agnostic, limiting adaptation to diverse behaviors and policy evolution.
Method
HiPS co-evolves universal rules and user-specific adaptive rules through Universal Strategy Distillation, Persona Delta Distillation, and Cross-Level Rule Flow.
Results
HiPS demonstrates consistent performance gains, with universal rules most important in-domain and adaptive mechanisms driving out-of-domain generalization.
Takeaways & Limitations
The shared-versus-personalized strategy boundary can be discovered empirically and updated using on-policy evidence rather than predefined.
Takeaways & Limitations
Evaluation focuses primarily on English-language conversational benchmarks, while multilingual and cross-lingual settings remain open areas for future work.
Abstract
from arXiv · showhide
Memory-augmented agents maintain compact user profiles throughout extended conversations, enabling personalized and consistent responses without the need to process the entire dialogue history. The quality of these user profiles relies on the underlying memory management strategy: at each step, the agent must determine what to retain, compress, or discard. However, existing methods typically employ a static, one-size-fits-all strategy established before training. In practice, the optimal memory decision is inherently user-specific and dynamically evolves alongside policy optimization. To address this, we propose \textbf{HiPS} (\textbf{Hi}erarchical \textbf{P}ersonalized \textbf{S}trategy), a framework that decouples memory management into a globally shared foundation and a user-specific adaptive tier. Specifically, HiPS employs \textbf{Universal Strategy} to extract shared principles from cross-persona trajectories, alongside \textbf{Persona Delta Distillation} to generate tailored rules for users whose behaviors diverge from general patterns. \textbf{Cross-Level Rule Flow} dynamically calibrates their boundary by promoting broadly validated personal rules and demoting contradicted global ones. The architecture establishes a co-evolution loop where a mechanism guarantees that all strategy refinements are anchored to task outcomes. Extensive experiments demonstrate consistent improvements over memory-augmented baselines.
1 Introduction
Existing memory strategies struggle to balance shared utility with user-specific needs and often remain static as policies evolve. HiPS addresses this by co-evolving universal and personalized rules from on-policy evidence, yielding consistent gains and different component priorities across domains.
- Memory-augmented agents must decide whether to retain, compress, or discard incoming context because raw dialogue cannot be retained or directly captures evolving preferences.
- Existing approaches are either static or user-agnostic, causing population-averaged learning signals to underrepresent niche behaviors and producing one-size-fits-all compromises.
- HiPS decomposes memory management into shared universal rules and per-user adaptive rules, discovering their boundary dynamically from on-policy evidence.
- Universal Strategy Distillation learns shared rules from cross-persona trajectories, while Persona Delta Distillation targets users whose behaviors diverge from population norms.
- Cross-Level Rule Flow promotes broadly validated personal rules and replaces contradicted global rules with targeted revisions, while strategy refinement remains tied to task outcomes.
- Universal rules dominate in-domain tasks, whereas adaptive mechanisms drive out-of-domain generalization, and experiments show consistent performance gains.
2 Related Work
Memory research includes memory banks, structured retrieval, personalization systems, RL-trained policies, and interpretable strategy methods. HiPS differs by combining user-indexed strategies with dynamic co-evolution rather than frozen global rules.
- Memory-bank systems segment, summarize, and selectively forget history, while tree- and graph-based indices improve retrieval access efficiency.
- Personalization-oriented systems maintain user profiles for downstream conditioning, extending memory management beyond generic historical retrieval.
- RL-based methods make memory operations learnable but encode management behavior implicitly in model parameters, limiting inspectability and editability.
- Strategy-based methods provide interpretable rules, but their rules are globally shared and frozen before training.
- HiPS uses user-indexed strategies that dynamically co-evolve with the policy under on-policy evidence.
3 Preliminary
The framework models memory updates as prompt-guided decisions over sequential dialogue chunks and decomposes the active strategy into shared and personalized components. These components co-evolve during training as evidence accumulates.
- The agent processes dialogue chunks sequentially while maintaining a compact memory state updated as new information arrives.
- Management rules are injected into the system prompt to regulate memory actions at each step.
- HiPS decomposes the active strategy into a shared component and a personalized component, allowing their partition to co-evolve with the policy.
- The shared component contains universally beneficial rules, while the personalized component contains rules tailored to user p’s behavioral patterns.
- After all K chunks, the agent generates a response conditioned on the terminal memory state mK.
4 Method
HiPS separates memory management into universal and user-specific tiers, then co-evolves their rules with policy rollouts using outcome-linked evidence. Its mechanisms gate personalization, move rules across tiers, select rules under a token budget, and align policy optimization with strategy adherence.
- Architecture: HiPS decomposes memory management into a shared universal tier and a user-specific adaptive tier, dynamically learning their boundary from on-policy evidence.Universal Strategy Distillation abstracts broadly applicable principles, while Persona Delta Distillation captures rules for users with divergent behaviors.
- Universal Strategy Distillation: Universal Strategy Distillation compares high- and low-reward trajectories and produces structured rule updates rather than free-form feedback.The updates use validation, hypothesis, and revision or retirement operations, with rules constrained to explicit management behaviors.
- Persona Delta Distillation: Predictive gain ranks rule importance and gates personalization, while users with divergence D(p) ≥ θdiv trigger Persona Delta Distillation.Users below the threshold bypass personalization because the universal strategy is considered sufficient; predictive gain is used for relative ranking and threshold-based gating, not as an absolute quality measure.
- Cross-Level Rule Flow: Cross-Level Rule Flow promotes broadly useful personalized rules into the universal tier and creates localized replacements when universal rules are revised or contradicted.Generalization occurs when supported persona rules recur across enough users, while specialization protects minority users who benefited from deprecated universal guidance.
- Strategy Injection and Co-Evolution: HiPS constrains the active strategy with a token budget, allocates capacity according to aggregate predictive gains, and aligns policy optimization with a persona-aware adherence reward.The active strategy directs rollouts, resulting trajectories provide feedback for refinement, and the combined GRPO reward adds adherence to answer reward.
- Strategy Injection and Co-Evolution: HiPS separates answer reward from adherence-based distillation inputs to prevent rule compliance from directly inflating the evidence used for strategy updates.Distillation buffers are ranked exclusively by answer reward, while adherence contributes to GRPO advantage computation and can influence later rollout behavior indirectly.
5 Experiments
HiPS is evaluated across personalized-memory benchmarks, ablations, persona profiles, backbone models, and context scales. It consistently outperforms baselines, while experiments show that universal and adaptive components matter differently across in-domain and out-of-domain settings.
- Overall Comparison with Baselines: HiPS consistently outperforms baselines across all twelve evaluation settings.The evaluation covers in-domain PersonaMem and out-of-domain PrefEval, PersonaBench, and PERMA settings.
- Generalization Across Settings: HiPS generalizes across in-domain and out-of-domain tasks, including explicit and implicit preference queries, noisy contexts, and domain crossings.The authors attribute this pattern to universal strategy distillation combined with persona delta distillation.
- Ablation Study: Ablations show an importance flip: USD and PG matter most in-domain, whereas Flow and Gate matter most out-of-domain.Removing Flow reduces PERMA C-S accuracy from 66.95 to 45.39, while removing USD reduces PersonaMem 128K accuracy from 62.01 to 53.70.
- Strategy Quality Analysis: 52.4% to 72.1%: universal-strategy accuracy rises during training as rules mature from tentative to supported and established.The rule count stabilizes at 6 established rules after step 480+, reflecting a create-validate-prune lifecycle.
- Strategy Quality Analysis: +6.0% average improvement: persona-specific strategies benefit 20 personas, but effects vary from +18.6% for P15 to -3.3% for P6.P15 benefits from a financial logging rule, whereas additional rules over-constrain P6 because universal strategies already manage that profile well.
- Cross-model Transfer: HiPS strategies transfer across backbone LLMs, and Su+∆p outperforms Su-only across models.GPT-4o-mini optimization achieves 71.48 on GPT-5 and 68.12 on Gemini 2.5 flash.
- Scaling Analysis: As dialogue tokens grow from 4K to 128K, memory-bank size increases from 1,000 to around 1,900 tokens while evolution time rises roughly linearly from 140 to 1,700 seconds.The bounded memory growth is attributed to consolidating preference-relevant information and filtering redundant content.
6 Conclusion
HiPS decouples memory management into a universal baseline and user-specific adaptive delta, updating their partition with online evidence. Its co-evolution of strategies and policies yields consistent gains over retrieval and RL-trained counterparts, especially as context grows.
- HiPS decouples memory management into a universal baseline and a user-specific adaptive delta, with the partition updated using online evidence.
- HiPS establishes a continuous co-evolution loop between multi-tiered strategies and active policies for personalized long-term agent memory.
- HiPS secures consistent performance gains over retrieval and RL-trained counterparts, with advantages becoming pronounced as dialogue context scales.
Limitations
The evaluation is concentrated on English conversational benchmarks and histories up to 1M tokens. Multilingual settings and extremely long-term personality drift remain open directions requiring further mechanisms.
- The evaluation primarily covers English-language conversational benchmarks, leaving multilingual and cross-lingual settings for future work.
- Experiments span histories up to 1M tokens, while years-long interactions may involve gradual baseline personality shifts requiring additional meta-distillation mechanisms.
A.1 PersonaMem Benchmark
PersonaMem evaluates long-term personalization across simulated personas, evolving preferences, diverse tasks, and multiple context scales. Its interaction histories range from approximately 32k to 1M tokens.
- PersonaMem evaluates long-term personalization for 20 simulated personas across 15 diverse real-world task domains.
- Each persona has static attributes plus dynamic traits and preferences, tested through seven types of in-situ personalization queries.
- Histories concatenate 10, 20, or 60 sessions, producing approximate context lengths of 32k, 128k, and 1M tokens.
- At evaluation time, models select responses conditioned on interaction histories to test adaptation to dynamic user profiles.
A.2 PrefEval Benchmark
PrefEval tests whether language models can infer, retrieve, and act on user preferences across long, multi-session conversations. Its classification experiments use explicit and implicit instances separated from queries by 50 distractor turns.
- PrefEval evaluates preference inference, long-context retrieval, preference following, and personalization proactiveness.
- The benchmark contains 1,000 unique preference-query pairs spanning 20 everyday topics across seven domains.
- Experiments use 1,000 explicit and 1,000 implicit multiple-choice instances with 50 intervening distractor turns.
A.4 PERMA Benchmark
PERMA is an event-driven long-context benchmark for testing whether personalized memory agents maintain, update, and synthesize changing persona states. It evaluates task completion, preference consistency, and informational confidence across structured conversational settings.
- Benchmark scope: PERMA evaluates personalized memory agents on maintaining, updating, and synthesizing dynamic persona states in realistic conversational environments.Its evaluation emphasizes task completion, preference consistency, and informational confidence.
- Dataset composition: The dataset contains 10 representative user profiles with 2,166 fine-grained preference details spanning 10 countries and 20 domains.
- Evaluation coverage: PERMA covers domains including Travel, Finance, Shopping, Entertainment, Messaging, and Calendar.
- Evaluation formats: The benchmark supports both 8-option multiple-choice evaluation with systematically ablated options and multiturn interaction with an LLM-based user simulator.The interactive setting terminates upon successful task completion.
- Evaluation design: Event-driven timelines and controlled interventions structure interactions to test long-horizon personalization.
B Comparison in Different Categories
Across PersonaMem and PrefEval category and domain breakdowns, HiPS generally performs best, especially on evolving or implicit preference tasks and longer interaction histories.
- PersonaMem: 73.49 at 32K and 62.01 at 128K are HiPS’s best overall PersonaMem scores across the two history scales.At 32K, HiPS leads the strongest baseline MemSkill by 73.49 vs. 64.45.
- PersonaMem: At 128K, HiPS ranks first on Recall facts (68.42), Suggest ideas (44.02), Latest prefs (71.82), Prefs evolve (71.55), Update reasons (72.49), Aligned recs (54.73), and New Scenarios (44.13).Recall facts is not a comparative strength at 32K, where MemAgent scores 72.87 versus HiPS’s 71.43.
- PersonaMem: 20.74 is Long Context’s overall score at 128K, while HiPS remains the best-performing memory-based method.The results indicate stronger relative performance for memory-based methods as histories lengthen.
- PrefEval: 89.20 is HiPS’s overall accuracy under Explicit Memory on PrefEval, where it ranks first across all evaluated domains.Domain scores include Travel 90.20, Entertain 94.20, Lifestyle 91.70, Shop 84.10, Education 86.30, Professional 85.20, and Pet 78.10.
- PrefEval: 69.40 is HiPS’s overall accuracy under Implicit Memory, with the clearest gains in Travel 71.90, Lifestyle 71.60, Shop 64.50, and Professional 63.40.Education is an exception: HiPS scores 65.10 versus MemSkill’s 65.80.
C Hyperparameters
HiPS tunes strategy-update frequencies, budgets, and thresholds to balance expressiveness, personalization, and efficiency, while co-evolving rules with policy rollouts and task outcomes.
- Hyperparameter choices: 200 strategy tokens is selected as the default budget because 300 yields only marginally higher accuracy while using more prompt tokens.
- Hyperparameter choices: 0.3 is the selected divergence threshold θdiv, balancing excessive personalization at 0.1 against over-filtering divergent users at 0.5.
- Rule lifecycle: Rules begin at Supported evidence level and can reach Established status through repeated validation, while contradicted rules may be revised, demoted, or pruned.Tentative rules await confirmation; Established rules receive the highest injection priority.
- Universal strategy: Universal Strategy Distillation compares high- and low-reward cross-user trajectories to identify generalizable memory-management patterns rather than user-specific content.Overlapping hypotheses with word overlap > 0.7 are treated as validations to prevent rule explosion.
- Hyperparameter roles: HiPS uses strategy token budget B and update frequencies K1 and K2 to control shared and personalized rule distillation.K1 governs Universal Strategy Distillation frequency, while K2 governs Persona Delta Distillation frequency.
- Personalized strategy: Persona Delta Distillation derives observable management rules for users whose behavior differs from universal rules, while avoiding identity- or topic-based conditions.
- Validation signals: Predictive gain ranks and gates rules rather than measuring absolute quality, while the adherence reward uses λ = 0.3 as a small auxiliary weight.Compliance is estimated with a lightweight keyword heuristic to avoid additional LLM calls during training.
- Co-evolution procedure: At each step, HiPS selects and injects a budgeted combination of universal rules Su and personalized deltas ∆p, then collects task and adherence rewards from rollouts.Distillation periodically updates personalized and universal rules, with broadly successful personalized rules elevated into the global standard.