Source-linked AI summary
Controllable Memory Usage: Balancing Anchoring and Innovation in Long-Term Human-Agent Interaction
Muzhao Tian, Zisu Huang, Xiaohua Wang, Jingwen Xu, Zhengkang Guo, Qi Qian, Yuanzhe Shen, Kaitao Song, Jiakang Yuan, Changze Lv, Xiaoqing Zheng
TL;DR
Long-horizon agents need memory for personalization, yet existing approaches can anchor outputs to history and lack fine-grained user control over memory dependence. The paper measures memory dependence and introduces SteeM, which dynamically steers outputs toward user-specified reliance levels. Across scenarios, SteeM achieves better preference alignment than prompting and memory masking, while transferring beyond the controlled setting with minimal impact on general performance.
Problem
Existing agents lack a real-time mechanism for users to regulate memory dependence, and may default to high reliance despite requests for less history influence.
Method
SteeM uses a realistic long-horizon interaction dataset, measures behavioral memory dependence, and trains preference-aligned SFT and RL models to follow target dependence values.
Results
SteeM consistently achieves substantially better preference alignment, transfers beyond the controlled setting with minimal impact on general performance, and outperforms direct memory masking in pairwise comparisons.
Takeaways & Limitations
Memory reliance can be treated as a user-controllable behavior dimension for balancing consistency and innovation in personalized agents.
Takeaways & Limitations
The evaluation models memory preference on a 1–5 ordinal scale and covers only Research and Tutoring scenarios, which may differ from real interactions.
Abstract
from arXiv · showhide
As LLM-based agents are increasingly used in long-term interactions, cumulative memory is critical for enabling personalization and maintaining stylistic consistency. However, most existing systems adopt an ``all-or-nothing'' approach to memory usage: incorporating all relevant past information can lead to \textit{Memory Anchoring}, where the agent is trapped by past interactions, while excluding memory entirely results in under-utilization and the loss of important interaction history. We show that an agent's reliance on memory can be modeled as an explicit and user-controllable dimension. We first introduce a behavioral metric of memory dependence to quantify the influence of past interactions on current outputs. We then propose \textbf{Stee}rable \textbf{M}emory Agent, \texttt{SteeM}, a framework that allows users to dynamically regulate memory reliance, ranging from a fresh-start mode that promotes innovation to a high-fidelity mode that closely follows interaction history. Experiments across different scenarios demonstrate that our approach consistently outperforms conventional prompting and rigid memory masking strategies, yielding a more nuanced and effective control for personalized human-agent collaboration.
1 Introduction
Long-term agents need memory for continuity, but static retrieval can anchor outputs to history and limit users’ ability to request independent reasoning. SteeM makes memory dependence an explicit, dynamically controllable dimension and improves alignment with user preferences.
- Persistent memory supports personalization and stylistic consistency by helping agents continue from prior profiles, preferences, and project states.
- Dynamic user needs range from faithfully inheriting prior constraints to adopting a fresh-eyed perspective that gives less weight to legacy context.
- Memory Anchoring occurs when accumulated interaction history overly constrains the agent, preventing the clean-slate reasoning requested by the user.
- Existing systems provide coarse on/off controls or manual masking, while prompting can leave historical stylistic and ideological biases leaking into responses.
- SteeM lets users dynamically regulate memory reliance from bracketed independent reasoning to high-fidelity adherence to historical context.
- SteeM outperforms prompt-based methods and memory masking, enabling more precise balance between memory awareness and reasoning independence across diverse long-horizon tasks.
2 Related Work
Prior work improves long-term personalization through alignment methods and retrievable memories, but it largely evaluates factual preferences and assumes consistency with history. This paper frames memory dependence as an additional, intent-dependent preference to evaluate and control.
- Alignment methods include representation engineering, prompt optimization, supervised fine-tuning, direct preference optimization, and reinforcement learning from preference rewards.
- Long-term conversation benchmarks show persistent difficulty tracking narratives, retrieving relevant context, and producing preference-aligned responses.
- Existing personalization studies mainly assess factual preference satisfaction and leave memory dependence underexplored.
- These studies generally assume per-query preferences remain consistent with prior interactions, although user intent can naturally diverge from historical patterns.
- Retrievable memory modules externalize interaction history and selectively construct query-relevant context to improve long-horizon continuity and personalization.
3 Understanding Memory Anchoring with Realistic Synthetic Data
The paper builds a synthetic long-horizon interaction setting and introduces a rubric-based measure of how strongly responses rely on query-specific memory. Human validation and model experiments show that existing LLMs default to high memory dependence, while prompting provides limited control.
- 3.1 Simulating Long-Horizon Interaction Histories: The synthetic pipeline models Research and Tutoring workflows as timelines of events and evolving artifacts, then grounds queries in specific events and targets.The dataset covers recurring tasks including Plan & Design, Revise, Analyze & Critique, and Concept Explanation.
- 3.1 Simulating Long-Horizon Interaction Histories: Each query receives memory comprising long-term profile information, relevant cross-session interactions, and recent intra-session history.These components are rewritten from query-relevant timeline events and artifacts into concise summaries.
- 3.1 Simulating Long-Horizon Interaction Histories: The resulting dataset contains over 7,000 events, 7,000 artifacts, and 10,000+ query-memory pairs, including a held-out test set of 1000 pairs.The pipeline is designed to support analysis of memory dependence and preference alignment.
- 3.2 Formulating Memory-Dependence Preference: MD-Score measures response reliance on memory on a 1–5 rubric spanning memory-agnostic to strongly memory-grounded behavior, with larger values indicating stronger reliance.The score is produced by an LLM-as-a-judge evaluator using human-aligned rubrics.
- 3.2 Formulating Memory-Dependence Preference: Human judgments support MD-Score as a proxy for memory dependence, with especially strong consistency when paired responses have a large score gap.Annotators compared responses generated under different prompting settings and models.
- 3.3 Memory Anchoring in Agent Generation: Across models, dependence scores concentrate at 4-5, and changing prompts from LOW to HIGH produces only marginal shifts.This persistent high dependence despite explicit instructions is termed Memory Anchoring.
4 Method
The method trains models to match user-specified memory-dependence preferences while preserving task and general response quality. It combines preference-aligned data generation, supervised fine-tuning, and reinforcement learning with a composite reward.
- 4 Method: The training objective minimizes alignment error so responses match the user’s query-specific preference for memory dependence while preserving task quality.The approach addresses models’ tendency to produce heavily memory-reliant responses even under low-dependence instructions.
- 4 Method: Preference-aligned data generation augments queries with target dependence levels, elicits preference-indicative rewrites, and creates aligned variants matching each response’s realized dependence.The final triples are (qalign, M(q), y), with the aligned preference set to the realized dependence score.
- 4 Method: Quality-preserving filtering scores candidates with task-oriented rubrics and a reward model, retaining a final 7000-example SFT set that is aligned and high-quality.Qwen3-4B and Qwen3-8B are fine-tuned on this set with token-level cross-entropy.
5 Experiments
Across scenarios and tasks, SteeM aligns realized memory dependence with user-preferred levels while preserving response quality and generalizing to unseen subjects. It also compares favorably with prompting and memory masking, offering more precise control over memory reliance.
- Alignment: SteeM consistently achieves substantially lower δalign than the baseline across Research and Tutoring scenarios and four shared tasks.Lower alignment error indicates closer matching between realized memory dependence and the user-preferred level.
- Alignment: Compared with the baseline, SteeM shifts realized-dependence distributions toward target-level diagonals instead of concentrating at high levels 4–5.The evaluation samples 100 queries per dependence level and visualizes target-conditioned realized levels as confusion-matrix heatmaps.
- Generalization: SteeM transfers preference-following behavior to unseen Medical and Humanities subjects, with the RL-enhanced variant generalizing more strongly than SFT alone.The larger gap between variants on unseen subjects indicates stronger transfer for the RL-enhanced model.
- Response Quality: SteeM maintains response quality comparable to the baseline and produces slightly higher reward scores in several cases.The evaluation uses Skywork-Reward-V2-Llama-3.1-8B and also reports AlpacaEval results, indicating minimal impact on general response quality.
- Memory Masking: Against memory masking, SteeM yields a consistent task-wise win-rate advantage while remaining competitive across Qwen3-8B and Qwen3-4B.Masking changes available information but cannot reliably regulate reliance strength and may remove critical constraints or facts.
- Case Study: In a low-memory case study, SteeM introduces larger departures from the historical pipeline, including adaptive sampling and progress-triggered transitions.The baseline largely follows historical decisions, whereas SteeM better matches the request for new ideas with low memory dependence.
6 Conclusion
The paper studies how much agents should rely on historical memory in long-horizon interactions and introduces SteeM to make that reliance controllable. SteeM improves preference alignment, transfers beyond the controlled setting, and minimally affects general performance.
- The study identifies memory anchoring as models’ tendency to default to high reliance on historical memory despite user intent.
- SteeM addresses this problem through preference-aligned supervised fine-tuning and reinforcement learning.
- SteeM achieves substantially better preference alignment than the compared approaches.
- The method transfers beyond the controlled long-horizon setting with minimal impact on general performance and outperforms direct memory masking in pairwise comparisons.
Limitations
The dataset is designed to approximate long-horizon projects but may not fully represent real human interactions. The study also uses a coarse 1–5 preference scale and covers only Research and Tutoring scenarios.
- The simulated long-horizon projects may still differ from real human interactions despite efforts to make them realistic.
- Memory-dependence preference is modeled on a 1–5 ordinal scale, whereas users may express richer and more nuanced constraints.
- The current setup covers only Research and Tutoring scenarios.
- Broader application settings and more diverse task distributions remain important directions for extending the data and evaluation.
A Dataset Details
The dataset simulates long-term Research and Tutoring projects as evolving timelines of events and artifacts, then attaches standardized queries and query-specific memories to them. This construction supports comparable analysis of controllable memory usage.
- Scenarios and Topics: The dataset contains Research and Tutoring project scenarios representing sustained human-agent collaboration.
- Iterative Timeline Synthesis: Each project timeline is synthesized iteratively, with dependency checks and coherence assessments governing event generation and artifact updates.
- Tasks and Queries: Queries use four shared categories: Plan & Design, Revise, Analyze & Critique, and Concept Explanation.
- Tasks and Queries: Each query is attached to a timeline event and target artifact, enabling history-agnostic and history-dependent solutions to be compared for the same task form.
- Query-Specific Memory Construction: Query-specific memory combines a user profile, relevant cross-session interactions, and recent within-session history.
- Dataset Statistics: The resulting dataset statistics are reported in Table 2 and Figure 8, while Table 3 lists the scenario event and artifact types.
B Training Details
SteeM is trained with supervised fine-tuning followed by reinforcement learning using specified optimization configurations. The training data comprises aligned SFT and RL samples, and AlpacaEval uses Skywork-Reward-V2-Llama-3.1-8B for scoring.
- Supervised Fine-Tuning: Supervised fine-tuning uses MS-SWIFT with batch size 64, learning rate 1 × 10^-5, and 3 training epochs.
- Reinforcement Learning: Reinforcement learning uses GRPO through EASYR1 with rollout batch size 32, update batch size 8, learning rate 5 × 10^-6, sequence length 6144, and 8 rollouts per prompt.
- Training Data: The training set contains 7000 aligned SFT samples and 2000 RL samples.
- Evaluation: AlpacaEval results are reported in Table 4 and scored with Skywork-Reward-V2-Llama-3.1-8B.
D Case Study
Table 5 presents a case study where SteeM generates a more creative, less memory-dependent refinement than the baseline for a low-memory request. The section also describes the human annotation procedure used to assess memory reliance.
- Case Study: SteeM produces a more creative, less memory-dependent PROJECT_METHOD update that better matches the user’s low-memory intent.Blue highlights identify historical-artifact following, while red highlights identify newly introduced ideas.
- Case Study: The baseline largely follows the historical pipeline with only minor additions, reflecting memory anchoring despite the user’s instruction.
- Case Study: SteeM introduces larger departures from the historical pipeline, including adaptive sampling and progress-triggered transitions.
- Annotation Protocol: The reported judging rubric assigns an integer MD-Score DR, with complete scale definitions and dimension-wise guidance provided in Table 7.
- Annotation Protocol: Human annotation compares two candidate responses sharing the same query and memory context, selecting which response relies more on the provided memory.The protocol annotates 1000 pairwise instances, randomly split into 10 shards with 100 instances assigned to each volunteer annotator.
H Comparison with Memory Masking
This section provides the prompts and evaluation materials for comparing SteeM with direct memory masking. The supplied materials define memory filtering and pairwise judging, while Table 6 compares δalign across training-query conditions.
- Evaluation: Table 6 compares δalign between training with tag-cued queries and training with NL-cued queries for SteeM, with lower values preferred.
- Memory Masking: The memory-masking user simulator filters available memory by dropping items that would push the assistant away from the desired output.
- Evaluation Materials: The supplied materials identify Figure 9 as the user-simulator prompt for memory masking and Figure 10 as the pairwise-comparison prompt.
- Pairwise Comparison: The pairwise comparison prompt asks a judge to choose the better answer given the user query, memory context, and two model responses.
- Evaluation Materials: Figure 11 presents the protocol for human pairwise annotation of memory reliance.