PaperlayerView exact revision on arXiv

Source-linked AI summary

Controllable Memory Usage: Balancing Anchoring and Innovation in Long-Term Human-Agent Interaction

Muzhao Tian, Zisu Huang, Xiaohua Wang, Jingwen Xu, Zhengkang Guo, Qi Qian, Yuanzhe Shen, Kaitao Song, Jiakang Yuan, Changze Lv, Xiaoqing Zheng

arXiv:2601.05107v1cs.AI

TL;DR

Long-term agents lack a real-time way for users to control how strongly memory shapes outputs. SteeM makes memory reliance a controllable dimension, ranging from independent reasoning to historical fidelity. It achieves substantially better preference alignment, transfers beyond controlled long-horizon settings with minimal impact on general performance, and outperforms direct memory masking in pairwise comparisons.

  • Problem

    Existing systems lack a real-time mechanism for users to regulate memory dependence, relying instead on static retrieval or coarse on/off and manual-masking controls.

    Source: §1 · pp. 1–2

  • Method

    SteeM makes memory reliance user-controllable across modes from independent reasoning to high-fidelity adherence, using preference-aligned supervised fine-tuning and reinforcement learning.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    SteeM achieves substantially better preference alignment, transfers beyond controlled long-horizon settings with minimal impact on general performance, and outperforms direct memory masking in pairwise comparisons.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The study offers an initial step toward practical, user-controllable memory reliance for personalized agents.

    Source: §6 · p. 9

    The simulated testbed may differ from real human interactions, represents preferences on a 1–5 ordinal scale, and covers only Research and Tutoring.

    Source: Limitations · p. 9

Abstract

from arXiv · show

As LLM-based agents are increasingly used in long-term interactions, cumulative memory is critical for enabling personalization and maintaining stylistic consistency. However, most existing systems adopt an ``all-or-nothing'' approach to memory usage: incorporating all relevant past information can lead to \textit{Memory Anchoring}, where the agent is trapped by past interactions, while excluding memory entirely results in under-utilization and the loss of important interaction history. We show that an agent's reliance on memory can be modeled as an explicit and user-controllable dimension. We first introduce a behavioral metric of memory dependence to quantify the influence of past interactions on current outputs. We then propose \textbf{Stee}rable \textbf{M}emory Agent, \texttt{SteeM}, a framework that allows users to dynamically regulate memory reliance, ranging from a fresh-start mode that promotes innovation to a high-fidelity mode that closely follows interaction history. Experiments across different scenarios demonstrate that our approach consistently outperforms conventional prompting and rigid memory masking strategies, yielding a more nuanced and effective control for personalized human-agent collaboration.

1 Introduction

SteeM treats reliance on long-term memory as a user-controlled behavior, letting users balance historical consistency with independent reasoning as their needs change. The paper reports that it outperforms prompt-based methods and memory masking across diverse long-horizon tasks.

1 Introduction

  • Long-term memory stores user profiles, preferences, and project states, helping agents maintain personalization and continuity across extended interactions.Retrieved context can help agents pick up where they left off rather than start from scratch.Source: §1 · p. 1
  • Static memory injection can cause Memory Anchoring, constraining agents to interaction history even when users want clean-slate reasoning.Retrieved records can strongly steer outputs toward similar responses, while binary toggles and manual masking do not regulate behavioral dependence in real time; prompting may still allow memory leakage.Source: §1 · pp. 1–2
  • SteeM outperforms prompt-based methods and memory masking, enabling a more precise balance between memory awareness and reasoning independence across diverse long-horizon tasks.The framework lets agents follow a target memory-dependence value across scenarios.Source: §1 · p. 2
  • SteeM lets users dynamically steer memory reliance from a bracketed mode prioritizing independent reasoning to a high-fidelity mode adhering closely to historical context.The paper frames memory dependence as a control axis for navigating consistency and innovation according to users’ immediate needs.Source: §1 · p. 2

2 Related Work

Related work addresses alignment to global response preferences, personalization in long-term conversations, and retrieval-based memory agents. These areas leave query-specific, dynamically varying memory reliance insufficiently addressed or controlled.

Alignment for Large Language Models

Evaluating Personalization in Long-term Conversations

Memory-Enhanced Personalized Agents

  • Memory-enhanced agents use persistent stores and selective retrieval to improve continuity and personalization, but offer limited user control over reliance on retrieved memories (Liu et al., 2025b; Zhang et al., 2025; Xiong et al., 2025).Examples include RMM’s multi-granularity summarization and retrospective retrieval refinement, LD-Agent’s tunable modules, and O-Mem’s dynamic profiles and hierarchical retrieval (Tan et al., 2025; Li et al., 2025; Wang et al., 2025).Source: §2 · p. 3

3 Understanding Memory Anchoring with Realistic Synthetic Data

This section builds synthetic long-horizon histories and a rubric-based measure of memory dependence, then finds that prompting alone does little to counter agents’ default high memory reliance.

3.1 Simulating Long-Horizon Interaction Histories

  • The pipeline models Research and Tutoring projects with 200 diverse topics each and four recurring task types, grounding queries in timeline events and evolving artifacts.It yields over 7,000 events, 7,000 artifacts, and 10,000+ query-memory pairs, with a held-out test set of 1,000 pairs; each query memory summarizes user goals and preferences, relevant cross-session interactions, and recent intra-session history.Source: §3.1 · pp. 3–4

3.2 Formulating Memory-Dependence Preference

  • The 1–5 rubric-based MD-Score uses an LLM judge to rate memory reliance, while MD-Pref specifies a query’s desired reliance level and alignment error measures the match.Rubrics span memory-agnostic to strongly memory-grounded behaviors, with higher MD-Scores indicating stronger reliance on query-specific memory.Source: §3.2 · p. 4

3.3 Memory Anchoring in Agent Generation

  • MD-Score rankings showed strong consistency with human judgments, especially for response pairs with larger score gaps, supporting the metric as a proxy for memory dependence.Human annotators judged which response relied more on the same query-specific memory, and the study reported agreement rate and rank correlation.Source: §3.3 · p. 5, Figure 3 · p. 5
  • Across all models, memory-dependence scores concentrated at 4–5, and switching prompts from LOW to HIGH yielded only marginal shifts, suggesting persistent high-dependence generation despite explicit user instructions (Memory Anchoring).Source: §3.3 · p. 5

4 Method

SteeM optimizes responses to match a query-specific memory-dependence preference, using preference-aligned supervised fine-tuning followed by reinforcement learning to preserve both alignment and task quality.

4.1 Problem Formulation

  • SteeM optimizes response generation to minimize the alignment error between realized memory dependence and the user’s query-specific preference.The response is generated from the query and its constructed memory; the objective is to match the preference p(q).Source: §4.1 · p. 5

4.2 Memory-Dependence Aligned Supervised Fine-Tuning

  • To address memory anchoring that undermines naive sample-and-filter, SteeM constructs preference-aligned training triples by matching rewritten queries to each candidate response’s realized dependence.A Gemini-2.5-Pro user simulator rewrites queries for target preferences 1–5, and four candidates are sampled per augmented query from Qwen3-8B and Qwen3-14B; the simulator then rewrites the original query to match each retained candidate’s realized dependence.Source: §4.2 · pp. 5–6
  • Quality filtering retains the highest-scoring candidates to form a 7,000-example SFT set, which is used to fine-tune Qwen3-4B and Qwen3-8B.Candidates are scored with task-oriented rubrics and a reward model before standard token-level cross-entropy fine-tuning.Source: §4.2 · p. 6

4.3 δalign-Guided Reinforcement Learning

  • After SFT, SteeM applies GRPO to 2,000 non-overlapping RL samples using rewards for dependence alignment, rubric-based task quality, and general response quality.The alignment reward is derived from δalign; RL inputs use preference-indicative queries generated with the SFT data pipeline.Source: §4.3 · p. 6

5 Experiments

The experiments assess whether SteeM can match users’ preferred memory dependence while preserving response quality and generalizing to unseen subjects. They compare it with a base model, rubric prompting, predefined tags, and memory masking.

5.1 Main Results

  • The main evaluation measures alignment with target memory dependence, response quality, and generalization to unseen subjects, comparing against the base model and rubric prompting.The test set comes from Section 3.1 and is augmented with preference-indicative queries.Source: §5.1 · p. 6

5.1.1 Steering Outputs Toward User-Preferred Memory Dependence

  • SteeM achieves substantially lower memory-dependence alignment error than the baseline across all scenarios and four shared tasks.Figure 4 shows the baseline concentrated at high realized dependence levels regardless of target, while SteeM shifts the distribution toward the target-level diagonal.Source: Table 1 · p. 7, §5.1.1 · p. 7, Figure 4 · p. 7
  • SteeM transfers preference-following behavior to unseen Medical and Humanities subjects, with the RL-enhanced variant generalizing more strongly than SFT alone.The reported generalization gap is larger than in Table 1.Source: Figure 5 · p. 7, §5.1.1 · p. 7

5.1.2 Preserving Response Quality

  • SteeM maintains response quality comparable to the baseline and sometimes scores slightly higher, with minimal impact on AlpacaEval performance.The response-quality evaluation uses Skywork-Reward-V2-Llama-3.1-8B (Liu et al., 2025a).Source: Figure 6 · p. 8, §5.1.2 · p. 8

5.2 Natural Expressions vs. Predefined Tags

  • Tag-conditioned training achieves slightly better alignment than SteeM but significantly degrades general performance on AlpacaEval.The variant replaces implicit preference cues with five predefined tags, from Minimal to Maximal.Source: §5.2 · p. 8

5.3 Comparison with Straightforward Binary Memory Masking

  • SteeM is competitive with memory masking and has a consistent win-rate advantage, while masking cannot reliably regulate reliance strength and may discard critical facts or constraints.Masking also places a heavy selection burden on users managing long, information-dense histories.Source: Figure 7 · p. 8, §5.3 · p. 8

5.4 Case Study

  • In the case study, SteeM better follows a low-memory request by proposing substantial departures from the historical project pipeline, whereas the baseline largely preserves it.SteeM’s examples include adaptive sampling and progress-triggered transitions.Source: §5.4 · p. 8

6 Conclusion

SteeM addresses memory anchoring by making historical-memory reliance user-controllable, achieving better preference alignment and transferring beyond controlled long-horizon interactions with minimal impact on general performance.

6 Conclusion

  • SteeM achieves substantially better preference alignment, transfers beyond controlled long-horizon interactions with minimal impact on general performance, and outperforms direct memory masking in pairwise comparisons.It is trained with preference-aligned SFT and RL.Source: §6 · p. 9
  • The paper identifies memory anchoring: models default to high memory reliance despite user intent in long-horizon interactions.The study examines how much an agent should rely on historical memory.Source: §6 · pp. 8–9
  • The authors build a realistic dataset simulating long-horizon interactions and present SteeM as an initial step toward practical, user-controllable memory reliance for personalized agents.Source: §6 · p. 9

Limitations

The testbed may differ from real human interactions, and its scope is limited by a coarse preference scale and evaluation in only two scenarios.

Limitations

  • The testbed aims to mimic realistic long-horizon projects but may still differ from real human interactions.The authors consider it a useful testbed for studying Memory Anchoring.Source: Limitations · p. 9
  • Memory-dependence preferences use a 1–5 ordinal scale, which may not capture users’ richer constraints; future work could use finer-grained or continuous preferences.Source: Limitations · p. 9
  • The setup covers only Research and Tutoring, so broader application settings and more diverse task distributions remain future directions.Source: Limitations · p. 9

A Dataset Details

The dataset models long-term research and tutoring projects through coherent event timelines, standardized queries, and query-specific memories. Its construction is designed to support comparable probes of memory dependence across project stages.

Scenarios and Topics

  • Each scenario contains 200 manually filtered topics, covering open-ended research or tutoring projects that evolve through their respective stages.Topics are proposed by Gemini-2.5-Pro; research spans planning, experimentation, analysis, and writing, while tutoring spans goal setting, lesson delivery, practice, and review.Source: A Dataset Details · p. 12

Events and Artifacts

  • Scenario-specific event types mark project milestones, while artifact types represent intermediate products that are iteratively updated throughout each project.Table 3 lists the event and artifact types used in the two scenarios.Source: A Dataset Details · p. 12

Iterative Timeline Synthesis

  • Gemini-2.5-Pro builds each project timeline iteratively, checking artifact prerequisites and global coherence before updating the artifact set or accepting the next event.Generation continues until a natural terminal state or a predefined maximum length; trajectories can include backtracking or refinement.Source: A Dataset Details · pp. 12–13

Tasks and Queries

  • Queries use four shared task categories and attach each task to an event and target artifact, enabling history-agnostic or history-dependent solutions to the same query form.Each query is generated as a natural user question issued immediately after its associated event completes.Source: A Dataset Details · p. 13

Query-Specific Memory Construction

  • Query-specific memory combines a long-term user profile, relevant cross-session or cross-topic interactions, and recent within-session history, summarized from timeline events and artifacts.The resulting memory and query form the retrieved context, allowing the amount of exposed history to vary.Source: A Dataset Details · p. 13

Dataset Statistics

  • The synthesis pipeline produces a diverse synthetic dataset, with subject and task distributions summarized in Figure 8.The dataset statistics are presented in Table 2 and Figure 8.Source: Figure 8 · p. 12, A Dataset Details · p. 13

B Training Details · C Response Quality

The appendix records the SFT and GRPO training setup and data counts, and notes the AlpacaEval reporting and reward model used for response-quality evaluation.

B Training Details

C Response Quality

D Case Study … G Human Annotation Protocol

The appendices illustrate SteeM’s low-memory steering in a case study and provide comparison details, the memory-dependence rubric, and the human annotation protocol.

D Case Study

  • In the PROJECT_METHOD refinement case, SteeM better matched the request for low memory dependence by adding substantial new ideas, while the baseline largely followed the historical pipeline.Examples of SteeM’s additions include adaptive sampling and progress-triggered transitions.Source: D Case Study · pp. 13–14, Table 5 · p. 14
  • The baseline’s minor add-ons followed the historical pipeline, illustrating memory anchoring despite the user’s low-memory instruction.SteeM’s more substantial departures reduced unintended memory-following.Source: D Case Study · p. 14

E Details for Natural Expression vs. Predefined-Tag Comparison

F Memory-Dependence Rubrics

  • The full rubric R for assigning integer memory-dependence MD-Scores is provided, including scale definitions and dimension-wise guidance in Table 7.Source: F Memory-Dependence Rubrics · p. 14

G Human Annotation Protocol

  • For human correlation analysis, annotators compared two candidate responses to the same query and memory and selected which relied more on the provided memory.The protocol covered 1,000 pairwise comparison instances; the exact prompt appears in Figure 11.Source: G Human Annotation Protocol · p. 14
  • The 1,000 instances were randomly split into 10 shards for 10 volunteer annotators, with 100 instances assigned to each.Estimated workload was about 45 seconds per judgment, or about 75 minutes per annotator; agreement and rank correlation with MD-Score are reported in Figure 3 (left).Source: G Human Annotation Protocol · p. 14

H Comparison with Memory Masking

This section provides materials for comparing SteeM with direct memory masking and for evaluating memory reliance, including prompts, a judging rubric, and an annotation protocol.

H Comparison with Memory Masking

Related papers

Loading 2601.05107v1…