PaperlayerView exact revision on arXiv

Source-linked AI summary

Memory-Aware Reward Modeling: Learning Human Preferences for LLM Memory Use

Yijiang River Dong, Zexue He, Cheng Yu, Tiancheng Hu, Michael J Ryan, Jan Batzner, Zheng Hui, Shaolong Wu, Orestis Papakyriakopoulos, Alex Pentland, Nigel Collier, Jiaxin Pei

arXiv:2610.31272v1cs.CLcs.AI

TL;DR

Memory benchmarks do not establish whether using available memories produces responses users prefer. The paper collects user-grounded judgments and trains a memory-aware reward model, achieving 72.8% accuracy on clear-preference comparisons with held-out participants.

  • Problem

    Memory evaluations primarily assess recall and recovery, leaving less clear whether using available memories produces responses users prefer.

    Source: §1 · p. 1

  • Method

    The paper collects pairwise judgments on responses to participants’ own queries and memories and trains a memory-aware reward model on rubric features.

    Source: §1 · p. 1, §7 · p. 8

  • Results

    72.8% accuracy on clear-preference comparisons with held-out participants, outperforming direct LLM judging and pretrained reward baselines.

    Source: §7 · p. 8

  • Takeaways & Limitations

    Memory-conditioned responses were not consistently preferred, and automatic evaluators favored them more often than users did.

    Source: §7 · p. 8

    The experiments evaluate memory-aware preference modeling offline; whether prediction improvements transfer to better memory-use behavior and user experience remains untested.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Memory enables language model assistants to maintain continuity across sessions and generate personalized responses, but knowing more about a user does not necessarily produce responses they prefer. Existing evaluations assess memory capabilities without directly establishing whether memory use benefits the user. To study this gap, we collect approximately 1,000 pairwise judgments from 87 participants using their own ChatGPT queries and personal memories. We find that memory-conditioned responses are not consistently preferred over responses generated without memory and in some cases, users strongly prefer the latter. Automatic evaluators also show limited agreement with users and favor memory-conditioned responses substantially more often than users do. Motivated by these gaps, we introduce a memory-aware reward model that learns from user judgments, achieving 72.8\% preference prediction accuracy and outperforming the existing baselines. Our findings motivate evaluation and optimization objectives that align memory use with the preferences of the people whose information is being used.

1 Introduction

Memory benchmarks measure recall and recovery, but do not establish whether using available memories produces responses users prefer. The paper studies this gap with judgments grounded in participants’ own queries and memories, and introduces a reward model trained on those preferences.

1 Introduction

  • Memory benchmarks assess compression and recovery, but accurate recall and preference inference do not ensure that using a memory is appropriate for the current request.Reusing a remembered preference can narrow an answer beyond what the current query warrants.Source: §1 · p. 1
  • Memory-conditioned responses do not consistently outperform responses without memory, and users sometimes strongly prefer answers generated without their personal context.The study collects approximately 1,000 response-pair judgments from 87 participants using their own queries and memories.Source: §1 · pp. 1–2
  • Automatic evaluators show limited agreement with users and select memory-conditioned responses substantially more often than users do.The evaluators’ prediction accuracy is close to the majority baseline.Source: §1 · p. 2
  • 72.8% accuracy, compared with 59.6% for a direct zero-shot LLM judge and 58.8% mean accuracy for pretrained rewards, using rubric-based reward modeling.The model learns preferences over response quality and the appropriateness of memory use from real users’ judgments.Source: §1 · p. 2
  • The paper contributes a dataset grounded in participants’ own queries and memories and a memory-aware preference model as an interpretable proxy for human preferences.The authors argue that human preference is a necessary reference for assessing memory’s value in personalized assistants.Source: §1 · p. 2

2 Related Work

Prior work develops long-term memory systems, personalization methods, and preference-based alignment, but leaves less clear how assistants should use available memories. This paper frames appropriate memory use as a user-preference learning problem.

Long-term LLM memory

  • Long-term memory benchmarks measure whether information can be retained, recovered, and updated, while the behavior expected once relevant memories are available is less clear.Prior evaluations include recall, multi-session reasoning, personalized responses, and tracking evolving states.Source: §2 · p. 3

Personalization and Over-personalization

  • Personalization methods condition generation on user profiles or past interactions, while studies of over-personalization identify unnecessary, distracting, sycophantic, or privacy-intrusive uses.The paper instead collects response-level preferences over controlled variations in memory-conditioned responses.Source: §2 · p. 3

RLHF and post-training

  • Preference-based alignment has expanded beyond single prompts, and this work extends it to cross-session memory with a rubric-based reward model conditioned on user memory.The model learns whether and how memory should influence the current response.Source: §2 · p. 3

3 Data Collection

The study collects preference judgments on paired responses to participants’ own queries and personal memories, then tests how consistently participants and models evaluate those responses.

3 Data Collection

  • 821 judgments from 87 participants remained after quality filtering, comprising 673 strict preferences and 148 ties.The original collection included 1,047 judgments, including 75 test–retest repeats excluded from training and evaluation.Source: §3 · p. 3

3.1 Personal Context Collection

  • Participants built editable memory profiles from their ChatGPT sessions and entered recent events, targeting 30 events.Profiles included personal attributes, preferences across about 10 topics, and up to 100 recent queries.Source: §3.1 · pp. 3–4

3.2 Response Pair Construction

  • For each query, the study retrieved nine memory entries and generated paired responses with the same model and query, differing in access to personal context.Memory-use prompts varied across conservative, ordinary, and aggressive regimes; candidate pairs were screened for substantial, memory-grounded differences.Source: §3.2 · p. 4

3.3 Response Preference Annotation

  • Participants judged side-by-side response pairs on a five-point preference scale without being instructed to favor greater personalization.The study excluded incomplete pairs, attention-check failures, and other abnormal annotation behavior.Source: §3.3 · p. 4

3.4 Annotator Consistency

  • 68.0% directional agreement was observed across 75 test–retest comparisons; agreement was higher for initially much preferences than slightly preferences, 73.9% versus 58.6%.Because slightly preferences were less consistent, testing used the much-better subset as a clearer preference signal.Source: §3.4 · p. 4, Table 1 · p. 4

4 Method

The method extends reward modeling from response quality to the appropriateness of using personal memory. It represents each memory/no-memory comparison with interpretable rubric features and learns how those features predict the user’s preference.

4.1 Preliminaries

  • Rubric-based reward models score explicit response criteria and aggregate evaluator judgments into a reward or preference.The method makes memory an explicit part of the comparison and models both response quality and memory-use appropriateness.Source: §4.1 · p. 5

4.2 Memory-Aware Preference Modeling

  • Each comparison is encoded with interpretable features that contrast the memory-conditioned and no-memory responses, including response-quality and memory-relevance scores.The feature entries are signed so larger values favor the memory-conditioned response.Source: §4.2 · p. 5
  • Rubric induction uses preference-labeled examples to propose and consolidate criteria, labeling each as memory-dependent or memory-agnostic.In one experiment, the resulting rubric had 12 features: three memory-related and nine response-quality features; induction and consolidation occur within each training fold.Source: §4.2 · p. 5
  • An L2-regularized logistic model learns feature weights to predict whether the annotator prefers the memory-conditioned response.Only the feature weights and intercept are learned; response generators and rubric evaluators remain fixed.Source: §4.2 · p. 5

5 Experimental Setup

The experimental setup evaluates the model using participant-separated cross-validation and compares it with feature-block ablations and automatic-evaluator baselines. It also reports selective prediction across several coverage levels.

5 Experimental Setup

  • The dataset retains 291 clear-preference pairs for rubric learning.The authors describe clear-preference pairs as providing a consistent preference signal.Source: §5 · p. 5
  • The evaluation uses five-fold cross-validation split by participant, repeated across ten random shuffles, with predictions averaged before accuracy is computed.Regularization strength and standardization are determined using training participants only.Source: §5 · p. 5
  • Feature-block ablations compare the full model with Memory only and Response only variants, while selective accuracy is reported at 25%, 50%, 75%, and 100% coverage.The ablations retain the same evaluator input; Table 6 specifies feature membership.Source: §5 · pp. 5–6
  • Table 2 reports annotator preferences by prompting regime and response generator, with percentages including ties.Generator comparisons are restricted to ordinary prompting to hold personalization instructions constant.Source: Table 2 · p. 6
  • Baselines include zero-shot and rubric-guided LLM judges, plus pretrained Skywork-Reward-V2 models scoring each response with the query and memories.LLM-judge results use either one presentation order or averaged A/B orders with response identities aligned.Source: §5 · p. 6

6 Results and Analysis

Across the screened comparisons, memory use did not consistently match users’ preferences, and existing evaluators often failed to recover their choices. A model learning to aggregate rubric features from user judgments outperformed the tested baselines, while its analyses showed that both memory appropriateness and response attributes matter.

6.1 User Preferences for Memory Use

  • Under ordinary prompting, users preferred memory-conditioned responses in 38.7% of comparisons and no-memory responses in 41.9%; preference for memory responses fell to 34.6% with aggressive prompting.Under aggressive prompting, users preferred no-memory responses in 49.3% of comparisons.Source: §6.1 · p. 6
  • Preference direction varied by generator: memory responses were preferred more often for GPT and Claude, while no-memory responses were preferred more often for Gemini.These findings concern screened comparisons prioritizing substantial, memory-grounded response differences.Source: §6.1 · p. 6

6.2 Automatic Evaluators Failed to Capture User Preferences

  • Direct GPT-5.2 judging reached 58.1% accuracy and Skywork with memories reached 52.9%, near or below the 55.0% majority baseline.Under two-order voting, the direct judge and Skywork selected memory-conditioned responses in 67.2% and 81.1% of comparisons, versus 45.0% for annotators.Source: §6.2 · pp. 6–7, Table 3 · p. 7
  • Rubric-based LLM judging reached 59.6% accuracy versus 58.1% for direct judging, a modest improvement that left substantial disagreement with users.The rubric highlighted appropriate memory use and non-use, but still required the evaluator to weigh competing criteria.Source: §6.2 · p. 7

6.3 Learning to Predict Response Preferences

  • 72.8% accuracy outperformed the majority baseline (55.0%), direct LLM judging (58.1%), and rubric-based LLM judging (59.6%).The model approached test–retest consistency of 73.9% and exceeded baselines across GPT-, Gemini-, and Claude-generated pairs.Source: §6.3 · p. 7
  • The model’s largest descriptive-fit coefficients favored more headings (+0.646), longer responses (+0.571), memory relevance (+0.386), and assumption proportionality (+0.323); unwarranted assumptions had a negative coefficient (−0.201).The five features were fit on 291 clear-preference pairs.Source: §6.3 · p. 7, Table 4 · p. 8
  • Combining memory-related and response-related features achieved 72.8% pooled accuracy, exceeding memory-only (62.5%) and response-only (67.0%) models.The full model achieved the highest accuracy for all three response generators.Source: §6.3 · pp. 7–8
  • At 75%, 50%, and 25% coverage, model accuracy rose from 72.8% to 76.2%, 83.7%, and 88.8%, respectively.The model ranks comparisons by confidence, allowing coverage to be traded for accuracy.Source: §6.3 · p. 8, Table 5 · p. 8

7 Conclusion

The study finds that memory-conditioned responses are not consistently preferred, and existing evaluators favor them more often than users do. A reward model learning from user judgments outperformed tested baselines, while extending this supervision to post-training remains future work.

7 Conclusion

  • In screened comparisons using participants’ own queries and memories, memory responses were not consistently preferred, and aggressive personalization received less favorable judgments.Existing evaluators favored memory-conditioned responses more often than users did.Source: §7 · p. 8
  • The memory-aware reward model achieved 72.8% accuracy on clear-preference comparisons with held-out participants, outperforming direct LLM judging and pretrained reward baselines.Combining memory-related criteria with general response attributes improved prediction over either feature group alone.Source: §7 · p. 8
  • The authors identify extending user-preference supervision to post-training as an important next step for assistants learning when remembered information improves responses.Source: §7 · p. 8

Limitations

The study’s findings are bounded by the user-context sources and single-turn setting represented in its data, and by offline evaluation of the reward model.

Limitations

  • The dataset uses profiles built from participants’ ChatGPT interactions and recent events, so findings do not yet cover the more diverse sources available to persistent assistants.Extending to those settings requires richer, multi-source, longitudinal collection with preferences grounded in that context.Source: Limitations · p. 8
  • The data and rubrics cover single-turn responses with retrieved memories supplied as context, not multi-turn conversations or agentic tasks.In those settings, appropriate memory use may depend on interaction trajectories, assistant actions, and changes in user goals or circumstances.Source: Limitations · pp. 8–9
  • The experiments evaluate memory-aware preference modeling offline, leaving its effects on post-training and user experience untested.The authors identify testing whether preference-prediction improvements translate into better memory-use behavior as a next step.Source: Limitations · p. 9

Ethical Considerations

The study describes participant safeguards and the collection and release procedures used to gather personal memories and response preferences. It emphasizes that predicted preference is not consent to handle personal information.

Ethical Considerations

  • Participants consented, could edit memory profiles, and were instructed to exclude sensitive information and direct identifiers; preference predictions are not consent to collect, retain, or disclose personal information.The study received ethics committee approval, and full conversation histories and attachments were not uploaded to the platform.Source: Ethical Considerations · p. 9

A Annotation Protocol and Participant Materials

  • The appendix provides collection interfaces and response-preference instructions to make the procedure inspectable, while noting that interface details varied across revisions.The screenshots use an example profile separate from participant records.Source: A Annotation Protocol and Participant… · p. 12

A.1 Profile Collection Interface

  • Step 1 asks participants to run an extraction prompt in their own account and inspect the returned profile before event collection.The displayed profile is illustrative.Source: §A.1 · p. 12, Figure 2 · p. 12

A.2 Event Collection Interface

  • Step 2 asks participants for specific events from the preceding month and supports batch recording, typing, and voice dictation toward a 30-event target.The interface provides examples of the required specificity and displays collection progress.Source: §A.2 · p. 13, Figure 3 · p. 13

A.3 Response Preference Instructions

  • Participants compare two responses that differ in memory use and choose which they would prefer as a user, with five options from much better to about the same.Response labels do not reveal the generation condition, and participants may add an optional explanation.Source: §A.3 · p. 14

A.4 PII Removal and De-identification Protocol

  • The release protocol combines identifier replacement and detail generalization with contextual human review across records, withholding comparisons if redaction changes the basis of preference.The protocol addresses combinations of details and third parties, but de-identification does not establish that personal narratives are anonymous.Source: §A.4 · p. 14

B.1 Preference for the response generated with memory

The appendix presents examples of preferences in both directions and describes the feature set used in a descriptive regression fit. The examples show that memory use is not uniformly preferred.

B.1 Preference for the response generated with memory

  • One participant slightly preferred the memory-conditioned response, which used viewing memories to suggest spreading the films across several evenings.The no-memory response instead asked for the titles and runtimes before giving a specific plan.Source: §B.1 · p. 15

B.2 Preference for the response generated without memory

  • For a presentation-planning query, the participant judged the response without retrieved memory much better than the noon-focused memory-conditioned schedule.The no-memory response proposed work blocks and a final pass; the memory-conditioned response scheduled the main block around the user’s reported noon productivity.Source: §B.2 · p. 16

C Rubric Regression Details

  • Table 6 lists twelve model features ordered by absolute standardized L2 coefficient in a descriptive fit to 291 clear-preference pairs, with C = 0.1.Positive coefficients favor the memory-conditioned response; the reported coefficients are from the full-data fit, not averages of nested-validation fits.Source: Table 6 · p. 17

D.1 Direct LLM Judging

Direct GPT-5.2 judging is assessed against user choices using paired response comparisons and order-aware voting. The judge favored memory-conditioned responses more often than annotators did.

D.1 Direct LLM Judging

  • The evaluation combines confidence-weighted predictions after mapping both orders to response identities and separately reports cases where both orders agree.Table 7 reports direct GPT-5.2 judging on the clear-preference stratum, with majority rates computed separately for each evaluation subset.Source: §D.1 · p. 18, Table 7 · p. 18
  • 22.2 percentage points separated the judge’s memory-conditioned choice rate from annotators’ (67.2% versus 45.0%), despite 77.7% agreement across A/B orders.Confidence correlated only .180 with correctness and .064 with selecting the memory-conditioned response.Source: §D.1 · p. 18
  • The rubric judge’s zero-shot GPT-5.2 prompt discouraged preferences based only on length or personal references and favored appropriate non-use over forced memory use.It scored both A/B orders for 291 pooled clear-preference comparisons; the direct prompt supplied no rubric to counter length or personalization preferences.Source: §D.1 · p. 18

D.2 Pretrained Skywork Rewards

This section examines pretrained Skywork reward-model scoring with and without supplied memories and documents the prompts used to generate responses under two personalization approaches. Ordinary instructions condition memory use on relevance and helpfulness, while aggressive instructions encourage proactive personalization and stylistic adaptation when memories do not directly apply; the model favored memory-conditioned responses more often when memories were supplied but scored higher overall without them.

D.2 Pretrained Skywork Rewards

  • 52.9% accuracy with memories was below the 55.0% majority baseline, while removing memories raised accuracy to 58.8%.With memories, accuracy was 63.6% on GPT-, 37.5% on Gemini-, and 73.3% on Claude-generated pairs; without memories, it was 55.5%, 66.9%, and 42.2%, respectively. The model selected memory-conditioned responses in 81.1% of comparisons where annotators selected them in 45.0%; without memories in its input, it did so in 19.6%.Source: §D.2 · p. 19

E.1 Complete Persona Extraction Prompt

  • Persona extraction prompts request privacy-protected, structured profiles and recent queries, while limiting extraction to accessible history and forbidding fabricated content.The prompt requests 30 recent first queries, but sessions depending on unavailable attachments or prior context are skipped, and fewer queries are allowed when history is insufficient. Outputs include separate profile and recent-activity memories plus a structured tag summary.Source: §E.1 · pp. 20, 22, 23

E.2 Response Generation Prompts

  • Response-generation templates define conservative, ordinary, and aggressive personalization, with {episode_block} filled by the memory context supplied to the generator.The templates retain the heading “top 20 by similarity,” although the audited snapshot’s full retrieved sets contain nine entries.Source: §E.2 · p. 24

E.2.1 Conservative Personalization

  • Conservative personalization uses memory only when it would meaningfully improve the specific answer and rejects unrelated personal context.For objective or factual queries, its instructions prohibit injecting the user’s personal life and prefer a clean generic answer over force-fitting context.Source: §E.2.1 · p. 24

E.2.2 Ordinary Personalization

  • Ordinary personalization instructs the assistant to use the supplied memories to personalize answers where relevant.Source: §E.2.2 · p. 24
  • Ordinary personalization advises naturally incorporating clearly relevant memories, personalizing factual answers’ framing or examples, and avoiding unrelated or verbatim memory use.The instructions leave judgment to the assistant about what actually helps and favor coherent integration over listing memories.Source: §E.2.2 · p. 24

E.2.3 Aggressive Personalization

  • Aggressive personalization directs the assistant to connect queries to the user’s context, explicitly reference relevant memories, and tailor even general answers.It frames personalization as the primary goal and urges proactive use of the supplied memories.Source: §E.2.3 · p. 25
  • When memories do not directly apply, aggressive instructions call for matching the user’s communication style and omitting only completely unrelated memories that would be harmful to force in.The prompt also recommends adapting tone and level of detail to the known profile.Source: §E.2.3 · p. 25

Related papers

Loading 2610.31272v1…