Source-linked AI summary
Learning Personalized Agents from Human Feedback
Kaiqu Liang, Julia Kruk, Shengyi Qian, Xianjun Yang, Shengjie Bi, Yuanshun Yao, Shaoliang Nie, Mingyang Zhang, Lijuan Liu, Jaime Fernández Fisac, Shuyan Zhou, Saghar Hosseini
TL;DR
PAHF tackles the difficulty of personalizing agents for new users and changing preferences, where prior approaches rely on static data. It uses explicit per-user memory with pre-action clarification and post-action feedback in a continual loop, evaluated in embodied manipulation and online shopping. Across the evaluation, PAHF consistently outperforms single-channel baselines while reducing initial personalization error and adapting to preference shifts.
Problem
Existing personalization approaches rely on static histories or profiles, limiting support for new users and changing preferences.
Method
PAHF uses explicit per-user memory updated through pre-action clarification, preference-grounded action, and post-action corrective feedback.
Results
PAHF consistently outperforms single-channel baselines across embodied manipulation and online shopping evaluations.
Takeaways & Limitations
Combining proactive clarification with reactive correction addresses complementary failure modes in continual personalization.
Takeaways & Limitations
Relying on post-action feedback alone forces learning through trial and error, causing larger initial personalization errors and higher user costs.
Abstract
from arXiv · showhide
Modern AI agents are powerful but often fail to align with the idiosyncratic, evolving preferences of individual users. Prior approaches typically rely on static datasets, either training implicit preference models on interaction history or encoding user profiles in external memory. However, these approaches struggle with new users and with preferences that change over time. We introduce Personalized Agents from Human Feedback (PAHF), a framework for continual personalization in which agents learn online from live interaction using explicit per-user memory. PAHF operationalizes a three-step loop: (1) seeking pre-action clarification to resolve ambiguity, (2) grounding actions in preferences retrieved from memory, and (3) integrating post-action feedback to update memory when preferences drift. To evaluate this capability, we develop a four-phase protocol and two benchmarks in embodied manipulation and online shopping. These benchmarks quantify an agent's ability to learn initial preferences from scratch and subsequently adapt to persona shifts. Our theoretical analysis and empirical results show that integrating explicit memory with dual feedback channels is critical: PAHF learns substantially faster and consistently outperforms both no-memory and single-channel baselines, reducing initial personalization error and enabling rapid adaptation to preference shifts.
1 Introduction
PAHF addresses the difficulty of aligning agents with complex, changing individual preferences by learning online from interaction. It combines explicit per-user memory with pre-action clarification and post-action feedback, evaluated across two benchmarks and four phases.
- Agents still struggle to align actions with individual preferences that are complex, idiosyncratic, and dynamic.
- Static personalization cannot immediately support new users or adapt reliably to real-time changes in human behavior.
- PAHF learns continually through explicit per-user memory updated from online interaction and human feedback.
- Its three-step loop queries memory and seeks clarification before ambiguous actions, grounds execution in retrieved preferences, then integrates corrective feedback afterward.
- The evaluation suite includes embodied manipulation and online shopping benchmarks with a four-phase protocol separating initial learning from adaptation under persona shifts.
2 Related Work
Related work spans preference alignment, memory-based personalization, and interactive-environment agents. PAHF is positioned against static or increasingly complex personalization approaches while evaluating continual feedback-driven adaptation.
- Prior alignment work includes reinforcement learning from human feedback and supervised fine-tuning on preference data without explicit reinforcement learning.
- Figure 2 illustrates continual personalization through pre-action learning, contextual correction, and post-action updating after preference drift.
- Personalized LLM agents commonly retrieve user histories into context and increasingly use reasoning or operation-selection procedures during interaction.
- Recent agent-memory methods emphasize long-term state tracking, summarization, and retrieval, whereas PAHF uses lightweight memory to isolate feedback-channel effects.
- Interactive-environment personalization has been studied for spatial arrangement, table settings, object navigation, planning, and flight recommendation.
3 Personalized Agents from Human Feedback
PAHF frames personalization as online learning over explicit per-user preference memory, addressing both incomplete knowledge and changing preferences. Its three-step loop combines clarification, memory-grounded action, and post-action updating, with theory showing complementary feedback channels reduce personalization error.
- Formalizing Continual Personalization: Partial observability creates known-uncertainty errors, whereas non-stationary preferences create miscalibration when stored preferences become outdated.The framework distinguishes incomplete memory for new or unfamiliar tasks from preference drift that can make an agent confidently wrong.
- Formalizing Continual Personalization: PAHF models continual personalization as learning a policy from interaction while maintaining an explicit, persistent estimate of each user’s preferences.The policy conditions actions on instructions, observations, and retrieved memory, aiming to minimize cumulative personalization error.
- PAHF Framework: The PAHF loop retrieves relevant memory, asks clarification questions when ambiguity remains, grounds the action in available context, and writes pre-action information before execution.This sequence resolves known uncertainty and preserves newly learned preferences for future rounds.
- PAHF Framework: Post-action feedback detects incorrect outcomes and updates memory to correct confidently wrong preferences, including revising one stored preference for another.The paper describes this reactive channel as uniquely necessary for correcting miscalibration caused by preference drift.
- Theoretical Justification for PAHF: Without post-action feedback, preference switches can cause Ω(T) expected mistakes, whereas updating after the first error at each switch limits mistakes to O(K).The result applies under unambiguous rounds with at most K preference switches.
- Theoretical Justification for PAHF: With up to k balanced m-ary pre-action questions on ambiguous rounds, expected errors on those rounds are O(γT · m^-k), and k = Θ(log_m T) gives E[R_T] = O(K + γ).The bound combines switch-related mistakes with residual ambiguity after questioning.
4 Evaluations
The evaluation tests continual personalization across embodied manipulation and online shopping using a four-phase protocol. It separates learning and testing of initial preferences from adaptation and testing after persona shifts, with metrics tracking success, feedback use, and cumulative personalization error.
- Evaluation Domains: The evaluation spans embodied manipulation and online shopping to test personalization across physical and digital agents.The embodied benchmark uses context-dependent selection and placement tasks, while shopping requires choosing among three products or abstaining under persona-specific acceptance policies.
- Embodied Manipulation Domain: Embodied scenarios use idiosyncratic, context-dependent personas, preventing correct actions from being inferred reliably through generic commonsense heuristics.Persona feedback arrives through clarification answers and post-action responses, and preferences can evolve across persona variants.
- Four-Phase Evaluation Protocol: The four-phase protocol first trains and tests initial preferences, then trains and tests adaptation after each user’s persona is replaced by a new one.The initial test has no human feedback, while the adaptation phases assess whether memory is revised and applied to new scenarios.
- Four-Phase Evaluation Protocol: The protocol separately quantifies initial learning and adaptation to preference drift through two interactive learning phases and two test phases.This separation is intended to provide a comprehensive view of continual personalization.
- Metrics: Success Rate measures correctly completed tasks, Feedback Frequency measures tasks using any human feedback, and ACPE tracks cumulative personalization error over learning iterations.ACPE ranges from 0 to 1, interpretable as 0% best and 100% worst.
5 Results
Across embodied manipulation and online shopping, the results show complementary roles for pre-action clarification and post-action feedback. PAHF combines both channels with explicit memory to reduce initial errors, adapt after preference drift, and achieve the strongest overall personalization.
- Phase 1: Initial learning: Pre-action clarification gives PAHF and Pre-action Only a higher first-interaction success rate and lower early ACPE than reactive or no-memory baselines.This warm start prevents initial personalization errors under partial observability.
- Adaptation under preference drift: Pre-action Only adapts poorly after preference drift, with limited Phase-3 success improvement, persistently higher ACPE, and infrequent clarification requests.Confident but outdated memory suppresses the ambiguity signal that would trigger new questions.
- Adaptation under preference drift: Post-action feedback produces steep success-rate improvements and sharp ACPE reductions across both domains during initial learning and preference shifts.Reactive updates help overwrite stale beliefs after mistakes.
- Comparing feedback channels: Post-action Only reaches 67.9% Phase-4 success in embodied manipulation versus 70.5% for PAHF, but incurs larger initial errors because it learns only after mistakes.Its reactive mechanism approaches PAHF after drift while imposing higher early user costs.
- PAHF: PAHF generally surpasses other methods in Phase 1, nearly matches Post-action Only after drift, and achieves the lowest ACPE across both domains.The learning curves cover success rate, feedback frequency, and ACPE in Phases 1 and 3 for embodied manipulation and online shopping.
6 Conclusion
PAHF addresses personalization under evolving user preferences by learning online with explicit memory and dual feedback channels. Across embodied manipulation and online shopping, it outperforms single-channel baselines, with pre-action feedback reducing early errors and post-action feedback enabling recovery after persona shifts.
- Conclusion: PAHF enables continual personalization through explicit memory, proactive pre-action queries, and reactive post-action corrections.The framework targets agents that must align with unique and evolving preferences rather than static user data.
- Conclusion: Across embodied manipulation and online shopping, PAHF consistently outperforms single-channel baselines under evolving user preferences.The reported results attribute early-error reduction to pre-action feedback and post-shift recovery to post-action feedback.
A Full Proofs for Section 3
The formal analysis models piecewise-stationary preferences, ambiguous rounds, and two feedback channels. Its lemmas show how balanced pre-queries shrink action error and why omitting post-action updates makes preference shifts indistinguishable in a no-ambiguity setting.
- Setting and assumptions: Preferences are piecewise stationary with at most K switches over T rounds, and ambiguity is defined by a Bayes-optimal action error threshold ε0.γ denotes the fraction of ambiguous rounds.
- Feedback model: On ambiguous rounds, pre-action feedback permits up to k truthful m-ary clarification questions, while post-action feedback supplies corrective updates after mistakes.The framework distinguishes proactive queries from reactive memory updates.
- Lemma 1: Under balanced m-ary questioning, k pre-queries reduce the posterior probability of a suboptimal induced action to at most m^-k.Each answer shrinks wrong-action posterior mass by a factor of at least m.
- Lemma 2: With no ambiguous rounds and no post-action updates, environments differing only in switch times induce identical internal-state and action-sequence distributions.The policy therefore cannot use its observations to distinguish when the preference switch occurred.
Proof of Proposition 1
Without post-action feedback, a policy cannot reliably detect a preference switch and suffers linear expected loss, whereas reactive updates limit mistakes to one per switch.
- Any policy that never uses post-action feedback incurs an expected loss lower bound of ⌊T^2/4⌋/T under a uniformly random switch time.
- A single hidden preference switch is sufficient to establish the lower bound for policies that never use post-action feedback.The proof fixes K = 1 and chooses the switch time uniformly, making the observation process independent of the switch time.
- For any fixed action sequence, choosing the wrong action before or after the switch contributes respectively the number of rounds on the opposite side of the switch.The two contributions are t for choosing action 0 after the switch and T −t for choosing action 1 before it.
- A reactive policy that updates after the first error following each switch makes at most one mistake per switch.After feedback identifies the new optimal action, the policy acts correctly on subsequent unambiguous rounds until the next switch.
Proof of Proposition 2
Under partial observability, pre-action clarification is necessary to avoid errors on ambiguous rounds, and repeated balanced questions reduce those errors exponentially in the number of questions.
- Without pre-action clarification, any policy incurs Ω(γT) errors when a fraction γ of rounds are ambiguous.Even the Bayes-optimal decision has a positive constant error probability on each unresolved ambiguous round.
- With up to k balanced m-ary pre-action questions per ambiguous round, the expected errors on those rounds are O(γT · m^-k).The posterior probability that the induced action is wrong is at most m^-k after the questions.
Proof of Theorem 1
The theorem combines reactive post-action updates for preference switches with pre-action clarification for ambiguity, yielding an expected mistake bound that scales with both sources of difficulty.
- On unambiguous rounds, immediate post-action updating after the first error following each switch yields O(K) mistakes.
- On ambiguous rounds, up to k balanced m-ary pre-queries contribute O(γT m^-k) mistakes.
- Combining both channels gives E[R_T] = O(K + γT m^-k), and choosing k = ⌈log_m T⌉ yields E[R_T] = O(K + γ).
B Limitations and Future work
The framework leaves memory architecture, feedback noise, agent reasoning ability, and interaction limits as important boundaries for future work.
- The study isolates feedback mechanisms rather than introducing novel memory designs, leaving scalability and context retention to future memory backends.
- Human feedback may be noisy or heterogeneous, and the framework does not explicitly handle inconsistent or mistaken feedback.
- The benchmarks remain challenging, particularly online shopping, where agents still struggle to achieve high success rates.
- The framework limits interaction for disambiguation to reflect user friction in real deployments.
C Details on Dataset
The benchmarks model personalization through conditional, evolving user preferences across embodied manipulation and online shopping. Their four-phase scenarios test initial learning, memory-based evaluation, and adaptation to changed preferences.
- Embodied Manipulation: The embodied benchmark uses 40 personas whose preferences depend on nine contextual categories rather than static user-item mappings.Categories include drinks, snacks, storage, location, temperature, environmental approach, health, social context, and time of day.
- Preference Drift: The Evolved embodied personas invert the original belief system, forcing agents to unlearn outdated correlations through feedback.Examples include shifting from herbal tea to energy drinks when drowsy and from high shelves to low shelves.
- Embodied Manipulation: Each embodied scenario combines objects, an ambiguous instruction, a context tag, and ground-truth intent derived from persona rules.The setup supports tasks involving selecting items or placing them in locations.
- Online Shopping: The shopping benchmark defines ten product categories with three feature dimensions each and ranks attribute values as Preferred, Acceptable, or Disliked.A product is correct when it maximizes overlap with preferred and acceptable attributes while avoiding disliked features.
- Preference Drift: Shopping preference drift re-samples feature attributes in Evolved phases, so previously preferred values can become disliked.The four-phase structure uses Original preferences in Phases 1–2 and Evolved preferences in Phases 3–4; Phase 3 preserves scenario structure while changing labels.
- Agent Protocol: Agent prompts require memory search for personalization, clarification for ambiguity, and a fixed Thought/Action response format.Actions are option letters or “Ask human,” with clarification required for subjective or underspecified requests.
E.1 Qualitative Results
The qualitative examples show how agents use clarification and memory to learn preferences, execute matching actions, and respond to preference changes. Single feedback channels expose complementary weaknesses: pre-action-only behavior can remain stale after drift, while post-action-only behavior can make an initial wrong guess.
- Preference Drift: After the user states that herbal tea is no longer preferred and coffee is preferred, the interaction supplies explicit evidence of preference drift.The updated preference is presented as a change from the previously stated preference.
- Single-Channel Baselines: The pre-action-only baseline is described as brittle under preference drift because prior preference information can remain unchanged after the user switches.The example shows a stored herbal-tea preference followed by a user declaration that coffee is now preferred.
- PAHF: When memory lacks a relevant preference, the agent asks the user to clarify before selecting among ambiguous drinks.After the user identifies herbal tea, the agent selects it and can use the clarified preference in later evaluation.
- PAHF: With a stored preference, the agent grounds an ambiguous request in memory and selects the corresponding option without new feedback.The examples select herbal tea when it is recorded as Avery’s preferred or favorite drink.
- Single-Channel Baselines: The post-action-only baseline can incur large initial personalization error by guessing coffee when memory contains no preference and correcting only after rejection.The user then reports herbal tea as the preferred drink, providing feedback after the incorrect action.
- Contextual Preferences: Context-dependent preferences require the agent to distinguish a general favorite from a condition-specific exception such as iced tea when thirsty.Without evidence of thirst, the qualitative example selects the general favorite, herbal tea.